Merge pull request #8546 from blackpiglet/skip_deprecation_message

Skip the deprecation message for the dry-run install CLI JSON output.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -5,7 +5,7 @@ about: Tell us about a problem you are experiencing
 ---
 
 **What steps did you take and what happened:**
-[A clear and concise description of what the bug is, and what commands you ran.)
+<!--A clear and concise description of what the bug is, and what commands you ran.-->
 
 
 **What did you expect to happen:**
@@ -25,7 +25,7 @@ Please provide the output of the following commands (Pasting long output into a
 
 
 **Anything else you would like to add:**
-[Miscellaneous information that will assist in solving the issue.]
+<!--Miscellaneous information that will assist in solving the issue.-->
 
 
 **Environment:**

.github/ISSUE_TEMPLATE/feature-enhancement-request.md

@@ -5,15 +5,15 @@ about: Suggest an idea for this project
 ---
 
 **Describe the problem/challenge you have**
-[A description of the current limitation/problem/challenge that you are experiencing.]
+<!--A description of the current limitation/problem/challenge that you are experiencing.-->
 
 
 **Describe the solution you'd like**
-[A clear and concise description of what you want to happen.]
+<!--A clear and concise description of what you want to happen.-->
 
 
 **Anything else you would like to add:**
-[Miscellaneous information that will assist in solving the issue.]
+<!--Miscellaneous information that will assist in solving the issue.-->
 
 
 **Environment:**

.github/auto-assignees.yml

@@ -9,16 +9,19 @@ reviewers:
 
   groups:
     maintainers:
-      - dsu-igeek
       - sseago
       - reasonerjt
       - ywk253100
       - blackpiglet
-      - qiuming-best
       - shubham-pampattiwar
+      - Lyndon-Li
+      - anshulahuja98
 
     tech-writer:
-      - a-mccarthy
+      - sseago
+      - reasonerjt
+      - ywk253100
+      - Lyndon-Li
 
 files:
   'site/**':

.github/dependabot.yml

@@ -0,0 +1,21 @@
+version: 2
+updates:
+  # Dependencies listed in .github/workflows
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    labels:
+      - "Dependencies"
+      - "github_actions"
+      - "kind/changelog-not-required"
+  # Dependencies listed in go.mod
+  - package-ecosystem: "gomod"
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "weekly"
+    labels:
+      - "kind/changelog-not-required"
+    ignore:
+      - dependency-name: "*"
+        update-types: ["version-update:semver-major", "version-update:semver-minor", "version-update:semver-patch"]

.github/labeler.yml

@@ -0,0 +1,33 @@
+# This file is used by Auto Label PRs action.
+# Works with https://github.com/actions/labeler/
+# Below this line, the keys are labels to be applied, and the values are the file globs to match against.
+# Anything in the `design` directory gets the `Design` label.
+Area/Design:
+  - changed-files:
+      - any-glob-to-any-file: design/*
+# Anything that has plugin infra will be labeled.
+# Individual plugins don't necessarily live here, though
+Area/Plugins:
+  - changed-files:
+      - any-glob-to-any-file: pkg/plugins/**/*
+Dependencies:
+  - changed-files:
+      - any-glob-to-any-file: go.mod
+Documentation:
+  - changed-files:
+      - any-glob-to-any-file: site/content/docs/**/*
+# Anything in the site directory gets the website label *EXCEPT* docs
+Website:
+  - all:
+      - changed-files:
+          - any-glob-to-any-file: site/**/*
+          - all-globs-to-all-files: '!site/content/docs/**/*'
+has-changelog:
+  - changed-files:
+      - any-glob-to-any-file: changelogs/**
+has-e2e-2tests:
+  - changed-files:
+      - any-glob-to-any-file: test/e2e/**/*
+has-unit-tests:
+  - changed-files:
+      - any-glob-to-any-file: pkg/**/*_test.go

.github/labels.yaml

@@ -0,0 +1,43 @@
+# This file is used by [prow github action](https://github.com/jpmcb/prow-github-actions/) in .github/workflows/prow-action.yml.
+# This file only has values for kind and area commands.
+area:
+  - CLI
+  - CSI
+  - Cloud/AWS
+  - Cloud/Azure
+  - Cloud/DigitalOcean
+  - Cloud/GCP
+  - Cloud/vSphere
+  - Design
+  - Documentation
+  - Filters
+  - Plugins
+  - Process
+  - Storage/Minio
+  - Storage/Cinder
+  - WindowsSupport
+  - datamover
+  - fs-backup
+  - fs-backup/deletion
+  - fs-backup/file-selectable
+  - fs-uploader
+  - kopia-integration
+  - migration
+  - multi-tenancy
+  - progress-monitoring
+  - resilience
+  - schedule
+  - storage/IBM-ObjectStorage
+  - upgrade
+  - volume-snapshot-dm
+kind:
+  - changelog-not-required
+  - question
+  - refactor
+  - requirement
+  - release-note
+  - release-blocker
+  - spike
+  - tech-debt
+  - usage-error
+  - voting

.github/labels.yml

@@ -1,41 +0,0 @@
-area:
-  - "Cloud/AWS"
-  - "Cloud/GCP"
-  - "Cloud/Azure"
-  - "Design"
-  - "Plugins"
-
-# Labels that can be applied to PRs with the /kind command
-kind:
-  - "changelog-not-required"
-  - "tech-debt"
-
-# Works with https://github.com/actions/labeler/
-# Below this line, the keys are labels to be applied, and the values are the file globs to match against.
-# Anything in the `design` directory gets the `Design` label.
-Area/Design:
-  - design/*
-
-# Anything in the site directory gets the website label *EXCEPT* docs
-Website:
-  - any: ["site/**/*", "!site/content/docs/**/*"]
-
-Documentation:
-  - site/content/docs/**/*
-
-Dependencies:
-  - go.mod
-
-# Anything that has plugin infra will be labeled.
-# Individual plugins don't necessarily live here, though
-Area/Plugins:
-  - "pkg/plugins/**/*"
-
-has-unit-tests:
-  - "pkg/**/*_test.go"
-
-has-e2e-2tests:
-  - "test/e2e/**/*"
-
-has-changelog:
-  - "changelogs/**"

.github/pull_request_template.md

@@ -9,5 +9,5 @@ Fixes #(issue)
 # Please indicate you've done the following:
 
 - [ ] [Accepted the DCO](https://velero.io/docs/v1.5/code-standards/#dco-sign-off). Commits without the DCO will delay acceptance.
-- [ ] [Created a changelog file](https://velero.io/docs/v1.5/code-standards/#adding-a-changelog) or added `/kind changelog-not-required` as a comment on this pull request.
+- [ ] [Created a changelog file (`make new-changelog`)](https://velero.io/docs/main/code-standards/#adding-a-changelog) or comment `/kind changelog-not-required` on this PR.
 - [ ] Updated the corresponding documentation in `site/content/docs/main`.

.github/stale.yml

@@ -1,44 +0,0 @@
-# Number of days of inactivity before an issue becomes stale
-daysUntilStale: 60
-# Number of days of inactivity before a stale issue is closed
-daysUntilClose: 14
-# Issues with these labels will never be considered stale
-exemptLabels:
-  - Epic
-  - Area/CLI
-  - Area/Cloud/AWS
-  - Area/Cloud/Azure
-  - Area/Cloud/GCP
-  - Area/Cloud/vSphere
-  - Area/CSI
-  - Area/Design
-  - Area/Documentation
-  - Area/Plugins
-  - Bug
-  - Enhancement/User
-  - kind/requirement
-  - kind/refactor
-  - kind/tech-debt
-  - limitation
-  - Needs investigation
-  - Needs triage
-  - Needs Product
-  - P0 - Hair on fire
-  - P1 - Important
-  - P2 - Long-term important
-  - P3 - Wouldn't it be nice if...
-  - Product Requirements
-  - Restic - GA
-  - Restic
-  - release-blocker
-  - Security
-# Label to use when marking an issue as stale
-staleLabel: staled
-# Comment to post when marking an issue as stale. Set to `false` to disable
-markComment: >
-  This issue has been automatically marked as stale because it has not had
-  recent activity. It will be closed if no further activity occurs. Thank you
-  for your contributions.
-# Comment to post when closing a stale issue. Set to `false` to disable
-closeComment: >
-  Closing the stale issue.

.github/workflows/auto_assign_prs.yml

@@ -13,7 +13,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Set the author of a PR as the assignee
-        uses: kentaro-m/auto-assign-action@v1.1.1
+        uses: kentaro-m/auto-assign-action@v2.0.0
         with:
           configuration-path: ".github/auto-assignees.yml"
           repo-token: "${{ secrets.GITHUB_TOKEN }}"

.github/workflows/auto_label_prs.yml

@@ -13,7 +13,7 @@ jobs:
   triage:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/labeler@v3
+      - uses: actions/labeler@v5
         with:
           repo-token: "${{ secrets.GITHUB_TOKEN }}"
-          configuration-path: .github/labels.yml
+          configuration-path: .github/labeler.yml

.github/workflows/auto_request_review.yml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Request a PR review based on files types/paths, and/or groups the author belongs to
-        uses: necojackarc/auto-request-review@v0.7.0
+        uses: necojackarc/auto-request-review@v0.13.0
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
           config: .github/auto-assignees.yml

.github/workflows/crds-verify-kind.yaml

@@ -11,15 +11,16 @@ jobs:
   build-cli:
     runs-on: ubuntu-latest
     steps:
+      - name: Check out the code
+        uses: actions/checkout@v4
       - name: Set up Go
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v5
         with:
-          go-version: 1.17
-        id: go
+          go-version-file: 'go.mod'
       # Look for a CLI that's made for this PR
       - name: Fetch built CLI
         id: cache
-        uses: actions/cache@v2
+        uses: actions/cache@v4
         env:
           cache-name: cache-velero-cli
         with:
@@ -29,27 +30,12 @@ jobs:
           # This key controls the prefixes that we'll look at in the cache to restore from
           restore-keys: |
             velero-${{ github.event.pull_request.number }}-
-
-      - name: Fetch cached go modules
-        uses: actions/cache@v2
-        if: steps.cache.outputs.cache-hit != 'true'
-        with:
-          path: ~/go/pkg/mod
-          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
-          restore-keys: |
-            ${{ runner.os }}-go-
-
-      - name: Check out the code
-        uses: actions/checkout@v2
-        if: steps.cache.outputs.cache-hit != 'true'
-
       # If no binaries were built for this PR, build it now.
       - name: Build Velero CLI
         if: steps.cache.outputs.cache-hit != 'true'
         run: |
           make local
-
-  # Check the common CLI against all kubernetes versions
+  # Check the common CLI against all Kubernetes versions
   crd-check:
     needs: build-cli
     runs-on: ubuntu-latest
@@ -57,19 +43,19 @@ jobs:
       matrix:
         # Latest k8s versions. There's no series-based tag, nor is there a latest tag.
         k8s:
-          - 1.16.15
-          - 1.17.17
-          - 1.18.15
-          - 1.19.7
-          - 1.20.2
-          - 1.21.1
-          - 1.22.0
+          - 1.23.17
+          - 1.24.17
+          - 1.25.16
+          - 1.26.13
+          - 1.27.10
+          - 1.28.6
+          - 1.29.1
     # All steps run in parallel unless otherwise specified.
     # See https://docs.github.com/en/actions/learn-github-actions/managing-complex-workflows#creating-dependent-jobs
     steps:
       - name: Fetch built CLI
         id: cache
-        uses: actions/cache@v2
+        uses: actions/cache@v4
         env:
           cache-name: cache-velero-cli
         with:
@@ -81,7 +67,7 @@ jobs:
             velero-${{ github.event.pull_request.number }}-
       - uses: engineerd/setup-kind@v0.5.0
         with:
-          version: "v0.11.1"
+          version: "v0.21.0"
           image: "kindest/node:v${{ matrix.k8s }}"
       - name: Install CRDs
         run: |

.github/workflows/e2e-test-kind.yaml

@@ -11,37 +11,27 @@ jobs:
   build:
     runs-on: ubuntu-latest
     steps:
+      - name: Check out the code
+        uses: actions/checkout@v4
       - name: Set up Go
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v5
         with:
-          go-version: 1.17
-        id: go
+          go-version-file: 'go.mod'
       # Look for a CLI that's made for this PR
       - name: Fetch built CLI
         id: cli-cache
-        uses: actions/cache@v2
+        uses: actions/cache@v4
         with:
           path: ./_output/bin/linux/amd64/velero
           # The cache key a combination of the current PR number and the commit SHA
           key: velero-cli-${{ github.event.pull_request.number }}-${{ github.sha }}
       - name: Fetch built image
         id: image-cache
-        uses: actions/cache@v2
+        uses: actions/cache@v4
         with:
           path: ./velero.tar
           # The cache key a combination of the current PR number and the commit SHA
           key: velero-image-${{ github.event.pull_request.number }}-${{ github.sha }}
-      - name: Fetch cached go modules
-        uses: actions/cache@v2
-        if: steps.cli-cache.outputs.cache-hit != 'true'
-        with:
-          path: ~/go/pkg/mod
-          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
-          restore-keys: |
-            ${{ runner.os }}-go-
-      - name: Check out the code
-        uses: actions/checkout@v2
-        if: steps.cli-cache.outputs.cache-hit != 'true' || steps.image-cache.outputs.cache-hit != 'true'
       # If no binaries were built for this PR, build it now.
       - name: Build Velero CLI
         if: steps.cli-cache.outputs.cache-hit != 'true'
@@ -53,63 +43,56 @@ jobs:
         run: |
           IMAGE=velero VERSION=pr-test make container
           docker save velero:pr-test -o ./velero.tar
-  # Run E2E test against all kubernetes versions on kind
+  # Run E2E test against all Kubernetes versions on kind
   run-e2e-test:
     needs: build
     runs-on: ubuntu-latest
     strategy:
       matrix:
         k8s:
-          # doesn't cover 1.15 as 1.15 doesn't support "apiextensions.k8s.io/v1" that is needed for the case
-          #- 1.15.12
-          - 1.16.15
-          - 1.17.17
-          - 1.18.20
-          - 1.19.16
-          - 1.20.15
-          - 1.21.12
-          - 1.22.9
-          - 1.23.6
-          - 1.24.0
+          - 1.23.17
+          - 1.24.17
+          - 1.25.16
+          - 1.26.13
+          - 1.27.10
+          - 1.28.6
+          - 1.29.1
+        labels:
+          # labels are used to filter running E2E cases
+          - Basic && (ClusterResource || NodePort || StorageClass)
+          - ResourceFiltering && !Restic
+          - ResourceModifier || (Backups && BackupsSync) || PrivilegesMgmt || OrderedResources
+          - (NamespaceMapping && Single && Restic) || (NamespaceMapping && Multiple && Restic)
       fail-fast: false
     steps:
-      - name: Set up Go
-        uses: actions/setup-go@v2
-        with:
-          go-version: 1.17
-        id: go
       - name: Check out the code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version-file: 'go.mod'
       - name: Install MinIO
         run:
           docker run -d --rm -p 9000:9000 -e "MINIO_ACCESS_KEY=minio" -e "MINIO_SECRET_KEY=minio123" -e "MINIO_DEFAULT_BUCKETS=bucket,additional-bucket" bitnami/minio:2021.6.17-debian-10-r7
       - uses: engineerd/setup-kind@v0.5.0
         with:
-          version: "v0.14.0"
+          version: "v0.21.0"
           image: "kindest/node:v${{ matrix.k8s }}"
       - name: Fetch built CLI
         id: cli-cache
-        uses: actions/cache@v2
+        uses: actions/cache@v4
         with:
           path: ./_output/bin/linux/amd64/velero
           key: velero-cli-${{ github.event.pull_request.number }}-${{ github.sha }}
       - name: Fetch built Image
         id: image-cache
-        uses: actions/cache@v2
+        uses: actions/cache@v4
         with:
           path: ./velero.tar
           key: velero-image-${{ github.event.pull_request.number }}-${{ github.sha }}
       - name: Load Velero Image
         run:
           kind load image-archive velero.tar
-      # always try to fetch the cached go modules as the e2e test needs it either
-      - name: Fetch cached go modules
-        uses: actions/cache@v2
-        with:
-          path: ~/go/pkg/mod
-          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
-          restore-keys: |
-            ${{ runner.os }}-go-
       - name: Run E2E test
         run: |
           cat << EOF > /tmp/credential
@@ -117,17 +100,28 @@ jobs:
           aws_access_key_id=minio
           aws_secret_access_key=minio123
           EOF
-          GOPATH=~/go CLOUD_PROVIDER=kind \
-              OBJECT_STORE_PROVIDER=aws BSL_CONFIG=region=minio,s3ForcePathStyle="true",s3Url=http://$(hostname -i):9000 \
-              CREDS_FILE=/tmp/credential BSL_BUCKET=bucket \
-              ADDITIONAL_OBJECT_STORE_PROVIDER=aws ADDITIONAL_BSL_CONFIG=region=minio,s3ForcePathStyle="true",s3Url=http://$(hostname -i):9000 \
-              ADDITIONAL_CREDS_FILE=/tmp/credential ADDITIONAL_BSL_BUCKET=additional-bucket \
-              GINKGO_FOCUS='Basic\].+\[ClusterResource' VELERO_IMAGE=velero:pr-test \
-              make -C test/e2e run
+
+          # Match kubectl version to k8s server version
+          curl -LO https://dl.k8s.io/release/v${{ matrix.k8s }}/bin/linux/amd64/kubectl
+          sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
+
+          GOPATH=~/go \
+              CLOUD_PROVIDER=kind \
+              OBJECT_STORE_PROVIDER=aws \
+              BSL_CONFIG=region=minio,s3ForcePathStyle="true",s3Url=http://$(hostname -i):9000 \
+              CREDS_FILE=/tmp/credential \
+              BSL_BUCKET=bucket \
+              ADDITIONAL_OBJECT_STORE_PROVIDER=aws \
+              ADDITIONAL_BSL_CONFIG=region=minio,s3ForcePathStyle="true",s3Url=http://$(hostname -i):9000 \
+              ADDITIONAL_CREDS_FILE=/tmp/credential \
+              ADDITIONAL_BSL_BUCKET=additional-bucket \
+              VELERO_IMAGE=velero:pr-test \
+              GINKGO_LABELS="${{ matrix.labels }}" \
+              make -C test/ run-e2e
         timeout-minutes: 30
       - name: Upload debug bundle
         if: ${{ failure() }}
-        uses: actions/upload-artifact@v2
+        uses: actions/upload-artifact@v4
         with:
           name: DebugBundle
-          path: /home/runner/work/velero/velero/test/e2e/debug-bundle*
+          path: /home/runner/work/velero/velero/test/e2e/debug-bundle*

.github/workflows/milestoned-issues.yml

@@ -1,18 +0,0 @@
-name: Add issues with a milestone to the milestone's board
-
-on:
-  issues:
-    types: [milestoned]
-
-jobs:
-  automate-project-columns:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: alex-page/github-project-automation-plus@v0.3.0
-        with:
-          # Do NOT add PRs to the board, as that's duplication. Their corresponding issue should be on the board.
-          if: ${{ !github.event.issue.pull_request }}
-          project: "${{ github.event.issue.milestone.title }}"
-          column: "To Do"
-          repo-token: ${{ secrets.GH_TOKEN }}
-

.github/workflows/nightly-trivy-scan.yml

@@ -0,0 +1,36 @@
+name: Trivy Nightly Scan
+on:
+  schedule:
+    - cron: '0 2 * * *' # run at 2 AM UTC
+
+jobs:
+  nightly-scan:
+    name: Trivy nightly scan
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # maintain the versions of Velero those need security scan
+        versions: [main]
+        # list of images that need scan
+        images: [velero, velero-restore-helper]
+    permissions:
+      security-events: write  # for github/codeql-action/upload-sarif to upload SARIF results
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Run Trivy vulnerability scanner
+        uses: aquasecurity/trivy-action@master
+        with:
+          image-ref: 'docker.io/velero/${{ matrix.images }}:${{ matrix.versions }}'
+          severity: 'CRITICAL,HIGH,MEDIUM'
+          format: 'template'
+          template: '@/contrib/sarif.tpl'
+          output: 'trivy-results.sarif'
+
+      - name: Upload Trivy scan results to GitHub Security tab
+        uses: github/codeql-action/upload-sarif@v3
+        with:
+          sarif_file: 'trivy-results.sarif'

.github/workflows/opened-issues-triage.yml

@@ -1,15 +0,0 @@
-name: Move new issues into Triage
-
-on:
-  issues:
-    types: [opened]
-
-jobs:
-  automate-project-columns:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: alex-page/github-project-automation-plus@v0.3.0
-        with:
-          project: "Velero Support Board"
-          column: "New"
-          repo-token: ${{ secrets.GH_TOKEN }}

.github/workflows/pr-changelog-check.yml

@@ -1,5 +1,9 @@
 name: Pull Request Changelog Check
-on: [pull_request]
+# by setting `on: [pull_request]`, that means action will be trigger when PR is opened, synchronize, reopened.
+# Add labeled and unlabeled events too.
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, labeled, unlabeled]
 jobs:
 
   build:
@@ -8,7 +12,7 @@ jobs:
     steps:
 
     - name: Check out the code
-      uses: actions/checkout@v2
+      uses: actions/checkout@v4
 
     - name: Changelog check
       if: ${{ !(contains(github.event.pull_request.labels.*.name, 'kind/changelog-not-required') || contains(github.event.pull_request.labels.*.name, 'Design') || contains(github.event.pull_request.labels.*.name, 'Website') || contains(github.event.pull_request.labels.*.name, 'Documentation'))}}

.github/workflows/pr-ci-check.yml

@@ -4,26 +4,21 @@ jobs:
   build:
     name: Run CI
     runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
     steps:
-      - name: Set up Go
-        uses: actions/setup-go@v2
-        with:
-          go-version: 1.17
-        id: go
       - name: Check out the code
-        uses: actions/checkout@v2
-      - name: Fetch cached go modules
-        uses: actions/cache@v2
+        uses: actions/checkout@v4
+      - name: Set up Go
+        uses: actions/setup-go@v5
         with:
-          path: ~/go/pkg/mod
-          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
-          restore-keys: |
-            ${{ runner.os }}-go-
+          go-version-file: 'go.mod'
       - name: Make ci
         run: make ci
       - name: Upload test coverage
-        uses: codecov/codecov-action@v2
+        uses: codecov/codecov-action@v4
         with:
           token: ${{ secrets.CODECOV_TOKEN }}
           files: coverage.out
           verbose: true
+          fail_ci_if_error: true

.github/workflows/pr-codespell.yml

@@ -8,13 +8,50 @@ jobs:
     steps:
 
     - name: Check out the code
-      uses: actions/checkout@v2
+      uses: actions/checkout@v4
 
     - name: Codespell
       uses: codespell-project/actions-codespell@master
       with:
         # ignore the config/.../crd.go file as it's generated binary data that is edited elswhere.
-        skip: .git,*.png,*.jpg,*.woff,*.ttf,*.gif,*.ico,./config/crd/v1beta1/crds/crds.go,./config/crd/v1/crds/crds.go,./go.sum
-        ignore_words_list: iam,aks,ist,bridget,ue,shouldnot
+        skip: .git,*.png,*.jpg,*.woff,*.ttf,*.gif,*.ico,./config/crd/v1beta1/crds/crds.go,./config/crd/v1/crds/crds.go,./config/crd/v2alpha1/crds/crds.go,./go.sum,./LICENSE
+        ignore_words_list: iam,aks,ist,bridget,ue,shouldnot,atleast,notin,sme,optin
         check_filenames: true
         check_hidden: true
+
+    - name: Velero.io word list check
+      shell: bash {0}
+      run: |
+        IGNORE_COMMENT="Velero.io word list : ignore"
+        FILES_TO_CHECK=$(find . -type f \
+          ! -path "./.git/*" \
+          ! -path "./site/content/docs/v*" \
+          ! -path "./changelogs/CHANGELOG-*" \
+          ! -path "./.github/workflows/pr-codespell.yml" \
+          ! -path "./site/static/fonts/Metropolis/Open Font License.md" \
+          ! -regex '.*\.\(png\|jpg\|woff\|ttf\|gif\|ico\|svg\)'
+        )
+        function check_word_in_files() {
+            local word=$1
+
+            xargs grep -Iinr "$word" <<< "$FILES_TO_CHECK" | \
+            grep -v "$IGNORE_COMMENT" | \
+            grep -i --color=always "$word" && \
+            EXIT_STATUS=1
+        }
+        function check_word_case_sensitive_in_files() {
+            local word=$1
+
+            xargs grep -Inr "$word" <<< "$FILES_TO_CHECK" | \
+            grep -v "$IGNORE_COMMENT" | \
+            grep --color=always "$word" && \
+            EXIT_STATUS=1
+        }
+        EXIT_STATUS=0
+        check_word_case_sensitive_in_files ' kubernetes '
+        check_word_in_files 'on-premise\b'
+        check_word_in_files 'back-up'
+        check_word_in_files 'plug-in'
+        check_word_in_files 'whitelist'
+        check_word_in_files 'blacklist'
+        exit $EXIT_STATUS

.github/workflows/pr-containers.yml

@@ -0,0 +1,37 @@
+name: build Velero containers on Dockerfile change
+
+on:
+  pull_request:
+    branches:
+      - 'main'
+      - 'release-**'
+    paths:
+      - 'Dockerfile'
+
+jobs:
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+      name: Checkout
+
+    - name: Set up QEMU
+      id: qemu
+      uses: docker/setup-qemu-action@v3
+      with:
+        platforms: all
+
+    - name: Set up Docker Buildx
+      id: buildx
+      uses: docker/setup-buildx-action@v3
+      with:
+        version: latest
+
+    # Although this action also calls docker-push.sh, it is not triggered
+    # by push, so BRANCH and TAG are empty by default. docker-push.sh will
+    # only build Velero image without pushing.
+    - name: Make Velero container without pushing to registry.
+      if: github.repository == 'vmware-tanzu/velero'
+      run: |
+        ./hack/docker-push.sh

.github/workflows/pr-goreleaser.yml

@@ -0,0 +1,29 @@
+name: Verify goreleaser change
+
+on:
+  pull_request:
+    branches:
+      - 'main'
+      - 'release-**'
+    paths:
+      - '.goreleaser.yml'
+      - 'hack/release-tools/goreleaser.sh'
+
+jobs:
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+      name: Checkout
+
+    - name: Verify .goreleaser.yml and try a dryrun release.
+      if: github.repository == 'vmware-tanzu/velero'
+      run: |
+        CHANGELOG=$(ls changelogs | sort -V -r | head -n 1)
+        GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }} \
+        REGISTRY=velero \
+        RELEASE_NOTES_FILE=changelogs/$CHANGELOG \
+        PUBLISH=false \
+        make release
+

.github/workflows/pr-linter-check.yml

@@ -6,9 +6,14 @@ jobs:
     name: Run Linter Check
     runs-on: ubuntu-latest
     steps:
-
-    - name: Check out the code
-      uses: actions/checkout@v2
-
-    - name: Linter check
-      run: make lint
+      - name: Check out the code
+        uses: actions/checkout@v4
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version-file: 'go.mod'
+      - name: Linter check
+        uses: golangci/golangci-lint-action@v6
+        with:
+          version: v1.57.2
+          args: --verbose

.github/workflows/prow-action.yml

@@ -9,12 +9,21 @@ jobs:
   execute:
     runs-on: ubuntu-latest
     steps:
-      - uses: jpmcb/prow-github-actions@v1.1.2
+      - uses: jpmcb/prow-github-actions@v1.1.3
         with:
-          # Only support /kind command for now.
           # TODO: before allowing the /lgtm command, see if we can block merging if changelog labels are missing.
-          prow-commands: "/area
-            /kind
+          prow-commands: |
+            /approve
+            /area
+            /assign
             /cc
-            /uncc"
+            /close
+            /hold
+            /kind
+            /milestone
+            /retitle
+            /remove
+            /reopen
+            /uncc
+            /unassign
           github-token: "${{ secrets.GITHUB_TOKEN }}"

.github/workflows/push-builder.yml

@@ -12,7 +12,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
 
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4
       with:
         # The default value is "1" which fetches only a single commit. If we merge PR without squash or rebase,
         # there are at least two commits: the first one is the merge commit and the second one is the real commit

.github/workflows/push.yml

@@ -14,57 +14,78 @@ jobs:
     name: Build
     runs-on: ubuntu-latest
     steps:
+      - name: Check out the code
+        uses: actions/checkout@v4
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version-file: 'go.mod'
+      - id: 'auth'
+        uses: google-github-actions/auth@v2
+        with:
+          credentials_json: '${{ secrets.GCS_SA_KEY }}'
+      - name: 'set up GCloud SDK'
+        uses: google-github-actions/setup-gcloud@v2
+      - name: 'use gcloud CLI'
+        run: |
+          gcloud info
+      - name: Set up QEMU
+        id: qemu
+        uses: docker/setup-qemu-action@v3
+        with:
+          platforms: all
+      - name: Set up Docker Buildx
+        id: buildx
+        uses: docker/setup-buildx-action@v3
+        with:
+          version: latest
+      - name: Build
+        run: |
+          make local
+          # Clean go cache to ease the build environment storage pressure.
+          go clean -modcache -cache
+      - name: Test
+        run: make test
+      - name: Upload test coverage
+        uses: codecov/codecov-action@v4
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          files: coverage.out
+          verbose: true
+      # Use the JSON key in secret to login gcr.io
+      - uses: 'docker/login-action@v3'
+        with:
+          registry: 'gcr.io' # or REGION.docker.pkg.dev
+          username: '_json_key'
+          password: '${{ secrets.GCR_SA_KEY }}'
+      # Only try to publish the container image from the root repo; forks don't have permission to do so and will always get failures.
+      - name: Publish container image
+        if: github.repository == 'vmware-tanzu/velero'
+        run: |
+          sudo swapoff -a
+          sudo rm -f /mnt/swapfile
+          docker system prune -a --force
+              
+          # Build and push Velero image to docker registry
+          docker login -u ${{ secrets.DOCKER_USER }} -p ${{ secrets.DOCKER_PASSWORD }}
+          VERSION=$(./hack/docker-push.sh | grep 'VERSION:' | awk -F: '{print $2}' | xargs)
 
-    - name: Set up Go
-      uses: actions/setup-go@v2
-      with:
-        go-version: 1.17
-      id: go
+          # Upload Velero image package to GCS
+          source hack/ci/build_util.sh
+          BIN=velero
+          RESTORE_HELPER_BIN=velero-restore-helper
+          GCS_BUCKET=velero-builds
+          VELERO_IMAGE=${BIN}-${VERSION}
+          VELERO_RESTORE_HELPER_IMAGE=${RESTORE_HELPER_BIN}-${VERSION}
+          VELERO_IMAGE_FILE=${VELERO_IMAGE}.tar.gz
+          VELERO_RESTORE_HELPER_IMAGE_FILE=${VELERO_RESTORE_HELPER_IMAGE}.tar.gz
+          VELERO_IMAGE_BACKUP_FILE=${VELERO_IMAGE}-'build.'${GITHUB_RUN_NUMBER}.tar.gz
+          VELERO_RESTORE_HELPER_IMAGE_BACKUP_FILE=${VELERO_RESTORE_HELPER_IMAGE}-'build.'${GITHUB_RUN_NUMBER}.tar.gz
 
-    - name: Check out code into the Go module directory
-      uses: actions/checkout@v2
+          cp ${VELERO_IMAGE_FILE} ${VELERO_IMAGE_BACKUP_FILE}
+          cp ${VELERO_RESTORE_HELPER_IMAGE_FILE} ${VELERO_RESTORE_HELPER_IMAGE_BACKUP_FILE}
 
-    - name: Set up QEMU
-      id: qemu
-      uses: docker/setup-qemu-action@v1
-      with:
-        platforms: all
-
-    - name: Set up Docker Buildx
-      id: buildx
-      uses: docker/setup-buildx-action@v1
-      with:
-        version: latest
-
-    - name: Build
-      run: make local
-
-    - name: Test
-      run: make test
-
-    - name: Upload test coverage
-      uses: codecov/codecov-action@v2
-      with:
-        token: ${{ secrets.CODECOV_TOKEN }}
-        files: coverage.out
-        verbose: true
-
-    # Only try to publish the container image from the root repo; forks don't have permission to do so and will always get failures.
-    - name: Publish container image
-      if: github.repository == 'vmware-tanzu/velero'
-      run: |
-        docker login -u ${{ secrets.DOCKER_USER }} -p ${{ secrets.DOCKER_PASSWORD }}
-        ./hack/docker-push.sh
-
-    # Use the JSON key in secret to login gcr.io
-    - uses: 'docker/login-action@v1'
-      with:
-        registry: 'gcr.io' # or REGION.docker.pkg.dev
-        username: '_json_key'
-        password: '${{ secrets.GCR_SA_KEY }}'
-
-    # Push image to GCR to facilitate some environments that have rate limitation to docker hub, e.g. vSphere.
-    - name: Publish container image to GCR
-      if: github.repository == 'vmware-tanzu/velero'
-      run: |
-        REGISTRY=gcr.io/velero-gcp ./hack/docker-push.sh
+          uploader ${VELERO_IMAGE_FILE} ${GCS_BUCKET}
+          uploader ${VELERO_RESTORE_HELPER_IMAGE_FILE} ${GCS_BUCKET}
+          uploader ${VELERO_IMAGE_BACKUP_FILE} ${GCS_BUCKET}
+          uploader ${VELERO_RESTORE_HELPER_IMAGE_BACKUP_FILE} ${GCS_BUCKET}

.github/workflows/rebase.yml

@@ -9,10 +9,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Checkout the latest code
-      uses: actions/checkout@v2
+      uses: actions/checkout@v4
       with:
         fetch-depth: 0
     - name: Automatic Rebase
-      uses: cirrus-actions/rebase@1.3.1
+      uses: cirrus-actions/rebase@1.8
       env:
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/stale-issues.yml

@@ -1,24 +1,23 @@
 name: "Close stale issues and PRs"
 on:
   schedule:
-    # First of every month
-    - cron: "30 1 * * *"
+    - cron: "30 1 * * *" # Every day at 1:30 UTC
 
 jobs:
   stale:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/stale@v3
+      - uses: actions/stale@v9.0.0
         with:
           repo-token: ${{ secrets.GITHUB_TOKEN }}
-          stale-issue-message: "This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. If a Velero team member has requested log or more information, please provide the output of the shared commands."
-          close-issue-message: "This issue was closed because it has been stalled for 5 days with no activity."
-          days-before-issue-stale: 30
-          days-before-issue-close: 5
+          stale-issue-message: "This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands."
+          close-issue-message: "This issue was closed because it has been stalled for 14 days with no activity."
+          days-before-issue-stale: 60
+          days-before-issue-close: 14
+          stale-issue-label: staled
           # Disable stale PRs for now; they can remain open.
           days-before-pr-stale: -1
           days-before-pr-close: -1
           # Only issues made after Feb 09 2021.
           start-date: "2021-09-02T00:00:00"
-          # Only make issues stale if they have these labels. Comma separated.
-          only-labels: "Needs info,Duplicate"
+          exempt-issue-labels: "Epic,Area/CLI,Area/Cloud/AWS,Area/Cloud/Azure,Area/Cloud/GCP,Area/Cloud/vSphere,Area/CSI,Area/Design,Area/Documentation,Area/Plugins,Bug,Enhancement/User,kind/requirement,kind/refactor,kind/tech-debt,limitation,Needs investigation,Needs triage,Needs Product,P0 - Hair on fire,P1 - Important,P2 - Long-term important,P3 - Wouldn't it be nice if...,Product Requirements,Restic - GA,Restic,release-blocker,Security,backlog"

.gitignore

@@ -38,6 +38,7 @@ _testmain.go
 # Hugo compiled data
 site/public
 site/resources
+site/.hugo_build.lock
 
 .vs
 
@@ -46,5 +47,10 @@ _tiltbuild
 tilt-resources/tilt-settings.json
 tilt-resources/velero_v1_backupstoragelocation.yaml
 tilt-resources/deployment.yaml
-tilt-resources/restic.yaml
+tilt-resources/node-agent.yaml
 tilt-resources/cloud
+
+# test generated files
+test/e2e/report.xml
+coverage.out
+__debug_bin*

.golangci.yaml

@@ -7,42 +7,16 @@ run:
   concurrency: 4
 
   # timeout for analysis, e.g. 30s, 5m, default is 1m
-  timeout: 5m
+  timeout: 20m
 
   # exit code when at least one issue was found, default is 1
   issues-exit-code: 1
 
-  # include test files or not, default is true
-  tests: true
-
-  # list of build tags, all linters use it. Default is empty list.
-  #build-tags:
-  #  - mytag
-
-  # which dirs to skip: issues from them won't be reported;
-  # can use regexp here: generated.*, regexp is applied on full path;
-  # default value is empty list, but default dirs are skipped independently
-  # from this option's value (see skip-dirs-use-default).
-  # "/" will be replaced by current OS file path separator to properly work
-  # on Windows.
-  #skip-dirs:
-  #  - src/external_libs
-  #  - autogenerated_by_my_lib
 
   # default is true. Enables skipping of directories:
   #   vendor$, third_party$, testdata$, examples$, Godeps$, builtin$
   skip-dirs-use-default: true
 
-  # which files to skip: they will be analyzed, but issues from them
-  # won't be reported. Default value is empty list, but there is
-  # no need to include all autogenerated files, we confidently recognize
-  # autogenerated files. If it's not please let us know.
-  # "/" will be replaced by current OS file path separator to properly work
-  # on Windows.
-  # skip-files:
-  #  - ".*\\.my\\.go$"
-  #  - lib/bad.go
-
   # by default isn't set. If set we pass it to "go list -mod={option}". From "go help modules":
   # If invoked with -mod=readonly, the go command is disallowed from the implicit
   # automatic updating of go.mod described above. Instead, it fails when any changes
@@ -62,7 +36,9 @@ run:
 # output configuration options
 output:
   # colored-line-number|line-number|json|tab|checkstyle|code-climate, default is "colored-line-number"
-  format: colored-line-number
+  formats:
+    - format: colored-line-number
+      path: stdout
 
   # print lines of code with issue, default is true
   print-issued-lines: true
@@ -117,7 +93,7 @@ linters-settings:
     # minimal length of string constant, 3 by default
     min-len: 3
     # minimal occurrences count to trigger, 3 by default
-    min-occurrences: 3
+    min-occurrences: 5
   gocritic:
     # Which checks should be enabled; can't be combined with 'disabled-checks';
     # See https://go-critic.github.io/overview#checks-overview
@@ -166,10 +142,8 @@ linters-settings:
     # minimal confidence for issues, default is 0.8
     min-confidence: 0.8
   gomnd:
-    settings:
-      mnd:
-        # the list of enabled checks, see https://github.com/tommy-muehle/go-mnd/#checks for description.
-        checks: argument,case,condition,operation,return,assign
+    # the list of enabled checks, see https://github.com/tommy-muehle/go-mnd/#checks for description.
+    checks: argument,case,condition,operation,return,assign
   gomodguard:
     allowed:
       modules:                                                        # List of allowed modules
@@ -188,7 +162,7 @@ linters-settings:
         #     reason: "testing if blocked version constraint works."  # Reason why the version constraint exists. (Optional)
   govet:
     # report about shadowed variables
-    check-shadowing: true
+    # check-shadowing: true
 
     # settings per analyzer
     settings:
@@ -207,12 +181,12 @@ linters-settings:
       - shadow
     disable-all: false
   depguard:
-    list-type: blacklist
+    list-type: blacklist # Velero.io word list : ignore
     include-go-root: false
     packages:
       - github.com/sirupsen/logrus
     packages-with-error-message:
-      # specify an error message to output when a blacklisted package is used
+      # specify an error message to output when a denylisted package is used
       - github.com/sirupsen/logrus: "logging is allowed only by logutils.Log"
   lll:
     # max line length, lines longer will be reported. Default is 120.
@@ -253,9 +227,21 @@ linters-settings:
     require-explanation: true
     # Enable to require nolint directives to mention the specific linter being suppressed. Default is false.
     require-specific: true
+  revive:
+    rules:
+      - name: unexported-return
+        disabled: true
+    
   rowserrcheck:
     packages:
       - github.com/jmoiron/sqlx
+  testifylint:
+      # TODO: enable them all
+      disable:
+        - go-require
+        - float-compare
+        - require-error
+      enable-all: true
   testpackage:
     # regexp pattern to skip files
     skip-regexp: (export|internal)_test\.go
@@ -294,76 +280,77 @@ linters-settings:
     # Allow leading comments to be separated with empty liens
     allow-separated-leading-comment: false
 
-  # The custom section can be used to define linter plugins to be loaded at runtime. See README doc
-  #  for more info.
-  # custom:
-    # Each custom linter should have a unique name.
-    #  example:
-      # The path to the plugin *.so. Can be absolute or local. Required for each custom linter
-      # path: /path/to/example.so
-      # The description of the linter. Optional, just for documentation purposes.
-      # description: This is an example usage of a plugin linter.
-      # Intended to point to the repo location of the linter. Optional, just for documentation purposes.
-      # original-url: github.com/golangci/example-linter
-
 linters:
-#  enable:
-#    - megacheck
-#    - govet
-# disable:
-#    - maligned
-#    - prealloc
   disable-all: true
-  presets:
-#    - bugs
-#    - unused
+  enable:
+    - asasalint
+    - asciicheck
+    - bidichk
+    - bodyclose
+    - dogsled
+    - durationcheck
+    - dupword
+    - errcheck
+    - exportloopref
+    - errchkjson
+    - goconst
+    - gofmt
+    - goheader
+    - goimports
+    - goprintffuncname
+    - gosec
+    - gosimple
+    - govet
+    - ginkgolinter
+    - importas
+    - ineffassign
+    - misspell
+    - nakedret
+    - nosprintfhostport
+    - nilerr
+    - noctx
+    - nolintlint
+    - revive
+    - staticcheck
+    - stylecheck
+    - testifylint
+    - thelper
+    - typecheck
+    - unconvert
+    - unparam
+    - unused
+    - usestdlibvars
+    - whitespace
   fast: false
 
 
-#issues:
-#  # List of regexps of issue texts to exclude, empty list by default.
-#  # But independently from this option we use default exclude patterns,
-#  # it can be disabled by `exclude-use-default: false`. To list all
-#  # excluded by default patterns execute `golangci-lint run --help`
-#  exclude:
-#    - abcdef
-#
-#  # Excluding configuration per-path, per-linter, per-text and per-source
-#  exclude-rules:
-#    # Exclude some linters from running on tests files.
-#    - path: _test\.go
-#      linters:
-#        - gocyclo
-#        - errcheck
-#        - dupl
-#        - gosec
-#
-#    # Exclude known linters from partially hard-vendored code,
-#    # which is impossible to exclude via "nolint" comments.
-#    - path: internal/hmac/
-#      text: "weak cryptographic primitive"
-#      linters:
-#        - gosec
-#
-#    # Exclude some staticcheck messages
-#    - linters:
-#        - staticcheck
-#      text: "SA9003:"
-#
-#    # Exclude lll issues for long lines with go:generate
-#    - linters:
-#        - lll
-#      source: "^//go:generate "
-
-  # Independently from option `exclude` we use default exclude patterns,
-  # it can be disabled by this option. To list all
-  # excluded by default patterns execute `golangci-lint run --help`.
-  # Default value for this option is true.
-  exclude-use-default: false
-
-  # The default value is false. If set to true exclude and exclude-rules
-  # regular expressions become case sensitive.
-  exclude-case-sensitive: false
+issues:
+  exclude-rules:
+    - linters:
+        - staticcheck
+      text: "DefaultVolumesToRestic" # No need to report deprecate for DefaultVolumesToRestic.
+    - path: ".*_test.go$"
+      linters:
+        - dupword
+        - errcheck
+        - goconst
+        - gosec
+        - govet
+        - staticcheck
+        - stylecheck
+        - unparam
+        - unused
+    - path: test/
+      linters:
+        - dupword
+        - errcheck
+        - goconst
+        - gosec
+        - nilerr
+        - staticcheck
+        - stylecheck
+        - unparam
+        - unused
 
   # The list of ids of default excludes to include or disable. By default it's empty.
   include:
@@ -375,19 +362,17 @@ linters:
   # Maximum count of issues with the same text. Set to 0 to disable. Default is 3.
   max-same-issues: 0
 
-  # Show only new issues: if there are unstaged changes or untracked files,
-  # only those changes are analyzed, else only changes in HEAD~ are analyzed.
-  # It's a super-useful option for integration of golangci-lint into existing
-  # large codebase. It's not practical to fix all existing issues at the moment
-  # of integration: much better don't allow issues in new code.
-  # Default is false.
-  new: false
-
   # Show only new issues created after git revision `REV`
-  new-from-rev: REV
+  # new-from-rev: origin/main
 
-  # Show only new issues created in git patch with set file path.
-  new-from-patch: path/to/patch/file
+  # which dirs to skip: issues from them won't be reported;
+  # can use regexp here: generated.*, regexp is applied on full path;
+  # default value is empty list, but default dirs are skipped independently
+  # from this option's value (see skip-dirs-use-default).
+  # "/" will be replaced by current OS file path separator to properly work
+  # on Windows.
+  exclude-dirs:
+    - pkg/plugin/generated/*
 
 severity:
   # Default value is empty string.

.goreleaser.yml

@@ -54,3 +54,10 @@ release:
     name: velero
   draft: true
   prerelease: auto
+
+git:
+  # What should be used to sort tags when gathering the current and previous
+  # tags if there are more than one tag in the same commit.
+  #
+  # Default: `-version:refname`
+  tag_sort: -version:creatordate

ADOPTERS.md

@@ -3,6 +3,7 @@
 If you're using Velero and want to add your organization to this list, 
 [follow these directions][1]!
 
+<a href="https://www.pitsdatarecovery.net/" border="0" target="_blank"><img alt="pitsdatarecovery.net" src="site/static/img/adopters/PITSGlobalDataRecoveryServices.svg" height="50"></a>
 <a href="https://www.bitgo.com" border="0" target="_blank"><img alt="bitgo.com" src="site/static/img/adopters/BitGo.svg" height="50"></a>&nbsp; &nbsp; &nbsp;
 <a href="https://www.nirmata.com" border="0" target="_blank"><img alt="nirmata.com" src="site/static/img/adopters/nirmata.svg" height="50"></a>&nbsp; &nbsp; &nbsp;
 <a href="https://kyma-project.io/" border="0" target="_blank"><img alt="kyma-project.io" src="site/static/img/adopters/kyma.svg" height="50"></a>&nbsp; &nbsp; &nbsp;
@@ -15,16 +16,17 @@ If you're using Velero and want to add your organization to this list,
 <a href="https://mayadata.io/" border="0" target="_blank"><img alt="mayadata.io" src="site/static/img/adopters/mayadata.svg" height="50"></a>&nbsp; &nbsp; &nbsp;
 <a href="https://www.replicated.com/" border="0" target="_blank"><img alt="replicated.com" src="site/static/img/adopters/replicated-logo-red.svg" height="50"></a>
 <a href="https://cloudcasa.io/" border="0" target="_blank"><img alt="cloudcasa.io" src="site/static/img/adopters/cloudcasa.svg" height="50"></a>
+<a href="https://azure.microsoft.com/" border="0" target="_blank"><img alt="azure.com" src="site/static/img/adopters/azure.svg" height="50"></a>
 ## Success Stories
 
 Below is a list of adopters of Velero in **production environments** that have
 publicly shared the details of how they use it.
 
 **[BitGo][20]**  
-BitGo uses Velero backup and restore capabilities to seamlessly provision and scale fullnode statefulsets on the fly as well as having it serve an integral piece for our kubernetes disaster-recovery story.
+BitGo uses Velero backup and restore capabilities to seamlessly provision and scale fullnode statefulsets on the fly as well as having it serve an integral piece for our Kubernetes disaster-recovery story.
 
 **[Bugsnag][30]**  
-We use Velero for managing backups of an internal instance of our on-premise clustered solution. We also recommend our users of [on-premise Bugsnag installations][31] use Velero for [managing their own backups][32].
+We use Velero for managing backups of an internal instance of our on-premise clustered solution. We also recommend our users of [on-premise Bugsnag installations](https://www.bugsnag.com/on-premise) use Velero for [managing their own backups](https://docs.bugsnag.com/on-premise/clustered/backup-restore/). <!-- Velero.io word list : ignore -->
 
 **[Banzai Cloud][60]**  
 [Banzai Cloud Pipeline][61] is a Kubernetes-based microservices platform that integrates services needed for Day-1 and Day-2 operations along with first-class support both for on-prem and hybrid multi-cloud deployments. We use Velero to periodically [backup and restore these clusters in case of disasters][62].
@@ -40,7 +42,9 @@ We have integrated our [solution with Velero][11] to provide our customers with
 Kyma [integrates with Velero][41] to effortlessly back up and restore Kyma clusters with all its resources. Velero capabilities allow Kyma users to define and run manual and scheduled backups in order to successfully handle a disaster-recovery scenario.
 
 **[Red Hat][50]**  
-Red Hat has developed the [Cluster Application Migration Tool][51] which uses [Velero and Restic][52] to drive the migration of applications between OpenShift clusters.
+Red Hat has developed 2 operators for the OpenShift platform:
+- [Migration Toolkit for Containers][51] (Crane): This operator uses [Velero and Restic][52] to drive the migration of applications between OpenShift clusters.
+- [OADP (OpenShift API for Data Protection) Operator][53]: This operator sets up and installs Velero on the OpenShift platform, allowing users to backup and restore applications.
 
 **[Dell EMC][70]**  
 For Kubernetes environments, [PowerProtect Data Manager][71] leverages the Container Storage Interface (CSI) framework to take snapshots to back up the persistent data or the data that the application creates e.g. databases. [Dell EMC leverages Velero][72] to backup the namespace configuration files (also known as Namespace meta data) for enterprise grade data protection.
@@ -59,7 +63,10 @@ Okteto integrates Velero in [Okteto Cloud][94] and [Okteto Enterprise][95] to pe
 Replicated uses the Velero open source project to enable snapshots in [KOTS][101] to backup Kubernetes manifests & persistent volumes. In addition to the default functionality that Velero provides, [KOTS][101] provides a detailed interface in the [Admin Console][102] that can be used to manage the storage destination and schedule, and to perform and monitor the backup and restore process.<br>
 
 **[CloudCasa][103]**<br>
-[Catalogic Software][104] integrates Velero with [CloudCasa][103] - A Smart Home in the Cloud for Backups. CloudCasa is a simple, scalable, cloud-native solution providing data protection and disaster recovery as a service. This solution is built using Kubernetes for protecting Kubernetes clusters.<br>
+[Catalogic Software][104] integrates Velero with [CloudCasa][103] - A Smart Home in the Cloud for Backups. CloudCasa is a full-featured, scalable, cloud-native solution providing Kubernetes data protection, disaster recovery, and migration as a service. An option to manage existing Velero instances and an enterprise self-hosted option are also available.<br>
+
+**[Microsoft Azure][105]**<br>
+[Azure Backup for AKS][106] is an Azure native, Kubernetes aware, Enterprise ready backup for containerized applications deployed on Azure Kubernetes Service (AKS). AKS Backup utilizes Velero to perform backup and restore operations to protect stateful applications in AKS clusters.<br>
 
 ## Adding your organization to the list of Velero Adopters
 
@@ -80,8 +87,6 @@ If you would like to add your logo to a future `Adopters of Velero` section on [
 [20]: https://bitgo.com
 
 [30]: https://bugsnag.com
-[31]: https://www.bugsnag.com/on-premise
-[32]: https://docs.bugsnag.com/on-premise/clustered/backup-restore/
 
 [40]: https://kyma-project.io
 [41]: https://kyma-project.io/docs/components/backup/#overview-overview
@@ -89,6 +94,7 @@ If you would like to add your logo to a future `Adopters of Velero` section on [
 [50]: https://redhat.com
 [51]: https://github.com/fusor/mig-operator
 [52]: https://github.com/fusor/mig-operator/blob/master/docs/usage/2.md
+[53]: https://github.com/openshift/oadp-operator
 
 [60]: https://banzaicloud.com
 [61]: https://banzaicloud.com/products/pipeline/
@@ -116,3 +122,6 @@ If you would like to add your logo to a future `Adopters of Velero` section on [
 
 [103]: https://cloudcasa.io/
 [104]: https://www.catalogicsoftware.com/
+
+[105]: https://azure.microsoft.com/
+[106]: https://learn.microsoft.com/azure/backup/backup-overview

CHANGELOG.md

@@ -1,7 +1,13 @@
 ## Current release:
-  * [CHANGELOG-1.9.md][19]
+  * [CHANGELOG-1.15.md][25]
 
 ## Older releases:
+  * [CHANGELOG-1.14.md][24]
+  * [CHANGELOG-1.13.md][23]
+  * [CHANGELOG-1.12.md][22]
+  * [CHANGELOG-1.11.md][21]
+  * [CHANGELOG-1.10.md][20]
+  * [CHANGELOG-1.9.md][19]
   * [CHANGELOG-1.8.md][18]
   * [CHANGELOG-1.7.md][17]
   * [CHANGELOG-1.6.md][16]
@@ -22,6 +28,12 @@
   * [CHANGELOG-0.3.md][1]
 
 
+[25]: https://github.com/vmware-tanzu/velero/blob/main/changelogs/CHANGELOG-1.15.md
+[24]: https://github.com/vmware-tanzu/velero/blob/main/changelogs/CHANGELOG-1.14.md
+[23]: https://github.com/vmware-tanzu/velero/blob/main/changelogs/CHANGELOG-1.13.md
+[22]: https://github.com/vmware-tanzu/velero/blob/main/changelogs/CHANGELOG-1.12.md
+[21]: https://github.com/vmware-tanzu/velero/blob/main/changelogs/CHANGELOG-1.11.md
+[20]: https://github.com/vmware-tanzu/velero/blob/main/changelogs/CHANGELOG-1.10.md
 [19]: https://github.com/vmware-tanzu/velero/blob/main/changelogs/CHANGELOG-1.9.md
 [18]: https://github.com/vmware-tanzu/velero/blob/main/changelogs/CHANGELOG-1.8.md
 [17]: https://github.com/vmware-tanzu/velero/blob/main/changelogs/CHANGELOG-1.7.md

CODE_OF_CONDUCT.md

@@ -5,7 +5,7 @@
 We as members, contributors, and leaders pledge to make participation in the Velero project and our
 community a harassment-free experience for everyone, regardless of age, body
 size, visible or invisible disability, ethnicity, sex characteristics, gender
-identity and expression, level of experience, education, socio-economic status,
+identity and expression, level of experience, education, socioeconomic status,
 nationality, personal appearance, race, religion, or sexual identity
 and orientation.
 

Dockerfile

@@ -11,50 +11,72 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-FROM --platform=$BUILDPLATFORM golang:1.17.13 as builder-env
+
+# Velero binary build section
+FROM --platform=$BUILDPLATFORM golang:1.22.10-bookworm AS velero-builder
 
 ARG GOPROXY
+ARG BIN
 ARG PKG
 ARG VERSION
+ARG REGISTRY
 ARG GIT_SHA
 ARG GIT_TREE_STATE
-ARG REGISTRY
+ARG TARGETOS
+ARG TARGETARCH
+ARG TARGETVARIANT
 
 ENV CGO_ENABLED=0 \
     GO111MODULE=on \
     GOPROXY=${GOPROXY} \
+    GOOS=${TARGETOS} \
+    GOARCH=${TARGETARCH} \
+    GOARM=${TARGETVARIANT} \
     LDFLAGS="-X ${PKG}/pkg/buildinfo.Version=${VERSION} -X ${PKG}/pkg/buildinfo.GitSHA=${GIT_SHA} -X ${PKG}/pkg/buildinfo.GitTreeState=${GIT_TREE_STATE} -X ${PKG}/pkg/buildinfo.ImageRegistry=${REGISTRY}"
 
 WORKDIR /go/src/github.com/vmware-tanzu/velero
 
 COPY . /go/src/github.com/vmware-tanzu/velero
 
-RUN apt-get update && apt-get install -y bzip2
+RUN mkdir -p /output/usr/bin && \
+    export GOARM=$( echo "${GOARM}" | cut -c2-) && \
+    go build -o /output/${BIN} \
+    -ldflags "${LDFLAGS}" ${PKG}/cmd/${BIN} && \
+    go build -o /output/velero-helper \
+    -ldflags "${LDFLAGS}" ${PKG}/cmd/velero-helper && \
+    go clean -modcache -cache
 
-FROM --platform=$BUILDPLATFORM builder-env as builder
+# Restic binary build section
+FROM --platform=$BUILDPLATFORM golang:1.22.10-bookworm AS restic-builder
 
+ARG BIN
 ARG TARGETOS
 ARG TARGETARCH
 ARG TARGETVARIANT
-ARG PKG
-ARG BIN
 ARG RESTIC_VERSION
 
-ENV GOOS=${TARGETOS} \
+ENV CGO_ENABLED=0 \
+    GO111MODULE=on \
+    GOPROXY=${GOPROXY} \
+    GOOS=${TARGETOS} \
     GOARCH=${TARGETARCH} \
     GOARM=${TARGETVARIANT}
 
+COPY . /go/src/github.com/vmware-tanzu/velero
+
 RUN mkdir -p /output/usr/bin && \
-    bash ./hack/download-restic.sh && \
-    export GOARM=$( echo "${GOARM}" | cut -c2-) && \
-    go build -o /output/${BIN} \
-    -ldflags "${LDFLAGS}" ${PKG}/cmd/${BIN}
+    export GOARM=$(echo "${GOARM}" | cut -c2-) && \
+    /go/src/github.com/vmware-tanzu/velero/hack/build-restic.sh && \
+    go clean -modcache -cache
 
-FROM gcr.io/distroless/base-debian11@sha256:99133cb0878bb1f84d1753957c6fd4b84f006f2798535de22ebf7ba170bbf434
+# Velero image packing section
+FROM paketobuildpacks/run-jammy-tiny:0.2.56
 
-LABEL maintainer="Nolan Brubaker <brubakern@vmware.com>"
+LABEL maintainer="Xun Jiang <jxun@vmware.com>"
 
-COPY --from=builder /output /
+COPY --from=velero-builder /output /
 
-USER nonroot:nonroot
+COPY --from=restic-builder /output /
+
+USER cnb:cnb
 

GOVERNANCE.md

@@ -107,6 +107,29 @@ Lazy consensus does _not_ apply to the process of:
 
 * Removal of maintainers from Velero
 
+## Deprecation Policy
+
+### Deprecation Process
+
+Any contributor may introduce a request to deprecate a feature or an option of a feature by opening a feature request issue in the vmware-tanzu/velero GitHub project. The issue should describe why the feature is no longer needed or has become detrimental to Velero, as well as whether and how it has been superseded. The submitter should give as much detail as possible.
+
+Once the issue is filed, a one-month discussion period begins. Discussions take place within the issue itself as well as in the community meetings. The person who opens the issue, or a maintainer, should add the date and time marking the end of the discussion period in a comment on the issue as soon as possible after it is opened. A decision on the issue needs to be made within this one-month period.
+
+The feature will be deprecated by a supermajority vote of 50% plus one of the project maintainers at the time of the vote tallying, which is 72 hours after the end of the community meeting that is the end of the comment period. (Maintainers are permitted to vote in advance of the deadline, but should hold their votes until as close as possible to hear all possible discussion.) Votes will be tallied in comments on the issue. 
+
+Non-maintainers may add non-binding votes in comments to the issue as well; these are opinions to be taken into consideration by maintainers, but they do not count as votes. 
+
+If the vote passes, the deprecation window takes effect in the subsequent release, and the removal follows the schedule. 
+
+### Schedule
+If depreciation proposal passes by supermajority votes, the feature is deprecated in the next minor release and the feature can be removed completely after two minor version or equivalent major version e.g., if feature gets deprecated in Nth minor version, then feature can be removed after N+2 minor version or its equivalent if the major version number changes.
+
+### Deprecation Window
+
+The deprecation window is the period from the release in which the deprecation takes effect through the release in which the feature is removed. During this period, only critical security vulnerabilities and catastrophic bugs should be fixed.
+
+**Note:** If a backup relies on a deprecated feature, then backups made with the last Velero release before this feature is removed must still be restorable in version `n+2`. For instance, something like restic feature support, that might mean that restic is removed from the list of supported uploader types in version `n` but the underlying implementation required to restore from a restic backup won't be removed until release `n+2`.
+
 ## Updating Governance
 
 All substantive changes in Governance require a supermajority agreement by all maintainers.

MAINTAINERS.md

@@ -4,15 +4,16 @@
 
 ## Maintainers
 
-| Maintainer | GitHub ID | Affiliation |
-| --------------- | --------- | ----------- |
-| Dave Smith-Uchida | [dsu-igeek](https://github.com/dsu-igeek) | [Kasten](https://github.com/kastenhq/) |
-| Scott Seago | [sseago](https://github.com/sseago) | [OpenShift](https://github.com/openshift)
-| Daniel Jiang | [reasonerjt](https://github.com/reasonerjt) | [VMware](https://www.github.com/vmware/)
-| Wenkai Yin | [ywk253100](https://github.com/ywk253100) | [VMware](https://www.github.com/vmware/) |
-| Xun Jiang | [blackpiglet](https://github.com/blackpiglet) | [VMware](https://www.github.com/vmware/) |
-| Ming Qiu | [qiuming-best](https://github.com/qiuming-best) | [VMware](https://www.github.com/vmware/) |
-| Shubham Pampattiwar | [shubham-pampattiwar](https://github.com/shubham-pampattiwar) | [OpenShift](https://github.com/openshift)
+| Maintainer          | GitHub ID                                                     | Affiliation                                      |
+|---------------------|---------------------------------------------------------------|--------------------------------------------------|
+| Scott Seago         | [sseago](https://github.com/sseago)                           | [OpenShift](https://github.com/openshift)        |
+| Daniel Jiang        | [reasonerjt](https://github.com/reasonerjt)                   | [VMware](https://www.github.com/vmware/)         |
+| Wenkai Yin          | [ywk253100](https://github.com/ywk253100)                     | [VMware](https://www.github.com/vmware/)         |
+| Xun Jiang           | [blackpiglet](https://github.com/blackpiglet)                 | [VMware](https://www.github.com/vmware/)         |
+| Ming Qiu            | [qiuming-best](https://github.com/qiuming-best)               | [VMware](https://www.github.com/vmware/)         |
+| Shubham Pampattiwar | [shubham-pampattiwar](https://github.com/shubham-pampattiwar) | [OpenShift](https://github.com/openshift)        |
+| Yonghui Li          | [Lyndon-Li](https://github.com/Lyndon-Li)                     | [VMware](https://www.github.com/vmware/)         |
+| Anshul Ahuja        | [anshulahuja98](https://github.com/anshulahuja98)             | [Microsoft Azure](https://www.github.com/azure/) |
 
 ## Emeritus Maintainers
 * Adnan Abdulhussein ([prydonius](https://github.com/prydonius))
@@ -24,14 +25,15 @@
 * Carlisia Thompson ([carlisia](https://github.com/carlisia))
 * Bridget McErlean ([zubron](https://github.com/zubron))
 * JenTing Hsiao ([jenting](https://github.com/jenting))
-
+* Dave Smith-Uchida ([dsu-igeek](https://github.com/dsu-igeek))
+  
 ## Velero Contributors & Stakeholders
 
-| Feature Area | Lead |
-| ----------------------------- | :---------------------: |
-| Architect | Dave Smith-Uchida (dsu-igeek) |
-| Technical Lead | Daniel Jiang (reasonerjt) |
-| Kubernetes CSI Liaison |  |
-| Deployment |  |
-| Community Management | Orlin Vasilev (OrlinVasilev) |
-| Product Management | Eleanor Millman (eleanor-millman) |
+| Feature Area           |                                         Lead                                         |
+|------------------------|:------------------------------------------------------------------------------------:|
+| Technical Lead         |               Daniel Jiang [reasonerjt](https://github.com/reasonerjt)               |
+| Kubernetes CSI Liaison |                                                                                      |
+| Deployment             |                                                                                      |
+| Community Management   |            Orlin Vasilev [OrlinVasilev](https://github.com/OrlinVasilev)             |
+| Product Management     | Pradeep Kumar Chaturvedi [pradeepkchaturvedi](https://github.com/pradeepkchaturvedi) |
+

Makefile

@@ -22,9 +22,11 @@ PKG := github.com/vmware-tanzu/velero
 
 # Where to push the docker image.
 REGISTRY ?= velero
+GCR_REGISTRY ?= gcr.io/velero-gcp
 
 # Image name
 IMAGE ?= $(REGISTRY)/$(BIN)
+GCR_IMAGE ?= $(GCR_REGISTRY)/$(BIN)
 
 # We allow the Dockerfile to be configurable to enable the use of custom Dockerfiles
 # that pull base images from different registries.
@@ -66,11 +68,23 @@ TAG_LATEST ?= false
 
 ifeq ($(TAG_LATEST), true)
 	IMAGE_TAGS ?= $(IMAGE):$(VERSION) $(IMAGE):latest
+	GCR_IMAGE_TAGS ?= $(GCR_IMAGE):$(VERSION) $(GCR_IMAGE):latest
 else
 	IMAGE_TAGS ?= $(IMAGE):$(VERSION)
+	GCR_IMAGE_TAGS ?= $(GCR_IMAGE):$(VERSION)
 endif
 
-ifeq ($(shell docker buildx inspect 2>/dev/null | awk '/Status/ { print $$2 }'), running)
+# check buildx is enabled only if docker is in path
+# macOS/Windows docker cli without Docker Desktop license: https://github.com/abiosoft/colima
+# To add buildx to docker cli: https://github.com/abiosoft/colima/discussions/273#discussioncomment-2684502
+ifeq ($(shell which docker 2>/dev/null 1>&2 && docker buildx inspect 2>/dev/null | awk '/Status/ { print $$2 }'), running)
+	BUILDX_ENABLED ?= true
+# if emulated docker cli from podman, assume enabled
+# emulated docker cli from podman: https://podman-desktop.io/docs/migrating-from-docker/emulating-docker-cli-with-podman
+# podman known issues:
+# - on remote podman, such as on macOS,
+#   --output issue: https://github.com/containers/podman/issues/15922
+else ifeq ($(shell which docker 2>/dev/null 1>&2 && cat $(shell which docker) | grep -c "exec podman"), 1)
 	BUILDX_ENABLED ?= true
 else
 	BUILDX_ENABLED ?= false
@@ -82,7 +96,7 @@ see: https://velero.io/docs/main/build-from-source/#making-images-and-updating-v
 endef
 
 # The version of restic binary to be downloaded
-RESTIC_VERSION ?= 0.13.1
+RESTIC_VERSION ?= 0.15.0
 
 CLI_PLATFORMS ?= linux-amd64 linux-arm linux-arm64 darwin-amd64 darwin-arm64 windows-amd64 linux-ppc64le
 BUILDX_PLATFORMS ?= $(subst -,/,$(ARCH))
@@ -96,9 +110,6 @@ else
 	GIT_TREE_STATE ?= clean
 endif
 
-# The default linters used by lint and local-lint
-LINTERS ?= "gosec,goconst,gofmt,goimports,unparam"
-
 ###
 ### These variables should not need tweaking.
 ###
@@ -107,27 +118,29 @@ platform_temp = $(subst -, ,$(ARCH))
 GOOS = $(word 1, $(platform_temp))
 GOARCH = $(word 2, $(platform_temp))
 GOPROXY ?= https://proxy.golang.org
+GOBIN=$$(pwd)/.go/bin
 
 # If you want to build all binaries, see the 'all-build' rule.
 # If you want to build all containers, see the 'all-containers' rule.
 all:
 	@$(MAKE) build
-	@$(MAKE) build BIN=velero-restic-restore-helper
+	@$(MAKE) build BIN=velero-restore-helper
 
 build-%:
 	@$(MAKE) --no-print-directory ARCH=$* build
-	@$(MAKE) --no-print-directory ARCH=$* build BIN=velero-restic-restore-helper
+	@$(MAKE) --no-print-directory ARCH=$* build BIN=velero-restore-helper
 
 all-build: $(addprefix build-, $(CLI_PLATFORMS))
 
-all-containers: container-builder-env
+all-containers:
 	@$(MAKE) --no-print-directory container
-	@$(MAKE) --no-print-directory container BIN=velero-restic-restore-helper
+	@$(MAKE) --no-print-directory container BIN=velero-restore-helper
 
 local: build-dirs
 # Add DEBUG=1 to enable debug locally
 	GOOS=$(GOOS) \
 	GOARCH=$(GOARCH) \
+	GOBIN=$(GOBIN) \
 	VERSION=$(VERSION) \
 	REGISTRY=$(REGISTRY) \
 	PKG=$(PKG) \
@@ -144,6 +157,7 @@ _output/bin/$(GOOS)/$(GOARCH)/$(BIN): build-dirs
 	$(MAKE) shell CMD="-c '\
 		GOOS=$(GOOS) \
 		GOARCH=$(GOARCH) \
+		GOBIN=$(GOBIN) \
 		VERSION=$(VERSION) \
 		REGISTRY=$(REGISTRY) \
 		PKG=$(PKG) \
@@ -163,6 +177,7 @@ shell: build-dirs build-env
 	@# under $GOPATH).
 	@docker run \
 		-e GOFLAGS \
+		-e GOPROXY \
 		-i $(TTY) \
 		--rm \
 		-u $$(id -u):$$(id -g) \
@@ -177,20 +192,6 @@ shell: build-dirs build-env
 		$(BUILDER_IMAGE) \
 		/bin/sh $(CMD)
 
-container-builder-env:
-ifneq ($(BUILDX_ENABLED), true)
-	$(error $(BUILDX_ERROR))
-endif
-	@docker buildx build \
-	--target=builder-env \
-	--build-arg=GOPROXY=$(GOPROXY) \
-	--build-arg=PKG=$(PKG) \
-	--build-arg=VERSION=$(VERSION) \
-	--build-arg=GIT_SHA=$(GIT_SHA) \
-	--build-arg=GIT_TREE_STATE=$(GIT_TREE_STATE) \
-	--build-arg=REGISTRY=$(REGISTRY) \
-	-f $(VELERO_DOCKERFILE) .
-
 container:
 ifneq ($(BUILDX_ENABLED), true)
 	$(error $(BUILDX_ERROR))
@@ -199,6 +200,8 @@ endif
 	--output=type=$(BUILDX_OUTPUT_TYPE) \
 	--platform $(BUILDX_PLATFORMS) \
 	$(addprefix -t , $(IMAGE_TAGS)) \
+	$(addprefix -t , $(GCR_IMAGE_TAGS)) \
+	--build-arg=GOPROXY=$(GOPROXY) \
 	--build-arg=PKG=$(PKG) \
 	--build-arg=BIN=$(BIN) \
 	--build-arg=VERSION=$(VERSION) \
@@ -208,6 +211,12 @@ endif
 	--build-arg=RESTIC_VERSION=$(RESTIC_VERSION) \
 	-f $(VELERO_DOCKERFILE) .
 	@echo "container: $(IMAGE):$(VERSION)"
+ifeq ($(BUILDX_OUTPUT_TYPE)_$(REGISTRY), registry_velero)
+	docker pull $(IMAGE):$(VERSION)
+	rm -f $(BIN)-$(VERSION).tar
+	docker save $(IMAGE):$(VERSION) -o $(BIN)-$(VERSION).tar
+	gzip -f $(BIN)-$(VERSION).tar
+endif
 
 SKIP_TESTS ?=
 test: build-dirs
@@ -227,27 +236,21 @@ endif
 
 lint:
 ifneq ($(SKIP_TESTS), 1)
-	@$(MAKE) shell CMD="-c 'hack/lint.sh $(LINTERS)'"
+	@$(MAKE) shell CMD="-c 'hack/lint.sh'"
 endif
 
 local-lint:
 ifneq ($(SKIP_TESTS), 1)
-	@hack/lint.sh $(LINTERS)
-endif
-
-lint-all:
-ifneq ($(SKIP_TESTS), 1)
-	@$(MAKE) shell CMD="-c 'hack/lint.sh $(LINTERS) true'"
-endif
-
-local-lint-all:
-ifneq ($(SKIP_TESTS), 1)
-	@hack/lint.sh $(LINTERS) true
+	@hack/lint.sh
 endif
 
 update:
 	@$(MAKE) shell CMD="-c 'hack/update-all.sh'"
 
+# update-crd is for development purpose only, it is faster than update, so is a shortcut when you want to generate CRD changes only
+update-crd:
+	@$(MAKE) shell CMD="-c 'hack/update-3generated-crd-code.sh'"	
+
 build-dirs:
 	@mkdir -p _output/bin/$(GOOS)/$(GOARCH)
 	@mkdir -p .go/src/$(PKG) .go/pkg .go/bin .go/std/$(GOOS)/$(GOARCH) .go/go-build .go/golangci-lint
@@ -359,7 +362,7 @@ serve-docs: build-image-hugo
 	-v "$$(pwd)/site:/srv/hugo" \
 	-it -p 1313:1313 \
 	$(HUGO_IMAGE) \
-	hugo server --bind=0.0.0.0 --enableGitInfo=false
+	server --bind=0.0.0.0 --enableGitInfo=false
 # gen-docs generates a new versioned docs directory under site/content/docs.
 # Please read the documentation in the script for instructions on how to use it.
 gen-docs:
@@ -367,4 +370,29 @@ gen-docs:
 
 .PHONY: test-e2e
 test-e2e: local
-	$(MAKE) -e VERSION=$(VERSION) -C test/e2e run
+	$(MAKE) -e VERSION=$(VERSION) -C test/ run-e2e
+
+.PHONY: test-perf
+test-perf: local
+	$(MAKE) -e VERSION=$(VERSION) -C test/ run-perf
+
+go-generate:
+	go generate ./pkg/...
+
+# requires an authenticated gh cli
+# gh: https://cli.github.com/
+# First create a PR
+# gh pr create --title 'Title name' --body 'PR body'
+# by default uses PR title as changelog body but can be overwritten like so
+# make new-changelog CHANGELOG_BODY="Changes you have made"
+new-changelog: GH_LOGIN ?= $(shell gh pr view --json author --jq .author.login 2> /dev/null)
+new-changelog: GH_PR_NUMBER ?= $(shell gh pr view --json number --jq .number 2> /dev/null)
+new-changelog: CHANGELOG_BODY ?= "$(shell gh pr view --json title --jq .title)"
+new-changelog:
+	@if [ "$(GH_LOGIN)" = "" ]; then \
+		echo "branch does not have PR or cli not logged in, try 'gh auth login' or 'gh pr create'"; \
+		exit 1; \
+	fi
+	@mkdir -p ./changelogs/unreleased/ && \
+	echo $(CHANGELOG_BODY) > ./changelogs/unreleased/$(GH_PR_NUMBER)-$(GH_LOGIN) && \
+	echo "\"$(CHANGELOG_BODY)\" added to ./changelogs/unreleased/$(GH_PR_NUMBER)-$(GH_LOGIN)"

OWNERS

@@ -0,0 +1,24 @@
+# This file is used by the [PROW action](https://github.com/jpmcb/prow-github-actions) to approve and merge PRs.
+# The file's format follows the [OWNERS SPEC](https://www.kubernetes.dev/docs/guide/owners/#owners-spec).
+
+# List of usernames who may use /lgtm
+reviewers:
+- @Lyndon-Li
+- @anshulahuja98
+- @blackpiglet
+- @qiuming-best
+- @reasonerjt
+- @shubham-pampattiwar
+- @sseago
+- @ywk253100
+
+# List of usernames who may use /approve
+approvers:
+- @Lyndon-Li
+- @anshulahuja98
+- @blackpiglet
+- @qiuming-best
+- @reasonerjt
+- @shubham-pampattiwar
+- @sseago
+- @ywk253100

README.md

@@ -1,11 +1,13 @@
 ![100]
 
 [![Build Status][1]][2] [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/3811/badge)](https://bestpractices.coreinfrastructure.org/projects/3811)
-
+![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/vmware-tanzu/velero)
 
 ## Overview
 
-Velero (formerly Heptio Ark) gives you tools to back up and restore your Kubernetes cluster resources and persistent volumes. You can run Velero with a public cloud platform or on-premises. Velero lets you:
+Velero (formerly Heptio Ark) gives you tools to back up and restore your Kubernetes cluster resources and persistent volumes. You can run Velero with a public cloud platform or on-premises. 
+
+Velero lets you:
 
 * Take backups of your cluster and restore in case of loss.
 * Migrate cluster resources to other clusters.
@@ -18,7 +20,7 @@ Velero consists of:
 
 ## Documentation
 
-[The documentation][29] provides a getting started guide and information about building from source, architecture, extending Velero, and more.
+[The documentation][29] provides a getting started guide and information about building from source, architecture, extending Velero and more.
 
 Please use the version selector at the top of the site to ensure you are using the appropriate documentation for your version of Velero.
 
@@ -38,21 +40,22 @@ See [the list of releases][6] to find out about feature changes.
 
 The following is a list of the supported Kubernetes versions for each Velero version.
 
-| Velero version | Expected Kubernetes version compatibility| Tested on Kubernetes version|
-|----------------|--------------------|--------------------|
-| 1.9            | 1.16-latest        |  1.20.5, 1.21.2, 1.22.5, 1.23, and 1.24 |
-| 1.8            | 1.16-latest        |  |
-| 1.6.3-1.7.1    | 1.12-latest        ||
-| 1.60-1.6.2     | 1.12-1.21          ||
-| 1.5            | 1.12-1.21          ||
-| 1.4            | 1.10-1.21          | |
+| Velero version | Expected Kubernetes version compatibility | Tested on Kubernetes version        |
+|----------------|-------------------------------------------|-------------------------------------|
+| 1.15           | 1.18-latest                               | 1.28.8, 1.29.8, 1.30.4 and 1.31.1   |
+| 1.14           | 1.18-latest                               | 1.27.9, 1.28.9, and 1.29.4          |
+| 1.13           | 1.18-latest                               | 1.26.5, 1.27.3, 1.27.8, and 1.28.3  |
+| 1.12           | 1.18-latest                               | 1.25.7, 1.26.5, 1.26.7, and 1.27.3  |
+| 1.11           | 1.18-latest                               | 1.23.10, 1.24.9, 1.25.5, and 1.26.1 |
 
 Velero supports IPv4, IPv6, and dual stack environments. Support for this was tested against Velero v1.8.
 
-The Velero maintainers are continuously working to expand testing coverage, but are not able to test every combination of Velero and supported Kubernetes versions for each Velero release. The table above is meant to track the current testing coverage and the expected supported Kubernetes versions for each Velero version. If you have a question about test coverage before v1.9, please reach out in the [#velero-users](https://kubernetes.slack.com/archives/C6VCGP4MT) Slack channel.
+The Velero maintainers are continuously working to expand testing coverage, but are not able to test every combination of Velero and supported Kubernetes versions for each Velero release. The table above is meant to track the current testing coverage and the expected supported Kubernetes versions for each Velero version.
 
 If you are interested in using a different version of Kubernetes with a given Velero version, we'd recommend that you perform testing before installing or upgrading your environment. For full information around capabilities within a release, also see the Velero [release notes](https://github.com/vmware-tanzu/velero/releases) or Kubernetes [release notes](https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG). See the Velero [support page](https://velero.io/docs/latest/support-process/) for information about supported versions of Velero.
 
+For each release, Velero maintainers run the test to ensure the upgrade path from n-2 minor release.  For example, before the release of v1.10.x, the test will verify that the backup created by v1.9.x and v1.8.x can be restored using the build to be tagged as v1.10.x.
+
 [1]: https://github.com/vmware-tanzu/velero/workflows/Main%20CI/badge.svg
 [2]: https://github.com/vmware-tanzu/velero/actions?query=workflow%3A"Main+CI"
 [4]: https://github.com/vmware-tanzu/velero/issues

Tiltfile

@@ -7,17 +7,19 @@ k8s_yaml([
     'config/crd/v1/bases/velero.io_downloadrequests.yaml',
     'config/crd/v1/bases/velero.io_podvolumebackups.yaml',
     'config/crd/v1/bases/velero.io_podvolumerestores.yaml',
-    'config/crd/v1/bases/velero.io_resticrepositories.yaml',
+    'config/crd/v1/bases/velero.io_backuprepositories.yaml',
     'config/crd/v1/bases/velero.io_restores.yaml',
     'config/crd/v1/bases/velero.io_schedules.yaml',
     'config/crd/v1/bases/velero.io_serverstatusrequests.yaml',
     'config/crd/v1/bases/velero.io_volumesnapshotlocations.yaml',
+    'config/crd/v2alpha1/bases/velero.io_datauploads.yaml',
+    'config/crd/v2alpha1/bases/velero.io_datadownloads.yaml',    
 ])
 
 # default values
 settings = {
     "default_registry": "docker.io/velero",
-    "enable_restic": False,
+    "use_node_agent": False,
     "enable_debug": False,
     "debug_continue_on_start": True,  # Continue the velero process by default when in debug mode
     "create_backup_locations": False,
@@ -34,9 +36,9 @@ k8s_yaml(kustomize('tilt-resources'))
 k8s_yaml('tilt-resources/deployment.yaml')
 if settings.get("enable_debug"):
     k8s_resource('velero', port_forwards = '2345')
-    # TODO: Need to figure out how to apply port forwards for all restic pods
-if settings.get("enable_restic"):
-    k8s_yaml('tilt-resources/restic.yaml')
+    # TODO: Need to figure out how to apply port forwards for all node-agent pods
+if settings.get("use_node_agent"):
+    k8s_yaml('tilt-resources/node-agent.yaml')
 if settings.get("create_backup_locations"):
     k8s_yaml('tilt-resources/velero_v1_backupstoragelocation.yaml')
 if settings.get("setup-minio"):
@@ -50,7 +52,7 @@ git_sha = str(local("git rev-parse HEAD", quiet = True, echo_off = True)).strip(
 
 tilt_helper_dockerfile_header = """
 # Tilt image
-FROM golang:1.17 as tilt-helper
+FROM golang:1.22.10 as tilt-helper
 
 # Support live reloading with Tilt
 RUN wget --output-document /restart.sh --quiet https://raw.githubusercontent.com/windmilleng/rerun-process-wrapper/master/restart.sh  && \
@@ -60,9 +62,9 @@ RUN wget --output-document /restart.sh --quiet https://raw.githubusercontent.com
 
 additional_docker_helper_commands = """
 # Install delve to allow debugging
-RUN go get github.com/go-delve/delve/cmd/dlv
+RUN go install github.com/go-delve/delve/cmd/dlv@latest
 
-RUN wget -qO- https://dl.k8s.io/v1.19.2/kubernetes-client-linux-amd64.tar.gz | tar xvz
+RUN wget -qO- https://dl.k8s.io/v1.25.2/kubernetes-client-linux-amd64.tar.gz | tar xvz
 RUN wget -qO- https://get.docker.com | sh
 """
 
@@ -103,12 +105,12 @@ local_resource(
 
 local_resource(
     "restic_binary",
-    cmd = 'cd ' + '.' + ';mkdir -p _tiltbuild/restic; BIN=velero GOOS=linux GOARCH=amd64 RESTIC_VERSION=0.13.1 OUTPUT_DIR=_tiltbuild/restic ./hack/download-restic.sh',
+    cmd = 'cd ' + '.' + ';mkdir -p _tiltbuild/restic; BIN=velero GOOS=linux GOARCH=amd64 GOARM="" RESTIC_VERSION=0.13.1 OUTPUT_DIR=_tiltbuild/restic ./hack/build-restic.sh',
 )
 
 # Note: we need a distro with a bash shell to exec into the Velero container
 tilt_dockerfile_header = """
-FROM ubuntu:focal as tilt
+FROM ubuntu:22.04 as tilt
 
 RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -qq -y ca-certificates tzdata && rm -rf /var/lib/apt/lists/*
 
@@ -216,7 +218,7 @@ def enable_provider(provider):
 
     # Note: we need a distro with a shell to do a copy of the plugin binary
     tilt_dockerfile_header = """
-    FROM ubuntu:focal as tilt
+    FROM ubuntu:22.04 as tilt
     WORKDIR /
     COPY --from=tilt-helper /start.sh .
     COPY --from=tilt-helper /restart.sh .

assets/README.md

@@ -1,6 +1,6 @@
 # Velero Assets
 
-This folder contains logo images for Velero in gray (for light backgrounds) and white (for dark backgrounds like black tshirts or dark mode!) – horizontal and stacked… in .eps and .svg.
+This folder contains logo images for Velero in gray (for light backgrounds) and white (for dark backgrounds like black t-shirts or dark mode!) – horizontal and stacked… in .eps and .svg.
 
 ## Some general guidelines for usage
 

changelogs/CHANGELOG-0.9.md

@@ -154,7 +154,7 @@
   * Skip completed jobs and pods when restoring (#463, @nrb)
   * Set namespace correctly when syncing backups from object storage (#472, @skriss)
   * When building on macOS, bind-mount volumes with delegated config (#478, @skriss)
-  * Add replica sets and daemonsets to cohabitating resources so they're not backed up twice (#482 #485, @skriss)
+  * Add replica sets and daemonsets to cohabiting resources so they're not backed up twice (#482 #485, @skriss)
   * Shut down the Ark server gracefully on SIGINT/SIGTERM (#483, @skriss)
   * Only back up resources that support GET and DELETE in addition to LIST and CREATE (#486, @nrb)
   * Show a better error message when trying to get an incomplete restore's logs (#496, @nrb)

changelogs/CHANGELOG-1.10.md

@@ -0,0 +1,190 @@
+  ## v1.10.0
+### 2022-11-23
+
+### Download
+https://github.com/vmware-tanzu/velero/releases/tag/v1.10.0
+
+### Container Image
+`velero/velero:v1.10.0`
+
+### Documentation
+https://velero.io/docs/v1.10/
+
+### Upgrading
+https://velero.io/docs/v1.10/upgrade-to-1.10/
+
+### Highlights
+
+#### Unified Repository and Kopia integration
+In this release, we introduced the Unified Repository architecture to build a data path where data movers and the backup repository are decoupled and a unified backup repository could serve various data movement activities.
+
+In this release, we also deeply integrate Velero with Kopia, specifically, Kopia's uploader modules are isolated as a generic file system uploader; Kopia's repository modules are encapsulated as the unified backup repository.
+
+For more information, refer to the [design document](https://github.com/vmware-tanzu/velero/blob/v1.10.0/design/unified-repo-and-kopia-integration/unified-repo-and-kopia-integration.md).
+
+#### File system backup refactor
+Velero's file system backup (a.k.s. pod volume backup or formerly restic backup) is refactored as the first user of the Unified Repository architecture. Specifically, we added a new path, the Kopia path, besides the existing Restic path. While Restic path is still available and set as default, you can opt in Kopia path by specifying the `uploader-type` parameter at installation time. Meanwhile, you are free to restore from existing backups under either path, Velero dynamically switches to the correct path to process the restore.
+
+Because of the new path, we renamed some modules and parameters, refer to the Break Changes section for more details.
+
+For more information, visit the [file system backup document](https://velero.io/docs/v1.10/file-system-backup/) and [v1.10 upgrade guide document](https://velero.io/docs/v1.10/upgrade-to-1.10/).
+
+Meanwhile, we've created a performance guide for both Restic path and Kopia path, which helps you to choose between the two paths and provides you the best practice to configure them under different scenarios. Please note that the results in the guide are based on our testing environments, you may get different results when testing in your own ones. For more information, visit the [performance guide document](https://velero.io/docs/v1.10/performance-guidance/).
+
+#### Plugin versioning V1 refactor
+In this release, Velero moves plugins BackupItemAction, RestoreItemAction and VolumeSnapshotterAction to version v1, this allows future plugin changes that do not support backward compatibility, so is a preparation for various complex tasks, for example, data movement tasks.
+For more information, refer to the [plugin versioning design document](https://github.com/vmware-tanzu/velero/blob/v1.10.0/design/plugin-versioning.md).
+
+#### Refactor the controllers using Kubebuilder v3
+In this release we continued our code modernization work, rewriting some controllers using Kubebuilder v3. This work is ongoing and we will continue to make progress in future releases.
+
+#### Add credentials to volume snapshot locations
+In this release, we enabled dedicate credentials options to volume snapshot locations so that you can specify credentials per volume snapshot location as same as backup storage location.
+
+For more information, please visit the [locations document](https://velero.io/docs/v1.10/locations/).
+
+#### CSI snapshot enhancements
+In this release we added several changes to enhance the robustness of CSI snapshot procedures, for example, some protection code for error handling, and a mechanism to skip exclusion checks so that CSI snapshot works with various backup resource filters.
+
+#### Backup schedule pause/unpause
+In this release, Velero supports to pause/unpause a backup schedule during or after its creation. Specifically:
+
+At creation time, you can specify `–paused` flag to `velero schedule create` command, if so, you will create a paused schedule that will not run until it is unpaused
+After creation, you can run `velero schedule pause` or `velero schedule unpause` command to pause/unpause a schedule
+
+#### Runtime and dependencies
+In order to fix CVEs, we changed Velero's runtime and dependencies as follows:
+
+Bump go runtime to v1.18.8
+Bump some core dependent libraries to newer versions
+Compile Restic (v0.13.1) with go 1.18.8 instead of packaging the official binary
+
+
+#### Breaking changes
+Due to file system backup refactor, below modules and parameters name have been changed in this release:
+
+`restic` daemonset is renamed to `node-agent`
+`resticRepository` CR is renamed to `backupRepository`
+`velero restic repo` command is renamed to `velero repo`
+`velero-restic-credentials` secret is renamed to `velero-repo-credentials`
+`default-volumes-to-restic` parameter is renamed to `default-volumes-to-fs-backup`
+`restic-timeout` parameter is renamed to `fs-backup-timeout`
+`default-restic-prune-frequency` parameter is renamed to `default-repo-maintain-frequency`
+
+#### Upgrade
+Due to the major changes of file system backup, the old upgrade steps are not suitable any more. For the new upgrade steps, visit [v1.10 upgrade guide document](https://velero.io/docs/v1.10/upgrade-to-1.10/).
+
+#### Limitations/Known issues
+In this release, Kopia backup repository (so the Kopia path of file system backup) doesn't support self signed certificate for S3 compatible storage. To track this problem, refer to this [Velero issue](https://github.com/vmware-tanzu/velero/issues/5123) or [Kopia issue](https://github.com/kopia/kopia/issues/1443). 
+
+Due to the code change in Velero, there will be some code change required in vSphere plugin, without which the functionality may be impacted.  Therefore, if you are using vSphere plugin in your workflow, please hold the upgrade until the issue [#485](https://github.com/vmware-tanzu/velero-plugin-for-vsphere/issues/485) is fixed in vSphere plugin.
+
+### All changes
+  
+  * Restore ClusterBootstrap before Cluster otherwise a new default ClusterBootstrap object is create for the cluster  (#5616, @ywk253100)
+  * Add compile restic binary for CVE fix  (#5574, @qiuming-best)
+  * Fix controller problematic log output  (#5572, @qiuming-best)
+  * Enhance the restore priorities list to support specifying the low prioritized resources that need to be restored in the last (#5535, @ywk253100)
+  * fix restic backup progress error (#5534, @qiuming-best)
+  * fix restic backup failure with self-signed certification backend storage (#5526, @qiuming-best)
+  * Add credential store in backup deletion controller to support VSL credential. (#5521, @blackpiglet)
+  * Fix issue 5505: the pod volume backups/restores except the first one fail under the kopia path if "AZURE_CLOUD_NAME" is specified (#5512, @Lyndon-Li)
+  * After Pod Volume Backup/Restore refactor, remove all the unreasonable appearance of "restic" word from documents (#5499, @Lyndon-Li)
+  * Refactor Pod Volume Backup/Restore doc to match the new behavior (#5484, @Lyndon-Li)
+  * Remove redundancy code block left by #5388. (#5483, @blackpiglet)
+  * Issue fix 5477: create the common way to support S3 compatible object storages that work for both Restic and Kopia; Keep the resticRepoPrefix parameter for compatibility (#5478, @Lyndon-Li)
+  * Update the k8s.io dependencies to 0.24.0.
+This also required an update to github.com/bombsimon/logrusr/v3.
+Removed the `WithClusterName` method
+as it is a "legacy field that was
+always cleared by the system and never used" as per upstream k8s
+https://github.com/kubernetes/apimachinery/blob/release-1.24/pkg/apis/meta/v1/types.go#L257-L259 (#5471, @kcboyle)
+  * Add v1.10 velero upgrade doc (#5468, @qiuming-best)
+  * Upgrade velero docker image to use go 1.18 and upgrade golangci-lint to 1.45.0 (#5459, @Lyndon-Li)
+  * Add VolumeSnapshot client back. (#5449, @blackpiglet)
+  * Change subcommand `velero restic repo` to `velero repo` (#5446, @allenxu404)
+  * Remove irrational "Restic" names in Velero code after the PVBR refactor (#5444, @Lyndon-Li)
+  * moved RIA execute input/output structs back to velero package (#5441, @sseago)
+  * Rename Velero pod volume restore init helper from "velero-restic-restore-helper" to "velero-restore-helper" (#5432, @Lyndon-Li)
+  * Skip the exclusion check for additional resources returned by BIA (#5429, @reasonerjt)
+  * Change B/R describe CLI to support Kopia (#5412, @allenxu404)
+  * Add nil check before execution of csi snapshot delete (#5401, @shubham-pampattiwar)
+  * update velero using klog to version v2.9.0 (#5396, @blackpiglet)
+  * Fix Test_prepareBackupRequest_BackupStorageLocation UT failure. (#5394, @blackpiglet)
+  * Rename Velero daemonset from "restic" to "node-agent" (#5390, @Lyndon-Li)
+  * Add some corner cases checking for CSI snapshot in backup controller. (#5388, @blackpiglet)
+  * Fix issue 5386: Velero providers a full URL as the S3Url while the underlying minio client only accept the host part of the URL as the endpoint and the schema should be specified separately. (#5387, @Lyndon-Li)
+  * Fix restore error with flag namespace-mappings (#5377, @qiuming-best)
+  * Pod Volume Backup/Restore Refactor: Rename parameters in CRDs and commands to remove "Restic" word (#5370, @Lyndon-Li)
+  * Added backupController's UT to test the prepareBackupRequest() method BackupStorageLocation processing logic (#5362, @niulechuan)
+  * Fix a repoEnsurer problem introduced by the refactor - The repoEnsurer didn't check "" state of BackupRepository, as a result, the function GetBackupRepository always returns without an error even though the ensreReady is specified. (#5359, @Lyndon-Li)
+  * Add E2E test for schedule backup (#5355, @danfengliu)
+  * Add useOwnerReferencesInBackup field doc for schedule. (#5353, @cleverhu)
+  * Clarify the help message for the default value of parameter --snapshot-volumes, when it's not set. (#5350, @blackpiglet)
+  * Fix restore cmd extraflag overwrite bug (#5347, @qiuming-best)
+  * Resolve gopkg.in/yaml.v3 vulnerabilities by upgrading gopkg.in/yaml.v3 to v3.0.1 (#5344, @kaovilai)
+  * Increase ensure restic repository timeout to 5m (#5335, @shubham-pampattiwar)
+  * Add opt-in and opt-out PersistentVolume backup to E2E tests (#5331, @danfengliu)
+  * Cancel downloadRequest when timeout without downloadURL (#5329, @kaovilai)
+  * Fix PVB finds wrong parent snapshot (#5322, @qiuming-best)
+  * Fix issue 4874 and 4752: check the daemonset pod is running in the node where the workload pod resides before running the PVB for the pod (#5319, @Lyndon-Li)
+  * plugin versioning v1 refactor for VolumeSnapshotter (#5318, @sseago)
+  * Change the status of restore to completed from partially failed when restore empty backup (#5314, @allenxu404)
+  * RestoreItemAction v1 refactoring for plugin api versioning (#5312, @sseago)
+  * Refactor the repoEnsurer code to use controller runtime client and wrap some common BackupRepository operations to share with other modules (#5308, @Lyndon-Li)
+  * Remove snapshot related lister, informer and client from backup controller. (#5299, @jxun)
+  * Remove github.com/apex/log logger. (#5297, @blackpiglet)
+  * change CSISnapshotTimeout from pointer to normal variables. (#5294, @cleverhu)
+  * Optimize code for restore exists resources. (#5293, @cleverhu)
+  * Add more detailed comments for labels columns. (#5291, @cleverhu)
+  * Add backup status checking in schedule controller. (#5283, @blackpiglet)
+  * Add changes for problems/enhancements found during smoking test for Kopia pod volume backup/restore (#5282, @Lyndon-Li)
+  * Support pause/unpause schedules (#5279, @ywk253100)
+  *  plugin/clientmgmt refactoring for BackupItemAction v1 (#5271, @sseago)
+  * Don't move velero v1 plugins to new proto dir (#5263, @sseago)
+  * Fill gaps for Kopia path of PVBR: integrate Repo Manager with Unified Repo; pass UploaderType to PVBR backupper and restorer; pass RepositoryType to BackupRepository controller and Repo Ensurer (#5259, @Lyndon-Li)
+  * Add csiSnapshotTimeout for describe backup (#5252, @cleverhu)
+  * equip gc controller with configurable frequency (#5248, @allenxu404)
+  * Fix nil pointer panic when restoring StatefulSets (#5247, @divolgin)
+  * Controller refactor code modifications. (#5241, @jxun)
+  * Fix edge cases for already exists resources (#5239, @shubham-pampattiwar)
+  * Check for empty ns list before checking nslist[0] (#5236, @sseago)
+  * Remove reference to non-existent doc (#5234, @reasonerjt)
+  * Add changes for Kopia Integration: Kopia Lib - method implementation. Add changes to write Kopia Repository logs to Velero log (#5233, @Lyndon-Li)
+  * Add changes for Kopia Integration: Kopia Lib - initialize Kopia repo (#5231, @Lyndon-Li)
+  * Uploader Implementation: Kopia backup and restore (#5221, @qiuming-best)
+  * Migrate backup sync controller from code-generator to kubebuilder. (#5218, @jxun)
+  * check vsc null pointer (#5217, @lilongfeng0902)
+  * Refactor GCController with kubebuilder (#5215, @allenxu404)
+  * Uploader Implementation: Restic backup and restore (#5214, @qiuming-best)
+  * Add parameter "uploader-type" to velero server (#5212, @reasonerjt)
+  * Add annotation "pv.kubernetes.io/migrated-to" for CSI checking. (#5181, @jxun)
+  * Add changes for Kopia Integration: Unified Repository Provider - method implementation (#5179, @Lyndon-Li)
+  * Treat namespaces with exclude label as excludedNamespaces
+Related issue: #2413 (#5178, @allenxu404)
+  * Reduce CRD size. (#5174, @jxun)
+  * Fix restic backups to multiple backup storage locations bug (#5172, @qiuming-best)
+  * Add changes for Kopia Integration: Unified Repository Provider - Repo Password (#5167, @Lyndon-Li)
+  * Skip registering "crd-remap-version" plugin when feature flag "EnableAPIGroupVersions" is set (#5165, @reasonerjt)
+  * Kopia uploader integration on shim progress uploader module (#5163, @qiuming-best)
+  * Add labeled and unlabeled events for PR changelog check action. (#5157, @jxun)
+  * VolumeSnapshotLocation refactor with kubebuilder. (#5148, @jxun)
+  * Delay CA file deletion in PVB controller. (#5145, @jxun)
+  * This commit splits the pkg/restic package into several packages to support Kopia integration works (#5143, @ywk253100)
+  * Kopia Integration: Add the Unified Repository Interface definition. Kopia Integration: Add the changes for Unified Repository storage config. Related Issues; #5076, #5080 (#5142, @Lyndon-Li)
+  * Update the CRD for kopia integration (#5135, @reasonerjt)
+  * Let "make shell xxx" respect GOPROXY (#5128, @reasonerjt)
+  * Modify BackupStoreGetter to avoid BSL spec changes (#5122, @sseago)
+  * Dump stack trace when the plugin server handles panic (#5110, @reasonerjt)
+  * Make CSI snapshot creation timeout configurable. (#5104, @jxun)
+  *  Fix bsl validation bug: the BSL is validated continually and doesn't respect the validation period configured (#5101, @ywk253100)
+  * Exclude "csinodes.storage.k8s.io" and "volumeattachments.storage.k8s.io" from restore by default. (#5064, @jxun)
+  * Move 'velero.io/exclude-from-backup' label string to const (#5053, @niulechuan)
+  * Modify Github actions. (#5052, @jxun)
+  * Fix typo in doc, in https://velero.io/docs/main/restore-reference/ "Restore order" section, "Mamespace" should be "Namespace". (#5051, @niulechuan)
+  * Delete opened issues triage action. (#5041, @jxun)
+  * When spec.RestoreStatus is empty, don't restore status (#5008, @sseago)
+  * Added DownloadTargetKindCSIBackupVolumeSnapshots for retrieving the signed URL to download only the `<backup name>`-csi-volumesnapshots.json.gz  and DownloadTargetKindCSIBackupVolumeSnapshotContents to download only `<backup name>`-csi-volumesnapshotcontents.json.gz in the DownloadRequest CR structure. These files are already present in the backup layout.  (#4980, @anshulahuja98)
+  * Refactor BackupItemAction proto and related code to backupitemaction/v1 package.  This is part of implementation of the plugin version design https://github.com/vmware-tanzu/velero/blob/main/design/plugin-versioning.md (#4943, @phuongatemc)
+  * Unified Repository Design (#4926, @Lyndon-Li)
+  * Add credentials to volume snapshot locations (#4864, @sseago)

changelogs/CHANGELOG-1.11.md

@@ -0,0 +1,126 @@
+## v1.11
+### 2023-04-07
+
+### Download
+https://github.com/vmware-tanzu/velero/releases/tag/v1.11.0
+
+### Container Image
+`velero/velero:v1.11.0`
+
+### Documentation
+https://velero.io/docs/v1.11/
+
+### Upgrading
+https://velero.io/docs/v1.11/upgrade-to-1.11/
+
+### Highlights
+
+#### BackupItemAction v2
+This feature implements the BackupItemAction v2. BIA v2 has two new methods: Progress() and Cancel() and modifies the Execute() return value.
+
+The API change is needed to facilitate long-running BackupItemAction plugin actions that may not be complete when the Execute() method returns. This will allow long-running BackupItemAction plugin actions to continue in the background while the Velero moves to the following plugin or the next item.
+
+#### RestoreItemAction v2
+This feature implemented the RestoreItemAction v2. RIA v2 has three new methods: Progress(), Cancel(), and AreAdditionalItemsReady(), and it modifies RestoreItemActionExecuteOutput() structure in the RIA return value.
+
+The Progress() and Cancel() methods are needed to facilitate long-running RestoreItemAction plugin actions that may not be complete when the Execute() method returns. This will allow long-running RestoreItemAction plugin actions to continue in the background while the Velero moves to the following plugin or the next item. The AreAdditionalItemsReady() method is needed to allow plugins to tell Velero to wait until the returned additional items have been restored and are ready for use in the cluster before restoring the current item.
+
+#### Plugin Progress Monitoring
+This is intended as a replacement for the previously-approved Upload Progress Monitoring design ([Upload Progress Monitoring](https://github.com/vmware-tanzu/velero/blob/main/design/upload-progress.md)) to expand the supported use cases beyond snapshot upload to include what was previously called Async Backup/Restore Item Actions.
+
+#### Flexible resource policy that can filter volumes to skip in the backup
+This feature provides a flexible policy to filter volumes in the backup without requiring patching any labels or annotations to the pods or volumes. This policy is configured as k8s ConfigMap and maintained by the users themselves, and it can be extended to more scenarios in the future. By now, the policy rules out volumes from backup depending on the CSI driver, NFS setting, volume size, and StorageClass setting. Please refer to [policy API design](https://github.com/vmware-tanzu/velero/blob/main/design/Implemented/handle-backup-of-volumes-by-resources-filters.md#api-design) for the policy's ConifgMap format. It is not guaranteed to work on unofficial third-party plugins as it may not follow the existing backup workflow code logic of Velero.
+
+#### Resource Filters that can distinguish cluster scope and namespace scope resources
+This feature adds four new resource filters for backup. The new filters are separated into cluster scope and namespace scope. Before this feature, Velero could not filter cluster scope resources precisely. This feature provides the ability and refactors existing resource filter parameters.
+
+#### Add a parameter for setting the Velero server connection with the k8s API server's timeout
+In Velero, some code pieces need to communicate with the k8s API server. Before v1.11, these code pieces used hard-code timeout settings. This feature adds a resource-timeout parameter in the velero server binary to make it configurable.
+
+#### Add resource list in the output of the restore describe command
+Before this feature, Velero restore didn't have a restored resources list as the Velero backup. It's not convenient for users to learn what is restored. This feature adds the resources list and the handling result of the resources (including created, updated, failed, and skipped).
+
+#### Refactor controllers with controller-runtime
+In v1.11, Backup Controller and Restore controller are refactored with controller-runtime. Till v1.11, all Velero controllers use the controller-runtime framework.
+
+#### Runtime and dependencies
+To fix CVEs and keep pace with Golang, Velero made changes as follows:
+* Bump Golang runtime to v1.19.8.
+* Bump several dependent libraries to new versions.
+* Compile Restic (v0.15.0) with Golang v1.19.8 instead of packaging the official binary.
+
+
+### Breaking changes
+* The Velero CSI plugin now determines whether to restore Volume's data from snapshots on the restore's restorePVs setting. Before v1.11, the CSI plugin doesn't check the restorePVs parameter setting. 
+
+
+### Limitations/Known issues
+* The Flexible resource policy that can filter volumes to skip in the backup is not guaranteed to work on unofficial third-party plugins because the plugins may not follow the existing backup workflow code logic of Velero. The ConfigMap used as the policy is supposed to be maintained by users.
+
+
+### All Changes
+* Modify new scope resource filters name. (#6089, @blackpiglet)
+* Make Velero not exits when EnableCSI is on and CSI snapshot not installed (#6062, @blackpiglet)
+* Restore Services before Clusters (#6057, @ywk253100)
+* Fixed backup deletion bug related to async operations (#6041, @sseago)
+* Update Golang version to v1.19 for branch main. (#6039, @blackpiglet)
+* Fix issue #5972, don't assume errorField as error type when dealing with logger.WithError (#6028, @Lyndon-Li)
+* distinguish between New and InProgress operations (#6012, @sseago)
+* Modify golangci.yaml file. Resolve found lint issues. (#6008, @blackpiglet)
+* Remove Reference of itemsnapshotter (#5997, @reasonerjt)
+* minor fixes for backup_operations_controller (#5996, @sseago)
+* RIAv2 async operations controller work (#5993, @sseago)
+* Follow-on fixes for BIAv2 controller work (#5971, @sseago)
+* Refactor backup controller based on the controller-runtime framework. (#5969, @qiuming-best)
+* Fix client wait problem after async operation change, velero backup/restore --wait should check a full list of the terminal status (#5964, @Lyndon-Li)
+* Fix issue #5935, refactor the logics for backup/restore persistent log, so as to remove the contest to gzip writer (#5956, @Lyndon-Li)
+* Switch the base image to distroless/base-nossl-debian11 to reduce the CVE triage efforts (#5939, @ywk253100)
+* Wait for additional items to be ready before restoring current item (#5933, @sseago)
+* Add configurable server setting for default timeouts (#5926, @eemcmullan)
+* Add warning/error result to cmd `velero backup describe` (#5916, @allenxu404)
+* Fix Dependabot alerts. Use 1.18 and 1.19 golang instead of patch image in dockerfile. Add release-1.10 and release-1.9 in Trivy daily scan. (#5911, @blackpiglet)
+* Update client-go to v0.25.6 (#5907, @kaovilai)
+* Limit the concurrent number for backup's VolumeSnapshot operation. (#5900, @blackpiglet)
+* Fix goreleaser issue for resolving tags and updated it's version. (#5899, @anshulahuja98)
+* This is to fix issue 5881, enhance the PVB tracker in two modes, Track and Taken (#5894, @Lyndon-Li)
+* Add labels for velero installed namespace to support PSA. (#5873, @blackpiglet)
+* Add restored resource list in the restore describe command (#5867, @ywk253100)
+* Add a json output to cmd velero backup describe  (#5865, @allenxu404)
+* Make restore controller adopting the controller-runtime framework. (#5864, @blackpiglet)
+* Replace k8s.io/apimachinery/pkg/util/clock with k8s.io/utils/clock (#5859, @hezhizhen)
+* Restore finalizer and managedFields of metadata during the restoration (#5853, @ywk253100)
+* BIAv2 async operations controller work (#5849, @sseago)
+* Add secret restore item action to handle service account token secret (#5843, @ywk253100)
+* Add new resource filters can separate cluster and namespace scope resources. (#5838, @blackpiglet)
+* Correct PVB/PVR Failed Phase patching during startup (#5828, @kaovilai)
+* bump up golang net to fix CVE-2022-41721 (#5812, @Lyndon-Li)
+* Update CRD descriptions for SnapshotVolumes and restorePVs (#5807, @shubham-pampattiwar)
+* Add mapped selected-node existence check (#5806, @blackpiglet)
+* Add option "--service-account-name" to install cmd (#5802, @reasonerjt)
+* Enable staticcheck linter. (#5788, @blackpiglet)
+* Set Kopia IgnoreUnknownTypes in ErrorHandlingPolicy to True for ignoring backup unknown file type (#5786, @qiuming-best)
+* Bump up Restic version to 0.15.0  (#5784, @qiuming-best)
+* Add File system backup related metrics to Grafana dashboard
+  - Add metrics backup_warning_total for record of total warnings
+  - Add metrics backup_last_status for record of last status of the backup  (#5779, @allenxu404)
+* Design for Handling backup of volumes by resources filters (#5773, @qiuming-best)
+* Add PR container build action, which will not push image. Add GOARM parameter. (#5771, @blackpiglet)
+* Fix issue 5458, track pod volume backup until the CR is submitted in case it is skipped half way (#5769, @Lyndon-Li)
+* Fix issue 5226, invalidate the related backup repositories whenever the backup storage info change in BSL (#5768, @Lyndon-Li)
+* Add Restic builder in Dockerfile, and keep the used built Golang image version in accordance with upstream Restic. (#5764, @blackpiglet)
+* Fix issue 5043, after the restore pod is scheduled, check if the node-agent pod is running in the same node. (#5760, @Lyndon-Li)
+* Remove restore controller's redundant client. (#5759, @blackpiglet)
+* Define itemoperations.json format and update DownloadRequest API (#5752, @sseago)
+* Add Trivy nightly scan. (#5740, @jxun)
+* Fix issue 5696, check if the repo is still openable before running the prune and forget operation, if not, try to reconnect the repo (#5715, @Lyndon-Li)
+* Fix error with Restic backup empty volumes (#5713, @qiuming-best)
+* new backup and restore phases to support async plugin operations:
+  - WaitingForPluginOperations
+  - WaitingForPluginOperationsPartiallyFailed (#5710, @sseago)
+* Prevent nil panic on exec restore hooks (#5675, @dymurray)
+* Fix CVEs scanned by trivy (#5653, @qiuming-best)
+* Publish backupresults json to enhance error info during backups. (#5576, @anshulahuja98)
+* RestoreItemAction v2 API implementation (#5569, @sseago)
+* add new RestoreItemAction of "velero.io/change-image-name" to handle the issue mentioned at #5519 (#5540, @wenterjoy)
+* BackupItemAction v2 API implementation (#5442, @sseago)
+* Proposal to separate resource filter into cluster scope and namespace scope (#5333, @blackpiglet)

changelogs/CHANGELOG-1.12.md

@@ -0,0 +1,134 @@
+## v1.12
+### 2023-08-18
+
+### Download
+https://github.com/vmware-tanzu/velero/releases/tag/v1.12.0
+
+### Container Image
+`velero/velero:v1.12.0`
+
+### Documentation
+https://velero.io/docs/v1.12/
+
+### Upgrading
+https://velero.io/docs/v1.12/upgrade-to-1.12/
+
+### Highlights
+
+#### CSI Snapshot Data Movement
+CSI Snapshot Data Movement refers to back up CSI snapshot data from the volatile and limited production environment into durable, heterogeneous, and scalable backup storage in a consistent manner; and restore the data to volumes in the original or alternative environment.
+
+CSI Snapshot Data Movement is useful in below scenarios:
+
+* For on-premises users, the storage usually doesn't support durable snapshots, so it is impossible/less efficient/cost ineffective to keep volume snapshots by the storage This feature helps to move the snapshot data to a storage with lower cost and larger scale for long time preservation.
+* For public cloud users, this feature helps users to fulfill the multiple cloud strategy. It allows users to back up volume snapshots from one cloud provider and preserve or restore the data to another cloud provider. Then users will be free to flow their business data across cloud providers based on Velero backup and restore
+
+CSI Snapshot Data Movement is built according to the Volume Snapshot Data Movement design ([Volume Snapshot Data Movement](https://github.com/vmware-tanzu/velero/blob/main/design/Implemented/unified-repo-and-kopia-integration/unified-repo-and-kopia-integration.md)). More details can be found in the design.
+
+#### Resource Modifiers
+In many use cases, customers often need to substitute specific values in Kubernetes resources during the restoration process like changing the namespace, changing the storage class, etc. 
+
+To address this need, Resource Modifiers (also known as JSON Substitutions) offer a generic solution in the restore workflow. It allows the user to define filters for specific resources and then specify a JSON patch (operator, path, value) to apply to the resource. This feature simplifies the process of making substitutions without requiring the implementation of a new RestoreItemAction plugin. More details can be found in Volume Snapshot Resource Modifiers design ([Resource Modifiers](https://github.com/vmware-tanzu/velero/blob/main/design/Implemented/json-substitution-action-design.md)).
+
+#### Multiple VolumeSnapshotClasses
+Prior to version 1.12, the Velero CSI plugin would choose the VolumeSnapshotClass in the cluster based on matching driver names and the presence of the "velero.io/csi-volumesnapshot-class" label. However,  this approach proved inadequate for many user scenarios.
+
+With the introduction of version 1.12, Velero now offers support for multiple VolumeSnapshotClasses in the CSI Plugin, enabling users to select a specific class for a particular backup. More details can be found in Multiple VolumeSnapshotClasses design ([Multiple VolumeSnapshotClasses](https://github.com/vmware-tanzu/velero/blob/main/design/Implemented/multiple-csi-volumesnapshotclass-support.md)).
+
+#### Restore Finalizer
+Before v1.12, the restore controller would only delete restore resources but wouldn’t delete restore data from the backup storage location when the command `velero restore delete` was executed. The only chance Velero deletes restores data from the backup storage location is when the associated backup is deleted.
+
+In this version, Velero introduces a finalizer that ensures the cleanup of all associated data for restores when running the command `velero restore delete`.
+
+#### Runtime and dependencies
+To fix CVEs and keep pace with Golang, Velero made changes as follows:
+* Bump Golang runtime to v1.20.7.
+* Bump several dependent libraries to new versions.
+* Bump Kopia to v0.13.
+
+
+### Breaking changes
+* Prior to v1.12, the parameter `uploader-type` for Velero installation had a default value of "restic". However, starting from this version, the default value has been changed to "kopia". This means that Velero will now use Kopia as the default path for file system backup.
+* The ways of setting CSI snapshot time have changed in v1.12. First, the sync waiting time for creating a snapshot handle in the CSI plugin is changed from the fixed 10 minutes into backup.Spec.CSISnapshotTimeout. The second, the async waiting time for VolumeSnapshot and VolumeSnapshotContent's status turning into `ReadyToUse` in operation uses the operation's timeout. The default value is 4 hours.
+* As from [Velero helm chart v4.0.0](https://github.com/vmware-tanzu/helm-charts/releases/tag/velero-4.0.0), it supports multiple BSL and VSL, and the BSL and VSL have changed from the map into a slice, and[ this breaking change](https://github.com/vmware-tanzu/helm-charts/pull/413) is not backward compatible. So it would be best to change the BSL and VSL configuration into slices before the Upgrade.
+
+
+### Limitations/Known issues
+* The Azure plugin supports Azure AD Workload identity way, but it only works for Velero native snapshots. It cannot support filesystem backup and snapshot data mover scenarios. 
+
+
+### All Changes
+* Fixes #6498. Get resource client again after restore actions in case resource's gv is changed. This is an improvement of pr #6499, to support group changes. A group change usually happens in a restore plugin which is used for resource conversion: convert a resource from a not supported gv to a supported gv (#6634, @27149chen)
+* Add API support for volMode block, only error for now. (#6608, @shawn-hurley)
+* Fix how the AWS credentials are obtained from configuration (#6598, @aws_creds)
+* Add performance E2E test (#6569, @qiuming-best)
+* Non default s3 credential profiles work on Unified Repository Provider (kopia) (#6558, @kaovilai)
+* Fix issue #6571, fix the problem for restore item operation to set the errors correctly so that they can be recorded by Velero restore and then reflect the correct status for Velero restore. (#6594, @Lyndon-Li)
+* Fix issue 6575, flush the repo after delete the snapshot, otherwise, the changes(deleting repo snapshot) cannot be committed to the repo. (#6587, @Lyndon-Li)
+* Delete moved snapshots when the backup is deleted (#6547, @reasonerjt)
+* check if restore crd exist before operating restores (#6544, @allenxu404)
+* Remove PVC's selector in backup's PVC action. (#6481, @blackpiglet)
+* Delete the expired deletebackuprequests that are stuck in "InProgress" (#6476, @reasonerjt)
+* Fix issue #6534, reset PVB CR's StorageLocation to the latest one during backup sync as same as the backup CR. Also fix similar problem with DataUploadResult for data mover restore. (#6533, @Lyndon-Li)
+* Fix issue #6519. Restrict the client manager of node-agent server to include only Velero resources from the server's namespace, otherwise, the controllers will try to reconcile CRs from all the installed Velero namespaces. (#6523, @Lyndon-Li)
+* Track the skipped PVC and print the summary in backup log  (#6496, @reasonerjt)
+* Add restore finalizer to clean up external resources (#6479, @allenxu404)
+* fix: Typos and add more spell checking rules to CI (#6415, @mateusoliveira43)
+* Add missing CompletionTimestamp and metrics when restore moved into terminal phase in restoreOperationsReconciler (#6397, @Nutrymaco)
+* Add support for resource Modifications in the restore flow. Also known as JSON Substitutions. (#6452, @anshulahuja98)
+* Remove dependency of the legacy client code from pkg/cmd directory part 2 (#6497, @blackpiglet)
+* Add data upload and download metrics (#6493, @allenxu404)
+* Fix issue 6490, If a backup/restore has multiple async operations and one operation fails while others are still in-progress, when all the operations finish, the backup/restore will be set as Completed falsely (#6491, @Lyndon-Li)
+* Velero Plugins no longer need kopia indirect dependency in their go.mod (#6484, @kaovilai)
+* Remove dependency of the legacy client code from pkg/cmd directory (#6469, @blackpiglet)
+* Add support for OpenStack CSI drivers topology keys (#6464, @openstack-csi-topology-keys)
+* Add exit code log and possible memory shortage warning log for Restic command failure. (#6459, @blackpiglet)
+* Modify DownloadRequest controller logic (#6433, @blackpiglet)
+* Add data download controller for data mover (#6436, @qiuming-best)
+* Fix hook filter display issue for backup describer (#6434, @allenxu404)
+* Retrieve DataUpload into backup result ConfigMap during volume snapshot restore. (#6410, @blackpiglet)
+* Design to add support for Multiple VolumeSnapshotClasses in CSI Plugin. (#5774, @anshulahuja98)
+* Clarify the deletion frequency for gc controller (#6414, @allenxu404)
+* Add unit tests for pkg/archive (#6396, @allenxu404)
+* Add UT for pkg/discovery (#6394, @qiuming-best)
+* Add UT for pkg/util (#6368, @Lyndon-Li)
+* Add the code for data mover restore expose (#6357, @Lyndon-Li)
+* Restore Endpoints before Services (#6315, @ywk253100)
+* Add warning message for volume snapshotter in data mover case. (#6377, @blackpiglet)
+* Add unit test for pkg/uploader (#6374, @qiuming-best)
+* Change kopia as the default path of PVB (#6370, @Lyndon-Li)
+* Do not persist VolumeSnapshot and VolumeSnapshotContent for snapshot DataMover case. (#6366, @blackpiglet)
+* Add data mover related options in CLI (#6365, @ywk253100)
+* Add dataupload controller (#6337, @qiuming-best)
+* Add UT cases for pkg/podvolume (#6336, @Lyndon-Li)
+* Remove Wait VolumeSnapshot to ReadyToUse logic. (#6327, @blackpiglet)
+* Enhance the code because of #6297, the return value of GetBucketRegion is not recorded, as a result, when it fails, we have no way to get the cause (#6326, @Lyndon-Li)
+* Skip updating status when CRDs are restored (#6325, @reasonerjt)
+* Include namespaces needed by namespaced-scope resources in backup. (#6320, @blackpiglet)
+* Update metrics when backup failed with validation error (#6318, @ywk253100)
+* Add the code for data mover backup expose (#6308, @Lyndon-Li)
+* Fix a PVR issue for generic data path -- the namespace remap was not honored, and enhance the code for better error handling (#6303, @Lyndon-Li)
+* Add default values for defaultItemOperationTimeout and itemOperationSyncFrequency in velero CLI  (#6298, @shubham-pampattiwar)
+* Add UT cases for pkg/repository (#6296, @Lyndon-Li)
+* Fix issue #5875. Since Kopia has supported IAM, Velero should not require static credentials all the time (#6283, @Lyndon-Li)
+* Fixed a bug where status.progress is not getting updated for backups. (#6276, @kkothule)
+* Add code change for async generic data path that is used by both PVB/PVR and data mover (#6226, @Lyndon-Li)
+* Add data mover CRD under v2alpha1, include DataUpload CRD and DataDownload CRD (#6176, @Lyndon-Li)
+* Remove any dataSource or dataSourceRef fields from PVCs in PVC BIA for cases of
+prior PVC restores with CSI (#6111, @eemcmullan)
+* Add the design for Volume Snapshot Data Movement (#5968, @Lyndon-Li)
+* Fix issue #5123, Kopia repository supports self-cert CA for S3 compatible storage. (#6268, @Lyndon-Li)
+* Bump up Kopia to v0.13 (#6248, @Lyndon-Li)
+* log volumes to backup to help debug why `IsPodRunning` is called. (#6232, @kaovilai)
+* Enable errcheck linter and resolve found issues (#6208, @blackpiglet)
+* Enable more linters, and remove mal-functioned milestoned issue action. (#6194, @blackpiglet)
+* Enable stylecheck linter and resolve found issues. (#6185, @blackpiglet)
+* Fix issue #6182. If pod is not running, don't treat it as an error, let it go and leave a warning. (#6184, @Lyndon-Li)
+* Enable staticcheck and resolve found issues (#6183, @blackpiglet)
+* Enable linter revive and resolve found errors: part 2 (#6177, @blackpiglet)
+* Enable linter revive and resolve found errors: part 1 (#6173, @blackpiglet)
+* Fix usestdlibvars and whitespace linters issues. (#6162, @blackpiglet)
+* Update Golang to v1.20 for main. (#6158, @blackpiglet)
+* Make GetPluginConfig accessible from other packages. (#6151, @tkaovila)
+* Ignore not found error during patching managedFields (#6136, @ywk253100)
+* Fix the goreleaser issues and add a new goreleaser action (#6109, @blackpiglet)

changelogs/CHANGELOG-1.13.md

@@ -0,0 +1,166 @@
+## v1.13
+### 2024-01-10
+
+### Download
+https://github.com/vmware-tanzu/velero/releases/tag/v1.13.0
+
+### Container Image
+`velero/velero:v1.13.0`
+
+### Documentation
+https://velero.io/docs/v1.13/
+
+### Upgrading
+https://velero.io/docs/v1.13/upgrade-to-1.13/
+
+### Highlights
+
+#### Resource Modifier Enhancement
+Velero introduced the Resource Modifiers in v1.12.0. This feature allows users to specify a ConfigMap with a set of rules to modify the resources during restoration. However, only the JSON Patch is supported when creating the rules, and JSON Patch has some limitations, which cannot cover all use cases. In v1.13.0, Velero adds new support for JSON Merge Patch and Strategic Merge Patch, which provide more power and flexibility and allow users to use the same ConfigMap to apply patches on the resources. More design details can be found in [Support JSON Merge Patch and Strategic Merge Patch in Resource Modifiers](https://github.com/vmware-tanzu/velero/blob/main/design/Implemented/merge-patch-and-strategic-in-resource-modifier.md) design. For instructions on how to use the feature, please refer to the [Resource Modifiers](https://velero.io/docs/v1.13/restore-resource-modifiers/) doc.
+
+#### Node-Agent Concurrency
+Velero data movement activities from fs-backups and CSI snapshot data movements run in Velero node-agent, so may be hosted by every node in the cluster and consume resources (i.e. CPU, memory, network bandwidth) from there. With v1.13, users are allowed to configure how many data movement activities (a.k.a, loads) run in each node globally or by node, so that users can better leverage the performance of Velero data movement activities and the resource consumption in the cluster. For more information, check the [Node-Agent Concurrency](https://velero.io/docs/v1.13/node-agent-concurrency/) document. 
+
+#### Parallel Files Upload Options
+Velero now supports configurable options for parallel files upload when using Kopia uploader to do fs-backups or CSI snapshot data movements which makes speed up backup possible.
+For more information, please check [Here](https://velero.io/docs/v1.13/backup-reference/#parallel-files-upload).
+
+#### Write Sparse Files Options
+If using fs-restore or CSI snapshot data movements, it’s supported to write sparse files during restore. For more information, please check [Here](https://velero.io/docs/v1.13/restore-reference/#write-sparse-files).
+
+#### Backup Describe
+In v1.13, the Backup Volume section is added to the velero backup describe command output. The backup Volumes section describes information for all the volumes included in the backup of various backup types, i.e. native snapshot, fs-backup, CSI snapshot, and CSI snapshot data movement. Particularly, the velero backup description now supports showing the information of CSI snapshot data movements, which is not supported in v1.12.
+
+Additionally, backup describe command will not check EnableCSI feature gate from client side, so if a backup has volumes with CSI snapshot or CSI snapshot data movement, backup describe command always shows the corresponding information in its output.
+
+#### Backup's new VolumeInfo metadata
+Create a new metadata file in the backup repository's backup name sub-directory to store the backup-including PVC and PV information. The information includes the backing-up method of the PVC and PV data, snapshot information, and status. The VolumeInfo metadata file determines how the PV resource should be restored. The Velero downstream software can also use this metadata file to get a summary of the backup's volume data information.
+
+#### Enhancement for CSI Snapshot Data Movements when Velero Pod Restart
+When performing backup and restore operations, enhancements have been implemented for Velero server pods or node agents to ensure that the current backup or restore process is not stuck or interrupted after restart due to certain exceptional circumstances.
+
+#### New status fields added to show hook execution details
+Hook execution status is now included in the backup/restore CR status and displayed in the backup/restore describe command output. Specifically, it will show the number of hooks which attempted to execute under the HooksAttempted  field and the number of hooks which failed to execute under the HooksFailed  field. 
+
+#### AWS SDK Bump Up
+Bump up AWS SDK for Go to version 2, which offers significant performance improvements in CPU and memory utilization over version 1.
+
+#### Azure AD/Workload Identity Support
+Azure AD/Workload Identity is the recommended approach to do the authentication with Azure services/AKS, Velero has introduced support for Azure AD/Workload Identity on the Velero Azure plugin side in previous releases, and in v1.13.0 Velero adds new support for Kopia operations(file system backup/data mover/etc.) with Azure AD/Workload Identity.
+
+#### Runtime and dependencies
+To fix CVEs and keep pace with Golang, Velero made changes as follows:
+* Bump Golang runtime to v1.21.6.
+* Bump several dependent libraries to new versions.
+* Bump Kopia to v0.15.0.
+
+
+### Breaking changes
+* Backup describe command: due to the backup describe output enhancement, some existing information (i.e. the output for native snapshot, CSI snapshot, and fs-backup) has been moved to the Backup Volumes section with some format changes.
+* API type changes: changes the field [DataMoverConfig](https://github.com/vmware-tanzu/velero/blob/v1.13.0/pkg/apis/velero/v2alpha1/data_upload_types.go#L54) in DataUploadSpec from `*map[string][string]`` to `map[string]string`
+* Velero install command: due to the issue [#7264](https://github.com/vmware-tanzu/velero/issues/7264), v1.13.0 introduces a break change that make the informer cache enabled by default to keep the actual behavior consistent with the helper message(the informer cache is disabled by default before the change).
+
+
+### Limitations/Known issues
+* The backup's VolumeInfo metadata doesn't have the information updated in the async operations. This function could be supported in v1.14 release. 
+
+### Note
+* Velero introduces the informer cache which is enabled by default. The informer cache improves the restore performance but may cause higher memory consumption. Increase the memory limit of the Velero pod or disable the informer cache by specifying the `--disable-informer-cache` option when installing Velero if you get the OOM error.
+
+### Deprecation announcement
+* The generated k8s clients, informers, and listers are deprecated in the Velero v1.13 release. They are put in the Velero repository's pkg/generated directory. According to the n+2 supporting policy, the deprecated are kept for two more releases. The pkg/generated directory should be deleted in the v1.15 release.
+* After the backup VolumeInfo metadata file is added to the backup, Velero decides how to restore the PV resource according to the VolumeInfo content. To support the backup generated by the older version of Velero, the old logic is also kept. The support for the backup without the VolumeInfo metadata file will be kept for two releases. The support logic will be deleted in the v1.15 release.
+
+### All Changes
+  * Make "disable-informer-cache" option false(enabled) by default to keep it consistent with the help message (#7294, @ywk253100)
+  * Fix issue #6928, remove snapshot deletion timeout for PVB (#7282, @Lyndon-Li)
+  * Do not set "targetNamespace" to namespace items (#7274, @reasonerjt)
+  * Fix issue #7244. By the end of the upload, check the outstanding incomplete snapshots and delete them by calling ApplyRetentionPolicy (#7245, @Lyndon-Li)
+  * Adjust the newline output of resource list in restore describer (#7238, @allenxu404)
+  * Remove the redundant newline in backup describe output (#7229, @allenxu404)
+  * Fix issue #7189, data mover generic restore - don't assume the first volume as the restore volume (#7201, @Lyndon-Li)
+  * Update CSIVolumeSnapshotsCompleted in backup's status and the metric
+during backup finalize stage according to async operations content. (#7184, @blackpiglet)
+  * Refactor DownloadRequest Stream function (#7175, @blackpiglet)
+  * Add `--skip-immediately` flag to schedule commands; `--schedule-skip-immediately` server and install (#7169, @kaovilai)
+  * Add node-agent concurrency doc and change the config name from dataPathConcurrency to loadCocurrency (#7161, @Lyndon-Li)
+  * Enhance hooks tracker by adding a returned error to record function (#7153, @allenxu404)
+  * Track the skipped PV when SnapshotVolumes set as false (#7152, @reasonerjt)
+  * Add more linters part 2. (#7151, @blackpiglet)
+  * Fix issue #7135, check pod status before checking node-agent pod status (#7150, @Lyndon-Li)
+  * Treat namespace as a regular restorable item (#7143, @reasonerjt)
+  * Allow sparse option for Kopia & Restic restore  (#7141, @qiuming-best)
+  * Use VolumeInfo to help restore the PV. (#7138, @blackpiglet)
+  * Node agent restart enhancement (#7130, @qiuming-best)
+  * Fix issue #6695, add describe for data mover backups (#7125, @Lyndon-Li)
+  * Add hooks status to backup/restore CR (#7117, @allenxu404)
+  * Include plugin name in the error message by operations (#7115, @reasonerjt)
+  * Fix issue #7068, due to a behavior of CSI external snapshotter, manipulations of VS and VSC may not be handled in the same order inside external snapshotter as the API is called. So add a protection finalizer to ensure the order (#7102, @Lyndon-Li)
+  * Generate VolumeInfo for backup. (#7100, @blackpiglet)
+  * Fix issue #7094, fallback to full backup if previous snapshot is not found (#7096, @Lyndon-Li)
+  * Fix issue #7068, due to an behavior of CSI external snapshotter, manipulations of VS and VSC may not be handled in the same order inside external snapshotter as the API is called. So add a protection finalizer to ensure the order (#7095, @Lyndon-Li)
+  * Skip syncing the backup which doesn't contain backup metadata (#7081, @ywk253100)
+  * Fix issue #6693, partially fail restore if CSI snapshot is involved but CSI feature is not ready, i.e., CSI feature gate is not enabled or CSI plugin is not installed. (#7077, @Lyndon-Li)
+  * Truncate the credential file to avoid the change of secret content messing it up (#7072, @ywk253100)
+  * Add VolumeInfo metadata structures. (#7070, @blackpiglet)
+  * improve discoveryHelper.Refresh() in restore (#7069, @27149chen)
+  * Add DataUpload Result and CSI VolumeSnapshot check for restore PV. (#7061, @blackpiglet)
+  * Add the implementation for design #6950, configurable data path concurrency (#7059, @Lyndon-Li)
+  * Make data mover fail early (#7052, @qiuming-best)
+  * Remove dependency of generated client part 3. (#7051, @blackpiglet)
+  * Update Backup.Status.CSIVolumeSnapshotsCompleted during finalize (#7046, @kaovilai)
+  * Remove the Velero generated client. (#7041, @blackpiglet)
+  * Fix issue #7027, data mover backup exposer should not assume the first volume as the backup volume in backup pod (#7038, @Lyndon-Li)
+  * Read information from the credential specified by BSL (#7034, @ywk253100)
+  * Fix #6857. Added check for matching Owner References when synchronizing backups, removing references that are not found/have mismatched uid. (#7032, @deefdragon)
+  * Add description markers for dataupload and datadownload CRDs (#7028, @shubham-pampattiwar)
+  * Add HealthCheckNodePort deletion logic for Service restore. (#7026, @blackpiglet)
+  * Fix inconsistent behavior of Backup and Restore hook execution (#7022, @allenxu404)
+  * Fix #6964. Don't use csiSnapshotTimeout (10 min) for waiting snapshot to readyToUse for data mover, so as to make the behavior complied with CSI snapshot backup (#7011, @Lyndon-Li)
+  * restore: Use warning when Create IsAlreadyExist and Get error (#7004, @kaovilai)
+  * Bump kopia to 0.15.0 (#7001, @Lyndon-Li)
+  * Make Kopia file parallelism configurable (#7000, @qiuming-best)
+  * Fix unified repository (kopia) s3 credentials profile selection (#6995, @kaovilai)
+  * Fix #6988, always get region from BSL if it is not empty (#6990, @Lyndon-Li)
+  * Limit PVC block mode logic to non-Windows platform. (#6989, @blackpiglet)
+  * It is a valid case that the Status.RestoreSize field in VolumeSnapshot is not set, if so, get the volume size from the source PVC to create the backup PVC (#6976, @Lyndon-Li)
+  * Check whether the action is a CSI action and whether CSI feature is enabled, before executing the action. (#6968, @blackpiglet)
+  * Add the PV backup information design document. (#6962, @blackpiglet)
+  * Change controller-runtime List option from MatchingFields to ListOptions (#6958, @blackpiglet)
+  * Add the design for node-agent concurrency (#6950, @Lyndon-Li)
+  * Import auth provider plugins (#6947, @0x113)
+  * Fix #6668, add a limitation for file system restore parallelism with other types of restores (CSI snapshot restore, CSI snapshot movement restore) (#6946, @Lyndon-Li)
+  * Add MSI Support for Azure plugin. (#6938, @yanggangtony)
+  * Partially fix #6734, guide Kubernetes' scheduler to spread backup pods evenly across nodes as much as possible, so that data mover backup could achieve better parallelism (#6926, @Lyndon-Li)
+  * Bump up aws sdk to aws-sdk-go-v2 (#6923, @reasonerjt)
+  * Optional check if targeted container is ready before executing a hook (#6918, @Ripolin)
+  * Support JSON Merge Patch and Strategic Merge Patch in Resource Modifiers (#6917, @27149chen)
+  * Fix issue 6913: Velero Built-in Datamover: Backup stucks in phase WaitingForPluginOperations when Node Agent pod gets restarted (#6914, @shubham-pampattiwar)
+  * Set ParallelUploadAboveSize as MaxInt64 and flush repo after setting up policy so that policy is retrieved correctly by TreeForSource (#6885, @Lyndon-Li)
+  * Replace the base image with paketobuildpacks image (#6883, @ywk253100)
+  * Fix issue #6859, move plugin depending podvolume functions to util pkg, so as to remove the dependencies to unnecessary repository packages like kopia, azure, etc. (#6875, @Lyndon-Li)
+  * Fix #6861. Only Restic path requires repoIdentifier, so for non-restic path, set the repoIdentifier fields as empty in PVB and PVR and also remove the RepoIdentifier column in the get output of PVBs and PVRs (#6872, @Lyndon-Li)
+  * Add volume types filter in resource policies (#6863, @qiuming-best)
+  * change the metrics backup_attempt_total default value to 1. (#6838, @yanggangtony)
+  * Bump kopia to v0.14 (#6833, @Lyndon-Li)
+  * Retry failed create when using generateName (#6830, @sseago)
+  * Fix issue #6786, always delete VSC regardless of the deletion policy (#6827, @Lyndon-Li)
+  * Proposal to support JSON Merge Patch and Strategic Merge Patch in Resource Modifiers (#6797, @27149chen)
+  * Fix the node-agent missing metrics-address defines. (#6784, @yanggangtony)
+  * Fix default BSL setting not work (#6771, @qiuming-best)
+  * Update restore controller logic for restore deletion (#6770, @ywk253100)
+  * Fix #6752: add namespace exclude check. (#6760, @blackpiglet)
+  * Fix issue #6753, remove the check for read-only BSL in restore async operation controller since Velero cannot fully support read-only mode BSL in restore at present (#6757, @Lyndon-Li)
+  * Fix issue #6647, add the --default-snapshot-move-data parameter to Velero install, so that users don't need to specify --snapshot-move-data per backup when they want to move snapshot data for all backups (#6751, @Lyndon-Li)
+  * Use old(origin) namespace in resource modifier conditions in case namespace may change during restore (#6724, @27149chen)
+  * Perf improvements for existing resource restore (#6723, @sseago)
+  * Remove schedule-related metrics on schedule delete (#6715, @nilesh-akhade)
+  * Kubernetes 1.27 new job label batch.kubernetes.io/controller-uid are deleted during restore per https://github.com/kubernetes/kubernetes/pull/114930 (#6712, @kaovilai)
+  * This pr made some improvements in Resource Modifiers: 1. add label selector 2. change the field name from groupKind to groupResource (#6704, @27149chen)
+  * Make Kopia support Azure AD (#6686, @ywk253100)
+  * Add support for block volumes with Kopia (#6680, @dzaninovic)
+  * Delete PartiallyFailed orphaned backups as well as Completed ones (#6649, @sseago)
+  * Add CSI snapshot data movement doc (#6637, @Lyndon-Li)
+  * Fixes #6636, skip subresource in resource discovery (#6635, @27149chen)
+  * Add `orLabelSelectors` for backup, restore commands (#6475, @nilesh-akhade)
+  * fix run preHook and postHook on completed pods (#5211, @cleverhu)

changelogs/CHANGELOG-1.14.md

@@ -0,0 +1,105 @@
+## v1.14
+
+### Download
+https://github.com/vmware-tanzu/velero/releases/tag/v1.14.0
+
+### Container Image
+`velero/velero:v1.14.0`
+
+### Documentation
+https://velero.io/docs/v1.14/
+
+### Upgrading
+https://velero.io/docs/v1.14/upgrade-to-1.14/
+
+### Highlights
+
+#### The maintenance work for kopia/restic backup repositories is run in jobs
+Since velero started using kopia as the approach for filesystem-level backup/restore, we've noticed an issue when velero connects to the kopia backup repositories and performs maintenance, it sometimes consumes excessive memory that can cause the velero pod to get OOM Killed.  To mitigate this issue, the maintenance work will be moved out of velero pod to a separate kubernetes job, and the user will be able to specify the resource request in "velero install".
+#### Volume Policies are extended to support more actions to handle volumes
+In an earlier release, a flexible volume policy was introduced to skip certain volumes from a backup.  In v1.14 we've made enhancement to this policy to allow the user to set how the volumes should be backed up.  The user will be able to set "fs-backup" or "snapshot" as value of “action" in the policy and velero will backup the volumes accordingly.  This enhancement allows the user to achieve a fine-grained control like "opt-in/out" without having to update the target workload.  For more details please refer to https://velero.io/docs/v1.14/resource-filtering/#supported-volumepolicy-actions
+#### Node Selection for Data Movement Backup
+In velero the data movement flow relies on datamover pods, and these pods may take substantial resources and keep running for a long time.  In v1.14, the user will be able to create a configmap to define the eligible nodes on which the datamover pods are launched.  For more details refer to https://velero.io/docs/v1.14/data-movement-backup-node-selection/ 
+#### VolumeInfo metadata for restored volumes
+In v1.13, we introduced volumeinfo metadata for backup to help velero CLI and downstream adopter understand how velero handles each volume during backup.  In v1.14, similar metadata will be persisted for each restore. velero CLI is also updated to bring more info in the output of "velero restore describe".
+#### "Finalizing" phase is introduced to restores
+The "Finalizing" phase is added to the state transition flow to restore, which helps us fix several issues: The labels added to PVs will be restored after the data in the PV is restored via volumesnapshotter.  The post restore hook will be executed after datamovement is finished.
+#### Certificate-based authentication support for Azure
+Besides the service principal with secret(password)-based authentication, Velero introduces the new support for service principal with certificate-based authentication in v1.14.0. This approach enables you to adopt a phishing resistant authentication by using conditional access policies, which better protects Azure resources and is the recommended way by Azure.
+
+### Runtime and dependencies
+* Golang runtime: v1.22.2
+* kopia: v0.17.0
+
+### Limitations/Known issues
+* For the external BackupItemAction plugins that take snapshots for PVs, such as vsphere plugin.  If the plugin checks the value of the field "snapshotVolumes" in the backup spec as a criteria for snapshot, the settings in the volume policy will not take effect.  For example, if the "snapshotVolumes" is set to False in the backup spec, but a volume meets the condition in the volume policy for "snapshot" action, because the plugin will not check the settings in the volume policy, the plugin will not take snapshot for the volume.  For more details please refer to #7818
+
+### Breaking changes
+* CSI plugin has been merged into velero repo in v1.14 release.  It will be installed by default as an internal plugin, and should not be installed via "–plugins " parameter in "velero install" command.
+* The default resource requests and limitations for node agent are removed in v1.14, to make the node agent pods have the QoS class of "BestEffort", more details please refer to #7391
+* There's a change in namespace filtering behavior during backup:  In v1.14, when the includedNamespaces/excludedNamespaces fields are not set and the labelSelector/OrLabelSelectors are set in the backup spec, the backup will only include the namespaces which contain the resources that match the label selectors, while in previous releases all namespaces will be included in the backup with such settings.  More details refer to #7105
+* Patching the PV in the "Finalizing" state may cause the restore to be in "PartiallyFailed" state when the PV is blocked in "Pending" state, while in the previous release the restore may end up being in "Complete" state. For more details refer to #7866
+
+### All Changes
+* Fix backup log to show error string, not index (#7805, @piny940)
+* Modify the volume helper logic. (#7794, @blackpiglet)
+* Add documentation for extension of volume policy feature (#7779, @shubham-pampattiwar)
+* Surface errors when waiting for backupRepository and timeout occurs (#7762, @kaovilai)
+* Add existingResourcePolicy restore CR validation to controller (#7757, @kaovilai)
+* Fix condition matching in resource modifier when there are multiple rules (#7715, @27149chen)
+* Bump up the version of KinD and k8s in github actions (#7702, @reasonerjt)
+* Implementation for Extending VolumePolicies to support more actions (#7664, @shubham-pampattiwar)
+* Migrate from `github.com/Azure/azure-storage-blob-go` to `github.com/Azure/azure-sdk-for-go/sdk/storage/azblob` (#7598, @mmorel-35)
+* When Included/ExcludedNamespaces are omitted, and LabelSelector or OrLabelSelector is used, namespaces without selected items are excluded from backup. (#7697, @blackpiglet)
+* Display CSI snapshot restores in restore describe (#7687, @reasonerjt)
+* Use specific credential rather than the credential chain for Azure (#7680, @ywk253100)
+* Modify hook docs for clarity on displaying hook execution results (#7679, @allenxu404)
+* Wait for results of restore exec hook executions in Finalizing phase instead of InProgress phase (#7619, @allenxu404)
+* migrating to `sdk/resourcemanager/**/arm**` from `services/**/mgmt/**` (#7596, @mmorel-35)
+* Bump up to go1.22 (#7666, @reasonerjt)
+* Fix issue #7648. Adjust the exposing logic to avoid exposing failure and snapshot leak when expose fails (#7662, @Lyndon-Li)
+* Track and persist restore volume info (#7630, @reasonerjt)
+* Check the existence of the namespaces provided in the "--include-namespaces" option (#7569, @ywk253100)
+* Add the finalization phase to the restore workflow (#7377, @allenxu404)
+* Upgrade the version of go plugin related libs/tools (#7373, @ywk253100)
+* Check resource Group Version and Kind is available in cluster before attempting restore to prevent being stuck. (#7322, @kaovilai)
+* Merge CSI plugin code into Velero. (#7609, @blackpiglet)
+* Fix issue #7391, remove the default constraint for node-agent pods (#7488, @Lyndon-Li)
+* Fix DataDownload fails during restore for empty PVC workload (#7521, @qiuming-best)
+* Add repository maintenance job (#7451, @qiuming-best)
+* Check whether the VolumeSnapshot's source PVC is nil before using it.
+  Skip populate VolumeInfo for data-moved PV when CSI is not enabled. (#7515, @blackpiglet)
+* Fix issue #7308, change the data path requeue time to 5 second for data mover backup/restore, PVB and PVR. (#7458, @Lyndon-Li)
+*  Patch newly dynamically provisioned PV with volume info to restore custom setting of PV (#7504, @allenxu404)
+* Adjust the logic for the backup_last_status metrics to stop incorrectly incrementing over time (#7445, @allenxu404)
+* dependabot: support github-actions updates (#7594, @mmorel-35)
+* Include the design for adding the finalization phase to the restore workflow (#7317, @allenxu404)
+* Fix issue #7211. Enable advanced feature capability and add support to concatenate objects for unified repo. (#7452, @Lyndon-Li)
+* Add design to introduce restore volume info (#7610, @reasonerjt)
+* Increase the k8s client QPS/burst to avoid throttling request errors (#7311, @ywk253100)
+* Support update the backup VolumeInfos by the Async ops result. (#7554, @blackpiglet)
+* FS backup create PodVolumeBackup when the backup excluded PVC,
+  so I added logic to skip PVC volume type when PVC is not included in the backup resources to be backed up. (#7472, @sbahar619)
+* Respect and use `credentialsFile` specified in BSL.spec.config when IRSA is configured over Velero Pod Environment credentials (#7374, @reasonerjt)
+* Move the native snapshot definition code into internal directory (#7544, @blackpiglet)
+* Fix issue #7036. Add the implementation of node selection for data mover backups (#7437, @Lyndon-Li)
+* Fix issue #7535, add the MustHave resource check during item collection and item filter for restore (#7585, @Lyndon-Li)
+* build(deps): bump json-patch to v5.8.0 (#7584, @mmorel-35)
+* Add confirm flag to velero plugin add (#7566, @kaovilai)
+* do not skip unknown gvr at the beginning and get new gr when kind is changed (#7523, @27149chen)
+* Fix snapshot leak for backup (#7558, @qiuming-best)
+* For issue #7036, add the document for data mover node selection (#7640, @Lyndon-Li)
+* Add design for Extending VolumePolicies to support more actions (#6956, @shubham-pampattiwar)
+* BackupRepositories associated with a BSL are invalidated when BSL is (re-)created. (#7380, @kaovilai)
+* Improve the concurrency for PVBs in different pods  (#7571, @ywk253100)
+* Bump up Kopia to v0.16.0 and open kopia repo with no index change (#7559, @Lyndon-Li)
+* Bump up the versions of several Kubernetes-related libs (#7489, @ywk253100)
+* Make parallel restore configurable (#7512, @qiuming-best)
+* Support certificate-based authentication for Azure (#7549, @ywk253100)
+* Fix issue #7281, batch delete snapshots in the same repo (#7438, @Lyndon-Li)
+* Add CRD name to error message when it is not ready to use (#7295, @josemarevalo)
+* Add the design for node selection for data mover backup (#7383, @Lyndon-Li)
+* Bump up aws-sdk to latest version to leverage Pod Identity credentials. (#7307, @guikcd)
+* Fix issue #7246. Document the behavior for repo snapshot deletion (#7622, @Lyndon-Li)
+* Fix issue #7583, set backupName optional for Restore CRD (#7617, @Lyndon-Li)
+

changelogs/CHANGELOG-1.15.md

@@ -0,0 +1,173 @@
+## v1.15.1
+
+### Download
+https://github.com/vmware-tanzu/velero/releases/tag/v1.15.1
+
+### Container Image
+`velero/velero:v1.15.1`
+
+### Documentation
+https://velero.io/docs/v1.15/
+
+### Upgrading
+https://velero.io/docs/v1.15/upgrade-to-1.15/
+
+### All Changes
+  * Fix backup post hook issue #8159 (caused by #7571): always execute backup post hooks after PVBs are handled (#8517, @ywk253100)
+  * Fix issue #8125, log diagnostic info for data mover exposers when expose timeout (#8511, @Lyndon-Li)
+  * Set hinting region to use for GetBucketRegion() in pkg/repository/config/aws.go (#8505, @kaovilai)
+  * Fix issue #8433, add third party labels to data mover pods when the same labels exist in node-agent pods (#8501, @Lyndon-Li)
+  * Fix issue #8485, add an accepted time so as to count the prepare timeout (#8496, @Lyndon-Li)
+  * Add SecurityContext to restore-helper (#8495, @reasonerjt)
+  * Add nil check for updating DataUpload VolumeInfo in finalizing phase. (#8465, @blackpiglet)
+  * Fix issue #8391, check ErrCancelled from suffix of data mover pod's termination message (#8404, @Lyndon-Li)
+  * Fix issue #8394, don't call closeDataPath in VGDP callbacks, otherwise, the VGDP cleanup will hang (#8402, @Lyndon-Li)
+  * Reduce minimum required go toolchain in release-1.15 go.mod (#8399, @kaovilai)
+  * Fix issue #8539, validate uploader types when o.CRDsOnly is set to false only since CRD installation doesn't rely on uploader types (#8540, @Lyndon-Li)
+
+
+## v1.15
+
+### Download
+https://github.com/vmware-tanzu/velero/releases/tag/v1.15.0
+
+### Container Image
+`velero/velero:v1.15.0`
+
+### Documentation
+https://velero.io/docs/v1.15/
+
+### Upgrading
+https://velero.io/docs/v1.15/upgrade-to-1.15/
+
+### Highlights
+#### Data mover micro service
+Data transfer activities for CSI Snapshot Data Movement are moved from node-agent pods to dedicate backupPods or restorePods. This brings many benefits such as:  
+- This avoids to access volume data through host path, while host path access is privileged and may involve security escalations, which are concerned by users.
+- This enables users to to control resource (i.e., cpu, memory) allocations in a granular manner, e.g., control them per backup/restore of a volume.
+- This enhances the resilience, crash of one data movement activity won't affect others.
+- This prevents unnecessary full backup because of host path changes after workload pods restart.
+- For more information, check the design https://github.com/vmware-tanzu/velero/blob/main/design/Implemented/vgdp-micro-service/vgdp-micro-service.md.
+
+#### Item Block concepts and ItemBlockAction (IBA) plugin
+Item Block concepts are introduced for resource backups to help to achieve multiple thread backups. Specifically, correlated resources are categorized in the same item block and item blocks could be processed concurrently in multiple threads.  
+ItemBlockAction plugin is introduced to help Velero to categorize resources into item blocks. At present, Velero provides built-in IBAs for pods and PVCs and Velero also supports customized IBAs for any resources.  
+In v1.15, Velero doesn't support multiple thread process of item blocks though item block concepts and IBA plugins are fully supported. The multiple thread support will be delivered in future releases.  
+For more information, check the design https://github.com/vmware-tanzu/velero/blob/main/design/backup-performance-improvements.md.  
+
+#### Node selection for repository maintenance job
+Repository maintenance are resource consuming tasks, Velero now allows you to configure the nodes to run repository maintenance jobs, so that you can run repository maintenance jobs in idle nodes or avoid them to run in nodes hosting critical workloads.  
+To support the configuration, a new repository maintenance configuration configMap is introduced.  
+For more information, check the document https://velero.io/docs/v1.15/repository-maintenance/.  
+
+#### Backup PVC read-only configuration
+In 1.15, Velero allows you to configure the data mover backupPods to read-only mount the backupPVCs. In this way, the data mover expose process could be significantly accelerated for some storages (i.e., ceph).  
+To support the configuration, a new backup PVC configuration configMap is introduced.  
+For more information, check the document https://velero.io/docs/v1.15/data-movement-backup-pvc-configuration/.  
+
+#### Backup PVC storage class configuration
+In 1.15, Velero allows you to configure the storageclass used by the data mover backupPods. In this way, the provision of backupPVCs don't need to adhere to the same pattern as workload PVCs, e.g., for a backupPVC, it only needs one replica, whereas, the a workload PVC may have multiple replicas.  
+To support the configuration, the same backup PVC configuration configMap is used.  
+For more information, check the document https://velero.io/docs/v1.15/data-movement-backup-pvc-configuration/.  
+
+#### Backup repository data cache configuration
+The backup repository may need to cache data on the client side during various repository operations, i.e., read, write, maintenance, etc. The cache consumes the root file system space of the pod where the repository access happens.  
+In 1.15, Velero allows you to configure the total size of the cache per repository. In this way, if your pod doesn't have enough space in its root file system, the pod won't be evicted due to running out of ephemeral storage.  
+To support the configuration, a new backup repository configuration configMap is introduced.  
+For more information, check the document https://velero.io/docs/v1.15/backup-repository-configuration/.  
+
+#### Performance improvements
+In 1.15, several performance related issues/enhancements are included, which makes significant performance improvements in specific scenarios:  
+- There was a memory leak of Velero server after plugin calls, now it is fixed, see issue https://github.com/vmware-tanzu/velero/issues/7925
+- The `client-burst/client-qps` parameters are automatically inherited to plugins, so that you can use the same velero server parameters to accelerate the plugin executions when large number of API server calls happen, see issue https://github.com/vmware-tanzu/velero/issues/7806
+- Maintenance of Kopia repository takes huge memory in scenarios that huge number of files have been backed up, Velero 1.15 has included the Kopia upstream enhancement to fix the problem, see issue https://github.com/vmware-tanzu/velero/issues/7510
+
+### Runtime and dependencies
+Golang runtime: v1.22.8  
+kopia: v0.17.0
+
+### Limitations/Known issues
+#### Read-only backup PVC may not work on SELinux environments
+Due to an issue of Kubernetes upstream, if a volume is mounted as read-only in SELinux environments, the read privilege is not granted to any user, as a result, the data mover backup will fail. On the other hand, the backupPVC must be mounted as read-only in order to accelerate the data mover expose process.  
+Therefore, a user option is added in the same backup PVC configuration configMap, once the option is enabled, the backupPod container will run as a super privileged container and disable SELinux access control. If you have concern in this super privileged container or you have configured [pod security admissions](https://kubernetes.io/docs/concepts/security/pod-security-admission/) and don't allow super privileged containers, you will not be able to use this read-only backupPVC feature and lose the benefit to accelerate the data mover expose process.  
+
+### Breaking changes
+#### Deprecation of Restic
+Restic path for fs-backup is in deprecation process starting from 1.15. According to [Velero deprecation policy](https://github.com/vmware-tanzu/velero/blob/v1.15/GOVERNANCE.md#deprecation-policy), for 1.15, if Restic path is used the backup/restore of fs-backup still creates and succeeds, but you will see warnings in below scenarios:  
+- When `--uploader-type=restic` is used in Velero installation
+- When Restic path is used to create backup/restore of fs-backup
+
+#### node-agent configuration name is configurable
+Previously, a fixed name is searched for node-agent configuration configMap. Now in 1.15, Velero allows you to customize the name of the configMap, on the other hand, the name must be specified by node-agent server parameter `node-agent-configmap`.  
+
+#### Repository maintenance job configurations in Velero server parameter are moved to repository maintenance job configuration configMap
+In 1.15, below Velero server parameters for repository maintenance jobs are moved to the repository maintenance job configuration configMap. While for back compatibility reason, the same Velero sever parameters are preserved as is. But the configMap is recommended and the same values in the configMap take preference if they exist in both places:  
+```
+--keep-latest-maintenance-jobs
+--maintenance-job-cpu-request
+--maintenance-job-mem-request
+--maintenance-job-cpu-limit
+--maintenance-job-mem-limit
+```
+
+#### Changing PVC selected-node feature is deprecated
+In 1.15, the [Changing PVC selected-node feature](https://velero.io/docs/v1.15/restore-reference/#changing-pvc-selected-node) enters deprecation process and will be removed in future releases according to [Velero deprecation policy](https://github.com/vmware-tanzu/velero/blob/v1.15/GOVERNANCE.md#deprecation-policy). Usage of this feature for any purpose is not recommended.  
+
+### All Changes
+  * add no-relabeling option to backupPVC configmap (#8288, @sseago)
+  * only set spec.volumes readonly if PVC is readonly for datamover (#8284, @sseago)
+  * Add labels to maintenance job pods (#8256, @shubham-pampattiwar)
+  * Add the Carvel package related resources to the restore priority list (#8228, @ywk253100)
+  * Reduces indirect imports for plugin/framework importers (#8208, @kaovilai)
+  * Add controller name to periodical_enqueue_source. The logger parameter now includes an additional field with the value of reflect.TypeOf(objList).String() and another field with the value of controllerName. (#8198, @kaovilai)
+  * Update Openshift SCC docs link (#8170, @shubham-pampattiwar)
+  * Partially fix issue #8138, add doc for node-agent memory preserve (#8167, @Lyndon-Li)
+  * Pass Velero server command args to the plugins (#8166, @ywk253100)
+  * Fix issue #8155, Merge Kopia upstream commits for critical issue fixes and performance improvements (#8158, @Lyndon-Li)
+  * Implement the Repo maintenance Job configuration. (#8145, @blackpiglet)
+  * Add document for data mover micro service (#8144, @Lyndon-Li)
+  * Fix issue #8134, allow to config resource request/limit for data mover micro service pods (#8143, @Lyndon-Li)
+  * Apply backupPVCConfig to backupPod volume spec (#8141, @shubham-pampattiwar)
+  * Add resource modifier for velero restore describe CLI (#8139, @blackpiglet)
+  * Fix issue #7620, add doc for backup repo config (#8131, @Lyndon-Li)
+  * Modify E2E and perf test report generated directory (#8129, @blackpiglet)
+  * Add docs for backup pvc config support (#8119, @shubham-pampattiwar)
+  * Delete generated k8s client and informer. (#8114, @blackpiglet)
+  * Add support for backup PVC configuration (#8109, @shubham-pampattiwar)
+  * ItemBlock model and phase 1 (single-thread) workflow changes (#8102, @sseago)
+  * Fix issue #8032, make node-agent configMap name configurable (#8097, @Lyndon-Li)
+  * Fix issue #8072, add the warning messages for restic deprecation (#8096, @Lyndon-Li)
+  * Fix issue #7620, add backup repository configuration implementation and support cacheLimit configuration for Kopia repo (#8093, @Lyndon-Li)
+  * Patch dbr's status when error happens (#8086, @reasonerjt)
+  * According to design #7576, after node-agent restarts, if a DU/DD is in InProgress status, re-capture the data mover ms pod and continue the execution (#8085, @Lyndon-Li)
+  * Updates to IBM COS documentation to match current version (#8082, @gjanders)
+  * Data mover micro service DUCR/DDCR controller refactor according to design #7576 (#8074, @Lyndon-Li)
+  * add retries with timeout to existing patch calls that moves a backup/restore from InProgress/Finalizing to a final status phase. (#8068, @kaovilai)
+  * Data mover micro service restore according to design #7576 (#8061, @Lyndon-Li)
+  * Internal ItemBlockAction plugins (#8054, @sseago)
+  * Data mover micro service backup according to design #7576 (#8046, @Lyndon-Li)
+  * Avoid wrapping failed PVB status with empty message. (#8028, @mrnold)
+  * Created new ItemBlockAction (IBA) plugin type (#8026, @sseago)
+  * Make PVPatchMaximumDuration timeout configurable (#8021, @shubham-pampattiwar)
+  * Reuse existing plugin manager for get/put volume info (#8012, @sseago)
+  * Data mover ms watcher according to design #7576 (#7999, @Lyndon-Li)
+  * New data path for data mover ms according to design #7576 (#7988, @Lyndon-Li)
+  * For issue #7700 and #7747, add the design for backup PVC configurations (#7982, @Lyndon-Li)
+  * Only get VolumeSnapshotClass when DataUpload exists. (#7974, @blackpiglet)
+  * Fix issue #7972, sync the backupPVC deletion in expose clean up (#7973, @Lyndon-Li)
+  * Expose the VolumeHelper to third-party plugins. (#7969, @blackpiglet)
+  * Check whether the volume's source is PVC before fetching its PV. (#7967, @blackpiglet)
+  * Check whether the namespaces specified in namespace filter exist. (#7965, @blackpiglet)
+  * Add design for backup repository configurations for issue #7620, #7301 (#7963, @Lyndon-Li)
+  * New data path for data mover ms according to design #7576 (#7955, @Lyndon-Li)
+  * Skip PV patch step in Restoe workflow for WaitForFirstConsumer VolumeBindingMode Pending state PVCs (#7953, @shubham-pampattiwar)
+  * Fix issue #7904, add the deprecation and limitation clarification for change PVC selected-node feature (#7948, @Lyndon-Li)
+  * Expose the VolumeHelper to third-party plugins. (#7944, @blackpiglet)
+  * Don't consider unschedulable pods unrecoverable (#7899, @sseago)
+  * Upgrade to robfig/cron/v3 to support time zone specification. (#7793, @kaovilai)
+  * Add the result in the backup's VolumeInfo. (#7775, @blackpiglet)
+  * Migrate from github.com/golang/protobuf to google.golang.org/protobuf (#7593, @mmorel-35)
+  * Add the design for data mover micro service (#7576, @Lyndon-Li)
+  * Descriptive restore error when restoring into a terminating namespace. (#7424, @kaovilai)
+  * Ignore missing path error in conditional match (#7410, @seanblong)
+  * Propose a deprecation process for velero (#5532, @shubham-pampattiwar)

changelogs/CHANGELOG-1.8.md

@@ -61,7 +61,7 @@ in progress for 1.9.
 * Add rbac and annotation test cases (#4455, @mqiu)
 * remove --crds-version in velero install command. (#4446, @jxun)
 * Upgrade e2e test vsphere plugin (#4440, @mqiu)
-* Fix e2e test failures for the inappropriate optimaze of velero install (#4438, @mqiu)
+* Fix e2e test failures for the inappropriate optimize of velero install (#4438, @mqiu)
 * Limit backup namespaces on test resource filtering cases (#4437, @mqiu)
 * Bump up Go to 1.17 (#4431, @reasonerjt)
 * Added `<backup name>`-itemsnapshots.json.gz to the backup format.  This file exists
@@ -103,7 +103,7 @@ Also added DownloadTargetKindBackupItemSnapshots for retrieving the signed URL t
 * Fix CVE-2020-29652 and CVE-2020-26160 (#4274, @ywk253100)
 * Refine tag-release.sh to align with change in release process (#4185, @reasonerjt)
 * Fix plugins incompatible issue in upgrade test (#4141, @danfengliu)
-* Verify group before treating resource as cohabitating (#4126, @sseago)
+* Verify group before treating resource as cohabiting (#4126, @sseago)
 * Added ItemSnapshotter plugin definition and plugin framework - addresses #3533.
   Part of the Upload Progress enhancement (#3533) (#4077, @dsmithuchida)
 * Add upgrade test in E2E test (#4058, @danfengliu)

changelogs/CHANGELOG-1.9.md

@@ -1,79 +1,3 @@
-## v1.9.3
-### 2022-11-03
-
-### Download
-https://github.com/vmware-tanzu/velero/releases/tag/v1.9.3
-
-### Container Image
-`velero/velero:v1.9.3`
-
-### Documentation
-https://velero.io/docs/v1.9/
-
-### Upgrading
-https://velero.io/docs/v1.9/upgrade-to-1.9/
-
-### All changes
-
-  * Enhance the restore priorities list to support specifying the low prioritized resources that need to be restored in the last (#5529, @ywk253100)
-  * Fix v1.9.3 CSI VolumeSnapshot status duplicate issue. (#5518, @blackpiglet)
-  * Bump up the distroless image to the latest version (#5500, @ywk253100)
-  * Add some corner cases checking for CSI snapshot in backup controller. (#5482, @blackpiglet)
-  * Skip the exclusion check for additional resources returned by BIA (#5406, @reasonerjt)
-  * Exclude "csinodes.storage.k8s.io" and "volumeattachments.storage.k8s.io" from restore by default. (#5448, @jxun)
-  * Update the k8s.io dependencies to 0.24.0 and Removed the `WithClusterName` method as it is a "legacy field that was always cleared by the system and never used" as per upstream k8s. (#5472, @kcboyle)
-
-## v1.9.2
-### 2022-09-14
-
-### Download
-https://github.com/vmware-tanzu/velero/releases/tag/v1.9.2
-
-### Container Image
-`velero/velero:v1.9.2`
-
-### Documentation
-https://velero.io/docs/v1.9/
-
-### Upgrading
-https://velero.io/docs/v1.9/upgrade-to-1.9/
-
-### All changes
-
-  * Fix CVE-2022-1962 by bumping up golang version to 1.17.13 (#5286, @qiuming-best)
-  * Fix code spell check fail (#5300, @qiuming-best)
-  * Fix nil pointer panic when restoring StatefulSets (#5301, @divolgin)
-  * Check for empty ns list before checking nslist[0] (#5302, @sseago)
-  * check vsc null pointer (#5303, @lilongfeng0902)
-  * Fix edge cases for already exists resources (#5304, @shubham-pampattiwar)
-  * Increase ensure restic repository timeout to 5m (#5336, @shubham-pampattiwar)
-  * Added DownloadTargetKindCSIBackupVolumeSnapshots for retrieving the signed URL to download only the `<backup name>`-csi-volumesnapshots.json.gz  and DownloadTargetKindCSIBackupVolumeSnapshotContents to download only `<backup name>`-csi-volumesnapshotcontents.json.gz in the DownloadRequest CR structure. These files are already present in the backup layout. (#5307, @anshulahuja98)
-## v1.9.1
-### 2022-08-03
-
-### Download
-https://github.com/vmware-tanzu/velero/releases/tag/v1.9.1
-
-### Container Image
-`velero/velero:v1.9.1`
-
-### Documentation
-https://velero.io/docs/v1.9/
-
-### Upgrading
-https://velero.io/docs/v1.9/upgrade-to-1.9/
-
-### All changes
-
-  * Fix bsl validation bug: the BSL is validated continually and doesn't respect the validation period configured (#5112, @ywk253100)
-  * Modify BackupStoreGetter to avoid BSL spec changes (#5134, @sseago)
-  * Delay CA file deletion in PVB controller. (#5150, @jxun)
-  * Skip registering "crd-remap-version" plugin when feature flag "EnableAPIGroupVersions" is set (#5173, @reasonerjt)
-  * Fix restic backups to multiple backup storage locations bug (#5175, @qiuming-best)
-  * Make CSI snapshot creation timeout configurable. (#5189, @jxun)
-  * Add annotation "pv.kubernetes.io/migrated-to" for CSI checking. (#5186, @jxun)
-  * Bump up base image and package version to fix CVEs. (#5202, @ywk253100)
-
 ## v1.9.0
 ### 2022-06-13
 
@@ -178,4 +102,3 @@ With bumping up the API to v1 in CSI plugin, the v0.3.0 CSI plugin will only wor
   * Fix E2E test [Backups][Deletion][Restic] on GCP. (#4968, @jxun)
   * Disable status as sub resource in CRDs (#4972, @ywk253100)
   * Add more information for failing to get path or snapshot in restic backup and restore. (#4988, @jxun)
-  * When spec.RestoreStatus is empty, don't restore status (#5015, @sseago)

cmd/velero-helper/velero-helper.go

@@ -0,0 +1,43 @@
+/*
+Copyright The Velero Contributors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import (
+	"fmt"
+	"os"
+	"time"
+)
+
+const (
+	// workingModePause indicates it is for general purpose to hold the pod under running state
+	workingModePause = "pause"
+)
+
+func main() {
+	if len(os.Args) < 2 {
+		fmt.Fprintln(os.Stderr, "ERROR: at least one argument must be provided, the working mode")
+		os.Exit(1)
+	}
+
+	switch os.Args[1] {
+	case workingModePause:
+		time.Sleep(time.Duration(1<<63 - 1))
+	default:
+		fmt.Fprintln(os.Stderr, "ERROR: wrong working mode provided")
+		os.Exit(1)
+	}
+}

cmd/velero-restore-helper/velero-restore-helper.go

@@ -18,7 +18,6 @@ package main
 
 import (
 	"fmt"
-	"io/ioutil"
 	"os"
 	"path/filepath"
 	"time"
@@ -34,18 +33,16 @@ func main() {
 	defer ticker.Stop()
 
 	for {
-		select {
-		case <-ticker.C:
-			if done() {
-				fmt.Println("All restic restores are done")
-				err := removeFolder()
-				if err != nil {
-					fmt.Println(err)
-				} else {
-					fmt.Println("Done cleanup .velero folder")
-				}
-				return
+		<-ticker.C
+		if done() {
+			fmt.Println("All restic restores are done")
+			err := removeFolder()
+			if err != nil {
+				fmt.Println(err)
+			} else {
+				fmt.Println("Done cleanup .velero folder")
 			}
+			return
 		}
 	}
 }
@@ -54,7 +51,7 @@ func main() {
 // within the .velero/ subdirectory whose name is equal to os.Args[1], or
 // false otherwise
 func done() bool {
-	children, err := ioutil.ReadDir("/restores")
+	children, err := os.ReadDir("/restores")
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "ERROR reading /restores directory: %s\n", err)
 		return false
@@ -69,14 +66,14 @@ func done() bool {
 		doneFile := filepath.Join("/restores", child.Name(), ".velero", os.Args[1])
 
 		if _, err := os.Stat(doneFile); os.IsNotExist(err) {
-			fmt.Printf("Not found: %s\n", doneFile)
+			fmt.Printf("The filesystem restore done file %s is not found yet. Retry later.\n", doneFile)
 			return false
 		} else if err != nil {
-			fmt.Fprintf(os.Stderr, "ERROR looking for %s: %s\n", doneFile, err)
+			fmt.Fprintf(os.Stderr, "ERROR looking filesystem restore done file %s: %s\n", doneFile, err)
 			return false
 		}
 
-		fmt.Printf("Found %s", doneFile)
+		fmt.Printf("Found the done file %s\n", doneFile)
 	}
 
 	return true
@@ -84,7 +81,7 @@ func done() bool {
 
 // remove .velero folder
 func removeFolder() error {
-	children, err := ioutil.ReadDir("/restores")
+	children, err := os.ReadDir("/restores")
 	if err != nil {
 		return err
 	}

cmd/velero/velero.go

@@ -20,7 +20,7 @@ import (
 	"os"
 	"path/filepath"
 
-	"k8s.io/klog"
+	"k8s.io/klog/v2"
 
 	"github.com/vmware-tanzu/velero/pkg/cmd"
 	"github.com/vmware-tanzu/velero/pkg/cmd/velero"

config/crd/v1/bases/velero.io_backuprepositories.yaml

@@ -0,0 +1,111 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: backuprepositories.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: BackupRepository
+    listKind: BackupRepositoryList
+    plural: backuprepositories
+    singular: backuprepository
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    - jsonPath: .spec.repositoryType
+      name: Repository Type
+      type: string
+    name: v1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: BackupRepositorySpec is the specification for a BackupRepository.
+            properties:
+              backupStorageLocation:
+                description: |-
+                  BackupStorageLocation is the name of the BackupStorageLocation
+                  that should contain this repository.
+                type: string
+              maintenanceFrequency:
+                description: MaintenanceFrequency is how often maintenance should
+                  be run.
+                type: string
+              repositoryConfig:
+                additionalProperties:
+                  type: string
+                description: RepositoryConfig is for repository-specific configuration
+                  fields.
+                nullable: true
+                type: object
+              repositoryType:
+                description: RepositoryType indicates the type of the backend repository
+                enum:
+                - kopia
+                - restic
+                - ""
+                type: string
+              resticIdentifier:
+                description: |-
+                  ResticIdentifier is the full restic-compatible string for identifying
+                  this repository.
+                type: string
+              volumeNamespace:
+                description: |-
+                  VolumeNamespace is the namespace this backup repository contains
+                  pod volume backups for.
+                type: string
+            required:
+            - backupStorageLocation
+            - maintenanceFrequency
+            - resticIdentifier
+            - volumeNamespace
+            type: object
+          status:
+            description: BackupRepositoryStatus is the current status of a BackupRepository.
+            properties:
+              lastMaintenanceTime:
+                description: LastMaintenanceTime is the last time maintenance was
+                  run.
+                format: date-time
+                nullable: true
+                type: string
+              message:
+                description: Message is a message about the current status of the
+                  BackupRepository.
+                type: string
+              phase:
+                description: Phase is the current state of the BackupRepository.
+                enum:
+                - New
+                - Ready
+                - NotReady
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}

config/crd/v1/bases/velero.io_backups.yaml

@@ -1,11 +1,9 @@
-
 ---
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
   annotations:
-    controller-gen.kubebuilder.io/version: v0.7.0
-  creationTimestamp: null
+    controller-gen.kubebuilder.io/version: v0.14.0
   name: backups.velero.io
 spec:
   group: velero.io
@@ -19,18 +17,24 @@ spec:
   - name: v1
     schema:
       openAPIV3Schema:
-        description: Backup is a Velero resource that represents the capture of Kubernetes
+        description: |-
+          Backup is a Velero resource that represents the capture of Kubernetes
           cluster state at a point in time (API objects and associated volume state).
         properties:
           apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
             type: string
           kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
             type: string
           metadata:
             type: object
@@ -38,24 +42,63 @@ spec:
             description: BackupSpec defines the specification for a Velero backup.
             properties:
               csiSnapshotTimeout:
-                description: CSISnapshotTimeout specifies the time used to wait for
-                  CSI VolumeSnapshot status turns to ReadyToUse during creation, before
-                  returning error as timeout. The default value is 10 minute.
+                description: |-
+                  CSISnapshotTimeout specifies the time used to wait for CSI VolumeSnapshot status turns to
+                  ReadyToUse during creation, before returning error as timeout.
+                  The default value is 10 minute.
                 type: string
-              defaultVolumesToRestic:
-                description: DefaultVolumesToRestic specifies whether restic should
-                  be used to take a backup of all pod volumes by default.
+              datamover:
+                description: |-
+                  DataMover specifies the data mover to be used by the backup.
+                  If DataMover is "" or "velero", the built-in data mover will be used.
+                type: string
+              defaultVolumesToFsBackup:
+                description: |-
+                  DefaultVolumesToFsBackup specifies whether pod volume file system backup should be used
+                  for all volumes by default.
+                nullable: true
                 type: boolean
+              defaultVolumesToRestic:
+                description: |-
+                  DefaultVolumesToRestic specifies whether restic should be used to take a
+                  backup of all pod volumes by default.
+
+
+                  Deprecated: this field is no longer used and will be removed entirely in future. Use DefaultVolumesToFsBackup instead.
+                nullable: true
+                type: boolean
+              excludedClusterScopedResources:
+                description: |-
+                  ExcludedClusterScopedResources is a slice of cluster-scoped
+                  resource type names to exclude from the backup.
+                  If set to "*", all cluster-scoped resource types are excluded.
+                  The default value is empty.
+                items:
+                  type: string
+                nullable: true
+                type: array
+              excludedNamespaceScopedResources:
+                description: |-
+                  ExcludedNamespaceScopedResources is a slice of namespace-scoped
+                  resource type names to exclude from the backup.
+                  If set to "*", all namespace-scoped resource types are excluded.
+                  The default value is empty.
+                items:
+                  type: string
+                nullable: true
+                type: array
               excludedNamespaces:
-                description: ExcludedNamespaces contains a list of namespaces that
-                  are not included in the backup.
+                description: |-
+                  ExcludedNamespaces contains a list of namespaces that are not
+                  included in the backup.
                 items:
                   type: string
                 nullable: true
                 type: array
               excludedResources:
-                description: ExcludedResources is a slice of resource names that are
-                  not included in the backup.
+                description: |-
+                  ExcludedResources is a slice of resource names that are not
+                  included in the backup.
                 items:
                   type: string
                 nullable: true
@@ -68,9 +111,9 @@ spec:
                     description: Resources are hooks that should be executed when
                       backing up individual instances of a resource.
                     items:
-                      description: BackupResourceHookSpec defines one or more BackupResourceHooks
-                        that should be executed based on the rules defined for namespaces,
-                        resources, and label selector.
+                      description: |-
+                        BackupResourceHookSpec defines one or more BackupResourceHooks that should be executed based on
+                        the rules defined for namespaces, resources, and label selector.
                       properties:
                         excludedNamespaces:
                           description: ExcludedNamespaces specifies the namespaces
@@ -87,17 +130,17 @@ spec:
                           nullable: true
                           type: array
                         includedNamespaces:
-                          description: IncludedNamespaces specifies the namespaces
-                            to which this hook spec applies. If empty, it applies
+                          description: |-
+                            IncludedNamespaces specifies the namespaces to which this hook spec applies. If empty, it applies
                             to all namespaces.
                           items:
                             type: string
                           nullable: true
                           type: array
                         includedResources:
-                          description: IncludedResources specifies the resources to
-                            which this hook spec applies. If empty, it applies to
-                            all resources.
+                          description: |-
+                            IncludedResources specifies the resources to which this hook spec applies. If empty, it applies
+                            to all resources.
                           items:
                             type: string
                           nullable: true
@@ -111,8 +154,8 @@ spec:
                               description: matchExpressions is a list of label selector
                                 requirements. The requirements are ANDed.
                               items:
-                                description: A label selector requirement is a selector
-                                  that contains values, a key, and an operator that
+                                description: |-
+                                  A label selector requirement is a selector that contains values, a key, and an operator that
                                   relates the key and values.
                                 properties:
                                   key:
@@ -120,17 +163,16 @@ spec:
                                       applies to.
                                     type: string
                                   operator:
-                                    description: operator represents a key's relationship
-                                      to a set of values. Valid operators are In,
-                                      NotIn, Exists and DoesNotExist.
+                                    description: |-
+                                      operator represents a key's relationship to a set of values.
+                                      Valid operators are In, NotIn, Exists and DoesNotExist.
                                     type: string
                                   values:
-                                    description: values is an array of string values.
-                                      If the operator is In or NotIn, the values array
-                                      must be non-empty. If the operator is Exists
-                                      or DoesNotExist, the values array must be empty.
-                                      This array is replaced during a strategic merge
-                                      patch.
+                                    description: |-
+                                      values is an array of string values. If the operator is In or NotIn,
+                                      the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                      the values array must be empty. This array is replaced during a strategic
+                                      merge patch.
                                     items:
                                       type: string
                                     type: array
@@ -142,21 +184,20 @@ spec:
                             matchLabels:
                               additionalProperties:
                                 type: string
-                              description: matchLabels is a map of {key,value} pairs.
-                                A single {key,value} in the matchLabels map is equivalent
-                                to an element of matchExpressions, whose key field
-                                is "key", the operator is "In", and the values array
-                                contains only "value". The requirements are ANDed.
+                              description: |-
+                                matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                operator is "In", and the values array contains only "value". The requirements are ANDed.
                               type: object
                           type: object
+                          x-kubernetes-map-type: atomic
                         name:
                           description: Name is the name of this hook.
                           type: string
                         post:
-                          description: PostHooks is a list of BackupResourceHooks
-                            to execute after storing the item in the backup. These
-                            are executed after all "additional items" from item actions
-                            are processed.
+                          description: |-
+                            PostHooks is a list of BackupResourceHooks to execute after storing the item in the backup.
+                            These are executed after all "additional items" from item actions are processed.
                           items:
                             description: BackupResourceHook defines a hook for a resource.
                             properties:
@@ -171,10 +212,9 @@ spec:
                                     minItems: 1
                                     type: array
                                   container:
-                                    description: Container is the container in the
-                                      pod where the command should be executed. If
-                                      not specified, the pod's first container is
-                                      used.
+                                    description: |-
+                                      Container is the container in the pod where the command should be executed. If not specified,
+                                      the pod's first container is used.
                                     type: string
                                   onError:
                                     description: OnError specifies how Velero should
@@ -185,9 +225,9 @@ spec:
                                     - Fail
                                     type: string
                                   timeout:
-                                    description: Timeout defines the maximum amount
-                                      of time Velero should wait for the hook to complete
-                                      before considering the execution a failure.
+                                    description: |-
+                                      Timeout defines the maximum amount of time Velero should wait for the hook to complete before
+                                      considering the execution a failure.
                                     type: string
                                 required:
                                 - command
@@ -197,10 +237,9 @@ spec:
                             type: object
                           type: array
                         pre:
-                          description: PreHooks is a list of BackupResourceHooks to
-                            execute prior to storing the item in the backup. These
-                            are executed before any "additional items" from item actions
-                            are processed.
+                          description: |-
+                            PreHooks is a list of BackupResourceHooks to execute prior to storing the item in the backup.
+                            These are executed before any "additional items" from item actions are processed.
                           items:
                             description: BackupResourceHook defines a hook for a resource.
                             properties:
@@ -215,10 +254,9 @@ spec:
                                     minItems: 1
                                     type: array
                                   container:
-                                    description: Container is the container in the
-                                      pod where the command should be executed. If
-                                      not specified, the pod's first container is
-                                      used.
+                                    description: |-
+                                      Container is the container in the pod where the command should be executed. If not specified,
+                                      the pod's first container is used.
                                     type: string
                                   onError:
                                     description: OnError specifies how Velero should
@@ -229,9 +267,9 @@ spec:
                                     - Fail
                                     type: string
                                   timeout:
-                                    description: Timeout defines the maximum amount
-                                      of time Velero should wait for the hook to complete
-                                      before considering the execution a failure.
+                                    description: |-
+                                      Timeout defines the maximum amount of time Velero should wait for the hook to complete before
+                                      considering the execution a failure.
                                     type: string
                                 required:
                                 - command
@@ -247,52 +285,81 @@ spec:
                     type: array
                 type: object
               includeClusterResources:
-                description: IncludeClusterResources specifies whether cluster-scoped
-                  resources should be included for consideration in the backup.
+                description: |-
+                  IncludeClusterResources specifies whether cluster-scoped resources
+                  should be included for consideration in the backup.
                 nullable: true
                 type: boolean
+              includedClusterScopedResources:
+                description: |-
+                  IncludedClusterScopedResources is a slice of cluster-scoped
+                  resource type names to include in the backup.
+                  If set to "*", all cluster-scoped resource types are included.
+                  The default value is empty, which means only related
+                  cluster-scoped resources are included.
+                items:
+                  type: string
+                nullable: true
+                type: array
+              includedNamespaceScopedResources:
+                description: |-
+                  IncludedNamespaceScopedResources is a slice of namespace-scoped
+                  resource type names to include in the backup.
+                  The default value is "*".
+                items:
+                  type: string
+                nullable: true
+                type: array
               includedNamespaces:
-                description: IncludedNamespaces is a slice of namespace names to include
-                  objects from. If empty, all namespaces are included.
+                description: |-
+                  IncludedNamespaces is a slice of namespace names to include objects
+                  from. If empty, all namespaces are included.
                 items:
                   type: string
                 nullable: true
                 type: array
               includedResources:
-                description: IncludedResources is a slice of resource names to include
+                description: |-
+                  IncludedResources is a slice of resource names to include
                   in the backup. If empty, all resources are included.
                 items:
                   type: string
                 nullable: true
                 type: array
+              itemOperationTimeout:
+                description: |-
+                  ItemOperationTimeout specifies the time used to wait for asynchronous BackupItemAction operations
+                  The default value is 4 hour.
+                type: string
               labelSelector:
-                description: LabelSelector is a metav1.LabelSelector to filter with
-                  when adding individual objects to the backup. If empty or nil, all
-                  objects are included. Optional.
+                description: |-
+                  LabelSelector is a metav1.LabelSelector to filter with
+                  when adding individual objects to the backup. If empty
+                  or nil, all objects are included. Optional.
                 nullable: true
                 properties:
                   matchExpressions:
                     description: matchExpressions is a list of label selector requirements.
                       The requirements are ANDed.
                     items:
-                      description: A label selector requirement is a selector that
-                        contains values, a key, and an operator that relates the key
-                        and values.
+                      description: |-
+                        A label selector requirement is a selector that contains values, a key, and an operator that
+                        relates the key and values.
                       properties:
                         key:
                           description: key is the label key that the selector applies
                             to.
                           type: string
                         operator:
-                          description: operator represents a key's relationship to
-                            a set of values. Valid operators are In, NotIn, Exists
-                            and DoesNotExist.
+                          description: |-
+                            operator represents a key's relationship to a set of values.
+                            Valid operators are In, NotIn, Exists and DoesNotExist.
                           type: string
                         values:
-                          description: values is an array of string values. If the
-                            operator is In or NotIn, the values array must be non-empty.
-                            If the operator is Exists or DoesNotExist, the values
-                            array must be empty. This array is replaced during a strategic
+                          description: |-
+                            values is an array of string values. If the operator is In or NotIn,
+                            the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                            the values array must be empty. This array is replaced during a strategic
                             merge patch.
                           items:
                             type: string
@@ -305,13 +372,13 @@ spec:
                   matchLabels:
                     additionalProperties:
                       type: string
-                    description: matchLabels is a map of {key,value} pairs. A single
-                      {key,value} in the matchLabels map is equivalent to an element
-                      of matchExpressions, whose key field is "key", the operator
-                      is "In", and the values array contains only "value". The requirements
-                      are ANDed.
+                    description: |-
+                      matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                      map is equivalent to an element of matchExpressions, whose key field is "key", the
+                      operator is "In", and the values array contains only "value". The requirements are ANDed.
                     type: object
                 type: object
+                x-kubernetes-map-type: atomic
               metadata:
                 properties:
                   labels:
@@ -320,40 +387,41 @@ spec:
                     type: object
                 type: object
               orLabelSelectors:
-                description: OrLabelSelectors is list of metav1.LabelSelector to filter
-                  with when adding individual objects to the backup. If multiple provided
+                description: |-
+                  OrLabelSelectors is list of metav1.LabelSelector to filter with
+                  when adding individual objects to the backup. If multiple provided
                   they will be joined by the OR operator. LabelSelector as well as
-                  OrLabelSelectors cannot co-exist in backup request, only one of
-                  them can be used.
+                  OrLabelSelectors cannot co-exist in backup request, only one of them
+                  can be used.
                 items:
-                  description: A label selector is a label query over a set of resources.
-                    The result of matchLabels and matchExpressions are ANDed. An empty
-                    label selector matches all objects. A null label selector matches
-                    no objects.
+                  description: |-
+                    A label selector is a label query over a set of resources. The result of matchLabels and
+                    matchExpressions are ANDed. An empty label selector matches all objects. A null
+                    label selector matches no objects.
                   properties:
                     matchExpressions:
                       description: matchExpressions is a list of label selector requirements.
                         The requirements are ANDed.
                       items:
-                        description: A label selector requirement is a selector that
-                          contains values, a key, and an operator that relates the
-                          key and values.
+                        description: |-
+                          A label selector requirement is a selector that contains values, a key, and an operator that
+                          relates the key and values.
                         properties:
                           key:
                             description: key is the label key that the selector applies
                               to.
                             type: string
                           operator:
-                            description: operator represents a key's relationship
-                              to a set of values. Valid operators are In, NotIn, Exists
-                              and DoesNotExist.
+                            description: |-
+                              operator represents a key's relationship to a set of values.
+                              Valid operators are In, NotIn, Exists and DoesNotExist.
                             type: string
                           values:
-                            description: values is an array of string values. If the
-                              operator is In or NotIn, the values array must be non-empty.
-                              If the operator is Exists or DoesNotExist, the values
-                              array must be empty. This array is replaced during a
-                              strategic merge patch.
+                            description: |-
+                              values is an array of string values. If the operator is In or NotIn,
+                              the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                              the values array must be empty. This array is replaced during a strategic
+                              merge patch.
                             items:
                               type: string
                             type: array
@@ -365,27 +433,55 @@ spec:
                     matchLabels:
                       additionalProperties:
                         type: string
-                      description: matchLabels is a map of {key,value} pairs. A single
-                        {key,value} in the matchLabels map is equivalent to an element
-                        of matchExpressions, whose key field is "key", the operator
-                        is "In", and the values array contains only "value". The requirements
-                        are ANDed.
+                      description: |-
+                        matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                        map is equivalent to an element of matchExpressions, whose key field is "key", the
+                        operator is "In", and the values array contains only "value". The requirements are ANDed.
                       type: object
                   type: object
+                  x-kubernetes-map-type: atomic
                 nullable: true
                 type: array
               orderedResources:
                 additionalProperties:
                   type: string
-                description: OrderedResources specifies the backup order of resources
-                  of specific Kind. The map key is the Kind name and value is a list
-                  of resource names separated by commas. Each resource name has format
-                  "namespace/resourcename".  For cluster resources, simply use "resourcename".
+                description: |-
+                  OrderedResources specifies the backup order of resources of specific Kind.
+                  The map key is the resource name and value is a list of object names separated by commas.
+                  Each resource name has format "namespace/objectname".  For cluster resources, simply use "objectname".
                 nullable: true
                 type: object
+              resourcePolicy:
+                description: ResourcePolicy specifies the referenced resource policies
+                  that backup should follow
+                properties:
+                  apiGroup:
+                    description: |-
+                      APIGroup is the group for the resource being referenced.
+                      If APIGroup is not specified, the specified Kind must be in the core API group.
+                      For any other third-party types, APIGroup is required.
+                    type: string
+                  kind:
+                    description: Kind is the type of resource being referenced
+                    type: string
+                  name:
+                    description: Name is the name of resource being referenced
+                    type: string
+                required:
+                - kind
+                - name
+                type: object
+                x-kubernetes-map-type: atomic
+              snapshotMoveData:
+                description: SnapshotMoveData specifies whether snapshot data should
+                  be moved
+                nullable: true
+                type: boolean
               snapshotVolumes:
-                description: SnapshotVolumes specifies whether to take cloud snapshots
-                  of any PV's referenced in the set of objects included in the Backup.
+                description: |-
+                  SnapshotVolumes specifies whether to take snapshots
+                  of any PV's referenced in the set of objects included
+                  in the Backup.
                 nullable: true
                 type: boolean
               storageLocation:
@@ -393,9 +489,19 @@ spec:
                   BackupStorageLocation where the backup should be stored.
                 type: string
               ttl:
-                description: TTL is a time.Duration-parseable string describing how
-                  long the Backup should be retained for.
+                description: |-
+                  TTL is a time.Duration-parseable string describing how long
+                  the Backup should be retained for.
                 type: string
+              uploaderConfig:
+                description: UploaderConfig specifies the configuration for the uploader.
+                nullable: true
+                properties:
+                  parallelFilesUpload:
+                    description: ParallelFilesUpload is the number of files parallel
+                      uploads to perform when using the uploader.
+                    type: integer
+                type: object
               volumeSnapshotLocations:
                 description: VolumeSnapshotLocations is a list containing names of
                   VolumeSnapshotLocations associated with this backup.
@@ -406,26 +512,45 @@ spec:
           status:
             description: BackupStatus captures the current status of a Velero backup.
             properties:
+              backupItemOperationsAttempted:
+                description: |-
+                  BackupItemOperationsAttempted is the total number of attempted
+                  async BackupItemAction operations for this backup.
+                type: integer
+              backupItemOperationsCompleted:
+                description: |-
+                  BackupItemOperationsCompleted is the total number of successfully completed
+                  async BackupItemAction operations for this backup.
+                type: integer
+              backupItemOperationsFailed:
+                description: |-
+                  BackupItemOperationsFailed is the total number of async
+                  BackupItemAction operations for this backup which ended with an error.
+                type: integer
               completionTimestamp:
-                description: CompletionTimestamp records the time a backup was completed.
-                  Completion time is recorded even on failed backups. Completion time
-                  is recorded before uploading the backup object. The server's time
-                  is used for CompletionTimestamps
+                description: |-
+                  CompletionTimestamp records the time a backup was completed.
+                  Completion time is recorded even on failed backups.
+                  Completion time is recorded before uploading the backup object.
+                  The server's time is used for CompletionTimestamps
                 format: date-time
                 nullable: true
                 type: string
               csiVolumeSnapshotsAttempted:
-                description: CSIVolumeSnapshotsAttempted is the total number of attempted
+                description: |-
+                  CSIVolumeSnapshotsAttempted is the total number of attempted
                   CSI VolumeSnapshots for this backup.
                 type: integer
               csiVolumeSnapshotsCompleted:
-                description: CSIVolumeSnapshotsCompleted is the total number of successfully
+                description: |-
+                  CSIVolumeSnapshotsCompleted is the total number of successfully
                   completed CSI VolumeSnapshots for this backup.
                 type: integer
               errors:
-                description: Errors is a count of all error messages that were generated
-                  during execution of the backup.  The actual errors are in the backup's
-                  log file in object storage.
+                description: |-
+                  Errors is a count of all error messages that were generated during
+                  execution of the backup.  The actual errors are in the backup's log
+                  file in object storage.
                 type: integer
               expiration:
                 description: Expiration is when this Backup is eligible for garbage-collection.
@@ -440,73 +565,96 @@ spec:
                 description: FormatVersion is the backup format version, including
                   major, minor, and patch version.
                 type: string
+              hookStatus:
+                description: HookStatus contains information about the status of the
+                  hooks.
+                nullable: true
+                properties:
+                  hooksAttempted:
+                    description: |-
+                      HooksAttempted is the total number of attempted hooks
+                      Specifically, HooksAttempted represents the number of hooks that failed to execute
+                      and the number of hooks that executed successfully.
+                    type: integer
+                  hooksFailed:
+                    description: HooksFailed is the total number of hooks which ended
+                      with an error
+                    type: integer
+                type: object
               phase:
                 description: Phase is the current state of the Backup.
                 enum:
                 - New
                 - FailedValidation
                 - InProgress
+                - WaitingForPluginOperations
+                - WaitingForPluginOperationsPartiallyFailed
+                - Finalizing
+                - FinalizingPartiallyFailed
                 - Completed
                 - PartiallyFailed
                 - Failed
                 - Deleting
                 type: string
               progress:
-                description: Progress contains information about the backup's execution
-                  progress. Note that this information is best-effort only -- if Velero
-                  fails to update it during a backup for any reason, it may be inaccurate/stale.
+                description: |-
+                  Progress contains information about the backup's execution progress. Note
+                  that this information is best-effort only -- if Velero fails to update it
+                  during a backup for any reason, it may be inaccurate/stale.
                 nullable: true
                 properties:
                   itemsBackedUp:
-                    description: ItemsBackedUp is the number of items that have actually
-                      been written to the backup tarball so far.
+                    description: |-
+                      ItemsBackedUp is the number of items that have actually been written to the
+                      backup tarball so far.
                     type: integer
                   totalItems:
-                    description: TotalItems is the total number of items to be backed
-                      up. This number may change throughout the execution of the backup
-                      due to plugins that return additional related items to back
-                      up, the velero.io/exclude-from-backup label, and various other
+                    description: |-
+                      TotalItems is the total number of items to be backed up. This number may change
+                      throughout the execution of the backup due to plugins that return additional related
+                      items to back up, the velero.io/exclude-from-backup label, and various other
                       filters that happen as items are processed.
                     type: integer
                 type: object
               startTimestamp:
-                description: StartTimestamp records the time a backup was started.
-                  Separate from CreationTimestamp, since that value changes on restores.
+                description: |-
+                  StartTimestamp records the time a backup was started.
+                  Separate from CreationTimestamp, since that value changes
+                  on restores.
                   The server's time is used for StartTimestamps
                 format: date-time
                 nullable: true
                 type: string
               validationErrors:
-                description: ValidationErrors is a slice of all validation errors
-                  (if applicable).
+                description: |-
+                  ValidationErrors is a slice of all validation errors (if
+                  applicable).
                 items:
                   type: string
                 nullable: true
                 type: array
               version:
-                description: 'Version is the backup format major version. Deprecated:
-                  Please see FormatVersion'
+                description: |-
+                  Version is the backup format major version.
+                  Deprecated: Please see FormatVersion
                 type: integer
               volumeSnapshotsAttempted:
-                description: VolumeSnapshotsAttempted is the total number of attempted
+                description: |-
+                  VolumeSnapshotsAttempted is the total number of attempted
                   volume snapshots for this backup.
                 type: integer
               volumeSnapshotsCompleted:
-                description: VolumeSnapshotsCompleted is the total number of successfully
+                description: |-
+                  VolumeSnapshotsCompleted is the total number of successfully
                   completed volume snapshots for this backup.
                 type: integer
               warnings:
-                description: Warnings is a count of all warning messages that were
-                  generated during execution of the backup. The actual warnings are
-                  in the backup's log file in object storage.
+                description: |-
+                  Warnings is a count of all warning messages that were generated during
+                  execution of the backup. The actual warnings are in the backup's log
+                  file in object storage.
                 type: integer
             type: object
         type: object
     served: true
     storage: true
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crd/v1/bases/velero.io_backupstoragelocations.yaml

@@ -1,11 +1,9 @@
-
 ---
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
   annotations:
-    controller-gen.kubebuilder.io/version: v0.7.0
-  creationTimestamp: null
+    controller-gen.kubebuilder.io/version: v0.14.0
   name: backupstoragelocations.velero.io
 spec:
   group: velero.io
@@ -42,14 +40,19 @@ spec:
           objects
         properties:
           apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
             type: string
           kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
             type: string
           metadata:
             type: object
@@ -83,8 +86,10 @@ spec:
                       valid secret key.
                     type: string
                   name:
-                    description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                      TODO: Add other useful fields. apiVersion, kind, uid?'
+                    description: |-
+                      Name of the referent.
+                      More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+                      TODO: Add other useful fields. apiVersion, kind, uid?
                     type: string
                   optional:
                     description: Specify whether the Secret or its key must be defined
@@ -92,6 +97,7 @@ spec:
                 required:
                 - key
                 type: object
+                x-kubernetes-map-type: atomic
               default:
                 description: Default indicates this location is the default backup
                   storage location.
@@ -132,29 +138,36 @@ spec:
               BackupStorageLocation
             properties:
               accessMode:
-                description: "AccessMode is an unused field. \n Deprecated: there
-                  is now an AccessMode field on the Spec and this field will be removed
-                  entirely as of v2.0."
+                description: |-
+                  AccessMode is an unused field.
+
+
+                  Deprecated: there is now an AccessMode field on the Spec and this field
+                  will be removed entirely as of v2.0.
                 enum:
                 - ReadOnly
                 - ReadWrite
                 type: string
               lastSyncedRevision:
-                description: "LastSyncedRevision is the value of the `metadata/revision`
-                  file in the backup storage location the last time the BSL's contents
-                  were synced into the cluster. \n Deprecated: this field is no longer
-                  updated or used for detecting changes to the location's contents
-                  and will be removed entirely in v2.0."
+                description: |-
+                  LastSyncedRevision is the value of the `metadata/revision` file in the backup
+                  storage location the last time the BSL's contents were synced into the cluster.
+
+
+                  Deprecated: this field is no longer updated or used for detecting changes to
+                  the location's contents and will be removed entirely in v2.0.
                 type: string
               lastSyncedTime:
-                description: LastSyncedTime is the last time the contents of the location
-                  were synced into the cluster.
+                description: |-
+                  LastSyncedTime is the last time the contents of the location were synced into
+                  the cluster.
                 format: date-time
                 nullable: true
                 type: string
               lastValidationTime:
-                description: LastValidationTime is the last time the backup store
-                  location was validated the cluster.
+                description: |-
+                  LastValidationTime is the last time the backup store location was validated
+                  the cluster.
                 format: date-time
                 nullable: true
                 type: string
@@ -173,9 +186,3 @@ spec:
     served: true
     storage: true
     subresources: {}
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crd/v1/bases/velero.io_deletebackuprequests.yaml

@@ -1,11 +1,9 @@
-
 ---
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
   annotations:
-    controller-gen.kubebuilder.io/version: v0.7.0
-  creationTimestamp: null
+    controller-gen.kubebuilder.io/version: v0.14.0
   name: deletebackuprequests.velero.io
 spec:
   group: velero.io
@@ -31,14 +29,19 @@ spec:
         description: DeleteBackupRequest is a request to delete one or more backups.
         properties:
           apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
             type: string
           kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
             type: string
           metadata:
             type: object
@@ -73,9 +76,3 @@ spec:
     served: true
     storage: true
     subresources: {}
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crd/v1/bases/velero.io_downloadrequests.yaml

@@ -1,11 +1,9 @@
-
 ---
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
   annotations:
-    controller-gen.kubebuilder.io/version: v0.7.0
-  creationTimestamp: null
+    controller-gen.kubebuilder.io/version: v0.14.0
   name: downloadrequests.velero.io
 spec:
   group: velero.io
@@ -19,18 +17,24 @@ spec:
   - name: v1
     schema:
       openAPIV3Schema:
-        description: DownloadRequest is a request to download an artifact from backup
-          object storage, such as a backup log file.
+        description: |-
+          DownloadRequest is a request to download an artifact from backup object storage, such as a backup
+          log file.
         properties:
           apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
             type: string
           kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
             type: string
           metadata:
             type: object
@@ -46,15 +50,20 @@ spec:
                     - BackupLog
                     - BackupContents
                     - BackupVolumeSnapshots
-                    - BackupItemSnapshots
+                    - BackupItemOperations
                     - BackupResourceList
+                    - BackupResults
                     - RestoreLog
                     - RestoreResults
+                    - RestoreResourceList
+                    - RestoreItemOperations
                     - CSIBackupVolumeSnapshots
                     - CSIBackupVolumeSnapshotContents
+                    - BackupVolumeInfos
+                    - RestoreVolumeInfo
                     type: string
                   name:
-                    description: Name is the name of the kubernetes resource with
+                    description: Name is the name of the Kubernetes resource with
                       which the file is associated.
                     type: string
                 required:
@@ -87,9 +96,3 @@ spec:
         type: object
     served: true
     storage: true
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crd/v1/bases/velero.io_podvolumebackups.yaml

@@ -1,11 +1,9 @@
-
 ---
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
   annotations:
-    controller-gen.kubebuilder.io/version: v0.7.0
-  creationTimestamp: null
+    controller-gen.kubebuilder.io/version: v0.14.0
   name: podvolumebackups.velero.io
 spec:
   group: velero.io
@@ -37,9 +35,9 @@ spec:
       jsonPath: .spec.volume
       name: Volume
       type: string
-    - description: Restic repository identifier for this backup
-      jsonPath: .spec.repoIdentifier
-      name: Restic Repo
+    - description: The type of the uploader to handle data transfer
+      jsonPath: .spec.uploaderType
+      name: Uploader Type
       type: string
     - description: Name of the Backup Storage Location where this backup should be
         stored
@@ -54,14 +52,19 @@ spec:
       openAPIV3Schema:
         properties:
           apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
             type: string
           kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
             type: string
           metadata:
             type: object
@@ -69,8 +72,9 @@ spec:
             description: PodVolumeBackupSpec is the specification for a PodVolumeBackup.
             properties:
               backupStorageLocation:
-                description: BackupStorageLocation is the name of the backup storage
-                  location where the restic repository is stored.
+                description: |-
+                  BackupStorageLocation is the name of the backup storage location
+                  where the backup repository is stored.
                 type: string
               node:
                 description: Node is the name of the node that the Pod is running
@@ -84,47 +88,73 @@ spec:
                     description: API version of the referent.
                     type: string
                   fieldPath:
-                    description: 'If referring to a piece of an object instead of
-                      an entire object, this string should contain a valid JSON/Go
-                      field access statement, such as desiredState.manifest.containers[2].
-                      For example, if the object reference is to a container within
-                      a pod, this would take on a value like: "spec.containers{name}"
-                      (where "name" refers to the name of the container that triggered
-                      the event) or if no container name is specified "spec.containers[2]"
-                      (container with index 2 in this pod). This syntax is chosen
-                      only to have some well-defined way of referencing a part of
-                      an object. TODO: this design is not final and this field is
-                      subject to change in the future.'
+                    description: |-
+                      If referring to a piece of an object instead of an entire object, this string
+                      should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2].
+                      For example, if the object reference is to a container within a pod, this would take on a value like:
+                      "spec.containers{name}" (where "name" refers to the name of the container that triggered
+                      the event) or if no container name is specified "spec.containers[2]" (container with
+                      index 2 in this pod). This syntax is chosen only to have some well-defined way of
+                      referencing a part of an object.
+                      TODO: this design is not final and this field is subject to change in the future.
                     type: string
                   kind:
-                    description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                    description: |-
+                      Kind of the referent.
+                      More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
                     type: string
                   name:
-                    description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                    description: |-
+                      Name of the referent.
+                      More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                     type: string
                   namespace:
-                    description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                    description: |-
+                      Namespace of the referent.
+                      More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
                     type: string
                   resourceVersion:
-                    description: 'Specific resourceVersion to which this reference
-                      is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                    description: |-
+                      Specific resourceVersion to which this reference is made, if any.
+                      More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency
                     type: string
                   uid:
-                    description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                    description: |-
+                      UID of the referent.
+                      More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids
                     type: string
                 type: object
+                x-kubernetes-map-type: atomic
               repoIdentifier:
-                description: RepoIdentifier is the restic repository identifier.
+                description: RepoIdentifier is the backup repository identifier.
                 type: string
               tags:
                 additionalProperties:
                   type: string
-                description: Tags are a map of key-value pairs that should be applied
-                  to the volume backup as tags.
+                description: |-
+                  Tags are a map of key-value pairs that should be applied to the
+                  volume backup as tags.
                 type: object
+              uploaderSettings:
+                additionalProperties:
+                  type: string
+                description: |-
+                  UploaderSettings are a map of key-value pairs that should be applied to the
+                  uploader configuration.
+                nullable: true
+                type: object
+              uploaderType:
+                description: UploaderType is the type of the uploader to handle the
+                  data transfer.
+                enum:
+                - kopia
+                - restic
+                - ""
+                type: string
               volume:
-                description: Volume is the name of the volume within the Pod to be
-                  backed up.
+                description: |-
+                  Volume is the name of the volume within the Pod to be backed
+                  up.
                 type: string
             required:
             - backupStorageLocation
@@ -137,10 +167,11 @@ spec:
             description: PodVolumeBackupStatus is the current status of a PodVolumeBackup.
             properties:
               completionTimestamp:
-                description: CompletionTimestamp records the time a backup was completed.
-                  Completion time is recorded even on failed backups. Completion time
-                  is recorded before uploading the backup object. The server's time
-                  is used for CompletionTimestamps
+                description: |-
+                  CompletionTimestamp records the time a backup was completed.
+                  Completion time is recorded even on failed backups.
+                  Completion time is recorded before uploading the backup object.
+                  The server's time is used for CompletionTimestamps
                 format: date-time
                 nullable: true
                 type: string
@@ -160,9 +191,10 @@ spec:
                 - Failed
                 type: string
               progress:
-                description: Progress holds the total number of bytes of the volume
-                  and the current number of backed up bytes. This can be used to display
-                  progress information about the backup operation.
+                description: |-
+                  Progress holds the total number of bytes of the volume and the current
+                  number of backed up bytes. This can be used to display progress information
+                  about the backup operation.
                 properties:
                   bytesDone:
                     format: int64
@@ -176,8 +208,10 @@ spec:
                   pod volume.
                 type: string
               startTimestamp:
-                description: StartTimestamp records the time a backup was started.
-                  Separate from CreationTimestamp, since that value changes on restores.
+                description: |-
+                  StartTimestamp records the time a backup was started.
+                  Separate from CreationTimestamp, since that value changes
+                  on restores.
                   The server's time is used for StartTimestamps
                 format: date-time
                 nullable: true
@@ -187,9 +221,3 @@ spec:
     served: true
     storage: true
     subresources: {}
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crd/v1/bases/velero.io_podvolumerestores.yaml

@@ -1,11 +1,9 @@
-
 ---
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
   annotations:
-    controller-gen.kubebuilder.io/version: v0.7.0
-  creationTimestamp: null
+    controller-gen.kubebuilder.io/version: v0.14.0
   name: podvolumerestores.velero.io
 spec:
   group: velero.io
@@ -25,6 +23,10 @@ spec:
       jsonPath: .spec.pod.name
       name: Pod
       type: string
+    - description: The type of the uploader to handle data transfer
+      jsonPath: .spec.uploaderType
+      name: Uploader Type
+      type: string
     - description: Name of the volume to be restored
       jsonPath: .spec.volume
       name: Volume
@@ -51,14 +53,19 @@ spec:
       openAPIV3Schema:
         properties:
           apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
             type: string
           kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
             type: string
           metadata:
             type: object
@@ -66,8 +73,9 @@ spec:
             description: PodVolumeRestoreSpec is the specification for a PodVolumeRestore.
             properties:
               backupStorageLocation:
-                description: BackupStorageLocation is the name of the backup storage
-                  location where the restic repository is stored.
+                description: |-
+                  BackupStorageLocation is the name of the backup storage location
+                  where the backup repository is stored.
                 type: string
               pod:
                 description: Pod is a reference to the pod containing the volume to
@@ -77,41 +85,69 @@ spec:
                     description: API version of the referent.
                     type: string
                   fieldPath:
-                    description: 'If referring to a piece of an object instead of
-                      an entire object, this string should contain a valid JSON/Go
-                      field access statement, such as desiredState.manifest.containers[2].
-                      For example, if the object reference is to a container within
-                      a pod, this would take on a value like: "spec.containers{name}"
-                      (where "name" refers to the name of the container that triggered
-                      the event) or if no container name is specified "spec.containers[2]"
-                      (container with index 2 in this pod). This syntax is chosen
-                      only to have some well-defined way of referencing a part of
-                      an object. TODO: this design is not final and this field is
-                      subject to change in the future.'
+                    description: |-
+                      If referring to a piece of an object instead of an entire object, this string
+                      should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2].
+                      For example, if the object reference is to a container within a pod, this would take on a value like:
+                      "spec.containers{name}" (where "name" refers to the name of the container that triggered
+                      the event) or if no container name is specified "spec.containers[2]" (container with
+                      index 2 in this pod). This syntax is chosen only to have some well-defined way of
+                      referencing a part of an object.
+                      TODO: this design is not final and this field is subject to change in the future.
                     type: string
                   kind:
-                    description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                    description: |-
+                      Kind of the referent.
+                      More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
                     type: string
                   name:
-                    description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                    description: |-
+                      Name of the referent.
+                      More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                     type: string
                   namespace:
-                    description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                    description: |-
+                      Namespace of the referent.
+                      More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
                     type: string
                   resourceVersion:
-                    description: 'Specific resourceVersion to which this reference
-                      is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                    description: |-
+                      Specific resourceVersion to which this reference is made, if any.
+                      More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency
                     type: string
                   uid:
-                    description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                    description: |-
+                      UID of the referent.
+                      More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids
                     type: string
                 type: object
+                x-kubernetes-map-type: atomic
               repoIdentifier:
-                description: RepoIdentifier is the restic repository identifier.
+                description: RepoIdentifier is the backup repository identifier.
                 type: string
               snapshotID:
                 description: SnapshotID is the ID of the volume snapshot to be restored.
                 type: string
+              sourceNamespace:
+                description: SourceNamespace is the original namespace for namaspace
+                  mapping.
+                type: string
+              uploaderSettings:
+                additionalProperties:
+                  type: string
+                description: |-
+                  UploaderSettings are a map of key-value pairs that should be applied to the
+                  uploader configuration.
+                nullable: true
+                type: object
+              uploaderType:
+                description: UploaderType is the type of the uploader to handle the
+                  data transfer.
+                enum:
+                - kopia
+                - restic
+                - ""
+                type: string
               volume:
                 description: Volume is the name of the volume within the Pod to be
                   restored.
@@ -121,15 +157,17 @@ spec:
             - pod
             - repoIdentifier
             - snapshotID
+            - sourceNamespace
             - volume
             type: object
           status:
             description: PodVolumeRestoreStatus is the current status of a PodVolumeRestore.
             properties:
               completionTimestamp:
-                description: CompletionTimestamp records the time a restore was completed.
-                  Completion time is recorded even on failed restores. The server's
-                  time is used for CompletionTimestamps
+                description: |-
+                  CompletionTimestamp records the time a restore was completed.
+                  Completion time is recorded even on failed restores.
+                  The server's time is used for CompletionTimestamps
                 format: date-time
                 nullable: true
                 type: string
@@ -145,9 +183,10 @@ spec:
                 - Failed
                 type: string
               progress:
-                description: Progress holds the total number of bytes of the snapshot
-                  and the current number of restored bytes. This can be used to display
-                  progress information about the restore operation.
+                description: |-
+                  Progress holds the total number of bytes of the snapshot and the current
+                  number of restored bytes. This can be used to display progress information
+                  about the restore operation.
                 properties:
                   bytesDone:
                     format: int64
@@ -157,7 +196,8 @@ spec:
                     type: integer
                 type: object
               startTimestamp:
-                description: StartTimestamp records the time a restore was started.
+                description: |-
+                  StartTimestamp records the time a restore was started.
                   The server's time is used for StartTimestamps
                 format: date-time
                 nullable: true
@@ -167,9 +207,3 @@ spec:
     served: true
     storage: true
     subresources: {}
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crd/v1/bases/velero.io_resticrepositories.yaml

@@ -1,94 +0,0 @@
-
----
-apiVersion: apiextensions.k8s.io/v1
-kind: CustomResourceDefinition
-metadata:
-  annotations:
-    controller-gen.kubebuilder.io/version: v0.7.0
-  creationTimestamp: null
-  name: resticrepositories.velero.io
-spec:
-  group: velero.io
-  names:
-    kind: ResticRepository
-    listKind: ResticRepositoryList
-    plural: resticrepositories
-    singular: resticrepository
-  scope: Namespaced
-  versions:
-  - additionalPrinterColumns:
-    - jsonPath: .metadata.creationTimestamp
-      name: Age
-      type: date
-    name: v1
-    schema:
-      openAPIV3Schema:
-        properties:
-          apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
-            type: string
-          kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-            type: string
-          metadata:
-            type: object
-          spec:
-            description: ResticRepositorySpec is the specification for a ResticRepository.
-            properties:
-              backupStorageLocation:
-                description: BackupStorageLocation is the name of the BackupStorageLocation
-                  that should contain this repository.
-                type: string
-              maintenanceFrequency:
-                description: MaintenanceFrequency is how often maintenance should
-                  be run.
-                type: string
-              resticIdentifier:
-                description: ResticIdentifier is the full restic-compatible string
-                  for identifying this repository.
-                type: string
-              volumeNamespace:
-                description: VolumeNamespace is the namespace this restic repository
-                  contains pod volume backups for.
-                type: string
-            required:
-            - backupStorageLocation
-            - maintenanceFrequency
-            - resticIdentifier
-            - volumeNamespace
-            type: object
-          status:
-            description: ResticRepositoryStatus is the current status of a ResticRepository.
-            properties:
-              lastMaintenanceTime:
-                description: LastMaintenanceTime is the last time maintenance was
-                  run.
-                format: date-time
-                nullable: true
-                type: string
-              message:
-                description: Message is a message about the current status of the
-                  ResticRepository.
-                type: string
-              phase:
-                description: Phase is the current state of the ResticRepository.
-                enum:
-                - New
-                - Ready
-                - NotReady
-                type: string
-            type: object
-        type: object
-    served: true
-    storage: true
-    subresources: {}
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crd/v1/bases/velero.io_schedules.yaml

@@ -1,11 +1,9 @@
-
 ---
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
   annotations:
-    controller-gen.kubebuilder.io/version: v0.7.0
-  creationTimestamp: null
+    controller-gen.kubebuilder.io/version: v0.14.0
   name: schedules.velero.io
 spec:
   group: velero.io
@@ -32,54 +30,114 @@ spec:
     - jsonPath: .metadata.creationTimestamp
       name: Age
       type: date
+    - jsonPath: .spec.paused
+      name: Paused
+      type: boolean
     name: v1
     schema:
       openAPIV3Schema:
-        description: Schedule is a Velero resource that represents a pre-scheduled
-          or periodic Backup that should be run.
+        description: |-
+          Schedule is a Velero resource that represents a pre-scheduled or
+          periodic Backup that should be run.
         properties:
           apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
             type: string
           kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
             type: string
           metadata:
             type: object
           spec:
             description: ScheduleSpec defines the specification for a Velero schedule
             properties:
+              paused:
+                description: Paused specifies whether the schedule is paused or not
+                type: boolean
               schedule:
-                description: Schedule is a Cron expression defining when to run the
-                  Backup.
+                description: |-
+                  Schedule is a Cron expression defining when to run
+                  the Backup.
                 type: string
+              skipImmediately:
+                description: |-
+                  SkipImmediately specifies whether to skip backup if schedule is due immediately from `schedule.status.lastBackup` timestamp when schedule is unpaused or if schedule is new.
+                  If true, backup will be skipped immediately when schedule is unpaused if it is due based on .Status.LastBackupTimestamp or schedule is new, and will run at next schedule time.
+                  If false, backup will not be skipped immediately when schedule is unpaused, but will run at next schedule time.
+                  If empty, will follow server configuration (default: false).
+                type: boolean
               template:
-                description: Template is the definition of the Backup to be run on
-                  the provided schedule
+                description: |-
+                  Template is the definition of the Backup to be run
+                  on the provided schedule
                 properties:
                   csiSnapshotTimeout:
-                    description: CSISnapshotTimeout specifies the time used to wait
-                      for CSI VolumeSnapshot status turns to ReadyToUse during creation,
-                      before returning error as timeout. The default value is 10 minute.
+                    description: |-
+                      CSISnapshotTimeout specifies the time used to wait for CSI VolumeSnapshot status turns to
+                      ReadyToUse during creation, before returning error as timeout.
+                      The default value is 10 minute.
                     type: string
-                  defaultVolumesToRestic:
-                    description: DefaultVolumesToRestic specifies whether restic should
-                      be used to take a backup of all pod volumes by default.
+                  datamover:
+                    description: |-
+                      DataMover specifies the data mover to be used by the backup.
+                      If DataMover is "" or "velero", the built-in data mover will be used.
+                    type: string
+                  defaultVolumesToFsBackup:
+                    description: |-
+                      DefaultVolumesToFsBackup specifies whether pod volume file system backup should be used
+                      for all volumes by default.
+                    nullable: true
                     type: boolean
+                  defaultVolumesToRestic:
+                    description: |-
+                      DefaultVolumesToRestic specifies whether restic should be used to take a
+                      backup of all pod volumes by default.
+
+
+                      Deprecated: this field is no longer used and will be removed entirely in future. Use DefaultVolumesToFsBackup instead.
+                    nullable: true
+                    type: boolean
+                  excludedClusterScopedResources:
+                    description: |-
+                      ExcludedClusterScopedResources is a slice of cluster-scoped
+                      resource type names to exclude from the backup.
+                      If set to "*", all cluster-scoped resource types are excluded.
+                      The default value is empty.
+                    items:
+                      type: string
+                    nullable: true
+                    type: array
+                  excludedNamespaceScopedResources:
+                    description: |-
+                      ExcludedNamespaceScopedResources is a slice of namespace-scoped
+                      resource type names to exclude from the backup.
+                      If set to "*", all namespace-scoped resource types are excluded.
+                      The default value is empty.
+                    items:
+                      type: string
+                    nullable: true
+                    type: array
                   excludedNamespaces:
-                    description: ExcludedNamespaces contains a list of namespaces
-                      that are not included in the backup.
+                    description: |-
+                      ExcludedNamespaces contains a list of namespaces that are not
+                      included in the backup.
                     items:
                       type: string
                     nullable: true
                     type: array
                   excludedResources:
-                    description: ExcludedResources is a slice of resource names that
-                      are not included in the backup.
+                    description: |-
+                      ExcludedResources is a slice of resource names that are not
+                      included in the backup.
                     items:
                       type: string
                     nullable: true
@@ -92,9 +150,9 @@ spec:
                         description: Resources are hooks that should be executed when
                           backing up individual instances of a resource.
                         items:
-                          description: BackupResourceHookSpec defines one or more
-                            BackupResourceHooks that should be executed based on the
-                            rules defined for namespaces, resources, and label selector.
+                          description: |-
+                            BackupResourceHookSpec defines one or more BackupResourceHooks that should be executed based on
+                            the rules defined for namespaces, resources, and label selector.
                           properties:
                             excludedNamespaces:
                               description: ExcludedNamespaces specifies the namespaces
@@ -111,16 +169,16 @@ spec:
                               nullable: true
                               type: array
                             includedNamespaces:
-                              description: IncludedNamespaces specifies the namespaces
-                                to which this hook spec applies. If empty, it applies
+                              description: |-
+                                IncludedNamespaces specifies the namespaces to which this hook spec applies. If empty, it applies
                                 to all namespaces.
                               items:
                                 type: string
                               nullable: true
                               type: array
                             includedResources:
-                              description: IncludedResources specifies the resources
-                                to which this hook spec applies. If empty, it applies
+                              description: |-
+                                IncludedResources specifies the resources to which this hook spec applies. If empty, it applies
                                 to all resources.
                               items:
                                 type: string
@@ -135,26 +193,25 @@ spec:
                                   description: matchExpressions is a list of label
                                     selector requirements. The requirements are ANDed.
                                   items:
-                                    description: A label selector requirement is a
-                                      selector that contains values, a key, and an
-                                      operator that relates the key and values.
+                                    description: |-
+                                      A label selector requirement is a selector that contains values, a key, and an operator that
+                                      relates the key and values.
                                     properties:
                                       key:
                                         description: key is the label key that the
                                           selector applies to.
                                         type: string
                                       operator:
-                                        description: operator represents a key's relationship
-                                          to a set of values. Valid operators are
-                                          In, NotIn, Exists and DoesNotExist.
+                                        description: |-
+                                          operator represents a key's relationship to a set of values.
+                                          Valid operators are In, NotIn, Exists and DoesNotExist.
                                         type: string
                                       values:
-                                        description: values is an array of string
-                                          values. If the operator is In or NotIn,
-                                          the values array must be non-empty. If the
-                                          operator is Exists or DoesNotExist, the
-                                          values array must be empty. This array is
-                                          replaced during a strategic merge patch.
+                                        description: |-
+                                          values is an array of string values. If the operator is In or NotIn,
+                                          the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                          the values array must be empty. This array is replaced during a strategic
+                                          merge patch.
                                         items:
                                           type: string
                                         type: array
@@ -166,22 +223,20 @@ spec:
                                 matchLabels:
                                   additionalProperties:
                                     type: string
-                                  description: matchLabels is a map of {key,value}
-                                    pairs. A single {key,value} in the matchLabels
-                                    map is equivalent to an element of matchExpressions,
-                                    whose key field is "key", the operator is "In",
-                                    and the values array contains only "value". The
-                                    requirements are ANDed.
+                                  description: |-
+                                    matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                                    map is equivalent to an element of matchExpressions, whose key field is "key", the
+                                    operator is "In", and the values array contains only "value". The requirements are ANDed.
                                   type: object
                               type: object
+                              x-kubernetes-map-type: atomic
                             name:
                               description: Name is the name of this hook.
                               type: string
                             post:
-                              description: PostHooks is a list of BackupResourceHooks
-                                to execute after storing the item in the backup. These
-                                are executed after all "additional items" from item
-                                actions are processed.
+                              description: |-
+                                PostHooks is a list of BackupResourceHooks to execute after storing the item in the backup.
+                                These are executed after all "additional items" from item actions are processed.
                               items:
                                 description: BackupResourceHook defines a hook for
                                   a resource.
@@ -197,10 +252,9 @@ spec:
                                         minItems: 1
                                         type: array
                                       container:
-                                        description: Container is the container in
-                                          the pod where the command should be executed.
-                                          If not specified, the pod's first container
-                                          is used.
+                                        description: |-
+                                          Container is the container in the pod where the command should be executed. If not specified,
+                                          the pod's first container is used.
                                         type: string
                                       onError:
                                         description: OnError specifies how Velero
@@ -211,10 +265,9 @@ spec:
                                         - Fail
                                         type: string
                                       timeout:
-                                        description: Timeout defines the maximum amount
-                                          of time Velero should wait for the hook
-                                          to complete before considering the execution
-                                          a failure.
+                                        description: |-
+                                          Timeout defines the maximum amount of time Velero should wait for the hook to complete before
+                                          considering the execution a failure.
                                         type: string
                                     required:
                                     - command
@@ -224,10 +277,9 @@ spec:
                                 type: object
                               type: array
                             pre:
-                              description: PreHooks is a list of BackupResourceHooks
-                                to execute prior to storing the item in the backup.
-                                These are executed before any "additional items" from
-                                item actions are processed.
+                              description: |-
+                                PreHooks is a list of BackupResourceHooks to execute prior to storing the item in the backup.
+                                These are executed before any "additional items" from item actions are processed.
                               items:
                                 description: BackupResourceHook defines a hook for
                                   a resource.
@@ -243,10 +295,9 @@ spec:
                                         minItems: 1
                                         type: array
                                       container:
-                                        description: Container is the container in
-                                          the pod where the command should be executed.
-                                          If not specified, the pod's first container
-                                          is used.
+                                        description: |-
+                                          Container is the container in the pod where the command should be executed. If not specified,
+                                          the pod's first container is used.
                                         type: string
                                       onError:
                                         description: OnError specifies how Velero
@@ -257,10 +308,9 @@ spec:
                                         - Fail
                                         type: string
                                       timeout:
-                                        description: Timeout defines the maximum amount
-                                          of time Velero should wait for the hook
-                                          to complete before considering the execution
-                                          a failure.
+                                        description: |-
+                                          Timeout defines the maximum amount of time Velero should wait for the hook to complete before
+                                          considering the execution a failure.
                                         type: string
                                     required:
                                     - command
@@ -276,27 +326,56 @@ spec:
                         type: array
                     type: object
                   includeClusterResources:
-                    description: IncludeClusterResources specifies whether cluster-scoped
-                      resources should be included for consideration in the backup.
+                    description: |-
+                      IncludeClusterResources specifies whether cluster-scoped resources
+                      should be included for consideration in the backup.
                     nullable: true
                     type: boolean
+                  includedClusterScopedResources:
+                    description: |-
+                      IncludedClusterScopedResources is a slice of cluster-scoped
+                      resource type names to include in the backup.
+                      If set to "*", all cluster-scoped resource types are included.
+                      The default value is empty, which means only related
+                      cluster-scoped resources are included.
+                    items:
+                      type: string
+                    nullable: true
+                    type: array
+                  includedNamespaceScopedResources:
+                    description: |-
+                      IncludedNamespaceScopedResources is a slice of namespace-scoped
+                      resource type names to include in the backup.
+                      The default value is "*".
+                    items:
+                      type: string
+                    nullable: true
+                    type: array
                   includedNamespaces:
-                    description: IncludedNamespaces is a slice of namespace names
-                      to include objects from. If empty, all namespaces are included.
+                    description: |-
+                      IncludedNamespaces is a slice of namespace names to include objects
+                      from. If empty, all namespaces are included.
                     items:
                       type: string
                     nullable: true
                     type: array
                   includedResources:
-                    description: IncludedResources is a slice of resource names to
-                      include in the backup. If empty, all resources are included.
+                    description: |-
+                      IncludedResources is a slice of resource names to include
+                      in the backup. If empty, all resources are included.
                     items:
                       type: string
                     nullable: true
                     type: array
+                  itemOperationTimeout:
+                    description: |-
+                      ItemOperationTimeout specifies the time used to wait for asynchronous BackupItemAction operations
+                      The default value is 4 hour.
+                    type: string
                   labelSelector:
-                    description: LabelSelector is a metav1.LabelSelector to filter
-                      with when adding individual objects to the backup. If empty
+                    description: |-
+                      LabelSelector is a metav1.LabelSelector to filter with
+                      when adding individual objects to the backup. If empty
                       or nil, all objects are included. Optional.
                     nullable: true
                     properties:
@@ -304,25 +383,25 @@ spec:
                         description: matchExpressions is a list of label selector
                           requirements. The requirements are ANDed.
                         items:
-                          description: A label selector requirement is a selector
-                            that contains values, a key, and an operator that relates
-                            the key and values.
+                          description: |-
+                            A label selector requirement is a selector that contains values, a key, and an operator that
+                            relates the key and values.
                           properties:
                             key:
                               description: key is the label key that the selector
                                 applies to.
                               type: string
                             operator:
-                              description: operator represents a key's relationship
-                                to a set of values. Valid operators are In, NotIn,
-                                Exists and DoesNotExist.
+                              description: |-
+                                operator represents a key's relationship to a set of values.
+                                Valid operators are In, NotIn, Exists and DoesNotExist.
                               type: string
                             values:
-                              description: values is an array of string values. If
-                                the operator is In or NotIn, the values array must
-                                be non-empty. If the operator is Exists or DoesNotExist,
-                                the values array must be empty. This array is replaced
-                                during a strategic merge patch.
+                              description: |-
+                                values is an array of string values. If the operator is In or NotIn,
+                                the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                the values array must be empty. This array is replaced during a strategic
+                                merge patch.
                               items:
                                 type: string
                               type: array
@@ -334,13 +413,13 @@ spec:
                       matchLabels:
                         additionalProperties:
                           type: string
-                        description: matchLabels is a map of {key,value} pairs. A
-                          single {key,value} in the matchLabels map is equivalent
-                          to an element of matchExpressions, whose key field is "key",
-                          the operator is "In", and the values array contains only
-                          "value". The requirements are ANDed.
+                        description: |-
+                          matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                          map is equivalent to an element of matchExpressions, whose key field is "key", the
+                          operator is "In", and the values array contains only "value". The requirements are ANDed.
                         type: object
                     type: object
+                    x-kubernetes-map-type: atomic
                   metadata:
                     properties:
                       labels:
@@ -349,40 +428,41 @@ spec:
                         type: object
                     type: object
                   orLabelSelectors:
-                    description: OrLabelSelectors is list of metav1.LabelSelector
-                      to filter with when adding individual objects to the backup.
-                      If multiple provided they will be joined by the OR operator.
-                      LabelSelector as well as OrLabelSelectors cannot co-exist in
-                      backup request, only one of them can be used.
+                    description: |-
+                      OrLabelSelectors is list of metav1.LabelSelector to filter with
+                      when adding individual objects to the backup. If multiple provided
+                      they will be joined by the OR operator. LabelSelector as well as
+                      OrLabelSelectors cannot co-exist in backup request, only one of them
+                      can be used.
                     items:
-                      description: A label selector is a label query over a set of
-                        resources. The result of matchLabels and matchExpressions
-                        are ANDed. An empty label selector matches all objects. A
-                        null label selector matches no objects.
+                      description: |-
+                        A label selector is a label query over a set of resources. The result of matchLabels and
+                        matchExpressions are ANDed. An empty label selector matches all objects. A null
+                        label selector matches no objects.
                       properties:
                         matchExpressions:
                           description: matchExpressions is a list of label selector
                             requirements. The requirements are ANDed.
                           items:
-                            description: A label selector requirement is a selector
-                              that contains values, a key, and an operator that relates
-                              the key and values.
+                            description: |-
+                              A label selector requirement is a selector that contains values, a key, and an operator that
+                              relates the key and values.
                             properties:
                               key:
                                 description: key is the label key that the selector
                                   applies to.
                                 type: string
                               operator:
-                                description: operator represents a key's relationship
-                                  to a set of values. Valid operators are In, NotIn,
-                                  Exists and DoesNotExist.
+                                description: |-
+                                  operator represents a key's relationship to a set of values.
+                                  Valid operators are In, NotIn, Exists and DoesNotExist.
                                 type: string
                               values:
-                                description: values is an array of string values.
-                                  If the operator is In or NotIn, the values array
-                                  must be non-empty. If the operator is Exists or
-                                  DoesNotExist, the values array must be empty. This
-                                  array is replaced during a strategic merge patch.
+                                description: |-
+                                  values is an array of string values. If the operator is In or NotIn,
+                                  the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                                  the values array must be empty. This array is replaced during a strategic
+                                  merge patch.
                                 items:
                                   type: string
                                 type: array
@@ -394,29 +474,55 @@ spec:
                         matchLabels:
                           additionalProperties:
                             type: string
-                          description: matchLabels is a map of {key,value} pairs.
-                            A single {key,value} in the matchLabels map is equivalent
-                            to an element of matchExpressions, whose key field is
-                            "key", the operator is "In", and the values array contains
-                            only "value". The requirements are ANDed.
+                          description: |-
+                            matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                            map is equivalent to an element of matchExpressions, whose key field is "key", the
+                            operator is "In", and the values array contains only "value". The requirements are ANDed.
                           type: object
                       type: object
+                      x-kubernetes-map-type: atomic
                     nullable: true
                     type: array
                   orderedResources:
                     additionalProperties:
                       type: string
-                    description: OrderedResources specifies the backup order of resources
-                      of specific Kind. The map key is the Kind name and value is
-                      a list of resource names separated by commas. Each resource
-                      name has format "namespace/resourcename".  For cluster resources,
-                      simply use "resourcename".
+                    description: |-
+                      OrderedResources specifies the backup order of resources of specific Kind.
+                      The map key is the resource name and value is a list of object names separated by commas.
+                      Each resource name has format "namespace/objectname".  For cluster resources, simply use "objectname".
                     nullable: true
                     type: object
+                  resourcePolicy:
+                    description: ResourcePolicy specifies the referenced resource
+                      policies that backup should follow
+                    properties:
+                      apiGroup:
+                        description: |-
+                          APIGroup is the group for the resource being referenced.
+                          If APIGroup is not specified, the specified Kind must be in the core API group.
+                          For any other third-party types, APIGroup is required.
+                        type: string
+                      kind:
+                        description: Kind is the type of resource being referenced
+                        type: string
+                      name:
+                        description: Name is the name of resource being referenced
+                        type: string
+                    required:
+                    - kind
+                    - name
+                    type: object
+                    x-kubernetes-map-type: atomic
+                  snapshotMoveData:
+                    description: SnapshotMoveData specifies whether snapshot data
+                      should be moved
+                    nullable: true
+                    type: boolean
                   snapshotVolumes:
-                    description: SnapshotVolumes specifies whether to take cloud snapshots
-                      of any PV's referenced in the set of objects included in the
-                      Backup.
+                    description: |-
+                      SnapshotVolumes specifies whether to take snapshots
+                      of any PV's referenced in the set of objects included
+                      in the Backup.
                     nullable: true
                     type: boolean
                   storageLocation:
@@ -424,9 +530,20 @@ spec:
                       a BackupStorageLocation where the backup should be stored.
                     type: string
                   ttl:
-                    description: TTL is a time.Duration-parseable string describing
-                      how long the Backup should be retained for.
+                    description: |-
+                      TTL is a time.Duration-parseable string describing how long
+                      the Backup should be retained for.
                     type: string
+                  uploaderConfig:
+                    description: UploaderConfig specifies the configuration for the
+                      uploader.
+                    nullable: true
+                    properties:
+                      parallelFilesUpload:
+                        description: ParallelFilesUpload is the number of files parallel
+                          uploads to perform when using the uploader.
+                        type: integer
+                    type: object
                   volumeSnapshotLocations:
                     description: VolumeSnapshotLocations is a list containing names
                       of VolumeSnapshotLocations associated with this backup.
@@ -435,8 +552,9 @@ spec:
                     type: array
                 type: object
               useOwnerReferencesInBackup:
-                description: UseOwnerReferencesBackup specifies whether to use OwnerReferences
-                  on backups created by this Schedule.
+                description: |-
+                  UseOwnerReferencesBackup specifies whether to use
+                  OwnerReferences on backups created by this Schedule.
                 nullable: true
                 type: boolean
             required:
@@ -447,11 +565,17 @@ spec:
             description: ScheduleStatus captures the current state of a Velero schedule
             properties:
               lastBackup:
-                description: LastBackup is the last time a Backup was run for this
+                description: |-
+                  LastBackup is the last time a Backup was run for this
                   Schedule schedule
                 format: date-time
                 nullable: true
                 type: string
+              lastSkipped:
+                description: LastSkipped is the last time a Schedule was skipped
+                format: date-time
+                nullable: true
+                type: string
               phase:
                 description: Phase is the current phase of the Schedule
                 enum:
@@ -460,8 +584,9 @@ spec:
                 - FailedValidation
                 type: string
               validationErrors:
-                description: ValidationErrors is a slice of all validation errors
-                  (if applicable)
+                description: |-
+                  ValidationErrors is a slice of all validation errors (if
+                  applicable)
                 items:
                   type: string
                 type: array
@@ -470,9 +595,3 @@ spec:
     served: true
     storage: true
     subresources: {}
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crd/v1/bases/velero.io_serverstatusrequests.yaml

@@ -1,11 +1,9 @@
-
 ---
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
   annotations:
-    controller-gen.kubebuilder.io/version: v0.7.0
-  creationTimestamp: null
+    controller-gen.kubebuilder.io/version: v0.14.0
   name: serverstatusrequests.velero.io
 spec:
   group: velero.io
@@ -21,18 +19,24 @@ spec:
   - name: v1
     schema:
       openAPIV3Schema:
-        description: ServerStatusRequest is a request to access current status information
-          about the Velero server.
+        description: |-
+          ServerStatusRequest is a request to access current status information about
+          the Velero server.
         properties:
           apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
             type: string
           kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
             type: string
           metadata:
             type: object
@@ -65,8 +69,9 @@ spec:
                 nullable: true
                 type: array
               processedTimestamp:
-                description: ProcessedTimestamp is when the ServerStatusRequest was
-                  processed by the ServerStatusRequestController.
+                description: |-
+                  ProcessedTimestamp is when the ServerStatusRequest was processed
+                  by the ServerStatusRequestController.
                 format: date-time
                 nullable: true
                 type: string
@@ -77,9 +82,3 @@ spec:
         type: object
     served: true
     storage: true
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crd/v1/bases/velero.io_volumesnapshotlocations.yaml

@@ -1,11 +1,9 @@
-
 ---
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
   annotations:
-    controller-gen.kubebuilder.io/version: v0.7.0
-  creationTimestamp: null
+    controller-gen.kubebuilder.io/version: v0.14.0
   name: volumesnapshotlocations.velero.io
 spec:
   group: velero.io
@@ -13,6 +11,8 @@ spec:
     kind: VolumeSnapshotLocation
     listKind: VolumeSnapshotLocationList
     plural: volumesnapshotlocations
+    shortNames:
+    - vsl
     singular: volumesnapshotlocation
   scope: Namespaced
   versions:
@@ -23,14 +23,19 @@ spec:
           snapshots.
         properties:
           apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
             type: string
           kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
             type: string
           metadata:
             type: object
@@ -43,6 +48,27 @@ spec:
                   type: string
                 description: Config is for provider-specific configuration fields.
                 type: object
+              credential:
+                description: Credential contains the credential information intended
+                  to be used with this location
+                properties:
+                  key:
+                    description: The key of the secret to select from.  Must be a
+                      valid secret key.
+                    type: string
+                  name:
+                    description: |-
+                      Name of the referent.
+                      More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+                      TODO: Add other useful fields. apiVersion, kind, uid?
+                    type: string
+                  optional:
+                    description: Specify whether the Secret or its key must be defined
+                    type: boolean
+                required:
+                - key
+                type: object
+                x-kubernetes-map-type: atomic
               provider:
                 description: Provider is the provider of the volume storage.
                 type: string
@@ -64,9 +90,3 @@ spec:
         type: object
     served: true
     storage: true
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crd/v2alpha1/bases/velero.io_datadownloads.yaml

@@ -0,0 +1,189 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: datadownloads.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: DataDownload
+    listKind: DataDownloadList
+    plural: datadownloads
+    singular: datadownload
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - description: DataDownload status such as New/InProgress
+      jsonPath: .status.phase
+      name: Status
+      type: string
+    - description: Time duration since this DataDownload was started
+      jsonPath: .status.startTimestamp
+      name: Started
+      type: date
+    - description: Completed bytes
+      format: int64
+      jsonPath: .status.progress.bytesDone
+      name: Bytes Done
+      type: integer
+    - description: Total bytes
+      format: int64
+      jsonPath: .status.progress.totalBytes
+      name: Total Bytes
+      type: integer
+    - description: Name of the Backup Storage Location where the backup data is stored
+      jsonPath: .spec.backupStorageLocation
+      name: Storage Location
+      type: string
+    - description: Time duration since this DataDownload was created
+      jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    - description: Name of the node where the DataDownload is processed
+      jsonPath: .status.node
+      name: Node
+      type: string
+    name: v2alpha1
+    schema:
+      openAPIV3Schema:
+        description: DataDownload acts as the protocol between data mover plugins
+          and data mover controller for the datamover restore operation
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: DataDownloadSpec is the specification for a DataDownload.
+            properties:
+              backupStorageLocation:
+                description: |-
+                  BackupStorageLocation is the name of the backup storage location
+                  where the backup repository is stored.
+                type: string
+              cancel:
+                description: |-
+                  Cancel indicates request to cancel the ongoing DataDownload. It can be set
+                  when the DataDownload is in InProgress phase
+                type: boolean
+              dataMoverConfig:
+                additionalProperties:
+                  type: string
+                description: DataMoverConfig is for data-mover-specific configuration
+                  fields.
+                type: object
+              datamover:
+                description: |-
+                  DataMover specifies the data mover to be used by the backup.
+                  If DataMover is "" or "velero", the built-in data mover will be used.
+                type: string
+              operationTimeout:
+                description: |-
+                  OperationTimeout specifies the time used to wait internal operations,
+                  before returning error as timeout.
+                type: string
+              snapshotID:
+                description: SnapshotID is the ID of the Velero backup snapshot to
+                  be restored from.
+                type: string
+              sourceNamespace:
+                description: |-
+                  SourceNamespace is the original namespace where the volume is backed up from.
+                  It may be different from SourcePVC's namespace if namespace is remapped during restore.
+                type: string
+              targetVolume:
+                description: TargetVolume is the information of the target PVC and
+                  PV.
+                properties:
+                  namespace:
+                    description: Namespace is the target namespace
+                    type: string
+                  pv:
+                    description: PV is the name of the target PV that is created by
+                      Velero restore
+                    type: string
+                  pvc:
+                    description: PVC is the name of the target PVC that is created
+                      by Velero restore
+                    type: string
+                required:
+                - namespace
+                - pv
+                - pvc
+                type: object
+            required:
+            - backupStorageLocation
+            - operationTimeout
+            - snapshotID
+            - sourceNamespace
+            - targetVolume
+            type: object
+          status:
+            description: DataDownloadStatus is the current status of a DataDownload.
+            properties:
+              completionTimestamp:
+                description: |-
+                  CompletionTimestamp records the time a restore was completed.
+                  Completion time is recorded even on failed restores.
+                  The server's time is used for CompletionTimestamps
+                format: date-time
+                nullable: true
+                type: string
+              message:
+                description: Message is a message about the DataDownload's status.
+                type: string
+              node:
+                description: Node is name of the node where the DataDownload is processed.
+                type: string
+              phase:
+                description: Phase is the current state of the DataDownload.
+                enum:
+                - New
+                - Accepted
+                - Prepared
+                - InProgress
+                - Canceling
+                - Canceled
+                - Completed
+                - Failed
+                type: string
+              progress:
+                description: |-
+                  Progress holds the total number of bytes of the snapshot and the current
+                  number of restored bytes. This can be used to display progress information
+                  about the restore operation.
+                properties:
+                  bytesDone:
+                    format: int64
+                    type: integer
+                  totalBytes:
+                    format: int64
+                    type: integer
+                type: object
+              startTimestamp:
+                description: |-
+                  StartTimestamp records the time a restore was started.
+                  The server's time is used for StartTimestamps
+                format: date-time
+                nullable: true
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}

config/crd/v2alpha1/bases/velero.io_datauploads.yaml

@@ -0,0 +1,214 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: datauploads.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: DataUpload
+    listKind: DataUploadList
+    plural: datauploads
+    singular: dataupload
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - description: DataUpload status such as New/InProgress
+      jsonPath: .status.phase
+      name: Status
+      type: string
+    - description: Time duration since this DataUpload was started
+      jsonPath: .status.startTimestamp
+      name: Started
+      type: date
+    - description: Completed bytes
+      format: int64
+      jsonPath: .status.progress.bytesDone
+      name: Bytes Done
+      type: integer
+    - description: Total bytes
+      format: int64
+      jsonPath: .status.progress.totalBytes
+      name: Total Bytes
+      type: integer
+    - description: Name of the Backup Storage Location where this backup should be
+        stored
+      jsonPath: .spec.backupStorageLocation
+      name: Storage Location
+      type: string
+    - description: Time duration since this DataUpload was created
+      jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    - description: Name of the node where the DataUpload is processed
+      jsonPath: .status.node
+      name: Node
+      type: string
+    name: v2alpha1
+    schema:
+      openAPIV3Schema:
+        description: DataUpload acts as the protocol between data mover plugins and
+          data mover controller for the datamover backup operation
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: DataUploadSpec is the specification for a DataUpload.
+            properties:
+              backupStorageLocation:
+                description: |-
+                  BackupStorageLocation is the name of the backup storage location
+                  where the backup repository is stored.
+                type: string
+              cancel:
+                description: |-
+                  Cancel indicates request to cancel the ongoing DataUpload. It can be set
+                  when the DataUpload is in InProgress phase
+                type: boolean
+              csiSnapshot:
+                description: If SnapshotType is CSI, CSISnapshot provides the information
+                  of the CSI snapshot.
+                nullable: true
+                properties:
+                  snapshotClass:
+                    description: SnapshotClass is the name of the snapshot class that
+                      the volume snapshot is created with
+                    type: string
+                  storageClass:
+                    description: StorageClass is the name of the storage class of
+                      the PVC that the volume snapshot is created from
+                    type: string
+                  volumeSnapshot:
+                    description: VolumeSnapshot is the name of the volume snapshot
+                      to be backed up
+                    type: string
+                required:
+                - storageClass
+                - volumeSnapshot
+                type: object
+              dataMoverConfig:
+                additionalProperties:
+                  type: string
+                description: DataMoverConfig is for data-mover-specific configuration
+                  fields.
+                nullable: true
+                type: object
+              datamover:
+                description: |-
+                  DataMover specifies the data mover to be used by the backup.
+                  If DataMover is "" or "velero", the built-in data mover will be used.
+                type: string
+              operationTimeout:
+                description: |-
+                  OperationTimeout specifies the time used to wait internal operations,
+                  before returning error as timeout.
+                type: string
+              snapshotType:
+                description: SnapshotType is the type of the snapshot to be backed
+                  up.
+                type: string
+              sourceNamespace:
+                description: |-
+                  SourceNamespace is the original namespace where the volume is backed up from.
+                  It is the same namespace for SourcePVC and CSI namespaced objects.
+                type: string
+              sourcePVC:
+                description: SourcePVC is the name of the PVC which the snapshot is
+                  taken for.
+                type: string
+            required:
+            - backupStorageLocation
+            - operationTimeout
+            - snapshotType
+            - sourceNamespace
+            - sourcePVC
+            type: object
+          status:
+            description: DataUploadStatus is the current status of a DataUpload.
+            properties:
+              completionTimestamp:
+                description: |-
+                  CompletionTimestamp records the time a backup was completed.
+                  Completion time is recorded even on failed backups.
+                  Completion time is recorded before uploading the backup object.
+                  The server's time is used for CompletionTimestamps
+                format: date-time
+                nullable: true
+                type: string
+              dataMoverResult:
+                additionalProperties:
+                  type: string
+                description: DataMoverResult stores data-mover-specific information
+                  as a result of the DataUpload.
+                nullable: true
+                type: object
+              message:
+                description: Message is a message about the DataUpload's status.
+                type: string
+              node:
+                description: Node is name of the node where the DataUpload is processed.
+                type: string
+              path:
+                description: Path is the full path of the snapshot volume being backed
+                  up.
+                type: string
+              phase:
+                description: Phase is the current state of the DataUpload.
+                enum:
+                - New
+                - Accepted
+                - Prepared
+                - InProgress
+                - Canceling
+                - Canceled
+                - Completed
+                - Failed
+                type: string
+              progress:
+                description: |-
+                  Progress holds the total number of bytes of the volume and the current
+                  number of backed up bytes. This can be used to display progress information
+                  about the backup operation.
+                properties:
+                  bytesDone:
+                    format: int64
+                    type: integer
+                  totalBytes:
+                    format: int64
+                    type: integer
+                type: object
+              snapshotID:
+                description: SnapshotID is the identifier for the snapshot in the
+                  backup repository.
+                type: string
+              startTimestamp:
+                description: |-
+                  StartTimestamp records the time a backup was started.
+                  Separate from CreationTimestamp, since that value changes
+                  on restores.
+                  The server's time is used for StartTimestamps
+                format: date-time
+                nullable: true
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}

config/crd/v2alpha1/crds/doc.go

@@ -0,0 +1,4 @@
+// Package crds embeds the controller-tools generated CRD manifests
+package crds
+
+//go:generate go run ../../../../hack/crd-gen/v1/main.go

config/rbac/role.yaml

@@ -1,9 +1,7 @@
-
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
-  creationTimestamp: null
   name: velero-perms
 rules:
 - apiGroups:
@@ -24,6 +22,26 @@ rules:
   - pods
   verbs:
   - get
+- apiGroups:
+  - velero.io
+  resources:
+  - backuprepositories
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - velero.io
+  resources:
+  - backuprepositories/status
+  verbs:
+  - get
+  - patch
+  - update
 - apiGroups:
   - velero.io
   resources:
@@ -31,6 +49,19 @@ rules:
   verbs:
   - create
   - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - velero.io
+  resources:
+  - backups/status
+  verbs:
+  - get
+  - patch
+  - update
 - apiGroups:
   - velero.io
   resources:
@@ -51,6 +82,46 @@ rules:
   - get
   - patch
   - update
+- apiGroups:
+  - velero.io
+  resources:
+  - datadownloads
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - velero.io
+  resources:
+  - datadownloads/status
+  verbs:
+  - get
+  - patch
+  - update
+- apiGroups:
+  - velero.io
+  resources:
+  - datauploads
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - velero.io
+  resources:
+  - datauploads/status
+  verbs:
+  - get
+  - patch
+  - update
 - apiGroups:
   - velero.io
   resources:
@@ -134,7 +205,7 @@ rules:
 - apiGroups:
   - velero.io
   resources:
-  - resticrepositories
+  - restores
   verbs:
   - create
   - delete
@@ -146,7 +217,7 @@ rules:
 - apiGroups:
   - velero.io
   resources:
-  - resticrepositories/status
+  - restores/status
   verbs:
   - get
   - patch
@@ -191,3 +262,15 @@ rules:
   - get
   - patch
   - update
+- apiGroups:
+  - velero.io
+  resources:
+  - volumesnapshotlocations
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch

design/CLI/PoC/base/CRDs.yaml

@@ -509,7 +509,7 @@ spec:
                       - CSIBackupVolumeSnapshotContents
                   type: string
                 name:
-                  description: Name is the name of the kubernetes resource with
+                  description: Name is the name of the Kubernetes resource with
                     which the file is associated.
                   type: string
               required:

design/CLI/PoC/base/deployment.yaml

@@ -57,7 +57,7 @@ spec:
         - emptyDir: {}
           name: scratch
 ---
-apiVersion: rbac.authorization.k8s.io/v1beta1
+apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata:
   labels:

design/CLI/PoC/overlays/plugins/node-agent.yaml

@@ -5,22 +5,22 @@ metadata:
   creationTimestamp: null
   labels:
     component: velero
-  name: restic
+  name: node-agent
   namespace: velero
 spec:
   selector:
     matchLabels:
-      name: restic
+      name: node-agent
   template:
     metadata:
       creationTimestamp: null
       labels:
         component: velero
-        name: restic
+        name: node-agent
     spec:
       containers:
         - args:
-            - restic
+            - node-agent
             - server
           command:
             - /velero
@@ -43,12 +43,15 @@ spec:
               value: /credentials/cloud
           image: velero/velero:latest
           imagePullPolicy: Always
-          name: restic
+          name: node-agent
           resources: {}
           volumeMounts:
             - mountPath: /host_pods
               mountPropagation: HostToContainer
               name: host-pods
+            - mountPath: /var/lib/kubelet/plugins
+              mountPropagation: HostToContainer
+              name: host-plugins
             - mountPath: /scratch
               name: scratch
             - mountPath: /credentials
@@ -60,6 +63,9 @@ spec:
         - hostPath:
             path: /var/lib/kubelet/pods
           name: host-pods
+        - hostPath:
+            path: /var/lib/kubelet/plugins
+          name: host-plugins
         - emptyDir: {}
           name: scratch
         - name: cloud-credentials

design/Implemented/Extend-VolumePolicies-to-support-more-actions.md

@@ -0,0 +1,344 @@
+# Extend VolumePolicies to support more actions
+
+## Abstract
+
+Currently, the [VolumePolicies feature](https://github.com/vmware-tanzu/velero/blob/main/design/Implemented/handle-backup-of-volumes-by-resources-filters.md) which can be used to filter/handle volumes during backup only supports the skip action on matching conditions. Users need more actions to be supported.
+
+## Background
+
+The `VolumePolicies` feature was introduced in Velero 1.11 as a flexible way to handle volumes. The main agenda of
+introducing the VolumePolicies feature was to improve the overall user experience when performing backup operations
+for volume resources, the feature enables users to group volumes according the `conditions` (criteria) specified and
+also lets you specify the `action` that velero needs to take for these grouped volumes during the backup operation.
+The limitation being that currently `VolumePolicies` only supports `skip` as an action, We want to extend the `action`
+functionality to support more usable options like `fs-backup` (File system backup) and `snapshot` (VolumeSnapshots).
+
+## Goals
+- Extending the VolumePolicies to support more actions like `fs-backup` (File system backup) and `snapshot` (VolumeSnapshots).
+- Improve user experience when backing up Volumes via Velero
+
+## Non-Goals
+- No changes to existing approaches to opt-in/opt-out annotations for volumes
+- No changes to existing `VolumePolicies` functionalities
+- No additions or implementations to support more granular actions like `snapshot-csi` and `snapshot-datamover`. These actions can be implemented as a future enhancement
+
+
+## Use-cases/Scenarios
+
+**Use-case 1:**
+- A user wants to use `snapshot` (volumesnapshots) backup option for all the csi supported volumes and `fs-backup` for the rest of the volumes.
+- Currently, velero supports this use-case but the user experience is not that great.
+- The user will have to individually annotate the volume mounting pod with the annotation "backup.velero.io/backup-volumes" for `fs-backup`
+- This becomes cumbersome at scale.
+- Using `VolumePolicies`, the user can just specify 2 simple `VolumePolicies` like for csi supported volumes as `snapshot` action and rest can be backed up`fs-backup` action:
+```yaml
+version: v1
+volumePolicies:
+  - conditions:
+      storageClass:
+        - gp2
+    action:
+      type: snapshot
+  - conditions: {}
+    action:
+      type: fs-backup
+```
+
+**Use-case 2:**
+- A user wants to use `fs-backup` for nfs volumes pertaining to a particular server
+- In such a scenario the user can just specify a `VolumePolicy` like:
+```yaml
+version: v1
+volumePolicies:
+- conditions:
+    nfs:
+      server: 192.168.200.90
+  action:
+    type: fs-backup
+```
+## High-Level Design
+- When the VolumePolicy action is set as `fs-backup` the backup workflow modifications would be:
+  - We call [backupItem() -> backupItemInternal()](https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_backupper.go#L95) on all the items that are to be backed up
+  - Here when we encounter [Pod as an item ](https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_backupper.go#L195)
+  - We will have to modify the backup workflow to account for the `fs-backup` VolumePolicy action
+
+
+- When the VolumePolicy action is set as `snapshot` the backup workflow modifications would be:
+  - Once again, We call [backupItem() -> backupItemInternal()](https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_backupper.go#L95) on all the items that are to be backed up
+  - Here when we encounter [Persistent Volume as an item](https://github.com/vmware-tanzu/velero/blob/d4128542590470b204a642ee43311921c11db880/pkg/backup/item_backupper.go#L253)
+  - And we call the [takePVSnapshot func](https://github.com/vmware-tanzu/velero/blob/d4128542590470b204a642ee43311921c11db880/pkg/backup/item_backupper.go#L508)
+  - We need to modify the takePVSnapshot function to account for the `snapshot` VolumePolicy action.
+  - In case of csi snapshots for PVC objects, these snapshot actions are taken by the velero-plugin-for-csi, we need to modify the [executeActions()](https://github.com/vmware-tanzu/velero/blob/512fe0dabdcb3bbf1ca68a9089056ae549663bcf/pkg/backup/item_backupper.go#L232) function to account for the `snapshot` VolumePolicy action.
+
+**Note:** `Snapshot` action can either be a native snapshot or a csi snapshot, as is the case with the current flow where velero itself makes the decision based on the backup CR.
+
+## Detailed Design
+- Update VolumePolicy action type validation to account for `fs-backup` and `snapshot` as valid VolumePolicy actions.
+- Modifications needed for `fs-backup` action:
+  - Now based on the specification of volume policy on backup request we will decide whether to go via legacy pod annotations approach or the newer volume policy based fs-backup action approach.
+  - If there is a presence of volume policy(fs-backup/snapshot) on the backup request that matches as an action for a volume we use the newer volume policy approach to get the list of the volumes for `fs-backup` action
+  - Else continue with the annotation based legacy approach workflow.
+
+- Modifications needed for `snapshot` action:
+  - In the [takePVSnapshot function](https://github.com/vmware-tanzu/velero/blob/d4128542590470b204a642ee43311921c11db880/pkg/backup/item_backupper.go#L508) we will check the PV fits the volume policy criteria and see if the associated action is `snapshot`
+  - If it is not snapshot then we skip the further workflow and avoid taking the snapshot of the PV
+  - Similarly, For csi snapshot of PVC object, we need to do similar changes in [executeAction() function](https://github.com/vmware-tanzu/velero/blob/512fe0dabdcb3bbf1ca68a9089056ae549663bcf/pkg/backup/item_backupper.go#L348). we will check the PVC fits the volume policy criteria and see if the associated action is `snapshot` via csi plugin
+  - If it is not snapshot then we skip the csi BIA execute action and avoid taking the snapshot of the PVC by not invoking the csi plugin action for the PVC
+
+**Note:**
+- When we are using the `VolumePolicy` approach for backing up the volumes then the volume policy criteria and action need to be specific and explicit, there is no default behavior, if a volume matches `fs-backup` action then `fs-backup` method will be used for that volume and similarly if the volume matches the criteria for `snapshot` action then the snapshot workflow will be used for the volume backup.
+- Another thing to note is the workflow proposed in this design uses the legacy `opt-in/opt-out` approach as a fallback option. For instance, the user specifies a VolumePolicy but for a particular volume included in the backup there are no actions(fs-backup/snapshot) matching in the volume policy for that volume, in such a scenario the legacy approach will be used for backing up the particular volume.
+- The relation between the `VolumePolicy` and the backup's legacy parameter `SnapshotVolumes`: 
+  - The `VolumePolicy`'s `snapshot` action matching for volume has higher priority. When there is a `snapshot` action matching for the selected volume, it will be backed by the snapshot way, no matter of the `backup.Spec.SnapshotVolumes` setting.
+  - If there is no `snapshot` action matching the selected volume in the `VolumePolicy`, then the volume will be backed up by `snapshot` way, if the `backup.Spec.SnapshotVolumes` is not set to false.
+- The relation between the `VolumePolicy` and the backup's legacy filesystem `opt-in/opt-out` approach:
+  - The `VolumePolicy`'s `fs-backup` action matching for volume has higher priority. When there is a `fs-backup` action matching for the selected volume, it will be backed by the fs-backup way, no matter of the `backup.Spec.DefaultVolumesToFsBackup` setting and the pod's `opt-in/opt-out` annotation setting.
+  - If there is no `fs-backup` action matching the selected volume in the `VolumePolicy`, then the volume will be backed up by the legacy `opt-in/opt-out` way.
+
+## Implementation
+
+- The implementation should be included in velero 1.14
+
+- We will introduce a `VolumeHelper` interface. It will consist of two methods:
+```go
+type VolumeHelper interface {
+	ShouldPerformSnapshot(obj runtime.Unstructured, groupResource schema.GroupResource) (bool, error)
+	ShouldPerformFSBackup(volume corev1api.Volume, pod corev1api.Pod) (bool, error)
+}
+```
+-  The `VolumeHelperImpl` struct will implement the `VolumeHelper` interface and will consist of the functions that we will use through the backup workflow to accommodate volume policies for PVs and PVCs.
+```go
+type volumeHelperImpl struct {
+	volumePolicy             *resourcepolicies.Policies
+	snapshotVolumes          *bool
+	logger                   logrus.FieldLogger
+	client                   crclient.Client
+	defaultVolumesToFSBackup bool
+	backupExcludePVC         bool
+}
+
+```
+
+- We will create an instance of the structure `volumeHelperImpl` in `item_backupper.go`
+```go
+	itemBackupper := &itemBackupper{
+		...
+		volumeHelperImpl: volumehelper.NewVolumeHelperImpl(
+			resourcePolicy,
+			backupRequest.Spec.SnapshotVolumes,
+			log,
+			kb.kbClient,
+			boolptr.IsSetToTrue(backupRequest.Spec.DefaultVolumesToFsBackup),
+			!backupRequest.ResourceIncludesExcludes.ShouldInclude(kuberesource.PersistentVolumeClaims.String()),
+		),
+	}
+```
+
+
+#### FS-Backup
+- Regarding `fs-backup` action to decide whether to use legacy annotation based approach or volume policy based approach:
+  - We will use the `vh.ShouldPerformFSBackup()` function from the `volumehelper` package
+  - Functions involved in processing `fs-backup` volume policy action will somewhat look like:
+
+```go
+func (v volumeHelperImpl) ShouldPerformFSBackup(volume corev1api.Volume, pod corev1api.Pod) (bool, error) {
+	if !v.shouldIncludeVolumeInBackup(volume) {
+		v.logger.Debugf("skip fs-backup action for pod %s's volume %s, due to not pass volume check.", pod.Namespace+"/"+pod.Name, volume.Name)
+		return false, nil
+	}
+
+	if v.volumePolicy != nil {
+		pvc, err := kubeutil.GetPVCForPodVolume(&volume, &pod, v.client)
+		if err != nil {
+			v.logger.WithError(err).Errorf("fail to get PVC for pod %s", pod.Namespace+"/"+pod.Name)
+			return false, err
+		}
+		pv, err := kubeutil.GetPVForPVC(pvc, v.client)
+		if err != nil {
+			v.logger.WithError(err).Errorf("fail to get PV for PVC %s", pvc.Namespace+"/"+pvc.Name)
+			return false, err
+		}
+
+		action, err := v.volumePolicy.GetMatchAction(pv)
+		if err != nil {
+			v.logger.WithError(err).Errorf("fail to get VolumePolicy match action for PV %s", pv.Name)
+			return false, err
+		}
+
+		if action != nil {
+			if action.Type == resourcepolicies.FSBackup {
+				v.logger.Infof("Perform fs-backup action for volume %s of pod %s due to volume policy match",
+					volume.Name, pod.Namespace+"/"+pod.Name)
+				return true, nil
+			} else {
+				v.logger.Infof("Skip fs-backup action for volume %s for pod %s because the action type is %s",
+					volume.Name, pod.Namespace+"/"+pod.Name, action.Type)
+				return false, nil
+			}
+		}
+	}
+
+	if v.shouldPerformFSBackupLegacy(volume, pod) {
+		v.logger.Infof("Perform fs-backup action for volume %s of pod %s due to opt-in/out way",
+			volume.Name, pod.Namespace+"/"+pod.Name)
+		return true, nil
+	} else {
+		v.logger.Infof("Skip fs-backup action for volume %s of pod %s due to opt-in/out way",
+			volume.Name, pod.Namespace+"/"+pod.Name)
+		return false, nil
+	}
+}
+```
+
+- The main function from the above will be called when we encounter Pods during the backup workflow:
+```go
+			for _, volume := range pod.Spec.Volumes {
+				shouldDoFSBackup, err := ib.volumeHelperImpl.ShouldPerformFSBackup(volume, *pod)
+				if err != nil {
+					backupErrs = append(backupErrs, errors.WithStack(err))
+				}
+				...
+			}
+```
+
+#### Snapshot (PV)
+
+- Making sure that `snapshot` action is skipped for PVs that do not fit the volume policy criteria, for this we will use the `vh.ShouldPerformSnapshot` from the `VolumeHelperImpl(vh)` receiver.
+```go
+func (v *volumeHelperImpl) ShouldPerformSnapshot(obj runtime.Unstructured, groupResource schema.GroupResource) (bool, error) {
+	// check if volume policy exists and also check if the object(pv/pvc) fits a volume policy criteria and see if the associated action is snapshot
+	// if it is not snapshot then skip the code path for snapshotting the PV/PVC
+	pvc := new(corev1api.PersistentVolumeClaim)
+	pv := new(corev1api.PersistentVolume)
+	var err error
+
+	if groupResource == kuberesource.PersistentVolumeClaims {
+		if err = runtime.DefaultUnstructuredConverter.FromUnstructured(obj.UnstructuredContent(), &pvc); err != nil {
+			return false, err
+		}
+
+		pv, err = kubeutil.GetPVForPVC(pvc, v.client)
+		if err != nil {
+			return false, err
+		}
+	}
+
+	if groupResource == kuberesource.PersistentVolumes {
+		if err = runtime.DefaultUnstructuredConverter.FromUnstructured(obj.UnstructuredContent(), &pv); err != nil {
+			return false, err
+		}
+	}
+
+	if v.volumePolicy != nil {
+		action, err := v.volumePolicy.GetMatchAction(pv)
+		if err != nil {
+			return false, err
+		}
+
+		// If there is a match action, and the action type is snapshot, return true,
+		// or the action type is not snapshot, then return false.
+		// If there is no match action, go on to the next check.
+		if action != nil {
+			if action.Type == resourcepolicies.Snapshot {
+				v.logger.Infof(fmt.Sprintf("performing snapshot action for pv %s", pv.Name))
+				return true, nil
+			} else {
+				v.logger.Infof("Skip snapshot action for pv %s as the action type is %s", pv.Name, action.Type)
+				return false, nil
+			}
+		}
+	}
+
+		// If this PV is claimed, see if we've already taken a (pod volume backup)
+	// snapshot of the contents of this PV. If so, don't take a snapshot.
+	if pv.Spec.ClaimRef != nil {
+		pods, err := podvolumeutil.GetPodsUsingPVC(
+			pv.Spec.ClaimRef.Namespace,
+			pv.Spec.ClaimRef.Name,
+			v.client,
+		)
+		if err != nil {
+			v.logger.WithError(err).Errorf("fail to get pod for PV %s", pv.Name)
+			return false, err
+		}
+
+		for _, pod := range pods {
+			for _, vol := range pod.Spec.Volumes {
+				if vol.PersistentVolumeClaim != nil &&
+					vol.PersistentVolumeClaim.ClaimName == pv.Spec.ClaimRef.Name &&
+					v.shouldPerformFSBackupLegacy(vol, pod) {
+					v.logger.Infof("Skipping snapshot of pv %s because it is backed up with PodVolumeBackup.", pv.Name)
+					return false, nil
+				}
+			}
+		}
+	}
+
+	if !boolptr.IsSetToFalse(v.snapshotVolumes) {
+		// If the backup.Spec.SnapshotVolumes is not set, or set to true, then should take the snapshot.
+		v.logger.Infof("performing snapshot action for pv %s as the snapshotVolumes is not set to false")
+		return true, nil
+	}
+
+	v.logger.Infof(fmt.Sprintf("skipping snapshot action for pv %s possibly due to no volume policy setting or snapshotVolumes is false", pv.Name))
+	return false, nil
+}
+```
+
+- The function `ShouldPerformSnapshot` will be used as follows in `takePVSnapshot` function of the backup workflow:
+```go
+	snapshotVolume, err := ib.volumeHelperImpl.ShouldPerformSnapshot(obj, kuberesource.PersistentVolumes)
+	if err != nil {
+		return err
+	}
+
+	if !snapshotVolume {
+		log.Info(fmt.Sprintf("skipping volume snapshot for PV %s as it does not fit the volume policy criteria specified by the user for snapshot action", pv.Name))
+		ib.trackSkippedPV(obj, kuberesource.PersistentVolumes, volumeSnapshotApproach, "does not satisfy the criteria for volume policy based snapshot action", log)
+		return nil
+	}
+```
+
+#### Snapshot (PVC)
+
+- Making sure that `snapshot` action is skipped for PVCs that do not fit the volume policy criteria, for this we will again use the `vh.ShouldPerformSnapshot` from the `VolumeHelperImpl(vh)` receiver.
+- We will pass the `VolumeHelperImpl(vh)` instance in `executeActions` method so that it is available to use.
+```go
+
+```
+- The above function will be used as follows in the `executeActions` function of backup workflow.
+- Considering the vSphere plugin doesn't support the VolumePolicy yet, don't use the VolumePolicy for vSphere plugin by now.
+```go
+		if groupResource == kuberesource.PersistentVolumeClaims {
+			if actionName == csiBIAPluginName {
+				snapshotVolume, err := ib.volumeHelperImpl.ShouldPerformSnapshot(obj, kuberesource.PersistentVolumeClaims)
+				if err != nil {
+					return nil, itemFiles, errors.WithStack(err)
+				}
+
+				if !snapshotVolume {
+					log.Info(fmt.Sprintf("skipping csi volume snapshot for PVC %s as it does not fit the volume policy criteria specified by the user for snapshot action", namespace+"/"+name))
+					ib.trackSkippedPV(obj, kuberesource.PersistentVolumeClaims, volumeSnapshotApproach, "does not satisfy the criteria for volume policy based snapshot action", log)
+					continue
+				}
+			}
+		}
+```
+
+
+## Future Implementation
+It makes sense to add more specific actions in the future, once we deprecate the legacy opt-in/opt-out approach to keep things simple. Another point of note is, csi related action can be
+easier to implement once we decide to merge the csi plugin into main velero code flow.
+In the future, we envision the following actions that can be implemented:
+- `snapshot-native`: only use volume snapshotter (native cloud provider snapshots), do nothing if not present/not compatible
+- `snapshot-csi`: only use csi-plugin, don't use volume snapshotter(native cloud provider snapshots), don't use datamover even if snapshotMoveData is true
+- `snapshot-datamover`: only use csi with datamover, don't use volume snapshotter (native cloud provider snapshots), use datamover even if snapshotMoveData is false
+
+**Note:** The above actions are just suggestions for future scope, we may not use/implement them as is. We could definitely merge these suggested actions as `Snapshot` actions and use volume policy parameters and criteria to segregate them instead of making the user explicitly supply the action names to such granular levels.
+
+## Related to Design
+
+[Handle backup of volumes by resources filters](https://github.com/vmware-tanzu/velero/blob/main/design/Implemented/handle-backup-of-volumes-by-resources-filters.md)
+
+## Alternatives Considered
+Same as the earlier design as this is an extension of the original VolumePolicies design

design/Implemented/backup-pvc-config.md

@@ -0,0 +1,94 @@
+# Backup PVC Configuration Design
+
+## Glossary & Abbreviation
+
+**Velero Generic Data Path (VGDP)**: VGDP is the collective modules that is introduced in [Unified Repository design][1]. Velero uses these modules to finish data transfer for various purposes (i.e., PodVolume backup/restore, Volume Snapshot Data Movement). VGDP modules include uploaders and the backup repository.  
+
+**Exposer**: Exposer is a module that is introduced in [Volume Snapshot Data Movement Design][2]. Velero uses this module to expose the volume snapshots to Velero node-agent pods or node-agent associated pods so as to complete the data movement from the snapshots.  
+
+**backupPVC**: The intermediate PVC created by the exposer for VGDP to access data from, see [Volume Snapshot Data Movement Design][2] for more details.  
+
+**backupPod**: The pod consumes the backupPVC so that VGDP could access data from the backupPVC, see [Volume Snapshot Data Movement Design][2] for more details.  
+
+**sourcePVC**: The PVC to be backed up, see [Volume Snapshot Data Movement Design][2] for more details. 
+
+## Background
+
+As elaberated in [Volume Snapshot Data Movement Design][2], a backupPVC may be created by the Exposer and the VGDP reads data from the backupPVC.  
+In some scenarios, users may need to configure some advanced settings of the backupPVC so that the data movement could work in best performance in their environments. Specifically:  
+- For some storage providers, when creating a read-only volume from a snapshot, it is very fast; whereas, if a writable volume is created from the snapshot, they need to clone the entire disk data, which is time consuming. If the backupPVC's `accessModes` is set as `ReadOnlyMany`, the volume driver is able to tell the storage to create a read-only volume, which may dramatically shorten the snapshot expose time. On the other hand,  `ReadOnlyMany` is not supported by all volumes. Therefore, users should be allowed to configure the `accessModes` for the backupPVC.  
+- Some storage providers create one or more replicas when creating a volume, the number of replicas is defined in the storage class. However, it doesn't make any sense to keep replicas when an intermediate volume used by the backup. Therefore, users should be allowed to configure another storage class specifically used by the backupPVC.  
+
+## Goals
+
+- Create a mechanism for users to specify various configurations for backupPVC    
+
+## Non-Goals
+
+## Solution
+
+We will use the ConfigMap specified by `velero node-agent` CLI's parameter `--node-agent-configmap` to host the backupPVC configurations.
+This configMap is not created by Velero, users should create it manually on demand. The configMap should be in the same namespace where Velero is installed. If multiple Velero instances are installed in different namespaces, there should be one configMap in each namespace which applies to node-agent in that namespace only.  
+Node-agent server checks these configurations at startup time and use it to initiate the related Exposer modules. Therefore, users could edit this configMap any time, but in order to make the changes effective, node-agent server needs to be restarted.  
+Inside the ConfigMap we will add one new kind of configuration as the data in the configMap, the name is ```backupPVC```.  
+Users may want to set different backupPVC configurations for different volumes, therefore, we define the configurations as a map and allow users to specific configurations by storage class. Specifically, the key of the map element is the storage class name used by the sourcePVC and the value is the set of configurations for the backupPVC created for the sourcePVC.   
+
+The data structure is as below:
+```go
+type Configs struct {
+	// LoadConcurrency is the config for data path load concurrency per node.
+	LoadConcurrency *LoadConcurrency `json:"loadConcurrency,omitempty"`
+
+	// LoadAffinity is the config for data path load affinity.
+	LoadAffinity []*LoadAffinity `json:"loadAffinity,omitempty"`
+
+	// BackupPVC is the config for backupPVC of snapshot data movement.
+	BackupPVC map[string]BackupPVC `json:"backupPVC,omitempty"`
+}
+
+type BackupPVC struct {
+	// StorageClass is the name of storage class to be used by the backupPVC.
+	StorageClass string `json:"storageClass,omitempty"`
+
+	// ReadOnly sets the backupPVC's access mode as read only.
+	ReadOnly bool `json:"readOnly,omitempty"`
+}
+```  
+
+### Sample
+A sample of the ConfigMap is as below:
+```json
+{
+    "backupPVC": {
+        "storage-class-1": {
+            "storageClass": "snapshot-storage-class",
+            "readOnly": true
+        },
+        "storage-class-2": {
+            "storageClass": "snapshot-storage-class"
+        },
+        "storage-class-3": {
+            "readOnly": true
+        }        
+    }
+}
+```
+
+To create the configMap, users need to save something like the above sample to a json file and then run below command:
+```
+kubectl create cm <ConfigMap name> -n velero --from-file=<json file name>
+``` 
+
+### Implementation
+The `backupPVC` is passed to the exposer and the exposer sets the related specification and create the backupPVC.  
+If `backupPVC.storageClass` doesn't exist or set as empty, the sourcePVC's storage class will be used.  
+If `backupPVC.readOnly` is set to true, `ReadOnlyMany` will be the only value set to the backupPVC's `accessModes`, otherwise, `ReadWriteOnce` is used.  
+
+Once `backupPVC.storageClass` is set, users must make sure that the specified storage class exists in the cluster and can be used the the backupPVC, otherwise, the corresponding DataUpload CR will stay in `Accepted` phase until the prepare timeout (by default 30min).   
+Once `backupPVC.readOnly` is set to true, users must make sure that the storage supports to create a `ReadOnlyMany` PVC from a snapshot, otherwise, the corresponding DataUpload CR will stay in `Accepted` phase until the prepare timeout (by default 30min).  
+
+Once above problems happen, the DataUpload CR is cancelled after prepare timeout and the backupPVC and backupPod will be deleted, so there is no way to tell the cause is one of the above problems or others.  
+To help the troubleshooting, we can add some diagnostic mechanism to discover the status of the backupPod before deleting it as a result of the prepare timeout.  
+
+[1]: unified-repo-and-kopia-integration/unified-repo-and-kopia-integration.md
+[2]: volume-snapshot-data-movement/volume-snapshot-data-movement.md

design/Implemented/backup-repo-config.md

@@ -0,0 +1,123 @@
+# Backup Repository Configuration Design
+
+## Glossary & Abbreviation
+
+**Backup Storage**: The storage to store the backup data. Check [Unified Repository design][1] for details.  
+**Backup Repository**: Backup repository is layered between BR data movers and Backup Storage to provide BR related features that is introduced in [Unified Repository design][1].    
+
+## Background
+
+According to the [Unified Repository design][1] Velero uses selectable backup repositories for various backup/restore methods, i.e., fs-backup, volume snapshot data movement, etc. To achieve the best performance, backup repositories may need to be configured according to the running environments.  
+For example, if there are sufficient CPU and memory resources in the environment, users may enable compression feature provided by the backup repository, so as to achieve the best backup throughput.  
+As another example, if the local disk space is not sufficient, users may want to constraint the backup repository's cache size, so as to prevent the repository from running out of the disk space.  
+Therefore, it is worthy to allow users to configure some essential parameters of the backup repsoitories, and the configuration may vary from backup repositories.  
+
+## Goals
+
+- Create a mechanism for users to specify configurations for backup repositories  
+
+## Non-Goals
+
+## Solution
+
+### BackupRepository CRD
+
+After a backup repository is initialized, a BackupRepository CR is created to represent the instance of the backup repository. The BackupRepository's spec is a core parameter used by Unified Repo modules when interactive with the backup repsoitory. Therefore, we can add the configurations into the BackupRepository CR called ```repositoryConfig```.  
+The configurations may be different varying from backup repositories, therefore, we will not define each of the configurations explicitly. Instead, we add a map in the BackupRepository's spec to take any configuration to be set to the backup repository.  
+
+During various operations to the backup repository, the Unified Repo modules will retrieve from the map for the specific configuration that is required at that time. So even though it is specified, a configuration may not be visited/hornored if the operations don't require it for the specific backup repository, this won't bring any issue. When and how a configuration is hornored is decided by the configuration itself and should be clarified in the configuration's specification.  
+
+Below is the new BackupRepository's spec after adding the configuration map:  
+```yaml
+          spec:
+            description: BackupRepositorySpec is the specification for a BackupRepository.
+            properties:
+              backupStorageLocation:
+                description: |-
+                  BackupStorageLocation is the name of the BackupStorageLocation
+                  that should contain this repository.
+                type: string
+              maintenanceFrequency:
+                description: MaintenanceFrequency is how often maintenance should
+                  be run.
+                type: string
+              repositoryConfig:
+                additionalProperties:
+                  type: string
+                description: RepositoryConfig contains configurations for the specific
+                  repository.
+                type: object
+              repositoryType:
+                description: RepositoryType indicates the type of the backend repository
+                enum:
+                - kopia
+                - restic
+                - ""
+                type: string
+              resticIdentifier:
+                description: |-
+                  ResticIdentifier is the full restic-compatible string for identifying
+                  this repository.
+                type: string
+              volumeNamespace:
+                description: |-
+                  VolumeNamespace is the namespace this backup repository contains
+                  pod volume backups for.
+                type: string
+            required:
+            - backupStorageLocation
+            - maintenanceFrequency
+            - resticIdentifier
+            - volumeNamespace
+            type: object
+```            
+
+### BackupRepository configMap
+
+The BackupRepository CR is not created explicitly by a Velero CLI, but created as part of the backup/restore/maintenance operation if the CR doesn't exist. As a result, users don't have any way to specify the configurations before the BackupRepository CR is created.  
+Therefore, a BackupRepository configMap is introduced as a template of the configurations to be applied to the backup repository CR.  
+When the backup repository CR is created by the BackupRepository controller, the configurations in the configMap are copied to the ```repositoryConfig``` field.   
+For an existing BackupRepository CR, the configMap is never visited, if users want to modify the configuration value, they should directly edit the BackupRepository CR.  
+
+The BackupRepository configMap is created by users in velero installation namespace. The configMap name must be specified in the velero server parameter ```--backup-repository-configmap```, otherwise, it won't effect.  
+If the configMap name is specified but the configMap doesn't exist by the time of a backup repository is created, the configMap name is ignored.  
+For any reason, if the configMap doesn't effect, nothing is specified to the backup repository CR, so the Unified Repo modules use the hard-coded values to configure the backup repository.  
+
+The BackupRepository configMap supports backup repository type specific configurations, even though users can only specify one configMap.  
+So in the configMap struct, multiple entries are supported, indexed by the backup repository type. During the backup repository creation, the configMap is searched by the repository type.  
+
+### Configurations
+
+With the above mechanisms, any kind of configuration could be added. Here list the configurations defined at present:  
+```cacheLimitMB```: specifies the size limit(in MB) for the local data cache. The more data is cached locally, the less data may be downloaded from the backup storage, so the better performance may be achieved. Practically, users can specify any size that is smaller than the free space so that the disk space won't run out. This parameter is for each repository connection, that is, users could change it before connecting to the repository. If a backup repository doesn't use local cache, this parameter will be ignored. For Kopia repository, this parameter is supported.  
+```enableCompression```: specifies to enable/disable compression for a backup repsotiory. Most of the backup repositories support the data compression feature, if it is not supported by a backup repository, this parameter is ignored. Most of the backup repositories support to dynamically enable/disable compression, so this parameter is defined to be used whenever creating a write connection to the backup repository, if the dynamically changing is not supported, this parameter will be hornored only when initializing the backup repository. For Kopia repository, this parameter is supported and can be dynamically modified.  
+
+### Sample
+Below is an example of the BackupRepository configMap with the configurations:     
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: <config-name>
+  namespace: velero
+data:
+  <repository-type-1>: |
+    {
+      "cacheLimitMB": 2048,
+      "enableCompression": true    
+    }
+  <repository-type-2>: |
+    {
+      "cacheLimitMB": 1,
+      "enableCompression": false    
+    }        
+```
+
+To create the configMap, users need to save something like the above sample to a file and then run below commands:  
+```
+kubectl apply -f <yaml file name>
+```  
+
+
+
+[1]: unified-repo-and-kopia-integration/unified-repo-and-kopia-integration.md

design/Implemented/backup-resources-order.md

@@ -6,7 +6,7 @@ During backup process, user may need to back up resources of specific type in so
 (Ex: primary-secondary database pods in a cluster).
 
 ## Goals
-- Enable user to specify an order of back up resources belong to specific resource type
+- Enable user to specify an order of backup resources belong to specific resource type
 
 ## Alternatives Considered
 - Use a plugin to backup an resources and all the sub resources.  For example use a plugin for StatefulSet and backup pods belong to the StatefulSet in specific order.  This plugin solution is not generic and requires plugin for each resource type.

design/Implemented/biav2-design.md

@@ -0,0 +1,103 @@
+# Design for BackupItemAction v2 API
+
+## Abstract
+This design includes the changes to the BackupItemAction (BIA) api design as required by the [Item Action Progress Monitoring](general-progress-monitoring.md) feature.
+The BIA v2 interface will have two new methods, and the Execute() return signature will be modified.
+If there are any additional BIA API changes that are needed in the same Velero release cycle as this change, those can be added here as well.
+
+## Background
+This API change is needed to facilitate long-running plugin actions that may not be complete when the Execute() method returns.
+It is an optional feature, so plugins which don't need this feature can simply return an empty operation ID and the new methods can be no-ops.
+This will allow long-running plugin actions to continue in the background while Velero moves on to the next plugin, the next item, etc.
+
+## Goals
+- Allow for BIA Execute() to optionally initiate a long-running operation and report on operation status.
+
+## Non Goals
+- Allowing velero control over when the long-running operation begins.
+
+
+## High-Level Design
+As per the [Plugin Versioning](plugin-versioning.md) design, a new BIAv2 plugin `.proto` file will be created to define the GRPC interface.
+v2 go files will also be created in `plugin/clientmgmt/backupitemaction` and `plugin/framework/backupitemaction`, and a new PluginKind will be created.
+The velero Backup process will be modified to reference v2 plugins instead of v1 plugins.
+An adapter will be created so that any existing BIA v1 plugin can be executed as a v2 plugin when executing a backup.
+
+## Detailed Design
+
+### proto changes (compiled into golang by protoc)
+
+The v2 BackupItemAction.proto will be like the current v1 version with the following changes:
+ExecuteResponse gets a new field:
+```
+message ExecuteResponse {
+    bytes item = 1;
+    repeated generated.ResourceIdentifier additionalItems = 2;
+    string operationID = 3;
+    repeated generated.ResourceIdentifier itemsToUpdate = 4;
+}
+```
+The BackupItemAction service gets two new rpc methods:
+```
+service BackupItemAction {
+    rpc AppliesTo(BackupItemActionAppliesToRequest) returns (BackupItemActionAppliesToResponse);
+    rpc Execute(ExecuteRequest) returns (ExecuteResponse);
+    rpc Progress(BackupItemActionProgressRequest) returns (BackupItemActionProgressResponse);
+    rpc Cancel(BackupItemActionCancelRequest) returns (google.protobuf.Empty);
+}
+```
+To support these new rpc methods, we define new request/response message types:
+```
+message BackupItemActionProgressRequest {
+    string plugin = 1;
+    string operationID = 2;
+    bytes backup = 3;
+}
+
+message BackupItemActionProgressResponse {
+    generated.OperationProgress progress = 1;
+}
+
+message BackupItemActionCancelRequest {
+    string plugin = 1;
+    string operationID = 2;
+    bytes backup = 3;
+}
+
+```
+One new shared message type will be added, as this will also be needed for v2 RestoreItemAction and VolmeSnapshotter:
+```
+message OperationProgress {
+    bool completed = 1;
+    string err = 2;
+    int64 nCompleted = 3;
+    int64 nTotal = 4;
+    string operationUnits = 5;
+    string description = 6;
+    google.protobuf.Timestamp started = 7;
+    google.protobuf.Timestamp updated = 8;
+}
+```
+
+In addition to the two new rpc methods added to the BackupItemAction interface, there is also a new `Name()` method. This one is only actually used internally by Velero to get the name that the plugin was registered with, but it still must be defined in a plugin which implements BackupItemActionV2 in order to implement the interface. It doesn't really matter what it returns, though, as this particular method is not delegated to the plugin via RPC calls. The new (and modified) interface methods for `BackupItemAction` are as follows:
+```
+type BackupItemAction interface {
+...
+	Name() string
+...
+	Execute(item runtime.Unstructured, backup *api.Backup) (runtime.Unstructured, []velero.ResourceIdentifier, string, []velero.ResourceIdentifier, error)
+	Progress(operationID string, backup *api.Backup) (velero.OperationProgress, error)
+	Cancel(operationID string, backup *api.Backup) error
+...
+}
+```
+
+A new PluginKind, `BackupItemActionV2`, will be created, and the backup process will be modified to use this plugin kind.
+See [Plugin Versioning](plugin-versioning.md) for more details on implementation plans, including v1 adapters, etc.
+
+
+## Compatibility
+The included v1 adapter will allow any existing BackupItemAction plugin to work as expected, with an empty operation ID returned from Execute() and no-op Progress() and Cancel() methods.
+
+## Implementation
+This will be implemented during the Velero 1.11 development cycle.

design/Implemented/cluster-scope-resource-filter.md

@@ -0,0 +1,402 @@
+# Proposal to add resource filters for backup can distinguish whether resource is cluster-scoped or namespace-scoped.
+
+- [Proposal to add resource filters for backup can distinguish whether resource is cluster-scoped or namespace-scoped.](#proposal-to-add-resource-filters-for-backup-can-distinguish-whether-resource-is-cluster-scoped-or-namespace-scoped)
+	- [Abstract](#abstract)
+	- [Background](#background)
+	- [Goals](#goals)
+	- [Non Goals](#non-goals)
+	- [High-Level Design](#high-level-design)
+		- [Parameters Rules](#parameters-rules)
+		- [Using scenarios:](#using-scenarios)
+			- [no namespace-scoped resources + some cluster-scoped resources](#no-namespace-scoped-resources--some-cluster-scoped-resources)
+			- [no namespace-scoped resources + all cluster-scoped resources](#no-namespace-scoped-resources--all-cluster-scoped-resources)
+			- [some namespace-scoped resources + no cluster-scoped resources](#some-namespace-scoped-resources--no-cluster-scoped-resources)
+				- [scenario 1](#scenario-1)
+				- [scenario 2](#scenario-2)
+				- [scenario 3](#scenario-3)
+				- [scenario 4](#scenario-4)
+			- [some namespace-scoped resources + only related cluster-scoped resources](#some-namespace-scoped-resources--only-related-cluster-scoped-resources)
+				- [scenario 1](#scenario-1-1)
+				- [scenario 2](#scenario-2-1)
+				- [scenario 3](#scenario-3-1)
+			- [some namespace-scoped resources + some additional cluster-scoped resources](#some-namespace-scoped-resources--some-additional-cluster-scoped-resources)
+				- [scenario 1](#scenario-1-2)
+				- [scenario 2](#scenario-2-2)
+				- [scenario 3](#scenario-3-2)
+				- [scenario 4](#scenario-4-1)
+			- [some namespace-scoped resources + all cluster-scoped resources](#some-namespace-scoped-resources--all-cluster-scoped-resources)
+				- [scenario 1](#scenario-1-3)
+				- [scenario 2](#scenario-2-3)
+				- [scenario 3](#scenario-3-3)
+			- [all namespace-scoped resources + no cluster-scoped resources](#all-namespace-scoped-resources--no-cluster-scoped-resources)
+			- [all namespace-scoped resources + some additional cluster-scoped resources](#all-namespace-scoped-resources--some-additional-cluster-scoped-resources)
+			- [all namespace-scoped resources + all cluster-scoped resources](#all-namespace-scoped-resources--all-cluster-scoped-resources)
+			- [describe command change](#describe-command-change)
+	- [Detailed Design](#detailed-design)
+	- [Alternatives Considered](#alternatives-considered)
+	- [Security Considerations](#security-considerations)
+	- [Compatibility](#compatibility)
+	- [Implementation](#implementation)
+	- [Open Issues](#open-issues)
+
+## Abstract
+The current filter (IncludedResources/ExcludedResources + IncludeClusterResources flag) is not enough for some special cases, e.g. all namespace-scoped resources + some kind of cluster-scoped resource and all namespace-scoped resources + cluster-scoped resource excludes.
+Propose to add a new group of resource filtering parameters, which can distinguish cluster-scoped and namespace-scoped resources.
+
+## Background
+There are two sets of resource filters for Velero: `IncludedNamespaces/ExcludedNamespaces` and `IncludedResources/ExcludedResources`. 
+`IncludedResources` means only including the resource types specified in the parameter. Both cluster-scoped and namespace-scoped resources are handled in this parameter by now.
+The k8s resources are separated into cluster-scoped and namespace-scoped.
+As a result, it's hard to include all resources in one group and only including specified resource in the other group.
+
+## Goals
+- Make Velero can support more complicated namespace-scoped and cluster-scoped resources filtering scenarios in backup.
+
+## Non Goals
+- Enrich the resource filtering rules, for example, advanced PV filtering and filtering by resource names.
+
+
+## High-Level Design
+Four new parameters are added into command `velero backup create`: `--include-cluster-scoped-resources`, `--exclude-cluster-scoped-resources`, `--include-namespace-scoped-resources` and `--exclude-namespace-scoped-resources`. 
+`--include-cluster-scoped-resources` and `--exclude-cluster-scoped-resources` are used to filter cluster-scoped resources included or excluded in backup per resource type.
+`--include-namespace-scoped-resources` and `--exclude-namespace-scoped-resources` are used to filter namespace-scoped resources included or excluded in backup per resource type.
+Restore and other code pieces also use resource filtering will be handled in future releases.
+
+### Parameters Rules
+
+* `--include-cluster-scoped-resources`, `--include-namespace-scoped-resources`, `--exclude-cluster-scoped-resources` and `--exclude-namespace-scoped-resources` valid value include `*` and comma separated string. Each element of the CSV string should a k8s resource name. The format should be `resource.group`, such as `storageclasses.storage.k8s.io.`.
+
+* `--include-cluster-scoped-resources`, `--include-namespace-scoped-resources`, `--exclude-cluster-scoped-resources` and `--exclude-namespace-scoped-resources` parameters are mutual exclusive with `--include-cluster-resources`, `--include-resources` and `--exclude-resources` parameters. If both sets of parameters are provisioned, validation failure should be returned.
+
+* `--include-cluster-scoped-resources` and `--exclude-cluster-scoped-resources` should only contain cluster-scoped resource type names. If namespace-scoped resource type names are included, they are ignored.
+
+* If there are conflicts between `--include-cluster-scoped-resources` and `--exclude-cluster-scoped-resources` specified resources type lists, `--exclude-cluster-scoped-resources` parameter has higher priority.
+
+* `--include-namespace-scoped-resources` and `--exclude-namespace-scoped-resources` should only contain namespace-scoped resource type names. If cluster-scoped resource type names are included, they are ignored.
+
+* If there are conflicts between `--include-namespace-scoped-resources` and `--exclude-namespace-scoped-resources` specified resources type lists, `--exclude-namespace-scoped-resources` parameter has higher priority.
+
+* If  `--include-namespace-scoped-resources` is not present, it means all namespace-scoped resources are included per resource type.
+
+* If both `--include-cluster-scoped-resources` and `--exclude-cluster-scoped-resources` are not present, it means no additional cluster-scoped resource is included per resource type, just as the existing `--include-cluster-resources` parameter not setting value. Cluster-scoped resources are related to the namespace-scoped resources, which means those are returned in the namespace-scoped resources' BackupItemAction's result AdditionalItems array, are still included in backup by default. Taking backing up PVC scenario as an example, PVC is namespace-scoped, PV is cluster-scoped. PVC's BIA will include PVC related PV into backup too.
+
+### Using scenarios:
+Please notice, if the scenario give the example of using old filtering parameters (`--include-cluster-resources`, `--include-resources` and `--exclude-resources`), that means the old parameters also work for this case. If old parameters example is not given, that means they don't work for this scenario, only new parameters (`--include-cluster-scoped-resources`, `--include-namespace-scoped-resources`, `--exclude-cluster-scoped-resources` and `--exclude-namespace-scoped-resources`) work.
+
+#### no namespace-scoped resources + some cluster-scoped resources
+The following command means backup no namespace-scoped resources and some cluster-scoped resources.
+
+``` bash
+velero backup create <backup-name>
+	--exclude-namespace-scoped-resources=*
+	--include-cluster-scoped-resources=storageclass
+```
+
+#### no namespace-scoped resources + all cluster-scoped resources
+The following command means backup no namespace-scoped resources and all cluster-scoped resources.
+
+``` bash
+velero backup create <backup-name>
+	--exclude-namespace-scoped-resources=*
+	--include-cluster-scoped-resources=*
+```
+
+#### some namespace-scoped resources + no cluster-scoped resources
+##### scenario 1
+The following commands mean backup all resources in namespaces default and kube-system, and no cluster-scoped resources.
+
+Example of new parameters:
+``` bash
+velero backup create <backup-name>
+	--include-namespaces=default,kube-system
+	--exclude-cluster-scoped-resources=*
+```
+
+Example of old parameters:
+``` bash
+velero backup create <backup-name>
+	--include-namespaces=default,kube-system
+	--include-cluster-resources=false
+```
+##### scenario 2
+The following commands mean backup PVC, Deployment, Service, Endpoint, Pod and ReplicaSet resources in all namespaces, and no cluster-scoped resources. Although PVC's related PV should be included, due to no cluster-scoped resources are included, so they are ruled out too.
+
+Example of new parameters:
+``` bash
+velero backup create <backup-name>
+	--include-namespace-scoped-resources=persistentvolumeclaim,deployment,service,endpoint,pod,replicaset
+	--exclude-cluster-scope-resources=*
+```
+
+Example of old parameters:
+``` bash
+velero backup create <backup-name>
+	--include-resources=persistentvolumeclaim,deployment,service,endpoint,pod,replicaset
+	--include-cluster-resources=false
+```
+##### scenario 3
+The following commands mean backup PVC, Deployment, Service, Endpoint, Pod and ReplicaSet resources in namespace default and kube-system, and no cluster-scoped resources. Although PVC's related PV should be included, due to no cluster-scoped resources are included, so they are ruled out too.
+
+Example of new parameters:
+``` bash
+velero backup create <backup-name>
+	--include-namespaces=default,kube-system
+	--include-namespace-scoped-resources=persistentvolumeclaim,deployment,service,endpoint,pod,replicaset
+	--exclude-cluster-scope-resources=*
+```
+
+Example of old parameters:
+``` bash
+velero backup create <backup-name>
+	--include-namespaces=default,kube-system
+	--include-resources=persistentvolumeclaim,deployment,service,endpoint,pod,replicaset
+	--include-cluster-resources=false
+```
+##### scenario 4
+The following commands mean backup all resources except Ingress type resources in all namespaces, and no cluster-scoped resources.
+
+Example of new parameters:
+``` bash
+velero backup create <backup-name>
+	--exclude-namespace-scoped-resources=ingress
+	--exclude-cluster-scoped-resources=*
+```
+
+Example of old parameters:
+``` bash
+velero backup create <backup-name>
+	--exclude-resources=ingress
+	--include-cluster-resources=false
+```
+
+#### some namespace-scoped resources + only related cluster-scoped resources
+##### scenario 1
+This means backup all resources in namespaces default and kube-system, and related cluster-scoped resources.
+``` bash
+velero backup create <backup-name>
+	--include-namespaces=default,kube-system
+```
+
+##### scenario 2
+This means backup pods and configmaps in namespaces default and kube-system, and related cluster-scoped resources.
+``` bash
+velero backup create <backup-name>
+	--include-namespaces=default,kube-system
+	--include-namespace-scoped-resources=pods,configmaps
+```
+
+##### scenario 3
+This means backup all resources except Ingress type resources in all namespaces, and related cluster-scoped resources.
+
+Example of new parameters:
+``` bash
+velero backup create <backup-name>
+	--exclude-namespace-scoped-resources=ingress
+```
+
+Example of old parameters:
+``` bash
+velero backup create <backup-name>
+	--exclude-resources=ingress
+```
+
+#### some namespace-scoped resources + some additional cluster-scoped resources
+##### scenario 1
+This means backup all resources in namespace in default, kube-system, and related cluster-scoped resources, plus all StorageClass resources.
+``` bash
+velero backup create <backup-name>
+	--include-namespaces=default,kube-system
+	--include-cluster-scoped-resources=storageclass
+```
+
+##### scenario 2
+This means backup PVC, Deployment, Service, Endpoint, Pod and ReplicaSet resources in all namespaces, and related cluster-scoped resources, plus all StorageClass resources, and PVC related PV.
+``` bash
+velero backup create <backup-name>
+	--include-namespace-scoped-resources=persistentvolumeclaim,deployment,service,endpoint,pod,replicaset
+	--include-cluster-scoped-resources=storageclass
+```
+
+##### scenario 3
+This means backup PVC, Deployment, Service, Endpoint, Pod and ReplicaSet resources in default and kube-system namespaces, and related cluster-scoped resources, plus all StorageClass resources, and PVC related PV.
+``` bash
+velero backup create <backup-name>
+	--include-namespace-scoped-resources=persistentvolumeclaim,deployment,service,endpoint,pod,replicaset
+	--include-namespaces=default,kube-system
+	--include-cluster-scoped-resources=storageclass
+```
+
+##### scenario 4
+This means backup PVC, Deployment, Service, Endpoint, Pod and ReplicaSet resources in default and kube-system namespaces, and related cluster-scoped resources, plus all cluster-scoped resources except StorageClass type resources.
+``` bash
+velero backup create <backup-name>
+	--include-namespace-scoped-resources=persistentvolumeclaim,deployment,service,endpoint,pod,replicaset
+	--include-namespaces=default,kube-system
+	--exclude-cluster-scoped-resources=storageclass
+```
+
+#### some namespace-scoped resources + all cluster-scoped resources
+##### scenario 1
+The following commands mean backup all resources in namespace in default, kube-system, and all cluster-scoped resources.
+
+Example of new parameters:
+``` bash
+velero backup create <backup-name>
+	--include-namespaces=default,kube-system
+	--include-cluster-scoped-resources=*
+```
+
+Example of old parameters:
+``` bash
+velero backup create <backup-name>
+	--include-namespaces=default,kube-system
+	--include-cluster-resources=true
+```
+
+##### scenario 2
+This means backup Deployment, Service, Endpoint, Pod and ReplicaSet resources in all namespaces, and all cluster-scoped resources.
+``` bash
+velero backup create <backup-name>
+	--include-namespace-scoped-resources=deployment,service,endpoint,pod,replicaset
+	--include-cluster-scoped-resources=*
+```
+
+##### scenario 3
+This means backup Deployment, Service, Endpoint, Pod and ReplicaSet resources in default and kube-system namespaces, and all cluster-scoped resources.
+``` bash
+velero backup create <backup-name>
+	--include-namespaces=default,kube-system
+	--include-namespace-scoped-resources=deployment,service,endpoint,pod,replicaset
+	--include-cluster-scoped-resources=*
+```
+
+#### all namespace-scoped resources + no cluster-scoped resources
+The following commands all mean backup all namespace-scoped resources and no cluster-scoped resources.
+
+Example of new parameters:
+``` bash
+velero backup create <backup-name>
+	--exclude-cluster-scoped-resources=*
+```
+
+Example of old parameters:
+``` bash
+velero backup create <backup-name>
+	--include-cluster-resources=false
+```
+
+#### all namespace-scoped resources + some additional cluster-scoped resources
+This command means backup all namespace-scoped resources, and related cluster-scoped resources, plus all PersistentVolume resources.
+``` bash
+velero backup create <backup-name>
+	--include-namespaces=*
+	--include-cluster-scoped-resources=persistentvolume
+```
+
+#### all namespace-scoped resources + all cluster-scoped resources
+The following commands have the same meaning: backup all namespace-scoped resources, and all cluster-scoped resources.
+``` bash
+velero backup create <backup-name>
+	--include-cluster-scoped-resources=*
+```
+
+``` bash
+velero backup create <backup-name>
+	--include-cluster-resources=true
+```
+
+#### describe command change
+In `velero backup describe` command, the four new parameters should be outputted too.
+``` bash
+ velero backup describe <backup-name>
+......
+
+Namespaces:
+  Included:  ns2
+  Excluded:  <none>
+
+Resources:
+  Included cluster-scoped:    StorageClass,PersistentVolume
+  Excluded cluster-scoped:    <none>
+  Included namespace-scoped:  default
+  Excluded namespace-scoped:  <none>
+......
+```
+
+**Note:** `velero restore` command doesn't support those four new parameter in Velero v1.11, but `velero schedule` supports the four new parameters through backup specification.
+
+## Detailed Design
+With adding `IncludedNamespaceScopedResources`, `ExcludedNamespaceScopedResources`, `IncludedClusterScopedResources` and `ExcludedClusterScopedResources`, the `BackupSpec` looks like:
+``` go
+type BackupSpec struct {
+    ......
+	// IncludedResources is a slice of resource names to include
+	// in the backup. If empty, all resources are included.
+	// +optional
+	// +nullable
+	IncludedResources []string `json:"includedResources,omitempty"`
+
+	// ExcludedResources is a slice of resource names that are not
+	// included in the backup.
+	// +optional
+	// +nullable
+	ExcludedResources []string `json:"excludedResources,omitempty"`
+
+	// IncludeClusterResources specifies whether cluster-scoped resources
+	// should be included for consideration in the backup.
+	// +optional
+	// +nullable
+	IncludeClusterResources *bool `json:"includeClusterResources,omitempty"`
+
+	// IncludedClusterScopedResources is a slice of cluster-scoped
+	// resource type names to include in the backup.
+	// If set to "*", all cluster scope resource types are included.
+	// The default value is empty, which means only related cluster 
+	// scope resources are included.
+	// +optional
+	// +nullable
+	IncludedClusterScopedResources []string `json:"includedClusterScopedResources,omitempty"`
+
+	// ExcludedClusterScopedResources is a slice of cluster-scoped 
+	// resource type names to exclude from the backup.
+	// If set to "*", all cluster scope resource types are excluded.
+	// +optional
+	// +nullable
+	ExcludedClusterScopedResources []string `json:"excludedClusterScopedResources,omitempty"`
+
+	// IncludedNamespaceScopedResources is a slice of namespace-scoped
+	// resource type names to include in the backup.
+	// The default value is "*".
+	// +optional
+	// +nullable
+	IncludedNamespaceScopedResources []string `json:"includedNamespaceScopedResources,omitempty"`
+
+	// ExcludedNamespaceScopedResources is a slice of namespace-scoped
+	// resource type names to exclude from the backup.
+	// If set to "*", all namespace scope resource types are excluded.
+	// +optional
+	// +nullable
+	ExcludedNamespaceScopedResources []string `json:"excludedNamespaceScopedResources,omitempty"`
+    ......
+}
+```
+
+## Alternatives Considered
+Proposal from Jibu Data [Issue 5120](https://github.com/vmware-tanzu/velero/issues/5120#issue-1304534563)
+
+## Security Considerations
+No security impact.
+
+## Compatibility
+The four new parameters cannot be mixed with existing resource filter parameters: `IncludedResources`, `ExcludedResources` and `IncludeClusterResources`.
+If the new parameters and old parameters both appears in command line, or are specified in backup spec, the command line and the backup should fail.
+
+## Implementation
+This change should be included into Velero v1.11.
+New parameters will coexist with `IncludedResources`, `ExcludedResources` and `IncludeClusterResources`.
+Plan to deprecate `IncludedResources`, `ExcludedResources` and `IncludeClusterResources` in future releases, but also open to the community's feedback.
+
+## Open Issues
+`LabelSelector/OrLabelSelectors` apply to namespace-scoped resources.
+It may be reasonable to make them also working on cluster-scoped resources.
+An issue is created to trace this topic [resource label selector not work for cluster-scoped resources](https://github.com/vmware-tanzu/velero/issues/5787)

design/Implemented/csi-snapshots.md

@@ -304,8 +304,8 @@ Without these objects, the provider-level snapshots cannot be located in order t
 
 
 [1]: https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/
-[2]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/pkg/apis/volumesnapshot/v1alpha1/types.go#L41
-[3]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/pkg/apis/volumesnapshot/v1alpha1/types.go#L161
+[2]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/client/apis/volumesnapshot/v1/types.go#L42
+[3]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/client/apis/volumesnapshot/v1/types.go#L262
 [4]: https://github.com/heptio/velero/blob/main/pkg/volume/snapshot.go#L21
 [5]: https://github.com/heptio/velero/blob/main/pkg/apis/velero/v1/pod_volume_backup.go#L88
 [6]: https://github.com/heptio/velero-csi-plugin/

design/Implemented/delete-item-action.md

@@ -175,7 +175,7 @@ If there are one or more, download the backup tarball from backup storage, untar
 
 ## Alternatives Considered
 
-Another proposal for higher level `DeleteItemActions` was initially included, which would require implementors to individually download the backup tarball themselves.
+Another proposal for higher level `DeleteItemActions` was initially included, which would require implementers to individually download the backup tarball themselves.
 While this may be useful long term, it is not a good fit for the current goals as each plugin would be re-implementing a lot of boilerplate.
 See the deletion-plugins.md file for this alternative proposal in more detail.
 

design/Implemented/existing-resource-policy_design.md

@@ -1,6 +1,6 @@
 # Add support for `ExistingResourcePolicy` to restore API
 ## Abstract
-Velero currently does not support any restore policy on kubernetes resources that are already present in-cluster. Velero skips over the restore of the resource if it already exists in the namespace/cluster irrespective of whether the resource present in the restore is the same or different from the one present on the cluster. It is desired that Velero gives the option to the user to decide whether or not the resource in backup should overwrite the one present in the cluster.
+Velero currently does not support any restore policy on Kubernetes resources that are already present in-cluster. Velero skips over the restore of the resource if it already exists in the namespace/cluster irrespective of whether the resource present in the restore is the same or different from the one present on the cluster. It is desired that Velero gives the option to the user to decide whether or not the resource in backup should overwrite the one present in the cluster.
 
 ## Background
 As of Today, Velero will skip over the restoration of resources that already exist in the cluster. The current workflow followed by Velero is (Using a `service` that is backed up for example):
@@ -145,7 +145,7 @@ type RestoreSpec struct {
 .
 .
 .
-// ExistingResourcePolicy specifies the restore behaviour for the kubernetes resource to be restored
+// ExistingResourcePolicy specifies the restore behaviour for the Kubernetes resource to be restored
 // +optional
 ExistingResourcePolicy PolicyType
 
@@ -167,7 +167,7 @@ type RestoreSpec struct {
 .
 .
 .
-// ExistingResourcePolicyConfig specifies the restore behaviour for a particular/list of kubernetes resource(s) to be restored
+// ExistingResourcePolicyConfig specifies the restore behaviour for a particular/list of Kubernetes resource(s) to be restored
 // +optional
 ExistingResourcePolicyConfig []PolicyConfiguration
 
@@ -205,11 +205,11 @@ type RestoreSpec struct {
 .
 .
 .
-// ExistingResourceDefaultPolicy specifies the default restore behaviour for the kubernetes resource to be restored
+// ExistingResourceDefaultPolicy specifies the default restore behaviour for the Kubernetes resource to be restored
 // +optional
 existingResourceDefaultPolicy PolicyType
 
-// ExistingResourcePolicyOverrides specifies the restore behaviour for a particular/list of kubernetes resource(s) to be restored
+// ExistingResourcePolicyOverrides specifies the restore behaviour for a particular/list of Kubernetes resource(s) to be restored
 // +optional
 existingResourcePolicyOverrides []PolicyConfiguration
 

design/Implemented/general-progress-monitoring.md

@@ -0,0 +1,579 @@
+# Plugin Progress Monitoring
+
+This is intended as a replacement for the previously-approved Upload Progress Monitoring design
+([Upload Progress Monitoring](upload-progress.md)) in order to expand the supported use cases beyond
+snapshot uploads to include what was previously called Async Backup/Restore Item Actions. This
+updated design should handle the combined set of use cases for those previously separate designs.
+
+Volume snapshotter plugin are used by Velero to take snapshots of persistent volume contents. 
+Depending on the underlying storage system, those snapshots may be available to use immediately, 
+they may be uploaded to stable storage internally by the plugin or they may need to be uploaded after
+the snapshot has been taken. We would like for Velero to continue on to the next part of the backup as quickly
+as possible but we would also like the backup to not be marked as complete until it is a usable backup.  We'd also
+eventually like to bring the control of upload under the control of Velero and allow the user to make decisions
+about the ultimate destination of backup data independent of the storage system they're using.
+
+We would also like any internal or third party Backup or Restore Item Action to have the option of
+making use of this same ability to run some external process without blocking the current backup or
+restore operation. Beyond Volume Snapshotters, this is also needed for data mover operations on both
+backup and restore, and potentially useful for other third party operations -- for example
+in-cluster registry image backup or restore could make use of this feature in a third party plugin).
+
+### Glossary
+- <b>BIA</b>: BackupItemAction
+- <b>RIA</b>: RestoreItemAction
+
+## Examples
+- AWS - AWS snapshots return quickly, but are then uploaded in the background and cannot be used until EBS moves
+the data into S3 internally.
+
+- vSphere - The vSphere plugin takes a local snapshot and then the vSphere plugin uploads the data to S3.  The local
+snapshot is usable before the upload completes.
+
+- Restic - Does not go through the volume snapshot path.  Restic backups will block Velero progress
+until completed. However, with the more generalized approach in the revised design, restic/kopia
+backup and restore *could* make use of this framework if their actions are refactored as
+Backup/RestoreItemActions.
+
+- Data Movers
+    - Data movers are asynchronous processes executed inside backup/restore item actions that applies to a specific Kubernetes resources. A common use case for data mover is to backup/restore PVCs whose data we want to move to some form of backup storage outside of using velero kopia/restic implementations.
+    - Workflow
+        - User takes velero backup of PVC A
+        - BIA plugin applies to PVCs with compatible storage driver
+        - BIA plugin triggers data mover
+	  - Most common use case would be for the plugin action to create a new CR which would
+            trigger an external controller action
+	  - Another possible use case would be for the plugin to run its own async action in a
+            goroutine, although this would be less resilient to plugin container restarts.
+        - BIA plugin returns
+        - Velero backup process continues
+        - Main velero backup process monitors running BIA threads via gRPC to determine if process is done and healthy
+
+
+## Primary changes from the original Upload Progress Monitoring design
+
+The most fundamental change here is that rather than proposing a new special-purpose
+SnapshotItemAction, the existing BackupItemAction plugin will be modified to accommodate an optional
+snapshot (or other item operation) ID return. The primary reasons for this change are as follows:
+
+1. The intended scope has moved beyond snapshot processing, so it makes sense to support
+asynchronous operations in other backup or restore item actions.
+
+2. We expect to have plugin API versioning implemented in Velero 1.10, making it feasible to
+implement changes in the existing plugin APIs now.
+
+3. We will need this feature on both backup and restore, meaning that if we took the "new plugin
+type" approach, we'd need two new plugin types.
+
+4. Other than the snapshot/operation ID return, the rest of the plugin processing is identical to
+Backup/RestoreItemActions. With separate plugin types, we'd have to repeat all of that logic
+(including dealing with additional items, etc.) twice.
+
+The other major change is that we will be applying this to both backups and restores, although the
+Volume Snapshotter use case only needs this on backup. This means that everything we're doing around
+backup phase and workflow will also need to be done for restore.
+
+Then there are various minor changes around terminology to make things more generic. Instead of
+"snapshotID", we'll have "operationID" (which for volume snapshotters will be a snapshot
+ID).
+
+## Goals
+
+- Enable monitoring of backup/restore item action operations that continue after snapshotting and other operations have completed
+- Keep non-usable backups and restores (upload/persistence has not finished, etc.) from appearing as completed
+- Make use of plugin API versioning functionality to manage changes to Backup/RestoreItemAction interfaces
+- Enable vendors to plug their own data movers into velero using BIA/RIA plugins
+
+## Non-goals
+- Today, Velero is unable to recover from an in progress backup when the velero server crashes (pod is deleted). This has an impact on running asynchronous processes, but it’s not something we intend to solve in this design.
+
+## Models
+
+### Internal configuration and management
+In this model, movement of the snapshot to stable storage is under the control of the snapshot
+plugin.  Decisions about where and when the snapshot gets moved to stable storage are not
+directly controlled by Velero.  This is the model for the current VolumeSnapshot plugins.
+
+### Velero controlled management
+In this model, the snapshot is moved to external storage under the control of Velero.  This
+enables Velero to move data between storage systems.  This also allows backup partners to use
+Velero to snapshot data and then move the data into their backup repository.
+
+## Backup and Restore phases
+
+Velero currently has backup/restore phases "InProgress" and "Completed".  A backup moves to the
+Completed phase when all of the volume snapshots have completed and the Kubernetes metadata has been
+written into the object store.  However, the actual data movement may be happening in the background
+after the backup has been marked "Completed".  The backup is not actually a stable backup until the
+data has been persisted properly.  In some cases (e.g. AWS) the backup cannot be restored from until
+the snapshots have been persisted.
+
+Once the snapshots have been taken, however, it is possible for additional backups or restores (as
+long as they don't use not-yet-completed backups) to be made without interference.  Waiting until
+all data has been moved before starting the next backup will slow the progress of the system without
+adding any actual benefit to the user.
+
+New backup/restore phases, "WaitingForPluginOperations" and
+"WaitingForPluginOperationsPartiallyFailed" will be introduced.  When a backup or restore has
+entered one of these phases, Velero is free to start another backup/restore.  The backup/restore
+will remain in the "WaitingForPluginOperations" phase until all BIA/RIA operations have completed
+(for example, for a volume snapshotter, until all data has been successfully moved to persistent
+storage).  The backup/restore will not fail once it reaches this phase, although an error return
+from a plugin could cause a backup or restore to move to "PartiallyFailed".  If the backup is
+deleted (cancelled), the plugins will attempt to delete the snapshots and stop the data movement -
+this may not be possible with all storage systems.
+
+In addition, for backups (but not restores), there will also be two additional phases, "Finalizing"
+and "FinalizingPartiallyFailed", which will handle any steps required after plugin operations have
+all completed. Initially, this will just include adding any required resources to the backup that
+might have changed during asynchronous operation execution, although eventually other cleanup
+actions could be added to this phase.
+
+### State progression
+
+![image](AsyncActionFSM.png)
+### New
+When a backup/restore request is initially created, it is in the "New" phase.  
+
+The next state is either "InProgress" or "FailedValidation"
+
+### FailedValidation
+If the backup/restore request is incorrectly formed, it goes to the "FailedValidation" phase and
+terminates
+
+### InProgress
+When work on the backup/restore begins, it moves to the "InProgress" phase.  It remains in the
+"InProgress" phase until all pre/post execution hooks have been executed, all snapshots have been
+taken and the Kubernetes metadata and backup/restore info is safely written to the object store
+plugin.
+
+In the current implementation, Restic backups will move data during the "InProgress" phase.  In the
+future, it may be possible to combine a snapshot with a Restic (or equivalent) backup which would
+allow for data movement to be handled in the "WaitingForPluginOperations" phase,
+
+The next phase would be "WaitingForPluginOperations" for backups or restores which have unfinished
+asynchronous plugin operations and no errors so far, "WaitingForPluginOperationsPartiallyFailed" for
+backups or restores which have unfinished asynchronous plugin operations at least one error,
+"Completed" for restores with no unfinished asynchronous plugin operations and no errors,
+"PartiallyFailed" for restores with no unfinished asynchronous plugin operations and at least one
+error, "Finalizing" for backups with no unfinished asynchronous plugin operations and no errors,
+"FinalizingPartiallyFailed" for backups with no unfinished asynchronous plugin operations and at
+least one error, or "PartiallyFailed".  Backups/restores which would have a final phase of
+"Completed" or "PartiallyFailed" may move to the "WaitingForPluginOperations" or
+"WaitingForPluginOperationsPartiallyFailed" state.  A backup/restore which will be marked "Failed"
+will go directly to the "Failed" phase.  Uploads may continue in the background for snapshots that
+were taken by a "Failed" backup/restore, but no progress will not be monitored or updated. If there
+are any operations in progress when a backup is moved to the "Failed" phase (although with the
+current workflow, that shouldn't happen), Cancel() should be called on these operations. When a
+"Failed" backup is deleted, all snapshots will be deleted and at that point any uploads still in
+progress should be aborted.
+
+### WaitingForPluginOperations (new)
+The "WaitingForPluginOperations" phase signifies that the main part of the backup/restore, including
+snapshotting has completed successfully and uploading and any other asynchronous BIA/RIA plugin
+operations are continuing.  In the event of an error during this phase, the phase will change to
+WaitingForPluginOperationsPartiallyFailed.  On success, the phase changes to
+"Finalizing" for backups and "Completed" for restores.  Backups cannot be
+restored from when they are in the WaitingForPluginOperations state.
+
+### WaitingForPluginOperationsPartiallyFailed (new)
+The "WaitingForPluginOperationsPartiallyFailed" phase signifies that the main part of the
+backup/restore, including snapshotting has completed, but there were partial failures either during
+the main part or during any async operations, including snapshot uploads.  Backups cannot be
+restored from when they are in the WaitingForPluginOperationsPartiallyFailed state.
+
+### Finalizing (new)
+The "Finalizing" phase signifies that asynchronous backup operations have all completed successfully
+and Velero is currently backing up any resources indicated by asynchronous plugins as items to back
+up after operations complete. Once this is done, the phase changes to Completed.  Backups cannot be
+restored from when they are in the Finalizing state.
+
+### FinalizingPartiallyFailed (new)
+
+The "FinalizingPartiallyFailed" phase signifies that, for a backup which had errors during initial
+processing or asynchronous plugin operation, asynchronous backup operations have all completed and
+Velero is currently backing up any resources indicated by asynchronous plugins as items to back up
+after operations complete. Once this is done, the phase changes to PartiallyFailed.  Backups cannot
+be restored from when they are in the FinalizingPartiallyFailed state.
+
+### Failed
+When a backup/restore has had fatal errors it is marked as "Failed" Backups in this state cannot be
+restored from.
+
+### Completed
+The "Completed" phase signifies that the backup/restore has completed, all data has been transferred
+to stable storage (or restored to the cluster) and any backup in this state is ready to be used in a
+restore.  When the Completed phase has been reached for a backup it is safe to remove any of the
+items that were backed up.
+
+### PartiallyFailed
+The "PartiallyFailed" phase signifies that the backup/restore has completed and at least part of the
+backup/restore is usable.  Restoration from a PartiallyFailed backup will not result in a complete
+restoration but pieces may be available.
+
+## Workflow
+
+When a Backup or Restore Action is executed, any BackupItemAction, RestoreItemAction, or
+VolumeSnapshot plugins will return operation IDs (snapshot IDs or other plugin-specific
+identifiers).  The plugin should be able to provide status on the progress for the snapshot and
+handle cancellation of the operation/upload if the snapshot is deleted.  If the plugin is restarted,
+the operation ID should remain valid.
+
+When all snapshots have been taken and Kubernetes resources have been persisted to the ObjectStorePlugin
+the backup will either have fatal errors or will be at least partially usable.
+
+If the backup/restore has fatal errors it will move to the "Failed" state and finish. If a
+backup/restore fails, the upload or other operation will not be cancelled but it will not be
+monitored either.  For backups in any phase, all snapshots will be deleted when the backup is
+deleted.  Plugins will cancel any data movement or other operations and remove snapshots and other
+associated resources when the VolumeSnapshotter DeleteSnapshot method or DeleteItemAction Execute
+method is called.
+
+Velero will poll the plugins for status on the operations when the backup/restore exits the
+"InProgress" phase and has no fatal errors.
+
+If any operations are not complete, the backup/restore will move to either WaitingForPluginOperations
+or WaitingForPluginOperationsPartiallyFailed or Failed.
+
+Post-snapshot and other operations may take a long time and Velero and its plugins may be restarted
+during this time.  Once a backup/restore has moved into the WaitingForPluginOperations or
+WaitingForPluginOperationsPartiallyFailed phase, another backup/restore may be started.
+
+While in the WaitingForPluginOperations or WaitingForPluginOperationsPartiallyFailed phase, the
+snapshots and item actions will be periodically polled.  When all of the snapshots and item actions
+have reported success, restores will move directly to the Completed or PartiallyFailed phase, and
+backups will move to the Finalizing or FinalizingPartiallyFailed phase, depending on whether the
+backup/restore was in the WaitingForPluginOperations or WaitingForPluginOperationsPartiallyFailed
+phase.
+
+While in the Finalizing or FinalizingPartiallyFailed phase, Velero will update the backup with any
+resources indicated by plugins that they must be added to the backup after operations are completed,
+and then the backup will move to the Completed or PartiallyFailed phase, depending on whether there
+are any backup errors.
+
+The Backup resources will be written to object storage at the time the backup leaves the InProgress
+phase, but it will not be synced to other clusters (or usable for restores in the current cluster)
+until the backup has entered a final phase: Completed, Failed or PartiallyFailed. During the
+Finalizing phases, a the backup resources will be updated with any required resources related to
+asynchronous plugins.
+
+## Reconciliation of InProgress backups
+
+InProgress backups will not have a `velero-backup.json` present in the object store.  During
+reconciliation, backups which do not have a `velero-backup.json` object in the object store will be
+ignored.
+
+## Plugin API changes
+
+### OperationProgress struct
+
+    type OperationProgress struct {
+        Completed bool                          // True when the operation has completed, either successfully or with a failure
+        Err string                              // Set when the operation has failed
+        NCompleted, NTotal int64                // Quantity completed so far and the total quantity associated with the operation in operationUnits
+                                                // For data mover and volume snapshotter use cases, this would be in bytes
+                                                // On successful completion, completed and total should be the same.
+        OperationUnits string                   // Units represented by completed and total -- for data mover and item
+	                                        // snapshotters, this will usually be bytes.
+        Description string                      // Optional description of operation progress
+        Started, Updated time.Time              // When the upload was started and when the last update was seen.  Not all
+                                                // systems retain when the upload was begun, return Time 0 (time.Unix(0, 0))
+                                                // if unknown.
+    }
+
+### VolumeSnapshotter changes
+
+Two new methods will be added to the VolumeSnapshotter interface:
+
+    Progress(snapshotID string) (OperationProgress, error)
+    Cancel(snapshotID string) (error)
+
+Progress will report the current status of a snapshot upload.  This should be callable at
+any time after the snapshot has been taken.  In the event a plugin is restarted, if the operationID
+(snapshot ID) continues to be valid it should be possible to retrieve the progress.
+
+`error` is set if there is an issue retrieving progress.  If the snapshot is has encountered an
+error during the upload, the error should be returned in OperationProgress and error should be nil.
+
+### BackupItemAction and RestoreItemAction plugin changes
+
+Currently CSI snapshots and the Velero Plugin for vSphere are implemented as BackupItemAction
+plugins.  While the majority of BackupItemAction plugins do not take snapshots or upload data, this
+functionality is useful for any longstanding plugin operation managed by an external
+process/controller so we will modify BackupItemAction and RestoreItemAction to optionally return an
+operationID in addition to the modified item.
+
+Velero can attempt to cancel an operation by calling the Cancel API call on the BIA/RIA. The plugin
+can then take any appropriate action as needed. Cancel will be called for unfinished operations on
+backup deletion, and possibly reaching timeout. Cancel is not intended to be used to delete/remove
+the results of completed actions and will have no effect on a completed action. Cancel has no return
+value apart from the standard Error return, but this should only be used for unexpected
+failures. Under normal operations, Cancel will simply return a nil error (and nothing else), whether
+or not the plugin is able to cancel the operation.
+
+_AsyncOperationsNotSupportedError_ should only be returned (by Progress) if the
+Backup/RestoreItemAction plugin should not be handling the item.  If the Backup/RestoreItemAction
+plugin should handle the item but, for example, the item/snapshot ID cannot be found to report
+progress, Progress will return an InvalidOperationIDError error rather than a populated
+OperationProgress struct. If the item action does not start an asynchronous operation, then
+operationID will be empty.
+
+Three new methods will be added to the BackupItemAction interface, and the Execute() return signature
+will be modified:
+
+	// Name returns the name of this BIA. Plugins which implement this interface must defined Name,
+	// but its content is unimportant, as it won't actually be called via RPC. Velero's plugin infrastructure
+	// will implement this directly rather than delegating to the RPC plugin in order to return the name
+	// that the plugin was registered under. The plugins must implement the method to complete the interface.
+	Name() string
+	// Execute allows the BackupItemAction to perform arbitrary logic with the item being backed up,
+	// including mutating the item itself prior to backup. The item (unmodified or modified)
+	// should be returned, along with an optional slice of ResourceIdentifiers specifying
+	// additional related items that should be backed up now, an optional operationID for actions which
+	// initiate asynchronous actions, and a second slice of ResourceIdentifiers specifying related items
+	// which should be backed up after all asynchronous operations have completed. This last field will be
+	// ignored if operationID is empty, and should not be filled in unless the resource must be updated in the
+	// backup after async operations complete (i.e. some of the item's Kubernetes metadata will be updated
+	// during the asynch operation which will be required during restore)
+	Execute(item runtime.Unstructured, backup *api.Backup) (runtime.Unstructured, []velero.ResourceIdentifier, string, []velero.ResourceIdentifier, error)
+    	
+    	// Progress  
+	Progress(operationID string, backup *api.Backup) (velero.OperationProgress, error)
+    	// Cancel  
+	Cancel(operationID string, backup *api.Backup) error
+
+Three new methods will be added to the RestoreItemAction interface, and the
+RestoreItemActionExecuteOutput struct will be modified:
+
+	// Name returns the name of this RIA. Plugins which implement this interface must defined Name,
+	// but its content is unimportant, as it won't actually be called via RPC. Velero's plugin infrastructure
+	// will implement this directly rather than delegating to the RPC plugin in order to return the name
+	// that the plugin was registered under. The plugins must implement the method to complete the interface.
+	Name() string
+	// Execute allows the ItemAction to perform arbitrary logic with the item being restored,
+	// including mutating the item itself prior to restore. The item (unmodified or modified)
+	// should be returned, an optional OperationID, along with an optional slice of ResourceIdentifiers
+	// specifying additional related items that should be restored, a warning (which will be
+	// logged but will not prevent the item from being restored) or error (which will be logged
+	// and will prevent the item from being restored) if applicable. If OperationID is specified
+    	// then velero will wait for this operation to complete before the restore is marked Completed.
+	Execute(input *RestoreItemActionExecuteInput) (*RestoreItemActionExecuteOutput, error)
+
+    	
+    	// Progress  
+	Progress(operationID string, restore *api.Restore) (velero.OperationProgress, error)
+
+        // Cancel  
+	Cancel(operationID string, restore *api.Restore) error
+    
+    // RestoreItemActionExecuteOutput contains the output variables for the ItemAction's Execution function.
+    type RestoreItemActionExecuteOutput struct {
+        // UpdatedItem is the item being restored mutated by ItemAction.
+        UpdatedItem runtime.Unstructured
+
+        // AdditionalItems is a list of additional related items that should
+        // be restored.
+        AdditionalItems []ResourceIdentifier
+
+        // SkipRestore tells velero to stop executing further actions
+        // on this item, and skip the restore step. When this field's
+        // value is true, AdditionalItems will be ignored.
+        SkipRestore bool
+
+        // OperationID is an identifier which indicates an ongoing asynchronous action which Velero will
+        // continue to monitor after restoring this item. If left blank, then there is no ongoing operation
+        OperationID string
+    }
+
+## Changes in Velero backup format
+
+No changes to the existing format are introduced by this change.  As part of the backup workflow changes, a
+`<backup-name>-itemoperations.json.gz` file will be added that contains the items and operation IDs
+(snapshotIDs) returned by VolumeSnapshotter and BackupItemAction plugins.  Also, the creation of the
+`velero-backup.json` object will not occur until the backup moves to one of the terminal phases
+(_Completed_, _PartiallyFailed_, or _Failed_).  Reconciliation should ignore backups that do not
+have a `velero-backup.json` object.
+
+The Backup/RestoreItemAction plugin identifier as well as the ItemID and OperationID will be stored
+in the `<backup-name>-itemoperations.json.gz`.  When checking for progress, this info will be used
+to select the appropriate Backup/RestoreItemAction plugin to query for progress. Here's an example
+of what a record for a datamover plugin might look like:
+```
+    {
+        "spec": {
+            "backupName": "backup-1",
+            "backupUID": "f8c72709-0f73-46e1-a071-116bc4a76b07",
+            "backupItemAction": "velero.io/volumesnapshotcontent-backup",
+            "resourceIdentifier": {
+	        "Group": "snapshot.storage.k8s.io",
+                "Resource": "VolumeSnapshotContent",
+                "Namespace": "my-app",
+                "Name": "my-volume-vsc"
+            },
+            "operationID": "<DataMoverBackup objectReference>",
+            "itemsToUpdate": [
+	      {
+	        "Group": "velero.io",
+                "Resource": "VolumeSnapshotBackup",
+                "Namespace": "my-app",
+                "Name": "vsb-1"
+              }
+	    ]
+	},
+        "status": {
+            "operationPhase": "Completed",
+            "error": "",
+            "nCompleted": 12345,
+            "nTotal": 12345,
+            "operationUnits": "byte",
+            "description": "",
+            "Created": "2022-12-14T12:00:00Z",
+            "Started": "2022-12-14T12:01:00Z",
+            "Updated": "2022-12-14T12:11:02Z"
+        },
+    }
+```
+
+The cluster that is creating the backup will have the Backup resource present and will be able to
+manage the backup before the backup completes.
+
+If the Backup resource is removed (e.g. Velero is uninstalled) before a backup completes and writes
+its `velero-backup.json` object, the other objects in the object store for the backup will be
+effectively orphaned.  This can currently happen but the current window is much smaller.
+
+### `<backup-name>-itemoperations.json.gz`
+The itemoperations file is similar to the existing `<backup-name>-itemsnapshots.json.gz`  Each snapshot taken via
+BackupItemAction will have a JSON record in the file.  Exact format TBD.
+
+This file will be uploaded to object storage at the end of processing all of the items in the
+backup, before the phase moves away from `InProgress`.
+
+## Changes to Velero restores
+
+A `<restore-name>-itemoperations.json.gz` file will be added that contains the items and operation
+IDs returned by RestoreItemActions. The format will be the same as the
+`<backup-name>-itemoperations.json.gz` generated for backups.
+
+This file will be uploaded to object storage at the end of processing all of the items in the
+restore, before the phase moves away from `InProgress`.
+
+## CSI snapshots
+
+For systems such as EBS, a snapshot is not available until the storage system has transferred the
+snapshot to stable storage.  CSI snapshots expose the _readyToUse_ state that, in the case of EBS,
+indicates that the snapshot has been transferred to durable storage and is ready to be used.  The
+CSI BackupItemAction.Progress method will poll that field and when completed, return completion.
+
+## vSphere plugin
+
+The vSphere Plugin for Velero uploads snapshots to S3 in the background.  This is also a
+BackupItemAction plugin, it will check the status of the Upload records for the snapshot and return
+progress.
+
+## Backup workflow changes
+
+The backup workflow remains the same until we get to the point where the `velero-backup.json` object
+is written.  At this point, Velero will
+run across all of the VolumeSnapshotter/BackupItemAction operations and call the _Progress_ method
+on each of them.
+
+If all backup item operations have finished (either successfully or failed), the backup will move to
+one of the finalize phases.
+
+If any of the snapshots or backup items are still being processed, the phase of the backup will be
+set to the appropriate phase (_WaitingForPluginOperations_ or
+_WaitingForPluginOperationsPartiallyFailed_), and the async backup operations controller will
+reconcile periodically and call Progress on any unfinished operations. In the event of any of the
+progress checks return an error, the phase will move to _WaitingForPluginOperationsPartiallyFailed_.
+
+Once all operations have completed, the backup will be moved to one of the finalize phases, and the
+backup finalizer controller will update the the `velero-backup.json`in the object store with any
+resources necessary after asynchronous operations are complete and the backup will move to the
+appropriate terminal phase.
+
+
+## Restore workflow changes
+
+The restore workflow remains the same until velero would currently move the backup into one of the
+terminal states. At this point, Velero will run across all of the RestoreItemAction operations and
+call the _Progress_ method on each of them.
+
+If all restore item operations have finished (either successfully or failed), the restore will be
+completed and the restore will move to the appropriate terminal phase and the restore will be
+complete.
+
+If any of the restore items are still being processed, the phase of the restore will be set to the
+appropriate phase (_WaitingForPluginOperations_ or _WaitingForPluginOperationsPartiallyFailed_), and
+the async restore operations controller will reconcile periodically and call Progress on any
+unfinished operations. In the event of any of the progress checks return an error, the phase will
+move to _WaitingForPluginOperationsPartiallyFailed_. Once all of the operations have completed, the
+restore will be moved to the appropriate terminal phase.
+
+## Restart workflow
+
+On restart, the Velero server will scan all Backup/Restore resources.  Any Backup/Restore resources
+which are in the _InProgress_ phase will be moved to the _Failed_ phase.  Any Backup/Restore
+resources in the _WaitingForPluginOperations_ or _WaitingForPluginOperationsPartiallyFailed_ phase
+will be treated as if they have been requeued and progress checked and the backup/restore will be
+requeued or moved to a terminal phase as appropriate.
+
+## Notes on already-merged code which may need updating
+
+Since this design is modifying a previously-approved design, there is some preparation work based on
+the earlier upload progress monitoring design that may need modification as a result of these
+updates. Here is a list of some of these items:
+
+1. Consts for the "Uploading" and "UploadingPartiallyFailed" phases have already been defined. These
+   will need to be removed when the "WaitingForPluginOperations" and
+   "WaitingForPluginOperationsPartiallyFailed" phases are defined.
+   - https://github.com/vmware-tanzu/velero/pull/3805
+1. Remove the ItemSnapshotter plugin APIs (and related code) since the revised design will reuse
+   VolumeSnapshotter and BackupItemAction plugins.
+   - https://github.com/vmware-tanzu/velero/pull/4077
+   - https://github.com/vmware-tanzu/velero/pull/4417
+1. UploadProgressFeatureFlag shouldn't be needed anymore. The current design won't really need a
+   feature flag here -- the new features will be added to V2 of the VolumeSnapshotter,
+   BackupItemAction, and RestoreItemAction plugins, and it will only be used if there are plugins which
+   return operation IDs.
+   - https://github.com/vmware-tanzu/velero/pull/4416
+1. Adds <backup-name>-itemsnapshots.gz file to backup (when provided) -- this is still part of the
+   revised design, so it should stay.
+   - https://github.com/vmware-tanzu/velero/pull/4429
+1. Upload Progress Monitoring and Item Snapshotter basic support: This PR is not yet merged, so
+   nothing will need to be reverted. While the implementation here will be useful in informing the
+   implementation of the new design, several things have changed in the design proposal since the PR
+   was written.
+   - https://github.com/vmware-tanzu/velero/pull/4467
+
+# Implementation tasks
+
+VolumeSnapshotter new plugin APIs  
+BackupItemAction new plugin APIs  
+RestoreItemAction new plugin APIs  
+New backup phases  
+New restore phases  
+Defer uploading `velero-backup.json`  
+AWS EBS plugin Progress implementation  
+Operation monitoring  
+Implementation of `<backup-name>-itemoperations.json.gz` file  
+Implementation of `<restore-name>-itemoperations.json.gz` file
+Restart logic  
+Change in reconciliation logic to ignore backups/restores that have not completed  
+CSI plugin BackupItemAction Progress implementation  
+vSphere plugin BackupItemAction Progress implementation (vSphere plugin team)  
+
+
+# Open Questions
+
+1. Do we need a Cancel operation for VolumeSnapshotter?
+   - From feedback, I'm thinking we probably don't need it. The only real purpose of Cancel
+     here is to tell the plugin that Velero won't be waiting anymore, so if there are any
+     required custom cancellation actions, now would be a good time to perform them. For snapshot
+     uploads that are already in proress, there's not really anything else to cancel.
+2. Should we actually write the backup *before* moving to the WaitingForPluginOperations or
+   WaitingForPluginOperationsPartiallyFailed phase rather than waiting until all operations
+   have completed? The operations in question won't affect what gets written to object storage
+   for the backup, and since we've already written the list of operations we're waiting for to
+   object storage, writing the backup now would make the process resilient to Velero restart if
+   it happens during WaitingForPluginOperations or WaitingForPluginOperationsPartiallyFailed
+

design/Implemented/handle-backup-of-volumes-by-resources-filters.md

@@ -0,0 +1,324 @@
+# Handle backup of volumes by resources filters
+ 
+## Abstract
+Currently, Velero doesn't have one flexible way to handle volumes.
+ 
+If users want to skip the backup of volumes or only backup some volumes in different namespaces in batch, currently they need to use the opt-in and opt-out approach one by one, or use label-selector but if it has big different labels on each different related pod, which is cumbersome when they have lots of volumes to handle with. it would be convenient if Velero could provide policies to handle the backup of volumes just by `some specific volumes conditions`.
+ 
+## Background
+As of Today, Velero has lots of filters to handle (backup or skip backup) resources including resources filters like `IncludedNamespaces, ExcludedNamespaces`, label selectors like `LabelSelector, OrLabelSelectors`, annotation like `backup.velero.io/must-include-additional-items` etc. But it's not enough flexible to handle volumes, we need one generic way to handle volumes.
+ 
+## Goals
+- Introducing flexible policies to handle volumes, and do not patch any labels or annotations to the pods or volumes.
+ 
+## Non-Goals
+- We only handle volumes for backup and do not support restore.
+- Currently, only handles volumes, and does not support other resources.
+- Only environment-unrelated and platform-independent general volumes attributes are supported, do not support volumes attributes related to a specific environment.
+ 
+## Use-cases/Scenarios
+### Skip backup volumes by some attributes
+Users want to skip PV with the requirements:
+- option to skip all PV data
+- option to skip specified PV type (RBD, NFS)
+- option to skip specified PV size
+- option to skip specified storage-class
+ 
+## High-Level Design
+First, Velero will provide the user with one YAML file template and all supported volume policies will be in.
+
+Second, writing your own configuration file by imitating the YAML template, it could be partial volume policies from the template.
+
+Third, create one configmap from your own configuration file, and the configmap should be in Velero install namespace.
+
+Fourth, create a backup with the command `velero backup create --resource-policies-configmap $policiesConfigmap`, which will reference the current backup to your volume policies. At the same time, Velero will validate all volume policies user imported, the backup will fail if the volume policies are not supported or some items could not be parsed.
+
+Fifth, the current backup CR will record the reference of volume policies configmap.
+
+Sixth, Velero first filters volumes by other current supported filters, at last, it will apply the volume policies to the filtered volumes to get the final matched volume to handle.
+ 
+## Detailed Design
+The volume resources policies should contain a list of policies which is the combination of conditions and related `action`, when target volumes meet the conditions, the related `action` will take effection.
+
+Below is the API Design for the user configuration:
+
+### API Design
+```go
+type VolumeActionType string
+
+const Skip VolumeActionType = "skip"
+
+// Action defined as one action for a specific way of backup
+type Action struct {
+	// Type defined specific type of action, it could be 'file-system-backup', 'volume-snapshot', or 'skip' currently
+	Type VolumeActionType `yaml:"type"`
+	// Parameters defined map of parameters when executing a specific action
+	// +optional
+	// +nullable
+	Parameters map[string]interface{} `yaml:"parameters,omitempty"`
+}
+
+// VolumePolicy defined policy to conditions to match Volumes and related action to handle matched Volumes
+type VolumePolicy struct {
+	// Conditions defined list of conditions to match Volumes
+	Conditions map[string]interface{} `yaml:"conditions"`
+	Action     Action                 `yaml:"action"`
+}
+
+// ResourcePolicies currently defined slice of volume policies to handle backup
+type ResourcePolicies struct {
+  Version        string         `yaml:"version"`
+	VolumePolicies []VolumePolicy `yaml:"volumePolicies"`
+	// we may support other resource policies in the future, and they could be added separately
+  	// OtherResourcePolicies: []OtherResourcePolicy
+}
+```
+
+The policies YAML config file would look like this:
+```yaml
+version: v1
+volumePolicies:
+# it's a list and if the input item matches the first policy, the latters will be ignored
+# each policy consists of a list of conditions and an action
+
+# each key in the object is one condition, and one policy will apply to resources that meet ALL conditions
+- conditions:
+    # capacity condition matches the volumes whose capacity falls into the range
+    capacity: "0,100Gi"
+    csi:
+      driver: aws.ebs.csi.driver
+      fsType: ext4
+    storageClass:
+    - gp2
+    - ebs-sc
+  action:
+    type: volume-snapshot
+    parameters:
+      # optional parameters which are custom-defined parameters when doing an action
+      volume-snapshot-timeout: "6h"
+- conditions:
+    capacity: "0,100Gi"
+    storageClass:
+    - gp2
+    - ebs-sc
+  action:
+    type: file-system-backup
+- conditions:
+    nfs:
+      server: 192.168.200.90
+  action:
+    # type of file-system-backup could be defined a second time
+    type: file-system-backup
+- conditions:
+    nfs: {}
+  action:
+    type: skip
+- conditions:
+    csi:
+      driver: aws.efs.csi.driver
+  action:
+    type: skip
+```
+ 
+### Filter rules
+#### VolumePolicies
+The whole resource policies consist of groups of volume policies.
+
+For one specific volume policy which is a combination of one action and serval conditions. which means one action and serval conditions are the smallest unit of volume policy.
+
+Volume policies are a list and if the target volumes match the first policy, the latter will be ignored, which would reduce the complexity of matching volumes especially when there are multiple complex volumes policies.
+
+#### Action
+`Action` defined one action for a specific way of backup:
+  - if choosing `Kopia` or `Restic`, the action value would be `file-system-backup`.
+  - if choosing volume snapshot, the action value would be `volume-snapshot`.
+  - if choosing skip backup of volume, the action value would be `skip`, and it will skip backup of volume no matter is `file-system-backup` or `volume-snapshot`.
+
+The policies could be extended for later other ways of backup, which means it may have some other `Action` value that will be assigned in the future.
+
+Both `file-system-backup` `volume-snapshot`, and `skip` could be partially or fully configured in the YAML file. And configuration could take effect only for the related action.
+
+#### Conditions
+The conditions are serials of volume attributes, the matched Volumes should meet all the volume attributes in one conditions configuration.
+
+##### Supported conditions
+In Velero 1.11, we want to support the volume attributes listed below:
+- capacity: matching volumes have the capacity that falls within this `capacity` range.
+- storageClass: matching volumes those with specified `storageClass`, such as `gp2`, `ebs-sc` in eks.
+- matching volumes that used specified volume sources.
+##### Parameters
+Parameters are optional for one specific action. For example, it could be `csi-snapshot-timeout: 6h` for CSI snapshot.
+
+#### Special rule definitions:
+- One single condition in `Conditions` with a specific key and empty value, which means the value matches any value. For example, if the `conditions.nfs` is `{}`, it means if `NFS` is used as `persistentVolumeSource` in Persistent Volume will be skipped no matter what the NFS server or NFS Path is.
+ 
+- The size of each single filter value should limit to 256 bytes in case of an unfriendly long variable assignment.
+ 
+- For capacity for PV or size for Volume, the value should include the lower value and upper value concatenated by commas. And it has several combinations below:
+  - "0,5Gi" or "0Gi,5Gi" which means capacity or size matches from 0 to 5Gi, including value 0 and value 5Gi
+  - ",5Gi" which is equal to "0,5Gi"
+  - "5Gi," which means capacity or size matches larger than 5Gi, including value 5Gi
+  - "5Gi" which is not supported and will be failed in validating configuration.
+ 
+### Configmap Reference
+Currently, resources policies are defined in `BackupSpec` struct, it will be more and more bloated with adding more and more filters which makes the size of `Backup` CR bigger and bigger, so we want to store the resources policies in configmap, and `Backup` CRD reference to current configmap.
+
+the `configmap` user created would be like this:
+```yaml
+apiVersion: v1
+data:
+ policies.yaml:
+  ----
+  version: v1
+  volumePolicies:
+  - conditions:
+      capacity: "0,100Gi"
+      csi:
+        driver: aws.ebs.csi.driver
+        fsType: ext4
+      storageClass:
+      - gp2
+      - ebs-sc
+    action:
+      type: volume-snapshot
+      parameters:
+        volume-snapshot-timeout: "6h"
+kind: ConfigMap
+metadata:
+ creationTimestamp: "2023-01-16T14:08:12Z"
+ name: backup01
+ namespace: velero
+ resourceVersion: "17891025"
+ uid: b73e7f76-fc9e-4e72-8e2e-79db717fe9f1
+```
+
+A new variable `resourcePolices` would be added into `BackupSpec`, it's value is assigned with the current resources policy configmap
+```yaml
+apiVersion: velero.io/v1
+kind: Backup
+metadata:
+  name: backup-1
+  spec:
+    resourcePolices:
+      refType: Configmap
+      ref: backup01
+    ...
+```
+The configmap only stores those assigned values, not the whole resources policies.
+ 
+The name of the configmap is `$BackupName`, and it's in Velero install namespace.
+ 
+#### Resource policies configmap related
+The life cycle of resource policies configmap is managed by the user instead of Velero, which could make it more flexible and easy to maintain.
+- The resource policies configmap will remain in the cluster until the user deletes it.
+- Unlike backup, the resource policies configmap will not sync to the new cluster. So if the user wants to use one resource policies that do not sync to the new cluster, the backup will fail with resource policies not found.
+- One resource policies configmap could be used by multiple backups.
+- If the backup referenced resource policies configmap is been deleted, it won't affect the already existing backups, but if the user wants to reference the deleted configmap to create one new backup, it will fail with resource policies not found.
+
+#### Versioning
+We want to introduce the version field in the YAML data to contain break changes. Therefore, we won't follow a semver paradigm, for example in v1.11 the data look like this:
+```yaml
+version: v1
+volumePolicies:
+  ....
+```
+Hypothetically, in v1.12 we add new fields like clusterResourcePolicies, the version will remain as v1 b/c this change is backward compatible:
+```yaml
+version: v1
+volumePolicies:
+  ....
+clusterResourcePolicies:
+  ....
+```
+Suppose in v1.13, we have to introduce a break change, at this time we will bump up the version:
+```yaml
+version: v2
+# This is just an example, we should try to avoid break change
+volume-policies: 
+  ....
+```
+We only support one version in Velero, so it won't be recognized if backup using a former version of YAML data.
+
+#### Multiple versions supporting
+To manage the effort for maintenance, we will only support one version of the data in Velero. Suppose that there is one break change for the YAML data in Velero v1.13, we should bump up the config version to v2, and v2 is only supported in v1.13. For the existing data with version: v1, it should migrate them when the Velero startup, this won't hurt the existing backup schedule CR as it only references the configmap. To make the migration easier, the configmap for such resource filter policies should be labeled manually before Velero startup like this, Velero will migrate the labeled configmap.
+
+We only support migrating from the previous version to the current version in case of complexity in data format conversion, which users could regenerate configmap in the new YAML data version, and it is easier to do version control.
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  labels:
+# This label can be optional but if this is not set, the backup will fail after the breaking change and the user will need to update the data manually 
+    velero.io/resource-filter-policies: "true"
+  name: example
+  namespace: velero
+data:
+  .....
+```
+### Display of resources policies
+As the resource policies configmap is referenced by backup CR, the policies in configmap are not so intuitive, so we need to integrate policies in configmap to the output of the command `velero backup describe`, and make it more readable.
+ 
+## Compatibility
+Currently, we have these resources filters:
+- IncludedNamespaces
+- ExcludedNamespaces
+- IncludedResources
+- ExcludedResources
+- LabelSelector
+- OrLabelSelectors
+- IncludeClusterResources
+- UseVolumeSnapshots
+- velero.io/exclude-from-backup=true
+- backup.velero.io/backup-volumes-excludes
+- backup.velero.io/backup-volumes
+- backup.velero.io/must-include-additional-items
+ 
+So it should be careful with the combination of volumes resources policies and the above resources filters.
+- When volume resource policies conflict with the above resource filters, we should respect the above resource filters. For example, if the user used the opt-out approach to `backup.velero.io/backup-volumes-excludes` annotation on the pod and also defined include volume in volumes resources filters configuration, we should respect the opt-out approach to skip backup of the volume.
+- If volume resource policies conflict with themselves, the first matched policy will be respect.
+
+## Implementation
+This implementation should be included in Velero v1.11.0
+
+Currently, in Velero v1.11.0 we only support `Action`
+ `skip`, and support `file-system-backup` and `volume-snapshot` for the later version. And `Parameters` in `Action` is also not supported in v1.11.0, we will support in a later version.
+
+In Velero 1.11, we supported Conditions and format listed below:
+ - capacity
+    ```yaml
+    capacity: "10Gi,100Gi" // match volume has the size between 10Gi and 100Gi
+    ```
+ - storageClass
+    ```yaml
+    storageClass: // match volume has the storage class gp2 or ebs-sc
+     - gp2
+     - ebs-sc
+    ```
+- volume sources (currently only support below format and attributes)
+  1. Specify the volume source name, the name could be `nfs`, `rbd`, `iscsi`, `csi` etc.
+     ```yaml
+     nfs : {} // match any volume has nfs volume source
+     
+     csi : {} // match any volume has csi volume source
+     ```
+
+  2. Specify details for the related volume source (currently we only support csi driver filter and nfs server or path filter)
+     ```yaml
+     csi: // match volume has nfs volume source and using `aws.efs.csi.driver`
+       driver: aws.efs.csi.driver 
+     
+     nfs: // match volume has nfs volume source and using below server and path
+       server: 192.168.200.90
+       path: /mnt/nfs
+     ```
+The conditions also could be extended in later versions, such as we could further supporting filtering other volume source detail not only NFS and CSI.
+
+## Alternatives Considered
+### Configmap VS CRD
+Here we support the user define the YAML config file and storing the resources policies into configmap, also we could define one resource's policies CRD and store policies imported from the user-defined config file in the related CR.
+
+But CRD is more like one kind of resource with status, Kubernetes API Server handles the lifecycle of a CR and handles it in different statuses. Compared to CRD, Configmap is more focused to store data.
+
+## Open Issues
+Should we support more than one version of filter policies configmap?

design/Implemented/json-substitution-action-design.md

@@ -0,0 +1,161 @@
+# Proposal to add support for Resource Modifiers (AKA JSON Substitutions) in Restore Workflow
+
+- [Proposal to add support for Resource Modifiers (AKA JSON Substitutions) in Restore Workflow](#proposal-to-add-support-for-resource-modifiers-aka-json-substitutions-in-restore-workflow)
+    - [Abstract](#abstract)
+    - [Goals](#goals)
+    - [Non Goals](#non-goals)
+    - [User Stories](#user-stories)
+        - [Scenario 1](#scenario-1)
+        - [Scenario 2](#scenario-2)
+    - [Detailed Design](#detailed-design)
+        - [Reference in velero API](#reference-in-velero-api)
+        - [ConfigMap Structure](#configmap-structure)
+        - [Operations supported by the JSON Patch library:](#operations-supported-by-the-json-patch-library)
+        - [Advance scenarios](#advance-scenarios)
+            - [Conditional patches using test operation](#conditional-patches-using-test-operation)
+    - [Alternatives Considered](#alternatives-considered)
+    - [Security Considerations](#security-considerations)
+    - [Compatibility](#compatibility)
+    - [Implementation](#implementation)
+    - [Future Enhancements](#future-enhancements)
+    - [Open Issues](#open-issues)
+
+## Abstract
+Currently velero supports substituting certain values in the K8s resources during restoration like changing the namespace, changing the storage class, etc. This proposal is to add generic support for JSON substitutions in the restore workflow. This will allow the user specify filters for particular resources and then specify a JSON patch (operator, path, value) to apply on a resource. This will allow the user to substitute any value in the K8s resource without having to write a new RestoreItemAction plugin for each kind of substitution.
+
+<!-- ## Background -->
+
+## Goals
+- Allow the user to specify a GroupResource, Name(optional), JSON patch for modification.
+- Allow the user to specify multiple JSON patch.
+
+## Non Goals
+- Deprecating the existing RestoreItemAction plugins for standard substitutions(like changing the namespace, changing the storage class, etc.)
+
+## User Stories
+
+### Scenario 1
+- Alice has a PVC which is encrypted using a DES(Disk Encryption Set - Azure example) mentioned in the PVC YAML through the StorageClass YAML. 
+- Alice wishes to restore this snapshot to a different cluster. The new cluster does not have access to the same DES to provision disk's out of the snapshot.
+- She wishes to use a different DES for all the PVCs which use the certain DES.
+- She can use this feature to substitute the DES in all StorageClass YAMLs with the new DES without having to create a fresh storageclass, or understanding the name of the storageclass.
+
+### Scenario 2
+- Bob has multi zone cluster where nodes are spread across zones. 
+- Bob has pinned certain pods to a particular zone using nodeSelector/ nodeaffinity on the pod spec.
+- In case of zone outage of the cloudprovider, Bob wishes to restore the workload to a different namespace in the same cluster, but change the zone pinning of the workload.
+- Bob can use this feature to substitute the nodeSelector/ nodeaffinity in the pod spec with the new zone pinning to quickly failover the workload to a different zone's nodes.
+
+## Detailed Design
+- The design and approach is inspired from [kubectl patch command](https://github.com/kubernetes/kubectl/blob/0a61782351a027411b8b45b1443ec3dceddef421/pkg/cmd/patch/patch.go#L102C2-L104C1)
+```bash
+# Update a container's image using a json patch with positional arrays
+kubectl patch pod valid-pod -type='json' -p='[{"op": "replace", "path": "/spec/containers/0/image", "value":"new image"}]'
+```
+- The user is expected to create a configmap with the desired Resource Modifications. Then the reference of the configmap will be provided in the RestoreSpec.
+- The core restore workflow before creating/updating a particular resource in the cluster will be checked against the filters provided and respective substitutions will be applied on it.
+
+### Reference in velero API 
+> Example of Reference to configmap in RestoreSpec
+```yaml
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: restore-1
+spec:
+    resourceModifier:
+      refType: Configmap
+      ref: resourcemodifierconfigmap
+```
+> Example CLI Command
+```bash
+velero restore create --from-backup backup-1 --resource-modifier-configmap resourcemodifierconfigmap
+```
+
+### Resource Modifier ConfigMap Structure
+- User first needs to provide details on which resources the JSON Substitutions need to be applied. 
+    - For this the user will provide 4 inputs - Namespaces(for NS Scoped resources), GroupResource (resource.group format similar to includeResources field in velero) and Name Regex(optional).
+    - If the user does not provide the Name, the JSON Substitutions will be applied to all the resources of the given Group and Kind under the given namespaces.
+
+- Further the use will specify the JSON Patch using the structure of kubectl's "JSON Patch" based inputs.
+- Sample data in ConfigMap
+```yaml
+version: v1
+resourceModifierRules:
+- conditions:
+    groupResource: persistentvolumeclaims
+    resourceNameRegex: "mysql.*"
+    namespaces:
+    - bar
+    - foo
+  patches:
+  - operation: replace
+    path: "/spec/storageClassName"
+    value: "premium"
+  - operation: remove
+    path: "/metadata/labels/test"
+```
+- The above configmap will apply the JSON Patch to all the PVCs in the namespaces bar and foo with name starting with mysql. The JSON Patch will replace the storageClassName with "premium" and remove the label "test" from the PVCs.
+- Note that the Namespace here is the original namespace of the backed up resource, not the new namespace where the resource is going to be restored.
+- The user can specify multiple JSON Patches for a particular resource. The patches will be applied in the order specified in the configmap. A subsequent patch is applied in order and if multiple patches are specified for the same path, the last patch will override the previous patches.
+- The user can specify multiple resourceModifierRules in the configmap. The rules will be applied in the order specified in the configmap. 
+
+> Users need to create one configmap in Velero install namespace from a YAML file that defined resource modifiers. The creating command would be like the below:
+```bash
+kubectl create cm <configmap-name> --from-file <yaml-file> -n velero
+```
+
+### Operations supported by the JSON Patch library: 
+- add
+- remove
+- replace
+- move
+- copy
+- test (covered below)
+
+### Advance scenarios
+#### **Conditional patches using test operation**
+ The `test` operation can be used to check if a particular value is present in the resource. If the value is present, the patch will be applied. If the value is not present, the patch will not be applied. This can be used to apply a patch only if a particular value is present in the resource. For example, if the user wishes to change the storage class of a PVC only if the PVC is using a particular storage class, the user can use the following configmap.
+```yaml
+version: v1
+resourceModifierRules:
+- conditions:
+    groupResource: persistentvolumeclaims.storage.k8s.io
+    resourceNameRegex: ".*"
+    namespaces:
+    - bar
+    - foo
+  patches:
+  - operation: test
+    path: "/spec/storageClassName"
+    value: "premium"
+  - operation: replace
+    path: "/spec/storageClassName"
+    value: "standard"
+```
+
+## Alternatives Considered
+1. JSON Path based addressal of json fields in the resource
+    - This was the initial planned approach, but there is no open source library which gives satisfactory edit functionality with support for all operators supported by the JsonPath RFC.
+    - We attempted modifying the  [https://kubernetes.io/docs/reference/kubectl/jsonpath/](https://kubernetes.io/docs/reference/kubectl/jsonpath/) but given the complexity of the code it did not make sense to change it since it would become a long term maintainability problem.
+1. RestoreItemAction for each kind of standard substitution
+    - Not an extensible design. If a new kind of substitution is required, a new RestoreItemAction needs to be written.
+1. RIA for JSON Substitution: The approach of doing JSON Substitution through a RestoreItemAction plugin was considered. But it is likely to have performance implications as the plugin will be invoked for all the resources.
+
+## Security Considerations
+No security impact.
+
+## Compatibility
+Compatibility with existing StorageClass mapping RestoreItemAction and similar plugins needs to be evaluated. 
+
+## Implementation
+- Changes in Restore CRD. Add a new field to the RestoreSpec to reference the configmap.
+- One example of where code will be modified:  https://github.com/vmware-tanzu/velero/blob/eeee4e06d209df7f08bfabda326b27aaf0054759/pkg/restore/restore.go#L1266 On the obj before Creation, we can apply the conditions to check if the resource is filtered out using given parameters. Then using JsonPatch provided, we can update the resource.
+- For Jsonpatch - https://github.com/evanphx/json-patch library is used.
+- JSON Patch RFC https://datatracker.ietf.org/doc/html/rfc6902
+
+## Future enhancements
+- Additional features such as wildcard support in path, regex match support in value, etc. can be added in future. This would involve forking the https://github.com/evanphx/json-patch library and adding the required features, since those features are not supported by the library currently and are not part of jsonpatch RFC.
+
+## Open Issues
+NA

design/Implemented/merge-patch-and-strategic-in-resource-modifier.md

@@ -0,0 +1,193 @@
+# Proposal to Support JSON Merge Patch and Strategic Merge Patch in Resource Modifiers
+
+- [Proposal to Support JSON Merge Patch and Strategic Merge Patch in Resource Modifiers](#proposal-to-support-json-merge-patch-and-strategic-merge-patch-in-resource-modifiers)
+    - [Abstract](#abstract)
+    - [Goals](#goals)
+    - [Non Goals](#non-goals)
+    - [User Stories](#user-stories)
+        - [Scenario 1](#scenario-1)
+        - [Scenario 2](#scenario-2)
+    - [Detailed Design](#detailed-design)
+        - [How to choose the right patch type](#how-to-choose-the-right-patch-type)
+        - [New Field MergePatches](#new-field-mergepatches)
+        - [New Field StrategicPatches](#new-field-strategicpatches)
+        - [Conditional Patches in ALL Patch Types](#conditional-patches-in-all-patch-types)
+        - [Wildcard Support for GroupResource](#wildcard-support-for-groupresource)
+        - [Helper Command to Generate Merge Patch and Strategic Merge Patch](#helper-command-to-generate-merge-patch-and-strategic-merge-patch)
+    - [Security Considerations](#security-considerations)
+    - [Compatibility](#compatibility)
+    - [Implementation](#implementation)
+    - [Future Enhancements](#future-enhancements)
+    - [Open Issues](#open-issues)
+
+## Abstract
+Velero introduced the concept of Resource Modifiers in v1.12.0. This feature allows the user to specify a configmap with a set of rules to modify the resources during restore. The user can specify the filters to select the resources and then specify the JSON Patch to apply on the resource. This feature is currently limited to the operations supported by JSON Patch RFC.
+This proposal is to add support for JSON Merge Patch and Strategic Merge Patch in the Resource Modifiers. This will allow the user to use the same configmap to apply JSON Merge Patch and Strategic Merge Patch on the resources during restore.
+
+## Goals
+- Allow the user to specify a JSON patch, JSON Merge Patch or Strategic Merge Patch for modification.
+- Allow the user to specify multiple JSON Patch, JSON Merge Patch or Strategic Merge Patch.
+- Allow the user to specify mixed JSON Patch, JSON Merge Patch and Strategic Merge Patch in the same configmap.
+
+## Non Goals
+- Deprecating the existing RestoreItemAction plugins for standard substitutions(like changing the namespace, changing the storage class, etc.)
+
+## User Stories
+
+### Scenario 1
+- Alice has some Pods and part of them have an annotation `{"for": "bar"}`.
+- Alice wishes to restore these Pods to a different cluster without this annotation.
+- Alice can use this feature to remove this annotation during restore.
+
+### Scenario 2
+- Bob has a Pod with several containers and one container with name nginx has an image `repo1/nginx`.
+- Bob wishes to restore this Pod to a different cluster, but new cluster can not access repo1, so he pushes the image to repo2.
+- Bob can use this feature to update the image of container nginx to `repo2/nginx` during restore.
+
+## Detailed Design
+- The design and approach is inspired by kubectl patch command and [this doc](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/).
+- New fields `MergePatches` and `StrategicPatches` will be added to the `ResourceModifierRule` struct to support all three patch types.
+- Only one of the three patch types can be specified in a single `ResourceModifierRule`.
+- Add wildcard support for `groupResource` in `conditions` struct.
+- The workflow to create Resource Modifier ConfigMap and reference it in RestoreSpec will remain the same as described in document [Resource Modifiers](https://github.com/vmware-tanzu/velero/blob/main/site/content/docs/main/restore-resource-modifiers.md).
+
+### How to choose the right patch type
+- [JSON Merge Patch](https://datatracker.ietf.org/doc/html/rfc7386) is a naively simple format, with limited usability. Probably it is a good choice if you are building something small, with very simple JSON Schema.
+- [JSON Patch](https://datatracker.ietf.org/doc/html/rfc6902) is a more complex format, but it is applicable to any JSON documents. For a comparison of JSON patch and JSON merge patch, see [JSON Patch and JSON Merge Patch](https://erosb.github.io/post/json-patch-vs-merge-patch/).
+- Strategic Merge Patch is a Kubernetes defined patch type, mainly used to process resources of type list. You can replace/merge a list, add/remove items from a list by key, change the order of items in a list, etc. Strategic merge patch is not supported for custom resources. For more details, see [this doc](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/).
+
+### New Field MergePatches
+MergePatches is a list to specify the merge patches to be applied on the resource. The merge patches will be applied in the order specified in the configmap. A subsequent patch is applied in order and if multiple patches are specified for the same path, the last patch will override the previous patches.
+
+Example of MergePatches in ResourceModifierRule
+```yaml
+version: v1
+resourceModifierRules:
+- conditions:
+    groupResource: pods
+    namespaces:
+    - ns1
+  mergePatches:
+  - patchData: |
+      {
+        "metadata": {
+          "annotations": {
+            "foo": null
+          }
+        }
+      }
+```
+- The above configmap will apply the Merge Patch to all the pods in namespace ns1 and remove the annotation `foo` from the pods.
+- Both json and yaml format are supported for the patchData.
+
+### New Field StrategicPatches
+StrategicPatches is a list to specify the strategic merge patches to be applied on the resource. The strategic merge patches will be applied in the order specified in the configmap. A subsequent patch is applied in order and if multiple patches are specified for the same path, the last patch will override the previous patches.
+
+Example of StrategicPatches in ResourceModifierRule
+```yaml
+version: v1
+resourceModifierRules:
+- conditions:
+    groupResource: pods
+    resourceNameRegex: "^my-pod$"
+    namespaces:
+    - ns1
+  strategicPatches:
+  - patchData: |
+      {
+        "spec": {
+          "containers": [
+            {
+              "name": "nginx",
+              "image": "repo2/nginx"
+            }
+          ]
+        }
+      }
+```
+- The above configmap will apply the Strategic Merge Patch to the pod with name my-pod in namespace ns1 and update the image of container nginx to `repo2/nginx`.
+- Both json and yaml format are supported for the patchData.
+
+### Conditional Patches in ALL Patch Types
+Since JSON Merge Patch and Strategic Merge Patch do not support conditional patches, we will use the `test` operation of JSON Patch to support conditional patches in all patch types by adding it to `Conditions` struct in `ResourceModifierRule`.
+
+Example of test in conditions
+```yaml
+version: v1
+resourceModifierRules:
+- conditions:
+    groupResource: persistentvolumeclaims.storage.k8s.io
+    matches:
+    - path: "/spec/storageClassName"
+      value: "premium"
+  mergePatches:
+  - patchData: |
+      {
+        "metadata": {
+          "annotations": {
+            "foo": null
+          }
+        }
+      }
+```
+- The above configmap will apply the Merge Patch to all the PVCs in all namespaces with storageClassName premium and remove the annotation `foo` from the PVCs.
+- You can specify multiple rules in the `matches` list. The patch will be applied only if all the matches are satisfied.
+
+### Wildcard Support for GroupResource
+The user can specify a wildcard for groupResource in the conditions' struct. This will allow the user to apply the patches for all the resources of a particular group or all resources in all groups. For example, `*.apps` will apply to all the resources in the `apps` group, `*` will apply to all the resources in all groups.
+
+### Helper Command to Generate Merge Patch and Strategic Merge Patch
+The patchData of Strategic Merge Patch is sometimes a bit complex for user to write. We can provide a helper command to generate the patchData for Strategic Merge Patch. The command will take the original resource and the modified resource as input and generate the patchData.
+It can also be used in JSON Merge Patch.
+
+Here is a sample code snippet to achieve this:
+```go
+package main
+
+import (
+	"fmt"
+
+	corev1 "k8s.io/api/core/v1"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+)
+
+func main() {
+	pod := &corev1.Pod{
+		Spec: corev1.PodSpec{
+			Containers: []corev1.Container{
+				{
+					Name:  "web",
+					Image: "nginx",
+				},
+			},
+		},
+	}
+	newPod := pod.DeepCopy()
+	patch := client.StrategicMergeFrom(pod)
+	newPod.Spec.Containers[0].Image = "nginx1"
+
+	data, _ := patch.Data(newPod)
+	fmt.Println(string(data))
+	// Output:
+	// {"spec":{"$setElementOrder/containers":[{"name":"web"}],"containers":[{"image":"nginx1","name":"web"}]}}
+}
+```
+
+## Security Considerations
+No security impact.
+
+## Compatibility
+Compatible with current Resource Modifiers.
+
+## Implementation
+- Use "github.com/evanphx/json-patch" to support JSON Merge Patch.
+- Use "k8s.io/apimachinery/pkg/util/strategicpatch" to support Strategic Merge Patch.
+- Use glob to support wildcard for `groupResource` in `conditions` struct.
+- Use `test` operation of JSON Patch to calculate the `matches` in `conditions` struct.
+
+## Future enhancements
+- add a Velero subcommand to generate/validate the patchData for Strategic Merge Patch and JSON Merge Patch.
+- add jq support for more complex conditions or patches, to meet the situations that the current conditions or patches can not handle. like [this issue](https://github.com/vmware-tanzu/velero/issues/6344)
+
+## Open Issues
+N/A

design/Implemented/move-gh-org.md

@@ -65,7 +65,7 @@ This page contains a pre-migration checklist for ensuring a repo migration goes
 
 #### Updating Netlify
 
-The settings for Netflify should remain the same, except that it now needs to be installed in the new repo. The instructions on how to install Netlify on the new repo are here: https://www.netlify.com/docs/github-permissions/.
+The settings for Netlify should remain the same, except that it now needs to be installed in the new repo. The instructions on how to install Netlify on the new repo are here: https://www.netlify.com/docs/github-permissions/.
 
 #### Communication strategy
 

design/Implemented/multiple-csi-volumesnapshotclass-support.md

@@ -0,0 +1,177 @@
+# Proposal to add support for Multiple VolumeSnapshotClasses in CSI Plugin
+
+- [Proposal to add support for Multiple VolumeSnapshotClasses in CSI Plugin](#proposal-to-add-support-for-multiple-volumesnapshotclasses-in-csi-plugin)
+	- [Abstract](#abstract)
+	- [Background](#background)
+	- [Goals](#goals)
+	- [Non Goals](#non-goals)
+    - [User Stories](#user-stories)
+        - [Scenario 1](#scenario-1)
+        - [Scenario 2](#scenario-2)
+    - [Detailed Design](#detailed-design)
+        - [Plugin Inputs Contract Changes](#plugin-inputs-contract-changes)
+        - [Using Plugin Inputs for CSI Plugin](#using-plugin-inputs-for-csi-plugin)
+        - [Annotations overrides on PVC for CSI Plugin](#annotations-overrides-on-pvc-for-csi-plugin)
+        - [Using Plugin Inputs for Other Plugins](#using-plugin-inputs-for-other-plugins)
+    - [Alternatives Considered](#alternatives-considered)
+    - [Security Considerations](#security-considerations)
+    - [Compatibility](#compatibility)
+    - [Implementation](#implementation)
+    - [Open Issues](#open-issues)
+
+
+## Abstract
+Currently the Velero CSI plugin chooses the VolumeSnapshotClass in the cluster that has the same driver name and also has the velero.io/csi-volumesnapshot-class label set on it. This global selection is not sufficient for many use cases. This proposal is to add support for multiple VolumeSnapshotClasses in CSI Plugin where the user can specify the VolumeSnapshotClass to use for a particular driver and backup.
+
+
+## Background
+The Velero CSI plugin chooses the VolumeSnapshotClass in the cluster that has the same driver name and also has the velero.io/csi-volumesnapshot-class label set on it. This global selection is not sufficient for many use cases. For example, if a cluster has multiple VolumeSnapshotClasses for the same driver, the user may want to use a VolumeSnapshotClass that is different from the default one. The user might also have different schedules set up for backing up different parts of the cluster and might wish to use different VolumeSnapshotClasses for each of these backups. 
+
+## Goals
+- Allow the user to specify the VolumeSnapshotClass to use for a particular driver and backup.
+
+## Non Goals
+- Deprecating existing VSC selection behaviour. (The current behaviour will remain the default behaviour if the user does not specify the VolumeSnapshotClass to use for a particular driver and backup.)
+
+
+## User Stories
+
+### Scenario 1
+- Consider Alice is a cluster admin and has a cluster with multiple VolumeSnapshotClasses for the same driver. Each VSC stores the snapshots taken in different ResourceGroup(Azure equivalent). 
+- Alice has configured multiple scheduled backups each covering a different set of namespaces, representing different apps owned by different teams. 
+- Alice wants to use a different VolumeSnapshotClass for each backup such that each snapshot goes in it's respective ResourceGroup to simply management of snapshots(COGS, RBAC etc).
+- In current velero, Alice can't achieve this as the CSI plugin will use the default VolumeSnapshotClass for the driver and all snapshots will go in the same ResourceGroup.
+- Proposed design will allow Alice to achieve this by specifying the VolumeSnapshotClass to use for a particular driver and backup/schedule.
+
+## Scenario 2
+- Bob is a cluster admin has PVCs storing different types of data. 
+- Most of the PVCs are used for storing non-sensitive application data. But certain PVCs store critical financial data.
+- For such PVCs Bob wants to use a VolumeSnapshotClass with certain encryption related parameters set. 
+- In current velero, Bob can't achieve this as the CSI plugin will use the default VolumeSnapshotClass for the driver and all snapshots will be taken using the same VolumeSnapshotClass.
+- Proposed design will allow Bob to achieve this by overriding the VolumeSnapshotClass to use for a particular driver and backup/schedule using annotations on those specific PVCs.
+
+
+## Detailed Design
+
+### Staged Approach: 
+
+### Stage 1 Approach
+#### Through Annotations
+    1. **Support VolumeSnapshotClass selection at backup/schedule level**
+    The user can  annotate the backup/ schedule with driver and VolumeSnapshotClass name. The CSI plugin will use the VolumeSnapshotClass specified in the annotation. If the annotation is not present, the CSI plugin will use the default VolumeSnapshotClass for the driver.
+
+        *example annotation on backup/schedule:*
+        ```yaml
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+        name: backup-1
+        annotations:
+            velero.io/csi-volumesnapshot-class_csi.cloud.disk.driver: csi-diskdriver-snapclass
+            velero.io/csi-volumesnapshot-class_csi.cloud.file.driver: csi-filedriver-snapclass
+            velero.io/csi-volumesnapshot-class_<driver name>: csi-snapclass
+        ```
+
+         To query the annotations on a backup: "velero.io/csi-volumesnapshot-class_'driver name'" - where driver names comes from the PVC's driver.
+
+    2. **Support VolumeSnapshotClass selection at PVC level**
+    The user can annotate the PVCs with driver and VolumeSnapshotClass name. The CSI plugin will use the VolumeSnapshotClass specified in the annotation. If the annotation is not present, the CSI plugin will use the default VolumeSnapshotClass for the driver. If the VolumeSnapshotClass provided is of a different driver, the CSI plugin will use the default VolumeSnapshotClass for the driver.
+
+        *example annotation on PVC:*
+        ```yaml
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+        name: pvc-1
+        annotations:
+            velero.io/csi-volumesnapshot-class: csi-diskdriver-snapclass
+            
+        ```
+    
+    Consider this as a override option in conjunction to part 1.
+
+**Note**: The user has to annotate the PVCs or backups with the VolumeSnapshotClass to use for each driver. This is not ideal for the user experience.
+    - **Mitigation**: We can extend Velero CLI to also annotate backups/schedules with the VolumeSnapshotClass to use for each driver. This will make it easier for the user to annotate the backups/schedules. This mitigation is not for the PVCs though, since PVCs is anyways a specific use case. Similar to : " kubectl run --image myimage --annotations="foo=bar" --annotations="another=one" mypod"
+    We can add support for  - velero backup create my-backup --annotations "velero.io/csi:csi.cloud.disk.driver=csi-diskdriver-snapclass"
+
+### Stage 2 Approach
+The above annotations route is to get started and for initial design closure/ implementation, north star is to either introduce CSI specific fields (considering that CSI might be a very core part of velero going forward) in the backup/restore CR OR leverage the pluginInputs field as being tracked in: https://github.com/vmware-tanzu/velero/pull/5981
+
+Refer section Alternatives 2. **Through generic property bag in the velero contracts**: in the design doc for more details on the pluginInputs field.
+
+
+## Alternatives Considered
+1. **Through CSI Specific Fields in Velero contracts**
+
+    **Considerations**
+    - Since CSI snapshotting is done through the plugin, we don't intend to bloat up the Backup Spec with CSI specific fields. 
+    - But considering that CSI Snapshotting is the way forward, we can debate if we should add a CSI section to the Backup Spec.
+
+
+    **Approach**: Similar to VolumeSnapshotLocation param in the Backup Spec, we can add a VolumeSnapshotClass param in the Backup Spec. This will allow the user to specify the VolumeSnapshotClass to use for the backup. The CSI plugin will use the VolumeSnapshotClass specified in the Backup Spec. If the VolumeSnapshotClass is not specified, the CSI plugin will use the default VolumeSnapshotClass for the driver.
+
+    *example of VolumeSnapshotClass param in the Backup Spec:*
+    ```yaml
+    apiVersion: velero.io/v1
+    kind: Backup
+    metadata:
+    name: backup-1
+    spec:
+        csiParameters: 
+            volumeSnapshotClasses: 
+                driver: csi.cloud.disk.driver
+                snapClass: csi-diskdriver-snapclass
+            timeout: 10m
+    ```
+
+1. **Through changes in velero contracts**
+    1. **Through configmap references.**
+    Currently even the storageclass mapping plugin expects the user to create a configmap which is used globally, and fetched through labels. This behaviour has same issue as the VolumeSnapshotClass selection. We can introduce a field in the velero contracts which allow passing configmap references for each plugin. And then the plugin can honour the configmap passed in as reference. The configmap can be used to pass the VolumeSnapshotClass to use for the backup, and also other parameters to tweak. This can help in making plugins more flexible while not depending on global behaviour.
+    
+
+        *example of configmap reference in the velero contracts:*
+        ```yaml
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+        name: backup-1
+        spec:
+            configmapRefs:
+            - name: csi-volumesnapshotclass-configmap
+            - namespace: velero
+            - plugin: velero.io/csi
+         ```
+
+    2. **Through generic property bag in the velero contracts**: We can introduce a field in the velero contracts which allow passing a generic property bag for each plugin. And then the plugin can honour the property bag passed in.
+
+
+        *example of property bag in the velero contracts:*
+        ```yaml
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+        name: backup-1
+        spec:
+            pluginInputs:
+            - name: velero.io/csi
+            - properties:
+                - key: csi.cloud.disk.driver
+                - value: csi-diskdriver-snapclass
+                - key: csi.cloud.file.driver
+                - value: csi-filedriver-snapclass
+        ```
+
+    **Note**: Both these approaches can also be used to tweak other parameters such as CSI Snapshotting Timeout/intervals. And further can be used by other plugins.
+
+
+## Security Considerations
+No security impact.
+
+## Compatibility
+Existing behaviour of csi plugin will be retained where it fetches the VolumeSnapshotClass through the label. This will be the default behaviour if the user does not specify the VolumeSnapshotClass.
+
+## Implementation
+TBD based on closure of high level design proposals.
+
+## Open Issues
+NA

design/Implemented/multiple-label-selectors_design.md

@@ -3,7 +3,7 @@
 As of today Velero supports filtering of resources based on single label selector per backup. It is desired that Velero
 support backing up of resources based on multiple labels (OR logic).
 
-**Note:** This solution is required because kubernetes label selectors only allow AND logic of labels.
+**Note:** This solution is required because Kubernetes label selectors only allow AND logic of labels.
 
 ## Background
 Currently, Velero's Backup/Restore API has a spec field `LabelSelector` which helps in filtering of resources based on

design/Implemented/node-agent-affinity.md

@@ -0,0 +1,132 @@
+# Node-agent Load Affinity Design
+
+## Glossary & Abbreviation
+
+**Velero Generic Data Path (VGDP)**: VGDP is the collective modules that is introduced in [Unified Repository design][1]. Velero uses these modules to finish data transfer for various purposes (i.e., PodVolume backup/restore, Volume Snapshot Data Movement). VGDP modules include uploaders and the backup repository.  
+
+**Exposer**: Exposer is a module that is introduced in [Volume Snapshot Data Movement Design][2]. Velero uses this module to expose the volume snapshots to Velero node-agent pods or node-agent associated pods so as to complete the data movement from the snapshots.    
+
+## Background
+
+Velero node-agent is a daemonset hosting controllers and VGDP modules to complete the concrete work of backups/restores, i.e., PodVolume backup/restore, Volume Snapshot Data Movement backup/restore.  
+Specifically, node-agent runs DataUpload controllers to watch DataUpload CRs for Volume Snapshot Data Movement backups, so there is one controller instance in each node. One controller instance takes a DataUpload CR and then launches a VGDP instance, which initializes a uploader instance and the backup repository connection, to finish the data transfer. The VGDP instance runs inside a node-agent pod or in a pod associated to the node-agent pod in the same node.  
+
+Varying from the data size, data complexity, resource availability, VGDP may take a long time and remarkable resources (CPU, memory, network bandwidth, etc.).  
+Technically, VGDP instances are able to run in any node that allows pod schedule. On the other hand, users may want to constrain the nodes where VGDP instances run for various reasons, below are some examples:  
+- Prevent VGDP instances from running in specific nodes because users have more critical workloads in the nodes  
+- Constrain VGDP instances to run in specific nodes because these nodes have more resources than others  
+- Constrain VGDP instances to run in specific nodes because the storage allows volume/snapshot provisions in these nodes only  
+
+Therefore, in order to improve the compatibility, it is worthy to configure the affinity of VGDP to nodes, especially for backups for which VGDP instances run frequently and centrally.  
+
+## Goals
+
+- Define the behaviors of node affinity of VGDP instances in node-agent for volume snapshot data movement backups  
+- Create a mechanism for users to specify the node affinity of VGDP instances for volume snapshot data movement backups  
+
+## Non-Goals
+- It is also beneficial to support VGDP instances affinity for PodVolume backup/restore, however, it is not possible since VGDP instances for PodVolume backup/restore should always run in the node where the source/target pods are created.  
+- It is also beneficial to support VGDP instances affinity for data movement restores, however, it is not possible in some cases. For example, when the `volumeBindingMode` in the StorageClass is `WaitForFirstConsumer`, the restore volume must be mounted in the node where the target pod is scheduled, so the VGDP instance must run in the same node. On the other hand, considering the fact that restores may not frequently and centrally run, we will not support data movement restores.  
+- As elaborated in the [Volume Snapshot Data Movement Design][2], the Exposer may take different ways to expose snapshots, i.e., through backup pods (this is the only way supported at present). The implementation section below only considers this approach currently, if a new expose method is introduced in future, the definition of the affinity configurations and behaviors should still work, but we may need a new implementation.  
+
+## Solution
+
+We will use the ConfigMap specified by `velero node-agent` CLI's parameter `--node-agent-configmap` to host the node affinity configurations.
+This configMap is not created by Velero, users should create it manually on demand. The configMap should be in the same namespace where Velero is installed. If multiple Velero instances are installed in different namespaces, there should be one configMap in each namespace which applies to node-agent in that namespace only.  
+Node-agent server checks these configurations at startup time and use it to initiate the related VGDP modules. Therefore, users could edit this configMap any time, but in order to make the changes effective, node-agent server needs to be restarted.  
+Inside the ConfigMap we will add one new kind of configuration as the data in the configMap, the name is ```loadAffinity```.  
+Users may want to set different LoadAffinity configurations according to different conditions (i.e., for different storages represented by StorageClass, CSI driver, etc.), so we define ```loadAffinity``` as an array. This is for extensibility consideration, at present, we don't implement multiple configurations support, so if there are multiple configurations, we always take the first one in the array.  
+
+The data structure is as below:
+```go
+type Configs struct {
+	// LoadConcurrency is the config for load concurrency per node.
+	LoadConcurrency *LoadConcurrency `json:"loadConcurrency,omitempty"`
+
+	// LoadAffinity is the config for data path load affinity.
+	LoadAffinity []*LoadAffinity `json:"loadAffinity,omitempty"`    
+}
+
+type LoadAffinity struct {
+    // NodeSelector specifies the label selector to match nodes
+    NodeSelector metav1.LabelSelector `json:"nodeSelector"`
+}
+```
+
+### Affinity
+Affinity configuration means allowing VGDP instances running in the nodes specified. There are two ways to define it:
+-  It could be defined by `MatchLabels` of `metav1.LabelSelector`. The labels defined in `MatchLabels` means a `LabelSelectorOpIn` operation by default, so in the current context, they will be treated as affinity rules.  
+- It could be defined by `MatchExpressions` of `metav1.LabelSelector`. The labels are defined in `Key` and `Values` of `MatchExpressions` and the `Operator` should be defined as `LabelSelectorOpIn` or `LabelSelectorOpExists`. 
+
+### Anti-affinity
+Anti-affinity configuration means preventing VGDP instances running in the nodes specified. Below is the way to define it:  
+- It could be defined by `MatchExpressions` of `metav1.LabelSelector`. The labels are defined in `Key` and `Values` of `MatchExpressions` and the `Operator` should be defined as `LabelSelectorOpNotIn` or `LabelSelectorOpDoesNotExist`.   
+
+### Sample
+A sample of the ConfigMap is as below:
+```json
+{
+    "loadAffinity": [
+        {
+            "nodeSelector": {
+                "matchLabels": {
+                    "beta.kubernetes.io/instance-type": "Standard_B4ms"
+                },
+                "matchExpressions": [
+                    {
+                        "key": "kubernetes.io/hostname",
+                        "values": [
+                            "node-1",
+                            "node-2",
+                            "node-3"
+                        ],
+                        "operator": "In"
+                    },
+                    {
+                        "key": "xxx/critial-workload",
+                        "operator": "DoesNotExist"
+                    }
+                ]          
+            }
+        }
+    ]
+}
+```
+This sample showcases two affinity configurations:
+- matchLabels: VGDP instances will run only in nodes with label key `beta.kubernetes.io/instance-type` and value `Standard_B4ms`
+- matchExpressions: VGDP instances will run in node `node1`, `node2` and `node3` (selected by `kubernetes.io/hostname` label)
+
+This sample showcases one anti-affinity configuration:
+- matchExpressions: VGDP instances will not run in nodes with label key `xxx/critial-workload`
+
+To create the configMap, users need to save something like the above sample to a json file and then run below command:
+```
+kubectl create cm <ConfigMap name> -n velero --from-file=<json file name>
+``` 
+
+### Implementation
+As mentioned in the [Volume Snapshot Data Movement Design][2], the exposer decides where to launch the VGDP instances. At present, for volume snapshot data movement backups, the exposer creates backupPods and the VGDP instances will be initiated in the nodes where backupPods are scheduled. So the loadAffinity will be translated (from `metav1.LabelSelector` to `corev1.Affinity`) and set to the backupPods.  
+
+It is possible that node-agent pods, as a daemonset, don't run in every worker nodes, users could fulfil this by specify `nodeSelector` or `nodeAffinity` to the node-agent daemonset spec. On the other hand, at present, VGDP instances must be assigned to nodes where node-agent pods are running. Therefore, if there is any node selection for node-agent pods, users must consider this into this load affinity configuration, so as to guarantee that VGDP instances are always assigned to nodes where node-agent pods are available. This is done by users, we don't inherit any node selection configuration from node-agent daemonset as we think daemonset scheduler works differently from plain pod scheduler, simply inheriting all the configurations may cause unexpected result of backupPod schedule.    
+Otherwise, if a backupPod are scheduled to a node where node-agent pod is absent, the corresponding DataUpload CR will stay in `Accepted` phase until the prepare timeout (by default 30min).  
+
+At present, as part of the expose operations, the exposer creates a volume, represented by backupPVC, from the snapshot. The backupPVC uses the same storageClass with the source volume. If the `volumeBindingMode` in the storageClass is `Immediate`, the volume is immediately allocated from the underlying storage without waiting for the backupPod. On the other hand, the loadAffinity is set to the backupPod's affinity. If the backupPod is scheduled to a node where the snapshot volume is not accessible, e.g., because of storage topologies, the backupPod won't get into Running state, concequently, the data movement won't complete.  
+Once this problem happens, the backupPod stays in `Pending` phase, and the corresponding DataUpload CR stays in `Accepted` phase until the prepare timeout (by default 30min). Below is an example of the backupPod's status when the problem happens:   
+```
+  status:
+    conditions:
+    - lastProbeTime: null
+      message: '0/2 nodes are available: 1 node(s) didn''t match Pod''s node affinity/selector,
+        1 node(s) had volume node affinity conflict. preemption: 0/2 nodes are available:
+        2 Preemption is not helpful for scheduling..'
+      reason: Unschedulable
+      status: "False"
+      type: PodScheduled
+    phase: Pending
+```    
+
+On the other hand, the backupPod is deleted after the prepare timeout, so there is no way to tell the cause is one of the above problems or others.  
+To help the troubleshooting, we can add some diagnostic mechanism to discover the status of the backupPod and node-agent in the same node before deleting it as a result of the prepare timeout.  
+
+[1]: Implemented/unified-repo-and-kopia-integration/unified-repo-and-kopia-integration.md
+[2]: volume-snapshot-data-movement/volume-snapshot-data-movement.md

design/Implemented/node-agent-concurrency.md

@@ -0,0 +1,131 @@
+# Node-agent Concurrency Design
+
+## Glossary & Abbreviation
+
+**Velero Generic Data Path (VGDP)**: VGDP is the collective of modules that is introduced in [Unified Repository design][1]. Velero uses these modules to finish data transfer for various purposes (i.e., PodVolume backup/restore, Volume Snapshot Data Movement). VGDP modules include uploaders and the backup repository.  
+
+## Background
+
+Velero node-agent is a daemonset hosting controllers and VGDP modules to complete the concrete work of backups/restores, i.e., PodVolume backup/restore, Volume Snapshot Data Movement backup/restore.  
+For example, node-agent runs DataUpload controllers to watch DataUpload CRs for Volume Snapshot Data Movement backups, so there is one controller instance in each node. One controller instance takes a DataUpload CR and then launches a VGDP instance, which initializes a uploader instance and the backup repository connection, to finish the data transfer. The VGDP instance runs inside the node-agent pod or in a pod associated to the node-agent pod in the same node.  
+
+Varying from the data size, data complexity, resource availability, VGDP may take a long time and remarkable resources (CPU, memory, network bandwidth, etc.).  
+Technically, VGDP instances are able to run in concurrent regardless of the requesters. For example, a VGDP instance for a PodVolume backup could run in parallel with another VGDP instance for a DataUpload. Then the two VGDP instances share the same resources if they are running in the same node.  
+
+Therefore, in order to gain the optimized performance with the limited resources, it is worthy to configure the concurrent number of VGDP per node. When the resources are sufficient in nodes, users can set a large concurrent number, so as to reduce the backup/restore time; otherwise, the concurrency should be reduced, otherwise, the backup/restore may encounter problems, i.e., time lagging, hang or OOM kill.  
+
+## Goals
+
+- Define the behaviors of concurrent VGDP instances in node-agent
+- Create a mechanism for users to specify the concurrent number of VGDP per node
+
+## Non-Goals
+- VGDP instances from different nodes always run in concurrent since in most common cases the resources are isolated. For special cases that some resources are shared across nodes, there is no support at present
+- In practice, restores run in prioritized scenarios, e.g., disaster recovery. However, the current design doesn't consider this difference, a VGDP instance for a restore is blocked if it reaches to the limit of the concurrency, even though the ones block it are for backups. If users do meet some problems here, they should consider to stop the backups first
+- Sometimes, users wants to totally block backups/restores from running in a specific node, this is out of the scope the current design. To archive this, more modules need to be considered (i.e., expoers of data movers), simply blocking the VGDP (e.g., by setting its concurrent number to 0) doesn't work. E.g., for a fs backup, VGDP instance must run in the node the source pod is running in, if we simply block from VGDP instance, the PodVolumeBackup CR is still submitted but never processed.  
+
+## Solution
+
+We introduce a ConfigMap specified by `velero node-agent` CLI's parameter `--node-agent-configmap` for users to specify the node-agent related configurations. This configMap is not created by Velero, users should create it manually on demand. The configMap should be in the same namespace where Velero is installed. If multiple Velero instances are installed in different namespaces, there should be one configMap in each namespace which applies to node-agent in that namespace only.  
+Node-agent server checks these configurations at startup time and use it to initiate the related VGDP modules. Therefore, users could edit this configMap any time, but in order to make the changes effective, node-agent server needs to be restarted.  
+The ConfigMap may be used for other purpose of configuring node-agent in future, at present, there is only one kind of configuration as the data in the configMap, the name is ```loadConcurrency```.  
+
+The data structure is as below:
+```go
+type Configs struct {
+	// LoadConcurrency is the config for load concurrency per node.
+	LoadConcurrency *LoadConcurrency `json:"loadConcurrency,omitempty"`
+}
+
+type LoadConcurrency struct {
+    // GlobalConfig specifies the concurrency number to all nodes for which per-node config is not specified
+    GlobalConfig int `json:"globalConfig,omitempty"`
+
+    // PerNodeConfig specifies the concurrency number to nodes matched by rules
+    PerNodeConfig []RuledConfigs `json:"perNodeConfig,omitempty"`
+}
+
+type RuledConfigs struct {
+    // NodeSelector specifies the label selector to match nodes
+    NodeSelector metav1.LabelSelector `json:"nodeSelector"`
+
+    // Number specifies the number value associated to the matched nodes
+    Number int `json:"number"`
+}
+```
+
+### Global concurrent number
+We allow users to specify a concurrent number that will be applied to all nodes if the per-node number is not specified. This number is set through ```globalConfig```.  
+The number starts from 1 which means there is no concurrency, only one instance of VGDP is allowed. There is no roof limit.    
+If this number is not specified or not valid, a hard-coded default value will be used, the value is set to 1. 
+
+### Per-node concurrent number
+We allow users to specify different concurrent number per node, for example, users can set 3 concurrent instances in Node-1, 2 instances in Node-2 and 1 instance in Node-3. This is for below considerations:
+- The resources may be different among nodes. Then users could specify smaller concurrent number for nodes with less resources while larger number for the ones with more resources
+- Help users to isolate critical environments. Users may run some critical workloads in some specified nodes, since VGDP instances may take large resource consumption, users may want to run less number of instances in the nodes with critical workloads
+
+The range of Per-node concurrent number is the same with Global concurrent number.  
+Per-node concurrent number is preferable to Global concurrent number, so it will overwrite the Global concurrent number for that node.  
+
+Per-node concurrent number is implemented through ```perNodeConfig``` field.  
+
+```perNodeConfig``` is a list of ```RuledConfigs``` each item of which matches one or more nodes by label selectors and specify the concurrent number for the matched nodes. This means, the nodes are identified by labels.  
+
+For example, the ```perNodeConfig`` could have below elements:
+```
+"nodeSelector: kubernetes.io/hostname=node1; number: 3"
+"nodeSelector: beta.kubernetes.io/instance-type=Standard_B4ms; number: 5"
+```
+The first element means the node with host name ```node1``` gets the Per-node concurrent number of 3.  
+The second element means all the nodes with label ```beta.kubernetes.io/instance-type``` of value ```Standard_B4ms``` get the Per-node concurrent number of 5. 
+At least one node is expected to have a label with the specified ```RuledConfigs``` element (rule). If no node is with this label, the Per-node rule makes no effect.  
+If one node falls into more than one rules, e.g., if node1 also has the label ```beta.kubernetes.io/instance-type=Standard_B4ms```, the smallest number (3) will be used.  
+
+### Sample
+A sample of the ConfigMap is as below:
+```json
+{
+    "loadConcurrency": {
+        "globalConfig": 2,
+        "perNodeConfig": [
+            {
+                "nodeSelector": {
+                    "matchLabels": {
+                        "kubernetes.io/hostname": "node1"
+                    }
+                },
+                "number": 3
+            },
+            {
+                "nodeSelector": {
+                    "matchLabels": {
+                        "beta.kubernetes.io/instance-type": "Standard_B4ms"
+                    }
+                },
+                "number": 5
+            }
+        ]
+    }
+}
+```
+To create the configMap, users need to save something like the above sample to a json file and then run below command:
+```
+kubectl create cm <ConfigMap name> -n velero --from-file=<json file name>
+```
+
+### Global data path manager
+As for the code implementation, data path manager is to maintain the total number of the running VGDP instances and ensure the limit is not excceeded. At present, there is one data path manager instance per controller, as a result, the concurrent numbers are calculated separately for each controller. This doesn't help to limit the concurrency among different requesters.  
+Therefore, we need to create one global data path manager instance server-wide, and pass it to different controllers. The instance will be created at node-agent server startup.  
+The concurrent number is required to initiate a data path manager, the number comes from either Per-node concurrent number or Global concurrent number.    
+Below are some prototypes related to data path manager:  
+
+```go
+func NewManager(cocurrentNum int) *Manager
+func (m *Manager) CreateFileSystemBR(jobName string, requestorType string, ctx context.Context, client client.Client, namespace string, callbacks Callbacks, log logrus.FieldLogger) (AsyncBR, error)
+```
+
+
+
+
+
+[1]: Implemented/unified-repo-and-kopia-integration/unified-repo-and-kopia-integration.md

design/Implemented/plugin-backup-and-restore-progress-design.md

@@ -393,7 +393,7 @@ Deletion of `VolumePluginBackup` CR can be delegated to plugin. Plugin can perfo
 ### 'core' Velero client/server required changes
 
 - Creation of the VolumePluginBackup/VolumePluginRestore CRDs at installation time
-- Persistence of VolumePluginBackup CRs towards the end of the back up operation
+- Persistence of VolumePluginBackup CRs towards the end of the backup operation
 - As part of backup synchronization, VolumePluginBackup CRs related to the backup will be synced.
 - Deletion of VolumePluginBackup when volumeshapshotter's DeleteSnapshot is called
 - Deletion of VolumePluginRestore as part of handling deletion of Restore CR

design/Implemented/plugin-versioning.md

@@ -135,8 +135,11 @@ type ObjectStore interface {
 
 The proto service definitions of the plugins will also be versioned and arranged by their plugin kind.
 Currently, all the proto definitions reside under `pkg/plugin/proto` in a file corresponding to their plugin kind.
-These files will be rearranged to be grouped by kind and then versioned: `pkg/plugin/proto/<plugin_kind>/<version>`.
-The scripts to compile the proto service definitions will need to be updated to place the generated Go code under a matching directory structure.
+These files will be rearranged to be grouped by kind and then versioned: `pkg/plugin/proto/<plugin_kind>/<version>`,
+except for the current v1 plugins. Those will remain in their current package/location for backwards compatibility.
+This will allow plugin images built with earlier versions of velero to work with the latest velero (for v1 plugins
+only). The go_package option will be added to all proto service definitions to allow the proto compilation script
+to place the generated go code for each plugin api version in the proper go package directory.
 
 It is not possible to import an existing proto service into a new one, so any methods will need to be duplicated across versions if they are required by the new version.
 The message definitions can be shared however, so these could be extracted from the service definition files and placed in a file that can be shared across all versions of the service.
@@ -238,7 +241,7 @@ In cases where the methods signatures remain the same, the adaptation layer will
 Examples where an adaptation may be safe:
 - A method signature is being changed to add a new parameter but the parameter could be optional (for example, adding a context parameter). The adaptation could call through to the method provided in the previous version but omit the parameter.
 - A method signature is being changed to remove a parameter, but it is safe to pass a default value to the previous version. The adaptation could call through to the method provided in the previous version but use a default value for the parameter.
-- A new method is being added but does not impact any existing behaviour of Velero (for example, a new method which will allow Velero to [wait for additional items to be ready](https://github.com/vmware-tanzu/velero/blob/main/design/wait-for-additional-items.md)). The adaptation would return a value which allows the existing behaviour to be performed.
+- A new method is being added but does not impact any existing behaviour of Velero (for example, a new method which will allow Velero to [wait for additional items to be ready](https://github.com/vmware-tanzu/velero/blob/main/design/Implemented/wait-for-additional-items.md)). The adaptation would return a value which allows the existing behaviour to be performed.
 - A method is being deleted as it is no longer used. The adaptation would call through to any methods which are still included but would omit the deleted method in the adaptation.
 
 Examples where an adaptation may not be safe:

design/Implemented/pv_backup_info.md

@@ -0,0 +1,186 @@
+# PersistentVolume backup information design
+
+## Abstract
+Create a new metadata file in the backup repository's backup name sub-directory to store the backup-including PVC and PV information. The information includes the way of backing up the PVC and PV data, snapshot information, and status. The needed snapshot status can also be recorded there, but the Velero-Native snapshot plugin doesn't provide a way to get the snapshot size from the API, so it's possible that not all snapshot size information is available.
+
+This new additional metadata file is needed when:
+* Get a summary of the backup's PVC and PV information, including how the data in them is backed up, or whether the data in them is skipped from backup.
+* Find out how the PVC and PV should be restored in the restore process.
+* Retrieve the PV's snapshot information for backup.
+
+## Background
+There is already a [PR](https://github.com/vmware-tanzu/velero/pull/6496) to track the skipped PVC in the backup. This design will depend on it and go further to get a summary of PVC and PV information, then persist into a metadata file in the backup repository.
+
+In the restore process, the Velero server needs to decide how the PV resource should be restored according to how the PV is backed up. The current logic is to check whether it's backed up by Velero-native snapshot, by file-system backup, or having `DeletionPolicy` set as `Delete`.
+
+The checks are made by the backup-generated PVBs or Snapshots. There is no generic way to find this information, and the CSI backup and Snapshot data movement backup are not covered.
+
+Another thing that needs noticing is when describing the backup, there is no generic way to find the PV's snapshot information.
+
+## Goals
+- Create a new metadata file to store backup's PVCs and PVs information and volume data backing up method. The file can be used to let downstream consumers generate a summary.
+- Create a generic way to let the Velero server know how the PV resources are backed up.
+- Create a generic way to let the Velero server find the PV corresponding snapshot information.
+
+## Non Goals
+- Unify how to get snapshot size information for all PV backing-up methods, and all other currently not ready PVs' information.
+
+## High-Level Design
+Create _backup-name_-volumes-info.json metadata file in the backup's repository. This file will be encoded to contain all the PVC and PV information included in the backup. The information covers whether the PV or PVC's data is skipped during backup, how its data is backed up, and the backed-up detail information.
+
+Please notice that the new metadata file includes all skipped volume information. This is used to address [the second phase needs of skipped volumes information](https://github.com/vmware-tanzu/velero/issues/5834#issuecomment-1526624211).
+
+The `restoreItem` function can decode the _backup-name_-volumes-info.json file to determine how to handle the PV resource. 
+
+## Detailed Design
+
+### The VolumeInfo structure
+_backup-name_-volumes-info.json file is a structure that contains an array of structure `VolumeInfo`.
+
+``` golang
+type VolumeInfo struct {
+    PVCName        string    // The PVC's name.
+    PVCNamespace   string    // The PVC's namespace.
+    PVName         string    // The PV name.
+    BackupMethod   string    // The way the volume data is backed up. The valid value includes `VeleroNativeSnapshot`, `PodVolumeBackup` and `CSISnapshot`.
+    SnapshotDataMoved bool   // Whether the volume's snapshot data is moved to specified storage.
+
+    Skipped         boolean   // Whether the Volume is skipped in this backup.
+    SkippedReason   string    // The reason for the volume is skipped in the backup.
+    StartTimestamp  *metav1.Time // Snapshot starts timestamp.
+
+    OperationID     string   // The Async Operation's ID.
+
+    CSISnapshotInfo          CSISnapshotInfo
+    SnapshotDataMovementInfo SnapshotDataMovementInfo
+    NativeSnapshotInfo       VeleroNativeSnapshotInfo
+    PVBInfo                  PodVolumeBackupInfo
+    PVInfo                   PVInfo
+}
+
+// CSISnapshotInfo is used for displaying the CSI snapshot status
+type CSISnapshotInfo struct {
+    SnapshotHandle  string       // It's the storage provider's snapshot ID for CSI.
+    Size            int64        // The snapshot corresponding volume size.
+
+    Driver          string  // The name of the CSI driver.
+    VSCName         string // The name of the VolumeSnapshotContent. 
+}
+
+// SnapshotDataMovementInfo is used for displaying the snapshot data mover status.
+type SnapshotDataMovementInfo struct {
+    DataMover        string    // The data mover used by the backup. The valid values are `velero` and ``(equals to `velero`).
+    UploaderType     string    // The type of the uploader that uploads the snapshot data. The valid values are `kopia` and `restic`.
+    RetainedSnapshot string    // The name or ID of the snapshot associated object(SAO). SAO is used to support local snapshots for the snapshot data mover, e.g. it could be a VolumeSnapshot for CSI snapshot data moign/pv_backup_info.
+    SnapshotHandle string  	   // It's the filesystem repository's snapshot ID.
+	
+}
+
+// VeleroNativeSnapshotInfo is used for displaying the Velero native snapshot status.
+type VeleroNativeSnapshotInfo struct {
+    SnapshotHandle      string       // It's the storage provider's snapshot ID for the Velero-native snapshot.
+
+    VolumeType string    // The cloud provider snapshot volume type.
+    VolumeAZ   string    // The cloud provider snapshot volume's availability zones.
+    IOPS       string    // The cloud provider snapshot volume's IOPS.
+}
+
+// PodVolumeBackupInfo is used for displaying the PodVolumeBackup snapshot status.
+type PodVolumeBackupInfo struct {
+    SnapshotHandle      string       // It's the file-system uploader's snapshot ID for PodVolumeBackup.
+    Size                int64        // The snapshot corresponding volume size.
+
+    UploaderType  string    // The type of the uploader that uploads the data. The valid values are `kopia` and `restic`.
+    VolumeName    string   // The PVC's corresponding volume name used by Pod: https://github.com/kubernetes/kubernetes/blob/e4b74dd12fa8cb63c174091d5536a10b8ec19d34/pkg/apis/core/types.go#L48
+    PodName       string   // The Pod name mounting this PVC.
+    PodNamespace  string   // The Pod namespace.
+    NodeName      string   // The PVB-taken k8s node's name.
+}
+
+// PVInfo is used to store some PV information modified after creation.
+// Those information are lost after PV recreation.
+type PVInfo struct {
+    ReclaimPolicy string            // ReclaimPolicy of PV. It could be different from the referenced StorageClass.
+    Labels        map[string]string // The PV's labels should be kept after recreation.
+}
+```
+
+### How the VolumeInfo array is generated.
+The function `persistBackup` has `backup *pkgbackup.Request` in parameters.
+From it, the `VolumeSnapshots`, `PodVolumeBackups`, `CSISnapshots`, `itemOperationsList`, and `SkippedPVTracker` can be read. All of them will be iterated and merged into the `VolumeInfo` array, and then persisted into backup repository in function `persistBackup`.
+
+Please notice that the change happened in async operations are not reflected in the new metadata file. The file only covers the volume changes happen in the Velero server process scope.
+
+A new methods are added to BackupStore to download the VolumeInfo metadata file.
+Uploading the metadata file is covered in the exiting `PutBackup` method.
+
+``` golang
+type BackupStore interface {
+    ...
+    GetVolumeInfos(name string) ([]*VolumeInfo, error)
+    ...
+}
+```
+
+### How the VolumeInfo array is used.
+
+#### Generate the PVC backed-up information summary
+The downstream tools can use this VolumeInfo array to format and display their volume information. This is not in the scope of this feature.
+
+#### Retrieve volume backed-up information for `velero backup describe` command
+The `velero backup describe` can also use this VolumeInfo array structure to display the volume information. The snapshot data mover volume should use this structure at first, then the Velero native snapshot, CSI snapshot, and PodVolumeBackup can also use this structure. The detailed implementation is also not in this feature's scope.
+
+#### Let restore know how to restore the PV
+In the function `restoreItem`, it will determine whether to restore the PV resource by checking it in the Velero native snapshots list, PodVolumeBackup list, and its DeletionPolicy. This logic is still kept. The logic will be used when the new `VolumeInfo` metadata cannot be found to support backward compatibility.
+
+``` golang
+	if groupResource == kuberesource.PersistentVolumes {
+		switch {
+		case hasSnapshot(name, ctx.volumeSnapshots):
+            ...
+        case hasPodVolumeBackup(obj, ctx):
+            ...
+        case hasDeleteReclaimPolicy(obj.Object):
+            ...
+        default:
+            ...
+```
+
+After introducing the VolumeInfo array, the following logic will be added.
+``` golang
+	if groupResource == kuberesource.PersistentVolumes {
+        volumeInfo := GetVolumeInfo(pvName)
+		switch volumeInfo.BackupMethod {
+		case VeleroNativeSnapshot:
+            ...
+        case PodVolumeBackup:
+            ...
+        case CSISnapshot:
+            ...
+        default:
+            // Need to check whether the volume is backed up by the SnapshotDataMover.
+            if volumeInfo.SnapshotDataMovement:
+
+            // Check whether the Velero server should restore the PV depending on the DeletionPolicy setting.
+            if volumeInfo.Skipped:
+```
+
+### How the VolumeInfo metadata file is deleted
+_backup-name_-volumes-info.json file is deleted during backup deletion.
+
+## Alternatives Considered
+The restore process needs more information about how the PVs are backed up to determine whether this PV should be restored. The released branches also need a similar function, but backporting a new feature into previous releases may not be a good idea, so according to [Anshul Ahuja's suggestion](https://github.com/vmware-tanzu/velero/issues/6595#issuecomment-1731081580), adding more cases here to support checking PV backed-up by CSI plugin and CSI snapshot data mover: https://github.com/vmware-tanzu/velero/blob/5ff5073cc3f364bafcfbd26755e2a92af68ba180/pkg/restore/restore.go#L1206-L1324.
+
+## Security Considerations
+There should be no security impact introduced by this design.
+
+## Compatibility
+After this design is implemented, there should be no impact on the existing [skipped PVC summary feature](https://github.com/vmware-tanzu/velero/pull/6496).
+
+To support older version backup, which doesn't have the VolumeInfo metadata file, the old logic, which is checking the Velero native snapshots list, PodVolumeBackup list, and PVC DeletionPolicy, is still kept, and supporting CSI snapshots and snapshot data mover logic will be added too.
+
+## Implementation
+This will be implemented in the Velero v1.13 development cycle.
+
+## Open Issues
+There are no open issues identified by now.

design/Implemented/pv_restore_info.md

@@ -0,0 +1,143 @@
+# Volume information for restore design
+
+## Background
+Velero has different ways to handle data in the volumes during restore.  The users want to have more clarity in terms of how
+the volumes are handled in restore process via either Velero CLI or other downstream product which consumes Velero.
+
+## Goals
+- Create new metadata to store the information of the restored volume, which will have the same life-cycle as the restore CR.
+- Consume the metadata in velero CLI to enable it display more details for volumes in the output of `velero restore describe --details`
+
+## Non Goals
+- Provide finer grained control of the volume restore process. The focus of the design is to enable displaying more details.
+- Persist additional metadata like podvolume, datadownloads etc to the restore folder in backup-location.
+
+## Design
+
+### Structure of the restore volume info
+The restore volume info will be stored in a file named like `${restore_name}-vol-info.json`. The content of the file will
+be a list of volume info objects, each of which will map to a volume that is restored, and will contain the information 
+like name of the restored PV/PVC, restore method and related objects to provide details depending on the way it's restored,
+it will look like this:
+```
+[
+  {
+    "pvcName": "nginx-logs-2",
+    "pvcNamespace": "nginx-app-restore",
+    "pvName": "pvc-e320d75b-a788-41a3-b6ba-267a553efa5e",
+    "restoreMethod": "PodVolumeRestore",
+    "snapshotDataMoved": false,
+    "pvrInfo": {
+      "snapshotHandle": "81973157c3a945a5229285c931b02c68",
+      "uploaderType": "kopia",
+      "volumeName": "nginx-logs",
+      "podName": "nginx-deployment-79b56c644b-mjdhp",
+      "podNamespace": "nginx-app-restore"
+    }
+  },
+  {
+    "pvcName": "nginx-logs-1",
+    "pvcNamespace": "nginx-app-restore",
+    "pvName": "pvc-98c151f4-df47-4980-ba6d-470842f652cc",
+    "restoreMethod": "CSISnapshot",
+    "snapshotDataMoved": false,
+    "csiSnapshotInfo": {
+      "snapshotHandle": "snap-01a3b21a5e9f85528",
+      "size": 2147483648,
+      "driver": "ebs.csi.aws.com",
+      "vscName": "velero-velero-nginx-logs-1-jxmbg-hx9x5"
+    }
+  }
+......  
+]
+```
+Each field will have the same meaning as the corresponding field in the backup volume info.  It will not have the fields 
+that were introduced to help with the backup process, like `pvInfo`, `dataupload` etc.
+
+### How the restore volume info is generated
+Two steps are involved in generating the restore volume info, the first is "collection", which is to gather the information 
+for restoration of the volumes, the second is "generation", which is to iterate through the data collected in the first step
+and generate the volume info list as is described above.
+
+Unlike backup, the CR objects created during the restore process will not be persisted to the backup storage location.  
+Therefore, to gather the information needed to generate volume information, we either need to collect the CRs in the middle
+of the restore process, or retrieve the objects based on the `resouce-list.json` of the restore via API server.
+The information to be collected are:
+- **PV/PVC mapping relationship:** It will be collected via the `restore-resource-list.json`, b/c at the time the json is ready, all
+PVCs and PVs are already created.
+- **Native snapshot information:** It will be collected in the restore workflow when each snapshot is restored.
+- **podvolumerestore CRs:** It will be collected in the restore workflow after each pvr is created.
+- **volumesnapshot CRs for CSI snapshot:** It will be collected in the step of collecting PVC info, by reading the `dataSource`
+field in the spec of the PVC.
+- **datadownload CRs** It will be collected in the phase of collecting PVC info, by querying the API-server to list the datadownload
+CRs labeled with the restore name.
+
+After the collection step, the generation step is relatively straight-forward, as we have all the information needed in 
+the data structures.  
+
+The whole collection and generation steps will be done with the "best-effort" manner, i.e. if there are any failures we 
+will only log the error in restore log, rather than failing the whole restore process, we will not put these errors or warnings
+into the `result.json`, b/c it won't impact the restored resources.
+
+Depending on the number of the restored PVCs the "collection" step may involve many API calls, but it's considered acceptable
+b/c at that time the resources are already created, so the actual RTO is not impacted.  By using the client of controller runtime
+we can make the collection step more efficient by using the cache of the API server.  We may consider to make improvements if 
+we observe performance issues, like using multiple go-routines in the collection.
+
+### Implementation
+Because the restore volume info shares the same data structures with the backup volume info, we will refactor the code in 
+package `internal/volume` to make the sub-components in backup volume info shared by both backup and restore volume info.  
+
+We'll introduce a struct called `RestoreVolumeInfoTracker` which encapsulates the logic of collecting and generating the restore volume info:
+```
+// RestoreVolumeInfoTracker is used to track the volume information during restore.
+// It is used to generate the RestoreVolumeInfo array.
+type RestoreVolumeInfoTracker struct {
+	*sync.Mutex
+	restore *velerov1api.Restore
+	log     logrus.FieldLogger
+	client  kbclient.Client
+	pvPvc   *pvcPvMap
+
+	// map of PV name to the NativeSnapshotInfo from which the PV is restored
+	pvNativeSnapshotMap map[string]NativeSnapshotInfo
+	// map of PV name to the CSISnapshot object from which the PV is restored
+	pvCSISnapshotMap map[string]snapshotv1api.VolumeSnapshot
+	datadownloadList *velerov2alpha1.DataDownloadList
+	pvrs             []*velerov1api.PodVolumeRestore
+}
+```
+The `RestoreVolumeInfoTracker` will be created when the restore request is initialized, and it will be passed to the `restoreContext`
+and carried over the whole restore process.  
+
+The `client` in this struct is to be used to query the resources in the restored namespace, and the current client in restore 
+reconciler only watches the resources in the namespace where velero is installed.  Therefore, we need to introduce the 
+`CrClient` which has the same life-cycle of velero server to the restore reconciler, because this is the client that watches all the 
+resources on the cluster.
+
+In addition to that, we will make small changes in the restore workflow to collect the information needed.  We'll make the 
+changes un-intrusive and make sure not to change the logic of the restore to avoid break change or regression.
+We'll also introduce routine changes in the package `pkg/persistence` to persist the restore volume info to the backup storage location.
+
+Last but not least, the `velero restore describe --details` will be updated to display the volume info in the output.  
+
+## Alternatives Considered
+There used to be suggestion that to provide more details about volume, we can query the `backup-vol-info.json` with the resource 
+identifier in `restore-resource-list.json`.  This will not work when there're resource modifiers involved in the restore process,
+which may change the metadata of PVC/PV.  In addition, we may add more detailed restore-specific information about the volumes that is not available
+in the `backup-vol-info.json`.  Therefore, the `restore-vol-info.json` is a better approach.
+
+## Security Considerations
+There should be no security impact introduced by this design.
+
+## Compatibility
+The restore volume info will be consumed by Velero CLI and downstream products for displaying details.  So the functionality 
+of backup and restore will not be impacted for restores created by older versions of Velero which do not have the restore volume info
+metadata.  The client should properly handle the case when the restore volume info does not exist.
+
+The data structures referenced by volume info is shared between both restore and backup and it's not versioned, so in the future 
+we must make sure there will only be incremental changes to the metadata, such that no break change will be introduced to the client.
+
+## Open Issues
+https://github.com/vmware-tanzu/velero/issues/7546
+https://github.com/vmware-tanzu/velero/issues/6478

design/Implemented/repo_maintenance_job_config.md

@@ -0,0 +1,311 @@
+# Repository maintenance job configuration design
+
+## Abstract
+Add this design to make the repository maintenance job can read configuration from a dedicate ConfigMap and make the Job's necessary parts configurable, e.g. `PodSpec.Affinity` and `PodSpec.Resources`.
+
+## Background
+Repository maintenance is split from the Velero server to a k8s Job in v1.14 by design [repository maintenance job](Implemented/repository-maintenance.md).
+The repository maintenance Job configuration was read from the Velero server CLI parameter, and it inherits the most of Velero server's Deployment's PodSpec to fill un-configured fields.
+
+This design introduces a new way to let the user to customize the repository maintenance behavior instead of inheriting from the Velero server Deployment or reading from `velero server` CLI parameters.
+The configurations added in this design including the resource limitations, node selection.
+It's possible new configurations are introduced in future releases based on this design.
+
+For the node selection, the repository maintenance Job also inherits from the Velero server deployment before, but the Job may last for a while and cost noneligible resources, especially memory.
+The users have the need to choose which k8s node to run the maintenance Job.
+This design reuses the data structure introduced by design [node-agent affinity configuration](Implemented/node-agent-affinity.md) to make the repository maintenance job can choose which node running on.
+
+## Goals
+- Unify the repository maintenance Job configuration at one place.
+- Let user can choose repository maintenance Job running on which nodes.
+
+## Non Goals
+- There was an [issue](https://github.com/vmware-tanzu/velero/issues/7911) to require the whole Job's PodSpec should be configurable. That's not in the scope of this design.
+- Please notice this new configuration is dedicated for the repository maintenance. Repository itself configuration is not covered.
+
+
+## Compatibility
+v1.14 uses the `velero server` CLI's parameter to pass the repository maintenance job configuration.
+In v1.15, those parameters are still kept, including `--maintenance-job-cpu-request`, `--maintenance-job-mem-request`, `--maintenance-job-cpu-limit`, `--maintenance-job-mem-limit`, and `--keep-latest-maintenance-jobs`.
+But the parameters read from the ConfigMap specified by `velero server` CLI parameter `--repo-maintenance-job-configmap` introduced by this design have a higher priority.
+
+If there `--repo-maintenance-job-configmap` is not specified, then the `velero server` parameters are used if provided.
+
+If the `velero server` parameters are not specified too, then the default values are used.
+* `--keep-latest-maintenance-jobs` default value is 3.
+* `--maintenance-job-cpu-request` default value is 0.
+* `--maintenance-job-mem-request` default value is 0.
+* `--maintenance-job-cpu-limit` default value is 0.
+* `--maintenance-job-mem-limit` default value is 0.
+
+## Deprecation
+Propose to deprecate the `velero server` parameters `--maintenance-job-cpu-request`, `--maintenance-job-mem-request`, `--maintenance-job-cpu-limit`, `--maintenance-job-mem-limit`, and `--keep-latest-maintenance-jobs` in release-1.15.
+That means those parameters will be deleted in release-1.17.
+After deletion, those resources-related parameters are replaced by the ConfigMap specified by `velero server` CLI's parameter `--repo-maintenance-job-configmap`.
+`--keep-latest-maintenance-jobs` is deleted from `velero server` CLI. It turns into a non-configurable internal parameter, and its value is 3.
+Please check [issue 7923](https://github.com/vmware-tanzu/velero/issues/7923) for more information why deleting this parameter.
+
+## Design
+This design introduces a new ConfigMap specified by `velero server` CLI parameter `--repo-maintenance-job-configmap` as the source of the repository maintenance job configuration. The specified ConfigMap is read from the namespace where Velero is installed.
+If the ConfigMap doesn't exist, the internal default values are used.
+
+Example of using the parameter `--repo-maintenance-job-configmap`:
+```
+velero server \
+    ...
+    --repo-maintenance-job-configmap repo-job-config
+    ...
+```
+
+**Notice**
+* Velero doesn't own this ConfigMap. If the user wants to customize the repository maintenance job, the user needs to create this ConfigMap.
+* Velero reads this ConfigMap content at starting a new repository maintenance job, so the ConfigMap change will not take affect until the next created job.
+
+### Structure
+The data structure is as below:
+```go
+type Configs struct {
+    // LoadAffinity is the config for data path load affinity.
+    LoadAffinity []*LoadAffinity `json:"loadAffinity,omitempty"`    
+
+    // PodResources is the config for the CPU and memory resources setting.
+    PodResources *kube.PodResources `json:"podResources,omitempty"`
+}
+
+type LoadAffinity struct {
+    // NodeSelector specifies the label selector to match nodes
+    NodeSelector metav1.LabelSelector `json:"nodeSelector"`
+}
+
+type PodResources struct {
+	CPURequest    string `json:"cpuRequest,omitempty"`
+	MemoryRequest string `json:"memoryRequest,omitempty"`
+	CPULimit      string `json:"cpuLimit,omitempty"`
+	MemoryLimit   string `json:"memoryLimit,omitempty"`
+}
+```
+
+The ConfigMap content is a map.
+If there is a key value as `global` in the map, the key's value is applied to all BackupRepositories maintenance jobs that cannot find their own specific configuration in the ConfigMap.
+The other keys in the map is the combination of three elements of a BackupRepository:
+* The namespace in which BackupRepository backs up volume data.
+* The BackupRepository referenced BackupStorageLocation's name.
+* The BackupRepository's type. Possible values are `kopia` and `restic`.
+
+Those three keys can identify a [unique BackupRepository](https://github.com/vmware-tanzu/velero/blob/2fc6300f2239f250b40b0488c35feae59520f2d3/pkg/repository/backup_repo_op.go#L32-L37).
+
+If there is a key match with BackupRepository, the key's value is applied to the BackupRepository's maintenance jobs.
+By this way, it's possible to let user configure before the BackupRepository is created.
+This is especially convenient for administrator configuring during the Velero installation.
+For example, the following BackupRepository's key should be `test-default-kopia`.
+
+``` yaml
+- apiVersion: velero.io/v1
+  kind: BackupRepository
+  metadata:
+    generateName: test-default-kopia-
+    labels:
+      velero.io/repository-type: kopia
+      velero.io/storage-location: default
+      velero.io/volume-namespace: test
+    name: test-default-kopia-kgt6n
+    namespace: velero
+  spec:
+    backupStorageLocation: default
+    maintenanceFrequency: 1h0m0s
+    repositoryType: kopia
+    resticIdentifier: gs:jxun:/restic/test
+    volumeNamespace: test
+```
+
+The `LoadAffinity` structure is reused from design [node-agent affinity configuration](Implemented/node-agent-affinity.md).
+It's possible that the users want to choose nodes that match condition A or condition B to run the job.
+For example, the user want to let the nodes is in a specified machine type or the nodes locate in the us-central1-x zones to run the job.
+This can be done by adding multiple entries in the `LoadAffinity` array.
+
+### Affinity Example
+A sample of the ConfigMap is as below:
+``` bash
+cat <<EOF > repo-maintenance-job-config.json
+{
+    "global": {
+        podResources: {
+            "cpuRequest": "100m",
+            "cpuLimit": "200m",
+            "memoryRequest": "100Mi",
+            "memoryLimit": "200Mi"
+        },
+        "loadAffinity": [
+            {
+                "nodeSelector": {
+                    "matchExpressions": [
+                        {
+                            "key": "cloud.google.com/machine-family",
+                            "operator": "In",
+                            "values": [
+                                "e2"
+                            ]
+                        }
+                    ]          
+                }
+            },
+            {
+                "nodeSelector": {
+                    "matchExpressions": [
+                        {
+                            "key": "topology.kubernetes.io/zone",
+                            "operator": "In",
+                            "values": [
+                                "us-central1-a",
+                                "us-central1-b",
+                                "us-central1-c"
+                            ]
+                        }
+                    ]          
+                }
+            }
+        ]
+    }
+}
+EOF
+```
+This sample showcases two affinity configurations:
+- matchLabels: maintenance job runs on nodes with label key `cloud.google.com/machine-family` and value `e2`.
+- matchLabels: maintenance job runs on nodes located in `us-central1-a`, `us-central1-b` and `us-central1-c`.
+The nodes matching one of the two conditions are selected.
+
+To create the configMap, users need to save something like the above sample to a json file and then run below command:
+```
+kubectl create cm repo-maintenance-job-config -n velero --from-file=repo-maintenance-job-config.json
+```
+
+### Value assigning rules
+If the Velero BackupRepositoryController cannot find the introduced ConfigMap, the following default values are used for repository maintenance job:
+``` go
+config := Configs {
+    // LoadAffinity is the config for data path load affinity.
+    LoadAffinity: nil,
+
+    // Resources is the config for the CPU and memory resources setting.
+    PodResources: &kube.PodResources{
+        // The repository maintenance job CPU request setting
+	    CPURequest:   "0m",
+
+        // The repository maintenance job memory request setting
+	    MemoryRequest:   "0Mi",
+
+        // The repository maintenance job CPU limit setting
+	    CPULimit:     "0m",
+
+        // The repository maintenance job memory limit setting
+	    MemoryLimit:     "0Mi",
+    },
+}
+```
+
+If the Velero BackupRepositoryController finds the introduced ConfigMap with only `global` element, the `global` value is used.
+
+If the Velero BackupRepositoryController finds the introduced ConfigMap with only element matches the BackupRepository, the matched element value is used.
+
+
+If the Velero BackupRepositoryController finds the introduced ConfigMap with both `global` element and element matches the BackupRepository, the matched element defined values overwrite the `global` value, and the `global` value is still used for matched element undefined values.
+
+For example, the ConfigMap content has two elements.
+``` json
+{
+    "global": {
+        "loadAffinity": [
+            {
+                "nodeSelector": {
+                    "matchExpressions": [
+                        {
+                            "key": "cloud.google.com/machine-family",
+                            "operator": "In",
+                            "values": [
+                                "e2"
+                            ]
+                        }
+                    ]          
+                }
+            },
+        ],
+        "podResources": {
+            "cpuRequest": "100m",
+            "cpuLimit": "200m",
+            "memoryRequest": "100Mi",
+            "memoryLimit": "200Mi"
+        }
+    },
+    "ns1-default-kopia": {
+        "podResources": {
+            "memoryRequest": "400Mi",
+            "memoryLimit": "800Mi"
+        }
+    }
+}
+```
+The config value used for BackupRepository backing up volume data in namespace `ns1`, referencing BSL `default`, and the type is `Kopia`:
+``` go
+config := Configs {
+    // LoadAffinity is the config for data path load affinity.
+    LoadAffinity: []*kube.LoadAffinity{
+        {
+			NodeSelector: metav1.LabelSelector{
+				MatchExpressions: []metav1.LabelSelectorRequirement{
+					{
+						Key:      "cloud.google.com/machine-family",
+						Operator: metav1.LabelSelectorOpIn,
+						Values:   []string{"e2"},
+					},
+				},
+			},
+		},
+    },
+    PodResources: &kube.PodResources{
+        // The repository maintenance job CPU request setting
+	    CPURequest:   "",
+        // The repository maintenance job memory request setting
+	    MemoryRequest:   "400Mi",
+        // The repository maintenance job CPU limit setting
+	    CPULimit:     "",
+        // The repository maintenance job memory limit setting
+	    MemoryLimit:     "800Mi",
+    }
+}
+```
+
+
+### Implementation
+During the Velero repository controller starts to maintain a repository, it will call the repository manager's `PruneRepo` function to build the maintenance Job.
+The ConfigMap specified by `velero server` CLI parameter `--repo-maintenance-job-configmap` is get to reinitialize the repository `MaintenanceConfig` setting.
+
+``` go
+	jobConfig, err := getMaintenanceJobConfig(
+		context.Background(),
+		m.client,
+		m.log,
+		m.namespace,
+		m.repoMaintenanceJobConfig,
+		repo,
+	)
+	if err != nil {
+        log.Infof("Cannot find the ConfigMap %s with error: %s. Use default value.",
+			m.namespace+"/"+m.repoMaintenanceJobConfig,
+			err.Error(),
+		)
+	}
+
+	log.Info("Start to maintenance repo")
+
+	maintenanceJob, err := m.buildMaintenanceJob(
+		jobConfig,
+		param,
+	)
+	if err != nil {
+		return errors.Wrap(err, "error to build maintenance job")
+	}
+```
+
+## Alternatives Considered
+An other option is creating each ConfigMap for a BackupRepository.
+This is not ideal for scenario that has a lot of BackupRepositories in the cluster.