Fix brew-update script, their commands changed (#2711)

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -14,11 +14,11 @@ about: Tell us about a problem you are experiencing
 **The output of the following commands will help us better understand what's going on**:
 (Pasting long output into a [GitHub gist](https://gist.github.com) or other pastebin is fine.)
 
-* `kubectl logs deployment/velero -n velero`
-* `velero backup describe <backupname>` or `kubectl get backup/<backupname> -n velero -o yaml`
-* `velero backup logs <backupname>`
-* `velero restore describe <restorename>` or `kubectl get restore/<restorename> -n velero -o yaml`
-* `velero restore logs <restorename>`
+- `kubectl logs deployment/velero -n velero`
+- `velero backup describe <backupname>` or `kubectl get backup/<backupname> -n velero -o yaml`
+- `velero backup logs <backupname>`
+- `velero restore describe <restorename>` or `kubectl get restore/<restorename> -n velero -o yaml`
+- `velero restore logs <restorename>`
 
 
 **Anything else you would like to add:**
@@ -28,7 +28,17 @@ about: Tell us about a problem you are experiencing
 **Environment:**
 
 - Velero version (use `velero version`): 
+- Velero features (use `velero client config get features`): 
 - Kubernetes version (use `kubectl version`):
 - Kubernetes installer & version:
 - Cloud provider or hardware configuration:
 - OS (e.g. from `/etc/os-release`):
+
+
+**Vote on this issue!**
+
+This is an invitation to the Velero community to vote on issues, you can see the project's [top voted issues listed here](https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).  
+Use the "reaction smiley face" up to the right of this comment to vote.
+
+- :+1: for "I would like to see this bug fixed as soon as possible"
+- :-1: for "There are more important bugs to focus on right now"

.github/ISSUE_TEMPLATE/feature-enhancement-request.md

@@ -23,3 +23,11 @@ about: Suggest an idea for this project
 - Kubernetes installer & version:
 - Cloud provider or hardware configuration:
 - OS (e.g. from `/etc/os-release`):
+
+**Vote on this issue!**
+
+This is an invitation to the Velero community to vote on issues, you can see the project's [top voted issues listed here](https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).  
+Use the "reaction smiley face" up to the right of this comment to vote.
+
+- :+1: for "The project would be better with this feature added"
+- :-1: for "This feature will not enhance the project in a meaningful way"

.github/workflows/pr-changelog-check.yml

@@ -0,0 +1,15 @@
+name: Pull Request Changelog Check
+on: [pull_request]
+jobs:
+
+  build:
+    name: Run Changelog Check
+    runs-on: ubuntu-latest
+    steps:
+
+    - name: Check out the code
+      uses: actions/checkout@v2
+
+    - name: Changelog check
+      if: ${{ !(contains(github.event.pull_request.labels.*.name, 'changelog-not-required') || contains(github.event.pull_request.labels.*.name, 'Design') || contains(github.event.pull_request.labels.*.name, 'Website') || contains(github.event.pull_request.labels.*.name, 'Documentation'))}}
+      run: ./hack/changelog-check.sh

.github/workflows/pr-ci-check.yml

@@ -0,0 +1,14 @@
+name: Pull Request CI Check
+on: [pull_request]
+jobs:
+
+  build:
+    name: Run CI
+    runs-on: ubuntu-latest
+    steps:
+
+    - name: Check out the code
+      uses: actions/checkout@v2
+
+    - name: Make ci
+      run: make ci

.github/workflows/pr-linter-check.yml

@@ -0,0 +1,14 @@
+name: Pull Request Linter Check
+on: [pull_request]
+jobs:
+
+  build:
+    name: Run Linter Check
+    runs-on: ubuntu-latest
+    steps:
+
+    - name: Check out the code
+      uses: actions/checkout@v2
+
+    - name: Linter check
+      run: make lint

.github/workflows/push-builder.yml

@@ -0,0 +1,22 @@
+name: build-image
+
+on:
+  push:
+    branches: [ master ]
+    paths:
+    - 'hack/build-image/Dockerfile'
+
+jobs:
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    steps:
+
+    - name: Build
+      run: make build-image
+
+    - name: Publish container image
+      run: |
+        docker login -u ${{ secrets.DOCKER_USER }} -p ${{ secrets.DOCKER_PASSWORD }}
+       
+        make push-build-image

.github/workflows/push.yml

@@ -0,0 +1,34 @@
+name: Master CI
+
+on:
+  push:
+    branches: [ master ]
+    tags:
+      - '*'
+
+jobs:
+
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    steps:
+
+    - name: Set up Go 1.14
+      uses: actions/setup-go@v2
+      with:
+        go-version: 1.14
+      id: go
+
+    - name: Check out code into the Go module directory
+      uses: actions/checkout@v2
+
+    - name: Build
+      run: make local
+
+    - name: Test
+      run: make test
+
+    - name: Publish container image
+      run: |
+        docker login -u ${{ secrets.DOCKER_USER }} -p ${{ secrets.DOCKER_PASSWORD }}
+        ./hack/docker-push.sh

.gitignore

@@ -28,6 +28,7 @@ debug
 
 /velero
 .idea/
+Tiltfile
 
 .container-*
 .vimrc
@@ -42,6 +43,7 @@ site/_site
 site/.sass-cache
 site/.jekyll
 site/.jekyll-metadata
+site/.jekyll-cache
 site/.bundle
 site/vendor
 .ruby-version

.goreleaser.yml

@@ -41,7 +41,7 @@ builds:
       - goos: windows
         goarch: ppc64le
     ldflags:
-      - -X "github.com/heptio/velero/pkg/buildinfo.Version={{ .Tag }}" -X "github.com/heptio/velero/pkg/buildinfo.GitSHA={{ .FullCommit }}" -X "github.com/heptio/velero/pkg/buildinfo.GitTreeState={{ .Env.GIT_TREE_STATE }}"
+      - -X "github.com/vmware-tanzu/velero/pkg/buildinfo.Version={{ .Tag }}" -X "github.com/vmware-tanzu/velero/pkg/buildinfo.GitSHA={{ .FullCommit }}" -X "github.com/vmware-tanzu/velero/pkg/buildinfo.GitTreeState={{ .Env.GIT_TREE_STATE }}"
 archives:
   - name_template: "{{ .ProjectName }}-{{ .Tag }}-{{ .Os }}-{{ .Arch }}"
     wrap_in_directory: true
@@ -52,7 +52,7 @@ checksum:
   name_template: 'CHECKSUM'
 release:
   github:
-    owner: heptio
+    owner: vmware-tanzu
     name: velero
   draft: true
   prerelease: auto

.travis.yml

@@ -1,11 +0,0 @@
-language: go
-
-go:
-  - 1.12.x
-
-sudo: required
-
-services:
-  - docker
-
-script: hack/ci-check.sh

ADOPTERS.md

@@ -0,0 +1,104 @@
+# Velero Adopters
+
+If you're using Velero and want to add your organization to this list, 
+[follow these directions][1]!
+
+<a href="https://www.bitgo.com" border="0" target="_blank"><img alt="bitgo.com" src="site/img/adopters/BitGo.svg" height="50"></a>&nbsp; &nbsp; &nbsp;
+<a href="https://www.nirmata.com" border="0" target="_blank"><img alt="nirmata.com" src="site/img/adopters/nirmata.svg" height="50"></a>&nbsp; &nbsp; &nbsp;
+<a href="https://kyma-project.io/" border="0" target="_blank"><img alt="kyma-project.io" src="site/img/adopters/kyma.svg" height="50"></a>&nbsp; &nbsp; &nbsp;
+<a href="https://redhat.com/" border="0" target="_blank"><img alt="redhat.com" src="site/img/adopters/redhat.svg" height="50"></a>&nbsp; &nbsp; &nbsp;
+<a href="https://dellemc.com/" border="0" target="_blank"><img alt="dellemc.com" src="site/img/adopters/DellEMC.png" height="50"></a>&nbsp; &nbsp; &nbsp;
+<a href="https://bugsnag.com/" border="0" target="_blank"><img alt="bugsnag.com" src="site/img/adopters/bugsnag.svg" height="50"></a>&nbsp; &nbsp; &nbsp;
+<a href="https://okteto.com/" border="0" target="_blank"><img alt="okteto.com" src="site/img/adopters/okteto.svg" height="50"></a>&nbsp; &nbsp; &nbsp;
+<a href="https://banzaicloud.com/" border="0" target="_blank"><img alt="banzaicloud.com" src="site/img/adopters/banzaicloud.svg" height="50"></a>&nbsp; &nbsp; &nbsp;
+<a href="https://sighup.io/" border="0" target="_blank"><img alt="sighup.io" src="site/img/adopters/sighup.svg" height="50"></a>&nbsp; &nbsp; &nbsp;
+<a href="https://mayadata.io/" border="0" target="_blank"><img alt="mayadata.io" src="site/img/adopters/mayadata.svg" height="50"></a>&nbsp; &nbsp; &nbsp;
+
+## Success Stories
+
+Below is a list of adopters of Velero in **production environments** that have
+publicly shared the details of how they use it.
+
+**[BitGo][20]**  
+BitGo uses Velero backup and restore capabilities to seamlessly provision and scale fullnode statefulsets on the fly as well as having it serve an integral piece for our kubernetes disaster-recovery story.
+
+**[Bugsnag][30]**  
+We use Velero for managing backups of an internal instance of our on-premise clustered solution. We also recommend our users of [on-premise Bugsnag installations][31] use Velero for [managing their own backups][32].
+
+**[Banzai Cloud][60]**  
+[Banzai Cloud Pipeline][61] is a Kubernetes-based microservices platform that integrates services needed for Day-1 and Day-2 operations along with first-class support both for on-prem and hybrid multi-cloud deployments. We use Velero to periodically [backup and restore these clusters in case of disasters][62].
+
+## Solutions built with Velero
+
+Below is a list of solutions where Velero is being used as a component.
+
+**[Nirmata][10]**  
+We have integrated our [solution with Velero][11] to provide our customers with out of box backup/DR.
+
+**[Kyma][40]**  
+Kyma [integrates with Velero][41] to effortlessly back up and restore Kyma clusters with all its resources. Velero capabilities allow Kyma users to define and run manual and scheduled backups in order to successfully handle a disaster-recovery scenario.
+
+**[Red Hat][50]**  
+Red Hat has developed the [Cluster Application Migration Tool][51] which uses [Velero and Restic][52] to drive the migration of applications between OpenShift clusters.
+
+**[Dell EMC][70]**  
+For Kubernetes environments, [PowerProtect Data Manager][71] leverages the Container Storage Interface (CSI) framework to take snapshots to back up the persistent data or the data that the application creates e.g. databases. [Dell EMC leverages Velero][72] to backup the namespace configuration files (also known as Namespace meta data) for enterprise grade data protection.
+
+**[SIGHUP][80]**  
+SIGHUP integrates Velero in its [Fury Kubernetes Distribution][81] providing predefined schedules and configurations to ensure an optimized disaster recovery experience.
+[Fury Kubernetes Disaster Recovery Module][82] is ready to be deployed into any Kubernetes cluster running anywhere.
+
+**[MayaData][90]**  
+MayaData is a large user of Velero as well as a contributor. MayaData offers a Data Agility platform called [OpenEBS Director][91], that helps customers confidently and easily manage stateful workloads in Kubernetes. Velero is one of the core software building block of the OpenEBS Director's [DMaaS or data migration as a service offering][92] used to enable data protection strategies.
+
+**[Okteto][93]**  
+Okteto integrates Velero in [Okteto Cloud][94] and [Okteto Enterprise][95] to periodically backup and restore our clusters for disaster recovery. Velero is also a core software building block to provide namespace cloning capabilities, a feature that allows our users cloning staging environments into their personal development namespace for providing production-like development environments.
+​
+## Adding your organization to the list of Velero Adopters
+
+If you are using Velero and would like to be included in the list of `Velero Adopters`, add an SVG version of your logo to the `site/img/adopters` directory in this repo and submit a [pull request][3] with your change. Name the image file something that reflects your company (e.g., if your company is called Acme, name the image acme.png). See this for an example [PR][4].
+
+### Adding a logo to velero.io
+
+If you would like to add your logo to a future `Adopters of Velero` section on [velero.io][2], follow the steps above to add your organization to the list of Velero Adopters. Our community will follow up and publish it to the [velero.io][2] website.
+
+[1]: #adding-a-logo-to-veleroio
+[2]: https://velero.io
+[3]: https://github.com/vmware-tanzu/velero/pulls
+[4]: https://github.com/vmware-tanzu/velero/pull/2242
+
+[10]: https://www.nirmata.com/2019/08/14/kubernetes-disaster-recovery-using-velero-and-nirmata/
+[11]: https://nirmata.com
+
+[20]: https://bitgo.com
+
+[30]: https://bugsnag.com
+[31]: https://www.bugsnag.com/on-premise
+[32]: https://docs.bugsnag.com/on-premise/clustered/backup-restore/
+
+[40]: https://kyma-project.io
+[41]: https://kyma-project.io/docs/components/backup/#overview-overview
+
+[50]: https://redhat.com
+[51]: https://github.com/fusor/mig-operator
+[52]: https://github.com/fusor/mig-operator/blob/master/docs/usage/2.md
+
+[60]: https://banzaicloud.com
+[61]: https://banzaicloud.com/products/pipeline/
+[62]: https://banzaicloud.com/blog/vault-backup-velero/
+
+[70]: https://dellemc.com
+[71]: https://dellemc.com/dataprotection
+[72]: https://www.dellemc.com/resources/en-us/asset/briefs-handouts/solutions/h18141-dellemc-dpd-kubernetes.pdf
+
+[80]: https://sighup.io
+[81]: https://github.com/sighupio/fury-distribution
+[82]: https://github.com/sighupio/fury-kubernetes-dr
+
+[90]: https://mayadata.io
+[91]: https://director.mayadata.io/
+[92]: https://help.mayadata.io/hc/en-us/articles/360033401591-DMaaS
+
+[93]: https://okteto.com
+[94]: https://cloud.okteto.com
+[95]: https://okteto.com/enterprise/

CHANGELOG.md

@@ -1,10 +1,13 @@
 ## Current release:
-  * [CHANGELOG-1.1.md][11]
+  * [CHANGELOG-1.4.md][14]
 
 ## Development release:
   * [Unreleased Changes][0]
 
 ## Older releases:
+  * [CHANGELOG-1.3.md][13]
+  * [CHANGELOG-1.2.md][12]
+  * [CHANGELOG-1.1.md][11]
   * [CHANGELOG-1.0.md][10]
   * [CHANGELOG-0.11.md][9]
   * [CHANGELOG-0.10.md][8]
@@ -17,15 +20,18 @@
   * [CHANGELOG-0.3.md][1]
 
 
-[11]: https://github.com/heptio/velero/blob/master/changelogs/CHANGELOG-1.1.md
-[10]: https://github.com/heptio/velero/blob/master/changelogs/CHANGELOG-1.0.md
-[9]: https://github.com/heptio/velero/blob/master/changelogs/CHANGELOG-0.11.md
-[8]: https://github.com/heptio/velero/blob/master/changelogs/CHANGELOG-0.10.md
-[7]: https://github.com/heptio/velero/blob/master/changelogs/CHANGELOG-0.9.md
-[6]: https://github.com/heptio/velero/blob/master/changelogs/CHANGELOG-0.8.md
-[5]: https://github.com/heptio/velero/blob/master/changelogs/CHANGELOG-0.7.md
-[4]: https://github.com/heptio/velero/blob/master/changelogs/CHANGELOG-0.6.md
-[3]: https://github.com/heptio/velero/blob/master/changelogs/CHANGELOG-0.5.md
-[2]: https://github.com/heptio/velero/blob/master/changelogs/CHANGELOG-0.4.md
-[1]: https://github.com/heptio/velero/blob/master/changelogs/CHANGELOG-0.3.md
-[0]: https://github.com/heptio/velero/blob/master/changelogs/unreleased
+[14]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-1.4.md
+[13]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-1.3.md
+[12]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-1.2.md
+[11]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-1.1.md
+[10]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-1.0.md
+[9]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-0.11.md
+[8]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-0.10.md
+[7]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-0.9.md
+[6]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-0.8.md
+[5]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-0.7.md
+[4]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-0.6.md
+[3]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-0.5.md
+[2]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-0.4.md
+[1]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-0.3.md
+[0]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/unreleased

CODE_OF_CONDUCT.md

@@ -1,37 +1,84 @@
-# Velero Community Code of Conduct
+# Contributor Covenant Code of Conduct
 
-## Contributor Code of Conduct
+## Our Pledge
 
-As contributors and maintainers of this project, and in the interest of fostering
-an open and welcoming community, we pledge to respect all people who contribute
-through reporting issues, posting feature requests, updating documentation,
-submitting pull requests or patches, and other activities.
+We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
 
-We are committed to making participation in this project a harassment-free experience for
-everyone, regardless of level of experience, gender, gender identity and expression,
-sexual orientation, disability, personal appearance, body size, race, ethnicity, age,
-religion, or nationality.
+We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
 
-Examples of unacceptable behavior by participants include:
+## Our Standards
 
-* The use of sexualized language or imagery
-* Personal attacks
-* Trolling or insulting/derogatory comments
+Examples of behavior that contributes to a positive environment for our community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
 * Public or private harassment
-* Publishing other's private information, such as physical or electronic addresses,
- without explicit permission
-* Other unethical or unprofessional conduct.
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
 
-Project maintainers have the right and responsibility to remove, edit, or reject
-comments, commits, code, wiki edits, issues, and other contributions that are not
-aligned to this Code of Conduct. By adopting this Code of Conduct, project maintainers
-commit themselves to fairly and consistently applying these principles to every aspect
-of managing this project. Project maintainers who do not follow or enforce the Code of
-Conduct may be permanently removed from the project team.
+## Enforcement Responsibilities
 
-This code of conduct applies both within project spaces and in public spaces
-when an individual is representing the project or its community.
+Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
 
-Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project maintainer(s).
+Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.
 
-This Code of Conduct is adapted from the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md) and [Contributor Covenant](http://contributor-covenant.org/version/1/2/0/), version 1.2.0.
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at [oss-coc@vmware.com](mailto:oss-coc@vmware.com). All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior,  harassment of an individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.0,
+available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at https://www.contributor-covenant.org/translations.

CONTRIBUTING.md

@@ -1,70 +1,3 @@
 # Contributing
 
-## CHANGELOG 
-
-Authors are expected to include a changelog file with their pull requests. The changelog file
-should be a new file created in the `changelogs/unreleased` folder. The file should follow the
-naming convention of `pr-username` and the contents of the file should be your text for the
-changelog.
-
-    velero/changelogs/unreleased   <- folder
-        000-username            <- file
-
-
-## DCO Sign off
-
-All authors to the project retain copyright to their work. However, to ensure
-that they are only submitting work that they have rights to, we are requiring
-everyone to acknowledge this by signing their work.
-
-Any copyright notices in this repo should specify the authors as "the Velero contributors".
-
-To sign your work, just add a line like this at the end of your commit message:
-
-```
-Signed-off-by: Joe Beda <joe@heptio.com>
-```
-
-This can easily be done with the `--signoff` option to `git commit`.
-
-By doing this you state that you can certify the following (from https://developercertificate.org/):
-
-```
-Developer Certificate of Origin
-Version 1.1
-
-Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
-1 Letterman Drive
-Suite D4700
-San Francisco, CA, 94129
-
-Everyone is permitted to copy and distribute verbatim copies of this
-license document, but changing it is not allowed.
-
-
-Developer's Certificate of Origin 1.1
-
-By making a contribution to this project, I certify that:
-
-(a) The contribution was created in whole or in part by me and I
-    have the right to submit it under the open source license
-    indicated in the file; or
-
-(b) The contribution is based upon previous work that, to the best
-    of my knowledge, is covered under an appropriate open source
-    license and I have the right under that license to submit that
-    work with modifications, whether created in whole or in part
-    by me, under the same open source license (unless I am
-    permitted to submit under a different license), as indicated
-    in the file; or
-
-(c) The contribution was provided directly to me by some other
-    person who certified (a), (b) or (c) and I have not modified
-    it.
-
-(d) I understand and agree that this project and the contribution
-    are public and that a record of the contribution (including all
-    personal information I submit with it, including my sign-off) is
-    maintained indefinitely and may be redistributed consistent with
-    this project or the open source license(s) involved.
-```
+Authors are expected to follow some guidelines when submitting PRs. Please see [our documentation](https://velero.io/docs/master/code-standards/) for details.

Dockerfile-fsfreeze-pause

@@ -1,19 +0,0 @@
-# Copyright 2018, 2019 the Velero contributors.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-FROM ubuntu:bionic
-
-LABEL maintainer="Steve Kriss <krisss@vmware.com>"
-
-ENTRYPOINT ["/bin/bash", "-c", "while true; do sleep 10000; done"]

Dockerfile-fsfreeze-pause-ppc64le

@@ -1,19 +0,0 @@
-# Copyright 2019 the Velero contributors.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-FROM ubuntu:bionic
-
-LABEL maintainer="Steve Kriss <krisss@vmware.com>"
-
-ENTRYPOINT ["/bin/bash", "-c", "while true; do sleep 10000; done"]

Dockerfile-velero

@@ -12,15 +12,15 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM ubuntu:bionic
+FROM ubuntu:focal
 
-LABEL maintainer="Steve Kriss <krisss@vmware.com>"
+LABEL maintainer="Nolan Brubaker <brubakern@vmware.com>"
 
 RUN apt-get update && \
     apt-get install -y --no-install-recommends ca-certificates wget bzip2 && \
-    wget --quiet https://github.com/restic/restic/releases/download/v0.9.4/restic_0.9.4_linux_amd64.bz2 && \
-    bunzip2 restic_0.9.4_linux_amd64.bz2 && \
-    mv restic_0.9.4_linux_amd64 /usr/bin/restic && \
+    wget --quiet https://github.com/restic/restic/releases/download/v0.9.6/restic_0.9.6_linux_amd64.bz2 && \
+    bunzip2 restic_0.9.6_linux_amd64.bz2 && \
+    mv restic_0.9.6_linux_amd64 /usr/bin/restic && \
     chmod +x /usr/bin/restic && \
     apt-get remove -y wget bzip2 && \
     rm -rf /var/lib/apt/lists/*
@@ -28,6 +28,6 @@ RUN apt-get update && \
 
 ADD /bin/linux/amd64/velero /velero
 
-USER nobody:nobody
+USER nobody:nogroup
 
 ENTRYPOINT ["/velero"]

Dockerfile-velero-arm

@@ -0,0 +1,23 @@
+# Copyright 2020 the Velero contributors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM arm32v7/ubuntu:focal
+
+ADD /bin/linux/arm/restic /usr/bin/restic
+
+ADD /bin/linux/arm/velero /velero
+
+USER nobody:nogroup
+
+ENTRYPOINT ["/velero"]

Dockerfile-velero-arm64

@@ -0,0 +1,23 @@
+# Copyright 2020 the Velero contributors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM arm64v8/ubuntu:focal
+
+ADD /bin/linux/arm64/restic /usr/bin/restic
+
+ADD /bin/linux/arm64/velero /velero
+
+USER nobody:nogroup
+
+ENTRYPOINT ["/velero"]

Dockerfile-velero-ppc64le

@@ -12,21 +12,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM ubuntu:bionic
+FROM ppc64le/ubuntu:focal
 
-LABEL maintainer="Steve Kriss <krisss@vmware.com>"
-
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends ca-certificates wget && \
-    wget --quiet https://oplab9.parqtec.unicamp.br/pub/ppc64el/restic/restic-0.9.4 && \
-    mv restic-0.9.4 /usr/bin/restic && \
-    chmod +x /usr/bin/restic && \
-    apt-get remove -y wget && \
-    rm -rf /var/lib/apt/lists/*
+LABEL maintainer="Prajyot Parab <prajyot.parab@ibm.com>"
 
+ADD /bin/linux/ppc64le/restic /usr/bin/restic
 
 ADD /bin/linux/ppc64le/velero /velero
 
-USER nobody:nobody
+USER nobody:nogroup
 
 ENTRYPOINT ["/velero"]

Dockerfile-velero-restic-restore-helper

@@ -12,12 +12,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM ubuntu:bionic
+FROM ubuntu:focal
 
-LABEL maintainer="Steve Kriss <krisss@vmware.com>"
+LABEL maintainer="Nolan Brubaker <brubakern@vmware.com>"
 
 ADD /bin/linux/amd64/velero-restic-restore-helper .
 
-USER nobody:nobody
+USER nobody:nogroup
 
 ENTRYPOINT [ "/velero-restic-restore-helper" ]

Dockerfile-velero-restic-restore-helper-arm

@@ -0,0 +1,21 @@
+# Copyright 2020 the Velero contributors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM arm32v7/ubuntu:focal
+
+ADD /bin/linux/arm/velero-restic-restore-helper .
+
+USER nobody:nogroup
+
+ENTRYPOINT [ "/velero-restic-restore-helper" ]

Dockerfile-velero-restic-restore-helper-arm64

@@ -0,0 +1,21 @@
+# Copyright 2020 the Velero contributors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM arm64v8/ubuntu:focal
+
+ADD /bin/linux/arm64/velero-restic-restore-helper .
+
+USER nobody:nogroup
+
+ENTRYPOINT [ "/velero-restic-restore-helper" ]

Dockerfile-velero-restic-restore-helper-ppc64le

@@ -12,12 +12,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM ubuntu:bionic
+FROM ppc64le/ubuntu:focal
 
-LABEL maintainer="Steve Kriss <krisss@vmware.com>"
+LABEL maintainer="Prajyot Parab <prajyot.parab@ibm.com>"
 
 ADD /bin/linux/ppc64le/velero-restic-restore-helper .
 
-USER nobody:nobody
+USER nobody:nogroup
 
 ENTRYPOINT [ "/velero-restic-restore-helper" ]

GOVERNANCE.md

@@ -0,0 +1,112 @@
+# Velero Governance
+
+This document defines the project governance for Velero.
+
+## Overview
+
+**Velero**, an open source project, is committed to building an open, inclusive, productive and self-governing open source community focused on building a high quality tool that enables users to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes. The community is governed by this document with the goal of defining how community should work together to achieve this goal.
+
+## Code Repositories
+
+The following code repositories are governed by Velero community and maintained under the `vmware-tanzu\Velero` organization.
+
+* **[Velero](https://github.com/vmware-tanzu/velero):** Main Velero codebase
+* **[Helm Chart](https://github.com/vmware-tanzu/helm-charts/tree/master/charts/velero):** The Helm chart for the Velero server component
+* **[Velero CSI Plugin](https://github.com/vmware-tanzu/velero-plugin-for-csi):** This repository contains Velero plugins for snapshotting CSI backed PVCs using the CSI beta snapshot APIs
+* **[Velero Plugin for vSphere](https://github.com/vmware-tanzu/velero-plugin-for-vsphere):** This repository contains the Velero Plugin for vSphere. This plugin is a volume snapshotter plugin that provides crash-consistent snapshots of vSphere block volumes and backup of volume data into S3 compatible storage.
+* **[Velero Plugin for AWS](https://github.com/vmware-tanzu/velero-plugin-for-aws):** This repository contains the plugins to support running Velero on AWS, including the object store plugin and the volume snapshotter plugin
+* **[Velero Plugin for GCP](https://github.com/vmware-tanzu/velero-plugin-for-gcp):** This repository contains the plugins to support running Velero on GCP, including the object store plugin and the volume snapshotter plugin
+* **[Velero Plugin for Azure](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure):** This repository contains the plugins to support running Velero on Azure, including the object store plugin and the volume snapshotter plugin
+* **[Velero Plugin Example](https://github.com/vmware-tanzu/velero-plugin-example):** This repository contains example plugins for Velero
+
+
+## Community Roles
+
+* **Users:** Members that engage with the Velero community via any medium (Slack, GitHub, mailing lists, etc.).
+* **Contributors:** Regular contributions to projects (documentation, code reviews, responding to issues, participation in proposal discussions, contributing code, etc.). 
+* **Maintainers**: The Velero project leaders. They are responsible for the overall health and direction of the project; final reviewers of PRs and responsible for releases. Some Maintainers are responsible for one or more components within a project, acting as technical leads for that component. Maintainers are expected to contribute code and documentation, review PRs including ensuring quality of code, triage issues, proactively fix bugs, and perform maintenance tasks for these components.
+
+### Maintainers
+
+New maintainers must be nominated by an existing maintainer and must be elected by a supermajority of existing maintainers. Likewise, maintainers can be removed by a supermajority of the existing maintainers or can resign by notifying one of the maintainers.
+
+### Supermajority
+
+A supermajority is defined as two-thirds of members in the group.
+A supermajority of [Maintainers](#maintainers) is required for certain
+decisions as outlined above. A supermajority vote is equivalent to the number of votes in favor being at least twice the number of votes against. For example, if you have 5 maintainers, a supermajority vote is 4 votes. Voting on decisions can happen on the mailing list, GitHub, Slack, email, or via a voting service, when appropriate. Maintainers can either vote "agree, yes, +1", "disagree, no, -1", or "abstain". A vote passes when supermajority is met. An abstain vote equals not voting at all.
+
+### Decision Making
+
+Ideally, all project decisions are resolved by consensus. If impossible, any
+maintainer may call a vote. Unless otherwise specified in this document, any
+vote will be decided by a supermajority of maintainers.
+
+Votes by maintainers belonging to the same company
+will count as one vote; e.g., 4 maintainers employed by fictional company **Valerium** will
+only have **one** combined vote. If voting members from a given company do not
+agree, the company's vote is determined by a supermajority of voters from that
+company. If no supermajority is achieved, the company is considered to have
+abstained.
+
+## Proposal Process
+
+One of the most important aspects in any open source community is the concept
+of proposals. Large changes to the codebase and / or new features should be
+preceded by a proposal in our community repo. This process allows for all
+members of the community to weigh in on the concept (including the technical
+details), share their comments and ideas, and offer to help. It also ensures
+that members are not duplicating work or inadvertently stepping on toes by
+making large conflicting changes.
+
+The project roadmap is defined by accepted proposals.
+
+Proposals should cover the high-level objectives, use cases, and technical
+recommendations on how to implement. In general, the community member(s)
+interested in implementing the proposal should be either deeply engaged in the
+proposal process or be an author of the proposal.
+
+The proposal should be documented as a separated markdown file pushed to the root of the 
+`design` folder in the [Velero](https://github.com/vmware-tanzu/velero/tree/master/design)
+repository via PR. The name of the file should follow the name pattern `<short
+meaningful words joined by '-'>_design.md`, e.g:
+`restore-hooks-design.md`.
+
+Use the [Proposal Template](https://github.com/vmware-tanzu/velero/blob/master/design/_template.md) as a starting point.
+
+### Proposal Lifecycle
+
+The proposal PR can follow the GitHub lifecycle of the PR to indicate its status:
+
+* **Open**: Proposal is created and under review and discussion.
+* **Merged**: Proposal has been reviewed and is accepted (either by consensus or through a vote).
+* **Closed**: Proposal has been reviewed and was rejected (either by consensus or through a vote).
+
+## Lazy Consensus
+
+To maintain velocity in a project as busy as Velero, the concept of [Lazy
+Consensus](http://en.osswiki.info/concepts/lazy_consensus) is practiced. Ideas
+and / or proposals should be shared by maintainers via
+GitHub with the appropriate maintainer groups (e.g.,
+`@vmware-tanzu/velero-maintainers`) tagged. Out of respect for other contributors,
+major changes should also be accompanied by a ping on Slack or a note on the
+Velero mailing list as appropriate. Author(s) of proposal, Pull Requests,
+issues, etc.  will give a time period of no less than five (5) working days for
+comment and remain cognizant of popular observed world holidays.
+
+Other maintainers may chime in and request additional time for review, but
+should remain cognizant of blocking progress and abstain from delaying
+progress unless absolutely needed. The expectation is that blocking progress
+is accompanied by a guarantee to review and respond to the relevant action(s)
+(proposals, PRs, issues, etc.) in short order.
+
+Lazy Consensus is practiced for all projects in the `Velero` org, including
+the main project repository and the additional repositories.
+
+Lazy consensus does _not_ apply to the process of:
+
+* Removal of maintainers from Velero
+
+## Updating Governance
+
+All substantive changes in Governance require a supermajority agreement by all maintainers.

Gopkg.toml

@@ -1,134 +0,0 @@
-
-# Gopkg.toml example
-#
-# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
-# for detailed Gopkg.toml documentation.
-#
-# required = ["github.com/user/thing/cmd/thing"]
-# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
-#
-# [[constraint]]
-#   name = "github.com/user/project"
-#   version = "1.0.0"
-#
-# [[constraint]]
-#   name = "github.com/user/project2"
-#   branch = "dev"
-#   source = "github.com/myfork/project2"
-#
-# [[override]]
-#  name = "github.com/x/y"
-#  version = "2.4.0"
-
-[prune]
-  unused-packages = true
-  non-go = true
-  go-tests = true
-
-#
-# Kubernetes packages
-#
-
-[[constraint]]
-  name = "k8s.io/kubernetes"
-  version = "~1.14"
-
-[[constraint]]
-  name = "k8s.io/client-go"
-  version = "kubernetes-1.14.0"
-
-[[constraint]]
-  name = "k8s.io/apimachinery"
-  version = "kubernetes-1.14.0"
-
-[[constraint]]
-  name = "k8s.io/api"
-  version = "kubernetes-1.14.0"
-
-[[constraint]]
-  name = "k8s.io/apiextensions-apiserver"
-  version = "kubernetes-1.14.0"
-
-# k8s.io/client-go kubernetes-1.14.0 uses v1.1.4
-[[override]]
-  name = "github.com/json-iterator/go"
-  version = "=1.1.4"
-
-#
-# Cloud provider packages
-#
-[[constraint]]
-  name = "github.com/aws/aws-sdk-go"
-  version = "1.13.12"
-
-[[constraint]]
-  name = "github.com/Azure/azure-sdk-for-go"
-  version = "~21.4.0"
-
-# k8s.io/client-go kubernetes-1.14.0 uses v11.1.2
-[[constraint]]
-  name = "github.com/Azure/go-autorest"
-  version = "11.1.2"
-
-[[constraint]]
-  name = "cloud.google.com/go"
-  version = "0.11.0"
-
-[[constraint]]
-  name = "google.golang.org/api"
-  version = "~v0.3.2"
-
-[[constraint]]
-  name = "golang.org/x/oauth2"
-  branch = "master"
-
-#
-# Third party packages
-#
-[[constraint]]
-  name = "github.com/robfig/cron"
-  revision = "df38d32658d8788cd446ba74db4bb5375c4b0cb3"
-
-[[constraint]]
-  name = "github.com/satori/go.uuid"
-  version = "~1.2.0"
-
-[[constraint]]
-  name = "github.com/spf13/afero"
-  branch = "master"
-
-[[constraint]]
-  name = "github.com/spf13/cobra"
-  version = "0.0.3"
-
-[[constraint]]
-  name = "github.com/spf13/pflag"
-  version = "1.0.2"
-
-[[constraint]]
-  name = "github.com/stretchr/testify"
-  version = "~1.2.2"
-
-[[constraint]]
-  name = "github.com/hashicorp/go-plugin"
-  revision = "a1bc61569a26c0f65865932c0d55743b0567c494"
-
-[[constraint]]
-  name = "github.com/golang/protobuf"
-  version = "~v1.3.1"
-
-[[constraint]]
-  name = "google.golang.org/grpc"
-  version = "~v1.19.0"
-
-[[constraint]]
-  name = "github.com/joho/godotenv"
-  version = "~v1.3.0"
-
-[[constraint]]
-  name = "github.com/gobwas/glob"
-  version = "~v0.2.3"
-
-[[override]]
-  name = "golang.org/x/sys"
-  branch = "master"

MAINTAINERS.md

@@ -0,0 +1,27 @@
+# Velero Maintainers
+
+[GOVERNANCE.md](https://github.com/vmware-tanzu/velero/blob/master/GOVERNANCE.md) describes governance guidelines and maintainer responsibilities.
+
+## Maintainers
+
+| Maintainer | GitHub ID | Affiliation |
+| --------------- | --------- | ----------- |
+| Carlisia Campos | [carlisia](https://github.com/carlisia) | [VMware](https://www.github.com/vmware/) |
+| Nolan Brubaker | [nrb](https://github.com/nrb) | [VMware](https://www.github.com/vmware/) |
+| Ashish Amarnath | [ashish-amarnath](https://github.com/ashish-amarnath) | [VMware](https://www.github.com/vmware/) |
+| Stephanie Bauman | [stephbman](https://github.com/stephbman) | [VMware](https://www.github.com/vmware/) |
+
+## Emeritus Maintainers
+* Adnan Abdulhussein ([prydonius](https://github.com/prydonius))
+* Andy Goldstein ([ncdc](https://github.com/ncdc))
+* Steve Kriss ([skriss](https://github.com/skriss))
+
+## Velero Contributors & Stakeholders
+
+| Feature Area | Lead |
+| ----------------------------- | :---------------------: |
+| Technical Lead | Nolan Brubaker (nrb) |
+| Kubernetes CSI Liaison | Nolan Brubaker (nrb), Ashish Amarnath (ashish-amarnath) |
+| Deployment | Carlisia Campos (carlisia), Carlos Tadeu Panato Junior (cpanato) | 
+| Community Management | Jonas Rosland (jonasrosland) |
+| Product Management | Stephanie Bauman (stephbman) |

Makefile

@@ -1,6 +1,6 @@
 # Copyright 2016 The Kubernetes Authors.
 #
-# Modifications Copyright 2017 the Velero contributors.
+# Modifications Copyright 2020 the Velero contributors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,10 +18,20 @@
 BIN ?= velero
 
 # This repo's root import path (under GOPATH).
-PKG := github.com/heptio/velero
+PKG := github.com/vmware-tanzu/velero
 
 # Where to push the docker image.
-REGISTRY ?= gcr.io/heptio-images
+REGISTRY ?= velero
+
+# Build image handling. We push a build image for every changed version of 
+# /hack/build-image/Dockerfile. We tag the dockerfile with the short commit hash
+# of the commit that changed it. When determining if there is a build image in
+# the registry to use we look for one that matches the current "commit" for the
+# Dockerfile else we make one.
+
+BUILDER_IMAGE_TAG := $(shell git log -1 --pretty=%h hack/build-image/Dockerfile)
+BUILDER_IMAGE := $(REGISTRY)/build-image:$(BUILDER_IMAGE_TAG)
+BUILDER_IMAGE_CACHED := $(shell docker images -q ${BUILDER_IMAGE} 2>/dev/null )
 
 # Which architecture to build - see $(ALL_ARCH) for options.
 # if the 'local' rule is being run, detect the ARCH from 'go env'
@@ -33,36 +43,60 @@ VERSION ?= master
 
 TAG_LATEST ?= false
 
+# The version of restic binary to be downloaded for power architecture
+RESTIC_VERSION ?= 0.9.6
+
+CLI_PLATFORMS ?= linux-amd64 linux-arm linux-arm64 darwin-amd64 windows-amd64 linux-ppc64le
+CONTAINER_PLATFORMS ?= linux-amd64 linux-ppc64le linux-arm linux-arm64
+MANIFEST_PLATFORMS ?= amd64 ppc64le arm arm64
+
+# set git sha and tree state
+GIT_SHA = $(shell git rev-parse HEAD)
+GIT_DIRTY = $(shell git status --porcelain 2> /dev/null)
+
+# The default linters used by lint and local-lint
+LINTERS ?= "gosec,goconst,gofmt,goimports,unparam"
+
 ###
 ### These variables should not need tweaking.
 ###
 
-CLI_PLATFORMS := linux-amd64 linux-arm linux-arm64 darwin-amd64 windows-amd64 linux-ppc64le
-CONTAINER_PLATFORMS := linux-amd64 linux-arm linux-arm64 linux-ppc64le
-
 platform_temp = $(subst -, ,$(ARCH))
 GOOS = $(word 1, $(platform_temp))
 GOARCH = $(word 2, $(platform_temp))
 
-# TODO(ncdc): support multiple image architectures once gcr.io supports manifest lists
 # Set default base image dynamically for each arch
 ifeq ($(GOARCH),amd64)
 		DOCKERFILE ?= Dockerfile-$(BIN)
+local-arch:
+	@echo "local environment for amd64 is up-to-date"
+endif
+ifeq ($(GOARCH),arm)
+		DOCKERFILE ?= Dockerfile-$(BIN)-arm
+local-arch:
+	@mkdir -p _output/bin/linux/arm/
+	@wget -q -O - https://github.com/restic/restic/releases/download/v$(RESTIC_VERSION)/restic_$(RESTIC_VERSION)_linux_arm.bz2 | bunzip2 > _output/bin/linux/arm/restic
+	@chmod a+x _output/bin/linux/arm/restic
+endif
+ifeq ($(GOARCH),arm64)
+		DOCKERFILE ?= Dockerfile-$(BIN)-arm64
+local-arch:
+	@mkdir -p _output/bin/linux/arm64/
+	@wget -q -O - https://github.com/restic/restic/releases/download/v$(RESTIC_VERSION)/restic_$(RESTIC_VERSION)_linux_arm64.bz2 | bunzip2 > _output/bin/linux/arm64/restic
+	@chmod a+x _output/bin/linux/arm64/restic
 endif
-#ifeq ($(GOARCH),arm)
-#		DOCKERFILE ?= Dockerfile.arm #armel/busybox
-#endif
-#ifeq ($(GOARCH),arm64)
-#		DOCKERFILE ?= Dockerfile.arm64 #aarch64/busybox
-#endif
 ifeq ($(GOARCH),ppc64le)
                 DOCKERFILE ?= Dockerfile-$(BIN)-ppc64le
+local-arch:
+	RESTIC_VERSION=$(RESTIC_VERSION) \
+        ./hack/get-restic-ppc64le.sh
 endif
 
-IMAGE = $(REGISTRY)/$(BIN)
+MULTIARCH_IMAGE = $(REGISTRY)/$(BIN)
+IMAGE ?= $(REGISTRY)/$(BIN)-$(GOARCH)
 
 # If you want to build all binaries, see the 'all-build' rule.
-# If you want to build all containers, see the 'all-container' rule.
+# If you want to build all containers, see the 'all-containers' rule.
 # If you want to build AND push all containers, see the 'all-push' rule.
 all:
 	@$(MAKE) build
@@ -70,18 +104,25 @@ all:
 
 build-%:
 	@$(MAKE) --no-print-directory ARCH=$* build
+	@$(MAKE) --no-print-directory ARCH=$* build BIN=velero-restic-restore-helper
 
-#container-%:
-#	@$(MAKE) --no-print-directory ARCH=$* container
+container-%:
+	@$(MAKE) --no-print-directory ARCH=$* container
+	@$(MAKE) --no-print-directory ARCH=$* container BIN=velero-restic-restore-helper
 
-#push-%:
-#	@$(MAKE) --no-print-directory ARCH=$* push
+push-%:
+	@$(MAKE) --no-print-directory ARCH=$* push
+	@$(MAKE) --no-print-directory ARCH=$* push BIN=velero-restic-restore-helper
 
 all-build: $(addprefix build-, $(CLI_PLATFORMS))
 
-#all-container: $(addprefix container-, $(CONTAINER_PLATFORMS))
+all-containers: $(addprefix container-, $(CONTAINER_PLATFORMS))
 
-#all-push: $(addprefix push-, $(CONTAINER_PLATFORMS))
+all-push: $(addprefix push-, $(CONTAINER_PLATFORMS))
+
+all-manifests:
+	@$(MAKE) manifest
+	@$(MAKE) manifest BIN=velero-restic-restore-helper
 
 local: build-dirs
 	GOOS=$(GOOS) \
@@ -89,6 +130,8 @@ local: build-dirs
 	VERSION=$(VERSION) \
 	PKG=$(PKG) \
 	BIN=$(BIN) \
+	GIT_SHA=$(GIT_SHA) \
+	GIT_DIRTY="$(GIT_DIRTY)" \
 	OUTPUT_DIR=$$(pwd)/_output/bin/$(GOOS)/$(GOARCH) \
 	./hack/build.sh
 
@@ -102,57 +145,43 @@ _output/bin/$(GOOS)/$(GOARCH)/$(BIN): build-dirs
 		VERSION=$(VERSION) \
 		PKG=$(PKG) \
 		BIN=$(BIN) \
+		GIT_SHA=$(GIT_SHA) \
+		GIT_DIRTY=\"$(GIT_DIRTY)\" \
 		OUTPUT_DIR=/output/$(GOOS)/$(GOARCH) \
 		./hack/build.sh'"
 
 TTY := $(shell tty -s && echo "-t")
 
-BUILDER_IMAGE := velero-builder
 
 # Example: make shell CMD="date > datefile"
-shell: build-dirs build-image
-	@# the volume bind-mount of $PWD/vendor/k8s.io/api is needed for code-gen to
-	@# function correctly (ref. https://github.com/kubernetes/kubernetes/pull/64567)
+shell: build-dirs build-env
+	@# bind-mount the Velero root dir in at /github.com/vmware-tanzu/velero
+	@# because the Kubernetes code-generator tools require the project to
+	@# exist in a directory hierarchy ending like this (but *NOT* necessarily
+	@# under $GOPATH).
 	@docker run \
 		-e GOFLAGS \
 		-i $(TTY) \
 		--rm \
 		-u $$(id -u):$$(id -g) \
-		-v "$$(pwd)/vendor/k8s.io/api:/go/src/k8s.io/api:delegated" \
+		-v "$$(pwd):/github.com/vmware-tanzu/velero:delegated" \
+		-v "$$(pwd)/_output/bin:/output:delegated" \
 		-v "$$(pwd)/.go/pkg:/go/pkg:delegated" \
 		-v "$$(pwd)/.go/std:/go/std:delegated" \
-		-v "$$(pwd):/go/src/$(PKG):delegated" \
-		-v "$$(pwd)/_output/bin:/output:delegated" \
 		-v "$$(pwd)/.go/std/$(GOOS)/$(GOARCH):/usr/local/go/pkg/$(GOOS)_$(GOARCH)_static:delegated" \
 		-v "$$(pwd)/.go/go-build:/.cache/go-build:delegated" \
-		-w /go/src/$(PKG) \
+		-v "$$(pwd)/.go/golangci-lint:/.cache/golangci-lint:delegated" \
+		-w /github.com/vmware-tanzu/velero \
 		$(BUILDER_IMAGE) \
 		/bin/sh $(CMD)
 
 DOTFILE_IMAGE = $(subst :,_,$(subst /,_,$(IMAGE))-$(VERSION))
 
-# Use a slightly customized build/push targets since we don't have a Go binary to build for the fsfreeze image
-build-fsfreeze: BIN = fsfreeze-pause
-build-fsfreeze:
-	@cp $(DOCKERFILE)  _output/.dockerfile-$(BIN).alpine
-	@docker build --pull -t $(IMAGE):$(VERSION) -f _output/.dockerfile-$(BIN).alpine _output
-	@docker images -q $(IMAGE):$(VERSION) > .container-$(DOTFILE_IMAGE)
-
-push-fsfreeze: BIN = fsfreeze-pause
-push-fsfreeze:
-	@docker push $(IMAGE):$(VERSION)
-ifeq ($(TAG_LATEST), true)
-	docker tag $(IMAGE):$(VERSION) $(IMAGE):latest
-	docker push $(IMAGE):latest
-endif
-	@docker images -q $(REGISTRY)/fsfreeze-pause:$(VERSION) > .container-$(DOTFILE_IMAGE)
-
 all-containers:
 	$(MAKE) container
 	$(MAKE) container BIN=velero-restic-restore-helper
-	$(MAKE) build-fsfreeze
 
-container: verify test .container-$(DOTFILE_IMAGE) container-name
+container: local-arch .container-$(DOTFILE_IMAGE) container-name
 .container-$(DOTFILE_IMAGE): _output/bin/$(GOOS)/$(GOARCH)/$(BIN) $(DOCKERFILE)
 	@cp $(DOCKERFILE) _output/.dockerfile-$(BIN)-$(GOOS)-$(GOARCH)
 	@docker build --pull -t $(IMAGE):$(VERSION) -f _output/.dockerfile-$(BIN)-$(GOOS)-$(GOARCH) _output
@@ -161,12 +190,6 @@ container: verify test .container-$(DOTFILE_IMAGE) container-name
 container-name:
 	@echo "container: $(IMAGE):$(VERSION)"
 
-all-push:
-	$(MAKE) push
-	$(MAKE) push BIN=velero-restic-restore-helper
-	$(MAKE) push-fsfreeze
-
-
 push: .push-$(DOTFILE_IMAGE) push-name
 .push-$(DOTFILE_IMAGE): .container-$(DOTFILE_IMAGE)
 	@docker push $(IMAGE):$(VERSION)
@@ -179,6 +202,20 @@ endif
 push-name:
 	@echo "pushed: $(IMAGE):$(VERSION)"
 
+manifest: .manifest-$(MULTIARCH_IMAGE) manifest-name
+.manifest-$(MULTIARCH_IMAGE):
+	@DOCKER_CLI_EXPERIMENTAL=enabled docker manifest create $(MULTIARCH_IMAGE):$(VERSION) \
+		$(foreach arch, $(MANIFEST_PLATFORMS), $(MULTIARCH_IMAGE)-$(arch):$(VERSION))
+	@DOCKER_CLI_EXPERIMENTAL=enabled docker manifest push --purge $(MULTIARCH_IMAGE):$(VERSION)
+ifeq ($(TAG_LATEST), true)
+	@DOCKER_CLI_EXPERIMENTAL=enabled docker manifest create $(MULTIARCH_IMAGE):latest \
+		$(foreach arch, $(MANIFEST_PLATFORMS), $(MULTIARCH_IMAGE)-$(arch):latest)
+	@DOCKER_CLI_EXPERIMENTAL=enabled docker manifest push --purge $(MULTIARCH_IMAGE):latest
+endif
+
+manifest-name:
+	@echo "pushed: $(MULTIARCH_IMAGE):$(VERSION)"
+
 SKIP_TESTS ?=
 test: build-dirs
 ifneq ($(SKIP_TESTS), 1)
@@ -195,28 +232,113 @@ ifneq ($(SKIP_TESTS), 1)
 	@$(MAKE) shell CMD="-c 'hack/verify-all.sh'"
 endif
 
+lint:
+ifneq ($(SKIP_TESTS), 1)
+	@$(MAKE) shell CMD="-c 'hack/lint.sh $(LINTERS)'"
+endif
+
+local-lint:
+ifneq ($(SKIP_TESTS), 1)
+	@hack/lint.sh $(LINTERS)
+endif
+
+lint-all:
+ifneq ($(SKIP_TESTS), 1)
+	@$(MAKE) shell CMD="-c 'hack/lint.sh $(LINTERS) true'"
+endif
+
+local-lint-all:
+ifneq ($(SKIP_TESTS), 1)
+	@hack/lint.sh $(LINTERS) true
+endif
+
 update:
 	@$(MAKE) shell CMD="-c 'hack/update-all.sh'"
 
 build-dirs:
 	@mkdir -p _output/bin/$(GOOS)/$(GOARCH)
-	@mkdir -p .go/src/$(PKG) .go/pkg .go/bin .go/std/$(GOOS)/$(GOARCH) .go/go-build
+	@mkdir -p .go/src/$(PKG) .go/pkg .go/bin .go/std/$(GOOS)/$(GOARCH) .go/go-build .go/golangci-lint
+
+build-env:
+	@# if we detect changes in dockerfile force a new build-image 
+	@# else if we dont have a cached image make one
+	@# finally use the cached image
+ifneq ($(shell git diff --quiet HEAD -- hack/build-image/Dockerfile; echo $$?), 0)
+	@echo "Local changes detected in hack/build-image/Dockerfile"
+	@echo "Preparing a new builder-image"
+	@make build-image
+else ifneq ($(BUILDER_IMAGE_CACHED),)
+	@echo "Using Cached Image: $(BUILDER_IMAGE)"
+else
+	@echo "Trying to pull build-image: $(BUILDER_IMAGE)"
+	docker pull -q $(BUILDER_IMAGE) || make build-image
+endif
 
 build-image:
-	cd hack/build-image && docker build --pull -t $(BUILDER_IMAGE) .
+	@# When we build a new image we just untag the old one.
+	@# This makes sure we don't leave the orphaned image behind.
+	@id=$$(docker image inspect  --format '{{ .ID }}' ${BUILDER_IMAGE} 2>/dev/null); \
+	cd hack/build-image && docker build --pull -t $(BUILDER_IMAGE) . ; \
+	new_id=$$(docker image inspect  --format '{{ .ID }}' ${BUILDER_IMAGE} 2>/dev/null); \
+	if [ "$$id" != "" ] && [ "$$id" != "$$new_id" ]; then \
+		docker rmi -f $$id || true; \
+	fi
+
+push-build-image:
+	@# this target will push the build-image it assumes you already have docker
+	@# credentials needed to accomplish this.
+	docker push $(BUILDER_IMAGE)
 
 clean:
+# if we have a cached image then use it to run go clean --modcache
+# this test checks if we there is an image id in the BUILDER_IMAGE_CACHED variable.
+ifneq ($(strip $(BUILDER_IMAGE_CACHED)),)
+	$(MAKE) shell CMD="-c 'go clean --modcache'"
+	docker rmi -f $(BUILDER_IMAGE) || true
+endif
 	rm -rf .container-* _output/.dockerfile-* .push-*
 	rm -rf .go _output
-	docker rmi $(BUILDER_IMAGE)
 
-ci: all verify test
+
+.PHONY: modules
+modules:
+	go mod tidy
+
+
+.PHONY: verify-modules
+verify-modules: modules
+	@if !(git diff --quiet HEAD -- go.sum go.mod); then \
+		echo "go module files are out of date, please commit the changes to go.mod and go.sum"; exit 1; \
+	fi
+
+
+ci: verify-modules verify all test
+
 
 changelog:
 	hack/changelog.sh
 
+# release builds a GitHub release using goreleaser within the build container.
+#
+# To dry-run the release, which will build the binaries/artifacts locally but
+# will *not* create a GitHub release:
+#		GITHUB_TOKEN=an-invalid-token-so-you-dont-accidentally-push-release \
+#		RELEASE_NOTES_FILE=changelogs/CHANGELOG-1.2.md \
+#		PUBLISH=false \
+#		make release
+#
+# To run the release, which will publish a *DRAFT* GitHub release in github.com/vmware-tanzu/velero 
+# (you still need to review/publish the GitHub release manually):
+#		GITHUB_TOKEN=your-github-token \ 
+#		RELEASE_NOTES_FILE=changelogs/CHANGELOG-1.2.md \
+#		PUBLISH=true \
+#		make release
 release:
-	hack/goreleaser.sh
+	$(MAKE) shell CMD="-c '\
+		GITHUB_TOKEN=$(GITHUB_TOKEN) \
+		RELEASE_NOTES_FILE=$(RELEASE_NOTES_FILE) \
+		PUBLISH=$(PUBLISH) \
+		./hack/goreleaser.sh'"
 
 serve-docs:
 	docker run \
@@ -245,7 +367,7 @@ serve-docs:
 # tagged version. Once the unstaged changes are ready, they can be added to the
 # staging area using 'git add' and then committed.
 #
-# To run gen-docs: "NEW_DOCS_VERSION=v1.1.0 make gen-docs"
+# To run gen-docs: "NEW_DOCS_VERSION=v1.4 VELERO_VERSION=v1.4.0 make gen-docs"
 #
 # **NOTE**: there are additional manual steps required to finalize the process of generating
 # a new versioned docs site. The full process is documented in site/README-JEKYLL.md.

PROJECT

@@ -0,0 +1,7 @@
+domain: io
+repo: github.com/vmware-tanzu/velero
+resources:
+- group: velero
+  kind: BackupStorageLocation
+  version: v1
+version: "2"

README.md

@@ -4,7 +4,7 @@
 
 ## Overview
 
-Velero (formerly Heptio Ark) gives you tools to back up and restore your Kubernetes cluster resources and persistent volumes. Velero lets you:
+Velero (formerly Heptio Ark) gives you tools to back up and restore your Kubernetes cluster resources and persistent volumes. You can run Velero with a public cloud platform or on-premises. Velero lets you:
 
 * Take backups of your cluster and restore in case of loss.
 * Migrate cluster resources to other clusters.
@@ -15,18 +15,9 @@ Velero consists of:
 * A server that runs on your cluster
 * A command-line client that runs locally
 
-You can run Velero in clusters on a cloud provider or on-premises. For detailed information, see [Compatible Storage Providers][99].
+## Documentation
 
-## Installation
-
-We strongly recommend that you use an [official release][6] of Velero. The tarballs for each release contain the
-`velero` command-line client. Follow the [installation instructions][28] to get started.
-
-_The code and sample YAML files in the master branch of the Velero repository are under active development and are not guaranteed to be stable. Use them at your own risk!_
-
-## More information
-
-[The documentation][29] provides a getting started guide, plus information about building from source, architecture, extending Velero, and more.
+[The documentation][29] provides a getting started guide and information about building from source, architecture, extending Velero, and more.
 
 Please use the version selector at the top of the site to ensure you are using the appropriate documentation for your version of Velero.
 
@@ -36,45 +27,24 @@ If you encounter issues, review the [troubleshooting docs][30], [file an issue][
 
 ## Contributing
 
-Thanks for taking the time to join our community and start contributing!
-
-Feedback and discussion are available on [the mailing list][24].
-
-### Before you start
-
-* Please familiarize yourself with the [Code of Conduct][8] before contributing.
-* See [CONTRIBUTING.md][5] for instructions on the developer certificate of origin that we require.
-* Read how [we're using ZenHub][26] for project and roadmap planning
-
-### Pull requests
-
-* We welcome pull requests. Feel free to dig through the [issues][4] and jump in.
+If you are ready to jump in and test, add code, or help with documentation, follow the instructions on our [Start contributing][31] documentation for guidance on how to setup Velero for development.
 
 ## Changelog
 
 See [the list of releases][6] to find out about feature changes.
 
-[1]: https://travis-ci.org/heptio/velero.svg?branch=master
-[2]: https://travis-ci.org/heptio/velero
-
-[4]: https://github.com/heptio/velero/issues
-[5]: https://github.com/heptio/velero/blob/master/CONTRIBUTING.md
-[6]: https://github.com/heptio/velero/releases
-
-[8]: https://github.com/heptio/velero/blob/master/CODE_OF_CONDUCT.md
+[1]: https://github.com/vmware-tanzu/velero/workflows/Master%20CI/badge.svg
+[2]: https://github.com/vmware-tanzu/velero/actions?query=workflow%3A"Master+CI"
+[4]: https://github.com/vmware-tanzu/velero/issues
+[6]: https://github.com/vmware-tanzu/velero/releases
 [9]: https://kubernetes.io/docs/setup/
 [10]: https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-with-homebrew-on-macos
 [11]: https://kubernetes.io/docs/tasks/tools/install-kubectl/#tabset-1
 [12]: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/README.md
 [14]: https://github.com/kubernetes/kubernetes
-
 [24]: https://groups.google.com/forum/#!forum/projectvelero
 [25]: https://kubernetes.slack.com/messages/velero
-[26]: https://velero.io/docs/zenhub
-
-[28]: https://velero.io/docs/install-overview
 [29]: https://velero.io/docs/
 [30]: https://velero.io/docs/troubleshooting
-
-[99]: https://velero.io/docs/support-matrix
-[100]: /site/docs/master/img/velero.png
+[31]: https://velero.io/docs/start-contributing
+[100]: https://velero.io/docs/master/img/velero.png

ROADMAP.md

@@ -0,0 +1,36 @@
+## Velero Roadmap
+
+### About this document
+This document provides a link to the [Velero Project board](https://app.zenhub.com/workspaces/velero-5c59c15e39d47b774b5864e3/board?repos=99143276,112385197,190224441,214524700,214524630,213946861) that serves as the up to date description of items that are in the release pipeline. The board has separate swim lanes based on prioritization. Most items are gathered from the community or include a feedback loop with the community. This should serve as a reference point for Velero users and contributors to understand where the project is heading, and help determine if a contribution could be conflicting with a longer term plan. You will need the ZenHub plugin to view the board.
+
+### How to help?
+Discussion on the roadmap can take place in threads under [Issues](https://github.com/vmware-tanzu/velero/issues) or in [community meetings](https://velero.io/community/). Please open and comment on an issue if you want to provide suggestions, use cases, and feedback to an item in the roadmap. Please review the roadmap to avoid potential duplicated effort.
+
+### How to add an item to the roadmap?
+One of the most important aspects in any open source community is the concept of proposals. Large changes to the codebase and / or new features should be preceded by a [proposal](https://github.com/vmware-tanzu/velero/blob/master/GOVERNANCE.md#proposal-process) in our repo. 
+For smaller enhancements, you can open an issue to track that initiative or feature request.
+We work with and rely on community feedback to focus our efforts to improve Velero and maintain a healthy roadmap.
+
+### Current Roadmap
+The following table includes the current roadmap for Velero. If you have any questions or would like to contribute to Velero, please attend a [community meeting](https://velero.io/community/) to discuss with our team. If you don't know where to start, we are always looking for contributors that will help us reduce technical, automation, and documentation debt.
+Please take the timelines & dates as proposals and goals. Priorities and requirements change based on community feedback, roadblocks encountered, community contributions, etc. If you depend on a specific item, we encourage you to attend community meetings to get updated status information, or help us deliver that feature by contributing to Velero.
+
+`Last Updated: May 2020`
+
+|Theme|Description|Timeline|
+|--|--|--|
+|Restic Improvements|Introduce improvements in annotating resources for Restic backup|August 2020|
+|Extensibility|Add restore hooks for enhanced recovery scenarios|August 2020|
+|CSI|Continue improving the CSI snapshot capabilities and participate in the upstream K8s CSI community|Long running (dependent on CSI working group)|
+|Backup/Restore|Improvements to long-running copy operations from a performance and reliability standpoint|August 2020|
+|UX|Improvements to install and configuration user experience|August 2020|
+|Restic Improvements|Improve the use of Restic in Velero and offer stable support|Dec 2020|
+|Perf & Scale|Introduce a scalable model by using a worker pod for each backup/restore operation and improve operations|Dec 2020|
+|Backup/Restore|Better backup and restore semantics for certain Kubernetes resources like stateful sets, operators|Dec 2020|
+|Security|Enable the use of custom credential providers|Dec 2020|
+|Self-Service & Multitenancy|Reduce friction by enabling developers to backup their namespaces via self-service. Introduce a Velero multi-tenancy model, enabling owners of namespaces to backup and restore within their access scope|Mar 2021|
+|Backup/Restore|Cross availability zone or region backup and restore|Mar 2021|
+|Application Consistency|Offer blueprints for backing up and restoring popular applications|May 2021|
+|Backup/Restore|Data only backup and restore|May 2021|
+|Backup/Restore|Introduce the ability to overwrite existing objects during a restore|May 2021|
+|Backup/Restore|What-if dry run for backup and restore|May 2021|

SUPPORT.md

@@ -1,5 +1,7 @@
 # Velero Support
 
-Thanks for trying out Velero! We welcome all feedback, please consider joining our mailing list: 
+Thanks for trying out Velero! We welcome all feedback, find all the ways to connect with us on our Community page:
 
-- [Mailing List](https://groups.google.com/forum/#!forum/projectvelero)
+- [Velero Community](https://velero.io/community/)
+
+You can find details on the Velero maintainers' support process [here](https://velero.io/docs/master/support-process/).

changelogs/CHANGELOG-1.2.md

@@ -0,0 +1,83 @@
+## v1.2.0
+#### 2019-11-07
+
+### Download
+https://github.com/vmware-tanzu/velero/releases/tag/v1.2.0
+
+### Container Image
+`velero/velero:v1.2.0`
+
+Please note that as of this release we are no longer publishing new container images to `gcr.io/heptio-images`. The existing ones will remain there for the foreseeable future.
+
+### Documentation
+https://velero.io/docs/v1.2.0/
+
+### Upgrading
+https://velero.io/docs/v1.2.0/upgrade-to-1.2/
+
+### Highlights
+## Moving Cloud Provider Plugins Out of Tree
+
+Velero has had built-in support for AWS, Microsoft Azure, and Google Cloud Platform (GCP)  since day 1. When Velero moved to a plugin architecture for object store providers and volume snapshotters in version 0.6, the code for these three providers was converted to use the plugin interface provided by this new architecture, but the cloud provider code still remained inside the Velero codebase. This put the AWS, Azure, and GCP plugins in a different position compared with other providers’ plugins, since they automatically shipped with the Velero binary and could include documentation in-tree.
+
+With version 1.2, we’ve extracted the AWS, Azure, and GCP plugins into their own repositories, one per provider. We now also publish one plugin image per provider. This change brings these providers to parity with other providers’ plugin implementations, reduces the size of the core Velero binary by not requiring each provider’s SDK to be included, and opens the door for the plugins to be maintained and released independently of core Velero.
+
+## Restic Integration Improvements
+
+We’ve continued to work on improving Velero’s restic integration. With this release, we’ve made the following enhancements:
+
+- Restic backup and restore progress is now captured during execution and visible to the user through the `velero backup/restore describe --details` command. The details are updated every 10 seconds. This provides a new level of visibility into restic operations for users.
+- Restic backups of persistent volume claims (PVCs) now remain incremental across the rescheduling of a pod. Previously, if the pod using a PVC was rescheduled, the next restic backup would require a full rescan of the volume’s contents. This improvement potentially makes such backups significantly faster.
+- Read-write-many volumes are no longer backed up once for every pod using the volume, but instead just once per Velero backup. This improvement speeds up backups and prevents potential restore issues due to multiple copies of the backup being processed simultaneously.
+
+
+## Clone PVs When Cloning a Namespace
+
+Before version 1.2, you could clone a Kubernetes namespace by backing it up and then restoring it to a different namespace in the same cluster by using the `--namespace-mappings` flag with the `velero restore create` command. However, in this scenario, Velero was unable to clone persistent volumes used by the namespace, leading to errors for users.
+
+In version 1.2, Velero automatically detects when you are trying to clone an existing namespace, and clones the persistent volumes used by the namespace as well. This doesn’t require the user to specify any additional flags for the `velero restore create` command.  This change lets you fully achieve your goal of cloning namespaces using persistent storage within a cluster.
+
+## Improved Server-Side Encryption Support
+
+To help you secure your important backup data, we’ve added support for more forms of server-side encryption of backup data on both AWS and GCP. Specifically:
+- On AWS, Velero now supports Amazon S3-managed encryption keys (SSE-S3), which uses AES256 encryption, by specifying `serverSideEncryption: AES256` in a backup storage location’s config.
+- On GCP, Velero now supports using a specific Cloud KMS key for server-side encryption by specifying `kmsKeyName: <key name>` in a backup storage location’s config.
+
+## CRD Structural Schema
+
+In Kubernetes 1.16, custom resource definitions (CRDs) reached general availability. Structural schemas are required for CRDs created in the `apiextensions.k8s.io/v1` API group. Velero now defines a structural schema for each of its CRDs and automatically applies it the user runs the `velero install` command.  The structural schemas enable the user to get quicker feedback when their backup, restore, or schedule request is invalid, so they can immediately remediate their request.
+
+### All Changes
+  * Ensure object store plugin processes are cleaned up after restore and after BSL validation during server start up (#2041, @betta1)
+  * bug fix: don't try to restore pod volume backups that don't have a snapshot ID (#2031, @skriss)
+  * Restore Documentation: Updated Restore Documentation with Clarification implications of removing restore object. (#1957, @nainav)
+  * add `--allow-partially-failed` flag to `velero restore create` for use with `--from-schedule` to allow partially-failed backups to be restored (#1994, @skriss)
+  * Allow backup storage locations to specify backup sync period or toggle off sync (#1936, @betta1)
+  * Remove cloud provider code (#1985, @carlisia)
+  * Restore action for cluster/namespace role bindings (#1974, @alexander-demichev)
+  * Add `--no-default-backup-location` flag to `velero install` (#1931, @Frank51)
+  * If includeClusterResources is nil/auto, pull in necessary CRDs in backupResource (#1831, @sseago)
+  * Azure: add support for Azure China/German clouds (#1938, @andyzhangx)
+  * Add a new required `--plugins` flag for `velero install` command. `--plugins` takes a list of container images to add as initcontainers. (#1930, @nrb)
+  * restic: only backup read-write-many PVCs at most once, even if they're annotated for backup from multiple pods. (#1896, @skriss)
+  * Azure: add support for cross-subscription backups (#1895, @boxcee)
+  * adds `insecureSkipTLSVerify` server config for AWS storage and `--insecure-skip-tls-verify` flag on client for self-signed certs (#1793, @s12chung)
+  * Add check to update resource field during backupItem (#1904, @spiffcs)
+  * Add `LD_LIBRARY_PATH` (=/plugins) to the env variables of velero deployment. (#1893, @lintongj)
+  * backup sync controller: stop using `metadata/revision` file, do a full diff of bucket contents vs. cluster contents each sync interval (#1892, @skriss)
+  * bug fix: during restore, check item's original namespace, not the remapped one, for inclusion/exclusion (#1909, @skriss)
+  * adds structural schema to Velero CRDs created on Velero install, enabling validation of Velero API fields (#1898, @prydonius)
+  * GCP: add support for specifying a Cloud KMS key name to use for encrypting backups in a storage location. (#1879, @skriss)
+  * AWS: add support for SSE-S3 AES256 encryption via `serverSideEncryption` config field in BackupStorageLocation (#1869, @skriss)
+  * change default `restic prune` interval to 7 days, add `velero server/install` flags for specifying an alternate default value. (#1864, @skriss)
+  * velero install: if `--use-restic` and `--wait` are specified, wait up to a minute for restic daemonset to be ready (#1859, @skriss)
+  * report restore progress in PodVolumeRestores and expose progress in the velero restore describe --details command (#1854, @prydonius)
+  * Jekyll Site updates - modifies documentation to use a wider layout; adds better markdown table formatting (#1848, @ccbayer)
+  * fix excluding additional items with the velero.io/exclude-from-backup=true label (#1843, @prydonius)
+  * report backup progress in PodVolumeBackups and expose progress in the velero backup describe --details command. Also upgrades restic to v0.9.5 (#1821, @prydonius)
+  * Add `--features` argument to all velero commands to provide feature flags that can control enablement of pre-release features. (#1798, @nrb)
+  * when backing up PVCs with restic, specify `--parent` flag to prevent full volume rescans after pod reschedules (#1807, @skriss)
+  * remove 'restic check' calls from before/after 'restic prune' since they're redundant (#1794, @skriss)
+  * fix error formatting due interpreting % as printf formatted strings (#1781, @s12chung)
+  * when using `velero restore create --namespace-mappings ...` to create a second copy of a namespace in a cluster, create copies of the PVs used (#1779, @skriss)
+  * adds --from-schedule flag to the `velero create backup` command to create a Backup from an existing Schedule (#1734, @prydonius)

changelogs/CHANGELOG-1.3.md

@@ -0,0 +1,116 @@
+## v1.3.2
+### 2020-04-03
+
+### Download
+https://github.com/vmware-tanzu/velero/releases/tag/v1.3.2
+
+### Container Image
+`velero/velero:v1.3.2`
+
+### Documentation
+https://velero.io/docs/v1.3.2/
+
+### Upgrading
+https://velero.io/docs/v1.3.2/upgrade-to-1.3/
+
+### All Changes
+* Allow `plugins/` as a valid top-level directory within backup storage locations. This directory is a place for plugin authors to store arbitrary data as needed. It is recommended to create an additional subdirectory under `plugins/` specifically for your plugin, e.g. `plugins/my-plugin-data/`. (#2350, @skriss)
+* bug fix: don't panic in `velero restic repo get` when last maintenance time is `nil` (#2315, @skriss)
+
+## v1.3.1
+### 2020-03-10
+
+### Download
+https://github.com/vmware-tanzu/velero/releases/tag/v1.3.1
+
+### Container Image
+`velero/velero:v1.3.1`
+
+### Documentation
+https://velero.io/docs/v1.3.1/
+
+### Upgrading
+https://velero.io/docs/v1.3.1/upgrade-to-1.3/
+
+### Highlights
+
+Fixed a bug that caused failures when backing up CustomResourceDefinitions with whole numbers in numeric fields.
+
+### All Changes
+ * Fix CRD backup failures when fields contained a whole number. (#2322, @nrb)
+
+
+## v1.3.0
+#### 2020-03-02
+
+### Download
+https://github.com/vmware-tanzu/velero/releases/tag/v1.3.0
+
+### Container Image
+`velero/velero:v1.3.0`
+
+### Documentation
+https://velero.io/docs/v1.3.0/
+
+### Upgrading
+https://velero.io/docs/v1.3.0/upgrade-to-1.3/
+
+### Highlights
+
+#### Custom Resource Definition Backup and Restore Improvements
+
+This release includes a number of related bug fixes and improvements to how Velero backs up and restores custom resource definitions (CRDs) and instances of those CRDs.
+
+We found and fixed three issues around restoring CRDs that were originally created via the `v1beta1` CRD API.  The first issue affected CRDs that  had the `PreserveUnknownFields` field set to `true`.  These CRDs could not be restored into 1.16+ Kubernetes clusters, because the `v1` CRD API does not allow this field to be set to `true`. We added code to the restore process to check for this scenario, to set the `PreserveUnknownFields` field to `false`, and to instead set `x-kubernetes-preserve-unknown-fields` to `true` in the OpenAPIv3 structural schema, per Kubernetes guidance. For more information on this, see the [Kubernetes documentation](https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#pruning-versus-preserving-unknown-fields). The second issue affected CRDs without structural schemas. These CRDs need to be backed up/restored through the `v1beta1` API, since all CRDs created through the `v1` API must have structural schemas. We added code to detect these CRDs and always back them up/restore them through the `v1beta1` API. Finally, related to the previous issue, we found that our restore code was unable to handle backups with multiple API versions for a given resource type, and we’ve remediated this as well.
+
+We also improved the CRD restore process to enable users to properly restore CRDs and instances of those CRDs in a single restore operation. Previously, users found that they needed to run two separate restores: one to restore the CRD(s), and another to restore instances of the CRD(s).  This was due to two deficiencies in the Velero code. First, Velero did not wait for a CRD to be fully accepted by the Kubernetes API server and ready for serving before moving on; and second, Velero did not refresh its cached list of available APIs in the target cluster after restoring CRDs, so it was not aware that it could restore instances of those CRDs.
+
+We fixed both of these issues by (1) adding code to wait for CRDs to be “ready” after restore before moving on, and (2) refreshing the cached list of APIs after restoring CRDs, so any instances of newly-restored CRDs could subsequently be restored.
+
+With all of these fixes and improvements in place, we hope that the CRD backup and restore experience is now seamless across all supported versions of Kubernetes.
+
+
+#### Multi-Arch Docker Images
+
+Thanks to community members [@Prajyot-Parab](https://github.com/Prajyot-Parab) and [@shaneutt](https://github.com/shaneutt), Velero now provides multi-arch container images by using Docker manifest lists.  We are currently publishing images for `linux/amd64`, `linux/arm64`, `linux/arm`, and `linux/ppc64le` in [our Docker repository](https://hub.docker.com/r/velero/velero/tags?page=1&name=v1.3&ordering=last_updated).
+
+Users don’t need to change anything other than updating their version tag - the v1.3 image is `velero/velero:v1.3.0`, and Docker will automatically pull the proper architecture for the host.
+
+For more information on manifest lists, see [Docker’s documentation](https://docs.docker.com/registry/spec/manifest-v2-2/). 
+
+
+#### Bug Fixes, Usability Enhancements, and More
+
+We fixed a large number of bugs and made some smaller usability improvements in this release. Here are a few highlights:
+
+- Support private registries with custom ports for the restic restore helper image ([PR #1999](https://github.com/vmware-tanzu/velero/pull/1999), [@cognoz](https://github.com/cognoz))
+- Use AWS profile from BackupStorageLocation when invoking restic ([PR #2096](https://github.com/vmware-tanzu/velero/pull/2096), [@dinesh](https://github.com/dinesh))
+- Allow restores from schedules in other clusters ([PR #2218](https://github.com/vmware-tanzu/velero/pull/2218), [@cpanato](https://github.com/cpanato))
+- Fix memory leak & race condition in restore code ([PR #2201](https://github.com/vmware-tanzu/velero/pull/2201), [@skriss](https://github.com/skriss))
+
+
+### All Changes
+  * Corrected the selfLink for Backup CR in site/docs/master/output-file-format.md (#2292, @RushinthJohn)
+  * Back up schema-less CustomResourceDefinitions as v1beta1, even if they are retrieved via the v1 endpoint. (#2264, @nrb)
+  * Bug fix: restic backup volume snapshot to the second location failed (#2244, @jenting)
+  * Added support of using PV name from volumesnapshotter('SetVolumeID') in case of PV renaming during the restore (#2216, @mynktl)
+  * Replaced deprecated helm repo url at all it appearance at docs. (#2209, @markrity)
+  * added support for arm and arm64 images (#2227, @shaneutt)
+  * when restoring from a schedule, validate by checking for backup(s) labeled with the schedule name rather than existence of the schedule itself, to allow for restoring from deleted schedules and schedules in other clusters (#2218, @cpanato)
+  * bug fix: back up server-preferred version of CRDs rather than always the `v1beta1` version (#2230, @skriss)
+  * Wait for CustomResourceDefinitions to be ready before restoring CustomResources. Also refresh the resource list from the Kubernetes API server after restoring CRDs in order to properly restore CRs. (#1937, @nrb)
+  * When restoring a v1 CRD with PreserveUnknownFields = True, make sure that the preservation behavior is maintained by copying the flag into the Open API V3 schema, but update the flag so as to allow the Kubernetes API server to accept the CRD without error. (#2197, @nrb)
+  * Enable pruning unknown CRD fields (#2187, @jenting)
+  * bump restic to 0.9.6 to fix some issues with non AWS standard regions (#2210, @Sh4d1)
+  * bug fix: fix race condition resulting in restores sometimes succeeding despite restic restore failures (#2201, @skriss)
+  * Bug fix: Check for nil LastMaintenanceTime in ResticRepository dueForMaintenance (#2200, @sseago)
+  * repopulate backup_last_successful_timestamp metrics for each schedule after server restart (#2196, @skriss)
+  * added support for ppc64le images and manifest lists (#1768, @prajyot)
+  * bug fix: only prioritize restoring `replicasets.apps`, not `replicasets.extensions` (#2157, @skriss)
+  * bug fix: restore both `replicasets.apps` *and* `replicasets.extensions` before `deployments` (#2120, @skriss)
+  * bug fix: don't restore cluster-scoped resources when restoring specific namespaces and IncludeClusterResources is nil (#2118, @skriss)
+  * Enableing Velero to switch credentials (`AWS_PROFILE`) if multiple s3-compatible backupLocations are present (#2096, @dinesh)
+  * bug fix: deep-copy backup's labels when constructing snapshot tags, so the PV name isn't added as a label to the backup (#2075, @skriss)
+  * remove the `fsfreeze-pause` image being published from this repo; replace it with `ubuntu:bionic` in the nginx example app (#2068, @skriss)
+  * add support for a private registry with a custom port in a restic-helper image (#1999, @cognoz)
+  * return better error message to user when cluster config can't be found via `--kubeconfig`, `$KUBECONFIG`, or in-cluster config (#2057, @skriss)

changelogs/CHANGELOG-1.4.md

@@ -0,0 +1,54 @@
+## v1.4.0
+### 2020-05-26
+
+### Download
+https://github.com/vmware-tanzu/velero/releases/tag/v1.4.0
+
+### Container Image
+`velero/velero:v1.4.0`
+
+### Documentation
+https://velero.io/docs/v1.4/
+
+### Upgrading
+https://velero.io/docs/v1.4/upgrade-to-1.4/
+
+### Highlights
+
+ * Added beta-level CSI support!
+ * Added custom CA certificate support
+ * Backup progress reporting
+ * Changed backup tarball format to support all versions of a given resource
+
+### All Changes
+  * increment restic volumesnapshot count after successful pvb create (#2542, @ashish-amarnath)
+  * Add details of CSI volumesnapshotcontents associated with a backup to `velero backup describe` when the `EnableCSI` feature flag is given on the velero client. (#2448, @nrb)
+  * Allow users the option to retrieve all versions of a given resource (instead of just the preferred version) from the API server with the `EnableAPIGroupVersions` feature flag. (#2373, @brito-rafa)
+  * Changed backup tarball format to store all versions of a given resource, updated backup tarball format to 1.1.0. (#2373, @brito-rafa)
+  * allow feature flags to be passed from install CLI (#2503, @ashish-amarnath)
+  * sync backups' CSI API objects into the cluster as part of the backup sync controller (#2496, @ashish-amarnath)
+  * bug fix: in error location logging hook, if the item logged under the `error` key doesn't implement the `error` interface, don't return an error since this is a valid scenario (#2487, @skriss)
+  * bug fix: in CRD restore plugin, don't use runtime.DefaultUnstructuredConverter.FromUnstructured(...) to avoid conversion issues when float64 fields contain int values (#2484, @skriss)
+  * during backup deletion also delete CSI volumesnapshotcontents that were created as a part of the backup but the associated volumesnapshot object does not exist (#2480, @ashish-amarnath)
+  * If plugins don't support the `--features` flag, don't pass it to them. Also, update the standard plugin server to ignore unknown flags. (#2479, @skriss)
+  * At backup time, if a CustomResourceDefinition appears to have been created via the v1beta1 endpoint, retrieve it from the v1beta1 endpoint instead of simply changing the APIVersion. (#2478, @nrb)
+  * update container base images from ubuntu:bionic to ubuntu:focal (#2471, @skriss)
+  * bug fix: when a resource includes/excludes list contains unresolvable items, don't remove them from the list, so that the list doesn't inadvertently end up matching *all* resources. (#2462, @skriss)
+  * Azure: add support for getting storage account key for restic directly from an environment variable (#2455, @jaygridley)
+  * Support to skip VSL validation for the backup having SnapshotVolumes set to false or created with `--snapshot-volumes=false` (#2450, @mynktl)
+  * report backup progress (number of items backed up so far out of an estimated total number of items) during backup in the logs and as status fields on the Backup custom resource (#2440, @skriss)
+  * bug fix: populate namespace in logs for backup errors (#2438, @skriss)
+  * during backup deletion also delete CSI volumesnapshots that were created as a part of the backup (#2411, @ashish-amarnath)
+  * bump Kubernetes module dependencies to v0.17.4 to get fix for https://github.com/kubernetes/kubernetes/issues/86149 (#2407, @skriss)
+  * bug fix: save PodVolumeBackup manifests to object storage even if the volume was empty, so that on restore, the PV is dynamically reprovisioned if applicable (#2390, @skriss)
+  * Adding new restoreItemAction for PVC to update the selected-node annotation (#2377, @mynktl)
+  * Added a --cacert flag to the install command to provide the CA bundle to use when verifying TLS connections to object storage (#2368, @mansam)
+  * Added a `--cacert` flag to the velero client describe, download, and logs commands to allow passing a path to a certificate to use when verifying TLS connections to object storage. Also added a corresponding client config option called `cacert` which takes a path to a certificate bundle to use as a default when `--cacert` is not specified. (#2364, @mansam)
+  * support setting a custom CA certificate on a BSL to use when verifying TLS connections (#2353, @mansam)
+  * adding annotations on backup CRD for k8s major, minor and git versions (#2346, @brito-rafa)
+  * When the EnableCSI feature flag is provided, upload CSI VolumeSnapshots and VolumeSnapshotContents to object storage as gzipped JSON. (#2323, @nrb)
+  * add CSI snapshot API types into default restore priorities (#2318, @ashish-amarnath)
+  * refactoring: wait for all informer caches to sync before running controllers (#2299, @skriss)
+  * refactor restore code to lazily resolve resources via discovery and eliminate second restore loop for instances of restored CRDs (#2248, @skriss)
+  * upgrade to go 1.14 and migrate from `dep` to go modules (#2214, @skriss)
+  * clarify the wording for restore describe for namespaces included

changelogs/unreleased/1781-s12chung

@@ -1 +0,0 @@
-fix error formatting due interpreting % as printf formatted strings

changelogs/unreleased/2550-skriss

@@ -0,0 +1 @@
+backup/restore: reinstantiate backup store just before uploading artifacts to ensure credentials are up-to-date

changelogs/unreleased/2561-carlisia

@@ -0,0 +1 @@
+Convert manifests + BSL api client to kubebuilder

changelogs/unreleased/2569-cblecker

@@ -0,0 +1 @@
+when creating new backup from schedule from cli, allow backup name to be automatically generated

changelogs/unreleased/2595-skriss

@@ -0,0 +1 @@
+log a warning instead of erroring if an additional item returned from a plugin can't be found in the Kubernetes API

changelogs/unreleased/2602-skriss

@@ -0,0 +1 @@
+Azure: support using `aad-pod-identity` auth when using restic

changelogs/unreleased/2610-nrb

@@ -0,0 +1 @@
+When a timeout string can't be parsed, log the error as a warning instead of silently consuming the error.

changelogs/unreleased/2611-ashish-amarnath

@@ -0,0 +1 @@
+implement option to back up all volumes by default with restic 

changelogs/unreleased/2613-ashish-amarnath

@@ -0,0 +1 @@
+add a CI check for a changelog file

changelogs/unreleased/2615-tbatard

@@ -0,0 +1 @@
+Add linter checks to Makefile

changelogs/unreleased/2620-mauilion

@@ -0,0 +1,6 @@
+improve builder image handling so that we don't rebuild each `make shell` 
+first check if there are pending changed on the build-image dockerfile if so build it.
+then check if there is an image in the registry if so pull it.
+then build an image cause we don't have a cached image. (this handles the backward compat case.)
+fix make clean to clear go mod cache before removing dirs (for containerized builds)
+

changelogs/unreleased/2625-tbatard

@@ -0,0 +1,3 @@
+Documentation: Update Jekyll to 4.1.0
+
+Switch from redcarpet to kramdown for Markdown renderer

changelogs/unreleased/2629-adamrushuk

@@ -0,0 +1 @@
+Update basic-install.md documentation: Add windows cli installation option via chocolatey

changelogs/unreleased/2635-jessicagreben

@@ -0,0 +1 @@
+Add a new supported provider for an object storage plugin for Storj

changelogs/unreleased/2636-ashish-amarnath

@@ -0,0 +1 @@
+move CSI plugin out of prototype into beta

changelogs/unreleased/2638-adamrushuk

@@ -0,0 +1 @@
+Update basic-install and release-instructions documentation for Windows Chocolatey package

changelogs/unreleased/2651-laverya

@@ -0,0 +1 @@
+bump Kubernetes module dependencies to v0.18.4 to fix https://github.com/vmware-tanzu/velero/issues/2540 by adding code compatibility with kubernetes v1.18

changelogs/unreleased/2660-carlisia

@@ -0,0 +1 @@
+Improve velero download doc

changelogs/unreleased/2674-carlisia

@@ -0,0 +1 @@
+Add a BSL controller to handle validation + update BSL status phase (validation removed from the server and no longer blocks when there's any invalid BSL)

changelogs/unreleased/2676-dthrasher

@@ -0,0 +1 @@
+updated acceptable values on cron schedule from 0-7 to 0-6

changelogs/unreleased/2683-ashish-amarnath

@@ -0,0 +1 @@
+capture version of the CRD prior before invoking the remap_crd_version backup item action

changelogs/unreleased/2696-nrb

@@ -0,0 +1 @@
+Adjust restic default time out to 4 hours and base pod resource requests to 500m CPU/512Mi memory.

changelogs/unreleased/2714-kathpeony

@@ -0,0 +1 @@
+Add backupValidationFailureTotal to metrics

cmd/velero/main.go

@@ -22,8 +22,8 @@ import (
 
 	"k8s.io/klog"
 
-	"github.com/heptio/velero/pkg/cmd"
-	"github.com/heptio/velero/pkg/cmd/velero"
+	"github.com/vmware-tanzu/velero/pkg/cmd"
+	"github.com/vmware-tanzu/velero/pkg/cmd/velero"
 )
 
 func main() {

config/crd/bases/velero.io_backups.yaml

@@ -0,0 +1,423 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.3.0
+  creationTimestamp: null
+  name: backups.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: Backup
+    listKind: BackupList
+    plural: backups
+    singular: backup
+  preserveUnknownFields: false
+  scope: Namespaced
+  validation:
+    openAPIV3Schema:
+      description: Backup is a Velero resource that respresents the capture of Kubernetes
+        cluster state at a point in time (API objects and associated volume state).
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: BackupSpec defines the specification for a Velero backup.
+          properties:
+            defaultVolumesToRestic:
+              description: DefaultVolumesToRestic specifies whether restic should
+                be used to take a backup of all pod volumes by default.
+              type: boolean
+            excludedNamespaces:
+              description: ExcludedNamespaces contains a list of namespaces that are
+                not included in the backup.
+              items:
+                type: string
+              nullable: true
+              type: array
+            excludedResources:
+              description: ExcludedResources is a slice of resource names that are
+                not included in the backup.
+              items:
+                type: string
+              nullable: true
+              type: array
+            hooks:
+              description: Hooks represent custom behaviors that should be executed
+                at different phases of the backup.
+              properties:
+                resources:
+                  description: Resources are hooks that should be executed when backing
+                    up individual instances of a resource.
+                  items:
+                    description: BackupResourceHookSpec defines one or more BackupResourceHooks
+                      that should be executed based on the rules defined for namespaces,
+                      resources, and label selector.
+                    properties:
+                      excludedNamespaces:
+                        description: ExcludedNamespaces specifies the namespaces to
+                          which this hook spec does not apply.
+                        items:
+                          type: string
+                        nullable: true
+                        type: array
+                      excludedResources:
+                        description: ExcludedResources specifies the resources to
+                          which this hook spec does not apply.
+                        items:
+                          type: string
+                        nullable: true
+                        type: array
+                      includedNamespaces:
+                        description: IncludedNamespaces specifies the namespaces to
+                          which this hook spec applies. If empty, it applies to all
+                          namespaces.
+                        items:
+                          type: string
+                        nullable: true
+                        type: array
+                      includedResources:
+                        description: IncludedResources specifies the resources to
+                          which this hook spec applies. If empty, it applies to all
+                          resources.
+                        items:
+                          type: string
+                        nullable: true
+                        type: array
+                      labelSelector:
+                        description: LabelSelector, if specified, filters the resources
+                          to which this hook spec applies.
+                        nullable: true
+                        properties:
+                          matchExpressions:
+                            description: matchExpressions is a list of label selector
+                              requirements. The requirements are ANDed.
+                            items:
+                              description: A label selector requirement is a selector
+                                that contains values, a key, and an operator that
+                                relates the key and values.
+                              properties:
+                                key:
+                                  description: key is the label key that the selector
+                                    applies to.
+                                  type: string
+                                operator:
+                                  description: operator represents a key's relationship
+                                    to a set of values. Valid operators are In, NotIn,
+                                    Exists and DoesNotExist.
+                                  type: string
+                                values:
+                                  description: values is an array of string values.
+                                    If the operator is In or NotIn, the values array
+                                    must be non-empty. If the operator is Exists or
+                                    DoesNotExist, the values array must be empty.
+                                    This array is replaced during a strategic merge
+                                    patch.
+                                  items:
+                                    type: string
+                                  type: array
+                              required:
+                              - key
+                              - operator
+                              type: object
+                            type: array
+                          matchLabels:
+                            additionalProperties:
+                              type: string
+                            description: matchLabels is a map of {key,value} pairs.
+                              A single {key,value} in the matchLabels map is equivalent
+                              to an element of matchExpressions, whose key field is
+                              "key", the operator is "In", and the values array contains
+                              only "value". The requirements are ANDed.
+                            type: object
+                        type: object
+                      name:
+                        description: Name is the name of this hook.
+                        type: string
+                      post:
+                        description: PostHooks is a list of BackupResourceHooks to
+                          execute after storing the item in the backup. These are
+                          executed after all "additional items" from item actions
+                          are processed.
+                        items:
+                          description: BackupResourceHook defines a hook for a resource.
+                          properties:
+                            exec:
+                              description: Exec defines an exec hook.
+                              properties:
+                                command:
+                                  description: Command is the command and arguments
+                                    to execute.
+                                  items:
+                                    type: string
+                                  minItems: 1
+                                  type: array
+                                container:
+                                  description: Container is the container in the pod
+                                    where the command should be executed. If not specified,
+                                    the pod's first container is used.
+                                  type: string
+                                onError:
+                                  description: OnError specifies how Velero should
+                                    behave if it encounters an error executing this
+                                    hook.
+                                  enum:
+                                  - Continue
+                                  - Fail
+                                  type: string
+                                timeout:
+                                  description: Timeout defines the maximum amount
+                                    of time Velero should wait for the hook to complete
+                                    before considering the execution a failure.
+                                  type: string
+                              required:
+                              - command
+                              type: object
+                          required:
+                          - exec
+                          type: object
+                        type: array
+                      pre:
+                        description: PreHooks is a list of BackupResourceHooks to
+                          execute prior to storing the item in the backup. These are
+                          executed before any "additional items" from item actions
+                          are processed.
+                        items:
+                          description: BackupResourceHook defines a hook for a resource.
+                          properties:
+                            exec:
+                              description: Exec defines an exec hook.
+                              properties:
+                                command:
+                                  description: Command is the command and arguments
+                                    to execute.
+                                  items:
+                                    type: string
+                                  minItems: 1
+                                  type: array
+                                container:
+                                  description: Container is the container in the pod
+                                    where the command should be executed. If not specified,
+                                    the pod's first container is used.
+                                  type: string
+                                onError:
+                                  description: OnError specifies how Velero should
+                                    behave if it encounters an error executing this
+                                    hook.
+                                  enum:
+                                  - Continue
+                                  - Fail
+                                  type: string
+                                timeout:
+                                  description: Timeout defines the maximum amount
+                                    of time Velero should wait for the hook to complete
+                                    before considering the execution a failure.
+                                  type: string
+                              required:
+                              - command
+                              type: object
+                          required:
+                          - exec
+                          type: object
+                        type: array
+                    required:
+                    - name
+                    type: object
+                  nullable: true
+                  type: array
+              type: object
+            includeClusterResources:
+              description: IncludeClusterResources specifies whether cluster-scoped
+                resources should be included for consideration in the backup.
+              nullable: true
+              type: boolean
+            includedNamespaces:
+              description: IncludedNamespaces is a slice of namespace names to include
+                objects from. If empty, all namespaces are included.
+              items:
+                type: string
+              nullable: true
+              type: array
+            includedResources:
+              description: IncludedResources is a slice of resource names to include
+                in the backup. If empty, all resources are included.
+              items:
+                type: string
+              nullable: true
+              type: array
+            labelSelector:
+              description: LabelSelector is a metav1.LabelSelector to filter with
+                when adding individual objects to the backup. If empty or nil, all
+                objects are included. Optional.
+              nullable: true
+              properties:
+                matchExpressions:
+                  description: matchExpressions is a list of label selector requirements.
+                    The requirements are ANDed.
+                  items:
+                    description: A label selector requirement is a selector that contains
+                      values, a key, and an operator that relates the key and values.
+                    properties:
+                      key:
+                        description: key is the label key that the selector applies
+                          to.
+                        type: string
+                      operator:
+                        description: operator represents a key's relationship to a
+                          set of values. Valid operators are In, NotIn, Exists and
+                          DoesNotExist.
+                        type: string
+                      values:
+                        description: values is an array of string values. If the operator
+                          is In or NotIn, the values array must be non-empty. If the
+                          operator is Exists or DoesNotExist, the values array must
+                          be empty. This array is replaced during a strategic merge
+                          patch.
+                        items:
+                          type: string
+                        type: array
+                    required:
+                    - key
+                    - operator
+                    type: object
+                  type: array
+                matchLabels:
+                  additionalProperties:
+                    type: string
+                  description: matchLabels is a map of {key,value} pairs. A single
+                    {key,value} in the matchLabels map is equivalent to an element
+                    of matchExpressions, whose key field is "key", the operator is
+                    "In", and the values array contains only "value". The requirements
+                    are ANDed.
+                  type: object
+              type: object
+            snapshotVolumes:
+              description: SnapshotVolumes specifies whether to take cloud snapshots
+                of any PV's referenced in the set of objects included in the Backup.
+              nullable: true
+              type: boolean
+            storageLocation:
+              description: StorageLocation is a string containing the name of a BackupStorageLocation
+                where the backup should be stored.
+              type: string
+            ttl:
+              description: TTL is a time.Duration-parseable string describing how
+                long the Backup should be retained for.
+              type: string
+            volumeSnapshotLocations:
+              description: VolumeSnapshotLocations is a list containing names of VolumeSnapshotLocations
+                associated with this backup.
+              items:
+                type: string
+              type: array
+          type: object
+        status:
+          description: BackupStatus captures the current status of a Velero backup.
+          properties:
+            completionTimestamp:
+              description: CompletionTimestamp records the time a backup was completed.
+                Completion time is recorded even on failed backups. Completion time
+                is recorded before uploading the backup object. The server's time
+                is used for CompletionTimestamps
+              format: date-time
+              nullable: true
+              type: string
+            errors:
+              description: Errors is a count of all error messages that were generated
+                during execution of the backup.  The actual errors are in the backup's
+                log file in object storage.
+              type: integer
+            expiration:
+              description: Expiration is when this Backup is eligible for garbage-collection.
+              format: date-time
+              nullable: true
+              type: string
+            formatVersion:
+              description: FormatVersion is the backup format version, including major,
+                minor, and patch version.
+              type: string
+            phase:
+              description: Phase is the current state of the Backup.
+              enum:
+              - New
+              - FailedValidation
+              - InProgress
+              - Completed
+              - PartiallyFailed
+              - Failed
+              - Deleting
+              type: string
+            progress:
+              description: Progress contains information about the backup's execution
+                progress. Note that this information is best-effort only -- if Velero
+                fails to update it during a backup for any reason, it may be inaccurate/stale.
+              nullable: true
+              properties:
+                itemsBackedUp:
+                  description: ItemsBackedUp is the number of items that have actually
+                    been written to the backup tarball so far.
+                  type: integer
+                totalItems:
+                  description: TotalItems is the total number of items to be backed
+                    up. This number may change throughout the execution of the backup
+                    due to plugins that return additional related items to back up,
+                    the velero.io/exclude-from-backup label, and various other filters
+                    that happen as items are processed.
+                  type: integer
+              type: object
+            startTimestamp:
+              description: StartTimestamp records the time a backup was started. Separate
+                from CreationTimestamp, since that value changes on restores. The
+                server's time is used for StartTimestamps
+              format: date-time
+              nullable: true
+              type: string
+            validationErrors:
+              description: ValidationErrors is a slice of all validation errors (if
+                applicable).
+              items:
+                type: string
+              nullable: true
+              type: array
+            version:
+              description: 'Version is the backup format major version. Deprecated:
+                Please see FormatVersion'
+              type: integer
+            volumeSnapshotsAttempted:
+              description: VolumeSnapshotsAttempted is the total number of attempted
+                volume snapshots for this backup.
+              type: integer
+            volumeSnapshotsCompleted:
+              description: VolumeSnapshotsCompleted is the total number of successfully
+                completed volume snapshots for this backup.
+              type: integer
+            warnings:
+              description: Warnings is a count of all warning messages that were generated
+                during execution of the backup. The actual warnings are in the backup's
+                log file in object storage.
+              type: integer
+          type: object
+      type: object
+  version: v1
+  versions:
+  - name: v1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crd/bases/velero.io_backupstoragelocations.yaml

@@ -0,0 +1,151 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.3.0
+  creationTimestamp: null
+  name: backupstoragelocations.velero.io
+spec:
+  additionalPrinterColumns:
+  - JSONPath: .status.phase
+    description: Backup Storage Location status such as Available/Unavailable
+    name: Phase
+    type: string
+  - JSONPath: .status.lastValidationTime
+    description: LastValidationTime is the last time the backup store location was
+      validated
+    name: Last Validated
+    type: date
+  - JSONPath: .metadata.creationTimestamp
+    name: Age
+    type: date
+  group: velero.io
+  names:
+    kind: BackupStorageLocation
+    listKind: BackupStorageLocationList
+    plural: backupstoragelocations
+    singular: backupstoragelocation
+  preserveUnknownFields: false
+  scope: Namespaced
+  subresources:
+    status: {}
+  validation:
+    openAPIV3Schema:
+      description: BackupStorageLocation is a location where Velero stores backup
+        objects
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: BackupStorageLocationSpec defines the desired state of a Velero
+            BackupStorageLocation
+          properties:
+            accessMode:
+              description: AccessMode defines the permissions for the backup storage
+                location.
+              enum:
+              - ReadOnly
+              - ReadWrite
+              type: string
+            backupSyncPeriod:
+              description: BackupSyncPeriod defines how frequently to sync backup
+                API objects from object storage. A value of 0 disables sync.
+              nullable: true
+              type: string
+            config:
+              additionalProperties:
+                type: string
+              description: Config is for provider-specific configuration fields.
+              type: object
+            objectStorage:
+              description: ObjectStorageLocation specifies the settings necessary
+                to connect to a provider's object storage.
+              properties:
+                bucket:
+                  description: Bucket is the bucket to use for object storage.
+                  type: string
+                caCert:
+                  description: CACert defines a CA bundle to use when verifying TLS
+                    connections to the provider.
+                  format: byte
+                  type: string
+                prefix:
+                  description: Prefix is the path inside a bucket to use for Velero
+                    storage. Optional.
+                  type: string
+              required:
+              - bucket
+              type: object
+            provider:
+              description: Provider is the provider of the backup storage.
+              type: string
+            validationFrequency:
+              description: ValidationFrequency defines how frequently to validate
+                the corresponding object storage. A value of 0 disables validation.
+              nullable: true
+              type: string
+          required:
+          - objectStorage
+          - provider
+          type: object
+        status:
+          description: BackupStorageLocationStatus defines the observed state of BackupStorageLocation
+          properties:
+            accessMode:
+              description: "AccessMode is an unused field. \n Deprecated: there is
+                now an AccessMode field on the Spec and this field will be removed
+                entirely as of v2.0."
+              enum:
+              - ReadOnly
+              - ReadWrite
+              type: string
+            lastSyncedRevision:
+              description: "LastSyncedRevision is the value of the `metadata/revision`
+                file in the backup storage location the last time the BSL's contents
+                were synced into the cluster. \n Deprecated: this field is no longer
+                updated or used for detecting changes to the location's contents and
+                will be removed entirely in v2.0."
+              type: string
+            lastSyncedTime:
+              description: LastSyncedTime is the last time the contents of the location
+                were synced into the cluster.
+              format: date-time
+              nullable: true
+              type: string
+            lastValidationTime:
+              description: LastValidationTime is the last time the backup store location
+                was validated the cluster.
+              format: date-time
+              nullable: true
+              type: string
+            phase:
+              description: Phase is the current state of the BackupStorageLocation.
+              enum:
+              - Available
+              - Unavailable
+              type: string
+          type: object
+      type: object
+  version: v1
+  versions:
+  - name: v1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crd/bases/velero.io_deletebackuprequests.yaml

@@ -0,0 +1,73 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.3.0
+  creationTimestamp: null
+  name: deletebackuprequests.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: DeleteBackupRequest
+    listKind: DeleteBackupRequestList
+    plural: deletebackuprequests
+    singular: deletebackuprequest
+  preserveUnknownFields: false
+  scope: Namespaced
+  validation:
+    openAPIV3Schema:
+      description: DeleteBackupRequest is a request to delete one or more backups.
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: DeleteBackupRequestSpec is the specification for which backups
+            to delete.
+          properties:
+            backupName:
+              type: string
+          required:
+          - backupName
+          type: object
+        status:
+          description: DeleteBackupRequestStatus is the current status of a DeleteBackupRequest.
+          properties:
+            errors:
+              description: Errors contains any errors that were encountered during
+                the deletion process.
+              items:
+                type: string
+              nullable: true
+              type: array
+            phase:
+              description: Phase is the current state of the DeleteBackupRequest.
+              enum:
+              - New
+              - InProgress
+              - Processed
+              type: string
+          type: object
+      type: object
+  version: v1
+  versions:
+  - name: v1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crd/bases/velero.io_downloadrequests.yaml

@@ -0,0 +1,94 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.3.0
+  creationTimestamp: null
+  name: downloadrequests.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: DownloadRequest
+    listKind: DownloadRequestList
+    plural: downloadrequests
+    singular: downloadrequest
+  preserveUnknownFields: false
+  scope: Namespaced
+  validation:
+    openAPIV3Schema:
+      description: DownloadRequest is a request to download an artifact from backup
+        object storage, such as a backup log file.
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: DownloadRequestSpec is the specification for a download request.
+          properties:
+            target:
+              description: Target is what to download (e.g. logs for a backup).
+              properties:
+                kind:
+                  description: Kind is the type of file to download.
+                  enum:
+                  - BackupLog
+                  - BackupContents
+                  - BackupVolumeSnapshots
+                  - BackupResourceList
+                  - RestoreLog
+                  - RestoreResults
+                  type: string
+                name:
+                  description: Name is the name of the kubernetes resource with which
+                    the file is associated.
+                  type: string
+              required:
+              - kind
+              - name
+              type: object
+          required:
+          - target
+          type: object
+        status:
+          description: DownloadRequestStatus is the current status of a DownloadRequest.
+          properties:
+            downloadURL:
+              description: DownloadURL contains the pre-signed URL for the target
+                file.
+              type: string
+            expiration:
+              description: Expiration is when this DownloadRequest expires and can
+                be deleted by the system.
+              format: date-time
+              nullable: true
+              type: string
+            phase:
+              description: Phase is the current state of the DownloadRequest.
+              enum:
+              - New
+              - Processed
+              type: string
+          type: object
+      type: object
+  version: v1
+  versions:
+  - name: v1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crd/bases/velero.io_podvolumebackups.yaml

@@ -0,0 +1,162 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.3.0
+  creationTimestamp: null
+  name: podvolumebackups.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: PodVolumeBackup
+    listKind: PodVolumeBackupList
+    plural: podvolumebackups
+    singular: podvolumebackup
+  preserveUnknownFields: false
+  scope: Namespaced
+  validation:
+    openAPIV3Schema:
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: PodVolumeBackupSpec is the specification for a PodVolumeBackup.
+          properties:
+            backupStorageLocation:
+              description: BackupStorageLocation is the name of the backup storage
+                location where the restic repository is stored.
+              type: string
+            node:
+              description: Node is the name of the node that the Pod is running on.
+              type: string
+            pod:
+              description: Pod is a reference to the pod containing the volume to
+                be backed up.
+              properties:
+                apiVersion:
+                  description: API version of the referent.
+                  type: string
+                fieldPath:
+                  description: 'If referring to a piece of an object instead of an
+                    entire object, this string should contain a valid JSON/Go field
+                    access statement, such as desiredState.manifest.containers[2].
+                    For example, if the object reference is to a container within
+                    a pod, this would take on a value like: "spec.containers{name}"
+                    (where "name" refers to the name of the container that triggered
+                    the event) or if no container name is specified "spec.containers[2]"
+                    (container with index 2 in this pod). This syntax is chosen only
+                    to have some well-defined way of referencing a part of an object.
+                    TODO: this design is not final and this field is subject to change
+                    in the future.'
+                  type: string
+                kind:
+                  description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                  type: string
+                name:
+                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                  type: string
+                namespace:
+                  description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                  type: string
+                resourceVersion:
+                  description: 'Specific resourceVersion to which this reference is
+                    made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                  type: string
+                uid:
+                  description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                  type: string
+              type: object
+            repoIdentifier:
+              description: RepoIdentifier is the restic repository identifier.
+              type: string
+            tags:
+              additionalProperties:
+                type: string
+              description: Tags are a map of key-value pairs that should be applied
+                to the volume backup as tags.
+              type: object
+            volume:
+              description: Volume is the name of the volume within the Pod to be backed
+                up.
+              type: string
+          required:
+          - backupStorageLocation
+          - node
+          - pod
+          - repoIdentifier
+          - volume
+          type: object
+        status:
+          description: PodVolumeBackupStatus is the current status of a PodVolumeBackup.
+          properties:
+            completionTimestamp:
+              description: CompletionTimestamp records the time a backup was completed.
+                Completion time is recorded even on failed backups. Completion time
+                is recorded before uploading the backup object. The server's time
+                is used for CompletionTimestamps
+              format: date-time
+              nullable: true
+              type: string
+            message:
+              description: Message is a message about the pod volume backup's status.
+              type: string
+            path:
+              description: Path is the full path within the controller pod being backed
+                up.
+              type: string
+            phase:
+              description: Phase is the current state of the PodVolumeBackup.
+              enum:
+              - New
+              - InProgress
+              - Completed
+              - Failed
+              type: string
+            progress:
+              description: Progress holds the total number of bytes of the volume
+                and the current number of backed up bytes. This can be used to display
+                progress information about the backup operation.
+              properties:
+                bytesDone:
+                  format: int64
+                  type: integer
+                totalBytes:
+                  format: int64
+                  type: integer
+              type: object
+            snapshotID:
+              description: SnapshotID is the identifier for the snapshot of the pod
+                volume.
+              type: string
+            startTimestamp:
+              description: StartTimestamp records the time a backup was started. Separate
+                from CreationTimestamp, since that value changes on restores. The
+                server's time is used for StartTimestamps
+              format: date-time
+              nullable: true
+              type: string
+          type: object
+      type: object
+  version: v1
+  versions:
+  - name: v1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crd/bases/velero.io_podvolumerestores.yaml

@@ -0,0 +1,145 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.3.0
+  creationTimestamp: null
+  name: podvolumerestores.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: PodVolumeRestore
+    listKind: PodVolumeRestoreList
+    plural: podvolumerestores
+    singular: podvolumerestore
+  preserveUnknownFields: false
+  scope: Namespaced
+  validation:
+    openAPIV3Schema:
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: PodVolumeRestoreSpec is the specification for a PodVolumeRestore.
+          properties:
+            backupStorageLocation:
+              description: BackupStorageLocation is the name of the backup storage
+                location where the restic repository is stored.
+              type: string
+            pod:
+              description: Pod is a reference to the pod containing the volume to
+                be restored.
+              properties:
+                apiVersion:
+                  description: API version of the referent.
+                  type: string
+                fieldPath:
+                  description: 'If referring to a piece of an object instead of an
+                    entire object, this string should contain a valid JSON/Go field
+                    access statement, such as desiredState.manifest.containers[2].
+                    For example, if the object reference is to a container within
+                    a pod, this would take on a value like: "spec.containers{name}"
+                    (where "name" refers to the name of the container that triggered
+                    the event) or if no container name is specified "spec.containers[2]"
+                    (container with index 2 in this pod). This syntax is chosen only
+                    to have some well-defined way of referencing a part of an object.
+                    TODO: this design is not final and this field is subject to change
+                    in the future.'
+                  type: string
+                kind:
+                  description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                  type: string
+                name:
+                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                  type: string
+                namespace:
+                  description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                  type: string
+                resourceVersion:
+                  description: 'Specific resourceVersion to which this reference is
+                    made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                  type: string
+                uid:
+                  description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                  type: string
+              type: object
+            repoIdentifier:
+              description: RepoIdentifier is the restic repository identifier.
+              type: string
+            snapshotID:
+              description: SnapshotID is the ID of the volume snapshot to be restored.
+              type: string
+            volume:
+              description: Volume is the name of the volume within the Pod to be restored.
+              type: string
+          required:
+          - backupStorageLocation
+          - pod
+          - repoIdentifier
+          - snapshotID
+          - volume
+          type: object
+        status:
+          description: PodVolumeRestoreStatus is the current status of a PodVolumeRestore.
+          properties:
+            completionTimestamp:
+              description: CompletionTimestamp records the time a restore was completed.
+                Completion time is recorded even on failed restores. The server's
+                time is used for CompletionTimestamps
+              format: date-time
+              nullable: true
+              type: string
+            message:
+              description: Message is a message about the pod volume restore's status.
+              type: string
+            phase:
+              description: Phase is the current state of the PodVolumeRestore.
+              enum:
+              - New
+              - InProgress
+              - Completed
+              - Failed
+              type: string
+            progress:
+              description: Progress holds the total number of bytes of the snapshot
+                and the current number of restored bytes. This can be used to display
+                progress information about the restore operation.
+              properties:
+                bytesDone:
+                  format: int64
+                  type: integer
+                totalBytes:
+                  format: int64
+                  type: integer
+              type: object
+            startTimestamp:
+              description: StartTimestamp records the time a restore was started.
+                The server's time is used for StartTimestamps
+              format: date-time
+              nullable: true
+              type: string
+          type: object
+      type: object
+  version: v1
+  versions:
+  - name: v1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crd/bases/velero.io_resticrepositories.yaml

@@ -0,0 +1,89 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.3.0
+  creationTimestamp: null
+  name: resticrepositories.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: ResticRepository
+    listKind: ResticRepositoryList
+    plural: resticrepositories
+    singular: resticrepository
+  preserveUnknownFields: false
+  scope: Namespaced
+  validation:
+    openAPIV3Schema:
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: ResticRepositorySpec is the specification for a ResticRepository.
+          properties:
+            backupStorageLocation:
+              description: BackupStorageLocation is the name of the BackupStorageLocation
+                that should contain this repository.
+              type: string
+            maintenanceFrequency:
+              description: MaintenanceFrequency is how often maintenance should be
+                run.
+              type: string
+            resticIdentifier:
+              description: ResticIdentifier is the full restic-compatible string for
+                identifying this repository.
+              type: string
+            volumeNamespace:
+              description: VolumeNamespace is the namespace this restic repository
+                contains pod volume backups for.
+              type: string
+          required:
+          - backupStorageLocation
+          - maintenanceFrequency
+          - resticIdentifier
+          - volumeNamespace
+          type: object
+        status:
+          description: ResticRepositoryStatus is the current status of a ResticRepository.
+          properties:
+            lastMaintenanceTime:
+              description: LastMaintenanceTime is the last time maintenance was run.
+              format: date-time
+              nullable: true
+              type: string
+            message:
+              description: Message is a message about the current status of the ResticRepository.
+              type: string
+            phase:
+              description: Phase is the current state of the ResticRepository.
+              enum:
+              - New
+              - Ready
+              - NotReady
+              type: string
+          type: object
+      type: object
+  version: v1
+  versions:
+  - name: v1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crd/bases/velero.io_restores.yaml

@@ -0,0 +1,189 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.3.0
+  creationTimestamp: null
+  name: restores.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: Restore
+    listKind: RestoreList
+    plural: restores
+    singular: restore
+  preserveUnknownFields: false
+  scope: Namespaced
+  validation:
+    openAPIV3Schema:
+      description: Restore is a Velero resource that represents the application of
+        resources from a Velero backup to a target Kubernetes cluster.
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: RestoreSpec defines the specification for a Velero restore.
+          properties:
+            backupName:
+              description: BackupName is the unique name of the Velero backup to restore
+                from.
+              type: string
+            excludedNamespaces:
+              description: ExcludedNamespaces contains a list of namespaces that are
+                not included in the restore.
+              items:
+                type: string
+              nullable: true
+              type: array
+            excludedResources:
+              description: ExcludedResources is a slice of resource names that are
+                not included in the restore.
+              items:
+                type: string
+              nullable: true
+              type: array
+            includeClusterResources:
+              description: IncludeClusterResources specifies whether cluster-scoped
+                resources should be included for consideration in the restore. If
+                null, defaults to true.
+              nullable: true
+              type: boolean
+            includedNamespaces:
+              description: IncludedNamespaces is a slice of namespace names to include
+                objects from. If empty, all namespaces are included.
+              items:
+                type: string
+              nullable: true
+              type: array
+            includedResources:
+              description: IncludedResources is a slice of resource names to include
+                in the restore. If empty, all resources in the backup are included.
+              items:
+                type: string
+              nullable: true
+              type: array
+            labelSelector:
+              description: LabelSelector is a metav1.LabelSelector to filter with
+                when restoring individual objects from the backup. If empty or nil,
+                all objects are included. Optional.
+              nullable: true
+              properties:
+                matchExpressions:
+                  description: matchExpressions is a list of label selector requirements.
+                    The requirements are ANDed.
+                  items:
+                    description: A label selector requirement is a selector that contains
+                      values, a key, and an operator that relates the key and values.
+                    properties:
+                      key:
+                        description: key is the label key that the selector applies
+                          to.
+                        type: string
+                      operator:
+                        description: operator represents a key's relationship to a
+                          set of values. Valid operators are In, NotIn, Exists and
+                          DoesNotExist.
+                        type: string
+                      values:
+                        description: values is an array of string values. If the operator
+                          is In or NotIn, the values array must be non-empty. If the
+                          operator is Exists or DoesNotExist, the values array must
+                          be empty. This array is replaced during a strategic merge
+                          patch.
+                        items:
+                          type: string
+                        type: array
+                    required:
+                    - key
+                    - operator
+                    type: object
+                  type: array
+                matchLabels:
+                  additionalProperties:
+                    type: string
+                  description: matchLabels is a map of {key,value} pairs. A single
+                    {key,value} in the matchLabels map is equivalent to an element
+                    of matchExpressions, whose key field is "key", the operator is
+                    "In", and the values array contains only "value". The requirements
+                    are ANDed.
+                  type: object
+              type: object
+            namespaceMapping:
+              additionalProperties:
+                type: string
+              description: NamespaceMapping is a map of source namespace names to
+                target namespace names to restore into. Any source namespaces not
+                included in the map will be restored into namespaces of the same name.
+              type: object
+            restorePVs:
+              description: RestorePVs specifies whether to restore all included PVs
+                from snapshot (via the cloudprovider).
+              nullable: true
+              type: boolean
+            scheduleName:
+              description: ScheduleName is the unique name of the Velero schedule
+                to restore from. If specified, and BackupName is empty, Velero will
+                restore from the most recent successful backup created from this schedule.
+              type: string
+          required:
+          - backupName
+          type: object
+        status:
+          description: RestoreStatus captures the current status of a Velero restore
+          properties:
+            errors:
+              description: Errors is a count of all error messages that were generated
+                during execution of the restore. The actual errors are stored in object
+                storage.
+              type: integer
+            failureReason:
+              description: FailureReason is an error that caused the entire restore
+                to fail.
+              type: string
+            phase:
+              description: Phase is the current state of the Restore
+              enum:
+              - New
+              - FailedValidation
+              - InProgress
+              - Completed
+              - PartiallyFailed
+              - Failed
+              type: string
+            validationErrors:
+              description: ValidationErrors is a slice of all validation errors (if
+                applicable)
+              items:
+                type: string
+              nullable: true
+              type: array
+            warnings:
+              description: Warnings is a count of all warning messages that were generated
+                during execution of the restore. The actual warnings are stored in
+                object storage.
+              type: integer
+          type: object
+      type: object
+  version: v1
+  versions:
+  - name: v1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crd/bases/velero.io_schedules.yaml

@@ -0,0 +1,379 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.3.0
+  creationTimestamp: null
+  name: schedules.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: Schedule
+    listKind: ScheduleList
+    plural: schedules
+    singular: schedule
+  preserveUnknownFields: false
+  scope: Namespaced
+  validation:
+    openAPIV3Schema:
+      description: Schedule is a Velero resource that represents a pre-scheduled or
+        periodic Backup that should be run.
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: ScheduleSpec defines the specification for a Velero schedule
+          properties:
+            schedule:
+              description: Schedule is a Cron expression defining when to run the
+                Backup.
+              type: string
+            template:
+              description: Template is the definition of the Backup to be run on the
+                provided schedule
+              properties:
+                defaultVolumesToRestic:
+                  description: DefaultVolumesToRestic specifies whether restic should
+                    be used to take a backup of all pod volumes by default.
+                  type: boolean
+                excludedNamespaces:
+                  description: ExcludedNamespaces contains a list of namespaces that
+                    are not included in the backup.
+                  items:
+                    type: string
+                  nullable: true
+                  type: array
+                excludedResources:
+                  description: ExcludedResources is a slice of resource names that
+                    are not included in the backup.
+                  items:
+                    type: string
+                  nullable: true
+                  type: array
+                hooks:
+                  description: Hooks represent custom behaviors that should be executed
+                    at different phases of the backup.
+                  properties:
+                    resources:
+                      description: Resources are hooks that should be executed when
+                        backing up individual instances of a resource.
+                      items:
+                        description: BackupResourceHookSpec defines one or more BackupResourceHooks
+                          that should be executed based on the rules defined for namespaces,
+                          resources, and label selector.
+                        properties:
+                          excludedNamespaces:
+                            description: ExcludedNamespaces specifies the namespaces
+                              to which this hook spec does not apply.
+                            items:
+                              type: string
+                            nullable: true
+                            type: array
+                          excludedResources:
+                            description: ExcludedResources specifies the resources
+                              to which this hook spec does not apply.
+                            items:
+                              type: string
+                            nullable: true
+                            type: array
+                          includedNamespaces:
+                            description: IncludedNamespaces specifies the namespaces
+                              to which this hook spec applies. If empty, it applies
+                              to all namespaces.
+                            items:
+                              type: string
+                            nullable: true
+                            type: array
+                          includedResources:
+                            description: IncludedResources specifies the resources
+                              to which this hook spec applies. If empty, it applies
+                              to all resources.
+                            items:
+                              type: string
+                            nullable: true
+                            type: array
+                          labelSelector:
+                            description: LabelSelector, if specified, filters the
+                              resources to which this hook spec applies.
+                            nullable: true
+                            properties:
+                              matchExpressions:
+                                description: matchExpressions is a list of label selector
+                                  requirements. The requirements are ANDed.
+                                items:
+                                  description: A label selector requirement is a selector
+                                    that contains values, a key, and an operator that
+                                    relates the key and values.
+                                  properties:
+                                    key:
+                                      description: key is the label key that the selector
+                                        applies to.
+                                      type: string
+                                    operator:
+                                      description: operator represents a key's relationship
+                                        to a set of values. Valid operators are In,
+                                        NotIn, Exists and DoesNotExist.
+                                      type: string
+                                    values:
+                                      description: values is an array of string values.
+                                        If the operator is In or NotIn, the values
+                                        array must be non-empty. If the operator is
+                                        Exists or DoesNotExist, the values array must
+                                        be empty. This array is replaced during a
+                                        strategic merge patch.
+                                      items:
+                                        type: string
+                                      type: array
+                                  required:
+                                  - key
+                                  - operator
+                                  type: object
+                                type: array
+                              matchLabels:
+                                additionalProperties:
+                                  type: string
+                                description: matchLabels is a map of {key,value} pairs.
+                                  A single {key,value} in the matchLabels map is equivalent
+                                  to an element of matchExpressions, whose key field
+                                  is "key", the operator is "In", and the values array
+                                  contains only "value". The requirements are ANDed.
+                                type: object
+                            type: object
+                          name:
+                            description: Name is the name of this hook.
+                            type: string
+                          post:
+                            description: PostHooks is a list of BackupResourceHooks
+                              to execute after storing the item in the backup. These
+                              are executed after all "additional items" from item
+                              actions are processed.
+                            items:
+                              description: BackupResourceHook defines a hook for a
+                                resource.
+                              properties:
+                                exec:
+                                  description: Exec defines an exec hook.
+                                  properties:
+                                    command:
+                                      description: Command is the command and arguments
+                                        to execute.
+                                      items:
+                                        type: string
+                                      minItems: 1
+                                      type: array
+                                    container:
+                                      description: Container is the container in the
+                                        pod where the command should be executed.
+                                        If not specified, the pod's first container
+                                        is used.
+                                      type: string
+                                    onError:
+                                      description: OnError specifies how Velero should
+                                        behave if it encounters an error executing
+                                        this hook.
+                                      enum:
+                                      - Continue
+                                      - Fail
+                                      type: string
+                                    timeout:
+                                      description: Timeout defines the maximum amount
+                                        of time Velero should wait for the hook to
+                                        complete before considering the execution
+                                        a failure.
+                                      type: string
+                                  required:
+                                  - command
+                                  type: object
+                              required:
+                              - exec
+                              type: object
+                            type: array
+                          pre:
+                            description: PreHooks is a list of BackupResourceHooks
+                              to execute prior to storing the item in the backup.
+                              These are executed before any "additional items" from
+                              item actions are processed.
+                            items:
+                              description: BackupResourceHook defines a hook for a
+                                resource.
+                              properties:
+                                exec:
+                                  description: Exec defines an exec hook.
+                                  properties:
+                                    command:
+                                      description: Command is the command and arguments
+                                        to execute.
+                                      items:
+                                        type: string
+                                      minItems: 1
+                                      type: array
+                                    container:
+                                      description: Container is the container in the
+                                        pod where the command should be executed.
+                                        If not specified, the pod's first container
+                                        is used.
+                                      type: string
+                                    onError:
+                                      description: OnError specifies how Velero should
+                                        behave if it encounters an error executing
+                                        this hook.
+                                      enum:
+                                      - Continue
+                                      - Fail
+                                      type: string
+                                    timeout:
+                                      description: Timeout defines the maximum amount
+                                        of time Velero should wait for the hook to
+                                        complete before considering the execution
+                                        a failure.
+                                      type: string
+                                  required:
+                                  - command
+                                  type: object
+                              required:
+                              - exec
+                              type: object
+                            type: array
+                        required:
+                        - name
+                        type: object
+                      nullable: true
+                      type: array
+                  type: object
+                includeClusterResources:
+                  description: IncludeClusterResources specifies whether cluster-scoped
+                    resources should be included for consideration in the backup.
+                  nullable: true
+                  type: boolean
+                includedNamespaces:
+                  description: IncludedNamespaces is a slice of namespace names to
+                    include objects from. If empty, all namespaces are included.
+                  items:
+                    type: string
+                  nullable: true
+                  type: array
+                includedResources:
+                  description: IncludedResources is a slice of resource names to include
+                    in the backup. If empty, all resources are included.
+                  items:
+                    type: string
+                  nullable: true
+                  type: array
+                labelSelector:
+                  description: LabelSelector is a metav1.LabelSelector to filter with
+                    when adding individual objects to the backup. If empty or nil,
+                    all objects are included. Optional.
+                  nullable: true
+                  properties:
+                    matchExpressions:
+                      description: matchExpressions is a list of label selector requirements.
+                        The requirements are ANDed.
+                      items:
+                        description: A label selector requirement is a selector that
+                          contains values, a key, and an operator that relates the
+                          key and values.
+                        properties:
+                          key:
+                            description: key is the label key that the selector applies
+                              to.
+                            type: string
+                          operator:
+                            description: operator represents a key's relationship
+                              to a set of values. Valid operators are In, NotIn, Exists
+                              and DoesNotExist.
+                            type: string
+                          values:
+                            description: values is an array of string values. If the
+                              operator is In or NotIn, the values array must be non-empty.
+                              If the operator is Exists or DoesNotExist, the values
+                              array must be empty. This array is replaced during a
+                              strategic merge patch.
+                            items:
+                              type: string
+                            type: array
+                        required:
+                        - key
+                        - operator
+                        type: object
+                      type: array
+                    matchLabels:
+                      additionalProperties:
+                        type: string
+                      description: matchLabels is a map of {key,value} pairs. A single
+                        {key,value} in the matchLabels map is equivalent to an element
+                        of matchExpressions, whose key field is "key", the operator
+                        is "In", and the values array contains only "value". The requirements
+                        are ANDed.
+                      type: object
+                  type: object
+                snapshotVolumes:
+                  description: SnapshotVolumes specifies whether to take cloud snapshots
+                    of any PV's referenced in the set of objects included in the Backup.
+                  nullable: true
+                  type: boolean
+                storageLocation:
+                  description: StorageLocation is a string containing the name of
+                    a BackupStorageLocation where the backup should be stored.
+                  type: string
+                ttl:
+                  description: TTL is a time.Duration-parseable string describing
+                    how long the Backup should be retained for.
+                  type: string
+                volumeSnapshotLocations:
+                  description: VolumeSnapshotLocations is a list containing names
+                    of VolumeSnapshotLocations associated with this backup.
+                  items:
+                    type: string
+                  type: array
+              type: object
+          required:
+          - schedule
+          - template
+          type: object
+        status:
+          description: ScheduleStatus captures the current state of a Velero schedule
+          properties:
+            lastBackup:
+              description: LastBackup is the last time a Backup was run for this Schedule
+                schedule
+              format: date-time
+              nullable: true
+              type: string
+            phase:
+              description: Phase is the current phase of the Schedule
+              enum:
+              - New
+              - Enabled
+              - FailedValidation
+              type: string
+            validationErrors:
+              description: ValidationErrors is a slice of all validation errors (if
+                applicable)
+              items:
+                type: string
+              type: array
+          type: object
+      type: object
+  version: v1
+  versions:
+  - name: v1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crd/bases/velero.io_serverstatusrequests.yaml

@@ -0,0 +1,85 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.3.0
+  creationTimestamp: null
+  name: serverstatusrequests.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: ServerStatusRequest
+    listKind: ServerStatusRequestList
+    plural: serverstatusrequests
+    singular: serverstatusrequest
+  preserveUnknownFields: false
+  scope: Namespaced
+  validation:
+    openAPIV3Schema:
+      description: ServerStatusRequest is a request to access current status information
+        about the Velero server.
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: ServerStatusRequestSpec is the specification for a ServerStatusRequest.
+          type: object
+        status:
+          description: ServerStatusRequestStatus is the current status of a ServerStatusRequest.
+          properties:
+            phase:
+              description: Phase is the current lifecycle phase of the ServerStatusRequest.
+              enum:
+              - New
+              - Processed
+              type: string
+            plugins:
+              description: Plugins list information about the plugins running on the
+                Velero server
+              items:
+                description: PluginInfo contains attributes of a Velero plugin
+                properties:
+                  kind:
+                    type: string
+                  name:
+                    type: string
+                required:
+                - kind
+                - name
+                type: object
+              nullable: true
+              type: array
+            processedTimestamp:
+              description: ProcessedTimestamp is when the ServerStatusRequest was
+                processed by the ServerStatusRequestController.
+              format: date-time
+              nullable: true
+              type: string
+            serverVersion:
+              description: ServerVersion is the Velero server version.
+              type: string
+          type: object
+      type: object
+  version: v1
+  versions:
+  - name: v1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crd/bases/velero.io_volumesnapshotlocations.yaml

@@ -0,0 +1,74 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.3.0
+  creationTimestamp: null
+  name: volumesnapshotlocations.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: VolumeSnapshotLocation
+    listKind: VolumeSnapshotLocationList
+    plural: volumesnapshotlocations
+    singular: volumesnapshotlocation
+  preserveUnknownFields: false
+  scope: Namespaced
+  validation:
+    openAPIV3Schema:
+      description: VolumeSnapshotLocation is a location where Velero stores volume
+        snapshots.
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: VolumeSnapshotLocationSpec defines the specification for a
+            Velero VolumeSnapshotLocation.
+          properties:
+            config:
+              additionalProperties:
+                type: string
+              description: Config is for provider-specific configuration fields.
+              type: object
+            provider:
+              description: Provider is the provider of the volume storage.
+              type: string
+          required:
+          - provider
+          type: object
+        status:
+          description: VolumeSnapshotLocationStatus describes the current status of
+            a Velero VolumeSnapshotLocation.
+          properties:
+            phase:
+              description: VolumeSnapshotLocationPhase is the lifecyle phase of a
+                Velero VolumeSnapshotLocation.
+              enum:
+              - Available
+              - Unavailable
+              type: string
+          type: object
+      type: object
+  version: v1
+  versions:
+  - name: v1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crd/crds/doc.go

@@ -0,0 +1,4 @@
+// Package crds embeds the controller-tools generated CRD manifests
+package crds
+
+//go:generate go run ../../../hack/crd-gen/main.go

config/rbac/role.yaml

@@ -0,0 +1,28 @@
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  creationTimestamp: null
+  name: manager-role
+rules:
+- apiGroups:
+  - velero.io
+  resources:
+  - backupstoragelocations
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - velero.io
+  resources:
+  - backupstoragelocations/status
+  verbs:
+  - get
+  - patch
+  - update

config/samples/velero_v1_backupstoragelocation.yaml

@@ -0,0 +1,16 @@
+apiVersion: velero.io/v1
+kind: BackupStorageLocation
+metadata:
+  creationTimestamp: null
+  labels:
+    component: velero
+  name: default
+  namespace: velero
+spec:
+  config:
+    region: minio
+    s3ForcePathStyle: "true"
+    s3Url: http://minio.velero.svc:9000
+  objectStorage:
+    bucket: velero
+  provider: aws

design/CLI/PoC/base/backupstoragelocations.yaml

@@ -0,0 +1,131 @@
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (unknown)
+  labels:
+    component: velero
+  name: backupstoragelocations.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: BackupStorageLocation
+    listKind: BackupStorageLocationList
+    plural: backupstoragelocations
+    singular: backupstoragelocation
+  scope: ""
+  validation:
+    openAPIV3Schema:
+      description: BackupStorageLocation is a location where Velero stores backup
+        objects.
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: BackupStorageLocationSpec defines the specification for a
+            Velero BackupStorageLocation.
+          properties:
+            accessMode:
+              description: AccessMode defines the permissions for the backup storage
+                location.
+              enum:
+              - ReadOnly
+              - ReadWrite
+              type: string
+            backupSyncPeriod:
+              description: BackupSyncPeriod defines how frequently to sync backup
+                API objects from object storage. A value of 0 disables sync.
+              nullable: true
+              type: string
+            config:
+              additionalProperties:
+                type: string
+              description: Config is for provider-specific configuration fields.
+              type: object
+            objectStorage:
+              description: ObjectStorageLocation specifies the settings necessary
+                to connect to a provider's object storage.
+              properties:
+                bucket:
+                  description: Bucket is the bucket to use for object storage.
+                  type: string
+                prefix:
+                  description: Prefix is the path inside a bucket to use for Velero
+                    storage. Optional.
+                  type: string
+              required:
+              - bucket
+              type: object
+            provider:
+              description: Provider is the provider of the backup storage.
+              type: string
+          required:
+          - objectStorage
+          - provider
+          type: object
+        status:
+          description: BackupStorageLocationStatus describes the current status
+            of a Velero BackupStorageLocation.
+          properties:
+            accessMode:
+              description: "AccessMode is an unused field. \n Deprecated: there
+                is now an AccessMode field on the Spec and this field will be removed
+                entirely as of v2.0."
+              enum:
+              - ReadOnly
+              - ReadWrite
+              type: string
+            lastSyncedRevision:
+              description: "LastSyncedRevision is the value of the `metadata/revision`
+                file in the backup storage location the last time the BSL's contents
+                were synced into the cluster. \n Deprecated: this field is no longer
+                updated or used for detecting changes to the location's contents
+                and will be removed entirely in v2.0."
+              type: string
+            lastSyncedTime:
+              description: LastSyncedTime is the last time the contents of the location
+                were synced into the cluster.
+              format: date-time
+              nullable: true
+              type: string
+            phase:
+              description: Phase is the current state of the BackupStorageLocation.
+              enum:
+              - Available
+              - Unavailable
+              type: string
+          type: object
+      type: object
+  version: v1
+  versions:
+  - name: v1
+    served: true
+    storage: true
+---
+apiVersion: velero.io/v1
+kind: BackupStorageLocation
+metadata:
+  creationTimestamp: null
+  labels:
+    component: velero
+  name: default
+  namespace: velero
+spec:
+  config:
+    region: minio
+    s3ForcePathStyle: "true"
+    s3Url: http://minio.velero.svc:9000
+  objectStorage:
+    bucket: velero
+  provider: aws

design/CLI/PoC/base/deployment.yaml

@@ -0,0 +1,89 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    component: velero
+  name: velero
+  namespace: velero
+spec:
+  selector:
+    matchLabels:
+      deploy: velero
+  strategy: {}
+  template:
+    metadata:
+      annotations:
+        prometheus.io/path: /metrics
+        prometheus.io/port: "8085"
+        prometheus.io/scrape: "true"
+      labels:
+        component: velero
+        deploy: velero
+    spec:
+      containers:
+        - args:
+            - server 
+          command:
+            - /velero
+          env:
+            - name: VELERO_SCRATCH_DIR
+              value: /scratch
+            - name: VELERO_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+            - name: LD_LIBRARY_PATH
+              value: /plugins
+          name: velero
+          image: velero/velero:latest
+          imagePullPolicy: Always
+          ports:
+            - containerPort: 8085
+              name: metrics
+          resources:
+            limits:
+              cpu: "1"
+              memory: 256Mi
+            requests:
+              cpu: 500m
+              memory: 128Mi
+          volumeMounts:
+            - mountPath: /scratch
+              name: scratch
+      restartPolicy: Always
+      serviceAccountName: velero
+      volumes:
+        - emptyDir: {}
+          name: scratch
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+  labels:
+    component: velero
+  name: velero
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cluster-admin
+subjects:
+  - kind: ServiceAccount
+    name: velero
+    namespace: velero
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  labels:
+    component: velero
+  name: velero
+  namespace: velero
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  labels:
+    component: velero
+  name: velero
+spec: {}

design/CLI/PoC/base/kustomization.yaml

@@ -0,0 +1,12 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - deployment.yaml
+  - CRDs.yaml
+  - backupstoragelocations.yaml
+  - volumesnapshotlocations.yaml # including so the velero server can run
+  - resticrepository.yaml # including so the velero server can runl
+  - podvolumes.yaml # including so the velero server can runl
+  - minio.yaml
+

design/CLI/PoC/base/minio.yaml

@@ -0,0 +1,107 @@
+# Copyright 2017 the Velero contributors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  namespace: velero
+  name: minio
+  labels:
+    component: minio
+spec:
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      component: minio
+  template:
+    metadata:
+      labels:
+        component: minio
+    spec:
+      volumes:
+      - name: storage
+        emptyDir: {}
+      - name: config
+        emptyDir: {}
+      containers:
+      - name: minio
+        image: minio/minio:latest
+        imagePullPolicy: IfNotPresent
+        args:
+        - server
+        - /storage
+        - --config-dir=/config
+        env:
+        - name: MINIO_ACCESS_KEY
+          value: "minio"
+        - name: MINIO_SECRET_KEY
+          value: "minio123"
+        ports:
+        - containerPort: 9000
+        volumeMounts:
+        - name: storage
+          mountPath: "/storage"
+        - name: config
+          mountPath: "/config"
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  namespace: velero
+  name: minio
+  labels:
+    component: minio
+spec:
+  # ClusterIP is recommended for production environments.
+  # Change to NodePort if needed per documentation,
+  # but only if you run Minio in a test/trial environment, for example with Minikube.
+  type: ClusterIP
+  ports:
+    - port: 9000
+      targetPort: 9000
+      protocol: TCP
+  selector:
+    component: minio
+
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  namespace: velero
+  name: minio-setup
+  labels:
+    component: minio
+spec:
+  template:
+    metadata:
+      name: minio-setup
+    spec:
+      restartPolicy: OnFailure
+      volumes:
+      - name: config
+        emptyDir: {}
+      containers:
+      - name: mc
+        image: minio/mc:latest
+        imagePullPolicy: IfNotPresent
+        command:
+        - /bin/sh
+        - -c
+        - "mc --config-dir=/config config host add velero http://minio:9000 minio minio123 && mc --config-dir=/config mb -p velero/velero"
+        volumeMounts:
+        - name: config
+          mountPath: "/config"

design/CLI/PoC/base/podvolumes.yaml

@@ -0,0 +1,297 @@
+---
+  apiVersion: apiextensions.k8s.io/v1beta1
+  kind: CustomResourceDefinition
+  metadata:
+    annotations:
+      controller-gen.kubebuilder.io/version: (unknown)
+    creationTimestamp: null
+    labels:
+      component: velero
+    name: podvolumebackups.velero.io
+  spec:
+    group: velero.io
+    names:
+      kind: PodVolumeBackup
+      listKind: PodVolumeBackupList
+      plural: podvolumebackups
+      singular: podvolumebackup
+    scope: ""
+    validation:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: PodVolumeBackupSpec is the specification for a PodVolumeBackup.
+            properties:
+              backupStorageLocation:
+                description: BackupStorageLocation is the name of the backup storage
+                  location where the restic repository is stored.
+                type: string
+              node:
+                description: Node is the name of the node that the Pod is running
+                  on.
+                type: string
+              pod:
+                description: Pod is a reference to the pod containing the volume to
+                  be backed up.
+                properties:
+                  apiVersion:
+                    description: API version of the referent.
+                    type: string
+                  fieldPath:
+                    description: 'If referring to a piece of an object instead of
+                      an entire object, this string should contain a valid JSON/Go
+                      field access statement, such as desiredState.manifest.containers[2].
+                      For example, if the object reference is to a container within
+                      a pod, this would take on a value like: "spec.containers{name}"
+                      (where "name" refers to the name of the container that triggered
+                      the event) or if no container name is specified "spec.containers[2]"
+                      (container with index 2 in this pod). This syntax is chosen
+                      only to have some well-defined way of referencing a part of
+                      an object. TODO: this design is not final and this field is
+                      subject to change in the future.'
+                    type: string
+                  kind:
+                    description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
+                    type: string
+                  name:
+                    description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                    type: string
+                  namespace:
+                    description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                    type: string
+                  resourceVersion:
+                    description: 'Specific resourceVersion to which this reference
+                      is made, if any. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency'
+                    type: string
+                  uid:
+                    description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                    type: string
+                type: object
+              repoIdentifier:
+                description: RepoIdentifier is the restic repository identifier.
+                type: string
+              tags:
+                additionalProperties:
+                  type: string
+                description: Tags are a map of key-value pairs that should be applied
+                  to the volume backup as tags.
+                type: object
+              volume:
+                description: Volume is the name of the volume within the Pod to be
+                  backed up.
+                type: string
+            required:
+              - backupStorageLocation
+              - node
+              - pod
+              - repoIdentifier
+              - volume
+            type: object
+          status:
+            description: PodVolumeBackupStatus is the current status of a PodVolumeBackup.
+            properties:
+              completionTimestamp:
+                description: CompletionTimestamp records the time a backup was completed.
+                  Completion time is recorded even on failed backups. Completion time
+                  is recorded before uploading the backup object. The server's time
+                  is used for CompletionTimestamps
+                format: date-time
+                nullable: true
+                type: string
+              message:
+                description: Message is a message about the pod volume backup's status.
+                type: string
+              path:
+                description: Path is the full path within the controller pod being
+                  backed up.
+                type: string
+              phase:
+                description: Phase is the current state of the PodVolumeBackup.
+                enum:
+                  - New
+                  - InProgress
+                  - Completed
+                  - Failed
+                type: string
+              progress:
+                description: Progress holds the total number of bytes of the volume
+                  and the current number of backed up bytes. This can be used to display
+                  progress information about the backup operation.
+                properties:
+                  bytesDone:
+                    format: int64
+                    type: integer
+                  totalBytes:
+                    format: int64
+                    type: integer
+                type: object
+              snapshotID:
+                description: SnapshotID is the identifier for the snapshot of the
+                  pod volume.
+                type: string
+              startTimestamp:
+                description: StartTimestamp records the time a backup was started.
+                  Separate from CreationTimestamp, since that value changes on restores.
+                  The server's time is used for StartTimestamps
+                format: date-time
+                nullable: true
+                type: string
+            type: object
+        type: object
+    version: v1
+    versions:
+      - name: v1
+        served: true
+        storage: true
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (unknown)
+  creationTimestamp: null
+  labels:
+    component: velero
+  name: podvolumerestores.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: PodVolumeRestore
+    listKind: PodVolumeRestoreList
+    plural: podvolumerestores
+    singular: podvolumerestore
+  scope: ""
+  validation:
+    openAPIV3Schema:
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: PodVolumeRestoreSpec is the specification for a PodVolumeRestore.
+          properties:
+            backupStorageLocation:
+              description: BackupStorageLocation is the name of the backup storage
+                location where the restic repository is stored.
+              type: string
+            pod:
+              description: Pod is a reference to the pod containing the volume to
+                be restored.
+              properties:
+                apiVersion:
+                  description: API version of the referent.
+                  type: string
+                fieldPath:
+                  description: 'If referring to a piece of an object instead of
+                    an entire object, this string should contain a valid JSON/Go
+                    field access statement, such as desiredState.manifest.containers[2].
+                    For example, if the object reference is to a container within
+                    a pod, this would take on a value like: "spec.containers{name}"
+                    (where "name" refers to the name of the container that triggered
+                    the event) or if no container name is specified "spec.containers[2]"
+                    (container with index 2 in this pod). This syntax is chosen
+                    only to have some well-defined way of referencing a part of
+                    an object. TODO: this design is not final and this field is
+                    subject to change in the future.'
+                  type: string
+                kind:
+                  description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
+                  type: string
+                name:
+                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                  type: string
+                namespace:
+                  description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                  type: string
+                resourceVersion:
+                  description: 'Specific resourceVersion to which this reference
+                    is made, if any. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency'
+                  type: string
+                uid:
+                  description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                  type: string
+              type: object
+            repoIdentifier:
+              description: RepoIdentifier is the restic repository identifier.
+              type: string
+            snapshotID:
+              description: SnapshotID is the ID of the volume snapshot to be restored.
+              type: string
+            volume:
+              description: Volume is the name of the volume within the Pod to be
+                restored.
+              type: string
+          required:
+            - backupStorageLocation
+            - pod
+            - repoIdentifier
+            - snapshotID
+            - volume
+          type: object
+        status:
+          description: PodVolumeRestoreStatus is the current status of a PodVolumeRestore.
+          properties:
+            completionTimestamp:
+              description: CompletionTimestamp records the time a restore was completed.
+                Completion time is recorded even on failed restores. The server's
+                time is used for CompletionTimestamps
+              format: date-time
+              nullable: true
+              type: string
+            message:
+              description: Message is a message about the pod volume restore's status.
+              type: string
+            phase:
+              description: Phase is the current state of the PodVolumeRestore.
+              enum:
+                - New
+                - InProgress
+                - Completed
+                - Failed
+              type: string
+            progress:
+              description: Progress holds the total number of bytes of the snapshot
+                and the current number of restored bytes. This can be used to display
+                progress information about the restore operation.
+              properties:
+                bytesDone:
+                  format: int64
+                  type: integer
+                totalBytes:
+                  format: int64
+                  type: integer
+              type: object
+            startTimestamp:
+              description: StartTimestamp records the time a restore was started.
+                The server's time is used for StartTimestamps
+              format: date-time
+              nullable: true
+              type: string
+          type: object
+      type: object
+  version: v1
+  versions:
+    - name: v1
+      served: true
+      storage: true

design/CLI/PoC/base/resticrepository.yaml

@@ -0,0 +1,85 @@
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (unknown)
+  creationTimestamp: null
+  labels:
+    component: velero
+  name: resticrepositories.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: ResticRepository
+    listKind: ResticRepositoryList
+    plural: resticrepositories
+    singular: resticrepository
+  scope: ""
+  validation:
+    openAPIV3Schema:
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: ResticRepositorySpec is the specification for a ResticRepository.
+          properties:
+            backupStorageLocation:
+              description: BackupStorageLocation is the name of the BackupStorageLocation
+                that should contain this repository.
+              type: string
+            maintenanceFrequency:
+              description: MaintenanceFrequency is how often maintenance should
+                be run.
+              type: string
+            resticIdentifier:
+              description: ResticIdentifier is the full restic-compatible string
+                for identifying this repository.
+              type: string
+            volumeNamespace:
+              description: VolumeNamespace is the namespace this restic repository
+                contains pod volume backups for.
+              type: string
+          required:
+            - backupStorageLocation
+            - maintenanceFrequency
+            - resticIdentifier
+            - volumeNamespace
+          type: object
+        status:
+          description: ResticRepositoryStatus is the current status of a ResticRepository.
+          properties:
+            lastMaintenanceTime:
+              description: LastMaintenanceTime is the last time maintenance was
+                run.
+              format: date-time
+              nullable: true
+              type: string
+            message:
+              description: Message is a message about the current status of the
+                ResticRepository.
+              type: string
+            phase:
+              description: Phase is the current state of the ResticRepository.
+              enum:
+                - New
+                - Ready
+                - NotReady
+              type: string
+          type: object
+      type: object
+  version: v1
+  versions:
+    - name: v1
+      served: true
+      storage: true

design/CLI/PoC/base/volumesnapshotlocations.yaml

@@ -0,0 +1,80 @@
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (unknown)
+  labels:
+    component: velero
+  name: volumesnapshotlocations.velero.io
+spec:
+  group: velero.io
+  names:
+    kind: VolumeSnapshotLocation
+    listKind: VolumeSnapshotLocationList
+    plural: volumesnapshotlocations
+    singular: volumesnapshotlocation
+  scope: ""
+  validation:
+    openAPIV3Schema:
+      description: VolumeSnapshotLocation is a location where Velero stores volume
+        snapshots.
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: VolumeSnapshotLocationSpec defines the specification for
+            a Velero VolumeSnapshotLocation.
+          properties:
+            config:
+              additionalProperties:
+                type: string
+              description: Config is for provider-specific configuration fields.
+              type: object
+            provider:
+              description: Provider is the provider of the volume storage.
+              type: string
+          required:
+            - provider
+          type: object
+        status:
+          description: VolumeSnapshotLocationStatus describes the current status
+            of a Velero VolumeSnapshotLocation.
+          properties:
+            phase:
+              description: VolumeSnapshotLocationPhase is the lifecyle phase of
+                a Velero VolumeSnapshotLocation.
+              enum:
+                - Available
+                - Unavailable
+              type: string
+          type: object
+      type: object
+  version: v1
+  versions:
+  - name: v1
+    served: true
+    storage: true
+---
+apiVersion: velero.io/v1
+kind: VolumeSnapshotLocation
+metadata:
+  creationTimestamp: null
+  labels:
+    component: velero
+  name: default
+  namespace: velero
+spec:
+  config:
+    region: us-east-2
+  provider: aws

design/CLI/PoC/overlays/plugins/aws-plugin.yaml

@@ -0,0 +1,40 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: velero
+spec:
+  selector:
+    matchLabels:
+      deploy: velero
+  template:
+    metadata:
+      labels:
+        component: velero
+        deploy: velero
+    spec:
+      containers:
+        - args:
+          - server
+          name: velero
+          env:
+            - name: AWS_SHARED_CREDENTIALS_FILE
+              value: /credentials/cloud
+          volumeMounts:
+            - mountPath: /plugins
+              name: plugins
+            - mountPath: /credentials
+              name: cloud-credential-aws
+      initContainers:
+        - image: velero/velero-plugin-for-aws:v1.0.1
+          imagePullPolicy: Always
+          name: velero-plugin-for-aws
+          volumeMounts:
+          - mountPath: /target
+            name: plugins
+      volumes:
+        - emptyDir: {}
+          name: plugins
+        - name: cloud-credential-aws
+          secret:
+            secretName: cloud-credential-aws

design/CLI/PoC/overlays/plugins/azure-plugin.yaml

@@ -0,0 +1,40 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: velero
+spec:
+  selector:
+    matchLabels:
+      deploy: velero
+  template:
+    metadata:
+      labels:
+        component: velero
+        deploy: velero
+    spec:
+      containers:
+        - args:
+          - server
+          name: velero
+          env:
+            - name: AZURE_SHARED_CREDENTIALS_FILE
+              value: /credentials/cloud
+          volumeMounts:
+            - mountPath: /plugins
+              name: plugins
+            - mountPath: /credentials
+              name: cloud-credential-azure
+      initContainers:
+        - image: velero/velero-plugin-for-microsoft-azure:v1.0.1
+          imagePullPolicy: Always
+          name: velero-plugin-for-microsoft-azure
+          volumeMounts:
+          - mountPath: /target
+            name: plugins
+      volumes:
+        - emptyDir: {}
+          name: plugins
+        - name: cloud-credential-azure
+          secret:
+            secretName: cloud-credential-azure

design/CLI/PoC/overlays/plugins/cloud

@@ -0,0 +1,3 @@
+[default]
+aws_access_key_id = minio
+aws_secret_access_key = minio123

design/CLI/PoC/overlays/plugins/kustomization.yaml

@@ -0,0 +1,24 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+bases:
+  - ../../base
+
+patchesStrategicMerge:
+  - aws-plugin.yaml # this patches the Velero deployment
+  # - azure-plugin.yaml # this patches the Velero deployment
+
+generatorOptions:
+  disableNameSuffixHash: true
+  labels: 
+    component: velero
+
+secretGenerator:
+- name: cloud-credentials
+  files:
+  - "cloud"
+
+
+
+
+

design/CLI/PoC/overlays/plugins/restic.yaml

@@ -0,0 +1,68 @@
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  creationTimestamp: null
+  labels:
+    component: velero
+  name: restic
+  namespace: velero
+spec:
+  selector:
+    matchLabels:
+      name: restic
+  template:
+    metadata:
+      creationTimestamp: null
+      labels:
+        component: velero
+        name: restic
+    spec:
+      containers:
+        - args:
+            - restic
+            - server
+          command:
+            - /velero
+          env:
+            - name: NODE_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: spec.nodeName
+            - name: VELERO_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+            - name: VELERO_SCRATCH_DIR
+              value: /scratch
+            - name: GOOGLE_APPLICATION_CREDENTIALS
+              value: /credentials/cloud
+            - name: AWS_SHARED_CREDENTIALS_FILE
+              value: /credentials/cloud
+            - name: AZURE_CREDENTIALS_FILE
+              value: /credentials/cloud
+          image: velero/velero:latest
+          imagePullPolicy: Always
+          name: restic
+          resources: {}
+          volumeMounts:
+            - mountPath: /host_pods
+              mountPropagation: HostToContainer
+              name: host-pods
+            - mountPath: /scratch
+              name: scratch
+            - mountPath: /credentials
+              name: cloud-credentials
+      securityContext:
+        runAsUser: 0
+      serviceAccountName: velero
+      volumes:
+        - hostPath:
+            path: /var/lib/kubelet/pods
+          name: host-pods
+        - emptyDir: {}
+          name: scratch
+        - name: cloud-credentials
+          secret:
+            secretName: cloud-credentials
+  updateStrategy: {}

design/_template.md

@@ -1,37 +1,35 @@
-# Design proposal template (replace with your proposal's title)
-
-Status: {Draft,Accepted,Declined}
-
-One to two sentences that describes the goal of this proposal.
-The reader should be able to tell by the title, and the opening paragraph, if this document is relevant to them.
+# Design proposal template `<replace with your proposal's title>`
 
 _Note_: The preferred style for design documents is one sentence per line.
 *Do not wrap lines*.
 This aids in review of the document as changes to a line are not obscured by the reflowing those changes caused and has a side effect of avoiding debate about one or two space after a period.
 
-## Goals
+_Note_: The name of the file should follow the name pattern `<short meaningful words joined by '-'>_design.md`, e.g:
+`listener-design.md`.
 
+## Abstract
+One to two sentences that describes the goal of this proposal and the problem being solved by the proposed change.
+The reader should be able to tell by the title, and the opening paragraph, if this document is relevant to them.
+
+## Background
+One to two paragraphs of exposition to set the context for this proposal.
+
+## Goals
 - A short list of things which will be accomplished by implementing this proposal.
 - Two things is ok.
 - Three is pushing it.
 - More than three goals suggests that the proposal's scope is too large.
 
 ## Non Goals
-
 - A short list of items which are:
 - a. out of scope
 - b. follow on items which are deliberately excluded from this proposal.
 
-## Background
-
-One to two paragraphs of exposition to set the context for this proposal.
 
 ## High-Level Design
-
 One to two paragraphs that describe the high level changes that will be made to implement this proposal.
 
 ## Detailed Design
-
 A detailed design describing how the changes to the product should be made.
 
 The names of types, fields, interfaces, and methods should be agreed on here, not debated in code review.
@@ -40,9 +38,16 @@ The same applies to changes in CRDs, YAML examples, and so on.
 Ideally the changes should be made in sequence so that the work required to implement this design can be done incrementally, possibly in parallel.
 
 ## Alternatives Considered
-
 If there are alternative high level or detailed designs that were not pursued they should be called out here with a brief explanation of why they were not pursued.
 
 ## Security Considerations
-
 If this proposal has an impact to the security of the product, its users, or data stored or transmitted via the product, they must be addressed here.
+
+## Compatibility
+A discussion of any compatibility issues that need to be considered
+
+## Implementation
+A description of the implementation, timelines, and any resources that have agreed to contribute.
+
+## Open Issues
+A discussion of issues relating to this proposal for which the author does not know the solution. This section may be omitted if there are none.

design/backup-resources-order.md

@@ -0,0 +1,40 @@
+## Backup Resources Order
+This document proposes a solution that allows user to specify a backup order for resources of specific resource type.
+
+## Background
+During backup process, user may need to back up resources of specific type in some specific order to ensure the resources were backup properly because these resources are related and ordering might be required to preserve the consistency for the apps to recover itself <20>from the backup image 
+(Ex: primary-secondary database pods in a cluster).
+
+## Goals
+- Enable user to specify an order of back up resources belong to specific resource type
+
+## Alternatives Considered
+- Use a plugin to backup an resources and all the sub resources.  For example use a plugin for StatefulSet and backup pods belong to the StatefulSet in specific order.  This plugin solution is not generic and requires plugin for each resource type.
+
+## High-Level Design
+User will specify a map of resource type to list resource names (separate by semicolons).  Each name will be in the format "namespaceName/resourceName" to enable ordering accross namespaces.  Based on this map, the resources of each resource type will be sorted by the order specified in the list of resources.  If a resource instance belong to that specific type but its name is not in the order list, then it will be put behind other resources that are in the list.
+
+### Changes to BackupSpec
+Add new field to BackupSpec
+
+    type BackupSpec struct {
+        ...
+        // OrderedResources contains a list of key-value pairs that represent the order
+        // of backup of resources that belong to specific resource type
+        // +optional
+        // +nullable
+        OrderedResources map[string]string
+    }
+
+### Changes to itemCollector
+Function getResourceItems collects all items belong to a specific resource type.  This function will be enhanced to check with the map to see whether the OrderedResources has specified the order for this resource type.  If such order exists, then sort the items by such order being process before return.
+
+### Changes to velero CLI
+Add new flag "--ordered-resources" to Velero backup create command which takes a string of key-values pairs which represents the map between resource type and the order of the items of such resource type. Key-value pairs are separated by semicolon, items in the value are separated by commas.
+
+Example:
+>velero backup create mybackup --ordered-resources "pod=ns1/pod1,ns1/pod2;persistentvolumeclaim=n2/slavepod,ns2/primarypod"
+
+## Open Issues
+- In the CLI, the design proposes to use commas to separate items of a resource type and semicolon to separate key-value pairs.  This follows the convention of using commas to separate items in a list (For example: --include-namespaces ns1,ns2).  However, the syntax for map in labels and annotations use commas to seperate key-value pairs.  So it introduces some inconsistency.
+- For pods that managed by Deployment or DaemonSet, this design may not work because the pods' name is randomly generated and if pods are restarted, they would have different names so the Backup operation may not consider the restarted pods in the sorting algorithm.  This problem will be addressed when we enhance the design to use regular expression to specify the OrderResources instead of exact match. 

design/cli-install-changes.md

@@ -0,0 +1,373 @@
+# Proposal for a more intuitive CLI to install and configure Velero
+
+Currently, the Velero CLI tool has a `install` command that configures numerous major and minor aspects of Velero. As a result, the combined set of flags for this `install` command makes it hard to intuit and reason about the different Velero components. This document proposes changes to improve the UX for installation and configuration in a way that would make it easier for the user to discover what needs to be configured by looking at what is available in the CLI rather then having to rely heavily on our documentation for the usage. At the same time, it is expected that the documentation update to reflect these changes will also make the documentation flow easier to follow.
+
+This proposal prioritizes discoverability and self-documentation over minimizing length or number of commands and flags.
+
+## Goals
+
+- Split flags currently under the `velero install` command into multiple commands, and group flags under commands in a way that allows a good level of discovery and self-documentation
+- Maintain compatibility with gitops practices (i.e. ability to generate a full set of yaml for install that can be stored in source control)
+- Have a clear path for deprecating commands
+
+## Non Goals
+
+- Introduce new CLI features 
+- Propose changes to the CLI that go beyond the functionality of install and configure
+- Optimize for shorter length or number of commands/flags
+
+## Background
+
+This document proposes users could benefit from a more intuitive and self-documenting CLI setup as compared to our existing CLI UX. Ultimately, it is proposed that a recipe-style CLI flow for installation, configuration and use would greatly contribute to this purpose.
+
+Also, the `install` command currently can be reused to update Velero deployment configurations. For server and restic related install and configurations, settings will be moved to under `velero config`.
+
+## High-Level Design
+
+The naming and organization of the proposed new CLI commands below have been inspired on the `kubectl` commands, particularly `kubectl set` and `kubectl config`.
+
+#### General CLI improvements
+
+These are improvements that are part of this proposal:
+- Go over all flags and document what is optional, what is required, and default values.
+- Capitalize all help messages
+
+#### Commands
+
+The organization of the commands follows this format:
+
+```
+velero [resource] [operation] [flags]
+```
+
+To conform with Velero's current practice: 
+- commands will also work by swapping the operation/resource.
+- the "object" of a command is an argument, and flags are strictly for modifiers (example: `backup get my-backup` and not `backup get --name my-backup`)
+
+All commands will include the `--dry-run` flag, which can be used to output yaml files containing the commands' configuration for resource creation or patching.
+
+`--dry-run                                    generate resources, but don't send them to the cluster. Use with -o. Optional.`
+
+The `--help` and `--output` flags will also be included for all commands, omitted below for brevity.
+
+Below is the proposed set of new commands to setup and configure Velero.
+
+1) `velero config`
+
+```
+      server                                         Configure up the namespace, RBAC, deployment, etc., but does not add any external plugins, BSL/VSL definitions. This would be the minimum set of commands to get the Velero server up and running and ready to accept other configurations.
+        --label-columns stringArray                  a comma-separated list of labels to be displayed as columns
+        --show-labels                                show labels in the last column
+        --image string                               image to use for the Velero and restic server pods. Optional. (default "velero/velero:latest")
+        --pod-annotations mapStringString            annotations to add to the Velero and restic pods. Optional. Format is key1=value1,key2=value2
+        --restore-only                               run the server in restore-only mode. Optional.
+        --pod-cpu-limit string                       CPU limit for Velero pod. A value of "0" is treated as unbounded. Optional. (default "1000m")
+        --pod-cpu-request string                     CPU request for Velero pod. A value of "0" is treated as unbounded. Optional. (default "500m")
+        --pod-mem-limit string                       memory limit for Velero pod. A value of "0" is treated as unbounded. Optional. (default "256Mi")
+        --pod-mem-request string                     memory request for Velero pod. A value of "0" is treated as unbounded. Optional. (default "128Mi")
+        --client-burst int                           maximum number of requests by the server to the Kubernetes API in a short period of time (default 30)
+        --client-qps float32                         maximum number of requests per second by the server to the Kubernetes API once the burst limit has been reached (default 20)
+        --default-backup-ttl duration                how long to wait by default before backups can be garbage collected (default 720h0m0s)
+        --disable-controllers strings                list of controllers to disable on startup. Valid values are backup,backup-sync,schedule,gc,backup-deletion,restore,download-request,restic-repo,server-status-request
+        --log-format                                 the format for log output. Valid values are text, json. (default text)
+        --log-level                                  the level at which to log. Valid values are debug, info, warning, error, fatal, panic. (default info)
+        --metrics-address string                     the address to expose prometheus metrics (default ":8085")
+        --plugin-dir string                          directory containing Velero plugins (default "/plugins")
+        --profiler-address string                    the address to expose the pprof profiler (default "localhost:6060")
+        --restore-only                               run in a mode where only restores are allowed; backups, schedules, and garbage-collection are all disabled. DEPRECATED: this flag will be removed in v2.0. Use read-only backup storage locations instead.
+        --restore-resource-priorities strings        desired order of resource restores; any resource not in the list will be restored alphabetically after the prioritized resources (default [namespaces,storageclasses,persistentvolumes,persistentvolumeclaims,secrets,configmaps,serviceaccounts,limitranges,pods,replicaset,customresourcedefinitions])
+        --terminating-resource-timeout duration      how long to wait on persistent volumes and namespaces to terminate during a restore before timing out (default 10m0s)
+
+      restic                                        Configuration for restic operations.
+        --default-prune-frequency duration          how often 'restic prune' is run for restic repositories by default. Optional.
+        --pod-annotations mapStringString           annotations to add to the Velero and restic pods. Optional. Format is key1=value1,key2=value2
+        --pod-cpu-limit string                      CPU limit for restic pod. A value of "0" is treated as unbounded. Optional. (default "0")
+        --pod-cpu-request string                    CPU request for restic pod. A value of "0" is treated as unbounded. Optional. (default "0")
+        --pod-mem-limit string                      memory limit for restic pod. A value of "0" is treated as unbounded. Optional. (default "0")
+        --pod-mem-request string                    memory request for restic pod. A value of "0" is treated as unbounded. Optional. (default "0")
+        --timeout duration                          how long backups/restores of pod volumes should be allowed to run before timing out (default 1h0m0s)
+        repo  
+          get                                         Get restic repositories
+```
+The `velero config server` command will create the following resources:
+
+```
+Namespace
+Deployment
+backups.velero.io
+backupstoragelocations.velero.io
+deletebackuprequests.velero.io
+downloadrequests.velero.io
+podvolumebackups.velero.io
+podvolumerestores.velero.io
+resticrepositories.velero.io
+restores.velero.io
+schedules.velero.io
+serverstatusrequests.velero.io
+volumesnapshotlocations.velero.io
+```
+
+Note: Velero will maintain the `velero server` command run by the Velero pod, which starts the Velero server deployment.
+
+2) `velero backup-location`
+Commands/flags for backup locations.
+
+```
+      set
+        --default string                                  sets the default backup storage location (default "default") (NEW, -- was `server --default-backup-storage-location; could be set as an annotation on the BSL)
+        --credentials mapStringString                     sets the name of the corresponding credentials secret for a provider. Format is provider:credentials-secret-name. (NEW)
+        --cacert-file mapStringString                     configuration to use for creating a secret containing a custom certificate for an S3 location of a plugin provider. Format is provider:path-to-file. (NEW)
+
+      create                                              NAME [flags]
+        --default                                         Sets this new location to be the new default backup location. Default is false. (NEW) 
+        --access-mode                                     access mode for the backup storage location. Valid values are ReadWrite,ReadOnly (default ReadWrite)
+        --backup-sync-period 0s                           how often to ensure all Velero backups in object storage exist as Backup API objects in the cluster. Optional. Set this to 0s to disable sync
+        --bucket string                                   name of the object storage bucket where backups should be stored. Required.
+        --config mapStringString                          configuration to use for creating a backup storage location. Format is key1=value1,key2=value2 (was also in `velero install --backup-location-config`). Required for Azure.
+        --provider string                                 provider name for backup storage. Required.
+        --label-columns stringArray                       a comma-separated list of labels to be displayed as columns
+        --labels mapStringString                          labels to apply to the backup storage location
+        --prefix string                                   prefix under which all Velero data should be stored within the bucket. Optional.
+        --provider string                                 name of the backup storage provider (e.g. aws, azure, gcp)
+        --show-labels                                     show labels in the last column
+        --credentials mapStringString                     sets the name of the corresponding credentials secret for a provider. Format is provider:credentials-secret-name. (NEW)
+        --cacert-file mapStringString                     configuration to use for creating a secret containing a custom certificate for an S3 location of a plugin provider. Format is provider:path-to-file. (NEW)
+
+      get                                                 Display backup storage locations
+        --default                                         displays the current default backup storage location (NEW)
+        --label-columns stringArray                       a comma-separated list of labels to be displayed as columns
+        -l, --selector string                             only show items matching this label selector
+        --show-labels                                     show labels in the last column
+
+```
+
+3) `velero snapshot-location`
+Commands/flags for snapshot locations.
+
+```
+     set
+        --default mapStringString                         sets the list of unique volume providers and default volume snapshot location (provider1:location-01,provider2:location-02,...) (NEW, -- was `server --default-volume-snapshot-locations; could be set as an annotation on the VSL)  
+        --credentials mapStringString                     sets the list of name of the corresponding credentials secret for providers. Format is (provider1:credentials-secret-name1,provider2:credentials-secret-name2,...) (NEW)
+
+      create                                              NAME [flags]
+        --default                                         Sets these new locations to be the new default snapshot locations. Default is false. (NEW)
+        --config  mapStringString                         configuration to use for creating a volume snapshot location. Format is key1=value1,key2=value2 (was also in `velero install --`snapshot-location-config`). Required.
+        --provider string                                 provider name for volume storage. Required.
+        --label-columns stringArray                       a comma-separated list of labels to be displayed as columns
+        --labels mapStringString                          labels to apply to the volume snapshot location
+        --provider string                                 name of the volume snapshot provider (e.g. aws, azure, gcp)
+        --show-labels                                     show labels in the last column
+        --credentials mapStringString                     sets the list of name of the corresponding credentials secret for providers. Format is (provider1:credentials-secret-name1,provider2:credentials-secret-name2,...) (NEW)
+
+      get                                                 Display snapshot locations
+        --default                                         list of unique volume providers and default volume snapshot location (provider1:location-01,provider2:location-02,...) (NEW -- was `server --default-volume-snapshot-locations`))
+```
+
+4) `velero plugin`
+Configuration for plugins.
+
+```
+      add stringArray                           IMAGES [flags] - add plugin container images to install into the Velero Deployment
+
+      get                                       get information for all plugins on the velero server (was `get`)
+        --timeout duration                      maximum time to wait for plugin information to be reported (default 5s)
+
+      remove                                    Remove a plugin [NAME | IMAGE] 
+
+      set
+        --credentials-file mapStringString      configuration to use for creating a secret containing the AIM credentials for a plugin provider. Format is provider:path-to-file. (was `secret-file`)
+        --no-secret                             flag indicating if a secret should be created. Must be used as confirmation if create --secret-file is not provided. Optional. (MOVED FROM install -- not sure we need it?)
+        --sa-annotations mapStringString        annotations to add to the Velero ServiceAccount for GKE. Add iam.gke.io/gcp-service-account=[GSA_NAME]@[PROJECT_NAME].iam.gserviceaccount.com for workload identity. Optional. Format is key1=value1,key2=value2
+```
+
+#### Example
+
+Considering this proposal, let's consider what a high-level documentation for getting Velero ready to do backups could look like for Velero users:
+
+After installing the Velero CLI:
+```
+velero config server [flags] (required)
+velero config restic [flags]
+velero plugin add IMAGES [flags] (add/config provider plugins)
+velero backup-location/snapshot-location create NAME [flags] (run `velero plugin --get` to see what kind of plugins are available; create locations)
+velero backup/restore/schedule create/get/delete NAME [flags]
+```
+
+The above recipe-style documentation should highlight 1) the main components of Velero, and, 2) the relationship/dependency between the main components
+
+### Deprecation
+
+#### Timeline
+
+In order to maintain compatibility with the current Velero version for a sufficient amount of time, and give users a chance to upgrade any install scripts they might have, we will keep the current `velero install` command in parallel with the new commands until the next major Velero version, which will be Velero 2.0. In the mean time, ia deprecation warning will be added to the `velero install` command. 
+
+#### Commands/flags deprecated or moved
+
+##### Velero Install 
+`velero install (DEPRECATED)`
+
+Flags moved to...
+
+...`velero config server`:
+```
+      --image string                               image to use for the Velero and restic server pods. Optional. (default "velero/velero:latest")
+      --label-columns stringArray                  a comma-separated list of labels to be displayed as columns
+      --pod-annotations mapStringString            annotations to add to the Velero and restic pods. Optional. Format is key1=value1,key2=value2
+      --show-labels                                show labels in the last column
+      --pod-cpu-limit string                CPU limit for Velero pod. A value of "0" is treated as unbounded. Optional. (default "1000m")
+      --pod-cpu-request string              CPU request for Velero pod. A value of "0" is treated as unbounded. Optional. (default "500m")
+      --pod-mem-limit string                memory limit for Velero pod. A value of "0" is treated as unbounded. Optional. (default "256Mi")
+      --pod-mem-request string              memory request for Velero pod. A value of "0" is treated as unbounded. Optional. (default "128Mi")
+```
+
+...`velero config restic`
+```
+      --default-prune-frequency duration    how often 'restic prune' is run for restic repositories by default. Optional.
+      --pod-cpu-limit string                CPU limit for restic pod. A value of "0" is treated as unbounded. Optional. (default "0")
+      --pod-cpu-request string              CPU request for restic pod. A value of "0" is treated as unbounded. Optional. (default "0")
+      --pod-mem-limit string                memory limit for restic pod. A value of "0" is treated as unbounded. Optional. (default "0")
+      --pod-mem-request string              memory request for restic pod. A value of "0" is treated as unbounded. Optional. (default "0")
+```
+
+...`backup-location create`
+```
+      --backup-location-config mapStringString     configuration to use for the backup storage location. Format is key1=value1,key2=value2
+      --bucket string                              name of the object storage bucket where backups should be stored
+      --prefix string                              prefix under which all Velero data should be stored within the bucket. Optional.
+```
+
+...`snapshot-location create`
+```
+      --snapshot-location-config mapStringString   configuration to use for the volume snapshot location. Format is key1=value1,key2=value2
+```
+
+...both `backup-location create` and `snapshot-location create`
+```
+      --provider string                            provider name for backup and volume storage
+```
+
+...`plugin`
+```
+      --plugins stringArray                        Plugin container images to install into the Velero Deployment
+      --sa-annotations mapStringString             annotations to add to the Velero ServiceAccount. Add iam.gke.io/gcp-service-account=[GSA_NAME]@[PROJECT_NAME].iam.gserviceaccount.com for workload identity. Optional. Format is key1=value1,key2=value2
+      --no-secret                                  flag indicating if a secret should be created. Must be used as confirmation if --secret-file is not provided. Optional.
+      --secret-file string  (renamed `credentials-file`)   file containing credentials for backup and volume provider. If not specified, --no-secret must be used for confirmation. Optional.
+```
+
+Flags to deprecate:
+```
+      --no-default-backup-location                 flag indicating if a default backup location should be created. Must be used as confirmation if --bucket or --provider are not provided. Optional.
+      --use-volume-snapshots                       whether or not to create snapshot location automatically. Set to false if you do not plan to create volume snapshots via a storage provider. (default true)
+      --wait                                       wait for Velero deployment to be ready. Optional.
+      --use-restic                                 (obsolete since now we have `velero config restic`)
+```
+
+##### Velero Server
+
+These flags will be moved to under `velero config server`:
+
+`velero server --default-backup-storage-location (DEPRECATED)` changed to `velero backup-location set --default`
+
+`velero server --default-volume-snapshot-locations (DEPRECATED)` changed to `velero snapshot-location set --default`
+
+The value for these flags will be stored as annotations.
+
+## Detailed Design
+
+#### Handling CA certs
+
+In anticipation of a new configuration implementation to handle custom CA certs (as per design doc https://github.com/vmware-tanzu/velero/blob/master/design/custom-ca-support.md), a new flag `velero storage-location create/set --cacert-file mapStringString` is proposed. It sets the configuration to use for creating a secret containing a custom certificate for an S3 location of a plugin provider. Format is provider:path-to-file.
+
+See discussion https://github.com/vmware-tanzu/velero/pull/2259#discussion_r384700723 for more clarification.
+
+#### Renaming "provider" to "location-plugin"
+
+As part of this change, we should change to use the term `location-plugin` instead of `provider`. The reasoning: in practice, we usually have 1 plugin per provider, and if there is an implementation for both object store and volume snapshotter for that provider, it will all be contained in the same plugin. When we handle plugins, we follow this logic. In other words, there's a plugin name (ex: `velero.io/aws`) and it can contain implementations of kind `ObjectStore` and/or `VolumeSnapshotter`.
+
+But when we handle BSL or VSL (and the CLI commands/flags that configure them), we use the term `provider`, which can cause ambiguity as if that is a kind of thing different from a plugin. If the plugin is the "thing" that contains the implementation for the desired provider, we should make it easier for the user to guess that and change BackupStorageLocation/VolumeSnapshotLocation `Spec.Provider` field to be called `Spec.Location-Plugin` and all related CLI command flags to `location-plugin`, and update the docs accordingly.
+
+This change will require a CRD version bump and deprecation cycle.
+
+#### GitOps Compatibility
+
+To maintain compatibility with gitops practices, each of the new commands will generate `yaml` output that can be stored in source control.
+
+For content examples, please refer to the files here:
+
+https://github.com/carlisia/velero/tree/c-cli-design/design/CLI/PoC
+
+Note: actual `yaml` file names are defined by the user.
+
+`velero config server` - base/deployment.yaml
+
+`velero config restic` - overlays/plugins/restic.yaml
+
+`velero backup-location create` - base/backupstoragelocations.yaml
+
+`velero snapshot-location create` - base/volumasnapshotlocations.yaml
+
+`velero plugin add velero/velero-plugin-for-aws:v1.0.1` - overlays/plugins/aws-plugin.yaml
+
+`velero plugin add velero/velero-plugin-for-microsoft-azure:v1.0.1` - overlay/plugins/azure-plugin.yaml
+
+These resources can be deployed/deleted using the included kustomize setup and running:
+
+```
+kubectl apply -k design/CLI/PoC/overlays/plugins/
+
+kubectl delete -k design/CLI/PoC/overlays/plugins/
+```
+
+Note: All CRDs, including the `ResticRepository`, may continue to be deployed at startup as it is now, or together with their respective instantiation.
+
+
+#### Changes to startup behavior
+
+To recap, this proposal redesigns the Velero CLI to make `velero install` obsolete, and instead breaks down the installation and configuration into separate commands. These are the major highlights:
+
+- Plugins will only be installed separately via `velero plugin add`
+- BSL/VSL will be continue to be configured separately, and now each will have an associated secret
+
+Since each BSL/VSL will have its own association with a secret, the user will no longer need to upload a new secret whenever changing to, or adding, a BSL/VSL for a provider that is different from the one in use. This will be done at setup time. This will make it easier to support any number of BSL/VSL combinations, with different providers each.
+
+The user will start up the Velero server on a cluster by using the command `velero config server`. This will create the Velero deployment resource with default values or values overwritten with flags, create the Velero CRDs, and anything else that is not specific to plugins or BSL/VSL.
+
+The Velero server will start up, verify that the deployment is running, that all CRDs were found, and log a message that it is waiting for a BSL to be configured. at this point, other operations, such as configuring restic, will be allowed. Velero should keep track of its status, ie, if it is ready to create backups or not. This could be a field `ServerStatus` added to `ServerStatusRequest`. Possible values could be [ready|waiting]. "ready" would mean there is at least 1 valid BSL, and "waiting" would be anything but that.
+
+When adding/configuring a BSL or VSL, we will allow creating locations, and continuously verify if there is a corresponding, valid plugin. When a valid match is found, mark the BSL/VSL as "ready". This would require adding a field to the BSL/VSL, or using the existing `Phase` field, and keep track of its status, possibly: [ready|waiting].
+
+With the first approach: the server would transition into "ready" (to create backups) as soon as there is one BSL. It would require a set sequence of actions, ie, first install the plugin, only then the user can successfully configure a BSL.
+
+With the second approach, the Velero server would continue looping and checking all existing BSLs for at least 1 with a "ready" status. Once it found that, it would set itself to "ready" also.
+
+Another new behavior that must be added: the server needs to identify when there no longer exists a valid BSL. At this point, it should change its status from "ready" to one that indicates it is not ready, maybe "waiting". With the first approach above, this would mean checking if there is still at least one BSL. With the second approach, it would require checking the status of all BSLs to find at least one with the status of "ready".
+
+As it is today, a valid VSL would not be required to create backups, unless the backup included a PV.
+
+To make it easier for the user to identify if their Velero server is ready to create backups or not, a `velero status` command should be added. This issue has been created some time ago for this purpose: https://github.com/vmware-tanzu/velero/issues/1094.
+
+## Alternatives Considered
+
+It seems that the vast majority of tools document their usage with `kubectl` and `yaml` files to install and configure their Kubernetes resources. Many of them also make use of Helm, and to a lesser extent some of them have their own CLI tools. 
+
+Amongst the tools that have their own CLI, not enough examples were found to establish a clear pattern of usage. It seems the most relevant priority should be to have output in `yaml` format.
+
+Any set of `yaml` files can also be arranged to use with Kustomize by creating/updating resources, and patching them using Kustomize functionalities.
+
+The way the Velero commands were arranged in this proposal with the ability to output corresponding `yaml` files, and the included Kustomize examples, makes it in line with the widely used practices for installation and configuration.
+
+Some CLI tools do not document their usage with Kustomize, one could assume it is because anyone with knowledge of Kustomize and `yaml` files would know how to use it.
+
+Here are some examples:
+
+https://github.com/jetstack/kustomize-cert-manager-demo
+
+https://github.com/istio/installer/tree/master/kustomize
+
+https://github.com/weaveworks/flagger/tree/master/kustomize
+
+https://github.com/jpeach/contour/tree/1c575c772e9fd747fba72ae41ab99bdae7a01864/kustomize (RFC)
+
+## Security Considerations
+
+N/A

design/csi-snapshots.md

@@ -0,0 +1,324 @@
+# CSI Snapshot Support
+
+The Container Storage Interface (CSI) [introduced an alpha snapshot API in Kubernetes v1.12][1].
+It will reach beta support in Kubernetes v1.17, scheduled for release in December 2019.
+This proposal documents an approach for integrating support for this snapshot API within Velero, augmenting its existing capabilities.
+
+## Goals
+
+- Enable Velero to backup and restore CSI-backed volumes using the Kubernetes CSI CustomResourceDefinition API
+
+## Non Goals
+
+- Replacing Velero's existing [VolumeSnapshotter][7] API
+- Replacing Velero's Restic support
+
+## Background
+
+Velero has had support for performing persistent volume snapshots since its inception.
+However, support has been limited to a handful of providers.
+The plugin API introduced in Velero v0.7 enabled the community to expand the number of supported providers.
+In the meantime, the Kubernetes sig-storage advanced the CSI spec to allow for a generic storage interface, opening up the possibility of moving storage code out of the core Kubernetes code base.
+The CSI working group has also developed a generic snapshotting API that any CSI driver developer may implement, giving users the ability to snapshot volumes from a standard interface.
+
+By supporting the CSI snapshot API, Velero can extend its support to any CSI driver, without requiring a Velero-specific plugin be written, easing the development burden on providers while also reaching more end users.
+
+## High-Level Design
+
+In order to support CSI's snapshot API, Velero must interact with the [`VolumeSnapshot`][2] and [`VolumeSnapshotContent`][3] CRDs.
+These act as requests to the CSI driver to perform a snapshot on the underlying provider's volume.
+This can largely be accomplished with Velero `BackupItemAction` and `RestoreItemAction` plugins that operate on these CRDs.
+
+Additionally, changes to the Velero server and client code are necessary to track `VolumeSnapshot`s that are associated with a given backup, similarly to how Velero tracks its own [`volume.Snapshot`][4] type.
+Tracking these is important for allowing users to see what is in their backup, and provides parity for the existing `volume.Snapshot` and [`PodVolumeBackup`][5] types.
+This is also done to retain the object store as Velero's source of truth, without having to query the Kubernetes API server for associated `VolumeSnapshot`s.
+
+`velero backup describe --details` will use the stored VolumeSnapshots to list CSI snapshots included in the backup to the user.
+
+## Detailed Design
+
+### Resource Plugins
+
+A set of [prototype][6] plugins was developed that informed this design.
+
+The plugins will be as follows:
+
+
+#### A `BackupItemAction` for `PersistentVolumeClaim`s, named `velero.io/csi-pvc` 
+
+This plugin will act directly on PVCs, since an implementation of Velero's VolumeSnapshotter does not have enough information about the StorageClass to properly create the `VolumeSnapshot` objects.
+
+The associated PV will be queried and checked for the presence of `PersistentVolume.Spec.PersistentVolumeSource.CSI`. (See the "Snapshot Mechanism Selection" section below).
+If this field is `nil`, then the plugin will return early without taking action.
+If the `Backup.Spec.SnapshotVolumes` value is `false`, the plugin will return early without taking action.
+
+Additionally, to prevent creating CSI snapshots for volumes backed up by restic, the plugin will query for all pods in the `PersistentVolumeClaim`'s namespace.
+It will then filter out the pods that have the PVC mounted, and inspect the `backup.velero.io/backup-volumes` annotation for the associated volume's name.
+If the name is found in the list, then the plugin will return early without taking further action.
+
+Create a `VolumeSnapshot.snapshot.storage.k8s.io` object from the PVC.
+Label the `VolumeSnapshot` object with the [`velero.io/backup-name`][10] label for ease of lookup later.
+Also set an ownerRef on the `VolumeSnapshot` so that cascading deletion of the Velero `Backup` will delete associated `VolumeSnapshots`.
+
+The CSI controllers will create a `VolumeSnapshotContent.snapshot.storage.k8s.io` object associated with the `VolumeSnapshot`.
+
+Associated `VolumeSnapshotContent` objects will be retrieved and updated with the [`velero.io/backup-name`][10] label for ease of lookup later.
+`velero.io/volume-snapshot-name` will be applied as a label to the PVC so that the `VolumeSnapshot` can be found easily for restore.
+
+`VolumeSnapshot`, `VolumeSnapshotContent`, and `VolumeSnapshotClass` objects would be returned as additional items to be backed up. GitHub issue [1566][18] represents this work.
+
+The `VolumeSnapshotContent.Spec.VolumeSnapshotSource.SnapshotHandle` field is the link to the underlying platform's on-disk snapshot, and must be preserved for restoration.
+
+The plugin will _not_ wait for the `VolumeSnapshot.Status.readyToUse` field to be `true` before returning.
+This field indicates that the snapshot is ready to use for restoration, and for different vendors can indicate that the snapshot has been made durable.
+However, the applications can proceed as soon as `VolumeSnapshot.Status.CreationTime` is set.
+This also maintains current Velero behavior, which allows applications to quiesce and resume quickly, with minimal interruption.
+
+Any sort of monitoring or waiting for durable snapshots, either Velero-native or CSI snapshots, are not covered by this proposal.
+
+```
+K8s object relationships inside of the backup tarball
++-----------------------+               +-----------------------+
+| PersistentVolumeClaim +-------------->+ PersistentVolume      |
++-----------+-----------+               +-----------+-----------+
+            ^                                       ^
+            |                                       |
+            |                                       |
+            |                                       |
++-----------+-----------+               +-----------+-----------+
+| VolumeSnapshot        +<------------->+ VolumeSnapshotContent |
++-----------------------+               +-----------------------+
+```
+
+#### A `RestoreItemAction` for `VolumeSnapshotContent` objects, named `velero.io/csi-vsc`
+
+On restore, `VolumeSnapshotContent` objects are cleaned so that they may be properly associated with IDs assigned by the target cluster.
+
+Only `VolumeSnapshotContent` objects with the `velero.io/backup-name` label will be processed, using the plugin's `AppliesTo` function.
+
+The metadata (excluding labels), `PersistentVolumeClaim.UUID`, and `VolumeSnapshotRef.UUID` fields will be cleared.
+The reference fields are cleared because the associated objects will get new UUIDs in the cluster.
+This also maps to the "import" case of [the snapshot API][1].
+
+This means the relationship between the `VolumeSnapshot` and `VolumeSnapshotContent` is
+one way until the CSI controllers rebind them.
+
+
+```
+K8s objects after the velero.io/csi-vsc plugin has run
++-----------------------+               +-----------------------+
+| PersistentVolumeClaim +-------------->+ PersistentVolume      |
++-----------------------+               +-----------------------+
+                                                     
+                                                     
++-----------------------+               +-----------------------+
+| VolumeSnapshot        +-------------->+ VolumeSnapshotContent |
++-----------------------+               +-----------------------+
+```
+
+#### A `RestoreItemAction` for `VolumeSnapshot` objects, named `velero.io/csi-vs`
+
+`VolumeSnapshot` objects must be prepared for importing into the target cluster by removing IDs and metadata associated with their origin cluster.
+
+Only `VolumeSnapshot` objects with the `velero.io/backup-name` label will be processed, using the plugin's `AppliesTo` function.
+
+Metadata (excluding labels) and `Source` (that is, the pointer to the `PersistentVolumeClaim`) fields on the object will be cleared.
+The `VolumeSnapshot.Spec.SnapshotContentName` is the link back to the `VolumeSnapshotContent` object, and thus the actual snapshot.
+The `Source` field indicates that a new CSI snapshot operation should be performed, which isn't relevant on restore.
+This follows the "import" case of [the snapshot API][1].
+
+The `Backup` associated with the `VolumeSnapshot` will be queried, and set as an ownerRef on the `VolumeSnapshot` so that deletion can cascade.
+
+```
++-----------------------+               +-----------------------+
+| PersistentVolumeClaim +-------------->+ PersistentVolume      |
++-----------------------+               +-----------------------+
+                                                     
+                                                     
++-----------------------+               +-----------------------+
+| VolumeSnapshot        +-------------->+ VolumeSnapshotContent |
++-----------------------+               +-----------------------+
+```
+
+#### A `RestoreItemAction` for `PersistentVolumeClaim`s named `velero.io/csi-pvc`
+
+On restore, `PersistentVolumeClaims` will need to be created from the snapshot, and thus will require editing before submission.
+
+Only `PersistentVolumeClaim` objects with the `velero.io/volume-snapshot-name` label will be processed, using the plugin's `AppliesTo` function.
+Metadata (excluding labels) will be cleared, and the `velero.io/volume-snapshot-name` label will be used to find the relevant `VolumeSnapshot`.
+A reference to the `VolumeSnapshot` will be added to the `PersistentVolumeClaim.DataSource` field.
+
+```
++-----------------------+                                        
+| PersistentVolumeClaim |                                        
++-----------------------+                                        
+                                                                 
++-----------------------+               +-----------------------+
+| VolumeSnapshot        +-------------->+ VolumeSnapshotContent |
++-----------------------+               +-----------------------+
+```
+
+#### VolumeSnapshotClasses
+
+No special logic is required to restore `VolumeSnapshotClass` objects.
+
+These plugins should be provided with Velero, as there will also be some changes to core Velero code to enable association of a `Backup` to the included `VolumeSnapshot`s.
+
+
+
+### Velero server changes
+
+Any non-plugin code changes must be behind a `EnableCSI` feature flag and the behavior will be opt-in until it's exited beta status.
+This will allow the development to continue on the feature while it's in pre-production state, while also reducing the need for long-lived feature branches.
+
+[`persistBackup`][8] will be extended to query for all `VolumeSnapshot`s associated with the backup, and persist the list to JSON.
+
+[`BackupStore.PutBackup`][9] will receive an additional argument, `volumeSnapshots io.Reader`, that contains the JSON representation of `VolumeSnapshots`.
+This will be written to a file named `csi-snapshots.json.gz`.
+
+[`defaultRestorePriorities`][11] should be rewritten to the following to accomodate proper association between the CSI objects and PVCs. `CustomResourceDefinition`s are moved up because they're necessary for creating the CSI CRDs. The CSI CRDs are created before `PersistentVolume`s and `PersistentVolumeClaim`s so that they may be used as data sources.
+GitHub issue [1565][17] represents this work.
+
+```go
+var defaultRestorePriorities = []string{
+    "namespaces",
+    "storageclasses",
+    "customresourcedefinitions",
+    "volumesnapshotclass.snapshot.storage.k8s.io",
+    "volumesnapshotcontents.snapshot.storage.k8s.io",
+    "volumesnapshots.snapshot.storage.k8s.io",
+    "persistentvolumes",
+    "persistentvolumeclaims",
+    "secrets",
+    "configmaps",
+    "serviceaccounts",
+    "limitranges",
+    "pods",
+    "replicaset",
+}
+```
+### Restic and CSI interaction
+
+Volumes found in a `Pod`'s `backup.velero.io/backup-volumes` list will use Velero's current Restic code path.
+This also means Velero will continue to offer Restic as an option for CSI volumes.
+
+The `velero.io/csi-pvc` BackupItemAction plugin will inspect pods in the namespace to ensure that it does not act on PVCs already being backed up by restic.
+
+This is preferred to modifying the PVC due to the fact that Velero's current backup process backs up PVCs and PVs mounted to pods at the same time as the pod.
+
+A drawback to this approach is that we're querying all pods in the namespace per PVC, which could be a large number.
+In the future, the plugin interface could be improved to have some sort of context argument, so that additional data such as our existing `resticSnapshotTracker` could be passed to plugins and reduce work.
+
+### Garbage collection and deletion
+
+To ensure that all created resources are deleted when a backup expires or is deleted, `VolumeSnapshot`s will have an `ownerRef` defined pointing to the Velero backup that created them.
+
+In order to fully delete these objects, each `VolumeSnapshotContent`s object will need to be edited to ensure the associated provider snapshot is deleted.
+This will be done by editing the object and setting `VolumeSnapshotContent.Spec.DeletionPolicy` to `Delete`, regardless of whether or not the default policy for the class is `Retain`.
+See the Deletion Policies section below.
+The edit will happen before making Kubernetes API deletion calls to ensure that the cascade works as expected.
+
+Deleting a Velero `Backup` or any associated CSI object via `kubectl` is unsupported; data will be lost or orphaned if this is done.
+
+### Other snapshots included in the backup
+
+Since `VolumeSnapshot` and `VolumeSnapshotContent` objects are contained within a Velero backup tarball, it is possible that all CRDs and on-disk provider snapshots have been deleted, yet the CRDs are still within other Velero backup tarballs.
+Thus, when a Velero backup that contains these CRDs is restored, the `VolumeSnapshot` and `VolumeSnapshotContent` objects are restored into the cluster, the CSI controllers will attempt to reconcile their state, and there are two possible states when the on-disk snapshot has been deleted:
+
+    1) If the driver _does not_ support the `ListSnapshots` gRPC method, then the CSI controllers have no way of knowing how to find it, and sets the `VolumeSnapshot.Status.readyToUse` field to `true`.
+    2) If the driver _does_ support the `ListSnapshots` gRPC method, then the CSI controllers will query the state of the on-disk snapshot, see it is missing, and set `VolumeSnapshot.Status.readyToUse` and `VolumeSnapshotContent.Status.readyToUse` fields to `false`.
+
+## Velero client changes
+
+To use CSI features, the Velero client must use the `EnableCSI` feature flag.
+
+[`DescribeBackupStatus`][13] will be extended to download the `csi-snapshots.json.gz` file for processing. GitHub Issue [1568][19] captures this work.
+
+A new `describeCSIVolumeSnapshots` function should be added to the [output][12] package that knows how to render the included `VolumeSnapshot` names referenced in the `csi-snapshots.json.gz` file.
+
+### Snapshot selection mechanism
+
+The most accurate, reliable way to detect if a PersistentVolume is a CSI volume is to check for a non-`nil` [`PersistentVolume.Spec.PersistentVolumeSource.CSI`][16] field.
+Using the [`volume.beta.kubernetes.io/storage-provisioner`][14] is not viable, since the usage is for any PVC that should be dynamically provisioned, and is _not_ limited to CSI implementations.
+It was [introduced with dynamic provisioning support][15] in 2016, predating CSI.
+
+In the `BackupItemAction` for PVCs, the associated PV will be queried and checked for the presence of `PersistentVolume.Spec.PersistentVolumeSource.CSI`.
+Volumes with any other `PersistentVolumeSource` set will use Velero's current VolumeSnapshotter plugin code path.
+
+### VolumeSnapshotLocations and VolumeSnapshotClasses
+
+Velero uses its own `VolumeSnapshotLocation` CRDs to specify configuration options for a given storage system.
+In Velero, this often includes topology information such as regions or availibility zones, as well as credential information.
+
+CSI volume snapshotting has a `VolumeSnapshotClass` CRD which also contains configuration options for a given storage system, but these options are not the same as those that Velero would use.
+Since CSI volume snapshotting is operating within the same storage system that manages the volumes already, it does not need the same topology or credential information that Velero does.
+
+As such, when used with CSI volumes, Velero's `VolumeSnapshotLocation` CRDs are not relevant, and could be omitted.
+
+This will create a separate path in our documentation for the time being, and should be called out explicitly.
+
+## Alternatives Considered
+
+* Implementing similar logic in a Velero VolumeSnapshotter plugin was considered.
+However, this is inappropriate given CSI's data model, which requires a PVC/PV's StorageClass.
+Given the arguments to the VolumeSnapshotter interface, the plugin would have to instantiate its own client and do queries against the Kubernetes API server to get the necessary information.
+
+This is unnecessary given the fact that the `BackupItemAction` and `RestoreItemAction` APIs can act directly on the appropriate objects.
+
+Additionally, the VolumeSnapshotter plugins and CSI volume snapshot drivers overlap - both produce a snapshot on backup and a PersistentVolume on restore.
+Thus, there's not a logical place to fit the creation of VolumeSnapshot creation in the VolumeSnapshotter interface.
+
+* Implement CSI logic directly in Velero core code.
+The plugins could be packaged separately, but that doesn't necessarily make sense with server and client changes being made to accomodate CSI snapshot lookup.
+
+* Implementing the CSI logic entirely in external plugins.
+As mentioned above, the necessary plugins for `PersistentVolumeClaim`, `VolumeSnapshot`, and `VolumeSnapshotContent` could be hosted out-out-of-tree from Velero.
+In fact, much of the logic for creating the CSI objects will be driven entirely inside of the plugin implementation.
+
+However, Velero currently has no way for plugins to communicate that some arbitrary data should be stored in or retrieved from object storage, such as list of all `VolumeSnapshot` objects associated with a given `Backup`.
+This is important, because to display snapshots included in a backup, whether as native snapshots or Restic backups, separate JSON-encoded lists are stored within the backup on object storage.
+Snapshots are not listed directly on the `Backup` to fit within the etcd size limitations.
+Additionally, there are no client-side Velero plugin mechanisms, which means that the `velero describe backup --details` command would have no way of displaying the objects to the user, even if they were stored.
+
+## Deletion Policies
+
+In order for underlying, provider-level snapshots to be retained similarly to Velero's current functionality, the `VolumeSnapshotContent.Spec.DeletionPolicy` field must be set to `Retain`.
+
+This is most easily accomplished by setting the `VolumeSnapshotClass.DeletionPolicy` field to `Retain`, which will be inherited by all `VolumeSnapshotContent` objects associated with the `VolumeSnapshotClass`.
+
+The current default for dynamically provisioned `VolumeSnapshotContent` objects is `Delete`, which will delete the provider-level snapshot when the `VolumeSnapshotContent` object representing it is deleted.
+Additionally, the `Delete` policy will cascade a deletion of a `VolumeSnapshot`, removing the associated `VolumeSnapshotContent` object.
+
+It is not currently possible to define a deletion policy on a `VolumeSnapshot` that gets passed to a `VolumeSnapshotContent` object on an individual basis.
+
+## Security Considerations
+
+This proposal does not significantly change Velero's security implications within a cluster.
+
+If a deployment is using solely CSI volumes, Velero will no longer need privileges to interact with volumes or snapshots, as these will be handled by the CSI driver.
+This reduces the provider permissions footprint of Velero.
+
+Velero must still be able to access cluster-scoped resources in order to back up `VolumeSnapshotContent` objects.
+Without these objects, the provider-level snapshots cannot be located in order to re-associate them with volumes in the event of a restore.
+
+
+
+[1]: https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/
+[2]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/pkg/apis/volumesnapshot/v1alpha1/types.go#L41
+[3]: https://github.com/kubernetes-csi/external-snapshotter/blob/master/pkg/apis/volumesnapshot/v1alpha1/types.go#L161
+[4]: https://github.com/heptio/velero/blob/master/pkg/volume/snapshot.go#L21
+[5]: https://github.com/heptio/velero/blob/master/pkg/apis/velero/v1/pod_volume_backup.go#L88
+[6]: https://github.com/heptio/velero-csi-plugin/
+[7]: https://github.com/heptio/velero/blob/master/pkg/plugin/velero/volume_snapshotter.go#L26
+[8]: https://github.com/heptio/velero/blob/master/pkg/controller/backup_controller.go#L560
+[9]: https://github.com/heptio/velero/blob/master/pkg/persistence/object_store.go#L46
+[10]: https://github.com/heptio/velero/blob/master/pkg/apis/velero/v1/labels_annotations.go#L21
+[11]: https://github.com/heptio/velero/blob/master/pkg/cmd/server/server.go#L471
+[12]: https://github.com/heptio/velero/blob/master/pkg/cmd/util/output/backup_describer.go
+[13]: https://github.com/heptio/velero/blob/master/pkg/cmd/util/output/backup_describer.go#L214
+[14]: https://github.com/kubernetes/kubernetes/blob/8ea9edbb0290e9de1e6d274e816a4002892cca6f/pkg/controller/volume/persistentvolume/util/util.go#L69
+[15]: https://github.com/kubernetes/kubernetes/pull/30285
+[16]: https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/core/types.go#L237
+[17]: https://github.com/heptio/velero/issues/1565
+[18]: https://github.com/heptio/velero/issues/1566
+[19]: https://github.com/heptio/velero/issues/1568

design/custom-ca-support.md

@@ -0,0 +1,132 @@
+# Custom CA Bundle Support for S3 Object Storage
+
+It is desired that Velero performs SSL verification on the Object Storage
+endpoint (BackupStorageLocation), but it is not guaranteed that the Velero
+container has the endpoints' CA bundle in it's system store. Velero needs to
+support the ability for a user to specify custom CA bundles at installation
+time and Velero needs to support a mechanism in the BackupStorageLocation
+Custom Resource to allow a user to specify a custom CA bundle. This mechanism
+needs to also allow Restic to access and use this custom CA bundle.
+
+## Goals
+
+- Enable Velero to be configured with a custom CA bundle at installation
+- Enable Velero support for custom CA bundles with S3 API BackupStorageLocations
+- Enable Restic to use the custom CA bundles whether it is configured at installation time or on the BackupStorageLocation
+- Enable Velero client to take a CA bundle as an argument
+
+## Non Goals
+
+- Support non-S3 providers
+
+## Background
+
+Currently, in order for Velero to perform SSL verification of the object
+storage endpoint the user must manually set the `AWS_CA_BUNDLE` environment
+variable on the Velero deployment. If the user is using Restic, the user has to
+either:
+1. Add the certs to the Restic container's system store
+1. Modify Velero to pass in the certs as a CLI parameter to Restic - requiring
+   a custom Velero deployment
+
+## High-Level Design
+
+There are really 2 methods of using Velero with custom certificates:
+1. Including a custom certificate at Velero installation
+1. Specifying a custom certificate to be used with a `BackupStorageLocation`
+
+### Specifying a custom cert at installation
+
+On the Velero deployment at install time, we can set the AWS environment variable
+`AWS_CA_BUNDLE` which will allow Velero to communicate over https with the
+proper certs when communicating with the S3 bucket. This means we will add the
+ability to specify a custom CA bundle at installation time. For more
+information, see "Install Command Changes".
+
+On the Restic daemonset, we will want to also mount this secret at a pre-defined
+location. In the `restic` pkg, the command to invoke restic will need to be
+updated to pass the path to the cert file that is mounted if it is specified in
+the config.
+
+This is good, but doesn't allow us to specify different certs when
+`BackupStorageLocation` resources are created.
+
+### Specifying a custom cert on BSL
+
+In order to support custom certs for object storage, Velero will add an
+additional field to the `BackupStorageLocation`'s provider `Config` resource to
+provide a secretRef which will contain the coordinates to a secret containing
+the relevant cert file for object storage. 
+
+In order for Restic to be able to consume and use this cert, Velero will need
+the ability to write the CA bundle somewhere in memory for the Restic pod to
+consume it.
+
+To accomplish this, we can look at the code for managing restic repository
+credentials. The way this works today is that the key is stored in a secret in
+the Velero namespace, and each time Velero executes a restic command, the
+contents of the secret are read and written out to a temp file. The path to
+this file is then passed to restic and removed afterwards. pass the path of the
+temp file to restic, and then remove the temp file afterwards. See ref #1 and #2.
+
+This same approach can be taken for CA bundles. The bundle can be stored in a
+secret which is referenced on the BSL and written to a temp file prior to
+invoking Restic.
+
+[1](https://github.com/vmware-tanzu/velero/blob/master/pkg/restic/repository_manager.go#L238-L245)
+[2](https://github.com/vmware-tanzu/velero/blob/master/pkg/restic/common.go#L168-L203)
+
+## Detailed Design
+
+The `AWS_CA_BUNDLE` environment variable works for the Velero deployment
+because this environment variable is passed into the AWS SDK which is used in
+the [plugin][1] to build up the config object. This means that a user can
+simply define the CA bundle in the deployment as an env var. This can be
+utilized for the installation of Velero with a custom cert by simply setting
+this env var to the contents of the CA bundle, or the env var can be mapped to
+a secret which is controlled at installation time. I recommend using a secret
+as it makes the Restic integration easier as well.
+
+At installation time, if a user has specified a custom cert then the Restic
+daemonset should be updated to include the secret mounted at a predefined path.
+We could optionally use the system store for all custom certs added at
+installation time. Restic supports using the custom certs [in addition][3] to
+the root certs.
+
+In the case of the BSL being created with a secret reference, then at runtime
+the secret will need to be consumed. This secret will be read and applied to
+the AWS `session` object. The `getSession()` function will need to be updated
+to take in the custom CA bundle so it can be passed [here][4].
+
+The Restic controller will need to be updated to write the contents of the CA
+bundle secret out to a temporary file inside of the restic pod.The restic
+[command invocation][2] will need to be updated to include the path to the file
+as an argument to the restic server using `--cacert`. For the path when a user
+defines a custom cert on the BSL, Velero will be responsible for updating the
+daemonset to include the secret mounted as a volume at a predefined path.
+
+Where we mount the secret is a fine detail, but I recommend mounting the certs
+to `/certs` to keep it in line with the other volume mount paths being used.
+
+### Install command changes
+
+The installation flags should be updated to include the ability to pass in a
+cert file. Then the install command would do the heavy lifting of creating a
+secret and updating the proper fields on the deployment and daemonset to mount
+the secret at a well defined path.
+
+### Velero client changes
+
+Since the Velero client is responsible for gathering logs and information about
+the Object Storage, this implementation should include a new flag `--cacert`
+which can be used when communicating with the Object Storage. Additionally, the
+user should be able to set this in their client configuration. The command
+would look like:
+```
+$ velero client config set cacert PATH
+```
+
+[1]: https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/master/velero-plugin-for-aws/object_store.go#L135
+[2]: https://github.com/vmware-tanzu/velero/blob/master/pkg/restic/command.go#L47
+[3]: https://github.com/restic/restic/blob/master/internal/backend/http_transport.go#L81
+[4]: https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/master/velero-plugin-for-aws/object_store.go#L154

design/generating-velero-crds-with-structural-schema.md

@@ -0,0 +1,164 @@
+# Generating Velero CRDs with structural schema support
+
+As the apiextensions.k8s.io API moves to GA, structural schema in Custom Resource Definitions (CRDs) will become required.
+
+This document proposes updating the CRD generation logic as part of `velero install` to include structural schema for each Velero CRD.
+
+## Goals
+
+- Enable structural schema and validation for Velero Custom Resources.
+
+## Non Goals
+
+- Update Velero codebase to use Kubebuilder for controller/code generation.
+- Solve for keeping CRDs in the Velero Helm chart up-to-date.
+
+## Background
+
+Currently, Velero CRDs created by the `velero install` command do not contain any structural schema.
+The CRD is simply [generated at runtime](https://github.com/heptio/velero/blob/8b0cf3855c2b8aa631cf22e63da0955f7b1d06a8/pkg/install/crd.go#L39) using the name and plurals from the [`velerov1api.CustomResources()`](https://github.com/heptio/velero/blob/8b0cf3855c2b8aa631cf22e63da0955f7b1d06a8/pkg/apis/velero/v1/register.go#L60) info.
+
+Updating the info returned by that method would be one way to add support for structural schema when generating the CRDs, but this would require manually describing the schema and would duplicate information from the API structs (e.g. comments describing a field).
+
+Instead, the [controller-tools](https://github.com/kubernetes-sigs/controller-tools) project from Kubebuilder provides tooling for generating CRD manifests (YAML) from the Velero API types.
+This document proposes adding _controller-tools_ to the project to automatically generate CRDs, and use these generated CRDs as part of `velero install`.
+
+## High-Level Design
+
+_controller-tools_ works by reading the Go files that contain the API type definitions.
+It uses a combination of the struct fields, types, tags and comments to build the OpenAPIv3 schema for the CRDs. The tooling makes some assumptions based on conventions followed in upstream Kubernetes and the ecosystem, which involves some changes to the Velero API type definitions, especially around optional fields.
+
+In order for _controller-tools_ to read the Go files containing Velero API type defintiions, the CRDs need to be generated at build time, as these files are not available at runtime (i.e. the Go files are not accessible by the compiled binary).
+These generated CRD manifests (YAML) will then need to be available to the `pkg/install` package for it to include when installing Velero resources.
+
+## Detailed Design
+
+### Changes to Velero API type definitions
+
+API type definitions need to be updated to correctly identify optional and required fields for each API type.
+Upstream Kubernetes defines all optional fields using the `omitempty` tag as well as a `// +optional` annotation above the field (e.g. see [PodSpec definition](https://github.com/kubernetes/api/blob/master/core/v1/types.go#L2835-L2838)).
+_controller-tools_ will mark a field as optional if it sees either the tag or the annotation, but to keep consistent with upstream, optional fields will be updated to use both indicators (as [suggested](https://github.com/kubernetes-sigs/kubebuilder/issues/479) by the Kubebuilder project).
+Additionally, upstream Kubernetes defines the metav1.ObjectMeta, metav1.ListMeta, Spec and Status as [optional on all types](https://github.com/kubernetes/api/blob/master/core/v1/types.go#L3517-L3531).
+Some Velero API types set the `omitempty` tag on Status, but not on other fields - these will all need to be updated to be made optional.
+
+Below is a list of the Velero API type fields and what changes (if any) will be made.
+Note that this only includes fields used in the spec, all status fields will become optional.
+
+| Type                            | Field                   | Changes                                                     |
+|---------------------------------|-------------------------|-------------------------------------------------------------|
+| BackupSpec                      | IncludedNamespaces      | make optional                                               |
+|                                 | ExcludedNamespaces      | make optional                                               |
+|                                 | IncludedResources       | make optional                                               |
+|                                 | ExcludedResources       | make optional                                               |
+|                                 | LabelSelector           | make optional                                               |
+|                                 | SnapshotVolumes         | make optional                                               |
+|                                 | TTL                     | make optional                                               |
+|                                 | IncludeClusterResources | make optional                                               |
+|                                 | Hooks                   | make optional                                               |
+|                                 | StorageLocation         | make optional                                               |
+|                                 | VolumeSnapshotLocations | make optional                                               |
+| BackupHooks                     | Resources               | make optional                                               |
+| BackupResourceHookSpec          | Name                    | none (required)                                             |
+|                                 | IncludedNamespaces      | make optional                                               |
+|                                 | ExcludedNamespaces      | make optional                                               |
+|                                 | IncludedResources       | make optional                                               |
+|                                 | ExcludedResources       | make optional                                               |
+|                                 | LabelSelector           | make optional                                               |
+|                                 | PreHooks                | make optional                                               |
+|                                 | PostHooks               | make optional                                               |
+| BackupResourceHook              | Exec                    | none (required)                                             |
+| ExecHook                        | Container               | make optional                                               |
+|                                 | Command                 | required, validation: MinItems=1                            |
+|                                 | OnError                 | make optional                                               |
+|                                 | Timeout                 | make optional                                               |
+| HookErrorMode                   |                         | validation: Enum                                            |
+| BackupStorageLocationSpec       | Provider                | none (required)                                             |
+|                                 | Config                  | make optional                                               |
+|                                 | StorageType             | none (required)                                             |
+|                                 | AccessMode              | make optional                                               |
+| StorageType                     | ObjectStorage           | make required                                               |
+| ObjectStorageLocation           | Bucket                  | none (required)                                             |
+|                                 | Prefix                  | make optional                                               |
+| BackupStorageLocationAccessMode |                         | validation: Enum                                            |
+| DeleteBackupRequestSpec         | BackupName              | none (required)                                             |
+| DownloadRequestSpec             | Target                  | none (required)                                             |
+| DownloadTarget                  | Kind                    | none (required)                                             |
+|                                 | Name                    | none (required)                                             |
+| DownloadTargetKind              |                         | validation: Enum                                            |
+| PodVolumeBackupSpec             | Node                    | none (required)                                             |
+|                                 | Pod                     | none (required)                                             |
+|                                 | Volume                  | none (required)                                             |
+|                                 | BackupStorageLocation   | none (required)                                             |
+|                                 | RepoIdentifier          | none (required)                                             |
+|                                 | Tags                    | make optional                                               |
+| PodVolumeRestoreSpec            | Pod                     | none (required)                                             |
+|                                 | Volume                  | none (required)                                             |
+|                                 | BackupStorageLocation   | none (required)                                             |
+|                                 | RepoIdentifier          | none (required)                                             |
+|                                 | SnapshotID              | none (required)                                             |
+| ResticRepositorySpec            | VolumeNamespace         | none (required)                                             |
+|                                 | BackupStorageLocation   | none (required)                                             |
+|                                 | ResticIdentifier        | none (required)                                             |
+|                                 | MaintenanceFrequency    | none (required)                                             |
+| RestoreSpec                     | BackupName              | none (required) - should be set to "" if using ScheduleName |
+|                                 | ScheduleName            | make optional                                               |
+|                                 | IncludedNamespaces      | make optional                                               |
+|                                 | ExcludedNamespaces      | make optional                                               |
+|                                 | IncludedResources       | make optional                                               |
+|                                 | ExcludedResources       | make optional                                               |
+|                                 | NamespaceMapping        | make optional                                               |
+|                                 | LabelSelector           | make optional                                               |
+|                                 | RestorePVs              | make optional                                               |
+|                                 | IncludeClusterResources | make optional                                               |
+| ScheduleSpec                    | Template                | none (required)                                             |
+|                                 | Schedule                | none (required)                                             |
+| VolumeSnapshotLocationSpec      | Provider                | none (required)                                             |
+|                                 | Config                  | make optional                                               |
+
+### Build-time generation of CRD manifests
+
+The build image will be updated as follows to include the _controller-tool_ tooling:
+
+
+```diff
+diff --git a/hack/build-image/Dockerfile b/hack/build-image/Dockerfile
+index b69a8c8a..07eac9c6 100644
+--- a/hack/build-image/Dockerfile
++++ b/hack/build-image/Dockerfile
+@@ -21,6 +21,8 @@ RUN mkdir -p /go/src/k8s.io && \
+     git clone -b kubernetes-1.15.3 https://github.com/kubernetes/apimachinery && \
+     # vendor code-generator go modules to be compatible with pre-1.15
+     cd /go/src/k8s.io/code-generator && GO111MODULE=on go mod vendor && \
++    go get -d sigs.k8s.io/controller-tools/cmd/controller-gen && \
++    cd /go/src/sigs.k8s.io/controller-tools && GO111MODULE=on go mod vendor && \
+     go get golang.org/x/tools/cmd/goimports && \
+     cd /go/src/golang.org/x/tools && \
+     git checkout 40a48ad93fbe707101afb2099b738471f70594ec && \
+```
+
+To tie in the CRD manifest generation with existing scripts/workflows, the `hack/update-generated-crd-code.sh` script will be updated to use _controller-tools_ to generate CRDs manifests after it generates the client code.
+
+The generated CRD manifests will be placed in the `pkg/generated/crds/manifests` folder.
+Similarly to client code generation, these manifests will be checked-in to the git repo.
+Checking in these manifests allows including documentation and schema changes to API types as part of code review.
+
+### Updating `velero install` to include generated CRD manifests
+
+As described above, CRD generation using _controller-tools_ will happen at build time due to need to inspect Go files.
+To enable the `velero install` to access the generated CRD manifests at runtime, the `pkg/generated/crds/manifests` folder will be embedded as binary data in the Velero binary (e.g. using a tool like [vfsgen](https://github.com/shurcooL/vfsgen) - see [POC branch](https://github.com/prydonius/velero/commit/4aa7413f97ce9b23e071b6054f600dd0c283351e)).
+
+`velero install` will then unmarshal the binary data as `unstructured.Unstructured` types and append them to the [resources list](https://github.com/heptio/velero/blob/8b0cf3855c2b8aa631cf22e63da0955f7b1d06a8/pkg/install/resources.go#L217) in place of the existing CRD generation.
+
+## Alternatives Considered
+
+Instead of generating and bundling CRD manifests, it could be possible to instead embed the `pkg/apis` package in the Velero binary.
+With this, _controller-tools_ could be run at runtime during `velero install` to generate the CRD manifests.
+However, this would require including _controller-tools_ as a dependency in the project, which might not be desirable as it is a developer tool.
+
+Another option, to avoid embedding static files in the binary, would be to generate the CRD manifest as one YAML file in CI and upload it as a release artifact (e.g. using GitHub releases).
+`velero install` could then download this file for the current version and use it on install.
+The downside here is that `velero install` becomes dependent on the GitHub network, and we lose visibility on changes to the CRD manifests in the Git history.
+
+## Security Considerations
+
+n/a

design/move-gh-org.md

@@ -0,0 +1,90 @@
+# Plan for moving the Velero GitHub repo into the VMware GitHub organization
+
+Currently, the Velero repository sits under the Heptio GitHub organization. With the acquisition of Heptio by VMware, it is due time that this repo moves to one of the VMware GitHub organizations. This document outlines a plan to move this repo to the VMware Tanzu (https://github.com/vmware-tanzu) organization.
+
+## Goals
+
+- List all steps necessary to have this repo fully functional under the new org
+
+## Non Goals
+
+- Highlight any step necessary around setting up the new organization and its members
+
+## Action items
+
+### Todo list
+
+#### Pre move
+
+- [ ] PR: Blog post communicating the move. https://github.com/heptio/velero/issues/1841. Who: TBD.
+- [ ] PR: Find/replace in all Go, script, yaml, documentation, and website files: `github.com/heptio/velero -> github.com/vmware-tanzu/velero`. Who: a Velero developer; TBD
+- [ ] PR: Update website with the correct GH links. Who: a Velero developer; TBD
+- [ ] PR: Change deployment and grpc-push scripts with the new location path. Who: a Velero developer; TBD
+- [ ] Delete branches not to be carried over (https://github.com/heptio/velero/branches/all). Who: Any of the current repo owners; TBD
+
+#### Move
+
+- [ ] Use GH UI to transfer the repository to the VMW org; must be accepted within a day. Who: new org owner; TBD
+- [ ] Make owners of this repo owners of repo in the new org. Who: new org owner; TBD
+- [ ] Update Travis CI. Who: Any of the new repo owners; TBD
+- [ ] Add DCO for signoff check (https://probot.github.io/apps/dco/). Who: Any of the new repo owners; TBD
+
+
+#### Post move
+
+- [ ] Each individual developer should point their origin to the new location: `git remote set-url origin git@github.com:vmware-tanzu/velero.git`
+- [ ] Transfer ZenHub. Who: Any of the new repo owners; TBD
+- [ ] Update Netlify deploy settings. Any of the new repo owners; TBD
+- [ ] GH app: Netlify integration. Who: Any of the new repo owners; TBD
+- [ ] GH app: Slack integration. Who: Any of the new repo owners; TBD
+- [ ] Add webhook: travis CI. Who: Any of the new repo owners; TBD
+- [ ] Add webhook: zenhub. Who: Any of the new repo owners; TBD
+- [ ] Move all 3 native provider plugins into their own individual repo. https://github.com/heptio/velero/issues/1537. Who: @carlisia.
+- [ ] Merge PRs from the "pre move" section
+- [ ] Create a team for the Velero core members (https://github.com/orgs/vmware-tanzu/teams/). Who: Any of the new repo owners; TBD
+
+### Notes/How-Tos
+
+#### Transfering the GH repository
+
+All action items needed for the repo transfer are listed in the Todo list above. For details about what gets moved and other info, this is the GH documentation: https://help.github.com/en/articles/transferring-a-repository
+
+[Pending] We will find out this week who will be the organization owner(s) who will accept this transfer in the new GH org. This organization owner will make all current owners in this repo owners in the new org Velero repo.
+
+#### Updating Travis CI
+
+Someone with owner permission on the new repository needs to go to their Travis CI account and authorize Travis CI on the repo. Here are instructions: https://docs.travis-ci.com/user/tutorial/.
+
+After this, webhook notifications can be added following these instructions: https://docs.travis-ci.com/user/notifications/#configuring-webhook-notifications.
+
+#### Transfering ZenHub
+
+Pre-requisite: A new Zenhub account must exist for a vmware or vmware-tanzu organization.
+
+This page contains a pre-migration checklist for ensuring a repo migration goes well with Zenhub: https://help.zenhub.com/support/solutions/articles/43000010366-moving-a-repo-cross-organization-or-to-a-new-organization. After this, webhooks can be added by following these instructions: https://github.com/ZenHubIO/API#webhooks.
+
+#### Updating Netlify
+
+The settings for Netflify should remain the same, except that it now needs to be installed in the new repo. The instructions on how to install Netlify on the new repo are here: https://www.netlify.com/docs/github-permissions/.
+
+#### Communication strategy
+
+[Pending] We will find out this week how this move will be communicated to the community. In particular, the Velero repository move might be tied to the move of our provider plugins into their own repos, also in the new org: https://github.com/heptio/velero/issues/1814.
+
+#### TBD
+
+Many items on the todo list must be done by a repository member with owner permission. This doesn't all need to be done by the same person obviously, but we should specify if @skriss wants to split these tasks with any other owner(s).
+
+#### Other notes
+
+Might want to exclude updating documentation prior to v1.0.0.
+GH documentation does not specify if branches on the server are also moved.
+All links to the original repository location are automatically redirected to the new location.
+
+## Alternatives Considered
+
+Alternatives such as moving Velero to its own organization, or even not moving at all, were considered. Collectively, however, the open source leadership decided it would be best to move it so it lives alongside other VMware supported cloud native related repositories.
+
+## Security Considerations
+
+- Ensure that only the Velero core team has maintainer/owner privileges.

design/move-plugin-repos.md

@@ -0,0 +1,129 @@
+# Plan to extract the provider plugins out of (the Velero) tree
+
+Currently, the Velero project contains in-tree plugins for three cloud providers: AWS, Azure, and GCP. The Velero team has decided to extract each of those plugins into their own separate repository. This document details the steps necessary to create the new repositories, as well as a general design for what each plugin project will look like.
+
+## Goals
+
+- Have 3 new repositories for each cloud provider plugin currently supported by the Velero team: AWS, Azure, and GCP
+- Have the currently in-tree cloud provider plugins behave like any other plugin external to Velero
+
+## Non Goals
+
+- Extend the Velero plugin framework capability in any way
+- Create GH repositories for any plugin other then the currently 3 in-tree plugins
+- Extract out any plugin that is not a cloud provider plugin (ex: item action related plugins)
+
+## Background
+
+With more and more providers wanting to support Velero, it gets more difficult to justify excluding those from being in-tree just as with the three original ones. At the same time, if we were to include any more plugins in-tree, it would ultimately become the responsibility of the Velero team to maintain an increasing number of plugins. This move aims to equalize the field so all plugins are treated equally. We also hope that, with time, developers interested in getting involved in the upkeep of those plugins will become active enough to be promoted to maintainers. Lastly, having the plugins live in their own individual repositories allows for iteration on them separately from the core codebase.
+
+## Action items
+
+### Todo list
+
+#### Repository creation
+
+- [ ] Use GH UI to create each repository in the new VMW org. Who: new org owner; TBD
+- [ ] Make owners of the Velero repo owners of each repo in the new org. Who: new org owner; TBD
+- [ ] Add Travis CI. Who: Any of the new repo owners; TBD
+- [ ] Add webhook: travis CI. Who: Any of the new repo owners; TBD
+- [ ] Add DCO for signoff check (https://probot.github.io/apps/dco/). Who: Any of the new repo owners; TBD
+
+#### Plugin changes
+
+- [ ] Modify Velero so it can install any of the provider plugins. https://github.com/heptio/velero/issues/1740 - Who: @nrb
+- [ ] Extract each provider plugin into their own repo. https://github.com/heptio/velero/issues/1537
+- [ ] Create deployment and gcr-push scripts with the new location path. Who: @carlisia
+- [ ] Add documentation for how to use the plugin. Who: @carlisia
+- [ ] Update Helm chart to install Velero using any of the provider plugins. https://github.com/heptio/velero/issues/1819
+- [ ] Upgrade script. https://github.com/heptio/velero/issues/1889.
+
+### Notes/How-Tos
+
+#### Creating the GH repository
+
+[Pending] The organization owner will make all current owners in the Velero repo also owners in each of the new org plugin repos.
+
+#### Setting up Travis CI
+
+Someone with owner permission on the new repository needs to go to their Travis CI account and authorize Travis CI on the repo. Here are instructions: https://docs.travis-ci.com/user/tutorial/.
+
+After this, any webhook notifications can be added following these instructions: https://docs.travis-ci.com/user/notifications/#configuring-webhook-notifications.
+
+## High-Level Design
+
+Each provider plugin will be an independent project, using the Velero library to implement their specific functionalities.
+
+The way Velero is installed will be changed to accommodate installing these plugins at deploy time, namely the Velero `install` command, as well as the Helm chart.
+
+Each plugin repository will need to have their respective images built and pushed to the same registry as the Velero images.
+
+## Detailed Design
+
+### Projects
+
+Each provider plugin will be an independent GH repository, named: `velero-plugin-aws`, `velero-plugin-azure`, and `velero-plugin-gcp`.
+
+Build of the project will be done the same way as with Velero, using Travis.
+
+Images for all the plugins will be pushed to the same repository as the Velero image, also using Travis.
+
+Releases of each of these plugins will happen in sync with releases of Velero. This will consist of having a tag in the repo and a tagged image build with the same release version as Velero so it makes it easy to identify what versions are compatible, starting at v1.2.
+
+Documentation for how to install and use the plugins will be augmented in the existing Plugins section of the Velero documentation.
+
+Documentation for how to use each plugin will reside in their respective repos. The navigation on the Velero documentation will be modified for easy discovery of the docs/images for these plugins.
+
+#### Version compatibility
+
+We will keep the major and minor release points in sync, but the plugins can have multiple minor dot something releases as long as it remains compatible with the corresponding major/minor release of Velero. Ex:
+
+| Velero  | Plugin  | Compatible?   |
+|---|---|---|
+|  v1.2 |  v1.2 | ✅  |
+|  v1.2 |  v1.2.3 | ✅  |
+|  v1.2 |  v1.3 | 🚫  |
+|  v1.3 |  v1.2 | 🚫  |
+|  v1.3 |  v1.3.3 | ✅  |
+
+### Installation
+
+As per https://github.com/heptio/velero/issues/1740, we will add a `plugins` flag to the Velero install command which will accept an array of URLs pointing to +1 images of plugins to be installed. The `velero plugin add` command should continue working as is, in specific, it should also allow the installation of any of the new 3 provider plugins. @nrb will provide specifics about how this change will be tackled, as well as what will be documented. Part of the work of adding the `plugins` flag will be removing the logic that adds `velero.io` name spacing to plugins that are added without it.
+
+The Helm chart that allows the installation of Velero will be modified to accept the array of plugin images with an added `plugins` configuration item.
+
+### Design code changes and considerations
+
+The naming convention to use for name spacing each plugin will be `velero.io`, since they are currently maintained by the Velero team.
+
+Install dep
+
+Question: are there any places outside the plugins where we depend on the cloud-provider SDKs? can we eliminate those dependencies too? x
+
+- the `restic` package uses the `aws`. SDK to get the bucket region for the AWS object store (https://github.com/carlisia/velero/blob/32d46871ccbc6b03e415d1e3d4ad9ae2268b977b/pkg/restic/config.go#L41)
+- could not find usage of the cloud provider SDKs anywhere else.
+
+Plugins such as the pod -> pvc -> pv backupitemaction ones make sense to stay in the core repo as they provide some important logic that just happens to be implemented in a plugin.
+
+### Upgrade
+
+The documentation for how to fresh install the out-of-tree plugin with Velero v1.2 will be specified together with the documentation for the install changes on issue https://github.com/heptio/velero/issues/1740.
+
+For upgrades, we will provide a script that will:
+
+- change the tag on the Velero deployment yaml for both the main image and any of th three plugins installed.
+- rename existing aws, azure or gcp plugin names to have the `velero.io/` namespace preceding the name (ex: `velero.io/aws).
+
+Alternatively, we could add CLI `velero upgrade` command that would make these changes. Ex: `velero upgrade 1.3` would upgrade from `v1.2` to `v1.3`.
+
+For upgrading:
+
+- Edit the provider field in the backupstoragelocations and volumesnapshotlocations CRDs to include the new namespace.
+
+## Alternatives Considered
+
+We considered having the plugins all live in the same GH repository. The downside of that approach is ending up with a binary and image bigger than necessary, since they would contain the SDKs of all three providers.
+
+## Security Considerations
+
+- Ensure that only the Velero core team has maintainer/owner privileges.

design/plugin-backup-and-restore-progress-design.md

@@ -0,0 +1,448 @@
+# Progress reporting for backups and restores handled by volume snapshotters
+
+Users face difficulty in knowing the progress of backup/restore operations of volume snapshotters. This is very similar to the issues faced by users to know progress for restic backup/restore, like, estimation of operation, operation in-progress/hung etc.
+
+Each plugin might be providing a way to know the progress, but, it need not uniform across the plugins.
+
+Even though plugins provide the way to know the progress of backup operation, this information won't be available to user during restore time on the destination cluster.
+
+So, apart from the issues like progress, status of operation, volume snapshotters have unique problems like
+- not being uniform across plugins
+- not knowing the backup information during restore operation
+- need to be optional as few plugins may not have a way to provide the progress information
+
+This document proposes an approach for plugins to follow to provide backup/restore progress, which can be used by users to know the progress.
+
+## Goals
+
+- Provide uniform way of visibility into backup/restore operations performed by volume snapshotters
+
+## Non Goals
+- Plugin implementation for this approach
+
+## Background
+
+(Omitted, see introduction)
+
+## High-Level Design
+
+### Progress of backup operation handled by volume snapshotter
+
+Progress will be updated by volume snapshotter in VolumePluginBackup CR which is specific to that backup operation.
+
+### Progress of restore operation handled by volume snapshotter
+
+Progress will be updated by volume snapshotter in VolumePluginRestore CR which is specific to that restore operation.
+
+## Detailed Design
+
+### Approach 1
+
+Existing `Snapshot` Go struct from `volume` package have most of the details related to backup operation performed by volumesnapshotters.
+This struct also gets backed up to backup location. But, this struct doesn't get synced on other clusters at regular intervals.
+It is currently synced only during restore operation, and velero CLI shows few of its contents.
+
+At a high level, in this approach, this struct will be converted to a CR by adding new fields (related to Progress tracking) to it, and gets rid of `volume.Snapshot` struct.
+
+Instead of backing up of Go struct, proposal is: to backup CRs to backup location, and sync them into other cluster by backupSyncController running in that cluster.
+
+#### VolumePluginBackup CR
+
+There is one addition to volume.SnapshotSpec, i.e., ProviderName to convert it to CR's spec. Below is the updated VolumePluginBackup CR's Spec:
+
+```
+type VolumePluginBackupSpec struct {
+	// BackupName is the name of the Velero backup this snapshot
+	// is associated with.
+	BackupName string `json:"backupName"`
+
+	// BackupUID is the UID of the Velero backup this snapshot
+	// is associated with.
+	BackupUID string `json:"backupUID"`
+
+	// Location is the name of the VolumeSnapshotLocation where this snapshot is stored.
+	Location string `json:"location"`
+
+	// PersistentVolumeName is the Kubernetes name for the volume.
+	PersistentVolumeName string `json:"persistentVolumeName"`
+
+	// ProviderVolumeID is the provider's ID for the volume.
+	ProviderVolumeID string `json:"providerVolumeID"`
+
+	// Provider is the Provider field given in VolumeSnapshotLocation
+	Provider string `json:"provider"`
+
+	// VolumeType is the type of the disk/volume in the cloud provider
+	// API.
+	VolumeType string `json:"volumeType"`
+
+	// VolumeAZ is the where the volume is provisioned
+	// in the cloud provider.
+	VolumeAZ string `json:"volumeAZ,omitempty"`
+
+	// VolumeIOPS is the optional value of provisioned IOPS for the
+	// disk/volume in the cloud provider API.
+	VolumeIOPS *int64 `json:"volumeIOPS,omitempty"`
+}
+```
+
+Few fields (except first two) are added to volume.SnapshotStatus to convert it to CR's status. Below is the updated VolumePluginBackup CR's status:
+```
+type VolumePluginBackupStatus struct {
+	// ProviderSnapshotID is the ID of the snapshot taken in the cloud
+	// provider API of this volume.
+	ProviderSnapshotID string `json:"providerSnapshotID,omitempty"`
+
+	// Phase is the current state of the VolumeSnapshot.
+	Phase SnapshotPhase `json:"phase,omitempty"`
+
+	// PluginSpecific are a map of key-value pairs that plugin want to provide
+	// to user to identify plugin properties related to this backup
+	// +optional
+	PluginSpecific map[string]string `json:"pluginSpecific,omitempty"`
+
+	// Message is a message about the volume plugin's backup's status.
+	// +optional
+	Message string `json:"message,omitempty"`
+
+	// StartTimestamp records the time a backup was started.
+	// Separate from CreationTimestamp, since that value changes
+	// on restores.
+	// The server's time is used for StartTimestamps
+	// +optional
+	// +nullable
+	StartTimestamp *metav1.Time `json:"startTimestamp,omitempty"`
+
+	// CompletionTimestamp records the time a backup was completed.
+	// Completion time is recorded even on failed backups.
+	// Completion time is recorded before uploading the backup object.
+	// The server's time is used for CompletionTimestamps
+	// +optional
+	// +nullable
+	CompletionTimestamp *metav1.Time `json:"completionTimestamp,omitempty"`
+
+	// Progress holds the total number of bytes of the volume and the current
+	// number of backed up bytes. This can be used to display progress information
+	// about the backup operation.
+	// +optional
+	Progress VolumeOperationProgress `json:"progress,omitempty"`
+}
+
+type VolumeOperationProgress struct {
+	TotalBytes int64
+	BytesDone int64
+}
+
+type VolumePluginBackup struct {
+	metav1.TypeMeta `json:",inline"`
+
+	// +optional
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	// +optional
+	Spec VolumePluginBackupSpec `json:"spec,omitempty"`
+
+	// +optional
+	Status VolumePluginBackupStatus `json:"status,omitempty"`
+}
+```
+
+For every backup operation of volume, Velero creates VolumePluginBackup CR before calling volumesnapshotter's CreateSnapshot API.
+
+In order to know the CR created for the particular backup of a volume, Velero adds following labels to CR:
+- `velero.io/backup-name` with value as Backup Name, and,
+- `velero.io/pv-name` with value as volume that is undergoing backup
+
+Backup name being unique won't cause issues like duplicates in identifying the CR.
+Labels will be set with the value returned from `GetValidName` function. (https://github.com/vmware-tanzu/velero/blob/master/pkg/label/label.go#L35).
+
+If Plugin supports showing progress of the operation it is performing, it does following:
+- finds the VolumePluginBackup CR related to this backup operation by using `tags` passed in CreateSnapshot call
+- updates the CR with the progress regularly.
+
+After return from `CreateSnapshot` in `takePVSnapshot`, currently Velero adds `volume.Snapshot` to `backupRequest`. Instead of this, CR will be added to `backupRequest`.
+During persistBackup call, this CR also will be backed up to backup location.
+
+In backupSyncController, it checks for any VolumePluginBackup CRs that need to be synced from backup location, and syncs them to cluster if needed.
+
+VolumePluginBackup will be useful as long as backed up data is available at backup location. When the Backup is deleted either by manually or due to expiry, VolumePluginBackup also can be deleted.
+
+`processRequest` of `backupDeletionController` will perform deletion of VolumePluginBackup before volumesnapshotter's DeleteSnapshot is called.
+
+#### Backward compatibility:
+
+Currently `volume.Snapshot` is backed up as `<backupname>-volumesnapshots.json.gz` file in the backup location.
+
+As the VolumePluginBackup CR is backed up instead of `volume.Snapshot`, to provide backward compatibility, CR will be backed as the same file i.e., `<backupname>-volumesnapshots.json.gz` file in the backup location.
+
+For backward compatibility on restore side, consider below possible cases wrt Velero version on restore side and format of json.gz file at object location:
+
+- older version of Velero, older json.gz file (backupname-volumesnapshots.json.gz)
+
+- older version of Velero, newer json.gz file
+
+- newer version of Velero, older json.gz file
+
+- newer version of Velero, newer json.gz file
+
+First and last should be fine.
+
+For second case, decode in `GetBackupVolumeSnapshots` on the restore side should fill only required fields of older version and should work.
+
+For third case, after decode, metadata.name will be empty. `GetBackupVolumeSnapshots` decodes older json.gz into the CR which goes fine.
+It will be modified to return []VolumePluginBackupSpec, and the changes are done accordingly in its caller.
+
+If decode fails in second case during implementation, this CR need to be backed up to different file. And, for backward compatibility, newer code should check for old file existence, and follow older code if exists. If it doesn't exists, check for newer file and follow the newer code.
+
+`backupSyncController` on restore clusters gets the `<backupname>-volumesnapshots.json.gz` object from backup location and decodes it to in-memory VolumePluginBackup CR. If its `metadata.name` is populated, controller creates CR. Otherwise, it will not create the CR on the cluster. It can be even considered to create CR on the cluster.
+
+#### VolumePluginRestore CR
+
+```
+// VolumePluginRestoreSpec is the specification for a VolumePluginRestore CR.
+type VolumePluginRestoreSpec struct {
+	// SnapshotID is the identifier for the snapshot of the volume.
+	// This will be used to relate with output in 'velero describe backup'
+	SnapshotID string `json:"snapshotID"`
+
+	// BackupName is the name of the Velero backup from which PV will be
+	// created.
+	BackupName string `json:"backupName"`
+
+	// Provider is the Provider field given in VolumeSnapshotLocation
+	Provider string `json:"provider"`
+
+	// VolumeType is the type of the disk/volume in the cloud provider
+	// API.
+	VolumeType string `json:"volumeType"`
+
+	// VolumeAZ is the where the volume is provisioned
+	// in the cloud provider.
+	VolumeAZ string `json:"volumeAZ,omitempty"`
+}
+
+// VolumePluginRestoreStatus is the current status of a VolumePluginRestore CR.
+type VolumePluginRestoreStatus struct {
+	// Phase is the current state of the VolumePluginRestore.
+	Phase string `json:"phase"`
+
+	// VolumeID is the PV name to which restore done
+	VolumeID string `json:"volumeID"`
+
+	// Message is a message about the volume plugin's restore's status.
+	// +optional
+	Message string `json:"message,omitempty"`
+
+	// StartTimestamp records the time a restore was started.
+	// Separate from CreationTimestamp, since that value changes
+	// on restores.
+	// The server's time is used for StartTimestamps
+	// +optional
+	// +nullable
+	StartTimestamp *metav1.Time `json:"startTimestamp,omitempty"`
+
+	// CompletionTimestamp records the time a restore was completed.
+	// Completion time is recorded even on failed restores.
+	// The server's time is used for CompletionTimestamps
+	// +optional
+	// +nullable
+	CompletionTimestamp *metav1.Time `json:"completionTimestamp,omitempty"`
+
+	// Progress holds the total number of bytes of the snapshot and the current
+	// number of restored bytes. This can be used to display progress information
+	// about the restore operation.
+	// +optional
+	Progress VolumeOperationProgress `json:"progress,omitempty"`
+
+	// PluginSpecific are a map of key-value pairs that plugin want to provide
+	// to user to identify plugin properties related to this restore
+	// +optional
+	PluginSpecific map[string]string `json:"pluginSpecific,omitempty"`
+}
+
+type VolumePluginRestore struct {
+	metav1.TypeMeta `json:",inline"`
+
+	// +optional
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	// +optional
+	Spec VolumePluginRestoreSpec `json:"spec,omitempty"`
+
+	// +optional
+	Status VolumePluginRestoreStatus `json:"status,omitempty"`
+}
+```
+
+For every restore operation, Velero creates VolumePluginRestore CR before calling volumesnapshotter's CreateVolumeFromSnapshot API.
+
+In order to know the CR created for the particular restore of a volume, Velero adds following labels to CR:
+- `velero.io/backup-name` with value as Backup Name, and,
+- `velero.io/snapshot-id` with value as snapshot id that need to be restored
+- `velero.io/provider` with value as `Provider` in `VolumeSnapshotLocation`
+
+Labels will be set with the value returned from `GetValidName` function. (https://github.com/vmware-tanzu/velero/blob/master/pkg/label/label.go#L35).
+
+Plugin will be able to identify CR by using snapshotID that it received as parameter of CreateVolumeFromSnapshot API, and plugin's Provider name.
+It updates the progress of restore operation regularly if plugin supports feature of showing progress.
+
+Velero deletes VolumePluginRestore CR when it handles deletion of Restore CR.
+
+### Approach 2
+
+This approach is different to approach 1 only with respect to Backup.
+
+#### VolumePluginBackup CR
+
+```
+// VolumePluginBackupSpec is the specification for a VolumePluginBackup CR.
+type VolumePluginBackupSpec struct {
+	// Volume is the PV name to be backed up.
+	Volume string `json:"volume"`
+
+	// Backup name
+	Backup string `json:"backup"`
+
+	// Provider is the Provider field given in VolumeSnapshotLocation
+	Provider string `json:"provider"`
+}
+
+// VolumePluginBackupStatus is the current status of a VolumePluginBackup CR.
+type VolumePluginBackupStatus struct {
+	// Phase is the current state of the VolumePluginBackup.
+	Phase string `json:"phase"`
+
+	// SnapshotID is the identifier for the snapshot of the volume.
+	// This will be used to relate with output in 'velero describe backup'
+	SnapshotID string `json:"snapshotID"`
+
+	// Message is a message about the volume plugin's backup's status.
+	// +optional
+	Message string `json:"message,omitempty"`
+
+	// StartTimestamp records the time a backup was started.
+	// Separate from CreationTimestamp, since that value changes
+	// on restores.
+	// The server's time is used for StartTimestamps
+	// +optional
+	// +nullable
+	StartTimestamp *metav1.Time `json:"startTimestamp,omitempty"`
+
+	// CompletionTimestamp records the time a backup was completed.
+	// Completion time is recorded even on failed backups.
+	// Completion time is recorded before uploading the backup object.
+	// The server's time is used for CompletionTimestamps
+	// +optional
+	// +nullable
+	CompletionTimestamp *metav1.Time `json:"completionTimestamp,omitempty"`
+
+	// PluginSpecific are a map of key-value pairs that plugin want to provide
+	// to user to identify plugin properties related to this backup
+	// +optional
+	PluginSpecific map[string]string `json:"pluginSpecific,omitempty"`
+
+	// Progress holds the total number of bytes of the volume and the current
+	// number of backed up bytes. This can be used to display progress information
+	// about the backup operation.
+	// +optional
+	Progress VolumeOperationProgress `json:"progress,omitempty"`
+}
+
+type VolumeOperationProgress struct {
+	TotalBytes int64
+	BytesDone int64
+}
+
+type VolumePluginBackup struct {
+	metav1.TypeMeta `json:",inline"`
+
+	// +optional
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	// +optional
+	Spec VolumePluginBackupSpec `json:"spec,omitempty"`
+
+	// +optional
+	Status VolumePluginBackupStatus `json:"status,omitempty"`
+}
+```
+
+For every backup operation of volume, volume snapshotter creates VolumePluginBackup CR in Velero namespace.
+It keep updating the progress of operation along with other details like Volume name, Backup Name, SnapshotID etc as mentioned in the CR.
+
+In order to know the CR created for the particular backup of a volume, volume snapshotters adds following labels to CR:
+- `velero.io/backup-name` with value as Backup Name, and,
+- `velero.io/volume-name` with value as volume that is undergoing backup
+
+Backup name being unique won't cause issues like duplicates in identifying the CR.
+
+Plugin need to sanitize the value that can be set for above labels. Label need to be set with the value returned from `GetValidName` function. (https://github.com/vmware-tanzu/velero/blob/master/pkg/label/label.go#L35).
+
+Though no restrictions are required on the name of CR, as a general practice, volume snapshotter can name this CR with the value same as return value of CreateSnapshot.
+
+After return from `CreateSnapshot` in `takePVSnapshot`, if VolumePluginBackup CR exists for particular backup of the volume, velero adds this CR to `backupRequest`.
+During persistBackup call, this CR also will be backed up to backup location.
+
+In backupSyncController, it checks for any VolumePluginBackup CRs that need to be synced from backup location, and syncs them to cluster if needed.
+
+`processRequest` of `backupDeletionController` will perform deletion of VolumePluginBackup before volumesnapshotter's DeleteSnapshot is called.
+
+Another alternative is:
+Deletion of `VolumePluginBackup` CR can be delegated to plugin. Plugin can perform deletion of VolumePluginBackup using the `snapshotID` passed in volumesnapshotter's DeleteSnapshot request.
+
+### 'core' Velero client/server required changes
+
+- Creation of the VolumePluginBackup/VolumePluginRestore CRDs at installation time
+- Persistence of VolumePluginBackup CRs towards the end of the back up operation
+- As part of backup synchronization, VolumePluginBackup CRs related to the backup will be synced.
+- Deletion of VolumePluginBackup when volumeshapshotter's DeleteSnapshot is called
+- Deletion of VolumePluginRestore as part of handling deletion of Restore CR
+- In case of approach 1,
+  - converting `volume.Snapshot` struct as CR and its related changes
+  - creation of VolumePlugin(Backup|Restore) CRs before calling volumesnapshotter's API
+  - `GetBackupVolumeSnapshots` and its callers related changes for change in return type from []volume.Snapshot to []VolumePluginBackupSpec.
+
+### Velero CLI required changes
+
+In 'velero describe' CLI, required CRs will be fetched from API server and its contents like backupName, PVName (if changed due to label size limitation), size of PV snapshot will be shown in the output.
+
+### API Upgrade
+When CRs gets upgraded, velero can support older API versions also (till they get deprecated) to identify the CRs that need to be persisted to backup location.
+However, it can provide preference over latest supported API.
+
+If new fields are added without changing API version, it won't cause any problem as these resources are intended to provide information, and, there is no reconciliation on these resources.
+
+### Compatibility of latest plugin with older version of Velero
+Plugin that supports this CR should handle the situation gracefully when CRDs are not installed. It can handle the errors occured during creation/updation of the CRs.
+
+## Limitations:
+
+Non K8s native plugins will not be able to implement this as they can not create the CRs.
+
+## Open Questions
+
+## Alternatives Considered
+
+### Add another method to VolumeSnapshotter interface 
+Above proposed approach have limitation that plugin need to be K8s native in order to create, update CRs.
+Instead, a new method for 'Progress' will be added to interface. Velero server regularly polls this 'Progress' method and updates VolumePluginBackup CR on behalf of plugin.
+
+But, this involves good amount of changes and needs a way for backward compatibility.
+
+As volume plugins are mostly K8s native, its fine to go ahead with current limiation.
+
+### Update Backup CR
+Instead of creating new CRs, plugins can directly update the status of Backup CR. But, this deviates from current approach of having seperate CRs like PodVolumeBackup/PodVolumeRestore to know operations progress.
+
+### Restricting on name rather than using labels
+Instead of using labels to identify the CR related to particular backup on a volume, restrictions can be placed on the name of VolumePluginBackup CR to be same as the value returned from CreateSnapshot.
+But, this can cause issue when volume snapshotter just crashed without returning snapshot id to velero.
+
+### Backing up VolumePluginBackup CR to different object
+If CR is backed up to different object other than `#backup-volumesnapshots.json.gz` in backup location, restore controller need to follow 'fall-back model'.
+It first need to check for new kind of object, and, if it doesn't exists, follow the old model. To avoid 'fall-back' model which prone to errors, VolumePluginBackup CR is backed to same location as that of `volume.Snapshot` location.
+
+## Security Considerations
+
+Currently everything runs under the same `velero` service account so all plugins have broad access, which would include being able to modify CRs created by another plugin.
+

examples/README.md

@@ -7,5 +7,5 @@ This directory contains sample YAML config files that can be used for exploring
 * `nginx-app/`: A sample nginx app that can be used to test backups and restores.
 
 
-[0]: /docs/get-started.md
+[0]: /docs/contributions/minio.md
 [1]: https://github.com/minio/minio

examples/nginx-app/base.yaml

@@ -37,7 +37,7 @@ spec:
         app: nginx
     spec:
       containers:
-      - image: nginx:1.7.9
+      - image: nginx:1.17.6
         name: nginx
         ports:
         - containerPort: 80