diff --git a/site/content/docs/main/maintainers.md b/site/content/docs/main/maintainers.md index c3af51d8c..579fb3109 100644 --- a/site/content/docs/main/maintainers.md +++ b/site/content/docs/main/maintainers.md @@ -24,6 +24,8 @@ Some tips for doing reviews: ## Creating a release Maintainers are expected to create releases for the project. We have parts of the process automated, and full [instructions](release-instructions). +We are working towards automating more the Velero testing, but there is still a need for manual testing as part of the release process. +The manual test cases for release testing are documented [here](./manual-testing). ## Community support Maintainers are expected to participate in the community support rotation. We have guidelines for how we handle the [support](support-process). diff --git a/site/content/docs/main/manual-testing.md b/site/content/docs/main/manual-testing.md new file mode 100644 index 000000000..d7d896883 --- /dev/null +++ b/site/content/docs/main/manual-testing.md @@ -0,0 +1,92 @@ +--- +title: "Manual Testing Requirements for Velero" +layout: docs +--- + +Although we have automated unit and end-to-end tests, there is still a need for Velero to undergo manual tests during a release. +This document outlines the manual test operations that Velero needs to correctly perform in order to be considered ready for release. + +## Current test cases + +The following are test cases that are currently performed as part of a Velero release. + +### Install + +- Verify that Velero CRDs are compatible with the earliest and latest versions of Kubernetes that we support: + - Kubernetes v1.12 + - Kubernetes v1.20 + +### Upgrade + +- Verify that Velero upgrade instructions work + +### Basic functionality + +The "Backup and Restore" test cases below describe general backup and restore functionality that needs to run successfully on all the following providers that we maintain plugins for: +- AWS +- GCP +- Microsoft Azure +- VMware vSphere + +#### Backup and Restore + +- Verify that a backup and restore using Volume Snapshots can be performed +- Verify that a backup and restore using Restic can be performed +- Verify that a backup of a cluster workload can be restored in a new cluster +- Verify that an installation using the latest version can be used to restore from backups created with the last 3 versions. + - e.g. Install Velero 1.6 and use it to restore backups from Velero v1.3, v1.4, v1.5. + +### Working with Multiple Providers + +The following are test cases that exercise Velero behaviour when interacting with multiple providers: + +- Verify that a backup and restore to multiple BackupStorageLocations using the same provider with unique credentials can be performed +- Verify that a backup and restore to multiple BackupStorageLocations using different providers with unique credentials can be performed +- Verify that a backup and restore that includes volume snapshots using different providers for the snapshots and object storage can be performed + - e.g. perform a backup and restore using AWS for the VolumeSnapshotLocation and Azure Blob Storage as the BackupStorageLocation + +## Future test cases + +The following are test cases that are not currently performed as part of a Velero release but cases that we will want to cover with future releases. + +### Schedules + +- Verify that schedules create a backup upon creation and create Backup resources at the correct frequency + +### Resource management + +- Verify that deleted backups are successfully removed from object storage +- Verify that backups that have been removed from object storage can still be deleted with `velero delete backup` +- Verify that Volume Snapshots associated with a deleted backup are removed +- Verify that backups that exceed their TTL are deleted +- Verify that existing backups in object storage are synced to Velero + +### Restic repository test cases + +- Verify that restic repository maintenance is performed as the specified interval + +### Backup Hooks + +- Verify that a pre backup hook provided via pod annotation is performed during backup +- Verify that a pre backup hook provided via Backup spec is performed during backup +- Verify that a post backup hook provided via pod annotation is performed during backup +- Verify that a post backup hook provided via Backup spec is performed during backup + +### Restore Hooks + +- Verify that an InitContainer restore hook provided via pod annotation is performed during restore +- Verify that an InitContainer restore hook provided via Restore spec is performed during restore +- Verify that an InitContainer restore hook provided via Restore spec is performed during restore that includes restoring restic volumes +- Verify that an Exec restore hook provided via pod annotation is performed during restore +- Verify that an Exec restore hook provided via Restore spec is performed during restore + + +#### Resource filtering + +- Verify that backups and restores correctly apply the following resource filters: + - `--include-namespaces` + - `--include-resources` + - `--include-cluster-resources` + - `--exclude-namespaces` + - `--exclude-resources` + - `velero.io/exclude-from-backup=true` label diff --git a/site/content/docs/v1.6/maintainers.md b/site/content/docs/v1.6/maintainers.md index 6223859c0..5e7071f55 100644 --- a/site/content/docs/v1.6/maintainers.md +++ b/site/content/docs/v1.6/maintainers.md @@ -24,6 +24,8 @@ Some tips for doing reviews: ## Creating a release Maintainers are expected to create releases for the project. We have parts of the process automated, and full [instructions](release-instructions). +We are working towards automating more the Velero testing, but there is still a need for manual testing as part of the release process. +The manual test cases for release testing are documented [here](./manual-testing). ## Community support Maintainers are expected to participate in the community support rotation. We have guidelines for how we handle the [support](support-process). diff --git a/site/content/docs/v1.6/manual-testing.md b/site/content/docs/v1.6/manual-testing.md new file mode 100644 index 000000000..d7d896883 --- /dev/null +++ b/site/content/docs/v1.6/manual-testing.md @@ -0,0 +1,92 @@ +--- +title: "Manual Testing Requirements for Velero" +layout: docs +--- + +Although we have automated unit and end-to-end tests, there is still a need for Velero to undergo manual tests during a release. +This document outlines the manual test operations that Velero needs to correctly perform in order to be considered ready for release. + +## Current test cases + +The following are test cases that are currently performed as part of a Velero release. + +### Install + +- Verify that Velero CRDs are compatible with the earliest and latest versions of Kubernetes that we support: + - Kubernetes v1.12 + - Kubernetes v1.20 + +### Upgrade + +- Verify that Velero upgrade instructions work + +### Basic functionality + +The "Backup and Restore" test cases below describe general backup and restore functionality that needs to run successfully on all the following providers that we maintain plugins for: +- AWS +- GCP +- Microsoft Azure +- VMware vSphere + +#### Backup and Restore + +- Verify that a backup and restore using Volume Snapshots can be performed +- Verify that a backup and restore using Restic can be performed +- Verify that a backup of a cluster workload can be restored in a new cluster +- Verify that an installation using the latest version can be used to restore from backups created with the last 3 versions. + - e.g. Install Velero 1.6 and use it to restore backups from Velero v1.3, v1.4, v1.5. + +### Working with Multiple Providers + +The following are test cases that exercise Velero behaviour when interacting with multiple providers: + +- Verify that a backup and restore to multiple BackupStorageLocations using the same provider with unique credentials can be performed +- Verify that a backup and restore to multiple BackupStorageLocations using different providers with unique credentials can be performed +- Verify that a backup and restore that includes volume snapshots using different providers for the snapshots and object storage can be performed + - e.g. perform a backup and restore using AWS for the VolumeSnapshotLocation and Azure Blob Storage as the BackupStorageLocation + +## Future test cases + +The following are test cases that are not currently performed as part of a Velero release but cases that we will want to cover with future releases. + +### Schedules + +- Verify that schedules create a backup upon creation and create Backup resources at the correct frequency + +### Resource management + +- Verify that deleted backups are successfully removed from object storage +- Verify that backups that have been removed from object storage can still be deleted with `velero delete backup` +- Verify that Volume Snapshots associated with a deleted backup are removed +- Verify that backups that exceed their TTL are deleted +- Verify that existing backups in object storage are synced to Velero + +### Restic repository test cases + +- Verify that restic repository maintenance is performed as the specified interval + +### Backup Hooks + +- Verify that a pre backup hook provided via pod annotation is performed during backup +- Verify that a pre backup hook provided via Backup spec is performed during backup +- Verify that a post backup hook provided via pod annotation is performed during backup +- Verify that a post backup hook provided via Backup spec is performed during backup + +### Restore Hooks + +- Verify that an InitContainer restore hook provided via pod annotation is performed during restore +- Verify that an InitContainer restore hook provided via Restore spec is performed during restore +- Verify that an InitContainer restore hook provided via Restore spec is performed during restore that includes restoring restic volumes +- Verify that an Exec restore hook provided via pod annotation is performed during restore +- Verify that an Exec restore hook provided via Restore spec is performed during restore + + +#### Resource filtering + +- Verify that backups and restores correctly apply the following resource filters: + - `--include-namespaces` + - `--include-resources` + - `--include-cluster-resources` + - `--exclude-namespaces` + - `--exclude-resources` + - `velero.io/exclude-from-backup=true` label