Commit Graph
84 Commits
Author SHA1 Message Date
Xun JiangandXun Jiang effbcba521 Add E2E test cases for backup VolumeInfo feature.
Signed-off-by: Xun Jiang <blackpigletbruce@gmail.com>
2024-02-22 16:18:55 +08:00
Xun Jiang 08dedd8b66 Rename volumeinfo metadata file.
Change from <backup-name>-volumeinfos.json.gz to
<backup-name>-volumeinfo.json.gz.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2024-01-03 11:22:49 +08:00
Xun Jiang 9be8eb0c6d Don't generate empty structure.
VolumeInfo contains several sub-structures. They are filled for
different scenarios. Do not generate empty structure for the
not filled sub-structures.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-12-21 14:53:03 +08:00
Xun Jiang 3dc202d30a Bump Golang library versions for v1.13 Restic to fix CVEs.
Bump golang.org/x/crypto version to v0.17.0.
Bump google.golang.org/grpc version to v1.56.3.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-12-20 10:31:48 +08:00
Xun Jiang 6222891d5b Fix shedule get and describe CLI issue.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-12-18 16:41:10 +08:00
Xun Jiang edb0860dd2 Fix issue #7163.
Update CSIVolumeSnapshotsCompleted in backup's status and the metric
during backup finalize stage according to async operations content.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-12-07 09:43:10 +08:00
Xun Jiang c8e76f4602 Fix the DownloadRequest context error.
Clean the DownloadRequest Stream function.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-12-05 13:29:23 +08:00
Xun Jiang 28df14d9d5 Modify restore logic.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-12-05 10:01:16 +08:00
Xun Jiang c77bec73bb Move VolumesInformation to an independant package.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-12-04 08:33:37 +08:00
Xun Jiang ca97248f2a Use VolumeInfo to help restore the PV.
Add VolumeInfo for left PVs during backup.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-12-04 08:33:37 +08:00
Xun Jiang f5c159ce56 Resolve linter issues.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-11-29 11:15:43 +08:00
Xun Jiang d70535b6d2 Add nolintlint linter.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-11-29 11:13:46 +08:00
Xun Jiang ec03d1ebce Add noctx linter.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-11-29 11:13:46 +08:00
Xun Jiang dbd1a12d9f Add nilerr and ginkgolinter linter.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-11-29 11:13:46 +08:00
Xun Jiang cddc11e000 Enable linter errchkjson.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-11-29 11:13:46 +08:00
Xun Jiang 3805a470a9 Enable dupword linter.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-11-29 11:13:46 +08:00
Xun Jiang 7f52321772 Generate VolumeInfo.
Remove CSI VolumeSnapshot listter and the informer.
Add download the VolumeInfos metadata for backup.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-11-22 09:40:38 +08:00
Xun Jiang b440a4f53f Add VolumeInfo metadata structures and object get method.
Modify design according to comments.
Add PVInfo structure.
Add backup VolumeInfo's object storage's put and get methods.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-11-17 17:23:47 +08:00
Xun Jiang 1fb0529d98 Add DataUpload Result and CSI VolumeSnapshot check for restore PV.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-11-06 22:40:03 +08:00
Xun Jiang 56b5e982d9 Remove dependency of generated client part 3
Replace generated discovery client with client-go client.
Remove generated client from PVR action.
Remove generated client from pkg/cmd directory.
Delete velero generate client from client factory.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-11-06 11:34:39 +08:00
Xun Jiang a221a88945 Remove dependency of generated client part 2
Remove dependecy of generate client from pkg/cmd/cli/snapshotLocation.
Remove the Velero generated informer from PVB and PVR. 
Remove dependency of generated client from pkg/podvolume directory.
Replace generated codec with runtime codec. 

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-11-03 17:11:36 +08:00
Xun Jiang 23b9484370 Add the PV backup information design document.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-11-02 10:14:16 +08:00
Xun Jiang a94918026c Add HealthCheckNodePort deletion logic in Service restore.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-10-27 14:13:52 +08:00
Xun Jiang 908e2c63ba Check whether the action is a CSI action and whether CSI feature is
enabled, before executing the action.

The DeleteItemAction is not checked, because the DIA doesn't have a
method to get the action's plugin name.
This should be OK, because the CSI will check whether the VS and VSC
have a backup name annotation. If the VS and VSC is not handled by
the CSI plugin, then they don't have the annotation.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-10-24 16:54:38 +08:00
Xun Jiang ecc6e1621e Make Windows build skip BlockMode code.
PVC block mode backup and restore introduced some OS specific
system calls. Those calls are not available for Windows, so
add both non Windows version and Windows version code, and
return error for block mode on the Windows platform.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-10-20 19:39:44 +08:00
Xun Jiang 79c75718ca Change controller-runtime List option from MatchingFields to ListOptions.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-10-19 17:09:12 +08:00
Xun Jiang a726329e82 Bump golang version.
Bump golang version to v1.21.
Bump golang.org/x/net version to v0.17.0 in Velero and Restic.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-10-13 16:30:23 +08:00
Xun Jiang 79e176086c Add some configurations to avoid ArgoCD pruning backups generated from schedule.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-10-10 21:06:48 +08:00
Xun Jiang dbc3ad7453 Update the Velero chocolatey package release procedure.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-10-10 20:29:29 +08:00
Xun Jiang 98a383d94a Add an working example for rbac.md.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-10-10 13:56:50 +08:00
Xun Jiang 64595cc0f7 Add go clean in Dockerfile and action.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-09-28 20:30:05 +08:00
Xun Jiang 79b810ed25 Fix #6752: add namespace exclude check.
Add PSA audit and warn labels.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-09-06 14:44:30 +08:00
Xun Jiang 33b21a380c Remove PVC's selector in backup's PVC action.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-07-27 10:09:00 +08:00
Xun Jiang ccece7c855 Add RetentionPolicy supporting limitation in documents.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-07-19 09:36:45 +08:00
Xun Jiang 4a222b76c6 Improve backup log command UT coverage.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-07-18 14:41:55 +08:00
Xun Jiang 9e515ac397 Remove dependency of the legacy client code from pkg/cmd directory part 2.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-07-17 17:04:53 +08:00
Xun Jiang 89d3ad4864 Remove dependency of the legacy client code from pkg/cmd directory.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-07-14 15:55:09 +08:00
Xun Jiang d7f1ea4fbd Add exit code log and possible memory shortage warning log for Restic command failure.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-07-05 21:35:52 +08:00
Xun Jiang 7b4d4c7275 Limit label "velero.io/pvc-namespace-name" length to 63.
1. Limit label length.
2. Modify UT accordingly.
3. Remove unnecessary const variable.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-07-04 15:10:34 +08:00
Xun Jiang 40b2ee1323 Modify DownloadRequest controller logic
1. Avoid patch DownloadRequest when it's deleted.
2. Add periodic enqueue resource for reconcile.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-07-03 17:11:48 +08:00
Xun Jiang e205e2122d Retrieve DataUpload into backup result ConfigMap during volume snapshot restore.
Fix issue #6117.
Add CSI plugin needs builder functions.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-06-30 11:18:36 +08:00
Xun Jiang ec4bb42117 Retrieve DataUpload into backup result ConfigMap during volume snapshot restore.
Fix issue #6117.
Add CSI plugin needs builder functions.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-06-30 11:13:04 +08:00
Xun Jiang a5c28ad423 Add warning message for volume snapshotter in data mover case.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-06-12 17:57:01 +08:00
Xun Jiang f8d9cfdb84 Make CSI snapshot creation timeout configurable for backup and schedule.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-08-08 17:07:26 +08:00
Xun Jiang 7af1e23614 Add annotation "pv.kubernetes.io/migrated-to" for CSI checking.
1. Also checking annotation "pv.kubernetes.io/migrated-to" to find out whether volume is provisioned by CSI.
2. Add UT cases.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-08-08 10:59:41 +08:00
Xun Jiang e8da5df57a Reduce CRD size.
1. Make the Restore hook.InitConatianer server side field pruing disable.
2. Remove restore patch in update-generate-crd-code.sh.
3. Modify related testcases.
4. Add Container fields validation in Restore Init hook.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-08-02 19:45:01 +08:00
Xun Jiang cd643bbac9 Add labeled and unlabeled events for PR changelog check action.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-07-28 17:45:26 +08:00
Xun Jiang 396e68b810 VolumeSnapshotLocation refactor with kubebuilder.
1. modify VSL CRD API file name. Add kubebuilder related marker.
2. Add Labels init code for VSL create command.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-07-25 15:46:43 +08:00
Xun Jiang 82ac228a01 Delay CA file deletion in PVB controller
Fix #5140.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-07-22 19:36:17 +08:00
Xun Jiang 7b320e71c9 Modify Github actions.
1. remove go.sum file from code spell check action.
2. change go version to 1.17 in CRD verify action, and add k8s 1.23 and 1.24 in verification list.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-06-27 15:15:19 +08:00
Xun Jiang cb273ae469 Delete opened issues triage action.
Because the column and project specified by this action do not exist anymore, and Velero team doesn't use this action to assign issue and triage anymore, remove this action.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-06-22 10:46:54 +08:00
Xun Jiang 806234e043 Add more information for failing to get path or snapshot in restic backup and restore.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-06-13 11:32:55 +08:00
Xun Jiang febe79f334 Fix E2E test [Backups][Deletion][Restic] on GCP.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-06-08 09:23:58 +08:00
Xun Jiang 3db40a58a4 Make velero completion zsh command output can be used by source command.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-05-17 10:01:46 +08:00
Xun Jiang 989a1e3ebb Refactor BSL controller with periodical enqueue source.
Add filter functions for PeriodicalEnqueueSource.
Move BSL's valication frequency check test case to PeriodicalEnqueueSource's test.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-05-11 15:31:47 +08:00
Xun Jiang a69bd8b1f3 Remove VolumeSnapshot created during backup, when the VolumeSnapshotClass's DeletionPolicy is set to Delete.
1. Delete VolumeSnapshot directly when DeletionPolicy set to Retain.
2. Change VolumeSnapshotContent's DeletionPolicy to Retain, then delete VolumeSnapshot. After that delete VolumeSnapshotContent and change VSC DeletionPolicy to Delete back, then re-create the VolumeSnapshotContent.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-05-05 18:30:59 +08:00
Xun Jiang 6421f8ad02 Make waiting VolumeSnapshot to ready process parallel.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-05-05 14:32:55 +08:00
Xun Jiang 22dd0b2ebb Modify CSI VolumeSnapshot metric related code.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-04-23 12:59:56 +08:00
Xun Jiang 8064421e83 Update integrated Restic version and add insecureSkipTLSVerify for Restic CLI
1. Add --insecure-tls for ResticManager's commands.
2. Add --insecure-tls in PodVolumeBackup and PodVolumeRestore controller.
3. Upgrade integrated Restic version to v0.13.1
4. Change --last flag in Restic command to --latest=1 due to Restic version update.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-04-18 21:29:33 +08:00
Xun Jiang 9aa4e9e860 Update according to comments. Remove csiVolumeSnapshotsStorageTotal related code.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-04-15 08:05:56 +00:00
Xun Jiang 4daeec7ab9 Update CRD and GRPC.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-04-09 12:35:09 +00:00
Xun Jiang 368a1ddf3c Add CSI VolumeSnapshot related metrics.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-04-09 00:57:28 +08:00
Xun Jiang 5c74aa22e6 Avoid duplicated snapshotting, when CSI is enabled
Fix #4758

Do not take snapshot for PV to avoid duplicated snapshotting, when CSI feature is enabled.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-04-07 23:32:48 +08:00
Xun Jiang fa646888ef Add parameter for e2e test to support modify kibishii install path.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-03-27 20:33:26 +08:00
Xun JiangandHoang, Phuong 291f0c17e8 Support multiple skip option for E2E test
The GINKGO_SKIP option is updated to string that can be separated by "." for "make test-e2e".

Signed-off-by: Xun Jiang <jxun@vmware.com>
Signed-off-by: Hoang, Phuong <phuong.n.hoang@dell.com>
2022-03-15 19:05:13 -04:00
Xun Jiang d694d40919 Support multiple skip option for E2E test
The GINKGO_SKIP option is updated to string that can be separated by "." for "make test-e2e".

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-03-08 17:04:32 +08:00
Xun Jiang 860e4e2c1a Remove gcp auth action. Use JSON key as docker login secret instead.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-03-01 15:16:52 +08:00
Xun Jiang 37412fe214 Change authentication method from workload identity provider to service account key.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-03-01 10:59:39 +08:00
Xun Jiang 0503b2675f Modify according to comments.
Change hook.Container's value in test case to ""

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-02-28 19:28:17 +08:00
Xun Jiang 9cb81d825e Add skip case support in e2e test
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-02-23 14:05:22 +08:00
Xun Jiang 4c8d1c2693 Modify according to comments
1. rename zoneSeparator to gkeZoneSeparator
2. add example of regional PV's node affinity. modify test case description.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-02-22 16:23:10 +08:00
Xun Jiang 8c2a07b906 Update --use-owner-references-in-backup description in velero command line
Specify the risk of this parameter set to true. Add the issue first reported about this topic which includeds the google document illustrates about it.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-02-21 22:37:48 +08:00
Xun Jiang 7be12a9220 [fix] Add regional PV support for GKE
fix #4663.
For GKE pv, when create backup, return all zones retrived from node affinity.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-02-21 22:04:21 +08:00
Xun Jiang 6d9004dc62 [fix] Avoid overwritten hook's exec.container parameter when running pod command executor
Fix #4499
When hook influnce multiple pods, current logic's first pod's container will overwrite the hook's exec.container parameter. That will cause the other pod fail on the hook executing.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-02-17 15:24:56 +08:00
Xun Jiang 9419aa91a7 Update e2e test image to gcr.io
By now, only busybox:latest is used by e2e. It is already upload to gcr.io/velero-gcp/busybox:latest
Change the image to gcr.io to avoid pulling rate limitation from docker hub.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-02-14 20:55:34 +08:00
Xun Jiang 01842a1552 Add GCR login actions
use google-github-actions/auth to login GCP. Login gcr.io with generated access token.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-02-10 15:42:18 +08:00
Xun Jiang 5844fca5af Add pushing image to GCR in github workflow
Push to GCR in github workflow to faciliate some environments that have rate limitation to docker hub, e.g. vSphere.
 <root@jxun-jumpserver.c.velero-gcp.internal>
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-02-09 16:27:25 +08:00
Xun Jiang bdde7585c8 Add changelog.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2021-12-17 16:16:03 +08:00
Xun Jiang 7ab4bfc632 Revert "Migrate backup sync controller from code-generator to kubebuilder (#4423)"
This reverts commit 5aaeb3ebbe.
2021-12-17 09:40:24 +08:00
Xun Jiang 706d142096 fix: remove --crds-version in velero install command
Due to only v1 CRD is supported in velero version 1.8, remove CRDs version choosing option.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2021-12-13 21:02:56 +08:00
Xun Jiang 303d3dcad3 feat: migrate kubebuilder from v2 to v3
1. remove config/crd/v1beta1
2. remove PROJECT file
3. update controller-gen and kubebuilder version
4. generate client and CRD file
5. add changelog and remove v1beta1 CRD generated code.
6. add kubebuilder test bundle setup command.
7. due to apiextensions.k8s.io/v1beta1 is not supported, only k8s after v1.16 is supported, so remove v1.15 check.
8. add CRD and k8s suppored version update in changelog.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2021-11-23 19:32:19 +08:00
Xun Jiang 2c240c2830 Use logrusr instead of zap.
logrusr is a open source convertor, which can convert logrus logger into logr.
By using logrusr, velero can use exsiting formatted logrus logger, other than introducing zap as a new logger.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2021-11-17 18:10:37 +08:00
Xun Jiang 8eee35a62e Fix: add init log option for velero controller-runtime manager.
fix for issue #3737
add log option for velero controller-runtime manager to log return error in reconcile loop.

Signed-off-by: Xun Jiang jxun@vmware.com
Signed-off-by: Xun Jiang <jxun@vmware.com>
2021-11-16 15:42:00 +08:00
Xun Jiang 7b89950031 Fix: add init log option for velero controller-runtime manager.
fix for issue #3737
add log option for velero controller-runtime manager to log return error in reconcile loop.

Signed-off-by: Xun Jiang jxun@vmware.com
Signed-off-by: Xun Jiang <jxun@vmware.com>
2021-11-16 15:37:43 +08:00