33 Commits

Author SHA1 Message Date
Scott Seago
9c1c7d20ff Minor refactoring
Signed-off-by: Scott Seago <sseago@redhat.com>
2025-12-02 16:39:10 -05:00
Scott Seago
91357b28c4 Move worker pool creation to backup reconcile.
ItemBlockWorkerPool is now created for each backup.

Signed-off-by: Scott Seago <sseago@redhat.com>
2025-12-02 16:38:41 -05:00
Joseph Antony Vaikath
975f647323 Wildcard ns implement (#9255)
Some checks failed
Run the E2E test on kind / get-go-version (push) Failing after 1m5s
Run the E2E test on kind / build (push) Has been skipped
Run the E2E test on kind / setup-test-matrix (push) Successful in 3s
Run the E2E test on kind / run-e2e-test (push) Has been skipped
Main CI / get-go-version (push) Successful in 11s
Main CI / Build (push) Failing after 25s
Close stale issues and PRs / stale (push) Successful in 14s
Trivy Nightly Scan / Trivy nightly scan (velero, main) (push) Failing after 1m36s
Trivy Nightly Scan / Trivy nightly scan (velero-plugin-for-aws, main) (push) Failing after 1m2s
Trivy Nightly Scan / Trivy nightly scan (velero-plugin-for-gcp, main) (push) Failing after 1m19s
Trivy Nightly Scan / Trivy nightly scan (velero-plugin-for-microsoft-azure, main) (push) Failing after 1m2s
* Add wildcard status fields

Signed-off-by: Joseph <jvaikath@redhat.com>

* Implement wildcard namespace expansion in item collector

- Introduced methods to get active namespaces and expand wildcard includes/excludes in the item collector.
- Updated getNamespacesToList to handle wildcard patterns and return expanded lists.
- Added utility functions for setting includes and excludes in the IncludesExcludes struct.
- Created a new package for wildcard handling, including functions to determine when to expand wildcards and to perform the expansion.

This enhances the backup process by allowing more flexible namespace selection based on wildcard patterns.

Signed-off-by: Joseph <jvaikath@redhat.com>

* Enhance wildcard expansion logic and logging in item collector

- Improved logging to include original includes and excludes when expanding wildcards.
- Updated the ShouldExpandWildcards function to check for wildcard patterns in excludes.
- Added comments for clarity in the expandWildcards function regarding pattern handling.

These changes enhance the clarity and functionality of the wildcard expansion process in the backup system.

Signed-off-by: Joseph <jvaikath@redhat.com>

* Add wildcard namespace fields to Backup CRD and update deepcopy methods

- Introduced `wildcardIncludedNamespaces` and `wildcardExcludedNamespaces` fields to the Backup CRD to support wildcard patterns for namespace inclusion and exclusion.
- Updated deepcopy methods to handle the new fields, ensuring proper copying of data during object manipulation.

These changes enhance the flexibility of namespace selection in backup operations, aligning with recent improvements in wildcard handling.

Signed-off-by: Joseph <jvaikath@redhat.com>

* Refactor Backup CRD to rename wildcard namespace fields

- Updated `BackupStatus` struct to rename `WildcardIncludedNamespaces` to `WildcardExpandedIncludedNamespaces` and `WildcardExcludedNamespaces` to `WildcardExpandedExcludedNamespaces` for clarity.
- Adjusted associated comments to reflect the new naming and ensure consistency in documentation.
- Modified deepcopy methods to accommodate the renamed fields, ensuring proper data handling during object manipulation.

These changes enhance the clarity and maintainability of the Backup CRD, aligning with recent improvements in wildcard handling.

Signed-off-by: Joseph <jvaikath@redhat.com>

* Fix

Signed-off-by: Joseph <jvaikath@redhat.com>

* Refactor where wildcard expansion happens

Signed-off-by: Joseph <jvaikath@redhat.com>

* Refactor Backup CRD and related components for expanded namespace handling

- Updated `BackupStatus` struct to rename fields for clarity: `WildcardExpandedIncludedNamespaces` and `WildcardExpandedExcludedNamespaces` are now `ExpandedIncludedNamespaces` and `ExpandedExcludedNamespaces`, respectively.
- Adjusted associated comments and deepcopy methods to reflect the new naming conventions.
- Removed the `getActiveNamespaces` function from the item collector, streamlining the namespace handling process.
- Enhanced logging during wildcard expansion to provide clearer insights into the process.

These changes improve the clarity and maintainability of the Backup CRD and enhance the functionality of namespace selection in backup operations.

Signed-off-by: Joseph <jvaikath@redhat.com>

* Refactor wildcard expansion logic in item collector and enhance testing

- Moved the wildcard expansion logic into a dedicated method, `expandNamespaceWildcards`, improving code organization and readability.
- Updated logging to provide detailed insights during the wildcard expansion process.
- Introduced comprehensive unit tests for wildcard handling, covering various scenarios and edge cases.
- Enhanced the `ShouldExpandWildcards` function to better identify wildcard patterns and validate inputs.

These changes improve the maintainability and robustness of the wildcard handling in the backup system.

Signed-off-by: Joseph <jvaikath@redhat.com>

* Enhance Restore CRD with expanded namespace fields and update logic

- Added `ExpandedIncludedNamespaces` and `ExpandedExcludedNamespaces` fields to the `RestoreStatus` struct to support expanded wildcard namespace handling.
- Updated the `DeepCopyInto` method to ensure proper copying of the new fields.
- Implemented logic in the restore process to expand wildcard patterns for included and excluded namespaces, improving flexibility in namespace selection during restores.
- Enhanced logging to provide insights into the expanded namespaces.

These changes improve the functionality and maintainability of the restore process, aligning with recent enhancements in wildcard handling.

Signed-off-by: Joseph <jvaikath@redhat.com>

* Refactor Backup and Restore CRDs to enhance wildcard namespace handling

- Renamed fields in `BackupStatus` and `RestoreStatus` from `ExpandedIncludedNamespaces` and `ExpandedExcludedNamespaces` to `IncludeWildcardMatches` and `ExcludeWildcardMatches` for clarity.
- Introduced a new field `WildcardResult` to record the final namespaces after applying wildcard logic.
- Updated the `DeepCopyInto` methods to accommodate the new field names and ensure proper data handling.
- Enhanced comments to reflect the changes and improve documentation clarity.

These updates improve the maintainability and clarity of the CRDs, aligning with recent enhancements in wildcard handling.

Signed-off-by: Joseph <jvaikath@redhat.com>

* Enhance wildcard namespace handling in Backup and Restore processes

- Updated `BackupRequest` and `Restore` status structures to include a new field `WildcardResult`, which captures the final list of namespaces after applying wildcard logic.
- Renamed existing fields to `IncludeWildcardMatches` and `ExcludeWildcardMatches` for improved clarity.
- Enhanced logging to provide detailed insights into the expanded namespaces and final results during backup and restore operations.
- Introduced a new utility function `GetWildcardResult` to streamline the selection of namespaces based on include/exclude criteria.

These changes improve the clarity and functionality of namespace selection in both backup and restore processes, aligning with recent enhancements in wildcard handling.

Signed-off-by: Joseph <jvaikath@redhat.com>

* Refactor namespace wildcard expansion logic in restore process

- Moved the wildcard expansion logic into a dedicated method, `expandNamespaceWildcards`, improving code organization and readability.
- Enhanced error handling and logging to provide detailed insights into the expanded namespaces during the restore operation.
- Updated the restore context with expanded namespace patterns and final results, ensuring clarity in the restore status.

These changes improve the maintainability and clarity of the restore process, aligning with recent enhancements in wildcard handling.

Signed-off-by: Joseph <jvaikath@redhat.com>

* Add checks for "*" in exclude

Signed-off-by: Joseph <jvaikath@redhat.com>

* Rebase

Signed-off-by: Joseph <jvaikath@redhat.com>

* Create NamespaceIncludesExcludes to get full NS listing for backup w/

Signed-off-by: Scott Seago <sseago@redhat.com>
Signed-off-by: Joseph <jvaikath@redhat.com>

* Add new NamespaceIncludesExcludes struct

Signed-off-by: Joseph <jvaikath@redhat.com>

* Move namespace expansion logic

Signed-off-by: Joseph <jvaikath@redhat.com>

* Update backup status with expansion

Signed-off-by: Joseph <jvaikath@redhat.com>

* Wildcard status update

Signed-off-by: Joseph <jvaikath@redhat.com>

* Skip ns check if wildcard expansion

Signed-off-by: Joseph <jvaikath@redhat.com>

* Move wildcard expansion to getResourceItems

Signed-off-by: Joseph <jvaikath@redhat.com>

* lint

Signed-off-by: Joseph <jvaikath@redhat.com>

* Changelog

Signed-off-by: Joseph <jvaikath@redhat.com>

* linting issues

Signed-off-by: Joseph <jvaikath@redhat.com>

* Remove wildcard restore to check if tests pass

Signed-off-by: Joseph <jvaikath@redhat.com>

* Fix namespace mapping test bug from lint fix

The previous commit (0a4aabcf4) attempted to fix linting issues by
using strings.Builder, but incorrectly wrote commas to a separate
builder and concatenated them at the end instead of between namespace
mappings.

This caused the namespace mapping string to be malformed:
  Before: ns-1:ns-1-mapped,ns-2:ns-2-mapped
  Bug:    ns-1:ns-1-mappedns-2:ns-2-mapped,,

The malformed string was parsed as a single mapping with an invalid
namespace name containing a colon, causing Kubernetes to reject it:
  "ns-1-mappedns-2:ns-2-mapped" is invalid

Fix by properly using strings.Builder to construct the mapping string
with commas between entries, addressing both the linting concern and
the functional bug.

Fixes the MultiNamespacesMappingResticTest and
MultiNamespacesMappingSnapshotTest failures.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Signed-off-by: Joseph <jvaikath@redhat.com>

* Fix wildcard namespace expansion edge cases

This commit fixes two bugs in the wildcard namespace expansion feature:

1. Empty wildcard results: When a wildcard pattern (e.g., "invalid*")
   matched no namespaces, the backup would incorrectly back up ALL
   namespaces instead of backing up nothing. This was because the empty
   includes list was indistinguishable from "no filter specified".

   Fix: Added wildcardExpanded flag to NamespaceIncludesExcludes to
   track when wildcard expansion has occurred. When true and the
   includes list is empty, ShouldInclude now correctly returns false.

2. Premature namespace filtering: An earlier attempt to fix bug #1
   filtered namespaces too early in collectNamespaces, breaking
   LabelSelector tests where namespaces should be included based on
   resources within them matching the label selector.

   Fix: Removed the premature filtering and rely on the existing
   filterNamespaces call at the end of getAllItems, which correctly
   handles both wildcard expansion and label selector scenarios.

The fixes ensure:
- Wildcard patterns matching nothing result in empty backups
- Label selectors still work correctly (namespace included if any
  resource in it matches the selector)
- State is preserved across multiple ResolveNamespaceList calls

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Signed-off-by: Joseph <jvaikath@redhat.com>

* Run wildcard expansion during backup processing

Signed-off-by: Joseph <jvaikath@redhat.com>

* Lint fix

Signed-off-by: Joseph <jvaikath@redhat.com>

* Improve coverage

Signed-off-by: Joseph <jvaikath@redhat.com>

* gofmt fix

Signed-off-by: Joseph <jvaikath@redhat.com>

* Add wildcard details to describe backup status

Signed-off-by: Joseph <jvaikath@redhat.com>

* Revert "Remove wildcard restore to check if tests pass"

This reverts commit 4e22c2af855b71447762cb0a9fab7e7049f38a5f.

Signed-off-by: Joseph <jvaikath@redhat.com>

* Add restore describe for wildcard namespaces Revert restore wildcard removal

Signed-off-by: Joseph <jvaikath@redhat.com>

* Add coverage

Signed-off-by: Joseph <jvaikath@redhat.com>

* Lint

Signed-off-by: Joseph <jvaikath@redhat.com>

* Remove unintentional changes

Signed-off-by: Joseph <jvaikath@redhat.com>

* Remove wildcard status fields and mentionsRemove usage of wildcard fields for backup and restore status.

Signed-off-by: Joseph <jvaikath@redhat.com>

* Remove status update changelog line

Signed-off-by: Joseph <jvaikath@redhat.com>

* Rename getNamespaceIncludesExcludes
Signed-off-by: Scott Seago <sseago@redhat.com>

Signed-off-by: Scott Seago <sseago@redhat.com>

* Rewrite brace pattern validation

Signed-off-by: Joseph <jvaikath@redhat.com>

* Different var for internal loop

Signed-off-by: Joseph <jvaikath@redhat.com>

---------

Signed-off-by: Joseph <jvaikath@redhat.com>
Signed-off-by: Scott Seago <sseago@redhat.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Scott Seago <sseago@redhat.com>
Co-authored-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
2025-12-02 12:28:03 -05:00
0xLeo258
1ebe357d18 Add built-in mutex for SynchronizedVSList && Update unit tests
Signed-off-by: 0xLeo258 <noixe0312@gmail.com>
2025-09-20 09:13:07 +08:00
0xLeo258
f2a27c3864 fix9247: Protect VolumeSnapshot field
Signed-off-by: 0xLeo258 <noixe0312@gmail.com>
2025-09-20 09:13:07 +08:00
Scott Seago
fcfb2fd9ee Implement parallel ItemBlock processing via backup_controller goroutines
Signed-off-by: Scott Seago <sseago@redhat.com>
2025-02-12 12:03:37 -05:00
Scott Seago
015b1e69f6 Make BackedUpItems thread safe
Signed-off-by: Scott Seago <sseago@redhat.com>
2024-12-03 15:23:45 -05:00
Scott Seago
9d6f4d2db5 ItemBlock model and phase 1 (single-thread) workflow changes
Signed-off-by: Scott Seago <sseago@redhat.com>
2024-09-03 19:04:18 -04:00
Daniel Jiang
0a280e5786 Track and persist restore volume info
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2024-04-11 17:32:18 +08:00
Xun Jiang
efb94ae610 Refactor the native snapshot definition code.
Signed-off-by: Xun Jiang <blackpigletbruce@gmail.com>
2024-03-20 15:38:07 +08:00
Xun Jiang
c77bec73bb Move VolumesInformation to an independant package.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-12-04 08:33:37 +08:00
Xun Jiang
ca97248f2a Use VolumeInfo to help restore the PV.
Add VolumeInfo for left PVs during backup.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-12-04 08:33:37 +08:00
Xun Jiang
7f52321772 Generate VolumeInfo.
Remove CSI VolumeSnapshot listter and the informer.
Add download the VolumeInfos metadata for backup.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-11-22 09:40:38 +08:00
Tiger Kaovilai
9311a4269b refactor backup snapshot status updates into UpdateBackupSnapshotsStatus() and run in backup_finalizer_controller
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
2023-11-02 15:30:35 -04:00
Daniel Jiang
2548b20db9 Track the skipped PV in a backup and print the summary in backup log (#6496)
Partially address the requirements in #5834

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2023-07-20 16:13:48 +08:00
qiuming
086dbd344f Handle backup of volume by resource policies (#5901)
* Handle backup of volume by resource policies

Signed-off-by: Ming <mqiu@vmware.com>
2023-03-21 14:39:25 +08:00
Daniel Jiang
458560795b Remove Reference of itemsnapshotter (#5997)
It will be removed as we choose to use BIAv2 for progress monitoring.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2023-03-21 09:05:29 +08:00
Xun Jiang
a1e4f54488 Add new resource filters can separate cluster and namespace scope resources.
Signed-off-by: Xun Jiang <blackpiglet@gmail.com>
2023-03-14 07:07:22 +00:00
Scott Seago
c3d1d83da5 BIAv2 async operations controller work
Signed-off-by: Scott Seago <sseago@redhat.com>
2023-03-01 11:49:17 -05:00
Scott Seago
9a54142257 BackupItemAction v2 API implementation
Signed-off-by: Scott Seago <sseago@redhat.com>
2023-01-09 18:36:47 -05:00
Xun Jiang/Bruce Jiang
ce247a3d90 Remove snapshot related lister, informer and client from backup controller. (#5299)
Signed-off-by: Xun Jiang <blackpiglet@gmail.com>
2022-09-23 15:44:39 +08:00
Xun Jiang
9aa4e9e860 Update according to comments. Remove csiVolumeSnapshotsStorageTotal related code.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-04-15 08:05:56 +00:00
Xun Jiang
4daeec7ab9 Update CRD and GRPC.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-04-09 12:35:09 +00:00
Xun Jiang
368a1ddf3c Add CSI VolumeSnapshot related metrics.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2022-04-09 00:57:28 +08:00
David L. Smith-Uchida
5677e04bb1 Consolidated code for resolving actions and plugins into ActionResolver (#4410)
* Consolidated code for resolving actions and plugins into ActionResolver.  Added BackupWithResolvers and
RestoreWithResolvers.  Introduces ItemSnapshooterResolver to bring ItemSnapshotter plugins into backup and
restore.  ItemSnapshotters are not used yet.

Added action_resolver_test

Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>

* Addressed review comments

Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>
2021-12-10 12:53:47 -05:00
Nolan Brubaker
bbcbde084d Create hook package (#2734)
* Move pkg/backup/item_hook_handler to internal/hoo

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add internal packages to test target

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-07-22 14:26:14 -07:00
Steve Kriss
48792ece1f Updates for org move to vmware-tanzu (#1920)
* update import paths to github.com/vmware-tanzu/...

Signed-off-by: Steve Kriss <krisss@vmware.com>

* update other GH org refs to vmware-tanzu

Signed-off-by: Steve Kriss <krisss@vmware.com>

* site and docs: update GH org to vmware-tanzu

Signed-off-by: Steve Kriss <krisss@vmware.com>

* update travis badge links on docs readmes

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-09-30 17:26:56 -04:00
Adnan Abdulhussein
faef20d45a ensure backup resource list per gvk is sorted
Signed-off-by: Adnan Abdulhussein <aadnan@vmware.com>
2019-08-08 13:53:40 -07:00
Adnan Abdulhussein
07525bd593 store backup resource list metadata in object storage (#1709)
* move backedUpItems to pkg/backup.Request struct

Signed-off-by: Adnan Abdulhussein <aadnan@vmware.com>

* construct resource itemKey field from gvk

Signed-off-by: Adnan Abdulhussein <aadnan@vmware.com>

* store backup resource list metadata in object storage

Signed-off-by: Adnan Abdulhussein <aadnan@vmware.com>

* remove debug log

Signed-off-by: Adnan Abdulhussein <aadnan@vmware.com>

* fix formatting

Signed-off-by: Adnan Abdulhussein <aadnan@vmware.com>

* add missing license blocks and split BackupInfo struct lines

Signed-off-by: Adnan Abdulhussein <aadnan@vmware.com>

* add test for checking BackedUpItems matches tarball contents

Signed-off-by: Adnan Abdulhussein <aadnan@vmware.com>

* add comment to explain test

Signed-off-by: Adnan Abdulhussein <aadnan@vmware.com>
2019-08-05 11:15:55 -06:00
KubeKween
3b9af8c654 Store PodVolumeBackups in obj storage & use as source of truth (#1577)
* Store PodVolumeBackups in object storage

Signed-off-by: Carlisia <carlisiac@vmware.com>
2019-07-24 15:51:20 -04:00
Nolan Brubaker
43714caaec Rename Ark to Velero!!!
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2019-02-04 17:35:22 -05:00
Steve Kriss
da9ed38c63 store volume snapshot info as JSON in backup storage
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-17 13:39:49 -06:00
Steve Kriss
df07b7dc9f update backup code to work with volume snapshot locations
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-10-17 13:36:50 -06:00