From 4c1457c3181eee14181362debfd5ba2d760a4d1c Mon Sep 17 00:00:00 2001 From: Joseph Date: Fri, 1 Aug 2025 13:51:44 -0400 Subject: [PATCH] Enhance wildcard namespace support in backup and restore design document - Updated the abstract to clarify the current limitations of namespace specifications in Velero. - Expanded the goals section to include specific objectives for implementing wildcard patterns in `--include-namespaces` and `--exclude-namespaces`. - Detailed the high-level design and implementation steps, including the addition of new status fields in the backup CRD and the creation of a utility package for wildcard expansion. - Addressed backward compatibility and known limitations regarding the use of wildcards alongside the existing "*" character. This update aims to provide a comprehensive overview of the proposed changes for improved namespace selection flexibility. Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 171 +++++++++++++++----- 1 file changed, 128 insertions(+), 43 deletions(-) diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md index b5dfd3618..fb3426719 100644 --- a/design/wildcard-namespace-support-design.md +++ b/design/wildcard-namespace-support-design.md @@ -1,79 +1,164 @@ -# Wildcard namespace includes/excludes support for backups and restores +# Wildcard Namespace Includes/Excludes Support for Backups and Restores ## Abstract -One to two sentences that describes the goal of this proposal and the problem being solved by the proposed change. -The reader should be able to tell by the title, and the opening paragraph, if this document is relevant to them. -Velero currently does not have any support for wildcard characters in the namespace spec. -It fully expects the namespaces to be string literals. +Velero currently does not support wildcard characters in namespace specifications, requiring all namespaces to be specified as string literals. The only exception is the standalone "*" character, which includes all namespaces and ignores excludes. -The only and notable exception is the "*" character by it's lonesome, which acts as an include all and ignore excludes option. -Internally Velero treats not specifying anything as the "*" case. - -This document details the approach to implementing wildcard namespaces, while keeping the "*" characters purpose intact for legacy purposes. +This document details the approach to implementing wildcard namespace support for `--include-namespaces` and `--exclude-namespaces` flags, while preserving the existing "*" behavior for backward compatibility. ## Background -This was raised in Issue [#1874](https://github.com/vmware-tanzu/velero/issues/1874) +This feature was requested in Issue [#1874](https://github.com/vmware-tanzu/velero/issues/1874) to provide more flexible namespace selection using wildcard patterns. ## Goals -- A short list of things which will be accomplished by implementing this proposal. -- Two things is ok. -- Three is pushing it. -- More than three goals suggests that the proposal's scope is too large. -- Add support for wildcard namespaces in --include-namespaces and --exclude-namespaces -- Ensure legacy "*" support is not affected +- Add support for wildcard patterns in `--include-namespaces` and `--exclude-namespaces` flags +- Ensure legacy "*" behavior remains unchanged for backward compatibility -## Non Goals -- A short list of items which are: -- a. out of scope -- b. follow on items which are deliberately excluded from this proposal. +## Non-Goals -- Completely rethinking the way "*" is treated and allowing it to work with wildcard excludes. +- Completely rethinking the way "*" is treated and allowing it to work with wildcard excludes +- Supporting complex regex patterns beyond basic glob patterns ## High-Level Design -One to two paragraphs that describe the high level changes that will be made to implement this proposal. -Points of interest are two funcs within the utility layer, in file `velero/pkg/backup/item_collector.go` -- [collectNamespaces](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L742) -- [getNamespacesToList](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L638) +The wildcard expansion implementation focuses on two key functions in `pkg/backup/item_collector.go`: -collectNamespaces gets all the active namespaces and matches it against the user spec for included namespaces (r.backupRequest.Backup.Spec.IncludedNamespaces) -This is an ideal point where wildcard expansion can take place. -The implementation would mean that just like "*", namespaces with wildcard symbols would also be passed through without an existence check. -The resolved namespaces are stored in new status fields on the backup. +- [`collectNamespaces`](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L742) - Retrieves all active namespaces and processes include/exclude filters +- [`getNamespacesToList`](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L638) - Resolves namespace includes/excludes to final list + +The `collectNamespaces` function is the ideal integration point because it: +- Already retrieves all active namespaces from the cluster +- Processes the user-specified namespace filters +- Can expand wildcard patterns against the complete namespace list +- Stores the resolved namespaces in new backup status fields for visibility + +This approach ensures wildcard namespaces are handled consistently with the existing "*" behavior, bypassing individual namespace existence checks. ## Detailed Design -A detailed design describing how the changes to the product should be made. -The names of types, fields, interfaces, and methods should be agreed on here, not debated in code review. -The same applies to changes in CRDs, YAML examples, and so on. +The implementation involves four main components that can be developed incrementally: -Ideally the changes should be made in sequence so that the work required to implement this design can be done incrementally, possibly in parallel. +### Add new status fields to the backup CRD to store expanded wildcard namespaces -1. Add new status fields to the backup CRD to store expanded wildcard namespaces +```go +// BackupStatus captures the current status of a Velero backup. +type BackupStatus struct { + // ... existing fields ... + + // ExpandedIncludedNamespaces records the expanded include wildcard namespaces + // +optional + // +nullable + ExpandedIncludedNamespaces []string `json:"expandedIncludedNamespaces,omitempty"` + + // ExpandedExcludedNamespaces records the expanded exclude wildcard namespaces + // +optional + // +nullable + ExpandedExcludedNamespaces []string `json:"expandedExcludedNamespaces,omitempty"` + + // ... other fields ... +} ``` -``` -2. Create a util package for wildcard expansion +**Implementation**: Added status fields `ExpandedIncludedNamespaces` and `ExpandedExcludedNamespaces` to `pkg/apis/velero/v1/backup_types.go` to track the resolved namespace lists after wildcard expansion. -3. If required, expand wildcards and replace the request's includes and excludes with expanded namespaces -4. Populate the expanded namespace status field with the namespaces. +### Create a util package for wildcard expansion + +**Implementation**: Created `pkg/util/wildcard/expand.go` package containing: + +- `ShouldExpandWildcards(includes, excludes []string) bool` - Determines if wildcard expansion is needed (excludes simple "*" case) +- `ExpandWildcards(activeNamespaces, includes, excludes []string) ([]string, []string, error)` - Main expansion function +- `containsWildcardPattern(pattern string) bool` - Detects wildcard patterns (`*`, `?`, `[abc]`, `{a,b,c}`) +- `validateWildcardPatterns(patterns []string) error` - Validates patterns and rejects unsupported regex symbols +- Uses `github.com/gobwas/glob` library for glob pattern matching + +**Supported patterns**: +- `*` (any characters) +- `?` (single character) +- `[abc]` (character classes) +- `{a,b,c}` (alternatives) + +**Unsupported**: Regex symbols `|()`, consecutive asterisks `**` + +### If required, expand wildcards and replace the request's includes and excludes with expanded namespaces + +**Implementation**: In `pkg/backup/item_collector.go`: + +```go +// collectNamespaces function (line 748-803) +if wildcard.ShouldExpandWildcards(namespaceSelector.GetIncludes(), namespaceSelector.GetExcludes()) { + if err := r.expandNamespaceWildcards(activeNamespacesList, namespaceSelector); err != nil { + return nil, errors.WithMessage(err, "failed to expand namespace wildcard patterns") + } +} +``` + +The expansion occurs when collecting namespaces, after retrieving all active namespaces from the cluster. The `expandNamespaceWildcards` method: +- Calls `wildcard.ExpandWildcards()` with active namespaces and original patterns +- Updates the namespace selector with expanded results using `SetIncludes()` and `SetExcludes()` +- Preserves backward compatibility by skipping expansion for simple "*" pattern + +**Performance Improvement**: As part of this implementation, active namespaces are stored in a hashset rather than being iterated for each resolved/literal namespace check. This eliminates a [nested loop anti-pattern](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L767) and improves performance. + +### Populate the expanded namespace status field with the namespaces + +**Implementation**: In `expandNamespaceWildcards` function (line 889-891): + +```go +// Record the expanded wildcard includes/excludes in the request status +r.backupRequest.Status.ExpandedIncludedNamespaces = expandedIncludes +r.backupRequest.Status.ExpandedExcludedNamespaces = expandedExcludes +``` + +The status fields are populated immediately after successful wildcard expansion, providing visibility into which namespaces were actually matched by the wildcard patterns. ## Alternatives Considered -If there are alternative high level or detailed designs that were not pursued they should be called out here with a brief explanation of why they were not pursued. + +Several implementation approaches were considered for the wildcard expansion logic: + +### 1. Wildcard expansion in `getNamespacesToList` + +This approach was ruled out because: +- The `collectNamespaces` function encounters wildcard patterns first in the processing flow +- `collectNamespaces` already has access to the complete list of active namespaces, eliminating the need for additional API calls +- Placing the logic in `getNamespacesToList` would require redundant namespace retrieval + +### 2. Client-side wildcard expansion + +Expanding wildcards on the CLI side was considered but rejected because: +- It would only work for command-line initiated backups +- Scheduled backups would not benefit from this approach +- The server-side approach provides consistent behavior across all backup initiation methods ## Security Considerations -If this proposal has an impact to the security of the product, its users, or data stored or transmitted via the product, they must be addressed here. + +This feature does not introduce any security vulnerabilities as it only affects namespace selection logic within the existing backup authorization framework. ## Compatibility -A discussion of any compatibility issues that need to be considered + +### Backward Compatibility + +The implementation maintains full backward compatibility with existing behavior: +- The standalone "*" character continues to work as before (include all namespaces, ignore excludes) +- Existing backup configurations remain unaffected + +### Known Limitations + +1. **Mixed "*" and wildcard usage**: When using the standalone "*" in includes, the legacy implementation takes precedence and wildcard expansion is skipped. This means you cannot combine "*" (all namespaces) in includes with wildcard patterns in excludes. + +2. **Selective exclusion limitation**: The current design does not support selective pattern-based exclusion when including all namespaces via "*". ## Implementation -A description of the implementation, timelines, and any resources that have agreed to contribute. + +The implementation follows the detailed design outlined above, with the following timeline: + +1. **Phase 1**: Add backup CRD status fields for expanded namespaces +2. **Phase 2**: Implement wildcard utility package with validation +3. **Phase 3**: Integrate expansion logic into backup item collection +4. **Phase 4**: Add status field population and logging ## Open Issues -A discussion of issues relating to this proposal for which the author does not know the solution. This section may be omitted if there are none. + +**Restore Integration**: Currently investigating how restore operations should handle wildcard-expanded backups. The proposed approach is to apply wildcard patterns against the expanded namespace list stored in the backup's status fields, ensuring consistency between backup and restore operations. \ No newline at end of file