mirror of
https://github.com/vmware-tanzu/velero.git
synced 2026-01-05 13:05:17 +00:00
Enhance wildcard namespace support in backup and restore design document
- Updated the abstract to clarify the current limitations of namespace specifications in Velero. - Expanded the goals section to include specific objectives for implementing wildcard patterns in `--include-namespaces` and `--exclude-namespaces`. - Detailed the high-level design and implementation steps, including the addition of new status fields in the backup CRD and the creation of a utility package for wildcard expansion. - Addressed backward compatibility and known limitations regarding the use of wildcards alongside the existing "*" character. This update aims to provide a comprehensive overview of the proposed changes for improved namespace selection flexibility. Signed-off-by: Joseph <jvaikath@redhat.com>
This commit is contained in:
Joseph
@@ -1,79 +1,164 @@
|
|||||||
|
|
||||||
# Wildcard namespace includes/excludes support for backups and restores
|
# Wildcard Namespace Includes/Excludes Support for Backups and Restores
|
||||||
|
|
||||||
## Abstract
|
## Abstract
|
||||||
One to two sentences that describes the goal of this proposal and the problem being solved by the proposed change.
|
|
||||||
The reader should be able to tell by the title, and the opening paragraph, if this document is relevant to them.
|
|
||||||
|
|
||||||
Velero currently does not have any support for wildcard characters in the namespace spec.
|
Velero currently does not support wildcard characters in namespace specifications, requiring all namespaces to be specified as string literals. The only exception is the standalone "*" character, which includes all namespaces and ignores excludes.
|
||||||
It fully expects the namespaces to be string literals.
|
|
||||||
|
|
||||||
The only and notable exception is the "*" character by it's lonesome, which acts as an include all and ignore excludes option.
|
This document details the approach to implementing wildcard namespace support for `--include-namespaces` and `--exclude-namespaces` flags, while preserving the existing "*" behavior for backward compatibility.
|
||||||
Internally Velero treats not specifying anything as the "*" case.
|
|
||||||
|
|
||||||
This document details the approach to implementing wildcard namespaces, while keeping the "*" characters purpose intact for legacy purposes.
|
|
||||||
|
|
||||||
## Background
|
## Background
|
||||||
This was raised in Issue [#1874](https://github.com/vmware-tanzu/velero/issues/1874)
|
|
||||||
|
|
||||||
|
This feature was requested in Issue [#1874](https://github.com/vmware-tanzu/velero/issues/1874) to provide more flexible namespace selection using wildcard patterns.
|
||||||
|
|
||||||
## Goals
|
## Goals
|
||||||
- A short list of things which will be accomplished by implementing this proposal.
|
|
||||||
- Two things is ok.
|
|
||||||
- Three is pushing it.
|
|
||||||
- More than three goals suggests that the proposal's scope is too large.
|
|
||||||
|
|
||||||
- Add support for wildcard namespaces in --include-namespaces and --exclude-namespaces
|
- Add support for wildcard patterns in `--include-namespaces` and `--exclude-namespaces` flags
|
||||||
- Ensure legacy "*" support is not affected
|
- Ensure legacy "*" behavior remains unchanged for backward compatibility
|
||||||
|
|
||||||
## Non Goals
|
## Non-Goals
|
||||||
- A short list of items which are:
|
|
||||||
- a. out of scope
|
|
||||||
- b. follow on items which are deliberately excluded from this proposal.
|
|
||||||
|
|
||||||
- Completely rethinking the way "*" is treated and allowing it to work with wildcard excludes.
|
- Completely rethinking the way "*" is treated and allowing it to work with wildcard excludes
|
||||||
|
- Supporting complex regex patterns beyond basic glob patterns
|
||||||
|
|
||||||
|
|
||||||
## High-Level Design
|
## High-Level Design
|
||||||
One to two paragraphs that describe the high level changes that will be made to implement this proposal.
|
|
||||||
|
|
||||||
Points of interest are two funcs within the utility layer, in file `velero/pkg/backup/item_collector.go`
|
The wildcard expansion implementation focuses on two key functions in `pkg/backup/item_collector.go`:
|
||||||
- [collectNamespaces](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L742)
|
|
||||||
- [getNamespacesToList](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L638)
|
|
||||||
|
|
||||||
collectNamespaces gets all the active namespaces and matches it against the user spec for included namespaces (r.backupRequest.Backup.Spec.IncludedNamespaces)
|
- [`collectNamespaces`](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L742) - Retrieves all active namespaces and processes include/exclude filters
|
||||||
This is an ideal point where wildcard expansion can take place.
|
- [`getNamespacesToList`](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L638) - Resolves namespace includes/excludes to final list
|
||||||
The implementation would mean that just like "*", namespaces with wildcard symbols would also be passed through without an existence check.
|
|
||||||
The resolved namespaces are stored in new status fields on the backup.
|
The `collectNamespaces` function is the ideal integration point because it:
|
||||||
|
- Already retrieves all active namespaces from the cluster
|
||||||
|
- Processes the user-specified namespace filters
|
||||||
|
- Can expand wildcard patterns against the complete namespace list
|
||||||
|
- Stores the resolved namespaces in new backup status fields for visibility
|
||||||
|
|
||||||
|
This approach ensures wildcard namespaces are handled consistently with the existing "*" behavior, bypassing individual namespace existence checks.
|
||||||
|
|
||||||
## Detailed Design
|
## Detailed Design
|
||||||
A detailed design describing how the changes to the product should be made.
|
|
||||||
|
|
||||||
The names of types, fields, interfaces, and methods should be agreed on here, not debated in code review.
|
The implementation involves four main components that can be developed incrementally:
|
||||||
The same applies to changes in CRDs, YAML examples, and so on.
|
|
||||||
|
|
||||||
Ideally the changes should be made in sequence so that the work required to implement this design can be done incrementally, possibly in parallel.
|
### Add new status fields to the backup CRD to store expanded wildcard namespaces
|
||||||
|
|
||||||
1. Add new status fields to the backup CRD to store expanded wildcard namespaces
|
```go
|
||||||
|
// BackupStatus captures the current status of a Velero backup.
|
||||||
|
type BackupStatus struct {
|
||||||
|
// ... existing fields ...
|
||||||
|
|
||||||
|
// ExpandedIncludedNamespaces records the expanded include wildcard namespaces
|
||||||
|
// +optional
|
||||||
|
// +nullable
|
||||||
|
ExpandedIncludedNamespaces []string `json:"expandedIncludedNamespaces,omitempty"`
|
||||||
|
|
||||||
|
// ExpandedExcludedNamespaces records the expanded exclude wildcard namespaces
|
||||||
|
// +optional
|
||||||
|
// +nullable
|
||||||
|
ExpandedExcludedNamespaces []string `json:"expandedExcludedNamespaces,omitempty"`
|
||||||
|
|
||||||
|
// ... other fields ...
|
||||||
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
```
|
**Implementation**: Added status fields `ExpandedIncludedNamespaces` and `ExpandedExcludedNamespaces` to `pkg/apis/velero/v1/backup_types.go` to track the resolved namespace lists after wildcard expansion.
|
||||||
2. Create a util package for wildcard expansion
|
|
||||||
|
|
||||||
3. If required, expand wildcards and replace the request's includes and excludes with expanded namespaces
|
### Create a util package for wildcard expansion
|
||||||
4. Populate the expanded namespace status field with the namespaces.
|
|
||||||
|
**Implementation**: Created `pkg/util/wildcard/expand.go` package containing:
|
||||||
|
|
||||||
|
- `ShouldExpandWildcards(includes, excludes []string) bool` - Determines if wildcard expansion is needed (excludes simple "*" case)
|
||||||
|
- `ExpandWildcards(activeNamespaces, includes, excludes []string) ([]string, []string, error)` - Main expansion function
|
||||||
|
- `containsWildcardPattern(pattern string) bool` - Detects wildcard patterns (`*`, `?`, `[abc]`, `{a,b,c}`)
|
||||||
|
- `validateWildcardPatterns(patterns []string) error` - Validates patterns and rejects unsupported regex symbols
|
||||||
|
- Uses `github.com/gobwas/glob` library for glob pattern matching
|
||||||
|
|
||||||
|
**Supported patterns**:
|
||||||
|
- `*` (any characters)
|
||||||
|
- `?` (single character)
|
||||||
|
- `[abc]` (character classes)
|
||||||
|
- `{a,b,c}` (alternatives)
|
||||||
|
|
||||||
|
**Unsupported**: Regex symbols `|()`, consecutive asterisks `**`
|
||||||
|
|
||||||
|
### If required, expand wildcards and replace the request's includes and excludes with expanded namespaces
|
||||||
|
|
||||||
|
**Implementation**: In `pkg/backup/item_collector.go`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// collectNamespaces function (line 748-803)
|
||||||
|
if wildcard.ShouldExpandWildcards(namespaceSelector.GetIncludes(), namespaceSelector.GetExcludes()) {
|
||||||
|
if err := r.expandNamespaceWildcards(activeNamespacesList, namespaceSelector); err != nil {
|
||||||
|
return nil, errors.WithMessage(err, "failed to expand namespace wildcard patterns")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The expansion occurs when collecting namespaces, after retrieving all active namespaces from the cluster. The `expandNamespaceWildcards` method:
|
||||||
|
- Calls `wildcard.ExpandWildcards()` with active namespaces and original patterns
|
||||||
|
- Updates the namespace selector with expanded results using `SetIncludes()` and `SetExcludes()`
|
||||||
|
- Preserves backward compatibility by skipping expansion for simple "*" pattern
|
||||||
|
|
||||||
|
**Performance Improvement**: As part of this implementation, active namespaces are stored in a hashset rather than being iterated for each resolved/literal namespace check. This eliminates a [nested loop anti-pattern](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L767) and improves performance.
|
||||||
|
|
||||||
|
### Populate the expanded namespace status field with the namespaces
|
||||||
|
|
||||||
|
**Implementation**: In `expandNamespaceWildcards` function (line 889-891):
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Record the expanded wildcard includes/excludes in the request status
|
||||||
|
r.backupRequest.Status.ExpandedIncludedNamespaces = expandedIncludes
|
||||||
|
r.backupRequest.Status.ExpandedExcludedNamespaces = expandedExcludes
|
||||||
|
```
|
||||||
|
|
||||||
|
The status fields are populated immediately after successful wildcard expansion, providing visibility into which namespaces were actually matched by the wildcard patterns.
|
||||||
|
|
||||||
## Alternatives Considered
|
## Alternatives Considered
|
||||||
If there are alternative high level or detailed designs that were not pursued they should be called out here with a brief explanation of why they were not pursued.
|
|
||||||
|
Several implementation approaches were considered for the wildcard expansion logic:
|
||||||
|
|
||||||
|
### 1. Wildcard expansion in `getNamespacesToList`
|
||||||
|
|
||||||
|
This approach was ruled out because:
|
||||||
|
- The `collectNamespaces` function encounters wildcard patterns first in the processing flow
|
||||||
|
- `collectNamespaces` already has access to the complete list of active namespaces, eliminating the need for additional API calls
|
||||||
|
- Placing the logic in `getNamespacesToList` would require redundant namespace retrieval
|
||||||
|
|
||||||
|
### 2. Client-side wildcard expansion
|
||||||
|
|
||||||
|
Expanding wildcards on the CLI side was considered but rejected because:
|
||||||
|
- It would only work for command-line initiated backups
|
||||||
|
- Scheduled backups would not benefit from this approach
|
||||||
|
- The server-side approach provides consistent behavior across all backup initiation methods
|
||||||
|
|
||||||
## Security Considerations
|
## Security Considerations
|
||||||
If this proposal has an impact to the security of the product, its users, or data stored or transmitted via the product, they must be addressed here.
|
|
||||||
|
This feature does not introduce any security vulnerabilities as it only affects namespace selection logic within the existing backup authorization framework.
|
||||||
|
|
||||||
## Compatibility
|
## Compatibility
|
||||||
A discussion of any compatibility issues that need to be considered
|
|
||||||
|
### Backward Compatibility
|
||||||
|
|
||||||
|
The implementation maintains full backward compatibility with existing behavior:
|
||||||
|
- The standalone "*" character continues to work as before (include all namespaces, ignore excludes)
|
||||||
|
- Existing backup configurations remain unaffected
|
||||||
|
|
||||||
|
### Known Limitations
|
||||||
|
|
||||||
|
1. **Mixed "*" and wildcard usage**: When using the standalone "*" in includes, the legacy implementation takes precedence and wildcard expansion is skipped. This means you cannot combine "*" (all namespaces) in includes with wildcard patterns in excludes.
|
||||||
|
|
||||||
|
2. **Selective exclusion limitation**: The current design does not support selective pattern-based exclusion when including all namespaces via "*".
|
||||||
|
|
||||||
## Implementation
|
## Implementation
|
||||||
A description of the implementation, timelines, and any resources that have agreed to contribute.
|
|
||||||
|
The implementation follows the detailed design outlined above, with the following timeline:
|
||||||
|
|
||||||
|
1. **Phase 1**: Add backup CRD status fields for expanded namespaces
|
||||||
|
2. **Phase 2**: Implement wildcard utility package with validation
|
||||||
|
3. **Phase 3**: Integrate expansion logic into backup item collection
|
||||||
|
4. **Phase 4**: Add status field population and logging
|
||||||
|
|
||||||
## Open Issues
|
## Open Issues
|
||||||
A discussion of issues relating to this proposal for which the author does not know the solution. This section may be omitted if there are none.
|
|
||||||
|
**Restore Integration**: Currently investigating how restore operations should handle wildcard-expanded backups. The proposed approach is to apply wildcard patterns against the expanded namespace list stored in the backup's status fields, ensuring consistency between backup and restore operations.
|
||||||
Reference in New Issue
Block a user