From 571a816a61f4b3e50ca357db63cd9ab2cd84b1b1 Mon Sep 17 00:00:00 2001 From: Joseph Date: Wed, 2 Jul 2025 11:45:57 -0400 Subject: [PATCH 01/13] Add design doc Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 146 ++++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 design/wildcard-namespace-support-design.md diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md new file mode 100644 index 000000000..b7b1b01ba --- /dev/null +++ b/design/wildcard-namespace-support-design.md @@ -0,0 +1,146 @@ +# Wildcard Namespace Support Design + +## Abstract +This proposal introduces wildcard pattern support for namespace inclusion and exclusion in Velero backups (e.g., `prod-*`, `*-staging`). +The implementation uses lazy evaluation within the existing `ShouldInclude()` method to resolve wildcards on-demand with request-scoped caching. +Based on [Issue #1874](https://github.com/vmware-tanzu/velero/issues/1874). + +## Background +- Currently, Velero users must explicitly list each namespace for backup operations +- In environments with many namespaces following naming conventions (e.g., `prod-app`, `prod-db`, `prod-cache`), this becomes: + - Cumbersome to maintain + - Error-prone to manage +- Users have requested wildcard support to enable patterns like `--include-namespaces "prod-*"` + +## Goals +- Enable wildcard pattern support for namespace includes and excludes in Velero backup specifications +- Maintain optimal performance with lazy evaluation and request-scoped caching +- Preserve original wildcard patterns in backup specifications for audit and readability purposes + +## Non Goals +- Support for complex regex patterns beyond basic glob-style wildcards (`*`) +- Persistent caching of namespace resolution across backup requests +- Real-time namespace discovery that changes during backup execution + +## High-Level Design + +**Core Approach:** We're making the existing concrete type (`*IncludesExcludes`) polymorphic so we can substitute our new lazy evaluation type (`*LazyNamespaceIncludesExcludes`) without changing any calling code. + +- Implementation at **backup request level** within the `ShouldInclude()` method +- Uses lazy evaluation with `LazyNamespaceIncludesExcludes` wrapper +- On-demand namespace resolution with thread-safe caching +- First call triggers Kubernetes API namespace enumeration and wildcard resolution +- Results cached for subsequent calls within the same backup request + +## Detailed Design + +### Polymorphic Interface Approach + +The key insight is that all existing backup code already calls the same 4 methods on namespace filtering: +- `ShouldInclude(namespace string) bool` - Core filtering logic +- `IncludesString() string` - Logging display +- `ExcludesString() string` - Logging display +- `IncludeEverything() bool` - Optimization checks + +By creating a `NamespaceIncludesExcludesInterface` with these methods, we can: +1. **Standard case**: Use existing `*IncludesExcludes` (no wildcards) +2. **Wildcard case**: Use new `*LazyNamespaceIncludesExcludes` (with K8s API enumeration) + +**No calling code changes needed** - the interface abstraction handles everything. + +**Cache Scope:** Single backup request only - automatic cleanup when request completes. + +### Implementation Strategy + +**Location:** `pkg/util/collections/includes_excludes.go` +- New interface defining the 4 required methods +- `LazyNamespaceIncludesExcludes` struct embedding `*IncludesExcludes` for fallback +- Lazy resolution with thread-safe caching using mutex +- Special case handling for lone `*` to preserve existing efficient behavior + +**Integration:** `pkg/backup/backup.go` +- Wildcard detection logic determines which implementation to return +- Lone `*` pattern → standard `IncludesExcludes` (preserve current behavior) +- Any other wildcards → lazy `LazyNamespaceIncludesExcludes` + +**Type Updates:** Change struct fields from concrete `*IncludesExcludes` to interface type +- `pkg/backup/request.go` - Request struct field type +- `pkg/backup/item_collector.go` - Function parameter types + +### Performance Characteristics +- **First `ShouldInclude()` call:** ~500ms (K8s API namespace enumeration + wildcard resolution) +- **Subsequent calls:** ~1ms (cached lookup with read lock) +- **Memory overhead:** Minimal (resolved namespace list stored once per backup request) +- **Concurrency:** Full concurrent read access to cached results + +## Alternatives Considered + +### CLI-Level Resolution +**Problem:** Resolving wildcards during `velero backup create` command + +**Why rejected:** +- **Lost User Intent:** Backup specs store resolved lists instead of original patterns +- **Audit Trail Issues:** Original wildcard intent not visible when examining backup specifications +- **CLI Complexity:** CLI requires cluster access and namespace enumeration capabilities + +### Server-Level (Controller) Resolution +**Problem:** Resolving wildcards in backup controller with persistent caching + +**Why rejected:** +- **Architectural Complexity:** Requires additional API schema changes for storing resolved namespace lists +- **Cache Management:** Need cache invalidation, storage, and lifecycle management +- **Limited Benefit:** Performance gain only applies to narrow controller reconciliation retry scenarios +- **State Management:** Introduces persistent state maintained across backup lifecycle + +### Request-Level (ShouldInclude) Resolution +**Chosen Approach:** Lazy evaluation within backup request processing + +**Benefits:** +- **Preserved Intent:** Original wildcard patterns remain in backup specifications +- **Optimal Performance:** First resolution (~500ms), subsequent calls (~1ms) with request-scoped caching +- **Clean Architecture:** No persistent state, no API schema changes, minimal code changes +- **Thread Safety:** Proper mutex usage for concurrent worker access +- **Scoped Lifetime:** Cache automatically cleaned up when backup request completes + +## Security Considerations +- Implementation requires Velero service account to have `list` permissions on namespace resources +- Aligns with existing Velero RBAC requirements +- No additional privileges or security surface area introduced + +## Addressing Implementation Concerns + +### Multiple Pattern Support +Multiple wildcards work naturally: `--include-namespaces "prod-*,staging-*,dev-*"` - each pattern evaluated independently during lazy resolution. + +### Mixed Literal and Wildcard Detection +Simple approach: strings containing `*` are wildcards, others use existing literal namespace logic. Zero breaking changes for existing validation. + +### Include/Exclude Conflict Detection +Runtime resolution simplifies conflicts - wildcards resolve to actual namespace lists first, then standard include/exclude precedence applies. + +### Backward Compatibility +Lazy evaluation triggers only when wildcards detected. Non-wildcard backups have zero overhead and identical behavior to current implementation. + +## Special Consideration: Existing `*` Behavior + +**Current Velero Behavior:** `--include-namespaces "*"` (the CLI default) means "include all namespaces" and uses special logic that doesn't enumerate namespaces - it simply bypasses namespace filtering entirely. + +**Potential Breaking Change:** Our wildcard implementation would treat `*` as a glob pattern, resolving it to a specific list of namespaces at backup start time, which changes the behavior from "include everything" to "include these specific namespaces." + +**Required Solution:** Special-case handling for the lone `*` pattern to preserve existing behavior by using original `IncludesExcludes` logic instead of wildcard resolution. + +This ensures that `--include-namespaces "*"` continues to work exactly as before, while enabling new wildcard patterns like `prod-*`, `*-staging`, etc. + +## Compatibility +- Full backward compatibility with existing backup specifications using literal namespace lists +- No changes required to CLI commands, existing backups, or restore operations + +## Implementation +The implementation consists of approximately 200 lines of new code across four files: +- `pkg/util/collections/includes_excludes.go`: Core lazy evaluation logic (~150 lines) +- `pkg/backup/backup.go`: Wildcard detection logic (~20 lines) +- `pkg/backup/request.go`: Interface type usage (~5 lines) +- `pkg/backup/item_collector.go`: Compatibility method calls (~25 lines) + +## Open Issues +None. \ No newline at end of file From 69e307918bdedd03e12838d018cea084b984f0f6 Mon Sep 17 00:00:00 2001 From: Joseph Date: Mon, 28 Jul 2025 08:42:26 -0400 Subject: [PATCH 02/13] Update design Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 261 ++++++++++++++++++++ 1 file changed, 261 insertions(+) diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md index b7b1b01ba..32245dc87 100644 --- a/design/wildcard-namespace-support-design.md +++ b/design/wildcard-namespace-support-design.md @@ -73,6 +73,267 @@ By creating a `NamespaceIncludesExcludesInterface` with these methods, we can: - **Memory overhead:** Minimal (resolved namespace list stored once per backup request) - **Concurrency:** Full concurrent read access to cached results +## Namespace Discovery Timing and Behavior + +### Snapshot Timing +**Wildcard patterns are resolved at backup start time** and remain fixed for the entire backup duration. This provides: +- **Consistent behavior**: All resources in a backup come from the same namespace set +- **Predictable results**: Backup contents don't change mid-execution +- **Performance**: No repeated namespace enumeration during backup processing + +### Runtime Namespace Changes +When namespaces are created or deleted during backup execution: + +**Newly Created Namespaces:** +- If `prod-new` is created after backup starts, it will **NOT** be included even if it matches `prod-*` +- The resolved namespace list is fixed at backup start time + +**Deleted Namespaces:** +- If a namespace matching the pattern is deleted during backup, the backup continues +- Resources already processed from that namespace remain in the backup +- Subsequent resource enumeration for that namespace may result in "not found" errors (handled gracefully) + - This should ideally fail so that the user can re-run it without a namespace being deleted while a backup is started which is rare. + +**User Expectations:** +This behavior should be explicitly documented with examples: +``` +# At backup start: namespaces [prod-app, prod-db] exist +velero backup create --include-namespaces "prod-*" + +# During backup: prod-cache namespace is created +# Result: prod-cache is NOT included in this backup +# Recommendation: Run another backup to capture newly created namespaces +``` + +## Pattern Complexity and Validation + +### Supported Patterns +**Basic Wildcard Support (`*` only):** +- `prefix-*` - Matches namespaces starting with "prefix-" +- `*-suffix` - Matches namespaces ending with "-suffix" +- `*-middle-*` - Matches namespaces containing "-middle-" +- `*` - Special case: matches all namespaces (preserves current behavior) + +### Unsupported Patterns +**Not supported in initial implementation:** +- `?` for single character matching (e.g., `prod-?-app`) +- Character classes (e.g., `prod-[abc]-app`) +- Regex patterns (e.g., `prod-\d+-app`) + +### Pattern Validation +**Creation-time validation:** +- Invalid patterns containing unsupported characters will be rejected at backup creation +- Validation occurs in CLI and API server admission controller +- Clear error messages guide users to supported patterns + +**Example validation errors:** +```bash +# Unsupported pattern +velero backup create --include-namespaces "prod-?-app" +# Error: Pattern 'prod-?-app' contains unsupported character '?'. Only '*' wildcards are supported. + +# Valid patterns +velero backup create --include-namespaces "prod-*,*-staging" +# Success: Patterns validated successfully +``` + +## Error Handling + +### Kubernetes API Failures +**Namespace enumeration failures:** +- If initial namespace list API call fails → backup fails with clear error message +- Transient failures are retried using standard Kubernetes client retry logic +- No fallback to cached/partial data to ensure consistent behavior + +**Error response example:** +``` +Error: Failed to enumerate namespaces for wildcard resolution: unable to connect to Kubernetes API +Backup creation aborted. Please verify cluster connectivity and try again. +``` + +### Zero Namespace Matches +**When wildcard patterns match no namespaces:** +- **Behavior**: Warning logged, backup proceeds with empty namespace set +- **User notification**: Warning in backup status and logs +- **Rationale**: Allows for valid scenarios (e.g., temporary namespace absence) + +**Warning example:** +``` +Warning: Wildcard pattern 'prod-*' matched 0 namespaces. Backup will include no namespaces from this pattern. +``` + +### Dry-Run Support +**Preview functionality:** +```bash +# New flag to preview wildcard resolution +velero backup create my-backup --include-namespaces "prod-*" --dry-run=wildcards + +# Output: +Wildcard pattern 'prod-*' would include namespaces: [prod-app, prod-db, prod-cache] +Wildcard pattern '*-staging' would include namespaces: [app-staging, db-staging] +Total namespaces: 5 +``` + +## Restore Operations + +### Wildcard Behavior During Restore +**Restore uses namespaces captured at backup time:** +- Wildcard patterns in backup specs are **not** re-evaluated during restore +- Restore operates on the concrete namespace list that was resolved during backup +- This ensures restore consistency even if cluster namespace state has changed + +**Implementation approach:** +1. **Backup metadata storage**: Store both original patterns and resolved namespace lists +2. **Restore processing**: Use resolved namespace lists, ignore original patterns +3. **Audit trail**: Both patterns and resolved lists visible in backup metadata + +**Example scenario:** +```yaml +# Original backup spec +includedNamespaces: ["prod-*"] + +# Stored in backup metadata +resolvedNamespaces: ["prod-app", "prod-db"] +originalPatterns: ["prod-*"] + +# During restore (even if prod-cache now exists) +# Only prod-app and prod-db are restored +``` + +### Disaster Recovery Scenarios +**Cross-cluster restore behavior:** +- Restore attempts to create resources in target namespaces +- If target namespaces don't exist, Velero creates them (existing behavior) +- Wildcard patterns are not re-evaluated against target cluster + +## Scheduled Backups + +### Namespace State Changes Between Runs +**Each scheduled backup run performs fresh wildcard resolution:** +- Pattern `prod-*` may include different namespaces in each backup run +- This allows scheduled backups to automatically capture newly created namespaces +- **Trade-off**: Backup contents may vary between runs vs. automatic inclusion of new resources + +**Storage implications:** +- Varying namespace sets between runs may affect deduplication efficiency +- Each backup stores its own resolved namespace list independently + +**Example behavior:** +``` +# Monday backup: prod-* matches [prod-app, prod-db] +# Tuesday: prod-cache namespace created +# Tuesday backup: prod-* matches [prod-app, prod-db, prod-cache] +``` + +**User expectations:** +- Document that scheduled backups automatically include newly matching namespaces +- Provide guidance on namespace naming conventions for predictable backup behavior + +## Testing Strategy + +### Unit Tests +**Pattern matching tests:** +```go +func TestWildcardPatterns(t *testing.T) { + tests := []struct { + pattern string + namespace string + expected bool + }{ + {"prod-*", "prod-app", true}, + {"prod-*", "staging-app", false}, + {"*-staging", "app-staging", true}, + {"*-test-*", "app-test-db", true}, + } + // ... test implementation +} +``` + +**Edge cases:** +- Empty pattern list +- Pattern with no matches +- Pattern matching single namespace +- Multiple overlapping patterns +- Special case lone `*` behavior + +### Integration Tests +**Kubernetes cluster scenarios:** +- Create namespaces, verify wildcard resolution +- Test namespace creation/deletion during backup +- Verify thread safety with concurrent backup operations +- Error scenarios (API failures, network issues) + +**Concurrency testing:** +- Multiple concurrent `ShouldInclude()` calls +- Thread safety verification +- Cache hit ratio measurement + +## Example Usage + +### CLI Usage +```bash +# Single wildcard pattern +velero backup create prod-backup --include-namespaces "prod-*" + +# Multiple patterns +velero backup create env-backup --include-namespaces "prod-*,staging-*,dev-*" + +# Mixed literal and wildcard +velero backup create mixed-backup --include-namespaces "prod-*,kube-system,monitoring" + +# Exclude patterns +velero backup create no-test --include-namespaces "*" --exclude-namespaces "*-test,*-temp" + +# Preview before creating +velero backup create my-backup --include-namespaces "prod-*" --dry-run=wildcards +``` + +### Backup Specification YAML +```yaml +apiVersion: velero.io/v1 +kind: Backup +metadata: + name: production-backup + namespace: velero +spec: + # Wildcard patterns in includedNamespaces + includedNamespaces: + - "prod-*" # All namespaces starting with "prod-" + - "production-*" # All namespaces starting with "production-" + - "critical-app" # Literal namespace (mixed with wildcards) + + # Wildcard patterns in excludedNamespaces + excludedNamespaces: + - "*-test" # Exclude any test namespaces + - "*-temp" # Exclude any temporary namespaces + + # Other backup configuration + storageLocation: default + volumeSnapshotLocations: + - default + includeClusterResources: false +``` + +### Stored Backup Metadata +```yaml +# What gets stored in backup metadata +apiVersion: velero.io/v1 +kind: Backup +metadata: + name: production-backup +status: + # Original user patterns preserved for audit + originalIncludePatterns: ["prod-*", "production-*", "critical-app"] + originalExcludePatterns: ["*-test", "*-temp"] + + # Resolved concrete namespace lists (used for restore) + resolvedIncludedNamespaces: ["prod-app", "prod-db", "production-web", "critical-app"] + resolvedExcludedNamespaces: ["prod-app-test", "staging-temp"] + + # Resolution timestamp + namespaceResolutionTime: "2024-01-15T10:30:00Z" +``` + ## Alternatives Considered ### CLI-Level Resolution From c0699c443bdcd9d0f859b01f826c108de0aab016 Mon Sep 17 00:00:00 2001 From: Joseph Date: Fri, 1 Aug 2025 13:25:44 -0400 Subject: [PATCH 03/13] New design doc Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 424 +++----------------- 1 file changed, 48 insertions(+), 376 deletions(-) diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md index 32245dc87..b5dfd3618 100644 --- a/design/wildcard-namespace-support-design.md +++ b/design/wildcard-namespace-support-design.md @@ -1,407 +1,79 @@ -# Wildcard Namespace Support Design + +# Wildcard namespace includes/excludes support for backups and restores ## Abstract -This proposal introduces wildcard pattern support for namespace inclusion and exclusion in Velero backups (e.g., `prod-*`, `*-staging`). -The implementation uses lazy evaluation within the existing `ShouldInclude()` method to resolve wildcards on-demand with request-scoped caching. -Based on [Issue #1874](https://github.com/vmware-tanzu/velero/issues/1874). +One to two sentences that describes the goal of this proposal and the problem being solved by the proposed change. +The reader should be able to tell by the title, and the opening paragraph, if this document is relevant to them. + +Velero currently does not have any support for wildcard characters in the namespace spec. +It fully expects the namespaces to be string literals. + +The only and notable exception is the "*" character by it's lonesome, which acts as an include all and ignore excludes option. +Internally Velero treats not specifying anything as the "*" case. + +This document details the approach to implementing wildcard namespaces, while keeping the "*" characters purpose intact for legacy purposes. ## Background -- Currently, Velero users must explicitly list each namespace for backup operations -- In environments with many namespaces following naming conventions (e.g., `prod-app`, `prod-db`, `prod-cache`), this becomes: - - Cumbersome to maintain - - Error-prone to manage -- Users have requested wildcard support to enable patterns like `--include-namespaces "prod-*"` +This was raised in Issue [#1874](https://github.com/vmware-tanzu/velero/issues/1874) + ## Goals -- Enable wildcard pattern support for namespace includes and excludes in Velero backup specifications -- Maintain optimal performance with lazy evaluation and request-scoped caching -- Preserve original wildcard patterns in backup specifications for audit and readability purposes +- A short list of things which will be accomplished by implementing this proposal. +- Two things is ok. +- Three is pushing it. +- More than three goals suggests that the proposal's scope is too large. + +- Add support for wildcard namespaces in --include-namespaces and --exclude-namespaces +- Ensure legacy "*" support is not affected ## Non Goals -- Support for complex regex patterns beyond basic glob-style wildcards (`*`) -- Persistent caching of namespace resolution across backup requests -- Real-time namespace discovery that changes during backup execution +- A short list of items which are: +- a. out of scope +- b. follow on items which are deliberately excluded from this proposal. + +- Completely rethinking the way "*" is treated and allowing it to work with wildcard excludes. + ## High-Level Design +One to two paragraphs that describe the high level changes that will be made to implement this proposal. -**Core Approach:** We're making the existing concrete type (`*IncludesExcludes`) polymorphic so we can substitute our new lazy evaluation type (`*LazyNamespaceIncludesExcludes`) without changing any calling code. +Points of interest are two funcs within the utility layer, in file `velero/pkg/backup/item_collector.go` +- [collectNamespaces](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L742) +- [getNamespacesToList](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L638) -- Implementation at **backup request level** within the `ShouldInclude()` method -- Uses lazy evaluation with `LazyNamespaceIncludesExcludes` wrapper -- On-demand namespace resolution with thread-safe caching -- First call triggers Kubernetes API namespace enumeration and wildcard resolution -- Results cached for subsequent calls within the same backup request +collectNamespaces gets all the active namespaces and matches it against the user spec for included namespaces (r.backupRequest.Backup.Spec.IncludedNamespaces) +This is an ideal point where wildcard expansion can take place. +The implementation would mean that just like "*", namespaces with wildcard symbols would also be passed through without an existence check. +The resolved namespaces are stored in new status fields on the backup. ## Detailed Design +A detailed design describing how the changes to the product should be made. -### Polymorphic Interface Approach +The names of types, fields, interfaces, and methods should be agreed on here, not debated in code review. +The same applies to changes in CRDs, YAML examples, and so on. -The key insight is that all existing backup code already calls the same 4 methods on namespace filtering: -- `ShouldInclude(namespace string) bool` - Core filtering logic -- `IncludesString() string` - Logging display -- `ExcludesString() string` - Logging display -- `IncludeEverything() bool` - Optimization checks +Ideally the changes should be made in sequence so that the work required to implement this design can be done incrementally, possibly in parallel. -By creating a `NamespaceIncludesExcludesInterface` with these methods, we can: -1. **Standard case**: Use existing `*IncludesExcludes` (no wildcards) -2. **Wildcard case**: Use new `*LazyNamespaceIncludesExcludes` (with K8s API enumeration) - -**No calling code changes needed** - the interface abstraction handles everything. - -**Cache Scope:** Single backup request only - automatic cleanup when request completes. - -### Implementation Strategy - -**Location:** `pkg/util/collections/includes_excludes.go` -- New interface defining the 4 required methods -- `LazyNamespaceIncludesExcludes` struct embedding `*IncludesExcludes` for fallback -- Lazy resolution with thread-safe caching using mutex -- Special case handling for lone `*` to preserve existing efficient behavior - -**Integration:** `pkg/backup/backup.go` -- Wildcard detection logic determines which implementation to return -- Lone `*` pattern → standard `IncludesExcludes` (preserve current behavior) -- Any other wildcards → lazy `LazyNamespaceIncludesExcludes` - -**Type Updates:** Change struct fields from concrete `*IncludesExcludes` to interface type -- `pkg/backup/request.go` - Request struct field type -- `pkg/backup/item_collector.go` - Function parameter types - -### Performance Characteristics -- **First `ShouldInclude()` call:** ~500ms (K8s API namespace enumeration + wildcard resolution) -- **Subsequent calls:** ~1ms (cached lookup with read lock) -- **Memory overhead:** Minimal (resolved namespace list stored once per backup request) -- **Concurrency:** Full concurrent read access to cached results - -## Namespace Discovery Timing and Behavior - -### Snapshot Timing -**Wildcard patterns are resolved at backup start time** and remain fixed for the entire backup duration. This provides: -- **Consistent behavior**: All resources in a backup come from the same namespace set -- **Predictable results**: Backup contents don't change mid-execution -- **Performance**: No repeated namespace enumeration during backup processing - -### Runtime Namespace Changes -When namespaces are created or deleted during backup execution: - -**Newly Created Namespaces:** -- If `prod-new` is created after backup starts, it will **NOT** be included even if it matches `prod-*` -- The resolved namespace list is fixed at backup start time - -**Deleted Namespaces:** -- If a namespace matching the pattern is deleted during backup, the backup continues -- Resources already processed from that namespace remain in the backup -- Subsequent resource enumeration for that namespace may result in "not found" errors (handled gracefully) - - This should ideally fail so that the user can re-run it without a namespace being deleted while a backup is started which is rare. - -**User Expectations:** -This behavior should be explicitly documented with examples: -``` -# At backup start: namespaces [prod-app, prod-db] exist -velero backup create --include-namespaces "prod-*" - -# During backup: prod-cache namespace is created -# Result: prod-cache is NOT included in this backup -# Recommendation: Run another backup to capture newly created namespaces +1. Add new status fields to the backup CRD to store expanded wildcard namespaces ``` -## Pattern Complexity and Validation - -### Supported Patterns -**Basic Wildcard Support (`*` only):** -- `prefix-*` - Matches namespaces starting with "prefix-" -- `*-suffix` - Matches namespaces ending with "-suffix" -- `*-middle-*` - Matches namespaces containing "-middle-" -- `*` - Special case: matches all namespaces (preserves current behavior) - -### Unsupported Patterns -**Not supported in initial implementation:** -- `?` for single character matching (e.g., `prod-?-app`) -- Character classes (e.g., `prod-[abc]-app`) -- Regex patterns (e.g., `prod-\d+-app`) - -### Pattern Validation -**Creation-time validation:** -- Invalid patterns containing unsupported characters will be rejected at backup creation -- Validation occurs in CLI and API server admission controller -- Clear error messages guide users to supported patterns - -**Example validation errors:** -```bash -# Unsupported pattern -velero backup create --include-namespaces "prod-?-app" -# Error: Pattern 'prod-?-app' contains unsupported character '?'. Only '*' wildcards are supported. - -# Valid patterns -velero backup create --include-namespaces "prod-*,*-staging" -# Success: Patterns validated successfully ``` +2. Create a util package for wildcard expansion -## Error Handling - -### Kubernetes API Failures -**Namespace enumeration failures:** -- If initial namespace list API call fails → backup fails with clear error message -- Transient failures are retried using standard Kubernetes client retry logic -- No fallback to cached/partial data to ensure consistent behavior - -**Error response example:** -``` -Error: Failed to enumerate namespaces for wildcard resolution: unable to connect to Kubernetes API -Backup creation aborted. Please verify cluster connectivity and try again. -``` - -### Zero Namespace Matches -**When wildcard patterns match no namespaces:** -- **Behavior**: Warning logged, backup proceeds with empty namespace set -- **User notification**: Warning in backup status and logs -- **Rationale**: Allows for valid scenarios (e.g., temporary namespace absence) - -**Warning example:** -``` -Warning: Wildcard pattern 'prod-*' matched 0 namespaces. Backup will include no namespaces from this pattern. -``` - -### Dry-Run Support -**Preview functionality:** -```bash -# New flag to preview wildcard resolution -velero backup create my-backup --include-namespaces "prod-*" --dry-run=wildcards - -# Output: -Wildcard pattern 'prod-*' would include namespaces: [prod-app, prod-db, prod-cache] -Wildcard pattern '*-staging' would include namespaces: [app-staging, db-staging] -Total namespaces: 5 -``` - -## Restore Operations - -### Wildcard Behavior During Restore -**Restore uses namespaces captured at backup time:** -- Wildcard patterns in backup specs are **not** re-evaluated during restore -- Restore operates on the concrete namespace list that was resolved during backup -- This ensures restore consistency even if cluster namespace state has changed - -**Implementation approach:** -1. **Backup metadata storage**: Store both original patterns and resolved namespace lists -2. **Restore processing**: Use resolved namespace lists, ignore original patterns -3. **Audit trail**: Both patterns and resolved lists visible in backup metadata - -**Example scenario:** -```yaml -# Original backup spec -includedNamespaces: ["prod-*"] - -# Stored in backup metadata -resolvedNamespaces: ["prod-app", "prod-db"] -originalPatterns: ["prod-*"] - -# During restore (even if prod-cache now exists) -# Only prod-app and prod-db are restored -``` - -### Disaster Recovery Scenarios -**Cross-cluster restore behavior:** -- Restore attempts to create resources in target namespaces -- If target namespaces don't exist, Velero creates them (existing behavior) -- Wildcard patterns are not re-evaluated against target cluster - -## Scheduled Backups - -### Namespace State Changes Between Runs -**Each scheduled backup run performs fresh wildcard resolution:** -- Pattern `prod-*` may include different namespaces in each backup run -- This allows scheduled backups to automatically capture newly created namespaces -- **Trade-off**: Backup contents may vary between runs vs. automatic inclusion of new resources - -**Storage implications:** -- Varying namespace sets between runs may affect deduplication efficiency -- Each backup stores its own resolved namespace list independently - -**Example behavior:** -``` -# Monday backup: prod-* matches [prod-app, prod-db] -# Tuesday: prod-cache namespace created -# Tuesday backup: prod-* matches [prod-app, prod-db, prod-cache] -``` - -**User expectations:** -- Document that scheduled backups automatically include newly matching namespaces -- Provide guidance on namespace naming conventions for predictable backup behavior - -## Testing Strategy - -### Unit Tests -**Pattern matching tests:** -```go -func TestWildcardPatterns(t *testing.T) { - tests := []struct { - pattern string - namespace string - expected bool - }{ - {"prod-*", "prod-app", true}, - {"prod-*", "staging-app", false}, - {"*-staging", "app-staging", true}, - {"*-test-*", "app-test-db", true}, - } - // ... test implementation -} -``` - -**Edge cases:** -- Empty pattern list -- Pattern with no matches -- Pattern matching single namespace -- Multiple overlapping patterns -- Special case lone `*` behavior - -### Integration Tests -**Kubernetes cluster scenarios:** -- Create namespaces, verify wildcard resolution -- Test namespace creation/deletion during backup -- Verify thread safety with concurrent backup operations -- Error scenarios (API failures, network issues) - -**Concurrency testing:** -- Multiple concurrent `ShouldInclude()` calls -- Thread safety verification -- Cache hit ratio measurement - -## Example Usage - -### CLI Usage -```bash -# Single wildcard pattern -velero backup create prod-backup --include-namespaces "prod-*" - -# Multiple patterns -velero backup create env-backup --include-namespaces "prod-*,staging-*,dev-*" - -# Mixed literal and wildcard -velero backup create mixed-backup --include-namespaces "prod-*,kube-system,monitoring" - -# Exclude patterns -velero backup create no-test --include-namespaces "*" --exclude-namespaces "*-test,*-temp" - -# Preview before creating -velero backup create my-backup --include-namespaces "prod-*" --dry-run=wildcards -``` - -### Backup Specification YAML -```yaml -apiVersion: velero.io/v1 -kind: Backup -metadata: - name: production-backup - namespace: velero -spec: - # Wildcard patterns in includedNamespaces - includedNamespaces: - - "prod-*" # All namespaces starting with "prod-" - - "production-*" # All namespaces starting with "production-" - - "critical-app" # Literal namespace (mixed with wildcards) - - # Wildcard patterns in excludedNamespaces - excludedNamespaces: - - "*-test" # Exclude any test namespaces - - "*-temp" # Exclude any temporary namespaces - - # Other backup configuration - storageLocation: default - volumeSnapshotLocations: - - default - includeClusterResources: false -``` - -### Stored Backup Metadata -```yaml -# What gets stored in backup metadata -apiVersion: velero.io/v1 -kind: Backup -metadata: - name: production-backup -status: - # Original user patterns preserved for audit - originalIncludePatterns: ["prod-*", "production-*", "critical-app"] - originalExcludePatterns: ["*-test", "*-temp"] - - # Resolved concrete namespace lists (used for restore) - resolvedIncludedNamespaces: ["prod-app", "prod-db", "production-web", "critical-app"] - resolvedExcludedNamespaces: ["prod-app-test", "staging-temp"] - - # Resolution timestamp - namespaceResolutionTime: "2024-01-15T10:30:00Z" -``` +3. If required, expand wildcards and replace the request's includes and excludes with expanded namespaces +4. Populate the expanded namespace status field with the namespaces. ## Alternatives Considered - -### CLI-Level Resolution -**Problem:** Resolving wildcards during `velero backup create` command - -**Why rejected:** -- **Lost User Intent:** Backup specs store resolved lists instead of original patterns -- **Audit Trail Issues:** Original wildcard intent not visible when examining backup specifications -- **CLI Complexity:** CLI requires cluster access and namespace enumeration capabilities - -### Server-Level (Controller) Resolution -**Problem:** Resolving wildcards in backup controller with persistent caching - -**Why rejected:** -- **Architectural Complexity:** Requires additional API schema changes for storing resolved namespace lists -- **Cache Management:** Need cache invalidation, storage, and lifecycle management -- **Limited Benefit:** Performance gain only applies to narrow controller reconciliation retry scenarios -- **State Management:** Introduces persistent state maintained across backup lifecycle - -### Request-Level (ShouldInclude) Resolution -**Chosen Approach:** Lazy evaluation within backup request processing - -**Benefits:** -- **Preserved Intent:** Original wildcard patterns remain in backup specifications -- **Optimal Performance:** First resolution (~500ms), subsequent calls (~1ms) with request-scoped caching -- **Clean Architecture:** No persistent state, no API schema changes, minimal code changes -- **Thread Safety:** Proper mutex usage for concurrent worker access -- **Scoped Lifetime:** Cache automatically cleaned up when backup request completes +If there are alternative high level or detailed designs that were not pursued they should be called out here with a brief explanation of why they were not pursued. ## Security Considerations -- Implementation requires Velero service account to have `list` permissions on namespace resources -- Aligns with existing Velero RBAC requirements -- No additional privileges or security surface area introduced - -## Addressing Implementation Concerns - -### Multiple Pattern Support -Multiple wildcards work naturally: `--include-namespaces "prod-*,staging-*,dev-*"` - each pattern evaluated independently during lazy resolution. - -### Mixed Literal and Wildcard Detection -Simple approach: strings containing `*` are wildcards, others use existing literal namespace logic. Zero breaking changes for existing validation. - -### Include/Exclude Conflict Detection -Runtime resolution simplifies conflicts - wildcards resolve to actual namespace lists first, then standard include/exclude precedence applies. - -### Backward Compatibility -Lazy evaluation triggers only when wildcards detected. Non-wildcard backups have zero overhead and identical behavior to current implementation. - -## Special Consideration: Existing `*` Behavior - -**Current Velero Behavior:** `--include-namespaces "*"` (the CLI default) means "include all namespaces" and uses special logic that doesn't enumerate namespaces - it simply bypasses namespace filtering entirely. - -**Potential Breaking Change:** Our wildcard implementation would treat `*` as a glob pattern, resolving it to a specific list of namespaces at backup start time, which changes the behavior from "include everything" to "include these specific namespaces." - -**Required Solution:** Special-case handling for the lone `*` pattern to preserve existing behavior by using original `IncludesExcludes` logic instead of wildcard resolution. - -This ensures that `--include-namespaces "*"` continues to work exactly as before, while enabling new wildcard patterns like `prod-*`, `*-staging`, etc. +If this proposal has an impact to the security of the product, its users, or data stored or transmitted via the product, they must be addressed here. ## Compatibility -- Full backward compatibility with existing backup specifications using literal namespace lists -- No changes required to CLI commands, existing backups, or restore operations +A discussion of any compatibility issues that need to be considered ## Implementation -The implementation consists of approximately 200 lines of new code across four files: -- `pkg/util/collections/includes_excludes.go`: Core lazy evaluation logic (~150 lines) -- `pkg/backup/backup.go`: Wildcard detection logic (~20 lines) -- `pkg/backup/request.go`: Interface type usage (~5 lines) -- `pkg/backup/item_collector.go`: Compatibility method calls (~25 lines) +A description of the implementation, timelines, and any resources that have agreed to contribute. ## Open Issues -None. \ No newline at end of file +A discussion of issues relating to this proposal for which the author does not know the solution. This section may be omitted if there are none. From 4c1457c3181eee14181362debfd5ba2d760a4d1c Mon Sep 17 00:00:00 2001 From: Joseph Date: Fri, 1 Aug 2025 13:51:44 -0400 Subject: [PATCH 04/13] Enhance wildcard namespace support in backup and restore design document - Updated the abstract to clarify the current limitations of namespace specifications in Velero. - Expanded the goals section to include specific objectives for implementing wildcard patterns in `--include-namespaces` and `--exclude-namespaces`. - Detailed the high-level design and implementation steps, including the addition of new status fields in the backup CRD and the creation of a utility package for wildcard expansion. - Addressed backward compatibility and known limitations regarding the use of wildcards alongside the existing "*" character. This update aims to provide a comprehensive overview of the proposed changes for improved namespace selection flexibility. Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 171 +++++++++++++++----- 1 file changed, 128 insertions(+), 43 deletions(-) diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md index b5dfd3618..fb3426719 100644 --- a/design/wildcard-namespace-support-design.md +++ b/design/wildcard-namespace-support-design.md @@ -1,79 +1,164 @@ -# Wildcard namespace includes/excludes support for backups and restores +# Wildcard Namespace Includes/Excludes Support for Backups and Restores ## Abstract -One to two sentences that describes the goal of this proposal and the problem being solved by the proposed change. -The reader should be able to tell by the title, and the opening paragraph, if this document is relevant to them. -Velero currently does not have any support for wildcard characters in the namespace spec. -It fully expects the namespaces to be string literals. +Velero currently does not support wildcard characters in namespace specifications, requiring all namespaces to be specified as string literals. The only exception is the standalone "*" character, which includes all namespaces and ignores excludes. -The only and notable exception is the "*" character by it's lonesome, which acts as an include all and ignore excludes option. -Internally Velero treats not specifying anything as the "*" case. - -This document details the approach to implementing wildcard namespaces, while keeping the "*" characters purpose intact for legacy purposes. +This document details the approach to implementing wildcard namespace support for `--include-namespaces` and `--exclude-namespaces` flags, while preserving the existing "*" behavior for backward compatibility. ## Background -This was raised in Issue [#1874](https://github.com/vmware-tanzu/velero/issues/1874) +This feature was requested in Issue [#1874](https://github.com/vmware-tanzu/velero/issues/1874) to provide more flexible namespace selection using wildcard patterns. ## Goals -- A short list of things which will be accomplished by implementing this proposal. -- Two things is ok. -- Three is pushing it. -- More than three goals suggests that the proposal's scope is too large. -- Add support for wildcard namespaces in --include-namespaces and --exclude-namespaces -- Ensure legacy "*" support is not affected +- Add support for wildcard patterns in `--include-namespaces` and `--exclude-namespaces` flags +- Ensure legacy "*" behavior remains unchanged for backward compatibility -## Non Goals -- A short list of items which are: -- a. out of scope -- b. follow on items which are deliberately excluded from this proposal. +## Non-Goals -- Completely rethinking the way "*" is treated and allowing it to work with wildcard excludes. +- Completely rethinking the way "*" is treated and allowing it to work with wildcard excludes +- Supporting complex regex patterns beyond basic glob patterns ## High-Level Design -One to two paragraphs that describe the high level changes that will be made to implement this proposal. -Points of interest are two funcs within the utility layer, in file `velero/pkg/backup/item_collector.go` -- [collectNamespaces](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L742) -- [getNamespacesToList](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L638) +The wildcard expansion implementation focuses on two key functions in `pkg/backup/item_collector.go`: -collectNamespaces gets all the active namespaces and matches it against the user spec for included namespaces (r.backupRequest.Backup.Spec.IncludedNamespaces) -This is an ideal point where wildcard expansion can take place. -The implementation would mean that just like "*", namespaces with wildcard symbols would also be passed through without an existence check. -The resolved namespaces are stored in new status fields on the backup. +- [`collectNamespaces`](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L742) - Retrieves all active namespaces and processes include/exclude filters +- [`getNamespacesToList`](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L638) - Resolves namespace includes/excludes to final list + +The `collectNamespaces` function is the ideal integration point because it: +- Already retrieves all active namespaces from the cluster +- Processes the user-specified namespace filters +- Can expand wildcard patterns against the complete namespace list +- Stores the resolved namespaces in new backup status fields for visibility + +This approach ensures wildcard namespaces are handled consistently with the existing "*" behavior, bypassing individual namespace existence checks. ## Detailed Design -A detailed design describing how the changes to the product should be made. -The names of types, fields, interfaces, and methods should be agreed on here, not debated in code review. -The same applies to changes in CRDs, YAML examples, and so on. +The implementation involves four main components that can be developed incrementally: -Ideally the changes should be made in sequence so that the work required to implement this design can be done incrementally, possibly in parallel. +### Add new status fields to the backup CRD to store expanded wildcard namespaces -1. Add new status fields to the backup CRD to store expanded wildcard namespaces +```go +// BackupStatus captures the current status of a Velero backup. +type BackupStatus struct { + // ... existing fields ... + + // ExpandedIncludedNamespaces records the expanded include wildcard namespaces + // +optional + // +nullable + ExpandedIncludedNamespaces []string `json:"expandedIncludedNamespaces,omitempty"` + + // ExpandedExcludedNamespaces records the expanded exclude wildcard namespaces + // +optional + // +nullable + ExpandedExcludedNamespaces []string `json:"expandedExcludedNamespaces,omitempty"` + + // ... other fields ... +} ``` -``` -2. Create a util package for wildcard expansion +**Implementation**: Added status fields `ExpandedIncludedNamespaces` and `ExpandedExcludedNamespaces` to `pkg/apis/velero/v1/backup_types.go` to track the resolved namespace lists after wildcard expansion. -3. If required, expand wildcards and replace the request's includes and excludes with expanded namespaces -4. Populate the expanded namespace status field with the namespaces. +### Create a util package for wildcard expansion + +**Implementation**: Created `pkg/util/wildcard/expand.go` package containing: + +- `ShouldExpandWildcards(includes, excludes []string) bool` - Determines if wildcard expansion is needed (excludes simple "*" case) +- `ExpandWildcards(activeNamespaces, includes, excludes []string) ([]string, []string, error)` - Main expansion function +- `containsWildcardPattern(pattern string) bool` - Detects wildcard patterns (`*`, `?`, `[abc]`, `{a,b,c}`) +- `validateWildcardPatterns(patterns []string) error` - Validates patterns and rejects unsupported regex symbols +- Uses `github.com/gobwas/glob` library for glob pattern matching + +**Supported patterns**: +- `*` (any characters) +- `?` (single character) +- `[abc]` (character classes) +- `{a,b,c}` (alternatives) + +**Unsupported**: Regex symbols `|()`, consecutive asterisks `**` + +### If required, expand wildcards and replace the request's includes and excludes with expanded namespaces + +**Implementation**: In `pkg/backup/item_collector.go`: + +```go +// collectNamespaces function (line 748-803) +if wildcard.ShouldExpandWildcards(namespaceSelector.GetIncludes(), namespaceSelector.GetExcludes()) { + if err := r.expandNamespaceWildcards(activeNamespacesList, namespaceSelector); err != nil { + return nil, errors.WithMessage(err, "failed to expand namespace wildcard patterns") + } +} +``` + +The expansion occurs when collecting namespaces, after retrieving all active namespaces from the cluster. The `expandNamespaceWildcards` method: +- Calls `wildcard.ExpandWildcards()` with active namespaces and original patterns +- Updates the namespace selector with expanded results using `SetIncludes()` and `SetExcludes()` +- Preserves backward compatibility by skipping expansion for simple "*" pattern + +**Performance Improvement**: As part of this implementation, active namespaces are stored in a hashset rather than being iterated for each resolved/literal namespace check. This eliminates a [nested loop anti-pattern](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L767) and improves performance. + +### Populate the expanded namespace status field with the namespaces + +**Implementation**: In `expandNamespaceWildcards` function (line 889-891): + +```go +// Record the expanded wildcard includes/excludes in the request status +r.backupRequest.Status.ExpandedIncludedNamespaces = expandedIncludes +r.backupRequest.Status.ExpandedExcludedNamespaces = expandedExcludes +``` + +The status fields are populated immediately after successful wildcard expansion, providing visibility into which namespaces were actually matched by the wildcard patterns. ## Alternatives Considered -If there are alternative high level or detailed designs that were not pursued they should be called out here with a brief explanation of why they were not pursued. + +Several implementation approaches were considered for the wildcard expansion logic: + +### 1. Wildcard expansion in `getNamespacesToList` + +This approach was ruled out because: +- The `collectNamespaces` function encounters wildcard patterns first in the processing flow +- `collectNamespaces` already has access to the complete list of active namespaces, eliminating the need for additional API calls +- Placing the logic in `getNamespacesToList` would require redundant namespace retrieval + +### 2. Client-side wildcard expansion + +Expanding wildcards on the CLI side was considered but rejected because: +- It would only work for command-line initiated backups +- Scheduled backups would not benefit from this approach +- The server-side approach provides consistent behavior across all backup initiation methods ## Security Considerations -If this proposal has an impact to the security of the product, its users, or data stored or transmitted via the product, they must be addressed here. + +This feature does not introduce any security vulnerabilities as it only affects namespace selection logic within the existing backup authorization framework. ## Compatibility -A discussion of any compatibility issues that need to be considered + +### Backward Compatibility + +The implementation maintains full backward compatibility with existing behavior: +- The standalone "*" character continues to work as before (include all namespaces, ignore excludes) +- Existing backup configurations remain unaffected + +### Known Limitations + +1. **Mixed "*" and wildcard usage**: When using the standalone "*" in includes, the legacy implementation takes precedence and wildcard expansion is skipped. This means you cannot combine "*" (all namespaces) in includes with wildcard patterns in excludes. + +2. **Selective exclusion limitation**: The current design does not support selective pattern-based exclusion when including all namespaces via "*". ## Implementation -A description of the implementation, timelines, and any resources that have agreed to contribute. + +The implementation follows the detailed design outlined above, with the following timeline: + +1. **Phase 1**: Add backup CRD status fields for expanded namespaces +2. **Phase 2**: Implement wildcard utility package with validation +3. **Phase 3**: Integrate expansion logic into backup item collection +4. **Phase 4**: Add status field population and logging ## Open Issues -A discussion of issues relating to this proposal for which the author does not know the solution. This section may be omitted if there are none. + +**Restore Integration**: Currently investigating how restore operations should handle wildcard-expanded backups. The proposed approach is to apply wildcard patterns against the expanded namespace list stored in the backup's status fields, ensuring consistency between backup and restore operations. \ No newline at end of file From 037db22afe7758cd2fdbac25cddd1c7ebec03567 Mon Sep 17 00:00:00 2001 From: Joseph Date: Fri, 1 Aug 2025 13:55:38 -0400 Subject: [PATCH 05/13] Refine wildcard namespace support design document - Clarified the use of the standalone `*` character in namespace specifications. - Ensured consistent formatting for `*` throughout the document. - Maintained focus on backward compatibility and limitations regarding wildcard usage. This update enhances the clarity and consistency of the design document for implementing wildcard namespace support in Velero. Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md index fb3426719..d6d529a02 100644 --- a/design/wildcard-namespace-support-design.md +++ b/design/wildcard-namespace-support-design.md @@ -3,9 +3,9 @@ ## Abstract -Velero currently does not support wildcard characters in namespace specifications, requiring all namespaces to be specified as string literals. The only exception is the standalone "*" character, which includes all namespaces and ignores excludes. +Velero currently does not support wildcard characters in namespace specifications, requiring all namespaces to be specified as string literals. The only exception is the standalone `*` character, which includes all namespaces and ignores excludes. -This document details the approach to implementing wildcard namespace support for `--include-namespaces` and `--exclude-namespaces` flags, while preserving the existing "*" behavior for backward compatibility. +This document details the approach to implementing wildcard namespace support for `--include-namespaces` and `--exclude-namespaces` flags, while preserving the existing `*` behavior for backward compatibility. ## Background @@ -14,11 +14,11 @@ This feature was requested in Issue [#1874](https://github.com/vmware-tanzu/vele ## Goals - Add support for wildcard patterns in `--include-namespaces` and `--exclude-namespaces` flags -- Ensure legacy "*" behavior remains unchanged for backward compatibility +- Ensure legacy `*` behavior remains unchanged for backward compatibility ## Non-Goals -- Completely rethinking the way "*" is treated and allowing it to work with wildcard excludes +- Completely rethinking the way `*` is treated and allowing it to work with wildcard excludes - Supporting complex regex patterns beyond basic glob patterns @@ -141,14 +141,14 @@ This feature does not introduce any security vulnerabilities as it only affects ### Backward Compatibility The implementation maintains full backward compatibility with existing behavior: -- The standalone "*" character continues to work as before (include all namespaces, ignore excludes) +- The standalone `*` character continues to work as before (include all namespaces, ignore excludes) - Existing backup configurations remain unaffected ### Known Limitations -1. **Mixed "*" and wildcard usage**: When using the standalone "*" in includes, the legacy implementation takes precedence and wildcard expansion is skipped. This means you cannot combine "*" (all namespaces) in includes with wildcard patterns in excludes. +1. **Mixed `*` and wildcard usage**: When using the standalone `*` in includes, the legacy implementation takes precedence and wildcard expansion is skipped. This means you cannot combine "*" (all namespaces) in includes with wildcard patterns in excludes. -2. **Selective exclusion limitation**: The current design does not support selective pattern-based exclusion when including all namespaces via "*". +2. **Selective exclusion limitation**: The current design does not support selective pattern-based exclusion when including all namespaces via `*`. ## Implementation From 571c9bd3ef1cc64fbee2f98467ee73731099ba94 Mon Sep 17 00:00:00 2001 From: Joseph Date: Sat, 2 Aug 2025 21:14:26 -0400 Subject: [PATCH 06/13] Enhance wildcard namespace support design document - Expanded the design to include detailed implementation steps for wildcard expansion in both backup and restore operations. - Added new status fields to the backup and restore CRDs to track expanded wildcard namespaces. - Clarified the approach to ensure backward compatibility with existing `*` behavior. - Addressed limitations and provided insights on restore operations handling wildcard-expanded backups. This update aims to provide a comprehensive and clear framework for implementing wildcard namespace support in Velero. Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 100 ++++++++++++++++---- 1 file changed, 82 insertions(+), 18 deletions(-) diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md index d6d529a02..c807ad347 100644 --- a/design/wildcard-namespace-support-design.md +++ b/design/wildcard-namespace-support-design.md @@ -3,9 +3,9 @@ ## Abstract -Velero currently does not support wildcard characters in namespace specifications, requiring all namespaces to be specified as string literals. The only exception is the standalone `*` character, which includes all namespaces and ignores excludes. +Velero currently does not support wildcard characters in namespace specifications, requiring all namespaces to be specified as string literals. The only exception is the standalone "*" character, which includes all namespaces and ignores excludes. -This document details the approach to implementing wildcard namespace support for `--include-namespaces` and `--exclude-namespaces` flags, while preserving the existing `*` behavior for backward compatibility. +This document details the approach to implementing wildcard namespace support for `--include-namespaces` and `--exclude-namespaces` flags, while preserving the existing "*" behavior for backward compatibility. ## Background @@ -13,17 +13,19 @@ This feature was requested in Issue [#1874](https://github.com/vmware-tanzu/vele ## Goals -- Add support for wildcard patterns in `--include-namespaces` and `--exclude-namespaces` flags -- Ensure legacy `*` behavior remains unchanged for backward compatibility +- Add support for wildcard patterns in `--include-namespaces` and `--exclude-namespaces` flags for both backup and restore +- Ensure legacy "*" behavior remains unchanged for backward compatibility ## Non-Goals -- Completely rethinking the way `*` is treated and allowing it to work with wildcard excludes +- Completely rethinking the way "*" is treated and allowing it to work with wildcard excludes - Supporting complex regex patterns beyond basic glob patterns ## High-Level Design +## Backup + The wildcard expansion implementation focuses on two key functions in `pkg/backup/item_collector.go`: - [`collectNamespaces`](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L742) - Retrieves all active namespaces and processes include/exclude filters @@ -37,32 +39,52 @@ The `collectNamespaces` function is the ideal integration point because it: This approach ensures wildcard namespaces are handled consistently with the existing "*" behavior, bypassing individual namespace existence checks. +## Restore + +The wildcard expansion implementation for restore operations focuses on the main execution flow in `pkg/restore/restore.go`: + +- [`execute`](https://github.com/vmware-tanzu/velero/blob/main/pkg/restore/restore.go#L430) - Main restore execution that parses backup contents and processes namespace filters +- [`extractNamespacesFromBackup`](https://github.com/vmware-tanzu/velero/blob/main/pkg/restore/restore.go#L2407) - Extracts available namespaces from backup tar contents + +The `execute` function is the ideal integration point because it: +- Already parses the backup tar file to understand available resources +- Processes the user-specified namespace filters for the restore operation +- Can expand wildcard patterns against namespaces that actually exist in the backup +- Stores the resolved namespaces in new restore status fields for visibility + +This approach ensures wildcard namespaces in restore operations are based on actual backup contents rather than original backup specifications, providing safety and consistency regardless of how the backup was created. + ## Detailed Design The implementation involves four main components that can be developed incrementally: -### Add new status fields to the backup CRD to store expanded wildcard namespaces +### Add new status fields to the backup and restore CRDs to store expanded wildcard namespaces ```go // BackupStatus captures the current status of a Velero backup. type BackupStatus struct { // ... existing fields ... - // ExpandedIncludedNamespaces records the expanded include wildcard namespaces + // IncludeWildcardMatches records the expanded include wildcard namespaces // +optional // +nullable - ExpandedIncludedNamespaces []string `json:"expandedIncludedNamespaces,omitempty"` + IncludeWildcardMatches []string `json:"includeWildcardMatches,omitempty"` - // ExpandedExcludedNamespaces records the expanded exclude wildcard namespaces + // ExcludeWildcardMatches records the expanded exclude wildcard namespaces // +optional // +nullable - ExpandedExcludedNamespaces []string `json:"expandedExcludedNamespaces,omitempty"` + ExcludeWildcardMatches []string `json:"excludeWildcardMatches,omitempty"` + + // WildcardResult records the final namespaces after applying wildcard include/exclude logic + // +optional + // +nullable + WildcardResult []string `json:"wildcardResult,omitempty"` // ... other fields ... } ``` -**Implementation**: Added status fields `ExpandedIncludedNamespaces` and `ExpandedExcludedNamespaces` to `pkg/apis/velero/v1/backup_types.go` to track the resolved namespace lists after wildcard expansion. +**Implementation**: Added status fields `IncludeWildcardMatches`, `ExcludeWildcardMatches`, and `WildcardResult` to `pkg/apis/velero/v1/backup_types.go` and `pkg/apis/velero/v1/restore_types.go` to track the resolved namespace lists after wildcard expansion. ### Create a util package for wildcard expansion @@ -84,6 +106,8 @@ type BackupStatus struct { ### If required, expand wildcards and replace the request's includes and excludes with expanded namespaces +### Backup: + **Implementation**: In `pkg/backup/item_collector.go`: ```go @@ -102,17 +126,55 @@ The expansion occurs when collecting namespaces, after retrieving all active nam **Performance Improvement**: As part of this implementation, active namespaces are stored in a hashset rather than being iterated for each resolved/literal namespace check. This eliminates a [nested loop anti-pattern](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L767) and improves performance. +### Restore + +**Implementation**: In `pkg/restore/restore.go`: + +```go +// Lines 478-509: Wildcard expansion in restore context +if wildcard.ShouldExpandWildcards(ctx.restore.Spec.IncludedNamespaces, ctx.restore.Spec.ExcludedNamespaces) { + availableNamespaces := extractNamespacesFromBackup(backupResources) + expandedIncludes, expandedExcludes, err := wildcard.ExpandWildcards( + availableNamespaces, + ctx.restore.Spec.IncludedNamespaces, + ctx.restore.Spec.ExcludedNamespaces, + ) + // Update restore context with expanded patterns + ctx.namespaceIncludesExcludes = collections.NewIncludesExcludes(). + Includes(expandedIncludes...). + Excludes(expandedExcludes...) +} +``` + +The restore expansion occurs after parsing the backup tar contents, using `extractNamespacesFromBackup` to determine which namespaces are actually available for restoration. This ensures wildcard patterns are applied against materialized backup contents rather than original backup specifications. + + + ### Populate the expanded namespace status field with the namespaces +#### Backup Status Fields + **Implementation**: In `expandNamespaceWildcards` function (line 889-891): ```go // Record the expanded wildcard includes/excludes in the request status -r.backupRequest.Status.ExpandedIncludedNamespaces = expandedIncludes -r.backupRequest.Status.ExpandedExcludedNamespaces = expandedExcludes +r.backupRequest.Status.IncludeWildcardMatches = expandedIncludes +r.backupRequest.Status.ExcludeWildcardMatches = expandedExcludes +r.backupRequest.Status.WildcardResult = wildcardResult ``` -The status fields are populated immediately after successful wildcard expansion, providing visibility into which namespaces were actually matched by the wildcard patterns. +#### Restore Status Fields + +**Implementation**: In `pkg/restore/restore.go` (lines 499-502): + +```go +// Record the expanded wildcard includes/excludes in the restore status +ctx.restore.Status.IncludeWildcardMatches = expandedIncludes +ctx.restore.Status.ExcludeWildcardMatches = expandedExcludes +ctx.restore.Status.WildcardResult = wildcardResult +``` + +The status fields are populated immediately after successful wildcard expansion, providing visibility into which namespaces were actually matched by the wildcard patterns and the final list of namespaces that will be processed. ## Alternatives Considered @@ -141,14 +203,14 @@ This feature does not introduce any security vulnerabilities as it only affects ### Backward Compatibility The implementation maintains full backward compatibility with existing behavior: -- The standalone `*` character continues to work as before (include all namespaces, ignore excludes) +- The standalone "*" character continues to work as before (include all namespaces, ignore excludes) - Existing backup configurations remain unaffected ### Known Limitations -1. **Mixed `*` and wildcard usage**: When using the standalone `*` in includes, the legacy implementation takes precedence and wildcard expansion is skipped. This means you cannot combine "*" (all namespaces) in includes with wildcard patterns in excludes. +1. **Mixed "*" and wildcard usage**: When using the standalone "*" in includes, the legacy implementation takes precedence and wildcard expansion is skipped. This means you cannot combine "*" (all namespaces) in includes with wildcard patterns in excludes. -2. **Selective exclusion limitation**: The current design does not support selective pattern-based exclusion when including all namespaces via `*`. +2. **Selective exclusion limitation**: The current design does not support selective pattern-based exclusion when including all namespaces via "*". ## Implementation @@ -161,4 +223,6 @@ The implementation follows the detailed design outlined above, with the followin ## Open Issues -**Restore Integration**: Currently investigating how restore operations should handle wildcard-expanded backups. The proposed approach is to apply wildcard patterns against the expanded namespace list stored in the backup's status fields, ensuring consistency between backup and restore operations. \ No newline at end of file +**Restore Integration**: Restore operations are completely decoupled from backup wildcard specifications and work safely with wildcard-created backups. The restore process reads the actual tar file contents to determine available namespaces, not the original backup spec. Since the tar file contains real, materialized namespaces (e.g., `test01`, `test02`) rather than wildcard patterns (e.g., `test*`), restore operations work with concrete namespace names. + +When wildcard patterns are specified in restore operations, they are expanded against the namespaces that actually exist in the backup tar file. This ensures that restore wildcard expansion is based on what was actually backed up, not what was originally intended to be backed up. From eb8b3828161b7073132a414680e6a107c8c87a03 Mon Sep 17 00:00:00 2001 From: Joseph Date: Sun, 3 Aug 2025 09:21:33 -0400 Subject: [PATCH 07/13] Update design Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md index c807ad347..3d475d3a5 100644 --- a/design/wildcard-namespace-support-design.md +++ b/design/wildcard-namespace-support-design.md @@ -226,3 +226,6 @@ The implementation follows the detailed design outlined above, with the followin **Restore Integration**: Restore operations are completely decoupled from backup wildcard specifications and work safely with wildcard-created backups. The restore process reads the actual tar file contents to determine available namespaces, not the original backup spec. Since the tar file contains real, materialized namespaces (e.g., `test01`, `test02`) rather than wildcard patterns (e.g., `test*`), restore operations work with concrete namespace names. When wildcard patterns are specified in restore operations, they are expanded against the namespaces that actually exist in the backup tar file. This ensures that restore wildcard expansion is based on what was actually backed up, not what was originally intended to be backed up. + +**Crucially, restore does not treat `*` as a catch all case**. +This means that if `*` is mentioned in restore, wildcard expansion is skipped owing to how it's treated in backup. From 56df64b625e6debbc724baeff98fda794fb0653c Mon Sep 17 00:00:00 2001 From: Joseph Date: Tue, 5 Aug 2025 11:15:09 -0400 Subject: [PATCH 08/13] Status fields are part of a struct Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md index 3d475d3a5..13d0fdcb3 100644 --- a/design/wildcard-namespace-support-design.md +++ b/design/wildcard-namespace-support-design.md @@ -158,9 +158,11 @@ The restore expansion occurs after parsing the backup tar contents, using `extra ```go // Record the expanded wildcard includes/excludes in the request status -r.backupRequest.Status.IncludeWildcardMatches = expandedIncludes -r.backupRequest.Status.ExcludeWildcardMatches = expandedExcludes -r.backupRequest.Status.WildcardResult = wildcardResult +r.backupRequest.Status.WildcardNamespaces = &velerov1api.WildcardNamespaceStatus{ + IncludeWildcardMatches: expandedIncludes, + ExcludeWildcardMatches: expandedExcludes, + WildcardResult: wildcardResult, +} ``` #### Restore Status Fields @@ -169,9 +171,11 @@ r.backupRequest.Status.WildcardResult = wildcardResult ```go // Record the expanded wildcard includes/excludes in the restore status -ctx.restore.Status.IncludeWildcardMatches = expandedIncludes -ctx.restore.Status.ExcludeWildcardMatches = expandedExcludes -ctx.restore.Status.WildcardResult = wildcardResult +ctx.restore.Status.WildcardNamespaces = &velerov1api.WildcardNamespaceStatus{ + IncludeWildcardMatches: expandedIncludes, + ExcludeWildcardMatches: expandedExcludes, + WildcardResult: wildcardResult, +} ``` The status fields are populated immediately after successful wildcard expansion, providing visibility into which namespaces were actually matched by the wildcard patterns and the final list of namespaces that will be processed. From 528392ac5bcef96a651b510e591e5fbcd2ba6066 Mon Sep 17 00:00:00 2001 From: Joseph Date: Tue, 5 Aug 2025 12:59:28 -0400 Subject: [PATCH 09/13] Added struct change Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 64 +++++++++++++++++---- 1 file changed, 54 insertions(+), 10 deletions(-) diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md index 13d0fdcb3..d95390812 100644 --- a/design/wildcard-namespace-support-design.md +++ b/design/wildcard-namespace-support-design.md @@ -60,31 +60,75 @@ The implementation involves four main components that can be developed increment ### Add new status fields to the backup and restore CRDs to store expanded wildcard namespaces +#### Backup + ```go -// BackupStatus captures the current status of a Velero backup. -type BackupStatus struct { - // ... existing fields ... - - // IncludeWildcardMatches records the expanded include wildcard namespaces +// WildcardNamespaceStatus contains information about wildcard namespace matching results +type WildcardNamespaceStatus struct { + // IncludeWildcardMatches records the namespaces that matched include wildcard patterns // +optional // +nullable IncludeWildcardMatches []string `json:"includeWildcardMatches,omitempty"` - // ExcludeWildcardMatches records the expanded exclude wildcard namespaces + // ExcludeWildcardMatches records the namespaces that matched exclude wildcard patterns // +optional // +nullable ExcludeWildcardMatches []string `json:"excludeWildcardMatches,omitempty"` // WildcardResult records the final namespaces after applying wildcard include/exclude logic - // +optional - // +nullable - WildcardResult []string `json:"wildcardResult,omitempty"` + // +optional + // +nullable + WildcardResult []string `json:"wildcardResult,omitempty"` +} + +// BackupStatus captures the current status of a Velero backup. +type BackupStatus struct { + // ... existing fields ... + + // WildcardNamespaces contains information about wildcard namespace processing + // +optional + // +nullable + WildcardNamespaces *WildcardNamespaceStatus `json:"wildcardNamespaces,omitempty"` // ... other fields ... } ``` -**Implementation**: Added status fields `IncludeWildcardMatches`, `ExcludeWildcardMatches`, and `WildcardResult` to `pkg/apis/velero/v1/backup_types.go` and `pkg/apis/velero/v1/restore_types.go` to track the resolved namespace lists after wildcard expansion. +#### Restore + +```go +// WildcardNamespaceStatus contains information about wildcard namespace matching results +type WildcardNamespaceStatus struct { + // IncludeWildcardMatches records the namespaces that matched include wildcard patterns + // +optional + // +nullable + IncludeWildcardMatches []string `json:"includeWildcardMatches,omitempty"` + + // ExcludeWildcardMatches records the namespaces that matched exclude wildcard patterns + // +optional + // +nullable + ExcludeWildcardMatches []string `json:"excludeWildcardMatches,omitempty"` + + // WildcardResult records the final namespaces after applying wildcard include/exclude logic + // +optional + // +nullable + WildcardResult []string `json:"wildcardResult,omitempty"` +} + +// RestoreStatus captures the current status of a Velero restore. +type RestoreStatus struct { + // ... existing fields ... + + // WildcardNamespaces contains information about wildcard namespace processing + // +optional + // +nullable + WildcardNamespaces *WildcardNamespaceStatus `json:"wildcardNamespaces,omitempty"` + + // ... other fields ... +} +``` + +**Implementation**: Added a structured `WildcardNamespaceStatus` type and `WildcardNamespaces` field to `pkg/apis/velero/v1/backup_types.go` and `pkg/apis/velero/v1/restore_types.go` to track the resolved namespace lists after wildcard expansion in a well-organized manner. ### Create a util package for wildcard expansion From 0b40702900cda4a5fb23d526e8d788772828e291 Mon Sep 17 00:00:00 2001 From: Joseph Date: Mon, 22 Sep 2025 17:31:22 -0400 Subject: [PATCH 10/13] Updates Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 43 +++++++++------------ 1 file changed, 19 insertions(+), 24 deletions(-) diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md index d95390812..3de0e155c 100644 --- a/design/wildcard-namespace-support-design.md +++ b/design/wildcard-namespace-support-design.md @@ -1,11 +1,13 @@ -# Wildcard Namespace Includes/Excludes Support for Backups and Restores +# Wildcard (*) support for Include/Exclude Namespaces ## Abstract -Velero currently does not support wildcard characters in namespace specifications, requiring all namespaces to be specified as string literals. The only exception is the standalone "*" character, which includes all namespaces and ignores excludes. +- Velero currently does **not** support wildcard characters in namespace specifications, they must be specified as string literals +- The only exception is the standalone `*` character, which includes all namespaces and ignores excludes -This document details the approach to implementing wildcard namespace support for `--include-namespaces` and `--exclude-namespaces` flags, while preserving the existing "*" behavior for backward compatibility. + - This design details an approach to implementing wildcard namespace support for `--include-namespaces` and `--exclude-namespaces` flags +- Preserves standalone `*` for backwards compatibility ## Background @@ -14,32 +16,33 @@ This feature was requested in Issue [#1874](https://github.com/vmware-tanzu/vele ## Goals - Add support for wildcard patterns in `--include-namespaces` and `--exclude-namespaces` flags for both backup and restore -- Ensure legacy "*" behavior remains unchanged for backward compatibility +- Ensure existing `*` behavior remains unchanged for backward compatibility ## Non-Goals - Completely rethinking the way "*" is treated and allowing it to work with wildcard excludes -- Supporting complex regex patterns beyond basic glob patterns +- Supporting complex regex patterns beyond basic glob patterns: this could be explored later ## High-Level Design -## Backup +### NamespaceIncludesExcludes struct -The wildcard expansion implementation focuses on two key functions in `pkg/backup/item_collector.go`: +- `NamespaceIncludesExcludes` is a wrapper around `IncludesExcludes` +- It has a field `activeNamespaces`, which will be used by the wildcard expansion logic to run the expansion -- [`collectNamespaces`](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L742) - Retrieves all active namespaces and processes include/exclude filters -- [`getNamespacesToList`](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L638) - Resolves namespace includes/excludes to final list +### Backup -The `collectNamespaces` function is the ideal integration point because it: -- Already retrieves all active namespaces from the cluster -- Processes the user-specified namespace filters -- Can expand wildcard patterns against the complete namespace list -- Stores the resolved namespaces in new backup status fields for visibility +The wildcard expansion implementation focuses on -This approach ensures wildcard namespaces are handled consistently with the existing "*" behavior, bypassing individual namespace existence checks. +- [`getNamespacesToList`](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L638) - Resolves namespace includes/excludes to final list. -## Restore +- Wildcard expansion is conditionally run when the `NamespaceIncludesExcludes.ResolveNamespaceList()` func is run. + - It overwrites the `NamespaceIncludeExclude` struct's `Includes` and `Excludes` with the expanded string literal namespaces + - If there are no candidate strings for expansion, it proceeds normally + + +### Restore The wildcard expansion implementation for restore operations focuses on the main execution flow in `pkg/restore/restore.go`: @@ -154,14 +157,6 @@ type RestoreStatus struct { **Implementation**: In `pkg/backup/item_collector.go`: -```go -// collectNamespaces function (line 748-803) -if wildcard.ShouldExpandWildcards(namespaceSelector.GetIncludes(), namespaceSelector.GetExcludes()) { - if err := r.expandNamespaceWildcards(activeNamespacesList, namespaceSelector); err != nil { - return nil, errors.WithMessage(err, "failed to expand namespace wildcard patterns") - } -} -``` The expansion occurs when collecting namespaces, after retrieving all active namespaces from the cluster. The `expandNamespaceWildcards` method: - Calls `wildcard.ExpandWildcards()` with active namespaces and original patterns From 14a6315667a2e3e2ead461f331fcabbaf2defa24 Mon Sep 17 00:00:00 2001 From: Joseph Date: Thu, 25 Sep 2025 04:09:59 -0400 Subject: [PATCH 11/13] update Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md index 3de0e155c..be293b39e 100644 --- a/design/wildcard-namespace-support-design.md +++ b/design/wildcard-namespace-support-design.md @@ -29,17 +29,25 @@ This feature was requested in Issue [#1874](https://github.com/vmware-tanzu/vele ### NamespaceIncludesExcludes struct - `NamespaceIncludesExcludes` is a wrapper around `IncludesExcludes` -- It has a field `activeNamespaces`, which will be used by the wildcard expansion logic to run the expansion +- It has a field `activeNamespaces`, which will be used to match wildcard patterns against active namespaces ### Backup -The wildcard expansion implementation focuses on +The wildcard expansion implementation expands the patterns before the normal flow. It focuses on two key functions in `pkg/backup/item_collector.go`: -- [`getNamespacesToList`](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L638) - Resolves namespace includes/excludes to final list. +- [`getResourceItems`](https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_collector.go#L356) - Integration point +- [`collectNamespaces`](https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_collector.go#L742) +- [`getNamespacesToList`](https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_collector.go#L638) -- Wildcard expansion is conditionally run when the `NamespaceIncludesExcludes.ResolveNamespaceList()` func is run. - - It overwrites the `NamespaceIncludeExclude` struct's `Includes` and `Excludes` with the expanded string literal namespaces - - If there are no candidate strings for expansion, it proceeds normally +The `getResourceItems` function is the ideal integration point, as the other two funcs depend on it + +Wildcard expansion is conditionally run when the [`NamespaceIncludesExcludes.ResolveNamespaceList()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/collections/includes_excludes.go#L146) function is called: +- It calls [`wildcard.ShouldExpandWildcards()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/wildcard/wildcard.go) to determine if expansion is needed +- If wildcards are detected, it uses [`wildcard.ExpandWildcards()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/wildcard/wildcard.go) to resolve them against active namespaces +- It overwrites the `NamespaceIncludesExcludes` struct's `Includes` and `Excludes` with the expanded literal namespaces using [`SetIncludes()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/collections/includes_excludes.go#L100) and [`SetExcludes()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/collections/includes_excludes.go#L107) +- If no wildcard patterns are found, it proceeds with normal glob-based processing via [`resolveNamespaceListTraditional()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/collections/includes_excludes.go#L170) + +This approach ensures wildcard namespaces are handled consistently with the existing "*" behavior, bypassing individual namespace existence checks. ### Restore From cf7a9495c5f82cfcc1bd3c2f0ae8d0150cc3761d Mon Sep 17 00:00:00 2001 From: Joseph Date: Thu, 25 Sep 2025 04:40:13 -0400 Subject: [PATCH 12/13] Leaner design Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 53 ++++++++------------- 1 file changed, 21 insertions(+), 32 deletions(-) diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md index be293b39e..6f81af668 100644 --- a/design/wildcard-namespace-support-design.md +++ b/design/wildcard-namespace-support-design.md @@ -3,11 +3,8 @@ ## Abstract -- Velero currently does **not** support wildcard characters in namespace specifications, they must be specified as string literals -- The only exception is the standalone `*` character, which includes all namespaces and ignores excludes - - - This design details an approach to implementing wildcard namespace support for `--include-namespaces` and `--exclude-namespaces` flags -- Preserves standalone `*` for backwards compatibility +- Velero currently does **not** support wildcard characters in namespace include/exclude specs, they must be specified as string literals +- This design details an approach to implementing wildcard namespace support ## Background @@ -20,7 +17,6 @@ This feature was requested in Issue [#1874](https://github.com/vmware-tanzu/vele ## Non-Goals -- Completely rethinking the way "*" is treated and allowing it to work with wildcard excludes - Supporting complex regex patterns beyond basic glob patterns: this could be explored later @@ -33,21 +29,28 @@ This feature was requested in Issue [#1874](https://github.com/vmware-tanzu/vele ### Backup -The wildcard expansion implementation expands the patterns before the normal flow. It focuses on two key functions in `pkg/backup/item_collector.go`: +The wildcard expansion implementation expands the patterns before the normal execution of the backup. +The goal here is ensuring wildcard namespaces are expanded before they're required. +It focuses on two key functions in `pkg/backup/item_collector.go`: -- [`getResourceItems`](https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_collector.go#L356) - Integration point - [`collectNamespaces`](https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_collector.go#L742) - [`getNamespacesToList`](https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_collector.go#L638) -The `getResourceItems` function is the ideal integration point, as the other two funcs depend on it +Call to expand wildcards is made at: +- [`getResourceItems`](https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_collector.go#L356) - Integration point + Wildcard expansion is conditionally run when the [`NamespaceIncludesExcludes.ResolveNamespaceList()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/collections/includes_excludes.go#L146) function is called: + +unc (nie *NamespaceIncludesExcludes) ResolveNamespaceList() ([]string, bool, error) { + - It calls [`wildcard.ShouldExpandWildcards()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/wildcard/wildcard.go) to determine if expansion is needed - If wildcards are detected, it uses [`wildcard.ExpandWildcards()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/wildcard/wildcard.go) to resolve them against active namespaces - It overwrites the `NamespaceIncludesExcludes` struct's `Includes` and `Excludes` with the expanded literal namespaces using [`SetIncludes()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/collections/includes_excludes.go#L100) and [`SetExcludes()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/collections/includes_excludes.go#L107) -- If no wildcard patterns are found, it proceeds with normal glob-based processing via [`resolveNamespaceListTraditional()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/collections/includes_excludes.go#L170) +- It returns the effective list of namespaces to be backed up and a bool indicating whether wildcard expansion occured + + -This approach ensures wildcard namespaces are handled consistently with the existing "*" behavior, bypassing individual namespace existence checks. ### Restore @@ -231,12 +234,11 @@ The status fields are populated immediately after successful wildcard expansion, Several implementation approaches were considered for the wildcard expansion logic: -### 1. Wildcard expansion in `getNamespacesToList` +### 1. Wildcard expansion in `getNamespacesToList` or `collectNamespaces` This approach was ruled out because: -- The `collectNamespaces` function encounters wildcard patterns first in the processing flow -- `collectNamespaces` already has access to the complete list of active namespaces, eliminating the need for additional API calls -- Placing the logic in `getNamespacesToList` would require redundant namespace retrieval +- Both funcs expect non-wildcard namespaces in the NamespaceIncludesExcludes struct +- Calling it at a higher level within getResourceItems is ideal ### 2. Client-side wildcard expansion @@ -254,29 +256,16 @@ This feature does not introduce any security vulnerabilities as it only affects ### Backward Compatibility The implementation maintains full backward compatibility with existing behavior: -- The standalone "*" character continues to work as before (include all namespaces, ignore excludes) -- Existing backup configurations remain unaffected +- The standalone "*" character continues to work as before ### Known Limitations -1. **Mixed "*" and wildcard usage**: When using the standalone "*" in includes, the legacy implementation takes precedence and wildcard expansion is skipped. This means you cannot combine "*" (all namespaces) in includes with wildcard patterns in excludes. - -2. **Selective exclusion limitation**: The current design does not support selective pattern-based exclusion when including all namespaces via "*". +N/A ## Implementation -The implementation follows the detailed design outlined above, with the following timeline: - -1. **Phase 1**: Add backup CRD status fields for expanded namespaces -2. **Phase 2**: Implement wildcard utility package with validation -3. **Phase 3**: Integrate expansion logic into backup item collection -4. **Phase 4**: Add status field population and logging +Aiming for 1.18 ## Open Issues -**Restore Integration**: Restore operations are completely decoupled from backup wildcard specifications and work safely with wildcard-created backups. The restore process reads the actual tar file contents to determine available namespaces, not the original backup spec. Since the tar file contains real, materialized namespaces (e.g., `test01`, `test02`) rather than wildcard patterns (e.g., `test*`), restore operations work with concrete namespace names. - -When wildcard patterns are specified in restore operations, they are expanded against the namespaces that actually exist in the backup tar file. This ensures that restore wildcard expansion is based on what was actually backed up, not what was originally intended to be backed up. - -**Crucially, restore does not treat `*` as a catch all case**. -This means that if `*` is mentioned in restore, wildcard expansion is skipped owing to how it's treated in backup. +N/A From 75f1817cba52b7cd36a56a8ece3d97669a884936 Mon Sep 17 00:00:00 2001 From: Joseph Date: Thu, 25 Sep 2025 09:28:20 -0400 Subject: [PATCH 13/13] Simplify Signed-off-by: Joseph --- design/wildcard-namespace-support-design.md | 272 +++++--------------- 1 file changed, 58 insertions(+), 214 deletions(-) diff --git a/design/wildcard-namespace-support-design.md b/design/wildcard-namespace-support-design.md index 6f81af668..45c4d4641 100644 --- a/design/wildcard-namespace-support-design.md +++ b/design/wildcard-namespace-support-design.md @@ -1,271 +1,115 @@ -# Wildcard (*) support for Include/Exclude Namespaces +# Wildcard Namespace Support ## Abstract -- Velero currently does **not** support wildcard characters in namespace include/exclude specs, they must be specified as string literals -- This design details an approach to implementing wildcard namespace support +Velero currently treats namespace patterns with glob characters as literal strings. This design adds wildcard expansion to support flexible namespace selection using patterns like `app-*` or `test-{dev,staging}`. ## Background -This feature was requested in Issue [#1874](https://github.com/vmware-tanzu/velero/issues/1874) to provide more flexible namespace selection using wildcard patterns. +Requested in [#1874](https://github.com/vmware-tanzu/velero/issues/1874) for more flexible namespace selection. ## Goals -- Add support for wildcard patterns in `--include-namespaces` and `--exclude-namespaces` flags for both backup and restore -- Ensure existing `*` behavior remains unchanged for backward compatibility +- Support glob pattern expansion in namespace includes/excludes +- Maintain backward compatibility with existing `*` behavior ## Non-Goals -- Supporting complex regex patterns beyond basic glob patterns: this could be explored later - +- Complex regex patterns beyond basic globs ## High-Level Design -### NamespaceIncludesExcludes struct +Wildcard expansion occurs early in both backup and restore flows, converting patterns to literal namespace lists before normal processing. -- `NamespaceIncludesExcludes` is a wrapper around `IncludesExcludes` -- It has a field `activeNamespaces`, which will be used to match wildcard patterns against active namespaces +### Backup Flow -### Backup +Expansion happens in `getResourceItems()` before namespace collection: +1. Check if wildcards exist using `ShouldExpandWildcards()` +2. Expand patterns against active cluster namespaces +3. Replace includes/excludes with expanded literal namespaces +4. Continue with normal backup processing -The wildcard expansion implementation expands the patterns before the normal execution of the backup. -The goal here is ensuring wildcard namespaces are expanded before they're required. -It focuses on two key functions in `pkg/backup/item_collector.go`: +### Restore Flow -- [`collectNamespaces`](https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_collector.go#L742) -- [`getNamespacesToList`](https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_collector.go#L638) +Expansion occurs in `execute()` after parsing backup contents: +1. Extract available namespaces from backup tar +2. Expand patterns against backup namespaces (not cluster namespaces) +3. Update restore context with expanded namespaces +4. Continue with normal restore processing -Call to expand wildcards is made at: -- [`getResourceItems`](https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_collector.go#L356) - Integration point - - -Wildcard expansion is conditionally run when the [`NamespaceIncludesExcludes.ResolveNamespaceList()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/collections/includes_excludes.go#L146) function is called: - -unc (nie *NamespaceIncludesExcludes) ResolveNamespaceList() ([]string, bool, error) { - -- It calls [`wildcard.ShouldExpandWildcards()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/wildcard/wildcard.go) to determine if expansion is needed -- If wildcards are detected, it uses [`wildcard.ExpandWildcards()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/wildcard/wildcard.go) to resolve them against active namespaces -- It overwrites the `NamespaceIncludesExcludes` struct's `Includes` and `Excludes` with the expanded literal namespaces using [`SetIncludes()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/collections/includes_excludes.go#L100) and [`SetExcludes()`](https://github.com/vmware-tanzu/velero/blob/main/pkg/util/collections/includes_excludes.go#L107) -- It returns the effective list of namespaces to be backed up and a bool indicating whether wildcard expansion occured - - - - - -### Restore - -The wildcard expansion implementation for restore operations focuses on the main execution flow in `pkg/restore/restore.go`: - -- [`execute`](https://github.com/vmware-tanzu/velero/blob/main/pkg/restore/restore.go#L430) - Main restore execution that parses backup contents and processes namespace filters -- [`extractNamespacesFromBackup`](https://github.com/vmware-tanzu/velero/blob/main/pkg/restore/restore.go#L2407) - Extracts available namespaces from backup tar contents - -The `execute` function is the ideal integration point because it: -- Already parses the backup tar file to understand available resources -- Processes the user-specified namespace filters for the restore operation -- Can expand wildcard patterns against namespaces that actually exist in the backup -- Stores the resolved namespaces in new restore status fields for visibility - -This approach ensures wildcard namespaces in restore operations are based on actual backup contents rather than original backup specifications, providing safety and consistency regardless of how the backup was created. +This ensures restore wildcards match actual backup contents, not current cluster state. ## Detailed Design -The implementation involves four main components that can be developed incrementally: +### Status Fields -### Add new status fields to the backup and restore CRDs to store expanded wildcard namespaces - -#### Backup +Add wildcard expansion tracking to backup and restore CRDs: ```go -// WildcardNamespaceStatus contains information about wildcard namespace matching results type WildcardNamespaceStatus struct { - // IncludeWildcardMatches records the namespaces that matched include wildcard patterns + // IncludeWildcardMatches records namespaces that matched include patterns // +optional - // +nullable IncludeWildcardMatches []string `json:"includeWildcardMatches,omitempty"` - - // ExcludeWildcardMatches records the namespaces that matched exclude wildcard patterns + + // ExcludeWildcardMatches records namespaces that matched exclude patterns // +optional - // +nullable ExcludeWildcardMatches []string `json:"excludeWildcardMatches,omitempty"` - - // WildcardResult records the final namespaces after applying wildcard include/exclude logic + + // WildcardResult records final namespaces after wildcard processing // +optional - // +nullable WildcardResult []string `json:"wildcardResult,omitempty"` } -// BackupStatus captures the current status of a Velero backup. +// Added to both BackupStatus and RestoreStatus type BackupStatus struct { - // ... existing fields ... - - // WildcardNamespaces contains information about wildcard namespace processing + // WildcardNamespaces contains wildcard expansion results // +optional - // +nullable WildcardNamespaces *WildcardNamespaceStatus `json:"wildcardNamespaces,omitempty"` - - // ... other fields ... } ``` -#### Restore +### Wildcard Expansion Package +New `pkg/util/wildcard/expand.go` package provides: + +- `ShouldExpandWildcards()` - Skip expansion for simple "*" case +- `ExpandWildcards()` - Main expansion function using `github.com/gobwas/glob` +- Pattern validation rejecting unsupported regex symbols + +**Supported patterns**: `*`, `?`, `[abc]`, `{a,b,c}` +**Unsupported**: `|()`, `**` + +### Implementation Details + +#### Backup Integration (`pkg/backup/item_collector.go`) + +Expansion in `getResourceItems()`: +- Call `wildcard.ExpandWildcards()` with cluster namespaces +- Update `NamespaceIncludesExcludes` with expanded results +- Populate status fields with expansion results + +#### Restore Integration (`pkg/restore/restore.go`) + +Expansion in `execute()`: ```go -// WildcardNamespaceStatus contains information about wildcard namespace matching results -type WildcardNamespaceStatus struct { - // IncludeWildcardMatches records the namespaces that matched include wildcard patterns - // +optional - // +nullable - IncludeWildcardMatches []string `json:"includeWildcardMatches,omitempty"` - - // ExcludeWildcardMatches records the namespaces that matched exclude wildcard patterns - // +optional - // +nullable - ExcludeWildcardMatches []string `json:"excludeWildcardMatches,omitempty"` - - // WildcardResult records the final namespaces after applying wildcard include/exclude logic - // +optional - // +nullable - WildcardResult []string `json:"wildcardResult,omitempty"` -} - -// RestoreStatus captures the current status of a Velero restore. -type RestoreStatus struct { - // ... existing fields ... - - // WildcardNamespaces contains information about wildcard namespace processing - // +optional - // +nullable - WildcardNamespaces *WildcardNamespaceStatus `json:"wildcardNamespaces,omitempty"` - - // ... other fields ... -} -``` - -**Implementation**: Added a structured `WildcardNamespaceStatus` type and `WildcardNamespaces` field to `pkg/apis/velero/v1/backup_types.go` and `pkg/apis/velero/v1/restore_types.go` to track the resolved namespace lists after wildcard expansion in a well-organized manner. - -### Create a util package for wildcard expansion - -**Implementation**: Created `pkg/util/wildcard/expand.go` package containing: - -- `ShouldExpandWildcards(includes, excludes []string) bool` - Determines if wildcard expansion is needed (excludes simple "*" case) -- `ExpandWildcards(activeNamespaces, includes, excludes []string) ([]string, []string, error)` - Main expansion function -- `containsWildcardPattern(pattern string) bool` - Detects wildcard patterns (`*`, `?`, `[abc]`, `{a,b,c}`) -- `validateWildcardPatterns(patterns []string) error` - Validates patterns and rejects unsupported regex symbols -- Uses `github.com/gobwas/glob` library for glob pattern matching - -**Supported patterns**: -- `*` (any characters) -- `?` (single character) -- `[abc]` (character classes) -- `{a,b,c}` (alternatives) - -**Unsupported**: Regex symbols `|()`, consecutive asterisks `**` - -### If required, expand wildcards and replace the request's includes and excludes with expanded namespaces - -### Backup: - -**Implementation**: In `pkg/backup/item_collector.go`: - - -The expansion occurs when collecting namespaces, after retrieving all active namespaces from the cluster. The `expandNamespaceWildcards` method: -- Calls `wildcard.ExpandWildcards()` with active namespaces and original patterns -- Updates the namespace selector with expanded results using `SetIncludes()` and `SetExcludes()` -- Preserves backward compatibility by skipping expansion for simple "*" pattern - -**Performance Improvement**: As part of this implementation, active namespaces are stored in a hashset rather than being iterated for each resolved/literal namespace check. This eliminates a [nested loop anti-pattern](https://github.com/vmware-tanzu/velero/blob/1535afb45e33a3d3820088e4189800a21ba55293/pkg/backup/item_collector.go#L767) and improves performance. - -### Restore - -**Implementation**: In `pkg/restore/restore.go`: - -```go -// Lines 478-509: Wildcard expansion in restore context -if wildcard.ShouldExpandWildcards(ctx.restore.Spec.IncludedNamespaces, ctx.restore.Spec.ExcludedNamespaces) { +if wildcard.ShouldExpandWildcards(includes, excludes) { availableNamespaces := extractNamespacesFromBackup(backupResources) expandedIncludes, expandedExcludes, err := wildcard.ExpandWildcards( - availableNamespaces, - ctx.restore.Spec.IncludedNamespaces, - ctx.restore.Spec.ExcludedNamespaces, - ) - // Update restore context with expanded patterns - ctx.namespaceIncludesExcludes = collections.NewIncludesExcludes(). - Includes(expandedIncludes...). - Excludes(expandedExcludes...) + availableNamespaces, includes, excludes) + // Update context and status } ``` -The restore expansion occurs after parsing the backup tar contents, using `extractNamespacesFromBackup` to determine which namespaces are actually available for restoration. This ensures wildcard patterns are applied against materialized backup contents rather than original backup specifications. - - - -### Populate the expanded namespace status field with the namespaces - -#### Backup Status Fields - -**Implementation**: In `expandNamespaceWildcards` function (line 889-891): - -```go -// Record the expanded wildcard includes/excludes in the request status -r.backupRequest.Status.WildcardNamespaces = &velerov1api.WildcardNamespaceStatus{ - IncludeWildcardMatches: expandedIncludes, - ExcludeWildcardMatches: expandedExcludes, - WildcardResult: wildcardResult, -} -``` - -#### Restore Status Fields - -**Implementation**: In `pkg/restore/restore.go` (lines 499-502): - -```go -// Record the expanded wildcard includes/excludes in the restore status -ctx.restore.Status.WildcardNamespaces = &velerov1api.WildcardNamespaceStatus{ - IncludeWildcardMatches: expandedIncludes, - ExcludeWildcardMatches: expandedExcludes, - WildcardResult: wildcardResult, -} -``` - -The status fields are populated immediately after successful wildcard expansion, providing visibility into which namespaces were actually matched by the wildcard patterns and the final list of namespaces that will be processed. - ## Alternatives Considered -Several implementation approaches were considered for the wildcard expansion logic: - -### 1. Wildcard expansion in `getNamespacesToList` or `collectNamespaces` - -This approach was ruled out because: -- Both funcs expect non-wildcard namespaces in the NamespaceIncludesExcludes struct -- Calling it at a higher level within getResourceItems is ideal - -### 2. Client-side wildcard expansion - -Expanding wildcards on the CLI side was considered but rejected because: -- It would only work for command-line initiated backups -- Scheduled backups would not benefit from this approach -- The server-side approach provides consistent behavior across all backup initiation methods - -## Security Considerations - -This feature does not introduce any security vulnerabilities as it only affects namespace selection logic within the existing backup authorization framework. +1. **Client-side expansion**: Rejected because it wouldn't work for scheduled backups +2. **Expansion in `collectNamespaces`**: Rejected because these functions expect literal namespaces ## Compatibility -### Backward Compatibility - -The implementation maintains full backward compatibility with existing behavior: -- The standalone "*" character continues to work as before - -### Known Limitations - -N/A +Maintains full backward compatibility - existing "*" behavior unchanged. ## Implementation -Aiming for 1.18 - -## Open Issues - -N/A +Target: Velero 1.18