This commit fixes the issue that when a namespace with label velero.io/exclude-from-backup: "true"
is added to the exludeNamespaces of a backup CR. There will be
duplicated entries of the namespace in the spec of the backup.
Signed-off-by: Daniel Jiang <daniel.jiang@broadcom.com>
* Fix backup-finalizer: do not set backup phase to Completed before PutBackupMetadata succeeds
Previously, the backup finalizer controller set backup.Status.Phase to
Completed/PartiallyFailed in-memory BEFORE calling PutBackupMetadata and
PutBackupContents. When these uploads failed (e.g., due to object lock
or immutability), the deferred patch function still wrote the terminal
phase to the Kubernetes API server, preventing the controller from
retrying the upload on the next reconcile.
This fix moves the phase assignment to AFTER both uploads succeed. A
DeepCopy of the backup is used to encode the JSON with the final phase
for object storage, while the in-memory backup object retains the
Finalizing phase until uploads complete.
Caveats:
- CompletionTimestamp is now captured before upload but only committed to
the API server after upload succeeds. On retry after a transient
failure, a new timestamp is generated, so the completion time reflects
when the upload finally succeeded rather than when finalization
processing completed.
- Metrics (RegisterBackupSuccess/RegisterBackupPartialFailure) are now
recorded after uploads succeed, so they accurately reflect only fully
persisted backups.
- The metadata uploaded to object storage contains the final phase and
completion timestamp via DeepCopy, so storage state is correct even
before the API server is patched.
Fixes#9645
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Add changelog for #9646
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Fix testifylint: use require.Error instead of assert.Error
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Address review feedback on backup-finalizer fix
- Add default guard for unhandled phase values in finalPhase switch
- Add retry with DefaultBackoff for PutBackupMetadata per reviewer request
- Replace brittle framework.BackupItemActionResolverV2{} mock with mock.Anything
- Add FinalizingPartiallyFailed test case for PutBackupContents failure
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Use bounded, object-storage-tuned backoff for backup-finalizer uploads
retry.DefaultBackoff is tuned for API server optimistic-concurrency
conflicts (4 steps, ~1.25s total) and gives up far too quickly for
object storage calls, which can see longer transient outages or
throttling (review feedback from blackpiglet). Replace it with a
dedicated, bounded backoff (1s base, 2x factor, 5 steps, ~31s total)
applied to both PutBackupMetadata and PutBackupContents.
Being bounded (rather than retrying forever) means a persistent
failure, e.g. an object-lock/immutability policy denying every write,
surfaces as an error within a bounded time instead of hanging the
reconcile indefinitely; controller-runtime requeues on error, so
retries continue across reconciles (review feedback from priyansh17).
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Fix PutBackupMetadata retry to re-read backupJSON each attempt
backupJSON is a bytes.Buffer, so passing it directly to
PutBackupMetadata drains it on the first read attempt. A retry after
a transient failure would then upload empty content instead of the
backup metadata. Wrap it in bytes.NewReader(backupJSON.Bytes()) inside
the retry closure so every attempt gets a fresh reader.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
---------
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Happy <yesreply@happy.engineering>
Update Request struct initialization in backup_test.go to use
SkippedVolumeTracker and NewSkipVolumeTracker following the rename
from SkippedPVTracker.
Signed-off-by: Adam Zhang <adam.zhang@broadcom.com>
PodVolumeRestores are only created for pods that Velero creates, so when
the pod already exists in the cluster the PVC-not-in-use pre-flight check
never runs and the volume data restore is skipped silently, while the
existing pod keeps consuming the PVC. Report an explicit pre-flight error
for such pods, aligned with the PVC CSI RIA behavior.
Signed-off-by: chlins <chlins.zhang@gmail.com>
Motivation:
Forget and BatchForget in pkg/repository/manager/manager.go accept a
caller-provided context.Context but ignore it, hardcoding
context.Background() when calling into the repository provider. This
means cancellation and timeouts set by callers (e.g. the backup
deletion controller) are silently dropped during repository connection
and snapshot deletion. Additionally, BatchForget returned a wrapped nil
instead of the real connection error when prd.BoostRepoConnect failed,
because it referenced an unrelated, already-nil err variable instead of
connectErr.
Approach:
Pass the caller's ctx through to prd.BoostRepoConnect, prd.Forget, and
prd.BatchForget in both Forget and BatchForget, instead of substituting
context.Background(). Fix BatchForget's connection-failure branch to
wrap and return connectErr instead of the stale err. Other methods on
manager (InitRepo, ConnectToRepo, PrepareRepo, PruneRepo, UnlockRepo)
don't accept a ctx parameter at all, so they are unaffected and out of
scope for this change.
Validation:
- go build ./pkg/repository/... and go build ./... pass.
- go vet ./pkg/repository/... is clean.
- go test ./pkg/repository/... passes, including three new tests added
to pkg/repository/manager/manager_test.go.
- golangci-lint run ./pkg/repository/... is clean.
- Confirmed the new tests reproduce both bugs: temporarily reverting
only manager.go and re-running go test ./pkg/repository/manager/...
made all three new tests fail (missing propagated context value and
cancellation, and a nil error returned where the real connect error
was expected); re-applying the fix makes them pass. This is a silent
behavior bug (broken context propagation and a swallowed error), not
a crash.
Report: https://github.com/velero-io/velero/issues/10551
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Assisted-by: claude-sonnet-5 (via Claude Code)
velero schedule create registers --annotations through BackupOptions.BindFlags,
but the Schedule ObjectMeta it builds only sets Labels, so the flag was accepted
and silently discarded.
This matters beyond the Schedule object itself: BackupBuilder.FromSchedule
falls back to schedule.Annotations when the template carries none, so every
backup generated by the schedule lost the annotations too. velero schedule
describe already prints these fields and the Schedule CRD already carries them,
so the create path was the only gap.
Same shape as #10526, which fixed the backup type being dropped on the same
struct literal.
Signed-off-by: Jeremy Schoemaker <jeremy@shoemoney.com>
Add includedNamespacesByLabel, excludedNamespacesByLabel, and
labelSelectorLogic to IncludeExcludePolicy in the ResourcePolicy
ConfigMap (realizes design in velero-io/velero#9772), letting a backup
select or exclude namespaces by label instead of (or in addition to)
name/wildcard.
The backup controller resolves label selectors against the live
namespace list once per backup, merges the results into
spec.includedNamespaces/excludedNamespaces, then proceeds through the
existing name-based filtering unchanged. A defaulted "*" include list
is replaced by the resolved set; an explicitly-configured include list
(including an explicit "*") is unioned with it instead, and stays
canonical rather than widening. Namespaces matching an exclude
selector are always subtracted from the merged includes, regardless of
how the includes were populated.
Because Velero's namespace-includes/excludes model requires at least
one name (an empty list means "match everything"), a selector that
resolves to zero namespaces is represented with a sentinel glob
pattern ("[-]*") guaranteed to match no real namespace, rather than an
empty list that would silently fall back to including/excluding
everything.
labelSelectorLogic ("AND"/"OR", case-insensitive) controls whether
multiple included/excluded label selectors are combined by
intersection or union; it is validated up front, including inside
ResolveNamespacesByLabel itself, so an invalid value fails fast instead
of silently falling through to OR semantics.
Namespace-selection-by-label and resource-selection-by-label act as
independent axes and do not affect each other, matching the design
discussion in #9772.
Known limitations:
- Selectors are evaluated once per backup against the namespace list
at that point in time; namespaces created or relabeled mid-backup
are not picked up.
- Backup-only for now; restore-side namespace mapping is unaffected.
Testing:
- Unit coverage in internal/resourcepolicies for validation, selector
resolution (including AND/OR logic, case-insensitivity, and
malformed-selector/invalid-logic error paths), and the no-match
sentinel.
- Unit coverage in pkg/controller for the merge logic between resolved
label selections and explicit/defaulted includes and excludes.
- End-to-end coverage in pkg/backup exercising the full backup
pipeline with label-selected namespaces, including the
velero.io/exclude-from-backup hard-exclusion interaction and the
zero-match/fully-excluded sentinel path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Modify according to comments.
* Rename the pvSkipTracker and related fields to indicate both PVC and PV are supported.
Signed-off-by: Xun Jiang <xun.jiang@broadcom.com>
* Fix schedule reconciler to compare SkipImmediately and LastSkipped by value
The reconciler compared spec.skipImmediately (*bool) and
status.lastSkipped (*metav1.Time) between the live object and its
DeepCopy by pointer, so both checks always reported a change once the
fields were set, and every reconcile issued a redundant Patch. Compare
by value instead (ptr.Equal / equality.Semantic.DeepEqual) and add a
regression test pinning that the skip-once flip persists with exactly
one patch, later reconciles of persisted state patch zero times, and
an explicit false is a no-op.
Signed-off-by: HeonJe LEE <lhjnano@gmail.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: lyndon-li <98304688+Lyndon-Li@users.noreply.github.com>
---------
Signed-off-by: HeonJe LEE <lhjnano@gmail.com>
Signed-off-by: lyndon-li <98304688+Lyndon-Li@users.noreply.github.com>
Co-authored-by: lyndon-li <98304688+Lyndon-Li@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The pre-flight check that verifies the existing PVC is still bound to the
backed-up volume compared PV names. A block data mover restore of a file
system volume recreates the PV under a new name (the volumeMode field is
immutable), so a second in-place restore of the same workload failed the
check even though the PVC was bound to the very same volume.
Record the CSI volume handle in the backup volume info (PVInfo) and
compare handles when both the backup and the bound PV record one; the
PV name remains the fallback for non-CSI volumes and for backups taken
before the handle was recorded. The handle reaches the PVC CSI RIA
through the same carrier annotation mechanism as the source size.
Signed-off-by: chlins <chlins.zhang@gmail.com>
* Fix backup queue permanently stuck when a dequeued backup completes during the patch
Motivation: backupQueueReconciler patched a dequeued backup to ReadyToStart and
only called backupTracker.AddReadyToStart on the next line. If backupReconciler
picked up that patch and completed the backup (e.g. immediate FailedValidation
while a BackupStorageLocation is briefly unavailable) before the queue
controller reached that line, backupReconciler's Add + deferred Delete ran
first, and the later AddReadyToStart re-inserted a tracker key nothing would
ever delete again. backupTracker is in-memory and never reconciled against
actual Backup phases, so RunningCount() stayed stuck at concurrentBackups and
every later reconcile, including the periodic recheck, was refused at that
gate -- the queue stopped dequeuing permanently until the deployment restarted.
Approach: record the backup as ReadyToStart in the tracker before patching it,
and roll that back if the patch itself fails, so the tracker entry always
exists before the backup can become visible to any other reconciler. Also
folds in two related fixes: the concurrency-refusal log line is now Info
instead of Debug so a stuck queue is visible at the default log level, and the
queue-position renumbering loop's error log (which built a logrus.Entry via
log.WithError(errors.Wrapf(...)) but never called a terminal method on it, so
it never actually logged anything) now emits properly.
Validation: go build ./pkg/controller/..., go vet ./pkg/controller/..., and
gofmt -l on both changed files are all clean. golangci-lint run
./pkg/controller/... reports no findings. go mod tidy produces a zero diff to
go.mod/go.sum, matching this repo's verify-modules check. Mirrored this repo's
own hack/test.sh invocation for this package (-short -vet=... -skip TestAPIs)
and it passes; TestAPIs is a separate envtest suite that needs a local
kubebuilder etcd binary not installed on this machine and fails identically on
an unmodified checkout, so it is a pre-existing environment gap, not a
regression. Added TestBackupQueueReconcilerTrackerNotLeakedWhenBackupCompletesDuringPatch,
which uses a controller-runtime fake client with a Patch interceptor to
simulate a racing reconciler completing the backup right after the
ReadyToStart patch lands; it fails (RunningCount leaks to 1) against the
pre-fix ordering and passes (RunningCount returns to 0) against the fix.
Report: https://github.com/velero-io/velero/issues/10519
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Assisted-by: claude-sonnet-5 (via Claude Code)
* Add changelog file for PR #10521
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
* Add test covering tracker rollback when ReadyToStart patch fails
Addresses review comment: verify backupTracker.RunningCount() returns
to 0 when the ReadyToStart patch itself errors, covering the Delete
rollback path alongside the existing race-condition regression test.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
---------
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Co-authored-by: Tiger Kaovilai <tkaovila@redhat.com>
* Update code to support namespace mapping when perform the in-place restore with block data mover
Update code to support namespace mapping when perform the in-p
lace restore with block data mover
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
* Do not assume a port name is a string when clearing node ports
deleteNodePorts reads the last-applied-configuration annotation, which
is free-form JSON controlled by whoever produced the backup, and cast
p["name"] to string without checking. A port whose name is a number
crashed the restore of that Service with an interface conversion panic.
Every sibling in the same loop already uses the comma-ok form.
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
* Add changelog file
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
* Convert name to string by Sprint.
Signed-off-by: Xun Jiang <xun.jiang@broadcom.com>
---------
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
Signed-off-by: Xun Jiang <xun.jiang@broadcom.com>
Co-authored-by: Xun Jiang <xun.jiang@broadcom.com>
Keep the existing opt-in behavior (consistent with velero backup create and velero restore create, where waiting is disabled by default and enabled with --wait), and clarify it in the command help text. Add tests pinning the --wait default (false) and flag parsing.
Signed-off-by: Mustafa Senoglu <mmustafasenoglu0@gmail.com>
* Remove toleration whitelist for PodVolumeBackup and data mover pods
Instead of filtering tolerations through a hardcoded allowlist
(ThirdPartyTolerations), inherit all tolerations from the node-agent
daemonset for PodVolumeBackup/Restore and DataUpload/Download pods,
and from the Velero deployment for maintenance jobs.
This enables backups and restores on nodes with custom NoExecute taints,
which was previously impossible since only two specific toleration keys
were whitelisted.
Fixes#9476
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
* Fix codespell: replace 'whitelist' with 'allowlist' in changelog
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
* Implement deduplication of tolerations and add unit tests for the new function
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Merge node-agent-configmap tolerations with third-party allowlist
Add a `tolerations` field to the node-agent-configmap so operators can
declare hosting-pod tolerations explicitly, per blackpiglet's review
feedback that tolerations shouldn't be read from the DaemonSet alone.
These are merged with (and deduplicated against) DaemonSet tolerations
matching the existing third-party allowlist
(kubernetes.azure.com/scalesetpriority, CriticalAddonsOnly), restoring
that allowlist per the follow-up suggestion to keep inheriting it
alongside the new config option.
The toleration dedup helper is moved from pkg/exposer to
pkg/util/kube (exported as DeduplicateTolerations) so it can be
shared with pkg/nodeagent without an import cycle.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Fix testifylint finding in TestGetTolerations
golangci-lint v2.12.0 (pinned in pr-linter-check.yml) flagged the
shared assert.Equal after the if/else as require-error: use require
for the error assertion so each branch is self-contained, matching
the pattern used elsewhere in this file.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Document toleration merge priority in GetTolerations
Per blackpiglet's review feedback: clarify that configured tolerations
take priority over allowlisted daemonset tolerations because they're
appended first and DeduplicateTolerations keeps only the first
occurrence of each exact (Key, Operator, Value, Effect) combination.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
---------
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Happy <yesreply@happy.engineering>
Compare the existing PVC's capacity against the source volume size
recorded in the backup volume info (#10506) before any side effect and
skip the volume when it is too small, so the restore fails early instead
of running out of space midway. For the block data mover the source size
is the device size; for the file system data movers it is the logical
size of the backed-up files, a lower bound since file system metadata is
not accounted for.
The file system path reads the size from the volume info already carried
in RestoreData. The PVC CSI RIA has no access to the volume info, so the
restore engine carries the size on the PVC item through a Velero-internal
annotation, the same mechanism as the selected-node carrier; both carrier
annotations are stripped before the item is created in the cluster.
The check is skipped when the source size is unknown (backups taken
before it was recorded) or the PVC's capacity is not reported.
Signed-off-by: chlins <chlins.zhang@gmail.com>
lyndon-liandGitHub
Lyndon-Li
Daniel JiangandGitHub
Xun Jiang/Bruce JiangandGitHub
Adam ZhangandGitHub
Chlins ZhangandGitHub
Pujitha Paladugu
Abhayraj Jaiswal
Xun Jiang
Jeremy Schoemaker
Wenkai Yin(尹文开)andGitHub
Mustafa Senoglu
Daniel Mungai