* Update CRDs and CLI to support in-place restore (#10038)
Update CRDs(Restore, DataDownload, PodVolumeRestore) and restore create CLI to support in-place restore
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
* Update Kopia(filesystem) uploader to support incremental and deleteExtraFile during restore (#10066)
Update Kopia(filesystem) uploader to support incremental and deleteExtraFile during restore
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
* Update Restore Exposer and PVC CSI to support in-place restore (#10104)
1. Update Restore Exposer to support exposing with existing PV for in-place restore
2. Update PVC CSI RIA to continue the restore process for in-place restore
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
* Update Block uploader to support increase restore (#10244)
Update Block uploader to support increase restore
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
* Update Exposer to recreate the target PV if the volume mode is different with the restore PVC (#10257)
Update Exposer to recreate the target PV if the volume mode is different with t
he restore PVC
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
* Preserve PVC selected-node annotation via carrier annotation for in-place restore
For in-place volume data restore, the existing PVC is deleted and
recreated. For StorageClasses with the WaitForFirstConsumer volume
binding mode, losing the volume.kubernetes.io/selected-node annotation
could let the scheduler place the recreated workload Pod in a different
zone than the original PV, leaving it stuck in ContainerCreating.
Instead of relying on RestoreItemAction execution order (the generic
PVC RIA unconditionally strips the selected-node annotation), the PVC
CSI RIA now captures the annotation from the existing PVC right before
deleting it and carries it on the target PVC via the Velero-internal
restore.velero.io/inplace-restore-selected-node annotation. The restore
engine translates the carrier back to the Kubernetes annotation after
all RestoreItemActions have run and always strips the carrier so it
never lands on the cluster.
This makes the behavior independent of RIA ordering: the Kubernetes
annotation is stripped by default on every path (including when the
target PVC does not exist and Velero falls back to provisioning a new
PVC), and preservation only happens when the CSI RIA explicitly
captured a value from the existing PVC.
Signed-off-by: chlins <chlins.zhang@gmail.com>
* Update the control path to make the in-place incremental restore with block data mover work E2E (#10410)
Update the control path to make the in-place incremental restore with block data mover work E2E
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
---------
Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
Signed-off-by: chlins <chlins.zhang@gmail.com>
Co-authored-by: chlins <chlins.zhang@gmail.com>
* Report a measured zero incremental instead of erasing it
A CBT incremental with an exactly zero delta -- nothing changed since
the parent -- was reported identically to a backup that moved the whole
device. `velero backup describe --details` printed only
"Moved data Size (bytes): 3221225472" with no incremental line, and
status.incrementalBytes was absent, for a run that transferred nothing.
The best possible CBT outcome displayed as the worst, and was
indistinguishable from a genuine full, a whole-device fallback, or a
backup predating incremental accounting.
The zero was being erased twice. Besides the API status fields,
datapath.BackupResult also carried omitempty, and that struct crosses a
JSON boundary from the data mover pod to the controller (see
micro_service_watcher.go), so the value was destroyed before the
controller could persist it. Every uploader always reports a figure
there, so 0 internally always means "transferred nothing" -- dropping
omitempty is sufficient and correct for that hop.
The API fields move to *int64 rather than just dropping omitempty. The
field shipped in v1.18.0-v1.18.2, so backups exist whose stored volume
info has no incrementalSize at all; with a plain int64 those unmarshal
to 0 and would render "Incremental data Size (bytes): 0", a false claim
of a perfect incremental on a run that never measured one. nil means not
measured, a pointer to 0 means measured zero. Both fields already carry
+optional, so the generated CRD schema is unchanged and no regeneration
is required.
Display gates relax from > 0 to != nil in all three places, including
volumesByPod.Add, whose signature takes *int64 now; the restore describer
passes nil, which is correct since restores measure no incremental.
Verified live: the same zero-delta scenario that reported <none> now
reports 0 and renders "Incremental data Size (bytes): 0", while an older
backup described with the new client still correctly prints no
incremental line at all.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
(cherry picked from commit 6c7aa9d588f6d5eab134d4ce19c92b838f45557c)
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* gofmt: fix import ordering in backup_test.go
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Regenerate CRDs for IncrementalBytes pointer type
make update-crd was missed in the original commit. Regenerated with
the pinned controller-gen v0.16.5 to avoid unrelated version-annotation
churn across other CRDs.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Add changelog for #10309
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Address review: make IncrementalBytes a pointer to preserve backward compat
Per Lyndon-Li's review on #10309: dropping omitempty on the plain int64
field breaks compatibility with a data mover from release-1.17 or
earlier that predates IncrementalBytes and never writes the key -- the
new controller would unmarshal a zero value ("nothing transferred")
instead of recognizing the field is simply absent ("not measured").
Switch to *int64 with omitempty restored:
- an old mover's omitted key unmarshals to nil ("not measured")
- a current mover's genuine zero still serializes the key, unmarshaling
to a non-nil pointer to 0 ("measured zero")
- nonzero values work exactly as before
- an old controller can still unmarshal a numeric value from a new mover
pkg/controller/data_upload_controller.go and pod_volume_backup_controller.go
assign the wire-struct field directly to their already-*int64,omitempty
CRD status field instead of re-wrapping it with ptr.To, since both are
now the same pointer type.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Fix CI: update marshal-fail test assertions for IncrementalBytes pointer
Both backup_micro_service_test.go files hardcoded the %v-formatted
zero-value BackupResult struct in an error-message assertion. Now that
IncrementalBytes is *int64, its zero value prints as <nil> instead of 0.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
---------
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Modify the logs.
Modify the CRD's data mover's comment.
Modify the resource policy's GetDataMover for default data mover case.
Signed-off-by: Xun Jiang <xun.jiang@broadcom.com>
* Add change-id and volume-id retrieve logic for both vks and vanilla k8s environment.
* Add change-id and volume-id support code in exposer.
Signed-off-by: Xun Jiang <xun.jiang@broadcom.com>
Change errors.Cause to errors.Is, because github.com/cockroachdb/errors
New() function create a error with error stack with depth 1, but
github.com/pkg/errors's New() function create error with no depth.
Signed-off-by: Xun Jiang <xun.jiang@broadcom.com>
Per review feedback on #9791, the previous revision still let a
DataUpload with an empty velero.io/backup-name label fall through to
genConfigmap, creating a ConfigMap that deleteMovedSnapshots can never
match back to a snapshot. The CM is useless and only adds etcd churn.
Treat the missing-label case the same way as the foreign-owner case:
warn and skip the ConfigMap creation. Use a distinct warn message so
operators can tell the two misconfiguration classes apart in logs
(missing-label vs. owner mismatch from a captured velero namespace).
Test for the missing-label case is updated to assert no ConfigMap is
created and a warn is emitted. The warn assertion is generalized to
match the per-case message substring instead of a fixed string.
Signed-off-by: Christian Schlichtherle <christian@schlichtherle.de>
Velero does not support self-protection: the velero namespace must
never be captured in a backup tarball. When it is, the tarball can
contain DataUpload CRs belonging to other backups, and the previous
revision of this change silently swallowed that case in the
DataUploadDeleteAction.
Per maintainer feedback, the action should make the misconfiguration
detectable rather than silent. Emit a warn-level log naming the
DataUpload, its owning backup-name label, and the executing backup,
and call out that the velero namespace should be excluded from
schedules. Continue to skip the snapshot-info ConfigMap creation so
that a mislabeled CM does not mask the real owning backup's snapshot
on deletion.
The test for the foreign-backup case now also asserts the warn is
emitted via a logrus test hook.
Signed-off-by: Christian Schlichtherle <cs@bsure-analytics.de>
When a backup tarball incidentally contains DataUpload CRs that belong to
a different backup (common when a schedule includes the velero namespace
where DataUploads live), DataUploadDeleteAction.Execute used to create a
"<du-name>-info" ConfigMap labeled with the *executing* backup's name
instead of the DataUpload's true owning backup. The ConfigMap is
created with Create-only semantics, so the wrong label is never
corrected.
deleteMovedSnapshots in the backup-deletion controller looks up these
ConfigMaps by velero.io/backup-name to discover which Kopia snapshots
to delete. With the wrong label, the real owning backup's expiry pass
finds no ConfigMaps for its DataUploads and silently leaves their Kopia
snapshots in object storage, leaking data over time.
Fix: in DataUploadDeleteAction.Execute, compare the DataUpload's
velero.io/backup-name label against input.Backup.Name (using
label.GetValidName to handle DNS-1035 truncation for long backup names).
If the label is present and differs, skip the DataUpload entirely; this
prevents the over-eager creation of misnamed ConfigMaps without changing
behavior for DataUploads that legitimately belong to the executing
backup, or for legacy DataUploads with no backup-name label.
Refs: #9472
Signed-off-by: Christian Schlichtherle <cs@bsure-analytics.de>
This commit introduces a deleteItemAction which writes a temporary configmap to
record the snapshot info so that the controller can trigger repo manager
to remove the snapshot
This process is a bit chatty and we should consider to refactor the code
so it's easier to connect to the repo directly in the DIA
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
R4mboandGitHub
lyndon-liandGitHub
Xun Jiang
Shubham PampattiwarandGitHub
Lyndon-Li
Xun Jiang/Bruce JiangandGitHub
Christian Schlichtherle
Christian Schlichtherle
Scott Seago
Matthieu MOREL
Xun Jiang
Daniel Jiang