Commit Graph
156 Commits
Author SHA1 Message Date
Lyndon-Li 44f09189c2 Merge branch 'main' into report-incremental-fallback 2026-09-09 14:19:59 +08:00
Yonghui Li 9d85334d40 refactor CBT retrievement to report the concrete error
Signed-off-by: Yonghui Li <lyonghui@vmware.com>
2026-09-09 14:14:54 +08:00
Yonghui Li 8e604b17b2 add ID to repo snapshot
Signed-off-by: Yonghui Li <lyonghui@vmware.com>
2026-09-08 18:15:34 +08:00
Lyndon-Li df709d39d6 report incremental fallback message
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-09-04 17:21:57 +08:00
Lyndon-Li 9687d1e30e Merge branch 'main' into report-incremental-fallback 2026-09-04 16:15:06 +08:00
Lyndon-Li 5146992b5b add UT for progress message
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-09-04 16:04:38 +08:00
Lyndon-Li 67923bca6f uploader report incremental fallback
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-09-04 14:17:28 +08:00
Lyndon-Li 7e968e10d8 UT for new bitmap implementation
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-09-04 13:48:17 +08:00
Lyndon-Li 6ec84c30f8 enhance CBT retrievement to indicate the result in error message
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-09-04 13:32:24 +08:00
Wenkai Yin(尹文开)andGitHub ccfdce30f9 Fall back to full restore rather than fail if fail to get the volume ID (#10465)
Run the E2E test on kind / setup-test-matrix (push) Failing after 4s
e2e-test-kind.yaml / extract (push) Failing after 7s
Run the E2E test on kind / get-go-version (push) Failing after 8s
Run the E2E test on kind / build (push) Skipped
Run the E2E test on kind / run-e2e-test (push) Skipped
push.yml / extract (push) Failing after 7s
Main CI / get-go-version (push) Failing after 8s
Main CI / Build (push) Skipped
Fall back to full restore rather than fail if fail to get the volum
e ID

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
2026-09-03 14:56:35 +08:00
Wenkai Yin(尹文开)andGitHub 07768e7b33 Add "IncrementalBytes" field to status of DataDownload and PVR to indicate data transferred by the incremental restore (#10421)
Add "IncrementalBytes" field to status of DataDownload and PVR to indicate data transferred by the incremental restore

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
2026-09-01 15:43:53 +08:00
b7d83a6f2b Cherry pick the in-place restore implementation PRs from feature branch to main (#10415)
* Update CRDs and CLI to support in-place restore (#10038)

Update CRDs(Restore, DataDownload, PodVolumeRestore) and restore create CLI to support in-place restore

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>

* Update Kopia(filesystem) uploader to support incremental and deleteExtraFile during restore (#10066)

Update Kopia(filesystem) uploader to support incremental and deleteExtraFile during restore

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>

* Update Restore Exposer and PVC CSI to support in-place restore (#10104)

1. Update Restore Exposer to support exposing with existing PV for in-place restore
2. Update PVC CSI RIA to continue the restore process for in-place restore

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>

* Update Block uploader to support increase restore (#10244)

Update Block uploader to support increase restore

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>

* Update Exposer to recreate the target PV if the volume mode is different with the restore PVC (#10257)

Update Exposer to recreate the target PV if the volume mode is different with t
he restore PVC

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>

* Preserve PVC selected-node annotation via carrier annotation for in-place restore

For in-place volume data restore, the existing PVC is deleted and
recreated. For StorageClasses with the WaitForFirstConsumer volume
binding mode, losing the volume.kubernetes.io/selected-node annotation
could let the scheduler place the recreated workload Pod in a different
zone than the original PV, leaving it stuck in ContainerCreating.

Instead of relying on RestoreItemAction execution order (the generic
PVC RIA unconditionally strips the selected-node annotation), the PVC
CSI RIA now captures the annotation from the existing PVC right before
deleting it and carries it on the target PVC via the Velero-internal
restore.velero.io/inplace-restore-selected-node annotation. The restore
engine translates the carrier back to the Kubernetes annotation after
all RestoreItemActions have run and always strips the carrier so it
never lands on the cluster.

This makes the behavior independent of RIA ordering: the Kubernetes
annotation is stripped by default on every path (including when the
target PVC does not exist and Velero falls back to provisioning a new
PVC), and preservation only happens when the CSI RIA explicitly
captured a value from the existing PVC.

Signed-off-by: chlins <chlins.zhang@gmail.com>

* Update the control path to make the in-place incremental restore with block data mover work E2E (#10410)

Update the control path to make the in-place incremental restore with block data mover work E2E

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>

---------

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
Signed-off-by: chlins <chlins.zhang@gmail.com>
Co-authored-by: chlins <chlins.zhang@gmail.com>
2026-08-26 10:21:32 -04:00
d2241adba0 Log the discovered parent snapshot ID, not the empty lookup parameter (#10305)
e2e-test-kind.yaml / extract (push) Failing after 8s
Run the E2E test on kind / get-go-version (push) Failing after 9s
Run the E2E test on kind / build (push) Skipped
Run the E2E test on kind / setup-test-matrix (push) Successful in 3s
Run the E2E test on kind / run-e2e-test (push) Skipped
push.yml / extract (push) Failing after 6s
Main CI / get-go-version (push) Failing after 7s
Main CI / Build (push) Skipped
getParentBackupInfo's parent-selection log lines interpolated the
parentSnapshot parameter. On the discovery branch (no explicit parent
passed by the caller) that parameter is empty by definition, so every
message about which parent was chosen, or why a run fell back to full,
printed no identifier at all -- e.g. "Using parent snapshot , start
time ...". This is the normal path for scheduled/incremental backups,
so the omission hit the common case, not an edge one.

Bind a parentID local that starts as the parameter but is overwritten
once a parent is actually resolved (explicit or discovered), and log
that instead. No behavior change.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 16:47:40 -04:00
339c8edda9 Detect block uploader cancellation through wrapped errors (#10308)
* Detect block uploader cancellation through wrapped errors

Cancelling a block data mover backup was reported as a failure: the
DataUpload ended Failed with an error message and the Backup went
PartiallyFailed, for a user-requested cancel.

The cause is a sentinel equality check. block.ErrCanceled is raised in
the write loop and then wrapped twice before it reaches the provider --
once in block/uploader.go ("error backing up bdev %s") and again in
block/snapshot.go ("Failed to run uploader backup for si %v") -- so
`err == block.ErrCanceled` can never be true and the ErrorCanceled
returns are unreachable. The filesystem provider avoids this by asking
the uploader for its state (kpUploader.IsCanceled()) rather than
inspecting the error.

Use errors.Is at both the backup and restore sites.

Adds TestBlockProviderCancelThroughWrappedError, which injects the
doubly-wrapped sentinel exactly as production builds it. Note the
assertion is require.ErrorIs, not ErrorContains: provider.ErrorCanceled
and block.ErrCanceled carry identical message text, so a substring
assertion passes whether or not the sentinel was recognised -- which is
why the existing test, injecting the bare sentinel, did not catch this.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
(cherry picked from commit 9d6c5da7a893068d424b0c7896638787c636e213)
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

* Add changelog for #10308

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

* lint: fix misspelling (recognised -> recognized)

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

---------

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 15:03:33 +08:00
Lyndon-Li 882ad944f3 Merge branch 'main' into block-uploader-restore-progress 2026-08-11 15:53:08 +08:00
Lyndon-Li bbb0f11f33 use source size in progress for block uploader restore
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-08-11 15:49:30 +08:00
Lyndon-Li c8127e243b object reader throughput improvement
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-08-11 14:07:59 +08:00
Lyndon-Li de87def9d9 enable object reader prefetch for block uploader
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-08-11 13:02:11 +08:00
Lyndon-Li a41cb11902 add prefetch options for repo interface
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-08-11 12:56:23 +08:00
Jay2006sawant 8ec4b22496 fix: return errors correctly in block restore validation and BatchForget
Signed-off-by: Jay2006sawant <jay242902@gmail.com>
2026-08-03 09:44:00 +05:30
lyndon-liandGitHub 23a0cbe163 add incremental size for block uploader (#10151)
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-08-04 16:00:45 +08:00
Lyndon-Li f22b7c86d7 upload progress every 10s
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-08-03 14:11:37 +08:00
Lyndon-Li c7d7801ae9 Merge branch 'main' into optimize-sub-object-description 2026-07-29 18:09:28 +08:00
Lyndon-Li d685b818ad refactor block uploader thread module
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-07-28 15:43:13 +08:00
Lyndon-Li 525fb9eaa2 Merge branch 'main' into optimize-sub-object-description 2026-07-28 15:14:15 +08:00
Lyndon-Li b635d3f8ed refactor block uploader thread module
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-07-28 13:20:34 +08:00
Lyndon-Li bc596da386 block uploader restore implementation
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-07-27 14:19:22 +08:00
Lyndon-Li 00f1626f7a block uploader restore implementation
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-07-24 14:02:19 +08:00
Lyndon-Li 236c45b443 block uploader restore UT
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-07-24 13:37:40 +08:00
Lyndon-Li 97978ed9b7 block uploader flush zero blocks
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-07-22 18:01:57 +08:00
Lyndon-Li 9c1d8bee71 block uploader restore data
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-07-22 18:00:13 +08:00
Lyndon-Li bec292e738 block uploader restore data
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-07-22 17:58:02 +08:00
Lyndon-Li 7bbd172684 persist source dev size
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-07-22 17:52:01 +08:00
Lyndon-Li 2b2aa061a8 optimize subobject description
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-07-20 14:17:34 +08:00
lyndon-liandGitHub a0749e7658 Block dev for restore (#10013)
Run the E2E test on kind / get-go-version (push) Failing after 1m3s
Run the E2E test on kind / build (push) Has been skipped
Run the E2E test on kind / setup-test-matrix (push) Successful in 3s
Run the E2E test on kind / run-e2e-test (push) Has been skipped
Main CI / get-go-version (push) Successful in 13s
Main CI / Build (push) Failing after 22s
* add block dev operations for block uploader restore

Signed-off-by: Lyndon-Li <lyonghui@vmware.com>

* add block dev operations for block uploader restore

Signed-off-by: Lyndon-Li <lyonghui@vmware.com>

---------

Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-07-17 16:35:19 +08:00
lyndon-liandGitHub b0d7ada06a Block dev for block uploader backup (#9994)
* add block dev operations for block data mover backup

Signed-off-by: Lyndon-Li <lyonghui@vmware.com>

* add block dev operations for block data mover backup

Signed-off-by: Lyndon-Li <lyonghui@vmware.com>

* Add block device operations for block uploader backup

Signed-off-by: Lyndon-Li <lyonghui@vmware.com>

* Add block device operations for block uploader backup

Signed-off-by: Lyndon-Li <lyonghui@vmware.com>

---------

Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-07-16 13:44:41 +08:00
Lyndon-Li 84bee82575 block uploader backup implementation
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-07-09 13:15:59 +08:00
Lyndon-Li ce1c1803b2 Merge branch 'main' into block-uploader-backup-implementation 2026-07-09 11:39:30 +08:00
Xun Jiang/Bruce JiangandGitHub 96ae6586d0 Merge pull request #9863 from blackpiglet/jxun/change-id
Support change-id and volume-id in backup workflow.
2026-07-02 11:32:15 +08:00
Xun Jiang 0bc06323bf Support change-id and volume-id in backup workflow.
* Add change-id and volume-id retrieve logic for both vks and vanilla k8s environment.
* Add change-id and volume-id support code in exposer.

Signed-off-by: Xun Jiang <xun.jiang@broadcom.com>
2026-07-01 15:19:52 +08:00
Lyndon-Li 8d23c7e813 block uploader backup implementation
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-06-30 17:03:39 +08:00
Lyndon-Li 502ec5f086 remove leading and trailing separator
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-06-30 16:22:55 +08:00
Lyndon-Li 39c745ef61 set totalSize from uploader
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-06-30 16:22:21 +08:00
Lyndon-Li df21463629 block uploader backup implementation
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-06-30 16:21:59 +08:00
Lyndon-Li 8df8709a8a block uploader snapshot implementation
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-06-30 14:14:23 +08:00
Lyndon-Li 82dbef2cd9 block uploader snapshot implementation
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-06-30 11:18:44 +08:00
Lyndon-Li f38bc20a0a block uploader snapshot implementation
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-06-29 15:02:10 +08:00
Lyndon-Li 5bef38dc95 block uploader snapshot implementation
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-06-25 08:31:32 +00:00
Lyndon-Li f4f897f669 load object from snapshot
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-06-25 15:36:33 +08:00
Lyndon-Li 845bd2dc4f block uploader snapshot implementation
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2026-06-25 15:27:18 +08:00