Commit Graph
135 Commits
Author SHA1 Message Date
2c6f45508c Report a measured zero incremental instead of erasing it (#10309)
Run the E2E test on kind / setup-test-matrix (push) Failing after 4s
e2e-test-kind.yaml / extract (push) Failing after 11s
Run the E2E test on kind / get-go-version (push) Failing after 12s
Run the E2E test on kind / build (push) Skipped
Run the E2E test on kind / run-e2e-test (push) Skipped
push.yml / extract (push) Failing after 7s
Main CI / get-go-version (push) Failing after 8s
Main CI / Build (push) Skipped
* Report a measured zero incremental instead of erasing it

A CBT incremental with an exactly zero delta -- nothing changed since
the parent -- was reported identically to a backup that moved the whole
device. `velero backup describe --details` printed only
"Moved data Size (bytes): 3221225472" with no incremental line, and
status.incrementalBytes was absent, for a run that transferred nothing.
The best possible CBT outcome displayed as the worst, and was
indistinguishable from a genuine full, a whole-device fallback, or a
backup predating incremental accounting.

The zero was being erased twice. Besides the API status fields,
datapath.BackupResult also carried omitempty, and that struct crosses a
JSON boundary from the data mover pod to the controller (see
micro_service_watcher.go), so the value was destroyed before the
controller could persist it. Every uploader always reports a figure
there, so 0 internally always means "transferred nothing" -- dropping
omitempty is sufficient and correct for that hop.

The API fields move to *int64 rather than just dropping omitempty. The
field shipped in v1.18.0-v1.18.2, so backups exist whose stored volume
info has no incrementalSize at all; with a plain int64 those unmarshal
to 0 and would render "Incremental data Size (bytes): 0", a false claim
of a perfect incremental on a run that never measured one. nil means not
measured, a pointer to 0 means measured zero. Both fields already carry
+optional, so the generated CRD schema is unchanged and no regeneration
is required.

Display gates relax from > 0 to != nil in all three places, including
volumesByPod.Add, whose signature takes *int64 now; the restore describer
passes nil, which is correct since restores measure no incremental.

Verified live: the same zero-delta scenario that reported <none> now
reports 0 and renders "Incremental data Size (bytes): 0", while an older
backup described with the new client still correctly prints no
incremental line at all.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
(cherry picked from commit 6c7aa9d588f6d5eab134d4ce19c92b838f45557c)
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

* gofmt: fix import ordering in backup_test.go

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

* Regenerate CRDs for IncrementalBytes pointer type

make update-crd was missed in the original commit. Regenerated with
the pinned controller-gen v0.16.5 to avoid unrelated version-annotation
churn across other CRDs.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

* Add changelog for #10309

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

* Address review: make IncrementalBytes a pointer to preserve backward compat

Per Lyndon-Li's review on #10309: dropping omitempty on the plain int64
field breaks compatibility with a data mover from release-1.17 or
earlier that predates IncrementalBytes and never writes the key -- the
new controller would unmarshal a zero value ("nothing transferred")
instead of recognizing the field is simply absent ("not measured").

Switch to *int64 with omitempty restored:
- an old mover's omitted key unmarshals to nil ("not measured")
- a current mover's genuine zero still serializes the key, unmarshaling
  to a non-nil pointer to 0 ("measured zero")
- nonzero values work exactly as before
- an old controller can still unmarshal a numeric value from a new mover

pkg/controller/data_upload_controller.go and pod_volume_backup_controller.go
assign the wire-struct field directly to their already-*int64,omitempty
CRD status field instead of re-wrapping it with ptr.To, since both are
now the same pointer type.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

* Fix CI: update marshal-fail test assertions for IncrementalBytes pointer

Both backup_micro_service_test.go files hardcoded the %v-formatted
zero-value BackupResult struct in an error-message assertion. Now that
IncrementalBytes is *int64, its zero value prints as <nil> instead of 0.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

---------

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 18:08:17 -04:00
Krishna AwasthiandGitHub 9d01d7f491 test: add verification for skippedPVTracker in backup tests (#10283)
Signed-off-by: opbot_xd <awasthikrishna23052005@gmail.com>
2026-08-17 13:39:03 +08:00
harshit sainiandGitHub 11a071637b Use k8s.io/api well-known label constants instead of hardcoded strings (#10279)
* refactor: use k8s.io/api well-known label constants

Several well-known Kubernetes label strings were hardcoded across the
codebase instead of using the constants already exported by
k8s.io/api/core/v1, which is an existing dependency:

  "kubernetes.io/hostname"        -> corev1api.LabelHostname
  "kubernetes.io/os"              -> corev1api.LabelOSStable
  "topology.kubernetes.io/zone"   -> corev1api.LabelTopologyZone

The local kube.NodeOSLabel and zoneLabel consts, which duplicated the
upstream values verbatim, are now defined in terms of the upstream
constants rather than repeating the literal. Both are kept: NodeOSLabel
is exported and referenced from four packages alongside NodeOSLinux and
NodeOSWindows, which have no upstream equivalent, and zoneLabel sits
beside the deprecated-label fallback it is compared against.

No functional change - every replacement is a constant with an identical
value.

Signed-off-by: Harshit saini <harshitsaini1188@gmail.com>

* Add changelog for #10279

Signed-off-by: Harshit saini <harshitsaini1188@gmail.com>

* Cover the selected-node path in createRestorePod

TestCreateRestorePod only exercised selectedNode == "", so the branch
that pins the restore pod to a node was never executed. Add a case with
a selected node and assert the resulting pod carries the hostname label
in its node selector.

Signed-off-by: Harshit saini <harshitsaini1188@gmail.com>

* Also use constants for the arch and deprecated zone labels

Extends the same replacement to the two remaining well-known labels
raised on the issue:

  "kubernetes.io/arch"                      -> corev1api.LabelArchStable
  "failure-domain.beta.kubernetes.io/zone"  -> corev1api.LabelFailureDomainBetaZone

zoneLabelDeprecated in item_backupper.go was the last local const still
repeating a literal that upstream already exports, so the zone pair now
reads consistently against k8s.io/api. The deprecation note upstream
applies to the label itself, not the constant; Velero reads that label
deliberately as the fallback for PVs created before the topology labels
existed.

Signed-off-by: Harshit saini <harshitsaini1188@gmail.com>

---------

Signed-off-by: Harshit saini <harshitsaini1188@gmail.com>
2026-08-17 13:37:44 +08:00
22ae12575c Fix excluded namespace objects leaking into backup with cross-namespa… (#10159)
* Fix excluded namespace objects leaking into backup with cross-namespace listing

Signed-off-by: Pragati <Pragati5-DEBUG@users.noreply.github.com>

* Add changelog for PR 10159

Signed-off-by: Pragati <Pragati5-DEBUG@users.noreply.github.com>

---------

Signed-off-by: Pragati <Pragati5-DEBUG@users.noreply.github.com>
Co-authored-by: Pragati <Pragati5-DEBUG@users.noreply.github.com>
2026-08-09 10:15:29 -05:00
a12b373e4c Support set-based filter label selectors (#10064)
Run the E2E test on kind / setup-test-matrix (push) Successful in 3s
e2e-test-kind.yaml / extract (push) Failing after 12s
Run the E2E test on kind / get-go-version (push) Failing after 13s
Run the E2E test on kind / build (push) Skipped
Run the E2E test on kind / run-e2e-test (push) Skipped
push.yml / extract (push) Successful in 10s
Main CI / get-go-version (push) Successful in 11s
Main CI / Build (push) Failing after 25s
* Support set-based filter label selectors

Use matchLabels/matchExpressions in fine-grained filters.

Signed-off-by: Adam Zhang <adam.zhang@broadcom.com>

* omit the details of resource policy for cli

The reason to not resolve and display CLI is because it may go
out of sync, we want to avoid display it to mislead users.

We may consider to cpature those information and display it
in later release.

Signed-off-by: Adam Zhang <adam.zhang@broadcom.com>

---------

Signed-off-by: Adam Zhang <adam.zhang@broadcom.com>
Co-authored-by: Scott Seago <sseago@redhat.com>
2026-07-24 16:03:51 -04:00
Adam Zhang 13f06026c0 caching the call for GetNamespaceFilter
Introduces a concurrent-safe sync.Map cache to the backup Request struct
to memoize GetNamespaceFilter results. This avoids re-evaluating glob
patterns for every item, significantly improving backup performance while
preserving the original exact-match precedence logic.

Signed-off-by: Adam Zhang <adam.zhang@broadcom.com>
2026-06-15 16:18:54 +08:00
Xun Jiang 49b670a791 Replace github.com/pkg/errors by github.com/cockroachdb/errors
Change errors.Cause to errors.Is, because github.com/cockroachdb/errors
New() function create a error with error stack with depth 1, but
github.com/pkg/errors's New() function create error with no depth.

Signed-off-by: Xun Jiang <xun.jiang@broadcom.com>
2026-06-10 15:55:57 +08:00
Adam Zhang f216f2497d address review comments regarding fallbacks
fix fallbacks in backup filter policies,
- refactor the code to start with global policies, and apply
  override if exists, and document the behaviour inline according
  to the design
- Ensure that unlisted cluster-scoped kinds properly fall back
  to global label selectors
- unlisted namespace-scoped kinds are explicitly skipped when
  evaluating policy label selectors

add test coverage for GetNamespaceFilter glob matching and ordering

Added a unit test to verify that GetNamespaceFilter correctly matches
compiled glob patterns against namespace strings, and properly honors
first-match semantics when a namespace matches multiple patterns.

Signed-off-by: Adam Zhang <adam.zhang@broadcom.com>
2026-06-08 15:50:05 +08:00
Adam Zhang c8bb3af761 implement fine-grained backup filter policies
implement backup logic to support backups with
NamespacedFilterPolicies and ClusterScopedFilterPolicy

Signed-off-by: Adam Zhang <adam.zhang@broadcom.com>
2026-06-08 15:50:05 +08:00
Scott Seago aedc0fe5e2 make update, changelog
Signed-off-by: Scott Seago <sseago@redhat.com>
2025-12-02 16:39:07 -05:00
Scott Seago 91357b28c4 Move worker pool creation to backup reconcile.
ItemBlockWorkerPool is now created for each backup.

Signed-off-by: Scott Seago <sseago@redhat.com>
2025-12-02 16:38:41 -05:00
Scott Seago 5fc76db8c0 Add incrementalSize to DU/PVB for reporting new/changed size
Signed-off-by: Scott Seago <sseago@redhat.com>
2025-10-27 15:38:31 -04:00
0xLeo258 1ebe357d18 Add built-in mutex for SynchronizedVSList && Update unit tests
Signed-off-by: 0xLeo258 <noixe0312@gmail.com>
2025-09-20 09:13:07 +08:00
Lyndon-Li 34f8b73507 bump up kopia to v0.21.1
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2025-07-22 15:56:04 +08:00
Matthieu MOREL 07ea14962c fix require-error rule from testifylint
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2025-06-23 15:39:54 +00:00
Xun JiangandXun Jiang/Bruce Jiang 2d71430e80 Skip namespace in terminating state in backup resource collection.
To make sure resources in terminating namespaces are not included.

Signed-off-by: Xun Jiang <xun.jiang@broadcom.com>
2025-05-29 10:32:32 +08:00
Matthieu MORELandGitHub c6a420bd3a chore: define common aliases for k8s packages (#8672)
Run the E2E test on kind / build (push) Failing after 6m48s
Run the E2E test on kind / setup-test-matrix (push) Successful in 3s
Run the E2E test on kind / run-e2e-test (push) Has been skipped
Main CI / Build (push) Failing after 35s
Close stale issues and PRs / stale (push) Successful in 8s
Trivy Nightly Scan / Trivy nightly scan (velero, main) (push) Failing after 1m11s
Trivy Nightly Scan / Trivy nightly scan (velero-plugin-for-aws, main) (push) Failing after 47s
Trivy Nightly Scan / Trivy nightly scan (velero-plugin-for-gcp, main) (push) Failing after 49s
Trivy Nightly Scan / Trivy nightly scan (velero-plugin-for-microsoft-azure, main) (push) Failing after 43s
* lchore: define common alias for k8s packages

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>

* Update .golangci.yaml

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>

* Update .golangci.yaml

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>

* Update .golangci.yaml

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>

---------

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2025-04-22 06:14:47 -04:00
Wenkai Yin(尹文开) 7aa8040c09 Run backup post hooks inside ItemBlock synchronously
Run backup post hooks inside ItemBlock synchronously as the ItemBlocks are handled asynchronously

Fixes #8516

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
2025-02-17 13:27:41 +08:00
Wenkai Yin(尹文开)andGitHub e3a64065f1 Merge pull request #8659 from sseago/parallel-itemblocks
Implement parallel ItemBlock processing via backup_controller goroutines
2025-02-14 10:42:14 +08:00
Scott Seago fcfb2fd9ee Implement parallel ItemBlock processing via backup_controller goroutines
Signed-off-by: Scott Seago <sseago@redhat.com>
2025-02-12 12:03:37 -05:00
Wenkai Yin(尹文开) cdcd6eb99d Fix WaitGroup panic issue
Make sure WaitGroup.Add() is called before WaitGroup.Done() to avoid WaitGroup panic issue

Fixes #8657

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
2025-02-12 13:56:05 +08:00
Matthieu MOREL cbba3bdde7 chore: enable use-any from revive
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2025-01-17 07:58:10 +01:00
Wenkai Yin(尹文开) 1f39943291 Check the PVB status via podvolume Backupper rather than calling API server to avoid API server issue
Check the PVB status via podvolume Backupper rather than calling API server to avoid API server issue

Fixes #8587

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
2025-01-13 12:56:26 +08:00
Wenkai Yin(尹文开) 14e71fa2cd Bug fix: increase the WaitGroup counter before start the goroutine
Bug fix: increase the WaitGroup counter before start the goroutine

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
2024-12-23 17:26:36 +08:00
Wenkai Yin(尹文开) c43fc42c25 Fix backup post hook issue
Fix backup post hook issue

Fixes #8159

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
2024-12-13 12:25:45 +08:00
Tiger KaovilaiandGitHub 04d6c79179 Merge pull request #8471 from vmware-tanzu/8440_fix_main
Run the E2E test on kind / build (push) Failing after 6m42s
Run the E2E test on kind / setup-test-matrix (push) Successful in 4s
Run the E2E test on kind / run-e2e-test (push) Has been skipped
Main CI / Build (push) Failing after 1m9s
[main] Add nil check for updating DataUpload VolumeInfo in finalizing phase
2024-12-05 01:17:19 -05:00
Scott Seago 015b1e69f6 Make BackedUpItems thread safe
Signed-off-by: Scott Seago <sseago@redhat.com>
2024-12-03 15:23:45 -05:00
Xun Jiang 226370d035 Add nil check for updating DataUpload VolumeInfo in finalizing phase.
Signed-off-by: Xun Jiang <xun.jiang@broadcom.com>
2024-12-03 10:50:55 +08:00
Scott Seago 9d6f4d2db5 ItemBlock model and phase 1 (single-thread) workflow changes
Signed-off-by: Scott Seago <sseago@redhat.com>
2024-09-03 19:04:18 -04:00
lyndon-liandGitHub 2b018272e6 fix linter check error (#8014)
Signed-off-by: Lyndon-Li <lyonghui@vmware.com>
2024-07-15 09:48:10 -04:00
Scott Seago dc286a38fc Reuse existing plugin manager for get/put volume info
Signed-off-by: Scott Seago <sseago@redhat.com>
2024-07-12 10:15:16 -04:00
Xun Jiang df28134e25 Add result in backup VolumeInfo.
Signed-off-by: Xun Jiang <blackpigletbruce@gmail.com>
2024-07-02 00:20:36 +08:00
Xun Jiang/Bruce JiangandGitHub d9c9f77860 Merge pull request #7671 from mmorel-35/testifylint/compare
testifylint: enable compares rule
2024-05-27 12:45:16 +08:00
Xun Jiang a91d2cb036 Modify the volume helper logic.
Signed-off-by: Xun Jiang <blackpigletbruce@gmail.com>
2024-05-23 09:57:21 +08:00
Matthieu MOREL 1010b04821 testifylint: enable compares rule
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2024-05-16 20:18:43 +00:00
Xun Jiang 884bcbec98 Fix the typecheck error reported by the lint GitHub action.
Signed-off-by: Xun Jiang <blackpigletbruce@gmail.com>
2024-04-19 18:41:16 +08:00
Xun Jiang 2eeaf4d55e Modify namespace filter logic for backup with label selector.
Signed-off-by: Xun Jiang <blackpigletbruce@gmail.com>
2024-04-18 10:30:59 +08:00
Daniel Jiang 0a280e5786 Track and persist restore volume info
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2024-04-11 17:32:18 +08:00
Xun Jiang/Bruce JiangandGitHub 8df4e6aded Merge branch 'main' into merge_csi
Signed-off-by: Xun Jiang/Bruce Jiang <59276555+blackpiglet@users.noreply.github.com>
2024-04-10 18:54:16 +08:00
clonefetchandGitHub 474dc824e7 chore: fix function names in comment (#7633)
Signed-off-by: clonefetch <c0217@outlook.com>
2024-04-10 15:15:29 +05:30
Xun Jiang 31e140919a Merge CSI plugin code.
Signed-off-by: Xun Jiang <blackpigletbruce@gmail.com>
2024-04-10 14:53:29 +08:00
Xun Jiang/Bruce JiangandGitHub 75962653c5 Merge pull request #7554 from blackpiglet/7357_fix
Support update the backup VolumeInfos by the Async ops result.
2024-04-01 11:05:33 +08:00
Xun Jiang 5462035469 Delete the unneeded pvRestorer action in
handleSkippedPVHasRetainPolicy

According to comment, calling executePVAction aims to reset PV's
claimRef, but the reset logic was moved into resetVolumeBindingInfo
since release-1.4.

Signed-off-by: Xun Jiang <blackpigletbruce@gmail.com>
2024-03-29 14:12:12 +08:00
Wenkai Yin(尹文开) 8d10b68eda Improve the concurrency for PVBs in different pods
Improve the concurrency for PVBs in different pods

Fixes #6676

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
2024-03-29 09:58:50 +08:00
Xun Jiang b06d7a467f Support update the backup VolumeInfos by the Async ops result.
1. Add PutBackupVolumeInfos method.
2. Add CompletionTimestamp in VolumeInfo.
3. Add Size in SnapshotDataMovementInfo.
4. Update CompletionTimpstmap, SnapshotHandle, RetainedSnapshot
   and Size in VolumeInfo on DataUpload Operation completes.

Signed-off-by: Xun Jiang <blackpigletbruce@gmail.com>
2024-03-28 19:52:44 +08:00
Matthieu MOREL ef04ef6361 golangci-lint: use exclude-rules instead of skip-files and skip-dirs
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2024-03-27 20:17:34 +00:00
Xun Jiang efb94ae610 Refactor the native snapshot definition code.
Signed-off-by: Xun Jiang <blackpigletbruce@gmail.com>
2024-03-20 15:38:07 +08:00
Wenkai Yin(尹文开) 8752c3a820 Bump up the versions of severel Kubernetes-related libs
Bump up the versions of severel Kubernetes-related libs

Signed-off-by: Wenkai Yin(尹文开) <yinw@vmware.com>
2024-03-05 13:09:38 +08:00
Xun Jiang c77bec73bb Move VolumesInformation to an independant package.
Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-12-04 08:33:37 +08:00
Xun Jiang ca97248f2a Use VolumeInfo to help restore the PV.
Add VolumeInfo for left PVs during backup.

Signed-off-by: Xun Jiang <jxun@vmware.com>
2023-12-04 08:33:37 +08:00