Dependabot security updates are not covered by version-update groups, so each
vulnerable dependency opens its own PR. Add a group with
applies-to: security-updates to both gomod entries so they arrive as a single
PR per module.
Signed-off-by: Tiger Kaovilai <passawit.kaovilai@gmail.com>
Co-authored-by: Claude Code (model: Claude Sonnet 5.5) <noreply@anthropic.com>
* ci: add CodeQL analysis for Go
The Scorecard SAST check scores 0 because the repository only uses
github/codeql-action to upload SARIF from Scorecard and Trivy, and does
not run any static analysis. Add a CodeQL workflow for Go that runs on
pushes and pull requests to main and weekly, with actions pinned by
commit SHA. Skipped on forks.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: Claude Sonnet 5.5) <noreply@anthropic.com>
* ci: use autobuild for CodeQL Go analysis
The Go extractor does not support build-mode none. Set up Go from go.mod
and use autobuild.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: Claude Sonnet 5.5) <noreply@anthropic.com>
---------
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Claude Code (model: Claude Sonnet 5.5) <noreply@anthropic.com>
Move the resourcemodifiers package from internal/ to pkg/ so that external
plugins, such as Restore Item Action (RIA) plugins, can import and reuse
Velero's built-in resource modifier implementation.
Due to Go's internal package visibility restrictions, external plugins
cannot import packages located under internal/. Relocating resourcemodifiers
to pkg/ allows plugins to evaluate and apply resource modifier rules to
in-memory resource objects prior to validation or restore execution,
ensuring consistent behavior with the Velero server without duplicate code.
Update all internal import paths and add an unreleased changelog entry.
Fixes#10668
Signed-off-by: Adam Zhang <adam.zhang@broadcom.com>
* docs: add data mover memory and upload concurrency tuning guidance
Explain that CPU, memory and upload concurrency are separate settings,
that data mover memory grows with --parallel-files-upload and
--parallel-blocks-upload, and how to tune them for the environment.
Add an example from one environment and link the guidance from the
backup, CSI data movement and fs-backup docs.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: Claude Opus 5.5) <noreply@anthropic.com>
* docs: note network bandwidth as a throughput factor for data mover concurrency
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: Claude Opus 5.5) <noreply@anthropic.com>
* docs: explain when raising or lowering data mover concurrency helps
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: Claude Opus 5.5) <noreply@anthropic.com>
* docs: add measured bandwidth and parallel-blocks-upload example for data mover tuning
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: Claude Opus 5.5) <noreply@anthropic.com>
---------
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Claude Code (model: Claude Opus 5.5) <noreply@anthropic.com>
* e2e: run the BSL deletion snapshot case on kind under CSI
The case was unskipped on kind by #10658 but left out of the matrix,
because nobody had checked whether it worked there. It does.
Run on kind with EnableCSI and Kibishii at the commit this workflow
pins:
BSL && Deletion && Snapshot Ran 1 of 70 Specs in 214.819 seconds
Kibishii installs from kubernetes/yaml/kind-csi, e2e-storage-class comes
from testdata/storage-class/kind-csi.yaml, and the snapshot checkpoint
reports EnableCSI with a real CSI snapshot handle rather than an empty
list, so the case verified a snapshot rather than passing with nothing to
check.
The row selects only the Snapshot variant. BSL && Deletion && FSBackup
is a separate case and stays out, since file system backup through a
deleted BackupStorageLocation has not been run here.
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* Add changelog for #10747
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
---------
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* e2e: add a target that runs the tests on a kind cluster it creates
Running the e2e tests locally means assembling a cluster, an object
store with two buckets, a credentials file, an address the cluster can
reach MinIO on, and a Velero image built and loaded, before any
GINKGO_LABELS is useful. That is a lot of setup to get wrong, and
nothing in the repo does it.
Add hack/run-e2e-kind.sh and a run-e2e-kind target that wraps it. It
creates the cluster, starts MinIO, builds and loads the image, runs the
suite, and removes what it created on the way out, including when the
tests fail or the run is interrupted. FAIL_FAST=true keeps the test bed
and prints how to remove it, as it does for the suite itself.
The cluster's kubeconfig is written to a temporary directory rather than
merged into the caller's, so ~/.kube/config is untouched and the current
context does not change.
The MinIO container is named after the cluster, and the script refuses to
start if that name is already taken rather than removing a container it
did not create. FEATURES=EnableCSI delegates to
hack/install-csi-hostpath.sh.
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* Add changelog for #10652
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* e2e: point the kind target's CSI runs at a CSI-backed StorageClass
The target installs csi-driver-host-path when EnableCSI is asked for, but
left the suite's own StorageClass alone. InstallStorageClasses falls back
to testdata/storage-class/<provider>.yaml when --storage-class-file is
not given, and the kind one provisions with rancher.io/local-path, so the
run installed a snapshot-capable driver and then created
e2e-storage-class on a provisioner that cannot snapshot. The example in
the script's own header could not have worked.
Pass kind-csi.yaml for a CSI run, which is the file the CSI job in CI
already uses. Non-CSI runs pass an empty value, which is what the
Makefile defaults to, so they are unchanged.
While here, match FEATURES as the comma-separated list the flag accepts
rather than comparing the whole value, so EnableCSI,EnableAPIGroupVersions
still installs the driver. Same reasoning as the Go side in #10661.
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* e2e: fix five faults in the kind runner found in review
Teardown deleted the cluster whether or not this run created it. The EXIT
trap is installed before the cluster exists, so an early exit, including
the check that refuses an existing MinIO container, removed a cluster
named velero-e2e that belonged to someone else. The script already
refused to remove a MinIO container it did not create, after an earlier
version deleted one that was in use; the cluster needed the same rule and
did not have it. It now refuses a cluster name that is taken and tracks
whether it created one.
MinIO was started with "server /data", which replaces the entrypoint. CI
passes no command and relies on the image starting the server itself, and
the bitnami image this script tells the user to build runs as UID 1001
and cannot create a data directory at the filesystem root, so the
recommended image could not come up. The command is now empty by default,
with MINIO_ARGS for an image that needs one, such as upstream
minio/minio, and the readiness failure says so.
The image name was taken with ${VELERO_IMAGE%%:*}, which strips from the
first colon, so localhost:5000/velero:e2e-local became "localhost" and
the load asked for a tag that was never built. Strip only the final tag.
The loaded tag was hardcoded to -linux-amd64, but the container target
appends the platform from go env when BUILD_OUTPUT_TYPE is docker, so an
arm64 workstation built -linux-arm64 and the load failed before any test
ran. Resolve the platform once and use it for both the load and the
suite.
Finally, the resolved tag was handed to the inner make through the
environment. A command-line assignment to the outer make propagates
through MAKEFLAGS and outranks the environment, so
"make -C test run-e2e-kind VELERO_IMAGE=velero:custom" ran the suite
against velero:custom while the script had loaded
velero:custom-linux-amd64. Pass it as an argument instead.
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* e2e: take only the architecture from go env, the image OS is always linux
The container target fixes ALL_OS to linux when BUILD_OUTPUT_TYPE is
docker and takes only the architecture from go env, so the tag it
produces ends in -linux-<arch> on every host. Deriving the OS from
go env GOOS instead meant a Darwin workstation computed
-darwin-<arch> and tried to load a tag that was never built, which
matters because a Darwin host still needs a Linux image to load into
kind.
Suggested in review.
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
---------
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* e2e: finish converting the CSI feature checks to FeatureEnabled
#10661 routed every check for EnableCSI through FeatureEnabled, which
splits the comma-separated --features value instead of comparing the
whole thing. One site was left on strings.EqualFold, because #10658
introduced it after #10661 had branched and the rebase applied cleanly
without noticing.
The result is that KibishiiVolumesAreHostPath and the Kind branch of
installKibishii, which are inverses of each other and sit twenty-odd
lines apart with comments saying so, disagree whenever a second feature
is enabled. With --features=EnableCSI,EnableAPIGroupVersions on kind the
mapping swaps in the CSI-backed overlay and the predicate still reports
the volumes as hostPath, so the five cases install a cluster they can
snapshot and then skip.
Convert the last site and test the predicate against the list, case and
single-feature shapes, which fails on the list cases before the change.
The test would not have run in CI either: hack/test.sh listed ./test but
not the other unit-testable packages under test/, so the two existing
tests in this package and the five in test/pkg/client and
test/util/velero were never executed. List all four. The e2e and perf
suites stay out, since they call RunSpecs and need a cluster.
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* Add changelog for #10678
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
---------
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* e2e: run the Kibishii file system backup cases on kind under CSI
The two Basic FSBackup cases were unskipped on kind by #10658 but left
out of the matrix, because whether a PodVolumeBackup would pick up
Kibishii's volumes on a CSI-backed class had not been checked. It does.
isHostPathVolume in pkg/podvolume/backupper.go matches pv.Spec.HostPath,
and the kind-csi overlay provisions through hostpath.csi.k8s.io, so the
PV carries pv.Spec.CSI and the check stops matching. The driver's name
has nothing to do with the field, which is what made this worth running
rather than reasoning about.
Both cases pass on kind with EnableCSI, with a PodVolumeBackup per
Kibishii PVC rather than volumes skipped:
Basic && FSBackup && !RetainPV Ran 2 of 70 Specs in 529.790 seconds
Basic && FSBackup && RetainPV Ran 1 of 70 Specs in 303.670 seconds
The second resolves overlays/sc-reclaim-policy/kind-csi, so the reclaim
overlay is exercised on the file system backup path too.
The row is Basic && FSBackup, which selects exactly those two cases and
nothing else.
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* Add changelog for #10680
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
---------
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
In the Volume Snapshot Retention section, the first listed case is the
storage that needs the parent snapshot to exist when GetMetadataDelta is
called, while the text below says the existing exposer works with Case 1
because the snapshot is always deleted, and that Case 2 needs the
snapshot to be retained. The volume policy text (Way 1 never retains,
Way 2 retains) and the exposer code comment that calls Ceph RBD a Case 2
storage follow that text too.
Swap the two list items so that Case 1 is the storage that doesn't need
the parent snapshot and Case 2 is the one that does.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Claude Code (model: Claude Opus 5.5) <noreply@anthropic.com>
* e2e: give the namespace-mapping cases real volumes on kind under CSI
Init had two kind branches that are not skips. The first replaces
Kibishii's dataset with every dimension zero, so generate.sh and verify.sh
run over nothing. The second forces --snapshot-volumes=false and clears
UseVolumeSnapshots, which also skips the else branch that would add
--default-volumes-to-fs-backup.
Both were right while Kibishii's PVs on kind were hostPath, since neither
a CSI snapshot nor a PodVolumeBackup would pick them up. The effect is
that the two NamespaceMapping FSBackup cases in the kind matrix, which run
on every PR, assert the namespace mapping and nothing about the volumes
despite their label.
Gate both on the same predicate as the other Kibishii cases, so a run with
EnableCSI gets the real dataset and a real snapshot, and add a
NamespaceMapping && Snapshot row to the CSI matrix to select the two cases
that were in no row before.
The FSBackup pair keeps its current behaviour: it runs in the non-CSI job
where Kibishii is still hostPath, so it still takes the zero-data path.
Moving it needs file system backup proven on a CSI-backed class first.
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* Add changelog for #10679
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
---------
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
Comment why unix.BLKZEROOUT must be used instead of a literal, and add a
flushZeroBlocks test with a non-zero prefilled file proving the write
fallback zeroes only the requested range.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: Claude Sonnet 5.5) <noreply@anthropic.com>
Other code in the kopia repository uses its logger without a nil check,
so drop the check and give the test a logger to cover the path.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: Claude Opus 5.5) <noreply@anthropic.com>
Deleting a kopia snapshot refreshes the repo when the snapshot is not
found, and failed when it was still not found. When a backup deletion
was interrupted after the snapshot was deleted, e.g. by a Velero server
restart, every retry of the deletion failed with "snapshot not found",
so the backup could never be deleted.
A snapshot that is still not found after the refresh is already deleted,
so return success.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
blkZeroOut hard coded BLKZEROOUT as 0x127b, which is BLKPBSZGET. The
ioctl therefore succeeded on block devices without zeroing anything, so
the conservative fallback that writes zero buffers never ran and block
data mover restores left the previous content in place for zero ranges
of the backup.
Take the request number from golang.org/x/sys/unix, which also gets the
arch specific value right (e.g. 0x2000127f on ppc64le).
Add a test that zeroes a range of a loop device (runs as root only) and
checks the type and number encoded in the request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
The CII Best Practices badge in the README still used the retired
bestpractices.coreinfrastructure.org domain, which only works via a
redirect. Link to bestpractices.dev directly and use the current
OpenSSF Best Practices name.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Claude Code (model: Claude Sonnet 5.5) <noreply@anthropic.com>
velero backup logs and velero restore logs check the phase before they
create a DownloadRequest, and only go ahead for Completed,
PartiallyFailed, Failed, WaitingForPluginOperations and
WaitingForPluginOperationsPartiallyFailed. Finalizing and
FinalizingPartiallyFailed are not on either list, so both commands
answer "logs ... are not available until it's finished processing" for
a backup or restore in those phases.
The log is already in object storage by then. The backup controller
writes it in persistBackup (PutBackup uploads the log first) before the
Backup is patched out of InProgress, and the restore controller calls
PutRestoreLog before it sets the WaitingForPluginOperations or
Finalizing phase. Neither finalizer controller writes the log again,
and the DownloadRequest controller signs URLs in both phases. Finalizing
also comes after WaitingForPluginOperations, which both commands
already accept, so a log that could be read while plugin operations
were running becomes unreadable again until the finalizer is done.
For restores that window is not short. The restore finalizer waits up
to the resource timeout for restored PVCs to bind before patching their
PVs, and waits for post-restore exec hooks, so a restore can stay in
Finalizing for minutes, and a restore stuck there is when its log is
most needed.
Add the two Finalizing phases to both allowlists. The new table tests
create a backup or restore in each phase and check that the phases with
a log get as far as requesting it, while New, InProgress and
FailedValidation are still refused.
Signed-off-by: Max Freedom Pollard <272618364+MaxFreedomPollard@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Fix restore resource priorities aliasing that can skip restoring resources
Priorities.Set returned HighPriorities and LowPriorities as slices of
one backing array when the value had a "-" in the middle, leaving
HighPriorities with spare capacity over the separator and the low list.
getOrderedResources appended to HighPriorities, so a backup with between
2 and 1+len(low) unprioritized resource types overwrote the low list in
place. That could drop a backed-up resource type from the restore order,
and because the server's priorities are shared by every restore, the
corrupted order persisted for later restores.
- Set: cap HighPriorities with full slice expressions.
- getOrderedResources: build a new slice instead of appending to the
caller's.
- String: build a fresh slice instead of appending to HighPriorities.
Signed-off-by: abd-ulbasit <basit@basit.engineer>
* Add changelog for #10665
Signed-off-by: abd-ulbasit <basit@basit.engineer>
---------
Signed-off-by: abd-ulbasit <basit@basit.engineer>
* e2e: fail the suite when a label filter selects no specs
A label expression that matches nothing is reported as a pass. Ginkgo
runs zero specs, prints SUCCESS!, and exits 0, so a run that tested
nothing is indistinguishable from one that tested everything asked of it:
$ GINKGO_LABELS='Basic && BackupsSync' make -C test/ run-e2e
Ran 0 of 70 Specs in 0.001 seconds
SUCCESS! -- 0 Passed | 0 Failed | 0 Pending | 70 Skipped
That expression looks reasonable and selects nothing, because the backup
sync case is labelled Backups rather than Basic. The same shape of
mistake kept the VolumeGroupSnapshot case from running for months, where
CSISnapshot did not select CSIVolumeGroupSnapshot because a bare token is
matched by equality rather than as a substring.
Pass --fail-on-empty, which ginkgo has supported for far longer than the
version in go.mod, so the above now ends in:
FAIL! - Detected no specs ran and --fail-on-empty is set
It cannot affect a run that selects specs and then skips them at run
time, which this suite does constantly on kind. The flag tests
specs.CountWithoutSkip(), and that Skip field is set only while the label
filter is applied, before anything runs; a Skip() call during a spec sets
the spec's state instead. Checked all five label rows the kind workflow
uses against every labelled Describe in the suite: the smallest selects
two specs, so none of them starts failing.
run-perf keeps the old behaviour. It takes its own labels for a different
workload, and an empty selection there is not what this is about.
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* Add changelog for #10674
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
---------
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* e2e: match --features as the comma-separated list it is
--features takes a comma-separated list, but every check for EnableCSI in
the suite compares the whole flag value against the single feature name.
With --features=EnableCSI,EnableAPIGroupVersions none of them match, so a
CSI run quietly behaves as a non-CSI one: BeforeSuite skips installing the
VolumeSnapshotClass, installKibishii picks the provider's plain overlay
rather than its CSI one, and BuildSnapshotCheckPointFromVolumeInfo returns
a checkpoint with EnableCSI false and no snapshot handles, so the CSI
snapshot count is never compared against the expected one.
One site already used strings.Contains instead, which handles a list but
matches any feature whose name merely starts with this one, and misses a
value that differs only in case.
Add FeatureEnabled, which splits on commas and compares each entry whole
and case-insensitively, and route all eleven call sites through it. The
table test covers the list, case and prefix shapes that the two previous
idioms each got wrong.
hack/test.sh listed only ./pkg/..., ./internal/... and ./cmd/..., so
nothing under test/ was ever unit tested and this table would not have run
in CI. Add ./test, which holds the suite's shared types and nothing that
needs a cluster. The e2e suites live in subpackages and stay unlisted.
Found while reviewing the kind-csi mapping in #10657, which copied the
EqualFold form from the Azure and AWS branches beside it.
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
* Add changelog for #10661
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
---------
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
e2e: run the RetainPV case too, and move the predicate beside the mapping
The RetainPV case was left out of the matrix on the assumption that it
needed checking first. It did not: it passes. Run on kind with
EnableCSI, Ran 1 of 70 Specs in 354 seconds, 1 passed, installing from
overlays/sc-reclaim-policy/kind-csi and taking two real CSI snapshots of
Kibishii's PVCs. So drop the exclusion rather than add a second row, and
the label becomes plain Basic && Snapshot, which selects the plain
snapshot case and the retained-PV one.
That overlay is the other half of the Kibishii change, and nothing
exercised it until now.
KibishiiVolumesAreHostPath moves from test/types.go to sit beside
installKibishii, since it is the inverse of that function's Kind branch
and the two have to agree on what enabling CSI means. Each now points at
the other in a comment. There is no import cycle: the kibishii package
already dot-imports test, and every call site already dot-imports
kibishii.
Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>
dependabot[bot]
Tiger KaovilaiandClaude Code
Xun Jiang/Bruce Jiang
lyndon-li
Adam Zhang
Daniel Jiang
Lyndon-Li
KrishhnaT
Wenkai Yin(尹文开)
jxj
Max Freedom PollardandClaude Opus 5.5
Abdul Basit
Harshil-1603