From 79e7d1a3c89a0bc2c62aeee9c65fbff8bdd8df70 Mon Sep 17 00:00:00 2001 From: KrishhnaT Date: Fri, 9 Oct 2026 02:29:28 +0530 Subject: [PATCH] e2e: document running the CSI cases on kind (#10630) * e2e: document running the CSI cases on kind The CSI cases are skipped on a plain kind cluster and nothing says how to change that. Describe what hack/install-csi-hostpath.sh installs and why kind needs it, how to run the CSI and VolumeGroupSnapshot label sets, and the StorageClass changes the cases need until those definitions are configurable, Kibishii's own class included. Also two failure modes that are hard to read from their symptoms: two VolumeGroupSnapshotClasses carrying the selector label, which makes Velero refuse to pick one, and a csi-snapshotter sidecar whose caches never sync, which makes snapshots hang rather than fail. Signed-off-by: krishhna24 * Add changelog for #10630 Signed-off-by: krishhna24 * e2e: note the WSL2 address and the exact label matching Two things that cost time to find out and are not written down anywhere. CI builds the BSL URL from $(hostname -i), which is correct on a runner but resolves to an address the cluster cannot reach under WSL2, so the BackupStorageLocation is unreachable and every backup fails to upload. Point readers at the kind network's gateway instead. A Ginkgo label filter matches a bare token by exact equality, not as a substring, so a filter naming CSISnapshot does not select a case labelled CSIVolumeGroupSnapshot. Say so next to the filter, because the failure mode is a case that silently never runs rather than an error. Signed-off-by: krishhna24 --------- Signed-off-by: krishhna24 --- changelogs/unreleased/10630-krishhna24 | 1 + test/e2e/README.md | 87 ++++++++++++++++++++++++++ 2 files changed, 88 insertions(+) create mode 100644 changelogs/unreleased/10630-krishhna24 diff --git a/changelogs/unreleased/10630-krishhna24 b/changelogs/unreleased/10630-krishhna24 new file mode 100644 index 000000000..e27850efb --- /dev/null +++ b/changelogs/unreleased/10630-krishhna24 @@ -0,0 +1 @@ +Document running the CSI e2e cases on a kind cluster diff --git a/test/e2e/README.md b/test/e2e/README.md index 2120e8b55..70f675ede 100644 --- a/test/e2e/README.md +++ b/test/e2e/README.md @@ -161,6 +161,74 @@ Stop kind cluster kind delete cluster ``` +#### CSI snapshots on kind + +kind has no CSI driver that can take snapshots: its default storage is the +local-path provisioner, which no CSI snapshotter can act on. The CSI cases are +therefore skipped on a plain kind cluster. + +`hack/install-csi-hostpath.sh` installs what they need: the snapshot CRDs, the +snapshot controller, the CSI sidecar RBAC and csi-driver-host-path, plus a +`csi-hostpath-sc` StorageClass. Versions are pinned in the script. + +``` bash +kind create cluster +./hack/install-csi-hostpath.sh +``` + +The script also turns on the `CSIVolumeGroupSnapshot` feature gate, which has to +be set on both the snapshot controller and the `csi-snapshotter` sidecar, and is +not enabled by either project's own manifests. Set +`ENABLE_VOLUME_GROUP_SNAPSHOT=false` to leave the group snapshot CRDs and gates +out. + +Then run the CSI cases with `FEATURES=EnableCSI`: + +``` bash +CLOUD_PROVIDER=kind \ +OBJECT_STORE_PROVIDER=aws \ +FEATURES=EnableCSI \ +BSL_CONFIG=region=minio,s3ForcePathStyle="true",s3Url=http://$(hostname -i):9000 \ +CREDS_FILE=/path/to/minio-creds \ +BSL_BUCKET=bucket \ +GINKGO_LABELS="BackupVolumeInfo && (CSISnapshot || CSIDataMover)" \ +make -C test/ run-e2e +``` + +`$(hostname -i)` is what CI uses, but it resolves to an address the cluster +cannot reach under WSL2, which leaves the BackupStorageLocation unreachable and +every backup failing to upload. Use the kind network's gateway instead: + +``` bash +docker network inspect kind -f '{{range .IPAM.Config}}{{.Gateway}} {{end}}' +``` + +Every CSI label has to be named in the filter. A Ginkgo label filter matches a +bare token by exact equality and not as a substring, so `CSISnapshot` does not +select a case labelled `CSIVolumeGroupSnapshot`. + +The PVCs those cases create have to land on a CSI-backed StorageClass. The suite +builds `e2e-storage-class` from the provider's file, and the kind one uses the +local-path provisioner, so until that definition is configurable the file has to +be pointed at `hostpath.csi.k8s.io` locally: + +``` bash +# local change, do not commit it: every other kind job reads this file +sed -i 's|^provisioner: rancher.io/local-path$|provisioner: hostpath.csi.k8s.io|' \ + test/testdata/storage-class/kind.yaml +``` + +Kibishii brings its own StorageClass for the cases that use it +(`kubernetes/yaml/kind/kibishiiKINDStorageClass.yaml` in +`vmware-tanzu-experiments/distributed-data-generator`), which is also local-path, +so those cases need the same change in the clone that `KIBISHII_DIRECTORY` points +at. + +`GINKGO_LABELS="BackupVolumeInfo && CSIVolumeGroupSnapshot"` runs the +VolumeGroupSnapshot case, which additionally needs a VolumeGroupSnapshotClass; +the suite installs one for providers that have the test data under +`test/testdata/volume-group-snapshot-class/`. + 1. Run Velero tests in an AWS cluster: ```bash BSL_PREFIX= \ @@ -419,6 +487,25 @@ Look for the ⛵ emoji printed at the end of each install and uninstall log. The ## `Failed to get bucket region` error If velero log shows `level=error msg="Failed to get bucket region, bucket: xbucket, error: operation error S3: HeadBucket, failed to resolve service endpoint, endpoint rule error, A region must be set when sending requests to S3." backup-storage-location=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-storage-location logSource="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:136" pluginName=velero-plugin-for-aws`, it means you need to set `BSL_CONFIG` to include `region=`. +## `multiple VolumeGroupSnapshotClasses found` error + +If a VolumeGroupSnapshot backup fails with `failed to determine +VolumeGroupSnapshotClass for CSI driver ...: multiple VolumeGroupSnapshotClasses +found`, the cluster has more than one VolumeGroupSnapshotClass carrying the +`velero.io/csi-volumegroupsnapshot-class=true` label and Velero cannot choose +between them. The suite removes the class it installs, so this usually means one +was left behind by hand. `kubectl get volumegroupsnapshotclass` shows them. + +## snapshots that never become ready + +If VolumeSnapshots stay `readyToUse: false` with nothing in the logs, check that +the `csi-snapshotter` sidecar synced its caches: +`kubectl logs csi-hostpathplugin-0 -c csi-snapshotter | grep "Caches populated"`. +A sidecar watching an API the cluster does not serve, for instance the +VolumeGroupSnapshot feature gate is on while the group snapshot CRDs are +missing, waits on a cache that never syncs rather than failing. +`hack/install-csi-hostpath.sh` checks for this at the end of its run. + ## fail fast If need to debug the failed test case, please set the `FAIL_FAST=true` for the `make test-e2e` CLI. If `FAIL_FAST` is enabled, the pipeline with failed test case will not delete the test bed,