e2e: document running the CSI cases on kind (#10630)

* e2e: document running the CSI cases on kind

The CSI cases are skipped on a plain kind cluster and nothing says how to
change that. Describe what hack/install-csi-hostpath.sh installs and why
kind needs it, how to run the CSI and VolumeGroupSnapshot label sets, and
the StorageClass changes the cases need until those definitions are
configurable, Kibishii's own class included.

Also two failure modes that are hard to read from their symptoms: two
VolumeGroupSnapshotClasses carrying the selector label, which makes
Velero refuse to pick one, and a csi-snapshotter sidecar whose caches
never sync, which makes snapshots hang rather than fail.

Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>

* Add changelog for #10630

Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>

* e2e: note the WSL2 address and the exact label matching

Two things that cost time to find out and are not written down anywhere.

CI builds the BSL URL from $(hostname -i), which is correct on a runner but
resolves to an address the cluster cannot reach under WSL2, so the
BackupStorageLocation is unreachable and every backup fails to upload. Point
readers at the kind network's gateway instead.

A Ginkgo label filter matches a bare token by exact equality, not as a
substring, so a filter naming CSISnapshot does not select a case labelled
CSIVolumeGroupSnapshot. Say so next to the filter, because the failure mode is
a case that silently never runs rather than an error.

Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>

---------

Signed-off-by: krishhna24 <krishhnatupedev@gmail.com>

changelogs/unreleased/10630-krishhna24

@@ -0,0 +1 @@
+Document running the CSI e2e cases on a kind cluster

test/e2e/README.md

@@ -161,6 +161,74 @@ Stop kind cluster
 kind delete cluster
 ```
 
+#### CSI snapshots on kind
+
+kind has no CSI driver that can take snapshots: its default storage is the
+local-path provisioner, which no CSI snapshotter can act on. The CSI cases are
+therefore skipped on a plain kind cluster.
+
+`hack/install-csi-hostpath.sh` installs what they need: the snapshot CRDs, the
+snapshot controller, the CSI sidecar RBAC and csi-driver-host-path, plus a
+`csi-hostpath-sc` StorageClass. Versions are pinned in the script.
+
+``` bash
+kind create cluster
+./hack/install-csi-hostpath.sh
+```
+
+The script also turns on the `CSIVolumeGroupSnapshot` feature gate, which has to
+be set on both the snapshot controller and the `csi-snapshotter` sidecar, and is
+not enabled by either project's own manifests. Set
+`ENABLE_VOLUME_GROUP_SNAPSHOT=false` to leave the group snapshot CRDs and gates
+out.
+
+Then run the CSI cases with `FEATURES=EnableCSI`:
+
+``` bash
+CLOUD_PROVIDER=kind \
+OBJECT_STORE_PROVIDER=aws \
+FEATURES=EnableCSI \
+BSL_CONFIG=region=minio,s3ForcePathStyle="true",s3Url=http://$(hostname -i):9000 \
+CREDS_FILE=/path/to/minio-creds \
+BSL_BUCKET=bucket \
+GINKGO_LABELS="BackupVolumeInfo && (CSISnapshot || CSIDataMover)" \
+make -C test/ run-e2e
+```
+
+`$(hostname -i)` is what CI uses, but it resolves to an address the cluster
+cannot reach under WSL2, which leaves the BackupStorageLocation unreachable and
+every backup failing to upload. Use the kind network's gateway instead:
+
+``` bash
+docker network inspect kind -f '{{range .IPAM.Config}}{{.Gateway}} {{end}}'
+```
+
+Every CSI label has to be named in the filter. A Ginkgo label filter matches a
+bare token by exact equality and not as a substring, so `CSISnapshot` does not
+select a case labelled `CSIVolumeGroupSnapshot`.
+
+The PVCs those cases create have to land on a CSI-backed StorageClass. The suite
+builds `e2e-storage-class` from the provider's file, and the kind one uses the
+local-path provisioner, so until that definition is configurable the file has to
+be pointed at `hostpath.csi.k8s.io` locally:
+
+``` bash
+# local change, do not commit it: every other kind job reads this file
+sed -i 's|^provisioner: rancher.io/local-path$|provisioner: hostpath.csi.k8s.io|' \
+  test/testdata/storage-class/kind.yaml
+```
+
+Kibishii brings its own StorageClass for the cases that use it
+(`kubernetes/yaml/kind/kibishiiKINDStorageClass.yaml` in
+`vmware-tanzu-experiments/distributed-data-generator`), which is also local-path,
+so those cases need the same change in the clone that `KIBISHII_DIRECTORY` points
+at.
+
+`GINKGO_LABELS="BackupVolumeInfo && CSIVolumeGroupSnapshot"` runs the
+VolumeGroupSnapshot case, which additionally needs a VolumeGroupSnapshotClass;
+the suite installs one for providers that have the test data under
+`test/testdata/volume-group-snapshot-class/`.
+
 1. Run Velero tests in an AWS cluster:
 ```bash
 BSL_PREFIX=<PREFIX_UNDER_BUCKET> \
@@ -419,6 +487,25 @@ Look for the ⛵ emoji printed at the end of each install and uninstall log. The
 ## `Failed to get bucket region` error
 If velero log shows `level=error msg="Failed to get bucket region, bucket: xbucket, error: operation error S3: HeadBucket, failed to resolve service endpoint, endpoint rule error, A region must be set when sending requests to S3." backup-storage-location=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-storage-location logSource="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:136" pluginName=velero-plugin-for-aws`, it means you need to set `BSL_CONFIG` to include `region=<region>`.
 
+## `multiple VolumeGroupSnapshotClasses found` error
+
+If a VolumeGroupSnapshot backup fails with `failed to determine
+VolumeGroupSnapshotClass for CSI driver ...: multiple VolumeGroupSnapshotClasses
+found`, the cluster has more than one VolumeGroupSnapshotClass carrying the
+`velero.io/csi-volumegroupsnapshot-class=true` label and Velero cannot choose
+between them. The suite removes the class it installs, so this usually means one
+was left behind by hand. `kubectl get volumegroupsnapshotclass` shows them.
+
+## snapshots that never become ready
+
+If VolumeSnapshots stay `readyToUse: false` with nothing in the logs, check that
+the `csi-snapshotter` sidecar synced its caches:
+`kubectl logs csi-hostpathplugin-0 -c csi-snapshotter | grep "Caches populated"`.
+A sidecar watching an API the cluster does not serve, for instance the
+VolumeGroupSnapshot feature gate is on while the group snapshot CRDs are
+missing, waits on a cache that never syncs rather than failing.
+`hack/install-csi-hostpath.sh` checks for this at the end of its run.
+
 ## fail fast
 If need to debug the failed test case, please set the `FAIL_FAST=true` for the `make test-e2e` CLI.
 If `FAIL_FAST` is enabled, the pipeline with failed test case will not delete the test bed,