diff --git a/changelogs/unreleased/10630-krishhna24 b/changelogs/unreleased/10630-krishhna24 new file mode 100644 index 000000000..e27850efb --- /dev/null +++ b/changelogs/unreleased/10630-krishhna24 @@ -0,0 +1 @@ +Document running the CSI e2e cases on a kind cluster diff --git a/test/e2e/README.md b/test/e2e/README.md index 2120e8b55..70f675ede 100644 --- a/test/e2e/README.md +++ b/test/e2e/README.md @@ -161,6 +161,74 @@ Stop kind cluster kind delete cluster ``` +#### CSI snapshots on kind + +kind has no CSI driver that can take snapshots: its default storage is the +local-path provisioner, which no CSI snapshotter can act on. The CSI cases are +therefore skipped on a plain kind cluster. + +`hack/install-csi-hostpath.sh` installs what they need: the snapshot CRDs, the +snapshot controller, the CSI sidecar RBAC and csi-driver-host-path, plus a +`csi-hostpath-sc` StorageClass. Versions are pinned in the script. + +``` bash +kind create cluster +./hack/install-csi-hostpath.sh +``` + +The script also turns on the `CSIVolumeGroupSnapshot` feature gate, which has to +be set on both the snapshot controller and the `csi-snapshotter` sidecar, and is +not enabled by either project's own manifests. Set +`ENABLE_VOLUME_GROUP_SNAPSHOT=false` to leave the group snapshot CRDs and gates +out. + +Then run the CSI cases with `FEATURES=EnableCSI`: + +``` bash +CLOUD_PROVIDER=kind \ +OBJECT_STORE_PROVIDER=aws \ +FEATURES=EnableCSI \ +BSL_CONFIG=region=minio,s3ForcePathStyle="true",s3Url=http://$(hostname -i):9000 \ +CREDS_FILE=/path/to/minio-creds \ +BSL_BUCKET=bucket \ +GINKGO_LABELS="BackupVolumeInfo && (CSISnapshot || CSIDataMover)" \ +make -C test/ run-e2e +``` + +`$(hostname -i)` is what CI uses, but it resolves to an address the cluster +cannot reach under WSL2, which leaves the BackupStorageLocation unreachable and +every backup failing to upload. Use the kind network's gateway instead: + +``` bash +docker network inspect kind -f '{{range .IPAM.Config}}{{.Gateway}} {{end}}' +``` + +Every CSI label has to be named in the filter. A Ginkgo label filter matches a +bare token by exact equality and not as a substring, so `CSISnapshot` does not +select a case labelled `CSIVolumeGroupSnapshot`. + +The PVCs those cases create have to land on a CSI-backed StorageClass. The suite +builds `e2e-storage-class` from the provider's file, and the kind one uses the +local-path provisioner, so until that definition is configurable the file has to +be pointed at `hostpath.csi.k8s.io` locally: + +``` bash +# local change, do not commit it: every other kind job reads this file +sed -i 's|^provisioner: rancher.io/local-path$|provisioner: hostpath.csi.k8s.io|' \ + test/testdata/storage-class/kind.yaml +``` + +Kibishii brings its own StorageClass for the cases that use it +(`kubernetes/yaml/kind/kibishiiKINDStorageClass.yaml` in +`vmware-tanzu-experiments/distributed-data-generator`), which is also local-path, +so those cases need the same change in the clone that `KIBISHII_DIRECTORY` points +at. + +`GINKGO_LABELS="BackupVolumeInfo && CSIVolumeGroupSnapshot"` runs the +VolumeGroupSnapshot case, which additionally needs a VolumeGroupSnapshotClass; +the suite installs one for providers that have the test data under +`test/testdata/volume-group-snapshot-class/`. + 1. Run Velero tests in an AWS cluster: ```bash BSL_PREFIX= \ @@ -419,6 +487,25 @@ Look for the ⛵ emoji printed at the end of each install and uninstall log. The ## `Failed to get bucket region` error If velero log shows `level=error msg="Failed to get bucket region, bucket: xbucket, error: operation error S3: HeadBucket, failed to resolve service endpoint, endpoint rule error, A region must be set when sending requests to S3." backup-storage-location=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-storage-location logSource="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:136" pluginName=velero-plugin-for-aws`, it means you need to set `BSL_CONFIG` to include `region=`. +## `multiple VolumeGroupSnapshotClasses found` error + +If a VolumeGroupSnapshot backup fails with `failed to determine +VolumeGroupSnapshotClass for CSI driver ...: multiple VolumeGroupSnapshotClasses +found`, the cluster has more than one VolumeGroupSnapshotClass carrying the +`velero.io/csi-volumegroupsnapshot-class=true` label and Velero cannot choose +between them. The suite removes the class it installs, so this usually means one +was left behind by hand. `kubectl get volumegroupsnapshotclass` shows them. + +## snapshots that never become ready + +If VolumeSnapshots stay `readyToUse: false` with nothing in the logs, check that +the `csi-snapshotter` sidecar synced its caches: +`kubectl logs csi-hostpathplugin-0 -c csi-snapshotter | grep "Caches populated"`. +A sidecar watching an API the cluster does not serve, for instance the +VolumeGroupSnapshot feature gate is on while the group snapshot CRDs are +missing, waits on a cache that never syncs rather than failing. +`hack/install-csi-hostpath.sh` checks for this at the end of its run. + ## fail fast If need to debug the failed test case, please set the `FAIL_FAST=true` for the `make test-e2e` CLI. If `FAIL_FAST` is enabled, the pipeline with failed test case will not delete the test bed,