diff --git a/sw-block/design/test/v3-phase-15-g15b-k8s-qa-test-instruction.md b/sw-block/design/test/v3-phase-15-g15b-k8s-qa-test-instruction.md index 9c506eb08..5afabf3ac 100644 --- a/sw-block/design/test/v3-phase-15-g15b-k8s-qa-test-instruction.md +++ b/sw-block/design/test/v3-phase-15-g15b-k8s-qa-test-instruction.md @@ -1,14 +1,14 @@ # V3 Phase 15 G15b Kubernetes Static PV QA Test Instruction **Date**: 2026-05-03 -**Status**: K8s lab instruction for `p15-g15b/k8s-static-pv@eb13105`; M02 re-run pending +**Status**: K8s lab instruction for `p15-g15b/k8s-static-pv@95b7217`; M02 re-run pending **Scope**: single-node Kubernetes static PV/PVC/pod smoke through real V3 daemons and CSI. --- ## Headline -At `seaweed_block@eb13105`, the G15b lab harness, image build inputs, and M02 DNS/log-preservation fixes are staged to prove: +At `seaweed_block@95b7217`, the G15b lab harness, image build inputs, M02 DNS/log-preservation fixes, CSI node `iscsi_tcp` module loading, and deterministic attachable-primary binding are staged to prove: ```text blockmaster + product-loop + r1/r2 blockvolume @@ -39,12 +39,17 @@ Known current local limitation: - On the current dev workstation, `kubectl` context `rancher-desktop` exists but API server is not reachable. This instruction needs QA or a running K8s lab. -M02 first-run blocker fixed: +M02 first-run blockers fixed: - `5375add` failed because `hostNetwork: true` blockvolume pods inherited host DNS and could not resolve `blockmaster.kube-system.svc.cluster.local`. - `eb13105` adds `dnsPolicy: ClusterFirstWithHostNet` to both blockvolume pods. - `eb13105` also collects daemon logs on every exit before cleanup, so failure evidence is preserved. +M02 second-run blockers fixed: + +- `eb13105` reached iSCSI attach but failed when M02 lacked the `iscsi_tcp` kernel module. `95b7217` adds a privileged CSI node init container that runs `modprobe iscsi_tcp || true` and adds `kmod` to the CSI image. +- after manual module load, the lab exposed a primary/listener mismatch: r2 could win authority while only r1 exposed the static loopback iSCSI target. `95b7217` changes the G9F-2 bridge so RF>1 verified placements produce a single deterministic frontend-primary `Bind` ask and pins r1/s1 as the first attachable placement slot. + --- ## Commands @@ -55,6 +60,8 @@ Pre-flight from the code repo: cd C:\work\seaweed_block_g9c git checkout p15-g15b/k8s-static-pv go test ./cmd/blockcsi -run TestG15b_Manifest -count=1 -v +go test ./core/host/master -run 'TestG9F2|TestG15b_ProductLoop' -count=1 -v +go test ./internal/testops ./core/host/master ./cmd/blockcsi -count=1 go test ./core/csi ./cmd/blockcsi ./core/host/volume ./core/host/master ./core/authority ./cmd/blockmaster ./cmd/blockvolume -count=1 ``` @@ -72,7 +79,7 @@ G15B_KIND_CLUSTER= bash scripts/build-g15b-images.sh "$PWD" Local image build result already verified at `5375add`: PASS, images `sw-block:local` and `sw-block-csi:local` built. -After pulling `eb13105`, rebuild images before rerun: +After pulling `95b7217`, rebuild images before rerun: ```bash bash scripts/build-g15b-images.sh "$PWD" @@ -102,8 +109,11 @@ Expected result: | Manifest attach path requires `ControllerPublish`. | `TestG15b_Manifest_CSIDriverRequiresAttach` | | Controller deploy uses external-attacher, not provisioner. | `TestG15b_Manifest_ControllerUsesAttacherNotProvisioner` | | Product stack uses G9G cluster-spec/product-loop and RF=2 r1/r2. | `TestG15b_Manifest_ProductStackSingleNodeLoopbackShape` | +| Product loop emits one deterministic frontend-primary Bind for RF=2 static placement. | `TestG15b_ProductLoop_RF2PlacementEmitsSingleDeterministicBind` | +| Attachable r1/s1 is the first placement slot and only r1 exposes the static iSCSI target. | `TestG15b_Manifest_AttachableReplicaIsFirstPlacementSlot` | | Static PV does not carry target endpoint truth. | `TestG15b_Manifest_StaticPVDoesNotEmbedTargetFacts` | | Node plugin has privileged host mount shape. | `TestG15b_Manifest_NodePluginPrivilegedShape` | +| Node plugin loads host `iscsi_tcp` before attach. | `TestG15b_Manifest_NodePluginLoadsISCSITCPModule` | | Pod write/read checksum path. | `scripts/run-g15b-k8s-static.sh` pod phase + `pod.log` | | Product logs captured for debug. | `blockmaster.log`, `blockvolume-r1.log`, `blockvolume-r2.log`, `blockcsi-controller.log` | diff --git a/sw-block/design/v3-phase-15-g15b-k8s-static-pv-mini-plan.md b/sw-block/design/v3-phase-15-g15b-k8s-static-pv-mini-plan.md index 93745523c..508e6a9fc 100644 --- a/sw-block/design/v3-phase-15-g15b-k8s-static-pv-mini-plan.md +++ b/sw-block/design/v3-phase-15-g15b-k8s-static-pv-mini-plan.md @@ -1,7 +1,7 @@ # V3 Phase 15 — G15b Kubernetes Static PV Mini-Plan **Date**: 2026-05-03 -**Status**: G15b-1 manifests implemented at `62325c9`; G15b-2 lab harness staged at `32b3a13`; image build inputs added at `5375add`; M02 first run found DNS/harness blockers; fixed at `eb13105`; Kubernetes re-run pending +**Status**: G15b-1 manifests implemented at `62325c9`; G15b-2 lab harness staged at `32b3a13`; image build inputs added at `5375add`; M02 first run found DNS/harness blockers fixed at `eb13105`; M02 second run found `iscsi_tcp` + primary/listener alignment blockers fixed at `95b7217`; Kubernetes re-run pending **Branch**: `p15-g15b/k8s-static-pv` from `ac49adb` **Goal**: prove a Kubernetes pod can consume a pre-provisioned V3 block volume through `cmd/blockcsi`, using real Kubernetes CSI control flow and real Linux iSCSI staging. @@ -162,7 +162,7 @@ Result: PASS on `62325c9`. ### G15b-2 — K8s Lab Harness -Status: **harness staged** at `seaweed_block@32b3a13`; image build inputs added at `seaweed_block@5375add`; DNS/logging fixes at `seaweed_block@eb13105`; real Kubernetes re-run pending. +Status: **harness staged** at `seaweed_block@32b3a13`; image build inputs added at `seaweed_block@5375add`; DNS/logging fixes at `seaweed_block@eb13105`; iSCSI TCP module loading + deterministic attachable-primary bridge fix at `seaweed_block@95b7217`; real Kubernetes re-run pending. Artifacts: @@ -200,6 +200,19 @@ Fix at `eb13105`: - changes `scripts/run-g15b-k8s-static.sh` so daemon logs are collected from the EXIT trap before cleanup; - adds `.gitattributes` to keep `*.sh` as LF on future checkouts. +M02 second-run findings: + +- M02 did not have `iscsi_tcp` loaded, so kubelet/CSI `NodeStage` failed with `iSCSI driver tcp is not loaded`; +- after manual `modprobe iscsi_tcp`, Kubernetes reached `ControllerPublish` and iSCSI login, but the volume still did not materialize because r2 won authority while only r1 exposed the static loopback iSCSI target; +- root cause: the G9F-2 product bridge mapped every RF=2 verified placement slot to a competing `Bind` ask for the same volume; authority's queue keeps latest desired ask per volume, so r2 could overwrite r1 nondeterministically. + +Fix at `95b7217`: + +- adds a privileged CSI node init container that runs `modprobe iscsi_tcp || true` and mounts host `/lib/modules`; +- adds `kmod` to the `sw-block-csi:local` image; +- changes the G9F-2 verified-placement bridge so RF>1 slots produce one deterministic frontend-primary `Bind` ask (first verified slot) instead of multiple competing `Bind` asks; +- pins the G15b static manifest assumption that the attachable iSCSI replica (`r1/s1`) is the first placement slot and that r2 does not expose a competing static iSCSI target. + Harness responsibilities: 1. Build V3 binaries/images for `blockmaster`, `blockvolume`, and `blockcsi`. @@ -218,10 +231,12 @@ Pass: - Pod writes and reads byte-equal data. - No dangling iSCSI session for the test IQN after cleanup. -Pre-flight verification green at `eb13105`: +Pre-flight verification green at `95b7217`: ```powershell go test ./cmd/blockcsi -run TestG15b_Manifest -count=1 -v +go test ./core/host/master -run 'TestG9F2|TestG15b_ProductLoop' -count=1 -v +go test ./internal/testops ./core/host/master ./cmd/blockcsi -count=1 go test ./core/csi ./cmd/blockcsi ./core/host/volume ./core/host/master ./core/authority ./cmd/blockmaster ./cmd/blockvolume -count=1 ``` @@ -236,7 +251,7 @@ Result: PASS; built `sw-block:local` and `sw-block-csi:local`. Not yet proven: - Kubernetes API server availability; -- image load path into the target cluster after rebuilding at `eb13105`; +- image load path into the target cluster after rebuilding at `95b7217`; - external-attacher calling `ControllerPublish`; - kubelet calling `NodeStage` / `NodePublish`; - pod checksum write/read.