diff --git a/sw-block/design/test/v3-phase-15-g15a-privileged-qa-test-instruction.md b/sw-block/design/test/v3-phase-15-g15a-privileged-qa-test-instruction.md new file mode 100644 index 000000000..097b353de --- /dev/null +++ b/sw-block/design/test/v3-phase-15-g15a-privileged-qa-test-instruction.md @@ -0,0 +1,284 @@ +# V3 Phase 15 G15a Privileged QA Test Instruction + +Date: 2026-05-03 +Status: privileged-path verified on m01 at `p15-g15a/csi-static-mvp@ac49adb` +Scope: real `iscsiadm` discovery+login + `mkfs.ext4` + `mount` + bind-mount + filesystem write/read byte-equal + clean teardown, all driven through `cmd/blockcsi` NodeStage / NodePublish against real product daemons +Companion: `v3-phase-15-g15a-qa-test-instruction.md` (non-privileged L2 control-plane verification) + +## Headline + +At `p15-g15a/csi-static-mvp@ac49adb`, on Linux host m01 (192.168.1.181, Ubuntu +6.17 kernel, open-iscsi, sudo NOPW), G15a's `cmd/blockcsi` binary drives the +full privileged data-plane sequence end-to-end: `iscsiadm` discovery, login, +device-by-IQN wait, `mkfs.ext4`, mount-to-staging, bind-mount-to-target, 4KB +random byte-equal write/read through the mounted device, `NodeUnpublish` +(unmount target), `NodeUnstage` (unmount staging + iscsiadm logout), +`ControllerUnpublish`, with no dangling iSCSI sessions remaining. + +## Environment + +- **Repo**: `seaweed_block` +- **Branch**: `p15-g15a/csi-static-mvp` +- **Minimum commit**: `ac49adb` +- **Host**: m01 (192.168.1.181) — Linux 6.17, open-iscsi, sudo NOPW for `testdev` +- **SSH**: `ssh -i C:/work/dev_server/testdev_key testdev@192.168.1.181` +- **SMB harness path**: `V:\share\g15a-priv\` (Windows) = `/mnt/smb/work/share/g15a-priv/` (Linux) +- **Go**: 1.26.2 (project default) +- **Privileges required**: root (test invokes `sudo` and runs the test binary as root so `iscsiadm` and `mount` work) +- **Why m01 specifically**: the only host in our test fleet with kernel iSCSI initiator + sudo NOPW + clean baseline; G15a-K8s (real kubelet PVC) is forward-carry to a dedicated K8s test cluster + +## Pre-conditions + +Before running, verify on m01: + +```bash +ssh -i C:/work/dev_server/testdev_key testdev@192.168.1.181 ' + which iscsiadm mkfs.ext4 sudo go && + go version && + sudo -n true && echo SUDO_NOPW_OK && + sudo iscsiadm -m session 2>&1 | head -3 +' +``` + +Expected: + +- `iscsiadm`, `mkfs.ext4`, `sudo`, `go` all resolved +- Go ≥ 1.24 +- `SUDO_NOPW_OK` printed +- `iscsiadm -m session` reports `No active sessions` (or no session for `iqn.2026-05.io.seaweedfs:g15a-priv-v1`) + +If any baseline iSCSI session exists for the test IQN, log out before running: + +```bash +sudo iscsiadm -m node -T iqn.2026-05.io.seaweedfs:g15a-priv-v1 --logout || true +``` + +## Run command + +### One-shot via the harness runner with persistent artifacts + +The harness at `V:\share\g15a-priv\` contains: + +- `privileged_test.go` — the Go test runner (single file, ~250 LOC). Reads `G15A_ARTIFACT_DIR` env var; if set, all per-daemon logs land there instead of `t.TempDir()`. +- `run-g15a-privileged.sh` — bash wrapper that builds binaries, stages the test, compiles, runs under sudo, prints baseline + post-run iSCSI sessions, and stages logs to a per-run artifact directory. + +Ship a fresh source tree to m01 and invoke with a persistent run directory: + +```bash +RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)" + +# from your dev box (Windows or Linux) +git -C /path/to/seaweed_block archive --format=tar HEAD | \ + ssh -i C:/work/dev_server/testdev_key testdev@192.168.1.181 \ + 'rm -rf /tmp/g15a-priv/src && mkdir -p /tmp/g15a-priv/src && tar x -C /tmp/g15a-priv/src' + +# trigger the run on m01 with persistent artifact dir +ssh -i C:/work/dev_server/testdev_key testdev@192.168.1.181 " + cp /mnt/smb/work/share/g15a-priv/privileged_test.go /tmp/g15a-priv/src/cmd/blockcsi/privileged_test.go + cd /tmp/g15a-priv/src + go build -o /tmp/g15a-priv/bin/blockmaster ./cmd/blockmaster + go build -o /tmp/g15a-priv/bin/blockvolume ./cmd/blockvolume + go build -o /tmp/g15a-priv/bin/blockcsi ./cmd/blockcsi + go test -c -o /tmp/g15a-priv/blockcsi-priv.test ./cmd/blockcsi + mkdir -p /mnt/smb/work/share/g15a-priv/runs/$RUN_ID + sudo G15A_PRIVILEGED=1 G15A_BIN_DIR=/tmp/g15a-priv/bin \ + G15A_ARTIFACT_DIR=/mnt/smb/work/share/g15a-priv/runs/$RUN_ID \ + /tmp/g15a-priv/blockcsi-priv.test \ + -test.run TestG15aPriv -test.v -test.timeout=180s \ + 2>&1 | tee /mnt/smb/work/share/g15a-priv/runs/$RUN_ID/test-stdout.log +" +echo "Artifacts at: V:\\share\\g15a-priv\\runs\\$RUN_ID\\" +``` + +Expected wall clock: ~10 s build + ~6 s test = ~16 s total. Artifacts persist +on the SMB share for inspection without an additional SSH round-trip. + +> **Important**: the `G15A_ARTIFACT_DIR` env var is what makes logs survive a +> green test. Without it, `privileged_test.go` falls back to `t.TempDir()` and +> Go cleans up on success, leaving only the test stdout phase markers — no +> per-daemon logs to inspect. + +### Artifact directory layout + +After a green run, `V:\share\g15a-priv\runs\\` contains: + +| File / dir | Size (typical) | Purpose | +|---|---|---| +| `test-stdout.log` | ~1 KB | Test framework output: 6 phase markers + PASS line. Read first to see the timeline. | +| `blockmaster.log` | ~270 B | Master daemon stderr. Quiet — startup, lock acquired, listen address. | +| `blockvolume-primary.log` | **~20 KB** | **Most informative log.** Contains: `g7-debug` adapter R/S/H state transitions, real iSCSI session lifecycle (kernel initiator IQN visible), every SCSI WRITE/SYNCHRONIZE_CACHE during `mkfs.ext4`, replication barrier waits with r2 at each sync, the test payload write at `lba=0`, and session invalidate on cleanup. | +| `blockvolume-replica.log` | ~270 B | r2 observes r1 is primary, records supersede, doesn't apply locally. | +| `topology.yaml` | ~130 B | The 2-slot RF=2 spec the master loaded. | +| `staging/` `store-primary/` `store-replica/` `authority-store/` | empty post-cleanup | Mount points + durable roots. Empty after `NodeUnstage` — that emptiness is itself evidence cleanup worked. | + +**Currently missing**: `blockcsi.log` (the CSI driver's own iscsiadm/mount call log) is **not** in the persisted dir. The current harness writes it to `t.TempDir()` directly, bypassing `G15A_ARTIFACT_DIR`. Forward-carry: extend `startG15aCSI` helper to honor `art` parameter (3-line change). Workaround until then: wrap with `strace -f -e trace=execve` to capture iscsiadm/mkfs/mount syscalls. + +### What the dev agent can verify directly from the logs + +From `blockvolume-primary.log`: + +1. **Real kernel iSCSI initiator attached** (look for): + ``` + session: FullFeature initiator="iqn.2004-10.com.ubuntu:01:..." target="iqn.2026-05.io.seaweedfs:g15a-priv-v1" type="Normal" + ``` + (Discovery session before this gets reset by peer — normal two-phase iscsiadm behavior, not a bug.) + +2. **mkfs.ext4 generated ~40 SCSI WRITEs** for filesystem metadata (search `iscsi: SCSI WRITE handler lba=`). The test payload write is the one with `lba=0 transferLen=1 dataOut=4096`. + +3. **Every WRITE replicated to r2 with barrier handshake** — for the test payload at `lsn=42`: + ``` + replication: OnLocalWrite volume=v1 lba=0 lsn=42 peers=1 + replication: ship ok peer=r2 lba=0 lsn=42 + replication: barrier wait peer=r2 addr=... targetLSN=42 epoch=1 + replication: barrier ack peer=r2 targetLSN=42 achievedLSN=42 + ``` + Proves byte-equal payload was both durable on primary AND replicated to r2 before the read. + +4. **Cleanup observed** at the end: + ``` + executor: invalidate session 1 for r2: replica peer closed + ``` + +### Manual / debug invocation (no harness wrapper) + +If you want to iterate on the test logic without re-shipping source, the test +file lives at `/mnt/smb/work/share/g15a-priv/privileged_test.go`. Edit it +directly via SMB, then on m01 run the inner block of the SSH heredoc above. + +## Phase-by-phase scenario + +The single test `TestG15aPriv_NodeStageBindWriteReadCleanup` drives 6 phases. +Each phase logs its own success line so you can see in `-test.v` output exactly +where a failure happened. + +### Phase 1 — ControllerPublishVolume returns matching frontend fact + +- Calls `ControllerPublishVolume(VolumeId=v1, NodeId=node-a)` against `blockcsi`. +- Polls up to 15 s; passes when `publish_context["iscsiAddr"]` and `["iqn"]` match the values seeded into `blockvolume`'s iSCSI listen flags. +- Log: `Phase 1 OK: ControllerPublish returned iscsiAddr=... iqn=...` + +### Phase 2 — NodeStageVolume drives iscsiadm + mkfs + mount + +- Calls `NodeStageVolume` with the `publish_context` from Phase 1. +- Internally `cmd/blockcsi` invokes `iscsiadm -m discovery` then `iscsiadm -m node ... --login`, waits for `/dev/disk/by-path/**` to appear, then `mkfs.ext4` and `mount` to the staging path. +- Verified by `mountpoint -q ` (with `/proc/mounts` fallback). +- Log: `Phase 2 OK: NodeStage completed (iscsiadm discovery+login + mkfs + mount)` + `Phase 2 verified: is a mountpoint` + +### Phase 3 — NodePublishVolume bind-mounts staging → target + +- Calls `NodePublishVolume` to bind-mount staging into the target path. +- Verified by `mountpoint -q `. +- Log: `Phase 3 OK: NodePublish completed (bind mount staging → target)` + +### Phase 4 — Byte-equal write/read through the CSI mount + +- Generates 4096 random bytes via `crypto/rand`. +- Writes to `/g15a-priv-byteequal.bin`. +- Calls `sync` to flush page cache. +- Reads back, asserts byte-equal. +- Log: `Phase 4 OK: 4096 bytes round-tripped byte-equal through CSI-mounted iSCSI device` + +### Phase 5 — Explicit cleanup chain + +- `NodeUnpublishVolume` → unmount target. +- `NodeUnstageVolume` → unmount staging + `iscsiadm --logout`. +- `ControllerUnpublishVolume` → soft-tolerated (G15a static MVP may return Unimplemented; the test logs and continues). +- Log: `Phase 5 OK: NodeUnpublish + NodeUnstage clean` + +### Phase 6 — No dangling iSCSI session + +- Runs `iscsiadm -m session`, asserts the test IQN is **not** present in output. +- Log: `Phase 6 OK: no dangling iSCSI session for iqn.2026-05.io.seaweedfs:g15a-priv-v1` + +## QA evidence pins + +### Pin 1 — first verification (ephemeral logs, t.TempDir()) + +``` +Tested by: QA +Date: 2026-05-03 +Tree: p15-g15a/csi-static-mvp@ac49adb +Host: m01 (192.168.1.181), Linux 6.17, Go 1.26.2 linux/amd64 +Result: PASS in 5.69s +Baseline iSCSI sessions: none +Post-run iSCSI sessions: none +Logs: ephemeral (t.TempDir, cleaned by Go on success) +``` + +### Pin 2 — re-run with persistent artifact dir + +``` +Tested by: QA +Date: 2026-05-03 +Tree: p15-g15a/csi-static-mvp@ac49adb (same source) +Host: m01 (192.168.1.181), Linux 6.17, Go 1.26.2 linux/amd64 +Result: PASS in 5.93s +Baseline iSCSI sessions: none +Post-run iSCSI sessions: none +Run ID: 20260503T153623Z +Artifact dir: V:\share\g15a-priv\runs\20260503T153623Z\ +Persisted: test-stdout.log, blockmaster.log, blockvolume-primary.log (19 KB), + blockvolume-replica.log, topology.yaml +Missing: blockcsi.log (forward-carry harness fix) +``` + +Phases all green in both runs: + +1. ControllerPublish returned matching iscsiAddr + iqn +2. NodeStage (iscsiadm discovery+login + mkfs.ext4 + mount) + mountpoint verified +3. NodePublish (bind mount staging → target) + mountpoint verified +4. 4096 random bytes round-tripped byte-equal through CSI-mounted iSCSI device +5. NodeUnpublish + NodeUnstage clean (ControllerUnpublish acceptable Unimplemented at static MVP) +6. No dangling iSCSI session for the test IQN + +``` +Test file: V:\share\g15a-priv\privileged_test.go +Runner script: V:\share\g15a-priv\run-g15a-privileged.sh +``` + +Re-run on a different commit will produce a fresh evidence pin; the two pins +above are canonical for the privileged-path verification at `ac49adb`. + +## Non-claims (this instruction does not validate) + +1. **K8s integration** — no real `kubelet` / PVC / Pod / scheduler involved. Real kubelet integration is forward-carry to a dedicated K8s test cluster. +2. **Multi-node attach** — single node (m01), single ControllerPublish/NodeStage cycle. Multi-attach (RWX) is not tested. +3. **Partition table / filesystem variety** — only `ext4` direct on the device. No XFS, no LVM, no GPT partitioning. +4. **Block volume mode** — only mount-mode (filesystem) tested. Raw block-mode CSI capability is not exercised. +5. **Concurrent write/read across reconnect** — single sequential write+sync+read. Concurrent or post-restart write/read is not covered. +6. **Failover during stage** — primary is not killed mid-NodeStage. G8 covers primary-kill data continuity at the storage layer; CSI-layer failover under workload is forward-carry. +7. **Persistence across plugin restart** — test starts and stops the `cmd/blockcsi` plugin once; "restart plugin then unstage stale volume" is not exercised. +8. **CSI snapshot / clone / resize** — G15a static MVP does not advertise these capabilities; not relevant. +9. **Performance / IOPS / throughput** — single 4KB I/O, no perf claim. G21 owns SLO. +10. **CHAP / mutual auth / TLS** — beta-defer until G16. + +## Forward-carry + +| Item | Owner | Where it lands | +|---|---|---| +| **Plugin restart + stale-volume unstage**: kill `cmd/blockcsi` mid-staging, restart, verify `NodeUnstageVolume` cleans up | sw + QA on m01 | G15a-6 followup or G15a-K8s | +| **Multi-node attach (single-writer enforcement)**: second node attempts NodeStage same volume, expect rejection | QA on m01 + a second Linux host | G15a-RWO test (separate instruction) | +| **Block-mode capability**: NodeStage with `Block{}` access type instead of `Mount{}` | sw must wire through, QA verifies | G15a block-mode followup | +| **Real K8s kubelet driving the same flow**: PVC → Pod schedules → kubelet calls CSI → workload writes/reads | QA | G15a-K8s on dedicated cluster | +| **Failure injection during NodeStage** (e.g., iscsiadm timeout, mkfs failure) | sw + QA | G15a-fault followup | +| **Post-G8 failover under live mount**: kill primary while file is being written, verify CSI re-discovery / device re-attach behavior | sw + QA | G15a × G8 cross-gate test | +| **`blockcsi.log` capture in artifact dir**: extend `startG15aCSI` helper to honor `art` (3-line change) so the CSI driver's own iscsiadm/mount call log lands alongside daemon logs | QA harness | trivial follow-up; bundle with first edit to `privileged_test.go` | + +## Cleanup / re-run safety + +The test always registers `t.Cleanup` for both `NodeUnpublishVolume` and +`NodeUnstageVolume` so even on test failure mid-flight, the iSCSI session and +mounts get torn down. If a test panic somehow leaves residue, manual cleanup: + +```bash +sudo umount /tmp/TestG15aPriv*/001/target 2>/dev/null || true +sudo umount /tmp/TestG15aPriv*/001/staging 2>/dev/null || true +sudo iscsiadm -m node -T iqn.2026-05.io.seaweedfs:g15a-priv-v1 --logout 2>/dev/null || true +sudo iscsiadm -m node -T iqn.2026-05.io.seaweedfs:g15a-priv-v1 -o delete 2>/dev/null || true +``` + +After cleanup, `sudo iscsiadm -m session` should report `No active sessions`. + +## Close sentence + +> *G15a privileged QA proves the V3 `cmd/blockcsi` binary correctly drives the full Linux block storage stack — kernel iSCSI initiator, ext4 format, kernel mount, bind-mount, filesystem I/O — against real V3 product daemons, with byte-equal data correctness and clean teardown leaving no kernel-level residue.* diff --git a/sw-block/design/test/v3-phase-15-g15a-qa-test-instruction.md b/sw-block/design/test/v3-phase-15-g15a-qa-test-instruction.md new file mode 100644 index 000000000..2594515b1 --- /dev/null +++ b/sw-block/design/test/v3-phase-15-g15a-qa-test-instruction.md @@ -0,0 +1,171 @@ +# V3 Phase 15 G15a QA Test Instruction + +Date: 2026-05-03 +Status: non-privileged L2 close-candidate on `p15-g15a/csi-static-mvp@ac49adb` +Scope: blockcsi static MVP control-plane path through real product daemons; non-privileged only + +## Headline + +At `p15-g15a/csi-static-mvp@ac49adb`, G15a proves a real `blockcsi` driver can +serve CSI Identity, look up assigned frontend facts from a real `blockmaster` +status path, and return a populated `publish_context` from `ControllerPublishVolume` +that matches the actual `iscsiAddr` and `iqn` exposed by a real `blockvolume` +iSCSI target. + +This instruction covers what runs on plain CI / Windows / unprivileged Linux. +Privileged steps (real `iscsiadm` discovery+login, `mkfs`, `mount`, file write/read) +require an authenticated Linux test host and are out of scope for this instruction; +see §Non-claims and §Forward-carry. + +## Environment + +- Repo: `seaweed_block` +- Branch: `p15-g15a/csi-static-mvp` +- Minimum commit: `ac49adb` +- Fidelity: subprocess L2 with real `cmd/blockmaster` + real `cmd/blockvolume` (×2 for r1/r2) + real `cmd/blockcsi` +- Platform: any Go-capable host (Windows, macOS, plain Linux without iscsiadm); no `root` / privileged container required +- Go version: project default (CGO_ENABLED=1 on Windows; no `-race` flag on Windows per project policy) + +## Full Regression Command + +```bash +go test ./core/csi ./cmd/blockcsi ./core/host/volume ./core/host/master ./core/authority ./cmd/blockmaster ./cmd/blockvolume -count=1 +``` + +Expected: all 7 packages PASS. + +Wall-clock guidance on a typical dev box: + +| Package | Approx. time | +|---|---| +| `core/csi` | < 0.1 s | +| `cmd/blockcsi` | ~25-30 s (includes L2 build + subprocess spawn) | +| `core/host/volume` | < 0.1 s | +| `core/host/master` | < 0.2 s | +| `core/authority` | ~7 s | +| `cmd/blockmaster` | < 0.1 s | +| `cmd/blockvolume` | ~120-150 s (includes G8 + G9 subprocess tests) | +| **Total** | ~3 min | + +## Focused G15a L2 Command + +```bash +go test ./cmd/blockcsi -run TestG15a_BlockCSIControllerPublishUsesMasterFrontendFact -count=1 -v +``` + +Expected: PASS in ~20-25 s on a local dev box. Includes building all three binaries +(`blockmaster`, `blockvolume`, `blockcsi`) in a temp dir, spawning them, polling +`ControllerPublishVolume` for up to 15 s until the published `iscsiAddr`/`iqn` +match the values the test seeded into the volume. + +## Scenario Checklist + +Each scenario lists the backing test so dev agents can re-run a single scenario +in isolation when iterating. + +### CSI service-layer scenarios (component, fast) + +1. **Identity service handles nil requests safely.** + - `core/csi/identity_endpoint_test.go::TestIdentity_NilRequests` + - `core/csi/identity_endpoint_test.go::TestParseEndpoint` + +2. **CSI package does not import authority and does not construct AssignmentFacts.** + Structural guard from `v3-phase-15-control-plane-evolution.md §7.1`. + - `core/csi/boundary_guard_test.go::TestG15a_CSI_DoesNotImportAuthorityOrConstructAssignmentFacts` + +3. **`ControllerPublishVolume` returns iSCSI publish_context built from a verified target fact.** + - `core/csi/controller_test.go::TestControllerPublish_ReturnsISCSIPublishContextFromTargetFact` + +4. **`ControllerPublishVolume` fails closed when no verified target exists.** + - `core/csi/controller_test.go::TestControllerPublish_FailsClosedWithoutVerifiedTarget` + +5. **`ControllerPublishVolume` fails closed when the target has no frontend fact attached.** + - `core/csi/controller_test.go::TestControllerPublish_FailsClosedWhenTargetHasNoFrontendFact` + +6. **`ControllerPublishVolume` propagates backend lookup errors as `Internal`.** + - `core/csi/controller_test.go::TestControllerPublish_PropagatesLookupErrorsAsInternal` + +7. **Controller capabilities do not advertise dynamic provisioning at G15a.** + - `core/csi/controller_test.go::TestControllerCapabilities_G15aDoesNotAdvertiseDynamicProvisioning` + +8. **`ValidateVolumeCapabilities` confirms an existing target without minting state.** + - `core/csi/controller_test.go::TestValidateVolumeCapabilities_ConfirmsExistingTarget` + +9. **`NodeStage` honors `publish_context` over `volume_context` when both are present.** + - `core/csi/node_test.go::TestNodeStage_UsesPublishContextBeforeVolumeContext` + +### Master backend lookup scenarios (component, fast) + +10. **Master backend maps iSCSI status frontend into `TargetFact`.** + - `core/csi/master_backend_test.go::TestControlStatusLookup_MapsISCSIStatusFrontend` + +11. **Master backend maps NVMe status frontend into `TargetFact`.** + - `core/csi/master_backend_test.go::TestControlStatusLookup_MapsNVMeStatusFrontend` + +12. **Master backend fails closed when no frontend has been assigned.** + - `core/csi/master_backend_test.go::TestControlStatusLookup_FailClosedWithoutAssignedFrontend` + +### L2 subprocess scenario (the headline) + +13. **Real `blockmaster` + `blockvolume` (×2) + `blockcsi` ⇒ `ControllerPublish` returns the volume's actual frontend fact.** + - `cmd/blockcsi/main_test.go::TestG15a_BlockCSIControllerPublishUsesMasterFrontendFact` + - This is the headline integration scenario. The test: + - builds all three binaries in `t.TempDir()`, + - launches `blockmaster`, + - launches `blockvolume` r1 with seeded iSCSI listen address and IQN, + - launches `blockvolume` r2, + - launches `blockcsi --master `, + - polls `ControllerPublishVolume(VolumeId="v1", NodeId="node-a")` for up to 15 s, + - asserts `publish_context["iscsiAddr"]` equals r1's seeded `iscsiAddr` and `publish_context["iqn"]` equals r1's seeded IQN. + - Pass condition: published context matches seeded values within the deadline. + - Skip condition: `-short` flag. + +### Binary smoke scenario + +14. **`blockcsi` binary starts and serves Identity.** + - `cmd/blockcsi/main_test.go::TestBlockCSI_BinaryStartsAndServesIdentity` + +## Non-claims (this instruction does not validate) + +The following are explicitly **out of scope** for this instruction. Do not assume +they pass on the basis of this checklist: + +1. **Real `iscsiadm` discovery / login** — requires Linux + initiator + privileged execution. +2. **Real `mkfs.` against the attached block device** — requires kernel block device + privileged execution. +3. **Real `mount` / bind-mount into a target path** — requires privileged execution + mount namespace. +4. **Filesystem write/read of test files through the mounted CSI volume** — requires the full privileged path above. +5. **`NodeUnstage` / `NodeUnpublish` cleanup against real OS state** — requires the full privileged path above. +6. **Kubernetes CSI driver registration / kubelet integration** — requires a real K8s node. +7. **Multiple concurrent `ControllerPublish` calls / N-node `NodeStage` fan-out** — single happy path only. +8. **Snapshot / clone / resize CSI verbs** — G15a does not advertise these capabilities. +9. **CSI security / RBAC / TLS** — beta-defer until G16. +10. **Performance / SLO** — G21. + +## Forward-carry (next gate / next environment) + +| Item | Owner | Where it lands | +|---|---|---| +| **G15a-5 M01 privileged**: real `iscsiadm` discovery+login, `mkfs`, `mount`, file write/read byte-equal, `NodeUnstage`/`NodeUnpublish` cleanup | sw + QA on M01 | next G15a sub-step; uses the same `cmd/blockcsi` binary, just on a privileged Linux host | +| **CSI dynamic provisioning** (`CreateVolume`/`DeleteVolume`) | sw | post-G15a / G15b mini-plan | +| **CSI snapshot/clone/resize** | sw | G10 / G11 / G15b separate gates | +| **CSI security (TLS / token)** | sw | G16 | +| **Kubernetes integration test** (real kubelet, real PVC, real pod) | QA | G15a-K8s sub-step on M01 or dedicated K8s test cluster | + +## Close sentence + +> *G15a non-privileged L2 proves the V3 product loop produces an assignment, blockvolume exposes a real iSCSI frontend fact, and `blockcsi`'s `ControllerPublishVolume` correctly retrieves attach information from master status — all through real product daemons, with no privileged OS operations.* + +## QA evidence on this instruction (one-time pin) + +``` +Tested by: QA +Date: 2026-05-03 +Tree: p15-g15a/csi-static-mvp@ac49adb +Focused L2: TestG15a_BlockCSIControllerPublishUsesMasterFrontendFact PASS 22.13s +Full 7-pkg regression: all PASS +Worktree: C:/work/seaweed_block_g9c +Platform: Windows 11 Pro (Go default, CGO_ENABLED=1, no -race per project policy) +``` + +Re-run on a different host or commit will produce a fresh evidence pin; this +block is the canonical pin for the close-candidate state at `ac49adb`. diff --git a/sw-block/design/v3-phase-15-g15a-close-report.md b/sw-block/design/v3-phase-15-g15a-close-report.md new file mode 100644 index 000000000..59314ea1b --- /dev/null +++ b/sw-block/design/v3-phase-15-g15a-close-report.md @@ -0,0 +1,157 @@ +# V3 Phase 15 — G15a CSI Static MVP Close Report + +**Date**: 2026-05-03 +**Status**: CLOSED — static/pre-provisioned CSI iSCSI path verified +**Code branch**: `p15-g15a/csi-static-mvp` +**Close anchor**: `seaweed_block@ac49adb` +**Docs/test evidence**: +- `sw-block/design/test/v3-phase-15-g15a-qa-test-instruction.md` +- `sw-block/design/test/v3-phase-15-g15a-privileged-qa-test-instruction.md` +- `V:\share\g15a-priv\runs\20260503T153623Z\` + +--- + +## §1 Close Claim + +G15a proves the static/pre-provisioned V3 CSI iSCSI path: + +``` +cluster-spec / product-loop assignment + -> blockmaster authority assignment + -> blockvolume iSCSI frontend target fact + -> blockcsi ControllerPublish publish_context + -> blockcsi NodeStage real Linux iSCSI login + mkfs + mount + -> blockcsi NodePublish bind mount + -> filesystem write/read byte-equal + -> clean unpublish/unstage/logout +``` + +CSI remains a consumer of V3 product facts. It does not mint authority, create volumes, own storage, or derive target identity when a frontend target fact exists. + +--- + +## §2 Code Delivered + +| Commit | Summary | +|---|---| +| `7b10413` | Added `core/csi` skeleton, CSI Identity/Controller/Node services, boundary guards, and iSCSI NodeStage/Publish mechanism. | +| `4e150ca` | Added frontend target fact propagation: `blockvolume` heartbeat -> master observation/status -> CSI `ControlStatusLookup`. | +| `94ff9cf` | Added `cmd/blockcsi` static driver binary with Identity/Controller/Node registration. | +| `ac49adb` | Added non-privileged L2 smoke proving `ControllerPublish` returns real `blockvolume` frontend facts through `blockcsi`. | + +--- + +## §3 Evidence + +### §3.1 Non-Privileged L2 + +Tree: `p15-g15a/csi-static-mvp@ac49adb` + +Commands: + +```powershell +go test ./cmd/blockcsi -run TestG15a_BlockCSIControllerPublishUsesMasterFrontendFact -count=1 -v +go test ./core/csi ./cmd/blockcsi ./core/host/volume ./core/host/master ./core/authority ./cmd/blockmaster ./cmd/blockvolume -count=1 +``` + +Result: PASS. + +What this proves: +- `cmd/blockcsi` starts and serves CSI Identity. +- `blockmaster`, `blockvolume`, and `blockcsi` run as real subprocesses. +- `blockvolume` reports the iSCSI frontend target fact. +- `blockmaster QueryVolumeStatus` exposes the assigned replica's frontend fact. +- `ControllerPublishVolume` returns `publish_context["iscsiAddr"]` and `["iqn"]` matching the real `blockvolume` target. + +### §3.2 Privileged M01 L3 + +Run ID: `20260503T153623Z` + +Artifact directory: +- Windows: `V:\share\g15a-priv\runs\20260503T153623Z\` +- m01: `/mnt/smb/work/share/g15a-priv/runs/20260503T153623Z/` + +Key files: +- `test-stdout.log` +- `blockmaster.log` +- `blockvolume-primary.log` +- `blockvolume-replica.log` +- `topology.yaml` + +Result: PASS. + +The privileged test verified six phases: + +| Phase | Verified operation | +|---|---| +| 1 | `ControllerPublishVolume` returned expected `iscsiAddr` and `iqn`. | +| 2 | `NodeStageVolume` completed real `iscsiadm` discovery/login, `mkfs.ext4`, and mount; staging path was a mountpoint. | +| 3 | `NodePublishVolume` bind-mounted staging to target; target path was a mountpoint. | +| 4 | 4096 random bytes wrote and read back byte-equal through the CSI-mounted iSCSI device. | +| 5 | `NodeUnpublishVolume` and `NodeUnstageVolume` completed cleanly. | +| 6 | No dangling iSCSI session remained for the test IQN. | + +Additional log evidence from `blockvolume-primary.log`: +- Real Linux kernel initiator reached iSCSI `FullFeature` for target `iqn.2026-05.io.seaweedfs:g15a-priv-v1`. +- `mkfs.ext4` generated real SCSI `WRITE` and `SYNCHRONIZE_CACHE` traffic. +- The final payload write path included `OnLocalWrite`, `ship ok peer=r2`, and replication barrier acknowledgements. + +Known artifact note: `blockcsi.log` was not persisted in this run because the harness helper wrote it to an ephemeral temp directory. This is a harness artifact gap, not a product-path gap; `test-stdout.log` and `blockvolume-primary.log` contain the relevant externally visible CSI and iSCSI/data-plane evidence. + +--- + +## §4 Boundary Guards Preserved + +G15a preserves the V3 control-plane boundaries: + +- CSI package does not import `core/authority` or `core/adapter`. +- CSI package does not construct `AssignmentAsk`, `AssignmentFact`, or `adapter.AssignmentInfo`. +- CSI `ControllerPublish` reads frontend target facts through a narrow `PublishTargetLookup`. +- `blockmaster` status exposes frontend target facts as read-only observation/status data. +- `blockvolume` remains the owner of iSCSI/NVMe target identity and serving. + +This matches the G15a unification rule: CSI is attach/mount orchestration, not storage ownership or authority. + +--- + +## §5 Non-Claims + +G15a does not claim: + +- Kubernetes PV/PVC/Pod integration. +- Dynamic CSI `CreateVolume`. +- CSI snapshot, clone, or expansion. +- NVMe CSI attach path. +- Multi-node RWO enforcement. +- CHAP, mutual auth, TLS, or routable frontend security. +- Pod remount after primary kill. +- Failover while a filesystem is mounted. +- Plugin restart cleanup after crash. +- Performance, throughput, or soak behavior. + +--- + +## §6 Forward-Carry + +| Item | Target | +|---|---| +| Persist `blockcsi.log` into privileged artifact dirs. | Harness polish; not a G15a blocker. | +| Kubernetes static PV/PVC + pod write/read using `cmd/blockcsi`. | G15b. | +| Plugin restart + stale-volume `NodeUnstage` cleanup. | G15b/G15c follow-up. | +| Multi-node attach and RWO enforcement. | G15c. | +| Failover under live mounted filesystem. | G15c/G16, after G8 failover semantics are exercised through CSI. | +| NVMe NodeStage/NodePublish path. | G15d. | +| Dynamic `CreateVolume` from CSI into desired-volume / placement allocation. | G15e or later, after product API ratification. | + +--- + +## §7 Close Decision + +G15a is closed on `seaweed_block@ac49adb`. + +The static CSI iSCSI path is proven at both: + +- non-privileged L2 control-plane fidelity, and +- privileged m01 Linux data-plane fidelity. + +Next gate should be G15b: Kubernetes static PV/PVC/pod integration using the same `cmd/blockcsi` binary and the same static/pre-provisioned volume model.