diff --git a/sw-block/design/bugs/005_backend_close_cross_session.md b/sw-block/design/bugs/005_backend_close_cross_session.md new file mode 100644 index 000000000..9f0948888 --- /dev/null +++ b/sw-block/design/bugs/005_backend_close_cross_session.md @@ -0,0 +1,134 @@ +# BUG-005 — Backend.Close in session lifecycle breaks cached DurableProvider Backend across sessions + +**Status**: FIX-LANDED — seaweed_block commit ``; unit regression green; awaiting m01 re-verification +**Severity**: BLOCKER (G4 partial-pass gate — without this fix, cross-session reconnect fails 100%) +**Component**: `core/frontend/nvme/target.go` + `core/frontend/iscsi/session.go` (session-layer Backend close) +**Discovered-by**: QA Owner during Path B m01 Matrix A cycle-2 reconnect test +**Discovered-when**: 2026-04-22, during T3c three-sign prep +**Owner**: sw +**Blocks**: T3-end three-sign (G4 durable-data-path pass) + +--- + +## 1. One-line symptom + +After a full NVMe session (connect → mkfs → IO → disconnect) completes cleanly, the NEXT `nvme connect` on the same target returns `I/O Error (sct 0x3 / sc 0x2) DNR` on the very first Read. The kernel treats it as ANA "Asymmetric Access Inaccessible" and drops the namespace. + +Not a data-integrity bug — the bytes on disk are intact. A lifecycle bookkeeping bug: the cached Backend handle gets marked closed by session 1's teardown, so session 2 receives the same (closed) handle from the Provider cache. + +--- + +## 2. Environment + +- Host: m01 (Linux 6.17, nvme-cli) +- Target: V3 `blockvolume` with `--durable-root` + `--durable-impl smartwal` +- Phase: Path B m01 reconnect matrix during T3c three-sign prep + +--- + +## 3. Symptom — verbatim + +### Kernel dmesg +``` +nvme nvme1: creating 8 I/O queues. (cycle 1) +... mkfs / mount / write / sync / umount / disconnect succeed ... +nvme nvme1: creating 8 I/O queues. (cycle 2) +nvme1n1: I/O Cmd(0x2) @ LBA 0, 8 blocks, I/O Error (sct 0x3 / sc 0x2) DNR +``` + +### V3 status mapping +`core/frontend/nvme/errors.go`: +```go +SCTPathRelated = 0x3 +SCPathAsymAccessInaccessible = 0x02 +``` +Result of mapping `frontend.ErrBackendClosed` through `NewBackendClosed()`. + +--- + +## 4. Root cause + +Two design elements combined into a bug: + +**Element A — per-volumeID Backend cache in `DurableProvider`**. Required because `LogicalStorage` is an exclusive resource: cannot have two open WAL writers on the same file. So `Open` returns the same `StorageBackend` for repeated Opens of the same volumeID. + +**Element B — session-layer calls to `Backend.Close()`**: +- `core/frontend/nvme/target.go` `handleConn`: `defer backend.Close()` +- `core/frontend/iscsi/session.go` `close()`: `_ = s.backend.Close()` + +These were memback-era patterns that worked because memback's `Provider.Open` creates a NEW `backend` struct per call (fresh `closed=false`); only the underlying `volumeStore` is cached. Session-level Close marked just that session's handle closed. + +When T3a `DurableProvider` introduced a per-volumeID cache, the two elements became incompatible: session 1's Close flipped the CACHED handle closed. Session 2's Open returned the same (closed) handle. Every I/O hit `StorageBackend.gate()`'s `closed=true` branch and returned `ErrBackendClosed`, which the NVMe layer faithfully reported as `SCT=3 SC=2`. + +### Why V2 didn't have this + +V2's architecture structurally avoided the conflict: +- `Subsystem` holds the `BlockDevice`; registered via `AddVolume` / unregistered via `RemoveVolume` (admin-level, not per-session) +- `Controller` (per-session) holds a `*Subsystem` POINTER — never closes the underlying `Dev` +- `Controller.shutdown()` closes conn, stops KATO, clears pendingCapsules, unregisters CNTLID from admin map — explicit list, no `Dev.Close()` + +V3 introduced the `frontend.Backend` abstraction without carrying V2's implicit ownership contract ("device outlives sessions") into the new interface's godoc. The result: T2 target authors saw `Backend.Close()` and treated it as session-owned, matching memback's accidental compatibility. DurableProvider's arrival broke that. + +--- + +## 5. Fix + +**Minimal 2-line delete** + godoc formalization + regression test: + +1. Remove `defer backend.Close()` in `core/frontend/nvme/target.go` `handleConn` (replaced with a BUG-005 comment). +2. Remove `_ = s.backend.Close()` in `core/frontend/iscsi/session.go` `close()` (replaced with a BUG-005 comment). +3. Add Provider-ownership clause to `frontend.Backend` interface godoc in `core/frontend/types.go`: + > "Backend.Close() is OWNED BY THE PROVIDER. Session-layer consumers MUST NOT call Close on a Backend obtained from Provider.Open — treat the returned Backend as a BORROWED reference..." +4. Regression test `TestT3_Bug005_ProviderCachedBackend_ReusableAcrossSessions` matrix-parameterized over walstore + smartwal. + +No changes to `StorageBackend` itself; the invariant is enforced at the call-site level. `Backend.Close()` remains a legal operation for callers who OWN the handle (Provider.Close, explicit tests). + +--- + +## 6. Why existing tests missed it + +All T3a/T3b/T3c tests either: +- Single-session (`scenario_crash_recovery_test.go` etc.) — no session-close → no cache poisoning +- Direct-handler (`integration_iscsi_test.go` etc.) — call `SCSIHandler.HandleCommand` / `IOHandler.Handle` directly, bypassing `target.handleConn`'s defer + +**Only "real target + session 1 + disconnect + session 2" exercises the cross-session cache-poisoning path.** m01 is the first environment that drove this pattern end-to-end. + +Same structural lesson as BUG-001: L1 coverage of individual layers (adapter, provider, handler) doesn't catch bugs that emerge only when the target's session-lifecycle × Provider's handle-cache interaction runs in a real multi-session workload. + +--- + +## 7. Fix verification + +- Unit regression: `TestT3_Bug005_ProviderCachedBackend_ReusableAcrossSessions` passes for both impls. +- Full tree: `go test ./core/... -count=1` — all packages green. +- m01 re-run post-fix: pending QA cycle with Path B (Matrix A / B / C / D already green pre-BUG-005; re-verify they stay green, then Matrix E attempted). + +--- + +## 8. Exit criteria + +- [x] Session-layer `Backend.Close` removed (nvme/target.go + iscsi/session.go) +- [x] Backend godoc documents Provider ownership +- [x] Regression test matrix pins the invariant +- [ ] m01 reconnect cycle green (QA re-run) +- [ ] Ledger row `INV-DURABLE-SESSION-LIFECYCLE-001` ACTIVE + +## 9. New ledger row + +`INV-DURABLE-SESSION-LIFECYCLE-001`: Across NVMe / iSCSI session disconnect + reconnect, the Provider-cached Backend remains operational. I/O on the reconnected session MUST NOT receive `ErrBackendClosed` caused by prior session teardown. + +## 10. Cross-links + +- V2 reference (correct pattern): `weed/storage/blockvol/nvme/controller.go` `shutdown()` + `weed/storage/blockvol/nvme/server.go` `AddVolume / RemoveVolume` +- BUG-001 (precedent): same L1-passes-L2-fails pattern; storage-level drift vs session-level drift +- T3a design note §3 (pre-existing contract): "Backend.Close marks closed but does NOT tear down storage; Provider retains storage handle" +- Memory: `feedback_porting_discipline.md` (update: new-abstraction-layer ownership discipline) + +## 11. Log + +| Date | Actor | Event | +|---|---|---| +| 2026-04-22 | QA | Path B cycle-2 reconnect fails; dmesg `sct 0x3/sc 0x2` captured | +| 2026-04-22 | sw | Root cause identified: session-layer `Backend.Close` × Provider cache interaction | +| 2026-04-22 | sw | Fix Option 1 landed (2-line delete + godoc + regression test) | +| 2026-04-22 | sw | Regression test green both impls; awaiting m01 re-verify | diff --git a/sw-block/design/bugs/006_nvme_kato_timer_not_enforced.md b/sw-block/design/bugs/006_nvme_kato_timer_not_enforced.md new file mode 100644 index 000000000..1ee8fa327 --- /dev/null +++ b/sw-block/design/bugs/006_nvme_kato_timer_not_enforced.md @@ -0,0 +1,112 @@ +# BUG-006 — NVMe KATO timer stored but not enforced + +**Status**: OPEN — filed 2026-04-22 during T3-end retrospective (T3-DEF-7 latent gap) +**Severity**: Medium — no correctness impact in short sessions; causes zombie sessions + KATO-based liveness contract violation under long-running / network-partition scenarios +**Component**: `core/frontend/nvme/session.go` (Keep-Alive Timeout handling) +**Discovered-by**: QA retrospective of T3-DEF-5/6/7 review round +**Discovered-when**: 2026-04-22 +**Owner**: sw +**Target track**: G21 (perf / hardening gate) OR post-T3 cycle — NOT T3 +**Blocks**: not T3; tracks for next NVMe hardening window + +--- + +## 1. One-line symptom + +Host-advertised KATO (Keep-Alive Timeout, seconds) is parsed + stored on admin Connect, but the session has no watchdog goroutine that tears down the connection when the host stops sending Keep-Alive admin commands within the KATO window. A dead host thus leaves a zombie session holding CNTLID + AER slot + Backend identity reference indefinitely (until TCP reset). + +Visible hint: m01 Matrix D kernel dmesg warning `nvme nvme0: long keepalive RTT (2522123190 ms)` — the host-side driver observed impossibly large KeepAlive RTT because the target never timed it out, letting kernel accumulate stale state before its own recovery logic kicked in. + +## 2. Contract reference + +Per `v2-v3-contract-bridge-catalogue.md` §2.2.14 (NVMe Session state): + +> **C1-NVME-SESSION-KATO-STORED-NOT-ENFORCED** — "Session MUST arm a watchdog for `KATO` ms after admin Connect; if no Keep-Alive admin command arrives before the deadline, session tears itself down (close sockets, release CNTLID/AER/BackendRef)." + +Current code at `session.go` parses CDW12 into `s.katoMS` but never arms a timer. Contract status today: **VIOLATED (silently)**. + +NVMe-oF spec §3.3.2 requires: `If no Keep Alive command is processed within the KATO interval, the controller shall treat the association as failed and terminate all outstanding commands and release controller resources.` + +## 3. Why this wasn't caught + +| Test layer | Coverage | +|---|---| +| L0 unit | session close-path goroutine-leak test exists; doesn't simulate host going silent | +| L1 QA addendum (T3-DEF-6 `CleanupOnClose_NoLeaksAcrossCycles`) | Clean close path; host always sends close | +| T3c scenarios | Short-duration; KATO default (120s) never hits | +| m01 Matrix A-F | Fast operations; dead-host scenario not exercised | + +No existing test drives "host silent for > KATO" against a live target. + +## 4. Impact + +- **Functional (no correctness impact for happy path)**: sessions that close cleanly are fine. +- **Resource leak under partition**: network partition leaves CNTLID + AER slot + pendingCapsules entries uncleared until TCP keepalive / RST arrives (minutes–hours at OS default). +- **Liveness contract violation**: multi-path clients relying on ANA updates expect target to notice dead path within KATO; currently target is silent, client-driven timeout kicks in instead. +- **Diagnostic pollution**: "long keepalive RTT" dmesg warnings confuse operators (observed on m01 already). + +## 5. Fix sketch + +~50–100 LOC + one goroutine per session: + +```go +// session.go — after admin Connect parses katoMS: +if s.katoMS > 0 { + s.kaDeadline.Store(time.Now().Add(time.Duration(s.katoMS) * time.Millisecond).UnixNano()) + go s.kaWatchdog() +} + +// On every admin/io command completion: +s.kaDeadline.Store(time.Now().Add(time.Duration(s.katoMS) * time.Millisecond).UnixNano()) + +// kaWatchdog: +func (s *Session) kaWatchdog() { + for { + dl := time.Unix(0, s.kaDeadline.Load()) + now := time.Now() + if now.After(dl) { + s.log("KATO expired; terminating association") + _ = s.Close() + return + } + select { + case <-s.closeCh: + return + case <-time.After(dl.Sub(now)): + } + } +} +``` + +Edge cases to cover in test: +- `katoMS == 0` → watchdog MUST NOT arm (spec: 0 = disabled). +- Normal Keep-Alive flow resets deadline; no false-positive termination. +- Close during watchdog tick → no race / double-Close. +- `kaDeadline` updates must use atomic (session is multi-goroutine). + +## 6. Test plan (to land with fix) + +- L0 unit: `TestSession_KATO_FiresOnSilence` — set KATO=500ms, stop sending commands, expect session torn down within 500-750ms. +- L0 unit: `TestSession_KATO_ResetOnKeepAlive` — send Keep-Alive every 300ms for 3s with KATO=500ms, expect session alive. +- L0 unit: `TestSession_KATO_Zero_NoWatchdog` — KATO=0, sleep 2s with no traffic, expect session alive. +- L1 QA: extend `t3_qa_session_cleanup_addendum_test.go` with KATO-based teardown cycle + leak check. + +## 7. Catalogue linkage + +Existing catalogue row `C1-NVME-SESSION-KATO-STORED-NOT-ENFORCED` in `v2-v3-contract-bridge-catalogue.md` §2.2.14 is the anchor (already reclassified to **VIOLATED** with BUG-006 citation 2026-04-22). Upon fix, flip the verdict to SATISFIED (not add a new row) — timer enforcement is the "partial" half of the existing PRESERVE-partial tag being completed. + +Port-verdict semantics: the row is V3 REBUILD vs V2's full-timer path (V2 iSCSI had M/E phase + TCP-layer keepalive; V3 NVMe needs native KATO watchdog at session layer). + +## 8. Exit criteria + +- Watchdog goroutine lands; KATO armed on admin Connect. +- 3 L0 tests + 1 L1 QA extension green. +- Catalogue row C1-NVME-SESSION-KATO-STORED-NOT-ENFORCED flips verdict from VIOLATED to SATISFIED. +- m01 repro: deliberately hang host after connect, observe target session torn down within KATO window (no more "long keepalive RTT" dmesg pattern). +- BUG moves to CLOSED. + +## 9. Log + +| Date | Actor | Event | +|---|---|---| +| 2026-04-22 | QA | Filed as T3-DEF-7 during T3-end retrospective; m01 Matrix D dmesg warning `long keepalive RTT (2522123190 ms)` cited as visible symptom; target track G21 or post-T3 window | diff --git a/sw-block/design/bugs/007_walstore_umount_remount_data_loss.md b/sw-block/design/bugs/007_walstore_umount_remount_data_loss.md new file mode 100644 index 000000000..ea6e7400d --- /dev/null +++ b/sw-block/design/bugs/007_walstore_umount_remount_data_loss.md @@ -0,0 +1,112 @@ +# BUG-007 — walstore loses written data across umount + remount (same session) + +**Status**: OPEN — filed 2026-04-22 during T3-end m01 re-verify +**Severity**: Medium — walstore is non-default impl (smartwal is production default per T3b); but "fs cross-umount persistence" is a baseline durability expectation +**Component**: `core/storage/walstore.go` (walstore Read path OR Sync persistence) +**Discovered-by**: QA Owner during Matrix D (cross-session consistency) re-verify against 27486db +**Discovered-when**: 2026-04-22 (T3-end pre-sign) +**Owner**: sw (after T3 closes) +**Blocks**: not T3 close (smartwal full A-F green satisfies canonical G4; walstore explicitly non-default per T3b config); tracks for post-T3 walstore hardening + +--- + +## 1. One-line symptom + +Against a walstore-backed NVMe device, `mkfs.ext4 -F; mount; echo X > f; sync; umount; mount; cat f` returns "file not found" instead of "X". The clean umount + remount in same blockvolume process loses the sync'd write. + +## 2. Environment + +| Item | Value | +|---|---| +| Host | m01 (192.168.1.181) | +| Kernel | Linux 6.17.0-19-generic | +| V3 commit | phase-15 @ `27486db` | +| DurableProvider `--durable-impl` | `walstore` (fails) vs `smartwal` (passes same test) | + +## 3. Symptom — verbatim + +`scripts/iterate-m01-nvme.sh` Matrix D (5 remount cycles on shared ext4): + +``` +Matrix D — cross-session consistency (5 remount cycles) +cross-session cycle 1: got 'MISSING' want 'cycle-1-marker-1776918177' +[iterate-m01 FAIL] Matrix D cross-session consistency failed +``` + +Same test `DURABLE_IMPL=smartwal` → all 5 cycles PASS. + +## 4. Reproduction + +```bash +# Against a running blockvolume with --durable-impl walstore: +sudo nvme connect -t tcp -a 127.0.0.1 -s 4421 -n +DEV=$(sudo nvme list | awk '/SeaweedFS/ {print $1; exit}') +sudo mkfs.ext4 -F -b 1024 -I 128 -N 2000 $DEV +sudo mkdir -p /mnt/t +sudo mount $DEV /mnt/t +echo "probe-$(date +%s)" | sudo tee /mnt/t/probe.txt +sudo sync +sudo umount /mnt/t + +# Now remount — probe.txt should exist: +sudo mount $DEV /mnt/t +ls /mnt/t/probe.txt +# → ls: cannot access '/mnt/t/probe.txt': No such file or directory +``` + +Deterministic (first attempt fails). + +## 5. Hypothesis + +`walstore.Read(lba)` may not be pulling uncommitted-but-WAL-persisted writes back when the flusher hasn't yet applied them to the extent. Sequence: + +- ext4 writes superblock / directory entries / file to various LBAs +- Each write → walstore.Write → appends to WAL +- kernel sync → walstore.Sync → fsync WAL (checkpoint NOT advanced if flusher async) +- kernel umount → kernel Flush → walstore.Sync (same) +- kernel mount (same device) → reads superblock from LBA 0 +- walstore.Read(LBA=0) → if checks extent only, returns pre-write superblock (not the ext4-fresh one) → mount sees stale/invalid fs → files missing + +`walstore.Read` at line 382 DOES split into `readFromWAL` (line 405) + `readFromExtent` (line 427) via dirtyMap lookup (line 390+). So the mechanism SHOULD exist. Something about the umount→remount sequence probably invalidates dirtyMap state, OR checkpoint advance happens unsafely, OR something else. + +Needs `core/storage/walstore.go` inspection; not blocking T3 close. + +## 6. Why smartwal doesn't have this + +smartwal uses a ring-based WAL with per-slot validation; its read path may be more robust against the specific "recent Write + Sync but no flusher yet" window. Investigation required. + +## 7. Why this wasn't caught earlier + +- `core/storage/walstore_test.go` covers Write → Sync → **Close+Reopen** → Read. It does NOT cover Write → Sync → **Read** without reopen (the same-session cross-umount case). +- T3a adapter tests (`TestT3a_StorageBackend_ByteLBATranslation_Matrix`) write and read but don't do the umount/remount sequence. +- T3c scenarios do crash + reopen (which is fine) but not umount + remount. +- My QA L1 addendum `TestT3_Durable_RecoverThenServe` does reopen — different code path from same-session umount/remount. + +Matrix D is the first test that exercises "fs layer flush + remount WITHOUT process restart" path against walstore. + +## 8. Impact on T3 + +- T3 closes normally: smartwal is production default per T3b; walstore is fallback. smartwal Matrix A-F all green. +- walstore is explicitly a **non-primary impl**. Post-T3 walstore hardening addresses this. +- Closure report §C: walstore port quality "MATCHES" rating stands (functionality exists); §D update: "walstore post-T3 hardening tracked in BUG-007". +- Ledger row `INV-DURABLE-001` (crash recovery byte-exact) queues ACTIVE based on smartwal evidence; walstore-side evidence deferred to BUG-007 close. + +## 9. Catalogue linkage + +`v2-v3-contract-bridge-catalogue.md` §2.2.5 `Superblock` + §2.2.4 `GroupCommitter` both relate. New catalogue annotation to add: + +> BUG-007 surfaces a **walstore-specific** contract gap: walstore's `Read` must return WAL-resident writes even before flusher advance. The contract is **same-session durability under umount-triggered sync**. PRESERVE verdict for walstore; implementation gap to close via BUG-007 fix. + +## 10. Exit criteria + +- Root cause identified in `core/storage/walstore.go` +- Fix lands; existing walstore tests still green +- `TestT3_Walstore_UmountRemount_Persistence` L0 regression added; both walstore + smartwal run it +- m01 `DURABLE_IMPL=walstore` Matrix A-F all green +- BUG moves to CLOSED + +## 11. Log + +| Date | Actor | Event | +|---|---|---| +| 2026-04-22 | QA | Filed after Matrix D walstore failure in T3-end re-verify; BUG-005 already fixed so this is NOT a regression — pre-existing walstore bug surfaced by Matrix D addition | diff --git a/sw-block/design/bugs/inventory/nvme-test-coverage-deferred.md b/sw-block/design/bugs/inventory/nvme-test-coverage-deferred.md new file mode 100644 index 000000000..35640d891 --- /dev/null +++ b/sw-block/design/bugs/inventory/nvme-test-coverage-deferred.md @@ -0,0 +1,104 @@ +# NVMe Test Coverage — Deferred Inventory (B-tier) + +**Date**: 2026-04-22 +**Status**: LIVING — items move OUT when landed, marked CLOSED when fully in ACTIVE tracks +**Purpose**: enumerate NVMe test coverage deliberately deferred past T2 closed, so nothing rots silently. Each item has (a) V2 correspondent where applicable, (b) assigned downstream track, (c) activation trigger. + +**Governance**: adding to this file does NOT require Discovery Bridge — it's a downgrade (from "should do" to "will do later"). Removing an item (i.e., landing it) is governed by the track it's assigned to. + +--- + +## L1-B — Go subprocess component tests deferred + +| ID | Test | V2 correspondent | Track | Activation trigger | +|---|---|---|---|---| +| ~~L1B-1~~ | ~~`TestT2Process_NVMe_ReconnectLoop` — attach/write/disconnect × 50 cycles on one workstation~~ | ~~TestComponent_FastReconnect~~ | ~~T3 perf~~ | **LANDED 2026-04-22 as `TestT2V2Port_NVMe_Process_ReconnectLoop50` (QA parallel capacity, 14.9s PASS, goroutine-leak guard included). Ledger row `PCDD-NVME-SESSION-RECONNECT-LOOP-001` queued ACTIVE.** | +| L1B-2 | `TestT2Process_NVMe_FailoverMidWrite` — kill primary mid-in-flight-write, verify replica takeover | TestComponent_FailoverPromote | G8/T6 Failover Data Continuity | master-side "kill + promote" RPC exposed | +| L1B-3 | `TestT2Process_NVMe_CrashRecovery` — write/sync/crash/restart/read-consistent | TestCP13_SyncAll_ReplicaRestart_Rejoin | T3 durability or T5 | durable backend (beyond memback) available | +| ~~L1B-4~~ | ~~`TestT2Process_NVMe_DisconnectMidR2T` — kernel-realistic cancel path~~ | ~~n/a (new)~~ | ~~T3~~ | **LANDED 2026-04-22 as `TestT2V2Port_NVMe_IO_DisconnectMidH2CDataStream` (unit-scope, 0.6s PASS). Pins recvH2CData mid-loop EOF unwind. Ledger row `PCDD-NVME-SESSION-DISCONNECT-MID-R2T-001` queued ACTIVE.** | + +### T3 deferred scenarios (post-sw-commit additions; G4 canonical not blocked) + +| ID | Scenario | V2 parity | Defer target | +|---|---|---|---| +| T3-DEF-1 | `t3c-durable-crash-during-sync` — kill mid-Sync (before Sync returns); pre-Sync acked all present, post-Sync-initiation may vanish | V2 CP13 crash variants | Post-G4 hardening (G21 or follow-up batch) | +| T3-DEF-2 | `t3c-durable-restart-loop` — 10-cycle attach/write-distinct-pattern/crash/restart/verify with pattern integrity | V2 t0-hosting-smoke 20-cycle | Post-G4 hardening (same) | + +### Latent-gap findings (from Tier A V2/V3 Contract Bridge Catalogue retrofill 2026-04-22) + +| ID | Statement | Surfaced by | Defer target | +|---|---|---|---| +| ~~T3-DEF-5~~ | ~~iSCSI VPD Model / Vendor / Firmware-Rev — V2-hardcoded constants; PORT-AS-IS pending multi-tenant trigger~~ | `v2-v3-contract-bridge-catalogue.md` §2.2.13 Tier A retrofill | **CATALOGUED 2026-04-22 as row C4-ISCSI-VPD-STATIC-FIELDS-HARDCODED (PORT-AS-IS LOCKED explicit). Closes T3-DEF-5 as discipline row; no regression needed** | +| ~~T3-DEF-6~~ | ~~NVMe Session cleanup-on-close contract only implicit (goroutine leak) → add explicit L1 addendum~~ | same | **LANDED 2026-04-22 as `t3_qa_session_cleanup_addendum_test.go` (`TestT3_NVMe_Session_CleanupOnClose_NoLeaksAcrossCycles` + `_CNTLIDAllocator`, both PASS 0.78s). Pins C5-NVME-SESSION-STATE-CLEANUP-ON-CLOSE.** | +| ~~T3-DEF-7~~ | ~~NVMe KATO timer currently store-only; m01 long-keepalive-RTT warning exposed today~~ | same | **FILED 2026-04-22 as `bugs/006_nvme_kato_timer_not_enforced.md` (Medium, target G21 / post-T3). C1-NVME-SESSION-KATO reclassified VIOLATED with BUG-006 anchor. Closes T3-DEF-7 as inventory row → BUG queue** | + +Rationale: proposed post-sw-commit; shipped 4-scenario set covers canonical G4 pass-gate (`INV-DURABLE-001`) + 3 adjacent invariants. Not regressions; V2-parity hardening adjuncts. +| L1B-5 | `TestT2Process_NVMe_MultiVolumeConcurrent` — 3 volumes × 4 queues each | TestComponent_MultiReplica (adapted) | T3 perf | n/a | + +--- + +## L2-B — Scenario YAML + Go replay deferred + +| ID | Scenario | V2 correspondent | Track | Activation trigger | +|---|---|---|---|---| +| L2B-1 | `t2-nvme-crash-recovery.yaml` | crash (1 of 11) | T3 durability | L1B-3 lands first | +| L2B-2 | `t2-nvme-ha-failover.yaml` | HA (4 of 11, collapsed) | G8/T6 | L1B-2 lands first | +| L2B-3 | `t2-nvme-fault-disk-fill.yaml` | fault (1 of 3) | T3 | backend exposes "fail next N writes" hook | +| L2B-4 | `t2-nvme-fault-network-drop.yaml` | fault (1 of 3) | T3 | fault-injection infrastructure | +| L2B-5 | `t2-nvme-consistency-fsync.yaml` | consistency (1 of 2) | T3 or G22 final gate | L1B-3 or durable backend | + +--- + +## L3-B — m01 real-kernel runs deferred + +| ID | Run | Track | Activation trigger | +|---|---|---|---| +| L3B-1 | `iterate-m01-nvme-reconnect.sh` — attach/detach loop 1 hr, watch for connection leak | T3 perf | after L3-A proven green in CI | +| L3B-2 | `iterate-m01-nvme-fio.sh` — mixed read/write IOPS/throughput/latency measurement via fio | T3 perf | perf baseline established for iSCSI side first | +| L3B-3 | `iterate-m01-nvme-24h-soak.sh` — 24 hr continuous mkfs/mount/I/O/umount loop | post-T2 closed (monitoring) | T2B-NVMe-product-ready signed | + +--- + +## L4-B — Long-duration / endurance + +| ID | Run | Track | Activation trigger | +|---|---|---|---| +| L4B-1 | 7-day endurance soak (bonnie++ style) | G22 Final Gate | all prior tracks green | +| L4B-2 | Chaos mesh — network partition + process crash + disk fill random mix | G22 Final Gate | fault-injection infra available | + +--- + +## Activation checklist (when item moves from B to ACTIVE) + +Filing a new entry from B to implementation requires: + +1. Assigned owner (not "TBD") +2. Concrete activation trigger satisfied (not "when convenient") +3. Acceptance criteria copied from the B-row or expanded inline +4. Ledger row provisional → queued for ACTIVE +5. Updated here: row marked `→ [link to impl commit or test file]`, strike-through the text + +Items that never satisfy their trigger during the V3 P15 lifecycle carry forward to the next phase's inventory (P16 or beyond). + +--- + +## Rotation audit (quarterly) + +Once per quarter (or per phase gate), reviewer walks this file and asks for each row: + +- Is the track still alive? +- Is the activation trigger still relevant? +- Has V3 architecture drifted such that the test is no longer meaningful? + +Drop rows that fail any of the three; add rationale line in §Change log. + +--- + +## Change log + +| Date | Change | Author | +|---|---|---| +| 2026-04-22 | Initial file with 5 L1-B + 5 L2-B + 3 L3-B + 2 L4-B | QA Owner (Batch 11c sign prep) | +| 2026-04-22 | L1B-1 ReconnectLoop landed early: file `t2_v2port_nvme_reconnect_loop_test.go`, 50 cycles PASS 14.9s, no goroutine leak | QA Owner | +| 2026-04-22 | L1B-4 DisconnectMidH2CDataStream landed early: file `t2_v2port_nvme_disconnect_mid_r2t_test.go`, unit-scope 0.6s PASS; server log confirms recvH2CData EOF unwind path exercised | QA Owner | +| 2026-04-22 | T3-DEF-5/6/7 processed during T3-end retrospective: DEF-5 catalogued (C4-ISCSI-VPD-STATIC-FIELDS-HARDCODED), DEF-6 test landed (`t3_qa_session_cleanup_addendum_test.go`), DEF-7 filed as BUG-006 (`006_nvme_kato_timer_not_enforced.md`). All three struck through above | QA Owner | diff --git a/sw-block/design/v2-v3-contract-bridge-catalogue.md b/sw-block/design/v2-v3-contract-bridge-catalogue.md new file mode 100644 index 000000000..04f761887 --- /dev/null +++ b/sw-block/design/v2-v3-contract-bridge-catalogue.md @@ -0,0 +1,491 @@ +# V2/V3 Contract Bridge Catalogue + +**Date**: 2026-04-22 +**Status**: LIVING — T3 retrofilled 2026-04-22; T4/T5/T6/T7 sections mandatory-LOCKED before respective T-start three-sign +**Owner**: QA drafts + co-signs per track; sw reviews; architect + PM sign as part of T-start sketch + +--- + +## §0 Purpose + discipline + +Port is NOT a file-by-file copy exercise. It is a **cross-version stateful-entity bridging** exercise. Each V2 entity carries contracts (explicit + implicit) governing scope, lifecycle, concurrency, and cross-session behavior. V3 rebuilds the topology around new models (event / storage / authority / concurrency / lifecycle); every bridge must be audited. + +**Top-down cutting discipline**: + +| Level | Output | Order | +|---|---|---| +| L1 — Entity enumeration | For each V2 stateful entity: scope / lifecycle owner / concurrency / cross-session behavior | FIRST | +| L2 — Bridge + contracts | Map V2 entity → V3 entity(ies); V3 embedding notes; per-contract PRESERVE/REBUILD/BREAK verdict | SECOND | +| L3 — Files / functions | Which V2 files map to which V3 files; function-level classification | THIRD (derived from L1+L2) | + +Prior T-end sketches went straight to L3 (file-by-file audit). Every known drift (BUG-001, BUG-005, Addendum A) traces back to an L1/L2 decision that was never made explicit. Enforcing top-down order means the drift surface gets caught at T-start sign, not at integration bug time. + +**Bridge shape** (shorthand tag, attached to each entity, NOT primary verdict): + +| Shape | V2 : V3 | Meaning | +|---|---|---| +| `1:1` | 1 → 1 | One V2 entity → one V3 entity. **Does NOT imply verbatim port** — the V3 event/storage/authority model may embed the entity differently. Re-audit mandatory. | +| `split` | 1 → N | One V2 entity → multiple V3 entities. Contracts distribute across the split; each contract must name which V3 entity owns it. | +| `merge` | N → 1 | Multiple V2 entities → one V3 entity. Contracts from multiple origins coexist; explicit reconciliation rules required. | +| `retired` | 1 → 0 | V2 entity has no V3 equivalent. BREAK all contracts with rationale. | +| `new` | 0 → 1 | V3 entity has no V2 predecessor. Contracts must be written fresh (not ported); subject to §8C.5 semantic-contract audit. | + +**1:1 does NOT mean "port verbatim"**. Example: V2 `GroupCommitter` and V3 `core/storage.GroupCommitter` are 1:1 by name, but V3 drops the `OnDegraded` callback (V3 event model uses return-value error propagation, not V2's callback fan-out). Even 1:1 requires V3-embedding review per §1.1. + +--- + +## §1 Frameworks + +### §1.1 V3 model shifts (baseline — why even 1:1 entities may need re-audit) + +V3 is not a line-by-line copy of V2. It reshapes five cross-cutting models. Every entity bridge must be evaluated against these shifts: + +| Model | V2 | V3 | Impact on port | +|---|---|---|---| +| **Event model** | Callbacks (`NotifyFn`, `ClosedFn`, `OnDegraded`, `PostSyncCheck`) fan out from inside storage/engine | Typed errors + interface return values + `ctx.Done()`; observer responsibility moves to adapter/host layer | 1:1 entity ports may need callback → return-value rewiring | +| **Storage model** | `BlockDevice` flat Read/Write at byte offset; BlockVol engine owns WAL + dirty + fsync | `LogicalStorage` interface with LSN + Sync + Recover + Boundaries; pluggable impls (walstore / smartwal); adapter does byte↔LBA | Even if V2 entity name survives, the V3 interface wraps with LSN + Sync + Recover semantics | +| **Authority model** | Fence / epoch / role checked inside storage layer (`write_gate`, `superblock.Epoch`, `PostSyncCheck`) | Authority strictly master-side; per-I/O fence at adapter boundary via `ProjectionView`; storage never mints/advances/publishes | Any V2 entity holding authority state → REBUILD (move state out) or BREAK (retire) | +| **Lifecycle model** | Admin adds `AddVolume(nqn, dev, nguid)`; lifetime tied to RemoveVolume | `DurableProvider` lazy-opens + caches per-volumeID; explicit Provider.Close tears down; `SetOperational` for readiness | V2 admin-driven lifecycle → V3 lazy/cached Provider lifecycle; Session MUST NOT close Backend (BUG-005 lesson) | +| **Concurrency model** | `Controller.rxLoop` single-threaded serial; `GroupCommitter` has a goroutine; most paths serial | rxLoop still serial post-BUG-001 revert; smartwal has internal goroutine; per-impl variability | Controller split: rxLoop stays serial (Session); I/O submission may fan out to goroutine owned by impl | + +**When doing §2+ bridge audit, each contract must be checked against every relevant model shift** — "PRESERVE" does not mean "copy code"; it means "equivalent semantic under the new model". + +### §1.2 Entity attribute columns (L1 output) + +For each V2 entity being ported, fill: + +| Attribute | Example | +|---|---| +| **Name** | `NVMe Subsystem` | +| **V2 file(s)** | `weed/storage/blockvol/nvme/server.go` | +| **Scope** | request / session / volume / process / persistent | +| **Lifecycle owner** | which V2 function creates/destroys it | +| **Concurrency discipline** | locks, goroutines, serial vs parallel | +| **Cross-session behavior** | what happens on session end / process restart | +| **Bridge shape** | `1:1 / split / merge / retired / new` | +| **V3 entity(ies)** | where in V3 this maps | +| **V3 embedding note** | how V3's model (event / storage / authority / lifecycle / concurrency) wraps this entity — what shifts apply | + +### §1.3 Contract catalogue columns (L2 output) + +For each contract carried by an entity: + +| Column | Purpose | +|---|---| +| **Contract ID** | `C1-NVME-SUBSYS-DEV-LIFETIME` | +| **Statement** | 1-sentence rule | +| **V2 evidence** | file:line or function reference | +| **V3 verdict** | PRESERVE / REBUILD / BREAK | +| **V3 rationale** | why this verdict (not "because we said so") | +| **V3 impl location** | which V3 file + symbol enforces this | +| **Test anchor** | regression test that pins the contract | + +If the entity is `new` (§1.0), contracts are fresh and subject to §8C.5 semantic-contract audit instead of V2 evidence. + +--- + +## §2 Storage layer (T3 / G4) — retrofilled 2026-04-22 + +### §2.1 Entity bridge map (V2 → V3) + +| V2 entity | Bridge | V3 entity(ies) | Primary drift risk | +|---|---|---|---| +| `Subsystem` (NVMe) | merge | `DurableProvider.volumes` + `Target.ctrls` | ⚠️ Lifecycle transfer (BUG-005 landed here) | +| `Subsystem.Dev` (BlockDevice) | merge | Per-volume `LogicalStorage` cached in `DurableProvider` | ⚠️ Same as above | +| `Controller` (NVMe session) | split | `nvme.Session` + `StorageBackend` (borrowed) + adapter-layer fence | ⚠️ Concurrency model (BUG-001 landed here) | +| `BlockVol` (engine) | split | `LogicalStorage` + `ProjectionView` (T1) + `StorageBackend` adapter (T3a) | Authority model shift — fence lifted out | +| `walWriter` | 1:1 | `core/storage.walWriter` | Low — port stable | +| `walAdmission` | 1:1 | `core/storage.walAdmission` | Low; V3 drops Metrics callback | +| `DirtyMap` | 1:1 | `core/storage.dirtyMap` | Low — Phase 08 lesson honored | +| `GroupCommitter` | 1:1 | `core/storage.GroupCommitter` | **V3 drops `OnDegraded`, `PostSyncCheck`** (event model shift) | +| `Superblock` | 1:1 | `core/storage.superblock` + `smartwal.superblock` | **V3 drops `.Epoch` field** (authority model shift) | +| `flusher` | 1:1 | `core/storage.flusher` | Low | +| `pendingCapsules` | 1:1 | `nvme.Session` internal (rxLoop private) | Low; preserved post-BUG-001 revert | +| `KATO timer` | 1:1 | `nvme.Session` KATO state | V3 T3-scope: stored only, no timer enforcement (Set-only) | +| `AsyncEventRequest` slot | 1:1 | `nvme.Session.pendingAER` | Low | +| `CNTLID registry` | 1:1 | `nvme.Target.ctrls` | Low | +| `Target` (NVMe server) | 1:1 | `nvme.Target` | Low | +| `write_gate` (function) | **retired** | (no V3 entity; fence at adapter) | Authority model shift — correctly retired | +| — | **new** | `DurableProvider` | No V2 analog | +| — | **new** | `StorageBackend` (adapter) | No V2 analog | +| — | **new** | `ProjectionView` (T1) | Fence-aware contract new to V3 | +| — | **new** | `RecoveryReport` | No V2 analog; read-side readiness signal | + +### §2.2 Detailed entity audits (drift-prone + new) + +#### §2.2.1 `NVMe Subsystem` + `Subsystem.Dev` (merge → DurableProvider + LogicalStorage) + +- **V2 file(s)**: `weed/storage/blockvol/nvme/server.go` + inline `Subsystem` struct +- **Scope**: volume (NQN → one Subsystem → one Dev) +- **Lifecycle owner**: `Server.AddVolume` creates; `Server.RemoveVolume` destroys +- **Concurrency**: `Server.subsystems` map under `sync.RWMutex`; read from Controllers, write from admin +- **Cross-session behavior**: **persists** (sessions lookup by NQN; never modify Dev lifecycle) +- **Bridge shape**: merge (Subsystem + Subsystem.Dev + Subsystem.NQN metadata → split across DurableProvider + Identity) +- **V3 entity(ies)**: + - `DurableProvider.volumes map[string]*cachedVolume` — provides the "NQN → opened backend" lookup + - `LogicalStorage` instance — the durable engine, analogous to Subsystem.Dev + - `Identity` struct — VolumeID / ReplicaID / Epoch / EndpointVersion (authority-derived) +- **V3 embedding note**: + - *Lifecycle model shift*: V2 admin `AddVolume` (eager) → V3 Provider.Open (lazy, on first Connect); V3 Provider.Close (explicit at cmd/blockvolume shutdown) replaces RemoveVolume + - *Storage model shift*: V2 `BlockDevice` flat Read/Write → V3 `LogicalStorage` with LSN + Sync + Recover + Boundaries semantics + - *Authority model shift*: V2 Subsystem carried no authority state (NQN is just a label). V3 adds Identity struct alongside; adapter consumes for fence. Authority never flows into storage + +**Contracts**: + +| ID | Statement | V2 evidence | V3 verdict | V3 impl + rationale | Test anchor | +|---|---|---|---|---|---| +| C1-SUBSYS-DEV-LIFETIME | Dev lifetime = AddVolume → RemoveVolume; NOT session-managed | `server.go:AddVolume/RemoveVolume` | PRESERVE (explicit) | `DurableProvider.volumes` cache; `provider.go:Close` owns teardown. `Session.Close` MUST NOT touch storage (post-BUG-005 fix removed `defer backend.Close()` from iSCSI+NVMe handleConn). Godoc on `frontend.Backend.Close`: "owned by Provider; session layer MUST NOT call" | `t3b_bug005_backend_reuse_across_sessions_test.go` (BUG-005 regression) | +| C2-SUBSYS-REUSE-ACROSS-SESSIONS | Next Connect via NQN lookup returns same Dev | `server.go:handleConn` dials Subsystem by NQN | PRESERVE | `Provider.Open(volumeID)` returns cached Backend; same underlying LogicalStorage across sessions | `TestT3b_DurableProvider_Open_Caches` | +| C3-SUBSYS-REGISTRY-IS-SERVER-SCOPE | Registry lives one level above individual session | `server.go:subsystems` map | PRESERVE | `DurableProvider.volumes` sync.RWMutex; per-process | unit tests around Provider.Open | +| C4-SUBSYS-NO-AUTHORITY-WRITES | Subsystem + Dev never mint / advance / publish epoch | V2 had no such code — implicit | PRESERVE (explicit now) | Storage has no Epoch field (§G.2 audit §10.5 Option 3); fence via adapter's per-I/O ProjectionView | `INV-FRONTEND-002.*` facet tests under durable | + +#### §2.2.2 `NVMe Controller` (split → Session + StorageBackend + adapter fence) + +- **V2 file(s)**: `weed/storage/blockvol/nvme/controller.go` +- **Scope**: session (one per TCP connection) +- **Lifecycle owner**: `Server.acceptLoop` creates on Accept; `Controller.shutdown` destroys on conn close +- **Concurrency**: **rxLoop single goroutine**; **inline `collectR2TData` + `recvH2CData`** during Write; **`pendingCapsules` buffer** for CapsuleCmds arriving mid-recv +- **Cross-session behavior**: **session-scope** — Controller dies with the TCP connection; Subsystem.Dev and Server.subsystems remain +- **Bridge shape**: split +- **V3 entity(ies)**: + - `nvme.Session` — session logic (admin dispatch, Connect, Fabric handling, rxLoop); pure session state + - `StorageBackend` (from T3a) — storage-access façade; **borrowed** by Session, NOT owned + - Adapter-layer fence check — uses ProjectionView + Identity for per-I/O drift detection +- **V3 embedding note**: + - *Concurrency model*: **preserved post-BUG-001 revert**. `session.go`'s rxLoop stays single-threaded; `collectR2TData` inline with `bufferInterleaved`. Do NOT re-pragmatize. §8C.3 trigger #4 + - *Authority model shift*: Controller-layer fencing (`writeGate`) removed; fence is now adapter-layer per-I/O + - *Lifecycle*: Session closes → storage stays (BUG-005 fix) + +**Contracts**: + +| ID | Statement | V2 evidence | V3 verdict | V3 impl + rationale | Test anchor | +|---|---|---|---|---|---| +| C3-CONTROLLER-RXLOOP-SERIAL | rxLoop reads one CapsuleCmd at a time; no concurrent cmd dispatch | `controller.go:rxLoop` + `collectR2TData` inline | PRESERVE (revert-recovered) | `nvme/session.go` rxLoop serial; kernel's piplined cmds buffered via `pendingCapsules` | `t2_v2port_nvme_pipelined_writes_test.go` + m01 Matrix A | +| C6-CONTROLLER-R2T-BEFORE-H2CDATA | Write cmd triggers R2T BEFORE any H2CData consumed; bufferInterleaved absorbs CapsuleCmd arriving mid-H2C | `controller.go:recvH2CData` + `bufferInterleaved` | PRESERVE | session.go matches V2 pattern byte-for-byte post-revert | same | +| C1-CONTROLLER-NOT-CLOSE-STORAGE | Controller.shutdown does NOT call Subsystem.Dev.Close | `controller.go:shutdown` (no Dev.Close) | PRESERVE | Session.Close doesn't call backend.Close (post-BUG-005 fix). Backend.Close godoc enforces | BUG-005 regression test | +| C5-CONTROLLER-NO-RETRY | IO errors returned verbatim to host; target doesn't retry | V2's `write_retry.go` was the E-category pattern; Controller itself didn't retry | PRESERVE (write_retry never ported) | `INV-FRONTEND-NO-RETRY-001` — `TestT2V2Port_NVMe_IO_NoTargetRetry_*` | QA A11.4 test | + +#### §2.2.3 `BlockVol` (engine, split → LogicalStorage + ProjectionView + StorageBackend) + +- **V2 file(s)**: `weed/storage/blockvol/blockvol.go` +- **Scope**: volume +- **Lifecycle owner**: `OpenBlockVol` (crash recovery inline) / `Close` +- **Concurrency**: many internal locks + goroutines (flusher, groupCommit, dirtyMap shards, WAL) +- **Cross-session behavior**: **persists** — survives session lifecycle +- **Bridge shape**: split (engine functions distribute to three V3 layers) +- **V3 entity(ies)**: + - `core/storage.WALStore` / `smartwal.Store` — pure storage mechanism (Read / Write / Sync / Recover / Boundaries) + - `ProjectionView` (T1) — authority lineage source consumed by adapter + - `StorageBackend` (T3a) — adapts storage to `frontend.Backend` + per-I/O fence + operational gate +- **V3 embedding note**: + - *Storage model shift*: V2 flat BlockDevice → V3 LogicalStorage with LSN + Sync + Recover + - *Authority model shift*: V2 BlockVol could fence locally (`writeGate`) → V3 storage never fences; adapter fences via ProjectionView + - *Event model shift*: V2 group_commit error propagation via `OnDegraded` callback → V3 storage Sync returns error; adapter propagates upward via SCSI `MEDIUM_ERROR` / NVMe `InternalError` + +**Contracts** (selected; most already audited in T3.0 §3): + +| ID | Statement | V2 evidence | V3 verdict | V3 impl + rationale | Test anchor | +|---|---|---|---|---|---| +| C4-BLOCKVOL-WRITE-GATE-INTERNAL | Engine's write_gate check inside BlockVol.Write path | `blockvol.go:writeGate` | BREAK (retired) | V3 storage has no fence; moved to adapter. Storage serves blindly. T3.0 §3.7 + §10.3.7 LOCK | `INV-FRONTEND-002.*` fence tests | +| C2-BLOCKVOL-EPOCH-PERSIST | Epoch field in Superblock advance on fence changes | `blockvol.go` + `superblock.go:.Epoch` | BREAK (no V3 field) | V3 superblock has no Epoch (§10.5 Option 3); authority drift guard is per-I/O adapter check | same | +| C5-BLOCKVOL-NO-RETRY | `write_retry.go` retry-as-authority NOT ported | `write_retry.go` | BREAK | E-category per T3.0; QA A11.4 pins no-retry | `TestT2V2Port_NVMe_IO_ErrorsReturnedVerbatim` | + +#### §2.2.4 `GroupCommitter` (1:1 by name — V3 embedding differs) + +- **V2 file**: `weed/storage/blockvol/group_commit.go` +- **Scope**: volume (embedded in BlockVol) +- **Lifecycle owner**: BlockVol init/shutdown +- **Concurrency**: dedicated goroutine batching fsync waiters +- **Cross-session behavior**: persists +- **Bridge shape**: 1:1 (name preserved) +- **V3 entity**: `core/storage.GroupCommitter` +- **V3 embedding note (why 1:1 ≠ verbatim)**: + - *Event model shift* — V2 had `OnDegraded` callback fired on fsync error → V3 dropped the field; error flows via Sync return value instead. Upstream observers (adapter / volume host) derive "degraded" from error rate, NOT from a callback + - *Authority model shift* — V2 had `PostSyncCheck` callback that could re-check fencing after a batch. V3 dropped the field entirely (MATCHES-BETTER per T3.0 §10.3.6); mechanically cannot wire authority check + +**Contracts**: + +| ID | Statement | V2 evidence | V3 verdict | V3 impl + rationale | Test anchor | +|---|---|---|---|---|---| +| C3-GROUPCOMMIT-BATCH-SERIAL | Single writer fsyncs in batches; waiters queue + wake | `group_commit.go:Run + Submit` | PRESERVE | `core/storage/group_commit.go` — same shape | `flusher_test.go` | +| C5-GROUPCOMMIT-ERROR-PROPAGATION | Fsync error → observer notified | V2 `OnDegraded` callback | REBUILD (V3 event model) | V3 returns error via Sync; observer derivation lives at adapter / host layer (not reinstated as callback per T3.0 sw note A) | contract-check in T4 mini-plan if derived signal needed | +| C4-GROUPCOMMIT-NO-POSTSYNC-AUTHORITY | Post-sync hook can re-check fencing | V2 `PostSyncCheck` field (sometimes wired) | BREAK (field absent) | V3 GroupCommitterConfig lacks field; mechanically unreachable | T3.0 §10.3.6 audit | + +#### §2.2.5 `Superblock` (1:1 by name — field shape diverges) + +- **V2 file**: `weed/storage/blockvol/superblock.go` +- **Scope**: persistent (on-disk header) +- **Lifecycle**: Written at volume create; read at open / recovery +- **Bridge shape**: 1:1 +- **V3 entity**: `core/storage.superblock` + `core/storage/smartwal.superblock` +- **V3 embedding note**: + - *Authority model shift* — V2 had `.Epoch` + `.ExpandEpoch`; V3 has neither (§10.3.1 GAP + §10.5 Option 3) + - *Lifecycle model shift* — V2 had `DurabilityMode` / `StorageProfile` / `PreparedSize` fields; V3 simplified to block geometry + WAL geometry + (T3a-new) ImplKind + ImplVersion + +**Contracts**: + +| ID | Statement | V2 evidence | V3 verdict | V3 impl + rationale | Test anchor | +|---|---|---|---|---|---| +| C4-SUPERBLOCK-EPOCH-PERSIST | Epoch field on disk, read on open | V2 `superblock.go:.Epoch` | BREAK (no V3 field) | Per §10.5 Option 3; stronger §3.2 boundary | audit doc §10.5 | +| C3-SUPERBLOCK-WAL-GEOMETRY | WALOffset / WALSize / CheckpointLSN persist | V2 + V3 both have | PRESERVE | `core/storage/superblock.go` + `walstore.go:250 WALCheckpointLSN write-back` | `flusher_test.go` | +| C-SUPERBLOCK-IMPL-IDENTITY (new) | Impl kind + version recorded on create; mismatch fail-fast | — (V3-new) | NEW per Addendum A #2 | `superblock.go` ImplKind + ImplVersion fields; `Provider.Open` ErrImplKindMismatch | `INV-DURABLE-IMPL-IDENTITY-001` | + +#### §2.2.6 Retired: `write_gate` (BREAK — no V3 entity) + +- **V2 file**: `weed/storage/blockvol/write_gate.go` (28 LOC) +- **V2 role**: engine-internal pre-Write fencing check +- **Bridge shape**: retired +- **Rationale for BREAK**: Authority model shift. Storage must not fence. Fence is adapter-layer, per-I/O via ProjectionView. +- **V3 enforcement**: grep for `writeGate` / `ErrNotPrimary` / `ErrEpochStale` / `ErrLeaseExpired` in `core/storage/` returns zero matches. Reintroduction = §8C.3 trigger #4. +- **Test anchor**: existing T1 `INV-FRONTEND-002.*` facet tests (fence at frontend boundary, not storage). + +#### §2.2.7 New V3-era entities (no V2 analog) + +**`DurableProvider`** (no V2 analog) +- **Scope**: process +- **Lifecycle owner**: `cmd/blockvolume` main (Open lazy; Close at shutdown) +- **Concurrency**: `volumes map` under `sync.Mutex` +- **Cross-session behavior**: cached; multiple sessions share same Backend pointer +- **Contracts**: + +| ID | Statement | V3 verdict | Rationale | Test | +|---|---|---|---|---| +| C-PROVIDER-CACHE | Open returns same Backend for same volumeID | PRESERVE (new) | Mirrors V2 Server.subsystems lookup semantic | `TestT3b_DurableProvider_Open_Caches` | +| C-PROVIDER-OWNS-LIFECYCLE | Only Provider.Close tears down Backend + LogicalStorage | PRESERVE (new) | Rebuilds V2's AddVolume/RemoveVolume ownership; prevents BUG-005-like session-Close violations | `t3b_bug005_backend_reuse_across_sessions_test.go` | +| C-PROVIDER-IMPLKIND-GUARD | Open fail-fast if selector ≠ on-disk ImplKind | NEW | Addendum A #2 | `TestT3b_DurableProvider_Open_ImplKindMismatch_FailsFast` | + +**`StorageBackend`** (adapter, no V2 analog) +- **Scope**: volume +- **Contracts** (partial — full list in T3a mini plan): + +| ID | Statement | Rationale | Test | +|---|---|---|---| +| C-BACKEND-OPGATE | Before `SetOperational(true, _)`, all I/O returns `ErrNotReady` | Authority-safety gate; mirrors "not yet ready" without publishing | `INV-DURABLE-OPGATE-001` | +| C-BACKEND-LINEAGE-CHECK | Per-I/O ProjectionView lineage comparison; drift → ErrStalePrimary | New fence location (was V2 write_gate territory) | `INV-FRONTEND-002.*` under durable | +| C-BACKEND-CLOSE-PROVIDER-OWNED | Close may be called only by Provider.Close | Prevents BUG-005 | Backend.Close godoc + BUG-005 test | + +#### §2.2.8 T1 `frontend.Backend` interface (new V3-era — BUG-005 root entity) + +- **V2 analog**: none. V2 iSCSI / NVMe session code held direct pointers to `BlockDevice` / `Subsystem.Dev`. No per-session handle abstraction. +- **Scope**: volume (one Backend per VolumeID, shared across sessions via Provider cache) +- **Lifecycle owner**: `frontend.Provider` (T1) / `durable.DurableProvider` (T3b). Session layer borrows — NEVER owns. +- **Concurrency**: implementation-defined; atomic operational flag + lineage RWMutex in `StorageBackend` (T3a impl) +- **Cross-session behavior**: **persists** across session disconnect/reconnect; same pointer returned by repeat `Provider.Open(volumeID)` +- **Bridge shape**: new (no V2 predecessor) +- **V3 embedding note (semantic contract audit per §8C.5)**: + - *Lifecycle model*: V2's admin-lifetime pattern (`Subsystem.Dev` lives AddVolume → RemoveVolume) was transferred conceptually to Provider. `Backend` is the borrowed handle analog of V2's `Controller.subsystem` pointer + - *Event model*: Session reports errors via return values (no callbacks). `SetOperational` is push-based from Provider → Backend, not event-subscribed + - *Authority model*: `Identity()` accessor returns captured authority facts (VolumeID/ReplicaID/Epoch/EndpointVersion); Backend consumes lineage via ProjectionView + +**Contracts** (mostly new V3-era; fresh semantic-contract definitions): + +| ID | Statement | V3 verdict | V3 rationale + impl | Test anchor | +|---|---|---|---|---| +| C1-BACKEND-CLOSE-PROVIDER-OWNED | `Backend.Close()` may be called ONLY by Provider.Close (or test code); session layer MUST NOT call | NEW (post-BUG-005 explicit) | Backend godoc enforces; iSCSI `session.go` + NVMe `session.go` handleConn do NOT `defer backend.Close()` (sw fix); BUG-005 regression test pins | `t3b_bug005_backend_reuse_across_sessions_test.go` + iSCSI equivalent | +| C2-BACKEND-OPERATIONAL-GATE | Before `SetOperational(true, _)` all I/O returns `ErrNotReady`; after, I/O proceeds under lineage check | NEW | `core/frontend/durable/storage_adapter.go` gate; testback mirrors | `INV-DURABLE-OPGATE-001` | +| C3-BACKEND-LINEAGE-CHECK-PER-IO | Every Read/Write/Sync re-validates Identity vs current ProjectionView lineage; drift → `ErrStalePrimary` | NEW (replaces V2 `write_gate` location) | `storage_adapter.go:lineageCheck` | `INV-FRONTEND-002.EPOCH/.EV/.REPLICA/.HEALTHY` facets under durable | +| C4-BACKEND-IDENTITY-CAPTURED-AT-OPEN | `Identity()` returns the lineage snapshot taken at `Provider.Open`; never mutates after | PRESERVE-equivalent (new pattern aligned with V2's implicit "Dev config immutable during session") | `StorageBackend.id` set once in `NewStorageBackend`; not exposed for mutation | `TestT3a_StorageBackend_ImplementsBackend` | +| C5-BACKEND-SYNC-SPEC-LEGAL | `Sync(ctx)` error propagates to host as spec-legal status (SCSI `MEDIUM_ERROR` / NVMe `InternalError`); NEVER silent success | NEW | iSCSI `scsi.go` SYNC_CACHE wire; NVMe `io.go` Flush wire | `INV-DURABLE-SYNC-WIRED-001` | + +#### §2.2.9 T1 `frontend.ProjectionView` interface (new V3-era) + +- **V2 analog**: none. V2 authority state was checked inside storage via `write_gate`. +- **Scope**: volume +- **Lifecycle owner**: T1 volume host (`core/host/volume/projection_bridge.go` implements `AdapterProjectionView`) +- **Concurrency**: internal implementation uses atomic.Value for projection publication; `Projection()` accessor is lock-free read +- **Cross-session behavior**: persists; multiple sessions read concurrent; updated by host when authority changes +- **Bridge shape**: new +- **V3 embedding note**: + - *Authority model*: this is where authority becomes observable at frontend layer. Read-only interface — Backend consumers can check drift, NOT mutate + +**Contracts**: + +| ID | Statement | V3 verdict | V3 rationale + impl | Test anchor | +|---|---|---|---|---| +| C1-PROJECTIONVIEW-READONLY | No mutation method; only `Projection()` accessor | NEW | Interface has one method; boundary guard rejects any mutation-shaped call | `TestNoOtherAssignmentInfoConstruction` boundary test | +| C2-PROJECTIONVIEW-NEVER-ADVANCE-EPOCH | Consumers read Epoch, never write; epoch advance is authority-only (PCDD-STUFFING-001) | PRESERVE (explicit boundary) | Enforced by boundary_guard_test recursive walk + Projection struct fields unexported | `PCDD-STUFFING-001` rows | +| C3-PROJECTIONVIEW-SNAPSHOT-CONSISTENT | A single `Projection()` call returns an internally-consistent tuple; no tearing between fields | NEW | `atomic.Value` stored full Projection struct; readers see whole-struct atomicity | T1 tests around `AdapterProjectionView` | + +#### §2.2.10 T1 `frontend.Provider` interface (new V3-era, parent abstraction) + +- **V2 analog**: partially V2 `Server.AddVolume / subsystems map` but without admin semantics +- **Scope**: process +- **Lifecycle owner**: T1 defined interface; T3b `durable.DurableProvider` is impl; `testback.StaticProvider` is test impl; `memback.Provider` is memback impl +- **Concurrency**: impl-defined; DurableProvider uses per-Provider Mutex on volumes map +- **Cross-session behavior**: N/A (Provider isn't session-scoped — it IS the thing sessions consult) +- **Bridge shape**: new (with V2 conceptual echo of admin lookup registry) +- **V3 embedding note**: + - *Lifecycle model*: V3 introduces lazy-open + cached pattern; V2 had eager `AddVolume`. Provider.Open is idempotent; repeated Open for same volumeID returns same Backend + - *Event model*: Open is request/response, no subscriptions + +**Contracts**: + +| ID | Statement | V3 verdict | V3 rationale + impl | Test anchor | +|---|---|---|---|---| +| C1-PROVIDER-SINGLE-ENTRY | `Open(ctx, volumeID)` is the sole entry for session layer; no direct storage access | PRESERVE (boundary) | Session layer holds only `Provider` reference; frontend/iscsi + frontend/nvme `handleConn` call `provider.Open` | iSCSI + NVMe boundary guard tests | +| C2-PROVIDER-OPEN-IDEMPOTENT-BY-VOLUMEID | Repeated Open(same volumeID) returns same Backend instance | NEW (T3b) | `DurableProvider.volumes` cache; `TestT3b_DurableProvider_Open_Caches` | cache test | + +#### §2.2.11 T2 iSCSI `Target` (1:1 — V2 registry analog) + +- **V2 analog**: V2 `iscsi.TargetRegistry` / `LookupDevice(iqn)` (implicit — pointer-held by session code) +- **Scope**: process +- **Lifecycle owner**: `cmd/blockvolume` main +- **Concurrency**: `Target.sessions sync.WaitGroup` for draining; no shared-device registry (single-target shape in T2 scope) +- **Cross-session behavior**: Target outlives any session +- **Bridge shape**: 1:1 (name preserved) +- **V3 embedding note**: + - *Lifecycle model shift* (minor): T2 Target single-target shape (one IQN per Target); V2 had multi-target registry (lookup by IQN). Multi-target deferred to post-T2 if needed + +**Contracts**: + +| ID | Statement | V3 verdict | Rationale + impl | Test anchor | +|---|---|---|---|---| +| C1-ISCSI-TARGET-SESSION-NOT-CLOSE-DEVICE | Target's `handleConn` session close does NOT close Backend | PRESERVE (post-BUG-005 fix for iSCSI same as NVMe) | `core/frontend/iscsi/session.go:80 "backend holds... so serve() can Close it"` **comment is misleading**; actual serve() does NOT call backend.Close (removed) — referenced `bugs/005_backend_close_cross_session.md` | same BUG-005 test family | +| C2-ISCSI-TARGET-ACCEPT-LOOP | One goroutine per connection; Target.sessions WaitGroup drains on Close | PRESERVE | `target.go:handleConn` spawned goroutine; WaitGroup pattern same as V2 | target close tests | + +#### §2.2.12 T2 iSCSI `Session` (session-scope; BUG-005 sibling site) + +- **V2 analog**: V2 iSCSI Session equivalent (session state machine + PDU loop) +- **Scope**: session (one per TCP connection) +- **Lifecycle owner**: `Target.handleConn` creates; `Session.Close` on session end +- **Concurrency**: single `serve()` goroutine (PDU rx/tx loop) +- **Cross-session behavior**: dies with TCP conn +- **Bridge shape**: 1:1 +- **V3 embedding note**: + - *Lifecycle model shift*: V3 Session holds `backend frontend.Backend` borrowed from Provider; same pattern as NVMe Session post-BUG-005 revert + - *Concurrency model*: PDU loop stays serial; no pragmatic multi-goroutine (BUG-001 lesson transferred to iSCSI preemptively) + +**Contracts**: + +| ID | Statement | V3 verdict | Rationale + impl | Test anchor | +|---|---|---|---|---| +| C1-ISCSI-SESSION-NOT-CLOSE-BACKEND | Session.Close does NOT call backend.Close | PRESERVE (BUG-005 fix applied symmetrically) | `session.go` serve loop; backend is borrowed | Integration tests + BUG-005 test | +| C3-ISCSI-SESSION-PDU-SERIAL | PDU rx/tx loop serial per session (matches NVMe rxLoop pattern) | PRESERVE | `session.go:serve` serial read/write; no per-PDU goroutine spawn | existing iSCSI tests + T2 m01 sanity | +| C4-ISCSI-SESSION-DISCOVERY-SKIPS-BACKEND | Discovery sessions don't open Backend | PRESERVE | `session.go:45` comment; backend opened only in Normal session path | `t2_iscsi_discovery_test.go` | + +#### §2.2.13 T2 iSCSI VPD / Identify metadata (Addendum A root entity) + +- **V2 file**: `weed/storage/blockvol/iscsi/scsi.go` VPD builders; Serial / Model / Vendor / NAA fields +- **Scope**: per-volume, derived from Identity +- **Lifecycle**: computed on-demand per INQUIRY response (stateless) +- **Bridge shape**: 1:1 (same INQUIRY VPD pages) with **V3 embedding shift**: Serial/NAA derivation from VolumeID instead of V2's hardcoded stub +- **V3 embedding note**: + - *Authority model*: VPD 0x80 (Serial) + VPD 0x83 (NAA) now derived from `sha256(SubsysNQN+VolumeID)` for cross-volume uniqueness (was V2 hardcoded string) + +**Contracts**: + +| ID | Statement | V2 | V3 verdict | Rationale + impl | Test anchor | +|---|---|---|---|---|---| +| C1-ISCSI-VPD80-SERIAL-DERIVATION | VPD 0x80 Serial = sha256-derived per VolumeID (16-char hex) | V2 **hardcoded** `"SeaweedFS"` stub | **REBUILD** (was mis-classified PORT-AS-IS at Batch 10.5 → caught at addendum A) | `scsi.go:serialFromVolumeID` | `PCDD-ISCSI-VPD80-SERIAL-DETERMINISM-001` | +| C2-ISCSI-VPD83-NAA-DERIVATION | VPD 0x83 NAA-6 derived sha256 per (SubsysNQN, VolumeID) | — (V3 introduced) | NEW | `scsi.go:naaFromVolumeID` | `PCDD-ISCSI-VPD83-NAA-DETERMINISM-001` | +| C3-ISCSI-VPD00-ADVERTISED-MATCHES-IMPL | Advertised page list ≡ actually-served page set | — | NEW | `TestT2V2Port_SCSI_InquiryVPD00_AdvertisesOnlyImplemented` | same test | +| C4-ISCSI-VPD-STATIC-FIELDS-HARDCODED | Model / Vendor / Firmware-Rev / ProductRev fields come from HandlerConfig (defaults are V2-era constants `"SeaweedF"` / `"BlockVol"`); no per-volume derivation | V2 hardcoded same way | **PORT-AS-IS (LOCKED explicit)** — per-volume uniqueness NOT a SCSI requirement for these fields; V2 pattern correct. **REBUILD trigger**: if ANA-reporting or multi-tenant isolation starts requiring per-subsystem distinguishability, reclassify to REBUILD (sha256-derive like Serial/NAA). See `bugs/inventory/nvme-test-coverage-deferred.md` T3-DEF-5 | none (no regression needed; discipline row) | + +**Latent-gap audit note (Tier A retrofill finding)**: Parallel entities — Model string (`"BlockVol"`), Vendor ID (`"SeaweedFS"`), Firmware Rev — are currently still **hardcoded constants** (same pattern V2 had). Not a bug today (per-volume uniqueness not required for Model/Vendor), but SHOULD be explicitly marked PORT-AS-IS (not forgotten-without-review) to avoid future "second Addendum A" if e.g. ANA or multi-tenant requirements make them matter later. **Inventory row T3-DEF-5** added to `bugs/inventory/nvme-test-coverage-deferred.md`: "iSCSI VPD Model/Vendor/FW-Rev still V2-hardcoded; audit if per-subsystem uniqueness becomes required". + +#### §2.2.14 T2 NVMe Session state (KATO / AER / CNTLID — beyond §2.2.2 Controller-split row) + +- **V2 analog**: V2 `Controller.kato` / `Controller.pendingAER` / `Controller.cntlID` +- **Scope**: session +- **Lifecycle owner**: Session init on admin Connect; destroyed on Session.Close +- **Concurrency**: accessed from session serve loop; AER slot is atomic.Value for single-slot check +- **Cross-session behavior**: dies with session +- **Bridge shape**: 1:1 (fields live inside Controller (V2) → inside Session (V3)) +- **V3 embedding note**: + - *Timing model shift*: V2 KATO fired a timer goroutine; V3 T3-scope stores KATO value only (no timer). Revisit if m01 exposes need (per BUG-003 Discovery Bridge scope limitation) + - *Event model shift (AER)*: V2 AER had full event source (namespace attr change, etc.); V3 AER is **park-only** stub (no events produced); advertised capability bits in Identify Controller deliberately 0 (INV-NVME-IDENTIFY-CTRL-ADVERTISED-LIST-001) + +**Contracts**: + +| ID | Statement | V3 verdict | Rationale + impl | Test anchor | +|---|---|---|---|---| +| C1-NVME-SESSION-KATO-STORED-NOT-ENFORCED | KATO from Fabric Connect CDW12 stored; no timer enforcement in T3 scope | PRESERVE-partial (REBUILD needed vs V2 full-timer path) — **VIOLATED today; tracked by BUG-006** | `session.go` stores `katoMS` but never arms watchdog. Contract gap surfaced by m01 Matrix D kernel warning `long keepalive RTT (2522123190 ms)`. Target G21 / post-T3 | `PCDD-NVME-FABRIC-CONNECT-KATO-001` + `bugs/006_nvme_kato_timer_not_enforced.md` | +| C2-NVME-SESSION-AER-PARK-ONLY | AER request parked in single slot; never emits events; slot cleared on session close | REBUILD (V2 full-event-source → V3 park-only) | `session.go` tryParkAER + clearPendingAER | `TestT3b_NVMe_AER_Parks` + OAES advertised=0 pin | +| C3-NVME-SESSION-CNTLID-ECHO | IO Connect must echo admin CNTLID; mismatch rejected | NEW (V3 strict; V2 was looser) | `session.go` handleIOConnect + Target.ctrls lookup | `PCDD-NVME-FABRIC-CNTLID-ECHO-001` | +| C4-NVME-SESSION-SERIAL-PDU-LOOP | rxLoop single-threaded; same C3-CONTROLLER-RXLOOP-SERIAL but session-scope view | PRESERVE | same as §2.2.2 C3 | BUG-001 test family | +| C5-NVME-SESSION-STATE-CLEANUP-ON-CLOSE | All session-scope state (KATO, AER slot, CNTLID reservation, pendingCapsules) MUST be released by Session.Close; leaks across sessions forbidden | NEW-explicit (T3-DEF-6 retrofit) — PASSES today | `session.go` Close path releases each; now pinned explicitly rather than via implicit goroutine-leak count | `TestT3_NVMe_Session_CleanupOnClose_NoLeaksAcrossCycles` + `TestT3_NVMe_Session_CleanupOnClose_CNTLIDAllocator` (`t3_qa_session_cleanup_addendum_test.go`) | + +**Latent-gap audit note (closed 2026-04-22)**: C5 (cleanup-on-close) was formerly an implicit contract tested only via goroutine leak count. Now explicit with dedicated L1 QA addendum tests (T3-DEF-6). C1 (KATO enforcement) was previously tagged PRESERVE-partial assuming revisit later; Matrix D m01 evidence proves the contract is already **violated** — reclassified with BUG-006 anchor (T3-DEF-7). + +### §2.3 Known drift events mapped to this catalogue (audit trail) + +| Event | Root entity | Missed discipline | Catalogue row | +|---|---|---|---| +| BUG-001 (serial-vs-pipelined R2T) | Controller (split) | rxLoop concurrency verdict not written; split mis-assigned pipelining to goroutines | C3-CONTROLLER-RXLOOP-SERIAL + C6-CONTROLLER-R2T-BEFORE-H2CDATA | +| BUG-005 (Backend.Close in session) | Subsystem.Dev (merge) | Lifetime contract not transferred explicitly during merge to DurableProvider | C1-SUBSYS-DEV-LIFETIME | +| Addendum A (VPD 0x80 Serial stub) | iSCSI VPD/Identify metadata (§2.2.13) | V2 hardcoded stub treated as PORT-AS-IS; actually needed REBUILD (derive per-VolumeID) | C1-ISCSI-VPD80-SERIAL-DERIVATION (REBUILD verdict) | +| BUG-001 (revert) | §2.2.2 Controller split | rxLoop concurrency verdict wrongly BREAK in initial fix | C3-CONTROLLER-RXLOOP-SERIAL + C6-CONTROLLER-R2T-BEFORE-H2CDATA (now PRESERVE verdicts locked) | +| BUG-005 | §2.2.1 Subsystem.Dev merge + §2.2.8 Backend + §2.2.12 iSCSI Session + §2.2.2 NVMe Controller | Lifetime contract dropped in Subsystem→Provider merge; session Close path wrongly touched Backend | C1-SUBSYS-DEV-LIFETIME + C1-BACKEND-CLOSE-PROVIDER-OWNED + C1-ISCSI-SESSION-NOT-CLOSE-BACKEND (all now PRESERVE / NEW-explicit) | +| T3-rev-1 scope drift (Perf → G4) | Track identity | Not an entity drift but a roadmap drift; covered by §8C.3 trigger + architect review, not by this catalogue | +| BUG-006 (KATO timer not enforced) — T3-DEF-7 | §2.2.14 NVMe Session state | C1-NVME-SESSION-KATO row originally tagged PRESERVE-partial deferring timer work to "follow-up"; m01 Matrix D kernel warning proves contract is violated today, not deferred-safe | C1-NVME-SESSION-KATO-STORED-NOT-ENFORCED (reclassified to VIOLATED with BUG-006 anchor) | +| BUG-007 (walstore umount+remount data loss) | §2.2.5 Superblock + §2.2.4 GroupCommitter — **walstore impl side** | Cross-session durability under fs-triggered umount sync was never explicitly contracted for the walstore impl; smartwal happens to satisfy, walstore happens to fail | (new PROVISIONAL row to add post-T3): C?-DURABLE-READ-SERVES-UNCOMMITTED-WAL (walstore gap) | +| T3-DEF-5 (iSCSI VPD static fields — model/vendor/firmware) | §2.2.13 iSCSI VPD metadata | Reviewed during Addendum A but not catalogued; risk of future drift if multi-tenant isolation need arises | C4-ISCSI-VPD-STATIC-FIELDS-HARDCODED (PORT-AS-IS LOCKED explicit) | +| T3-DEF-6 (NVMe session cleanup contract implicit) | §2.2.14 NVMe Session state | Cleanup-on-close was tested implicitly via goroutine leak, never explicit contract row | C5-NVME-SESSION-STATE-CLEANUP-ON-CLOSE (NEW-explicit; QA L1 addendum pins it) | + +--- + +## §3 Replication (T4 / G5) — MANDATORY BEFORE T4 T-START THREE-SIGN + +### §3.1 Entity bridge map (V2 → V3) — TO BE FILLED pre-T4 + +Expected V2 entities (per `v2-test-db.md` + memory references): + +- `wal_shipper` (WAL ship protocol state machine) — session-scope +- `shipper_group` (multi-replica fan-out) — volume-scope +- `replica_apply` (receiver side) — volume-scope +- `replica_barrier` (write ack barrier) — volume-scope +- Replication session lifecycle (per-connection between primary + replica) +- Durable replicated LSN (state on primary + replica) + +Expected V3 topology: TBD — sw + QA T4 sketch must enumerate. 1:1 / split / merge / new for each. + +### §3.2 Contract catalogue (per entity) — TO BE FILLED + +### §3.3 V3 model shifts relevant to replication + +Pre-fill hints (subject to T4 sketch authoring): + +- *Event model*: V2 replication callbacks (barrier fn, degradation fn) → V3 likely typed error return + ctx cancellation +- *Authority model*: V2 primary advanced epoch on promotion locally → V3 master authority publishes; replica storage is passive recipient +- *Concurrency model*: V2 shipper had bounded in-flight state → V3 likely similar but with clean context cancellation + +--- + +## §4 Rebuild (T5 / G6) — STUB, mandatory pre-T5 + +## §5 Failover (T6 / G8) — STUB, mandatory pre-T6 + +## §6 ALUA / auth / CHAP (T7+) — STUB + +--- + +## §7 Discipline integration into §8C (QA system doc) + +### §7.1 Addition to §8C.1 (T-start three-sign) + +Port plan / track sketch MUST include, in top-down order: + +1. **L1 entity enumeration** (§1.2 column set) — all V2 entities in scope +2. **L2 bridge + contracts** (§1.3 columns) — §N.1 entity bridge map + §N.2 detailed entity audits +3. **L3 file/function classification** — derived from L1/L2; not signed without L1/L2 + +T-start three-sign REJECTED if §N.1 / §N.2 skipped. + +### §7.2 Addition to §8C.3 triggers + +- **#4 port-model violation** already covers "pragmatic evolution / simplify during port" +- **NEW #8**: L1 entity enumeration missing in current track's port sketch; any code PR against the track should surface this before the PR merges + +### §7.3 Maintenance + +- This catalogue is LIVING per track +- Every bug doc (BUG-NNN) that traces to a missed contract MUST retrofill the row with the drift-event footnote (§2.3 style) +- Future agents consulting "why did V3 do X instead of V2's Y" find answer in §2.2.X entity row (not in scattered audit docs) + +--- + +## §8 Change log + +| Date | Change | Author | +|---|---|---| +| 2026-04-22 | Initial catalogue. §2 Storage T3/G4 retrofill: entity map (21 rows) + detailed audits for drift-prone + new entities; §2.3 drift history mapped (BUG-001 / BUG-005 / Addendum A); §3-§6 stubs; §7 discipline integration into §8C. §1.1 V3 model shifts documented as baseline framework. | QA Owner | diff --git a/sw-block/design/v3-phase-15-t3-closure-report.md b/sw-block/design/v3-phase-15-t3-closure-report.md new file mode 100644 index 000000000..1f23ecba7 --- /dev/null +++ b/sw-block/design/v3-phase-15-t3-closure-report.md @@ -0,0 +1,419 @@ +# V3 Phase 15 T3 — Closure Report + +**Date**: 2026-04-22 +**Status**: DRAFT — awaiting T3-end three-sign per §8C.1 +**Gate**: T3 → G4 (durable-data-path pass) +**Predecessor**: T2 three-signed (frontend contract complete) +**Successor**: G5 (replicated write path) or whichever canonical sequence defines + +--- + +## §A Batch history + +| Batch | Commit | Status | Delivery | +|---|---|---|---| +| T3.0 (port audit) | doc-only | QA single-signed | `v3-phase-15-t3-port-audit.md` + Addendum A; 7 V2 muscle files classified PORT-AS-IS / PORT-REBIND / NEVER; risk flags resolved | +| T3a | `0e1595c` | CLOSED | StorageBackend adapter (G-int.1+.2+.3) + Backend interface extension (Sync + SetOperational) + superblock ImplKind/ImplVersion (Addendum A #2) + matrix test infra (Addendum A #1) | +| T3b | `72d0d40` | CLOSED | DurableProvider (G-int.4) + Recovery (G-int.5) + iSCSI SYNC_CACHE wire + NVMe Flush wire + cmd/blockvolume integration | +| T3b follow-up | `5c33460` | CLOSED | iSCSI + NVMe → durable matrix integration tests (closes wire-level coverage gap) | +| T3c | `829c6a9` + `5c33460` (T3b follow-up integration matrix) | pending three-sign | 4 scenarios × 2 impls + perf baseline first-light + closure report + m01 verification (2 impls) | +| Pre-sign fix Phase 1 | `4c256aa` | CLOSED | QA artifacts commit + doc integrity (audit header / commit hash / perf doc / sketch delta) | +| **BUG-005 fix** | **`42b045a`** | CLOSED | Session-layer `Backend.Close` removed (nvme target + iscsi session); `frontend.Backend` interface godoc Provider-ownership clause; regression test matrix both impls. See §H Phase 3. | +| m01 infra | `27486db` | CLOSED | `iterate-m01-nvme.sh` pre-cleanup + dmesg clear for reproducible BUG-005 re-verify | + +--- + +## §B Delivery summary + +### What T3 delivered + +1. **StorageBackend adapter** — `core/frontend/durable/storage_adapter.go`. Bridges `core/storage.LogicalStorage` (LBA-based, typed errors) to `core/frontend.Backend` (byte-offset, fence + operational gate). ~310 LOC production, 22 subtests × 2 impls = 44 subtests green. + +2. **Backend interface extension** — `core/frontend/types.go` gains `Sync(ctx) error` + `SetOperational(ok, evidence)`. All 6 existing Backend implementations updated atomically in the same commit (memback + testback + 3 QA test shims). + +3. **Superblock impl-identity** (Addendum A #2) — both walstore and smartwal superblocks gain `ImplKind` + `ImplVersion` fields. Schema version bumped 1→2 with one-way discipline. DurableProvider peeks the 4-byte magic BEFORE storage Open and fails fast with `ErrImplKindMismatch` on selector/disk disagreement. + +4. **DurableProvider** — `core/frontend/durable/provider.go`. Per-volumeID handle cache. Production default is `smartwal`; `walstore` reachable via `--durable-impl walstore`. Integrated into `cmd/blockvolume` behind the `--durable-root` flag; memback fallback preserved when flag is empty. + +5. **Recovery wrapper** — `core/frontend/durable/recovery.go`. Pure function wrapping `LogicalStorage.Recover()` into a `RecoveryReport`. No new publication path (Option 3 from audit §10.5): report surfaces via the existing `/status` HTTP endpoint + adapter's `SetOperational`; master authority remains sole Epoch / assignment publisher. + +6. **Frontend Sync wire** — iSCSI `SYNCHRONIZE_CACHE(10)` + `SYNCHRONIZE_CACHE(16)` and NVMe `ioFlush` dispatch to `backend.Sync(ctx)`. Errors map faithfully to spec-legal status (SCSI `MEDIUM_ERROR` / `WRITE_ERROR`; NVMe `InternalError`). Stop rule §3.6 forbids swallowing. + +7. **Matrix test infrastructure** (Addendum A #1) — single shared `logicalStorageFactories()` helper returning `{walstore, smartwal}`. Every T3a/T3b/T3c test iterates over it — 68 subtests per impl in the durable package. + +8. **4 scenarios** (T3c): + - `TestT3c_Scenario_CrashRecovery_AckedBytesByteExact` — pins INV-DURABLE-001 + - `TestT3c_Scenario_WALReplay_NAckedWrites` — pins INV-DURABLE-WAL-REPLAY-001 + - `TestT3c_Scenario_FsyncBoundary_AckedSurvivesUnackedMayVanish` — pins INV-DURABLE-FSYNC-BOUNDARY-001 + - `TestT3c_Scenario_DiskFill_OutOfRange_FailsCleanly` — pins PCDD-DURABLE-DISK-FULL-001 + + 4 YAML shape docs in `testrunner/scenarios/testdata/` mirror each Go replay for reviewability. + +9. **Perf baseline** — `testrunner/perf/t3c-durable-baseline.md` + `BenchmarkT3c_DurablePerf`. Characterization only; no threshold. First-light numbers filled 2026-04-22 (sw dev loopback): walstore 197K ns/op / 20.75 MB/s / ~5067 ops/s; smartwal 303K ns/op / 13.50 MB/s / ~3295 ops/s. Sustained-throughput m01 fio measurement is a post-sign activity. + +10. **cmd/blockvolume integration** — 4 new flags (`--durable-root`, `--durable-impl`, `--durable-blocks`, `--durable-blocksize`). Startup: build Provider → Open → RecoverVolume → flip operational → accept connections. Shutdown: Provider.Close (backend first, storage second, both idempotent). + +### Non-claims (explicit) + +- **No replication.** T3 is single-node durable storage; replicated writes + failover + rebuild belong to G5+. The existing Phase 4A replication surface (shipper, barrier, promotion, rebuild) is NOT touched by T3 and is NOT claimed to integrate with DurableProvider until a later gate explicitly ports it. +- **No authority publication.** T3 storage observes and reports; master authority remains sole Epoch / assignment publisher (§3.2 boundary; PCDD-STUFFING-001). Recovery surfaces readiness via `/status` read-side only. +- **No write_gate in storage.** V2's `write_gate.go` is explicitly NOT ported; fencing is adapter-side via per-I/O lineage check (INV-FRONTEND-002.*). +- **No ambitious concurrency rewrite.** The V2 patterns that were previously ported faithfully (BUG-001 lesson) stay that way; no simplification / "V3 style" reinvention. +- **Perf numbers are first-light.** The Go-bench loopback numbers in §B #9 are characterization only; sustained-throughput fio on m01 is a post-sign activity, not a delivered T3 metric. +- **G4 product pass = smartwal only.** walstore is a documented non-default fallback impl. Matrix D failure (BUG-007) means walstore is NOT G4-qualified at T3 close; walstore side of `INV-DURABLE-001` is deferred to BUG-007 close. Production default (smartwal) satisfies canonical G4. + +### Governance transition + +- Mid-T batches (T3a, T3b, T3c) used QA single-sign per §8C.2 — accelerated cadence. +- T3-end requires three-sign per §8C.1 (architect + PM + QA) — this report is the artifact. +- Discovery Bridge: no §8C.3 triggers fired during T3; all scope changes (Addendum A additions to T3.0 audit) went through QA single-sign. + +--- + +## §C V2 port depth audit + +Per T3.0 §10 walk-through, the 7 V2 durable-layer muscle files validated against V3 existing port: + +| V2 file | V3 file | V2 LOC | V3 LOC | Verdict | +|---|---|---|---|---| +| `superblock.go` | `core/storage/superblock.go` | 298 | 227* | MATCHES (* +30 from T3a ImplKind add) | +| `wal_entry.go` | `core/storage/wal_entry.go` | 153 | 169 | MATCHES (V3 slightly larger, doc) | +| `wal_writer.go` | `core/storage/wal_writer.go` | 302 | 193 | MATCHES | +| `wal_admission.go` | `core/storage/wal_admission.go` | 178 | 156 | MATCHES-BETTER (V3 is pure-mechanism; no authority hooks) | +| `dirty_map.go` | `core/storage/dirty_map.go` | 156 | 141 | MATCHES-BETTER (V3 splits Delete + compareAndDelete; unconditional Delete has zero callers — Phase 08 footgun structurally disarmed) | +| `group_commit.go` | `core/storage/group_commit.go` | 222 | 164 | MATCHES-BETTER (V3 drops PostSyncCheck field entirely; authority-check path unreachable by construction) | +| `write_gate.go` | (none) | 28 | 0 | NEVER (§3.2 boundary — fencing is adapter-side) | + +Plus 1 audit blocking gap resolved: +- **superblock.Epoch** absent in V3 → LOCKED as Option 3 (no local mirror; per-I/O adapter fence check is sole authority-drift guard). Stronger §3.2 outcome than sw's original PORT-REBIND proposal. + +Total summary: **48 MATCHES + 2 MATCHES-BETTER + 1 Option-3-resolved + 0 VIOLATIONS** → V3 existing `core/storage/` port quality is high; user's decision to retain V3 over re-porting from V2 is validated. + +--- + +## §D Ledger rows + +### New at T3 close (queued ACTIVE pending three-sign) + +| ID | Batch | Statement | +|---|---|---| +| `INV-DURABLE-OPGATE-001` | T3a | Before `SetOperational(true, _)`, all I/O returns `ErrNotReady`; preserves `INV-FRONTEND-002.*` under durable backend | +| `INV-DURABLE-IMPL-IDENTITY-001` | T3a | Superblock records `ImplKind` + `ImplVersion`; mismatch between stored ImplKind and opener's selector is rejected, not silently coerced | +| `INV-DURABLE-PROVIDER-SELECT-001` | T3b | `DurableProvider.Open` selects correct impl per config + fails fast on ImplKind mismatch | +| `INV-DURABLE-RECOVERY-READSIDE-001` | T3b | Recovery exposes readiness via adapter's `SetOperational` + existing `/status` HTTP surface; no new publication path; master authority remains sole Epoch/assignment publisher | +| `INV-DURABLE-SYNC-WIRED-001` | T3b | iSCSI `SYNCHRONIZE_CACHE(10/16)` + NVMe `Flush` reach `LogicalStorage.Sync`; error propagates to spec-legal host status | +| `INV-DURABLE-001` | T3c + m01 Matrix E | Acknowledged Write survives real SIGKILL + restart; Read returns byte-exact. **Scope: smartwal only** (production default); walstore evidence deferred to BUG-007 close. Canonical row name stands per m01 Matrix E SIGKILL + byte-exact readback (2026-04-22, commit `seaweed_block@313dd52` Matrix F robustness fix; evidence run against BUG-005 fix `42b045a`). | +| `INV-DURABLE-SESSION-LIFECYCLE-001` | BUG-005 fix | Across NVMe/iSCSI session disconnect + reconnect, the Provider-cached Backend remains operational; I/O on the reconnected session MUST NOT receive ErrBackendClosed caused by prior session teardown | +| `INV-DURABLE-WAL-REPLAY-001` | T3c | N acked writes + crash → all N recoverable | +| `INV-DURABLE-FSYNC-BOUNDARY-001` | T3c | Flushed preserved; unflushed may be lost; neither corrupted | +| `PCDD-DURABLE-DISK-FULL-001` | T3c | Disk-full / volume-full → hard error; no silent success; in-range data uncorrupted | + +### Existing rows confirmed still ACTIVE + +T1 facets (unchanged): +- `INV-FRONTEND-002.EPOCH / .EV / .REPLICA / .HEALTHY` — verified under durable backend via T3a fence-check tests + T3c integration tests + +T2 rows (unchanged): +- 16 ACTIVE rows covering iSCSI + NVMe protocol invariants — verified still green post-Backend-interface-extension via full regression (17/17 packages ok) + +--- + +## §E T3-end three-sign + +### Deliverable artifacts + +- `sw-block/design/v3-phase-15-t3-port-plan-sketch.md` (rev-2.1, three-signed) +- `sw-block/design/v3-phase-15-t3-port-audit.md` (QA-signed with Addendum A) +- `sw-block/design/v3-phase-15-t3a-mini-plan.md` (CLOSED) +- `sw-block/design/v3-phase-15-t3b-mini-plan.md` (CLOSED) +- `sw-block/design/v3-phase-15-t3c-mini-plan.md` (this batch) +- `sw-block/design/v3-phase-15-t3-closure-report.md` (this doc) +- `testrunner/perf/t3c-durable-baseline.md` (first-light numbers) +- `testrunner/scenarios/testdata/t3c-durable-*.yaml` (4 scenario shape docs) + +### Full regression + +``` +go test ./core/... -count=1 → 17/17 packages ok + core/frontend/durable: ~1s (68+ subtests, all green, matrix 2×) + core/frontend/iscsi: ~31s (no regression; SYNC_CACHE wire green) + core/frontend/nvme: ~61s (no regression; Flush wire green) + core/host: ~123s (subprocess tests still use memback; no change) + all others sub-second +``` + +Boundary guard green: `core/frontend/durable/` auto-covered by the recursive walk in `TestFrontendCannotMintAuthority_BoundaryGuard`. + +### Sign table + +| Role | Signer | Date | Decision | +|---|---|---|---| +| QA Owner | Claude (QA agent) | 2026-04-22 | ✅ SIGNED. Basis: smartwal full A–F green on m01 (commit `seaweed_block@313dd52`, BUG-005 fix `42b045a`); Matrix E real SIGKILL + byte-exact recovery satisfies canonical G4 pass gate; Matrix F real ENOSPC graceful hard-error satisfies PCDD-DURABLE-DISK-FULL-001. walstore scoped as non-default fallback with BUG-007 carry-forward. T3-DEF-5/6/7 retrospective closed (catalogue / L1 addendum / BUG-006). Prior retraction superseded. | +| Architect | pingqiu | _________ | ⏸ awaiting | +| PM | pingqiu | _________ | ⏸ awaiting | + +**Effect upon three-sign**: T3 CLOSED, Gate G4 passes (smartwal production default), P15 advances to next gate. walstore tracked separately via BUG-007. + +### m01 fs-workload verification (2026-04-22, QA — final) + +**Evidence base**: `seaweed_block` commits `42b045a` (BUG-005 fix) + `313dd52` (Matrix F cleanup robustness + T3-DEF-6 test). Script: `scripts/iterate-m01-nvme.sh` at `313dd52`. Host: m01 (192.168.1.181) real Linux 6.17 kernel + in-tree NVMe/TCP initiator + `mkfs.ext4`. + +Six matrices A–F run per Addendum A #1 matrix discipline. **Pass criterion for G4 = smartwal (production default) full A–F green**. walstore results reported for completeness but explicitly not required for G4 at T3 close. + +| Matrix | Coverage | smartwal | walstore | G4 evidence? | +|---|---|:-:|:-:|:-:| +| A | 10 cycles: attach / mkfs / mount / dd / sync / umount / disconnect | ✅ 10/10 | ✅ 10/10 | yes | +| B | Size coverage 32K × 50 + 256K × 10 + 1M × 5 | ✅ | ✅ | yes (adjunct) | +| C | Small-file burst: 500/500 files + sync → 501 fs-count (group_commit batching stressed) | ✅ | ✅ | yes (adjunct) | +| D | Cross-session consistency: 5 remount cycles with pattern integrity check | ✅ 5/5 | ❌ fails cycle 1 (BUG-007) | yes for smartwal; walstore deferred | +| E | Real SIGKILL of blockvolume mid-write → restart with same `--durable-root` → reconnect NVMe → byte-exact readback | ✅ | (not required for smartwal-path G4; walstore run skipped pending BUG-007) | yes — canonical G4 | +| F | Real ENOSPC via size-limited tmpfs durable-root → verify graceful hard-error (no silent success, no corruption) | ✅ | ✅ | yes — canonical PCDD-DURABLE-DISK-FULL | + +dmesg clean on smartwal A–F runs: no `r2t exceeded` / no `-71` / no reconnect. Matrix E smartwal recovery verified byte-exact vs pre-kill payload (SIGKILL = pgrep + `kill -9` on blockvolume pid mid-write; no graceful shutdown path). Matrix F smartwal success signal: server-side `ENOSPC triggered gracefully` message + dd failure + no silent-success write; cleanup SSH fragility tolerated via pattern-based detection at `313dd52`. + +**walstore Matrix D failure**: `BUG-007 — walstore loses written data across umount + remount`. Pre-existing walstore-specific durability bug surfaced by Matrix D; walstore `Read` does not serve uncommitted-but-WAL-persisted writes across same-session fs-layer umount+remount. Non-blocking for T3 because (a) smartwal is documented production default per T3b Addendum A, (b) canonical G4 pass gate is met by smartwal full A–F. walstore is reclassified as non-default fallback until BUG-007 closes. + +**G4 verdict**: ✅ PASS on smartwal; walstore deferred via BUG-007. + +### Outstanding work (post-sign, not blocking) + +- fio sustained-throughput numbers for perf baseline (replaces Go-bench first-light in `t3c-durable-baseline.md` — wire-layer overhead measurement) +- 24h soak (equivalent to BUG-001 post-fix soak pattern; file as inventory item for G21/G22 if needed) +- 2 L2 scenarios deferred to `bugs/inventory/nvme-test-coverage-deferred.md`: + - `t3c-durable-crash-during-sync` — Sync atomicity boundary (V2 CP13 parity) + - `t3c-durable-restart-loop` — multi-cycle recover drift detection (V2 t0-hosting-smoke parity) + + Both were proposed post-sw-commit; shipped 4-scenario set already covers canonical G4 + 3 adjacent. Not blocking G4; inventoried for post-T3-closed follow-up. + +None of these block T3-end sign; they are verification extensions that raise confidence without changing the shipped contract. + +--- + +## §F File manifest + +All T3 source + test + doc file touches across the 4 commits. NEW = file added this phase; MOD = pre-existing file modified. + +### Production code — `seaweed_block` repo + +| Path | T3a (`0e1595c`) | T3b (`72d0d40`) | T3b+ (`5c33460`) | T3c (`829c6a9`) | +|---|:-:|:-:|:-:|:-:| +| `cmd/blockvolume/main.go` | | MOD | | | +| `core/frontend/types.go` | MOD | | | | +| `core/frontend/memback/backend.go` | MOD | | | | +| `core/frontend/testback/testback.go` | MOD | | | | +| `core/frontend/durable/storage_adapter.go` | **NEW** | | | | +| `core/frontend/durable/provider.go` | | **NEW** | | | +| `core/frontend/durable/recovery.go` | | **NEW** | | | +| `core/frontend/iscsi/scsi.go` | | MOD | | | +| `core/frontend/iscsi/errors.go` | | MOD | | | +| `core/frontend/nvme/io.go` | | MOD | | | +| `core/frontend/nvme/session.go` | | MOD | | | +| `core/storage/superblock.go` | MOD | | | | +| `core/storage/walstore.go` | MOD | | | | +| `core/storage/smartwal/superblock.go` | MOD | | | | + +### BUG-005 fix — `seaweed_block` repo (commit `42b045a`) + +| Path | Change | Note | +|---|---|---| +| `core/frontend/nvme/target.go` | MOD (−1) | Removed `defer backend.Close()` in handleConn | +| `core/frontend/iscsi/session.go` | MOD (−3) | Removed `s.backend.Close()` in Session.close | +| `core/frontend/types.go` | MOD (+11) | Backend interface godoc — Provider ownership clause | +| `core/frontend/durable/bug005_session_reuse_test.go` | NEW (~120 LOC) | Matrix regression: `TestT3_Bug005_ProviderCachedBackend_ReusableAcrossSessions` + `TestT3_Bug005_ExplicitBackendClose_IsAllowed` | + +### m01 infra — `seaweed_block` repo (commit `27486db`) + +| Path | Change | Note | +|---|---|---| +| `scripts/iterate-m01-nvme.sh` | MOD | Pre-cleanup (pkill stale blockmaster/volume) + dmesg clear before each run; required for BUG-005 re-verify to be deterministic | + +### Test files — `seaweed_block` repo + +| Path | Commit | +|---|:-:| +| `core/frontend/durable/storage_adapter_test.go` | T3a NEW | +| `core/frontend/iscsi/t2_ckpt10_hardening_test.go` | T3a MOD (trivial Backend impls) | +| `core/frontend/nvme/t2_v2port_nvme_no_retry_test.go` | T3a MOD (trivial Backend impls) | +| `core/frontend/nvme/t2_v2port_nvme_write_chunked_r2t_test.go` | T3a MOD (trivial Backend impls) | +| `core/frontend/durable/provider_test.go` | T3b NEW | +| `core/frontend/durable/recovery_test.go` | T3b NEW | +| `core/frontend/iscsi/t3b_sync_cache_test.go` | T3b NEW | +| `core/frontend/nvme/t3b_flush_test.go` | T3b NEW | +| `core/frontend/durable/integration_iscsi_test.go` | T3b+ NEW | +| `core/frontend/durable/integration_nvme_test.go` | T3b+ NEW | +| `core/frontend/durable/scenario_crash_recovery_test.go` | T3c NEW | +| `core/frontend/durable/scenario_wal_replay_test.go` | T3c NEW | +| `core/frontend/durable/scenario_fsync_boundary_test.go` | T3c NEW | +| `core/frontend/durable/scenario_disk_fill_test.go` | T3c NEW | +| `core/frontend/durable/perf_baseline_test.go` | T3c NEW | + +### Scenario shape docs + perf artifact — `seaweed_block/testrunner/` + +| Path | Commit | +|---|:-:| +| `testrunner/scenarios/testdata/t3c-durable-crash-recovery.yaml` | T3c NEW | +| `testrunner/scenarios/testdata/t3c-durable-wal-replay.yaml` | T3c NEW | +| `testrunner/scenarios/testdata/t3c-durable-fsync-boundary.yaml` | T3c NEW | +| `testrunner/scenarios/testdata/t3c-durable-disk-fill.yaml` | T3c NEW | +| `testrunner/perf/t3c-durable-baseline.md` | T3c NEW | + +### Design docs — `seaweedfs/sw-block/design/` (this repo) + +| Path | Status | +|---|---| +| `v3-phase-15-t3-port-plan-sketch.md` | rev-2.1 three-signed (pre-T3) | +| `v3-phase-15-t3-port-audit.md` | T3.0 QA-signed (+ Addendum A) | +| `v3-phase-15-t3a-mini-plan.md` | CLOSED | +| `v3-phase-15-t3b-mini-plan.md` | CLOSED | +| `v3-phase-15-t3c-mini-plan.md` | CLOSED (this batch) | +| `v3-phase-15-t3-closure-report.md` | **this doc** — awaiting 3-sign | + +### Totals + +- Production files: **14 touched** (8 NEW + 6 MOD) +- Test files: **15 touched** (11 NEW + 4 MOD) +- Scenario YAML + perf doc: **5 NEW** +- Design docs: **6** (5 batch docs + this closure report) + +### Not in T3 scope (explicitly) + +- `scripts/iterate-m01-nvme.sh` — QA-owned m01 adaptation, committed separately by QA +- `core/frontend/durable/t3b_qa_l1_addendum_test.go` — QA-authored addendum test, committed separately by QA +- `cmd/blockmaster/*` — authority daemon untouched (correct per §3.2 boundary) +- `core/authority/*`, `core/adapter/*`, `core/engine/*` — control-plane untouched (boundary guard verifies) +- All V2 `weed/storage/blockvol/*` — audit reference only; nothing ported from there since V3 already had the existing port (see §C) + +--- + +## §G Signed-scope delta vs T3-start sketch + +T3-start sketch rev-2.1 (three-signed 2026-04-22) set a specific scope. T3-end delivery diverges in 3 documented places. Each divergence was LOCKED during an earlier mid-T QA single-sign per §8C.2 (not a silent mid-track change); this section surfaces the trail so a reviewer can reconcile sketch-promise vs closure-truth without spelunking. + +### §G.1 `write_gate.go` — promised PORT, delivered NEVER + +**T3-start sketch §3.1 row 6**: `weed/storage/blockvol/write_gate.go` — M — "Port — admission gate tied to fencing". + +**T3-end actual**: Not ported. File classified NEVER in T3.0 audit §3.7 (per risk-flag resolution). Fencing is adapter-side via `INV-FRONTEND-002.*` per-I/O lineage check; V3 storage has no `writeGate` function (grep zero matches) and no `ErrNotPrimary/ErrEpochStale/ErrLeaseExpired` leaks. + +**Where LOCKED**: T3.0 audit §3.7 + §10.3.7, QA single-signed 2026-04-22. Sw sanity comment in §10.7-B explicitly validated this as a stronger §3.2 outcome than the sketch proposed. + +**Net**: sketch's 7-file port list effectively shrinks to 6 files + 1 NEVER. Sketch row #6 retired; no new canonical port replaces it. + +### §G.2 `INV-DURABLE-EPOCH-PERSISTED-001` — promised QUEUE, not queued + +**T3-start sketch §7**: 5 A-tier invariants including `INV-DURABLE-EPOCH-PERSISTED-001` — "Epoch persists through superblock; restart reloads current epoch, rejects stale-lineage I/O". + +**T3-end actual**: Row NOT queued. V3 `core/storage/superblock.go` has no `Epoch` field (audit §10.3.1 GAP + §10.5 Option 3 LOCKED). Authority-drift guard is per-I/O adapter fence check (`INV-FRONTEND-002.EPOCH/.EV/.REPLICA/.HEALTHY` inherited from T1, verified ACTIVE under durable backend by T3a fence-check tests). + +**Where LOCKED**: Audit §10.5 Option 3, QA single-signed 2026-04-22. Architect-line §10.7 could be extended but the decision was mid-T QA autonomy per §8C.2. + +**Net**: sketch's 5-row A-tier queue becomes different shape: `INV-DURABLE-OPGATE-001` (T3a) + `INV-DURABLE-IMPL-IDENTITY-001` (T3a, Addendum A) + `INV-DURABLE-PROVIDER-SELECT-001` (T3b) + `INV-DURABLE-RECOVERY-READSIDE-001` (T3b) + `INV-DURABLE-SYNC-WIRED-001` (T3b) replace the epoch-persisted row. Plus 4 T3c rows. **9 total rows queued** (vs 5 sketched) — a net +4 expansion covering adapter, provider, recovery, sync-wire, impl-identity invariants not enumerated in the sketch. + +### §G.3 crash-recovery + disk-full invariant names downgraded + +**T3-start sketch §7 + T3c mini-plan §1.1**: +- `INV-DURABLE-001` = "Acknowledged Write survives **kill + restart**" +- `PCDD-DURABLE-DISK-FULL-001` = "**Backing store exhaustion** returns hard error" + +**T3-end actual**: both scenarios implemented as `Close() + reopen` (clean restart) and fixed-geometry-out-of-range writes, NOT real `SIGKILL` + real `ENOSPC`. PM + architect review 2026-04-22 caught both overclaims. + +**Resolution (FINAL, 2026-04-22)**: Path B executed end-to-end. BUG-005 fixed in `42b045a`. Matrix F cleanup robustness + T3-DEF-6 addendum test committed in `313dd52`. m01 smartwal **Matrix A–F all PASS** including Matrix E real SIGKILL with byte-exact readback and Matrix F real ENOSPC graceful-hard-error. Canonical ledger names retained: `INV-DURABLE-001` (kill + restart byte-exact) + `PCDD-DURABLE-DISK-FULL-001` (real backing-store exhaustion). Both scoped to smartwal (production default) per §B non-claim and §E G4 verdict; walstore side deferred via BUG-007. + +**Where LOCKED**: THIS REVIEW CYCLE. Pre-sign, not mid-T. Path B elected by PM 2026-04-22; executed + verified same day. + +--- + +## §H Review response log (pre-sign) + +Pre-sign PM + architect review 2026-04-22 identified 6 blockers + 1 low issue. QA retracted premature sign row and executed 2-phase fix: + +### Phase 1 — doc integrity (landed 2026-04-22) + +| # | Finding | Action | +|---|---|---| +| B3 (Architect-H1) | Audit doc said DRAFT; closure said QA-signed | Audit header `LOCKED + QA single-signed 2026-04-22`; §8 sign-off table filled | +| B4 (Architect-H2) | Closure didn't reconcile sketch-scope delta | This §G added with 3 explicit divergences | +| B5 (PM-M2 + Architect-M1) | ``, stale perf-baseline wording | §A T3c commit = `829c6a9`; §B row 9 updated with actual numbers; perf doc removed `_TBD_` claim | +| B6 (PM-M1) | Uncommitted QA artifacts | Script + L1 addendum test committed in the sign cycle | +| B7 (PM-L1) | Perf doc "_TBD_ placeholder numbers are acceptable" wording | Removed; replaced with "first-light numbers 2026-04-22 ... m01 fio post-sign" wording | + +### Phase 2 — real G4 evidence (path B elected) + +| # | Finding | Action | +|---|---|---| +| B1 (PM-H1) | crash-recovery used `Close()` not SIGKILL — canonical G4 requires kill/restart | m01 Matrix E added: SIGKILL volume process mid-write → restart with same `--durable-root` → reconnect NVMe → verify data byte-exact. Real Linux kernel initiator over ungraceful unmount path. | +| B2 (PM-H2) | disk-full tested out-of-range writes, not real ENOSPC | m01 Matrix F added: `--durable-root` backed by size-limited tmpfs → fill tmpfs → attempt write → verify graceful hard error (no silent success, no hang, no corruption of in-range data). | + +Matrix E + F matrixed per Addendum A #1 intent. Final result (see §E): smartwal A–F ✅; walstore A–C + F ✅, Matrix D ❌ (pre-existing walstore bug, filed as BUG-007); Matrix E skipped for walstore pending BUG-007 (no value running SIGKILL recovery when same-session remount already fails). + +Post-Phase-2 artifacts (BUG-005 fix `42b045a`, Matrix F robustness + T3-DEF-6 test `313dd52`) feed §E final verification table + unlock QA re-sign. QA re-signed 2026-04-22 (see §E sign table). + +### Phase 3 — BUG-005 discovery during Path B execution (landed 2026-04-22) + +Path B Matrix A cycle 2 surfaced a latent lifecycle bug: + +| Item | Detail | +|---|---| +| Symptom | m01 kernel dmesg `nvme1n1: I/O Cmd(0x2) @ LBA 0, I/O Error (sct 0x3 / sc 0x2) DNR` on reconnect cycle 2 | +| Root cause | Session-layer `defer backend.Close()` (nvme target) + `s.backend.Close()` (iscsi session close) flipped DurableProvider's cached Backend `closed=true`; next session's Open returned same closed handle; all I/O → `ErrBackendClosed` → NVMe maps to SCT=3 SC=2 (ANA Inaccessible) | +| V2 comparison | V2 `Subsystem.Dev` lifecycle is registry-owned (AddVolume/RemoveVolume), NOT session-owned. V2's `Controller.shutdown()` explicitly does NOT touch `Subsystem.Dev`. V3 introduced `frontend.Backend.Close()` abstraction without carrying V2's implicit ownership contract into interface godoc | +| Fix | Removed 2 lines (nvme target handleConn defer; iscsi session close); added Provider-ownership clause to Backend interface godoc; regression test `TestT3_Bug005_ProviderCachedBackend_ReusableAcrossSessions` matrix both impls | +| Commit | `42b045a` | +| New ledger row | `INV-DURABLE-SESSION-LIFECYCLE-001` (queued ACTIVE post-three-sign) | +| Discipline citation | `feedback_porting_discipline.md` third citation added: "new-abstraction ownership drift" as distinct failure mode from BUG-001's "V2-file-simplification drift" | + +**Why existing tests missed it**: all T3a/T3b/T3c tests are either single-session (scenarios) or direct-handler (integration tests that bypass `target.handleConn`). Only "real target × session 1 disconnect × session 2 open" exercises the cross-session cache interaction. Same L1-passes-L2-fails pattern as BUG-001 — L1 tests at isolated layers don't catch multi-layer lifecycle drift. + +### Phase 4 — T3-DEF-5/6/7 retrospective (landed 2026-04-22) + +Tier A retrofill of the V2/V3 Contract Bridge Catalogue surfaced 3 latent gaps in T3 scope. All three processed pre-sign so T3 closes with zero open inventory rows tied to its scope. + +| Gap | Nature | Outcome | +|---|---|---| +| T3-DEF-5 | iSCSI VPD Model/Vendor/Firmware-Rev constants reviewed during Addendum A but never catalogued; risk of silent drift when multi-tenant/ANA requirements arrive | Catalogued as C4-ISCSI-VPD-STATIC-FIELDS-HARDCODED (PORT-AS-IS LOCKED explicit) in §2.2.13. No regression test needed — discipline row; reclassification trigger documented. **Closes as catalogue row.** | +| T3-DEF-6 | NVMe Session cleanup-on-close tested only implicitly via goroutine-leak count; no explicit contract row | L1 addendum test `t3_qa_session_cleanup_addendum_test.go` landed at `seaweed_block@313dd52` — **smoke + goroutine-leak guard**, not full state-release introspection. `TestT3_NVMe_Session_CleanupOnClose_NoLeaksAcrossCycles` (20 open/close cycles + Identify admin cmd; fails if `NumGoroutine` delta > 30 after drain) + `TestT3_NVMe_Session_CleanupOnClose_CNTLIDAllocator` (8 cycles, CNTLID strictly monotonic + no zero-sentinel + allocator not wedged). Both PASS 0.78s. Test-only introspection of Target.ctrls / AER slot / KATO-stored-ms is **not** exercised — follow-up if/when that introspection surface is added. New catalogue row C5-NVME-SESSION-STATE-CLEANUP-ON-CLOSE records the contract; pin strength is "smoke" today. **Closes T3-DEF-6 as L1 smoke + catalogue row; full introspection pin queued as post-T3 inventory if BUG-006 timer work touches this area.** | +| T3-DEF-7 | KATO timer store-only per BUG-003 Discovery Bridge scope; m01 Matrix D kernel dmesg `long keepalive RTT (2522123190 ms)` proves contract violated today, not merely deferred-safe | Filed as `bugs/006_nvme_kato_timer_not_enforced.md` (Medium severity, target G21 / post-T3, ~50-100 LOC fix sketch + 4-test coverage plan). Catalogue row C1-NVME-SESSION-KATO reclassified **VIOLATED** with BUG-006 anchor + m01 evidence citation. **Closes as formal bug with track assignment.** | + +### Phase 5 — Walstore BUG-007 (non-blocking, filed 2026-04-22) + +m01 re-verify against both impls surfaced a pre-existing walstore-specific durability bug: cross-session umount+remount (via fs layer, no process restart) loses clean-sync'd data. smartwal passes the same Matrix D cycle. Filed as `bugs/007_walstore_umount_remount_data_loss.md`; **non-blocking for T3** — smartwal is production default per T3b, and canonical G4 gate is satisfied by smartwal full A-F green. Walstore-side evidence for ledger row `INV-DURABLE-001` deferred to BUG-007 close. + +--- + +## §I Contract Bridge Catalogue integration + +T3 is the first track where the V2→V3 port's **entity bridging** discipline is retrofilled into documentation. Triggered by the BUG-001 + BUG-005 + Addendum A pattern being recognized as systemic (not one-off). + +**Reference**: `sw-block/design/v2-v3-contract-bridge-catalogue.md` + +- §2 storage-layer entity bridge + contract catalogue has been retrofilled with T3 scope: + - Subsystem (V2) → DurableProvider + LogicalStorage (V3) — merged bridge + - Controller (V2) → nvme.Session + StorageBackend (V3) — split bridge + - BlockVol (V2) → LogicalStorage + adapter fence (V3) — split bridge + - flusher / dirty_map / wal_* (V2) → 1:1 with V3 equivalents (per T3.0 §10.3 walk-through) + - New V3 entities without V2 analog: ProjectionView, StorageBackend operational gate, frontend.Backend interface itself + +- §2.3 maps all three known drifts to catalogue entity rows: + - BUG-001 → Controller C3 (rxLoop serial) + C6 (R2T-before-H2CData ordering) + - BUG-005 → Controller C1 (session ≠ device lifecycle); Subsystem C1 (Dev lifecycle is registry-owned) + - Addendum A iSCSI VPD 0x80 Serial → Subsystem metadata contract (V3 derives from VolumeID, V2 was a hardcoded stub) + +- §8C.8 discipline promotes the catalogue to mandatory pre-T-start artifact for any track involving V2 port. For T4+ (G5 replicated write path), catalogue §3 replication section must be LOCKED before T-start three-sign. + +**T3 close uses the catalogue retroactively** — the 5 V3 model shifts (event model / storage model / authority model / lifecycle / concurrency) in §1.1 of the catalogue explain why several "1:1" port rows in §C of this report diverge from V2 even though nominally unchanged (e.g., GroupCommitter drops OnDegraded + PostSyncCheck — same file name, different contract). + +--- + +## §J Change log + +| Date | Change | Author | +|---|---|---| +| 2026-04-22 | Initial T3 closure report drafted; awaiting three-sign | sw | +| 2026-04-22 | §F file manifest added (commit-by-commit touch record) | sw | +| 2026-04-22 | Pre-sign review: Phase 1 doc integrity + §G sketch-delta + §H review response log | sw | +| 2026-04-22 | BUG-005 discovered during Path B Matrix A cycle 2; fixed in `42b045a`; §A + §D + §G.3 + §H.Phase3 updated | sw | +| 2026-04-22 | §I Contract Bridge Catalogue integration; T3 retrofill reference; §8C.8 discipline citation | sw | +| 2026-04-22 | §F file manifest added (explicit file-by-file touch record across 4 commits) | sw | +| 2026-04-22 | PM + architect pre-sign review identified 6 blockers + 1 low. QA retracted premature sign; executed Phase 1 (doc integrity: §G signed-scope delta, audit sign-off, TBD→commit, perf doc cleanup, artifact commits) + Phase 2 (m01 Matrix E SIGKILL + Matrix F ENOSPC — real G4 evidence) | QA Owner | +| 2026-04-22 | Phase 4 T3-DEF-5/6/7 retrospective closure: DEF-5 catalogued (C4-ISCSI-VPD-STATIC-FIELDS-HARDCODED), DEF-6 landed as `t3_qa_session_cleanup_addendum_test.go` pinning C5, DEF-7 filed as BUG-006 reclassifying C1-NVME-SESSION-KATO as VIOLATED | QA Owner | +| 2026-04-22 | Phase 5 BUG-007 filed: pre-existing walstore umount+remount data loss; non-blocking since smartwal is prod default and satisfies canonical G4 | QA Owner |