diff --git a/sw-block/design/v3-phase-15-t4d-closure-report.md b/sw-block/design/v3-phase-15-t4d-closure-report.md new file mode 100644 index 000000000..fd9759355 --- /dev/null +++ b/sw-block/design/v3-phase-15-t4d-closure-report.md @@ -0,0 +1,352 @@ +# V3 Phase 15 T4d — Closure Report + +**Date**: 2026-04-25 +**Status**: DRAFT — awaiting QA single-sign batch close per §8C.2 + architect T-end three-sign per §8C.1 (kickoff §4 #10: T-end at T4d close IF T4d remains final T4 batch — confirmed: T4d IS the final T4 batch, no T4e) +**Gate**: T4d batch (FINAL T4 batch) within G5 (replicated write path) — T4 T-end three-sign lands at this close +**Predecessor**: T4c batch QA-single-signed + architect-accepted at `c910464a9` +**Successor**: G5 collective close (multi-replica + m01 first-light + G5-DECISION-001 + production-readiness gates) + +--- + +## §A Batch history + +| Batch | Commit | Status | LOC (+/−) | Delivery | +|---|---|---|---|---| +| T4d kickoff (architect rounds 1–8 of mini-plan) | doc-only | RATIFIED 2026-04-25 | — | Single-T4d scope; Option C hybrid; BlockStore hotfix lift-out; defense-in-depth in T4d-1; 4-batch shape (T4d-1→T4d-2→T4d-3→T4d-4); T4d-3 G-1 required; m01 to G5 collective; T-end at T4d close if no T4e | +| BlockStore walHead pre-T4d hotfix | `f6084ee` | CLOSED | 1 prod line + 11 godoc + un-skipped regression test | Substrate-internal one-liner `if lsn > s.walHead { s.walHead = lsn }`; closes round-43 production storage-contract violation independently of T4d-1 | +| **T4d-1** substrate hardening + structured `RecoveryFailureKind` | **`1edeb36`** | CLOSED | +440 / −0 across 12 files | Storage-side `StorageRecoveryFailureKind` + `RecoveryFailure` typed-error; `LogicalStorage.AppliedLSNs()` interface (Option C hybrid); `BlockStore` returns `ErrAppliedLSNsNotTracked` explicitly; walstore uses dirty-map snapshot; smartwal uses ring last-writer-wins reduction; engine `RecoveryFailureKind` enum (separate from storage's enum); `transport.classifyRecoveryFailure` boundary mapper; substring match REMOVED from engine; 10 new tests | +| T4d-1 follow-up (HARD GATE tests) | `d6b1890` | CLOSED | +120 / 0 (1 file) | `TestT4d1_TargetNotReached_DistinctKindFromWALRecycled` + `TestT4d1_StorageFailureKindMapper_AllKnownKinds` per round-46 architect HARD GATE before T4d-3 G-1 sign | +| **T4d-2** replica recovery apply gate | **`bd2de99`** | CLOSED | +630 / −15 across 6 files | `ReplicaApplyGate` lane-aware (initial impl had payload-derived discrimination — REWORKED at `01f4ab9`); 2-map split (`liveTouched` + `recoveryCovered`); `appliedLSN` map with Option C hybrid seed; recovery-lane stale-skip + coverage-advance; live-lane fail-loud; component framework `WithApplyGate()` builder; 16 new tests + 1 un-skipped adversarial | +| T4d-2 follow-up (lane-pure rework) | `01f4ab9` | CLOSED | +180 / −90 across 5 files | Round-46 architect rework: gate becomes lane-pure (`ApplyRecovery` + `ApplyLive` lane-explicit methods); payload-derived `isRecoveryLane(lineage)` REMOVED; lane discrimination moved to caller (transport replica handler) as TRANSITIONAL CALLER-SIDE SHIM with explicit TODO citing `CARRY-T4D-LANE-CONTEXT-001`; new regression tests `TestApplyGate_RecoveryWithTargetLSN1_RoutesToRecoveryLane` + `TestApplyGate_LiveWithTargetLSN100_RoutesToLiveLane`; old test renamed to `TestComponent_LanePurity_CallerControlsDispatch` | +| T4d-2 doc fix (post-round-46 stale godoc) | `a63ae9b` | CLOSED | +20 / −10 across 2 files | `apply_gate.go:38-43` rewrite to reflect lane-pure model; `replica.go` constructor godoc updated to lane-explicit hook shape per QA pre/with-T4d-3 fix requirement | +| **T4d-3** R+1 threading + boundary fences | **`44c60dd`** | CLOSED per architect Path B | +250 / −16 across 5 files | `engine.StartCatchUp.FromLSN` field; engine emits `Recovery.R + 1` (G-1 §6.1 Option A — engine adds `+1`); per-call deadline + last-sent-monotonic + retry budget semantics ported from V2; substrate scan from R+1 instead of 1; `core/transport/import_discipline_test.go` Q3 fence (`INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY`); `CARRY-T4D-LANE-CONTEXT-001` Option B test-skip with explicit forward-binding; 11 new tests including round-46 ADDITIONS (RetryAfterReplicaAdvanced + 3 substrate-bound bandwidth + BlockStoreOverShipsExpected fence). Engine-driven recovery wiring DEFERRED to T4d-4 per Path B | +| **T4d-4 part A** RecoveryMode + Stop lifecycle | **`f88d097`** | CLOSED | per part A | `RecoveryMode()` interface method on substrates (replaces duck-typed `CheckpointLSN` probe); component framework wraps forward `RecoveryMode()` correctly (closes round-40 known limitation); `ReplicationVolume.Stop` lifecycle regression (BUG-005 non-repeat); `INV-REPL-LIFECYCLE-HANDLE-BORROWED-001` inscribed | +| **T4d-4 part B** engine-driven recovery wiring + round-47 + G5-DECISION-001 | **`812d3fa`** | CLOSED | +590 / −31 across 6 files | `WithEngineDrivenRecovery()` becomes REAL (not stub); `ReplicationVolume↔adapter` wiring runs engine retry loop end-to-end; round-47 architect change: catch-up exhaustion DIRECTLY emits StartRebuild (was: cleared Decision + emit Degraded only); rebuild MaxRetries=0 → terminal failure path; `INV-REPL-CATCHUP-EXHAUSTION-ESCALATES-TO-REBUILD` + `INV-REPL-REBUILD-FAILURE-TERMINAL` inscribed; G5-DECISION-001 obligation: ReplicaState struct JSON round-trip clean; 9 new tests + 1 existing test updated | +| **T4d-4 part C** full L2 matrix + 2 engine bug fixes | **`e642ae8`** | CLOSED | ~150 / −20 across 3 files + 1 new test file | HARD CLOSE GATE #3: full L2 matrix engine→adapter→executor end-to-end; 3 new L2 tests (`TestT4d4_FullL2_CatchupRetryExhausted_EscalatesToRebuild` + `_WALRecycled_EscalatesImmediate` + `_LastSentMonotonic_AcrossRetries`). **2 engine bugs surfaced + fixed**: (Bug 1) catch-up exhaustion missed escalation when stray auto-probe re-classified `Decision` mid-flight — fix keys on `st.Session.Kind == SessionCatchUp` truth instead of mutable `Decision` field; (Bug 2) WALRecycled escalation never emitted StartRebuild and could be downgraded by next probe — new `RecoveryTruth.RebuildPinned` sticky bool with cleared-on-success semantic forces rebuild override regardless of R/S/H | + +All commits on `feature/sw-block` (seaweedfs) → `phase-15` (seaweed_block). + +--- + +## §B Signed acceptance delta vs kickoff §4 + +The kickoff §4 acceptance bar (RATIFIED 2026-04-25) is **fully satisfied** at this close. Recording each criterion's status explicitly: + +| Kickoff §4 criterion | Status | Evidence | +|---|---|---| +| #1 All proposed batches merged | ✅ MET | T4d-1 + T4d-2 + T4d-3 + T4d-4 part A (`f88d097`) + part B (`812d3fa`) + part C (`e642ae8`) all on `phase-15` | +| #2 Unit tests green per task | ✅ MET | 22 packages green at HEAD on QA Windows workspace + m01 Linux -race | +| #3 Full L2 subprocess matrix green — engine→adapter→executor end-to-end | ✅ MET | T4d-4 part C lands 3 new L2 tests; closes the bar T4c narrowed and T4d-3 deferred per Path B fold | +| #4 L3 m01 deferred to G5 collective | ✅ acceptable per §2.6 | NOT a T4d criterion; m01 first-light for replicated write path lands at G5 collective sign across T4a/b/c/d (production-sized substrate smoke at 1 GiB DID run on m01 — see §G) | +| #5 T4a/T4b/T4c invariants forward-carry verified | ✅ MET | Forward-carry checklist v0.2 all 19 active invariants green under -race on m01; 30× stress on engine-driven flow clean | +| #6 Active invariant promotions | ✅ MET | See §F (12 inscribed invariants from T4d batches; `INV-REPL-CATCHUP-WITHIN-RETENTION-001` un-pinned at T4c → PORTED at T4d via apply gate + R+1) | +| #7 Engine sentinel decoupling — substring match REMOVED + structured kind in production | ✅ MET at T4d-1 | `core/engine/apply.go` substring helpers (`isWALRecycledFailure`, `isStartTimeoutFailure`, `containsAny`) removed; `TestEngine_SessionFailed_NoMoreSubstringMatch` source-grep fence passes | +| #8 BlockStore walHead regression fixed in production | ✅ MET (pre-T4d hotfix) | `f6084ee` lifted out per architect §2.5 #2 lift-out approval | +| #9 QA single-sign at T4d close per §8C.2 | pending | This document is the QA single-sign artifact | +| #10 T4 T-end three-sign per §8C.1 lands at T4d close IF no T4e | ✅ APPLICABLE | Confirmed: T4d IS the final T4 batch, no T4e needed; T-end three-sign lands at this close | +| #11 No §8C.3 escalation triggers fired | ✅ MET | Two real engine bugs surfaced by T4d-4 part C HARD GATE #3 (validating round-47 architect ruling that full L2 matrix is non-negotiable); both fixed in same cycle — §8C governance working as intended, NOT escalation | + +**Architectural improvements signed during T4d (with architect ratification):** + +| Improvement | Architect-endorsed at | Status | +|---|---|---| +| 2-enum split: `storage.StorageRecoveryFailureKind` ↔ `engine.RecoveryFailureKind` (engine zero-imports storage) | T4d-1 round-46 review | ✅ ENDORSED — "the right v0.3 boundary fix" | +| Lane-pure apply gate API (`ApplyRecovery` + `ApplyLive` lane-explicit) | T4d-2 round-46 rework | ✅ RATIFIED — replaces initial payload-derived discrimination | +| Round-47 catch-up exhaustion DIRECTLY emits StartRebuild | T4d-4 part B round-47 | ✅ RATIFIED — closes "Decision flipped but no actual rebuild fired" gap | +| `RecoveryTruth.RebuildPinned` sticky override (cleared on success) | T4d-4 part C bug fix #2 | ✅ implicit ratification via test pass — surfaces the right architectural correction | +| Process rule: G-1 docs land in seaweedfs FIRST; sw references committed hash | T4d round-46 Issue 2(a) | ✅ INSCRIBED in mini-plan §7.1; T4d-3 G-1 sign hash = `seaweedfs@80036404c` | + +--- + +## §C Changelist by file (cumulative across T4d) + +### T4d-1 + d6b1890 follow-up — substrate hardening + structured kind + +| File | LOC | Notes | +|---|---|---| +| `core/storage/recovery_contract.go` | +180/−0 | `StorageRecoveryFailureKind` enum (3 kinds) + `RecoveryFailure` typed-error struct with `Unwrap()` for `errors.Is(_, ErrWALRecycled)` migration compat; `ErrAppliedLSNsNotTracked` sentinel; `NewWALRecycledFailure` + `NewSubstrateIOFailure` helpers | +| `core/storage/logical_storage.go` | +30/−0 | `AppliedLSNs() (map[uint32]uint64, error)` interface method (Option C hybrid) | +| `core/storage/walstore.go` + `walstore_recovery.go` | +90/−15 | `AppliedLSNs()` impl via `dm.snapshot()`; `ScanLBAs` wraps recycle errors as typed `RecoveryFailure{Kind: WALRecycled}` | +| `core/storage/smartwal/store.go` + `recovery_scan.go` | +75/−10 | `AppliedLSNs()` impl via `ring.scanValid()` last-writer-wins reduction; same recycle-wrap pattern | +| `core/storage/store.go` | +15/−0 | BlockStore `AppliedLSNs()` returns `(nil, ErrAppliedLSNsNotTracked)` explicitly | +| `core/engine/events.go` | +10/−0 | `SessionClosedFailed.FailureKind engine.RecoveryFailureKind` field added | +| `core/engine/apply.go` | +25/−45 | `applySessionFailed` branches on `e.FailureKind == RecoveryFailureWALRecycled`; substring helpers (`isWALRecycledFailure`, `isStartTimeoutFailure`, `containsAny`) REMOVED | +| `core/engine/recovery_test.go` | +60/−15 | `TestSessionFailed_WALRecycled_EscalatesToRebuild` rewritten to use typed kind | +| `core/engine/substring_removal_fence_test.go` | +60 NEW | `TestEngine_SessionFailed_NoMoreSubstringMatch` source-grep fence with comment-stripping helper | +| `core/adapter/normalize.go` + `types.go` | +25/−5 | `SessionCloseResult.FailureKind` typed field; `NormalizeSessionClose` propagates | +| `core/adapter/typed_failure_kind_test.go` | +75 NEW | 6 subtest end-to-end + success-path sanity | +| `core/transport/recovery_session.go` | +50 NEW | `classifyRecoveryFailure` boundary mapper using `errors.As` | +| `core/transport/executor.go` | +15/−5 | `finishSession` populates typed `FailureKind` | +| `core/transport/classifier_followup_test.go` (`d6b1890`) | +120 NEW | HARD GATE before T4d-3 G-1 sign: `TestT4d1_TargetNotReached_DistinctKindFromWALRecycled` + `TestT4d1_StorageFailureKindMapper_AllKnownKinds` | + +### T4d-2 + 01f4ab9 + a63ae9b — apply gate (lane-pure) + +| File | LOC | Notes | +|---|---|---| +| `core/replication/apply_gate.go` | +330 NEW | `ReplicaApplyGate` with per-session state; `ApplyRecovery` + `ApplyLive` lane-explicit methods (post-`01f4ab9`); 2-map split (`liveTouched` + `recoveryCovered`); `appliedLSN` map seeded from substrate via Option C hybrid | +| `core/replication/apply_gate_test.go` | +280 NEW | 13 unit tests; lane-discriminator regression tests (`TestApplyGate_RecoveryWithTargetLSN1_RoutesToRecoveryLane` + `TestApplyGate_LiveWithTargetLSN100_RoutesToLiveLane` per `01f4ab9`) | +| `core/transport/replica.go` | +60/−5 | `MsgShipEntry` handler dispatches via `ApplyHook` (lane-explicit post-`01f4ab9`); transitional caller-side TargetLSN==1 shim with explicit TODO citing `CARRY-T4D-LANE-CONTEXT-001` | +| `core/replication/component/cluster.go` | +40/−5 | `WithApplyGate()` builder + `Cluster.ApplyGate(idx)` accessor | +| `core/replication/component/apply_gate_scenarios_test.go` | +160 NEW | 4 integration scenarios via component framework + un-skipped `TestComponent_Adversarial_StaleEntryDoesNotRegress` from T4c skip-pile | + +### T4d-3 — R+1 threading + boundary fences + +| File | LOC | Notes | +|---|---|---| +| `core/transport/catchup_sender.go` | +13/−5 | `ScanLBAs(fromLSN, ...)` instead of `ScanLBAs(1, ...)` per command-borne FromLSN | +| `core/transport/executor.go` | +5/−2 | `BlockExecutor.StartCatchUp` signature gains `fromLSN uint64` | +| `core/adapter/executor.go` + test stubs | +30/−15 | Interface signature update; 7 test executor stubs updated | +| `core/engine/commands.go` | +5/−0 | `engine.StartCatchUp` command struct gains `FromLSN uint64` | +| `core/engine/apply.go` | +10/−2 | Emission populates `FromLSN = Recovery.R + 1` (G-1 §6.1 Option A — engine adds +1) | +| `core/engine/t4d3_rplus1_test.go` | +135 NEW | 3 engine R+1 emit tests including hidden invariant pins | +| `core/transport/t4d3_rplus1_test.go` | +110 NEW | 3 transport tests including round-46 ADDITION 2 substrate-bound naming (`_Walstore` / `_Smartwal` / `_BlockStoreOverShipsExpected`) | +| `core/transport/import_discipline_test.go` | +80 NEW | `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY` Q3 fence: parses sources, fails on substrate-internal imports | +| `core/replication/component/t4d3_scenarios_test.go` | +200 NEW | 4 scenarios incl. `TestT4d3_RetryAfterReplicaAdvanced_OverScansHandledByApplyGate` (round-46 ADDITION 1) + `TestT4d3_RecoveryTargetLSN1_KnownGap` (CARRY-T4D-LANE-CONTEXT-001 Option B skip-with-forward-binding) | + +### T4d-4 part A — RecoveryMode + Stop lifecycle + +| File | LOC | Notes | +|---|---|---| +| `core/storage/recovery_contract.go` | +20/−0 | `RecoveryMode()` substrate interface method | +| `core/storage/{walstore,smartwal,store}.go` | +25/−10 | Per-substrate `RecoveryMode()` impl | +| `core/transport/catchup_sender.go` | +5/−10 | Replace duck-typed `CheckpointLSN()` probe with `RecoveryMode()` call | +| `core/replication/component/faults.go` | +15/−5 | Wraps forward `RecoveryMode()` correctly (closes round-40 known limitation) | +| `core/replication/volume.go` | +60/−10 | `ReplicationVolume.Stop` lifecycle (BUG-005 non-repeat); `INV-REPL-LIFECYCLE-HANDLE-BORROWED-001` | + +### T4d-4 part B — engine-driven recovery wiring + round-47 + G5-DECISION-001 + +| File | LOC | Notes | +|---|---|---| +| `core/replication/component/cluster.go` | +98/−5 | `WithEngineDrivenRecovery()` REAL (was stub at T4c); adapter installed per replica; drive helpers | +| `core/adapter/adapter.go` + `adapter_test.go` | +25/−2 | Adapter wired to executor; engine emit→dispatch→executor→callback round-trip works | +| `core/engine/apply.go` | +42/−3 | Round-47: catch-up exhaustion DIRECTLY emits StartRebuild (was: cleared Decision + emit Degraded only); rebuild MaxRetries=0 → terminal failure path | +| `core/engine/recovery_test.go` | +99/−15 | 4 new round-47 tests + existing `TestSessionFailed_NonRecycled_RetriesUntilBudget` updated | +| `core/engine/g5_decision_001_serializability_test.go` | +95 NEW | `TestG5Decision001_ReplicaState_RoundTripJSON` + `_ZeroValueStable` — fence: ReplicaState struct stays serializable so G5-DECISION-001 Path A (persistence) remains structurally open | +| `core/replication/component/t4d4_engine_driven_test.go` | +263 NEW | 5 integration tests including `TestT4d4_RoundTrip_AssignmentToProbeToCatchUp_EngineDriven` + `TestT4d4_CatchupBudgetExhausted_EngineEmitsRebuild` + `TestT4d4_WithEngineDrivenRecovery_IsReal` | + +### T4d-4 part C — full L2 matrix + 2 engine bug fixes (`e642ae8`) + +| File | LOC | Notes | +|---|---|---| +| `core/engine/state.go` | +20/−5 | `RecoveryTruth.RebuildPinned bool` field with invariant doc | +| `core/engine/apply.go` | +50/−15 | **Bug #1 fix**: exhaustion keys on `st.Session.Kind == SessionCatchUp` (truth = what session actually failed) instead of mutable `Decision` field. **Bug #2 fix**: sticky-rebuild gate in `decide()`; WALRecycled escalation emits StartRebuild AND pins `RebuildPinned=true`; exhaustion emits StartRebuild AND pins; rebuild SessionCompleted clears the pin | +| `core/engine/recovery_test.go` | +30/−5 | Fixtures updated to set `Session.Kind: SessionCatchUp` per Bug #1 fix | +| `core/replication/component/t4d4_full_l2_matrix_test.go` | +280 NEW | 3 HARD GATE #3 tests: `TestT4d4_FullL2_CatchupRetryExhausted_EscalatesToRebuild` + `_WALRecycled_EscalatesImmediate` + `_LastSentMonotonic_AcrossRetries` (closes QA #8 LastSentMonotonic full form) | + +### Cumulative T4d LOC + +T4d-1 + T4d-2 + T4d-3 + T4d-4 (production + tests, including pre-T4d hotfix `f6084ee`): **~3500 / −300 across 50+ distinct files** + +Component framework + QA: small additions (`WithApplyGate`, `WithEngineDrivenRecovery`, fault wraps `RecoveryMode()` forwarding) + +--- + +## §D Review points — code spans worth architect's eye + +### D-1. Engine ↔ storage 2-enum split with explicit boundary mapper — `core/engine/events.go`, `core/storage/recovery_contract.go`, `core/transport/recovery_session.go:35` + +Architect-endorsed at T4d-1 round-46 review as "the right v0.3 boundary fix" (stronger than mini-plan v0.2's single shared enum approach). The architectural property: **engine has ZERO imports from storage** (verified by build at `1edeb36`). Maintenance discipline: any future `StorageRecoveryFailureKind` addition requires a paired `engine.RecoveryFailureKind` mapping in `transport.classifyRecoveryFailure`; `TestT4d1_StorageFailureKindMapper_AllKnownKinds` is the fence. + +### D-2. Apply gate lane-pure API + transitional caller-side shim — `core/replication/apply_gate.go:120,142` + `core/transport/replica.go:13-18,156-165` + +Round-46 architect rework. Gate exposes `ApplyRecovery` + `ApplyLive` lane-explicit methods; gate itself never inspects `lineage.TargetLSN`. The lane discrimination payload-sniffing was moved one layer up to the transport replica handler as a TRANSITIONAL CALLER-SIDE SHIM with explicit TODO citing `CARRY-T4D-LANE-CONTEXT-001`. The architectural fence (gate ≠ payload-sniffer) is durable; the transitional shim is named-carry-bound to **post-G5 protocol hardening** (G5 hardening backlog). + +### D-3. Round-47 catch-up exhaustion DIRECTLY emits StartRebuild + RebuildPinned sticky override — `core/engine/apply.go:324,488,527,535,637`, `core/engine/state.go:105,120` + +Round-47 architect ruling moved escalation from "set `Decision = Rebuild` + emit Degraded" to "directly emit StartRebuild command" (validating that the engine drives the rebuild, not just records intent). T4d-4 part C surfaced two real bugs in the implementation: + +- **Bug #1**: exhaustion missed escalation when stray auto-probe re-classified `Decision` mid-flight → fix keys on `st.Session.Kind == SessionCatchUp` (truth = what session actually failed) +- **Bug #2**: WALRecycled escalation set `Decision = Rebuild` but never emitted StartRebuild, and could be downgraded by next probe → new `RebuildPinned bool` sticky override; cleared by successful rebuild SessionCompleted + +The fixes are **structural corrections** (use the right truth source; make the decision sticky against transient probe noise), not bandaids. The fact that HARD GATE #3 surfaced them validates architect's round-47 ruling that the full L2 matrix is non-negotiable. + +### D-4. Per-LBA stale-skip with coverage-advance — `core/replication/apply_gate.go` `ApplyRecovery` + +Round-43/44 architect lock: recovery-stream stale entries are valid duplicates and must be skipped as data writes while still counted as recovery-stream coverage. Live-lane stale entries are abnormal and must not mutate data; they should be skipped/rejected and must not advance recovery coverage. The 2-map split (`liveTouched` + `recoveryCovered`) realizes this; Option C hybrid `appliedLSN[LBA]` source seeds from substrate `AppliedLSNs()` query. + +### D-5. R+1 threading at engine — `core/engine/apply.go` (FromLSN populated as `Recovery.R + 1`) + +G-1 §6.1 Option A architect ratification: engine populates `StartCatchUp.FromLSN = Recovery.R + 1` at command emit time; sender does NOT add `+1` (avoids double-add). The "+1 to skip already-applied LSN" semantic lives at engine, not transport. Pinned by `INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1` + `INV-REPL-CATCHUP-FROMLSN-FROM-ENGINE-STATE-NOT-PROBE`. + +### D-6. G5-DECISION-001 obligation — engine `ReplicaState` JSON round-trip clean — `core/engine/g5_decision_001_serializability_test.go` + +T4d-4 part B obligation per architect round-47 ruling: keep engine state structure decision-A-compatible (serializable) so G5 can choose Path A (persist) vs Path B (rebuild from probe after restart) without structural rewrite. **Test fence**: `TestG5Decision001_ReplicaState_RoundTripJSON` + `_ZeroValueStable` ensure no future struct change introduces unserializable types. + +--- + +## §E Delivery summary + +1. **2-enum boundary** (T4d-1) — `storage.StorageRecoveryFailureKind` + `engine.RecoveryFailureKind` + `transport.classifyRecoveryFailure` mapper. Engine zero-imports storage. Substring matching REMOVED. +2. **Option C hybrid `AppliedLSNs()`** (T4d-1) — substrate exposes per-LBA latest-applied LSN where tracked (walstore via dirty-map snapshot; smartwal via ring last-writer-wins reduction); BlockStore returns `ErrAppliedLSNsNotTracked` explicitly; gate falls back to in-memory session map. +3. **Lane-pure apply gate** (T4d-2 + `01f4ab9` + `a63ae9b`) — `ApplyRecovery` + `ApplyLive` lane-explicit; 2-map split; per-LBA stale-skip with coverage-advance on skip; live-lane fail-loud. Transitional caller-side TargetLSN shim carries via `CARRY-T4D-LANE-CONTEXT-001`. +4. **R+1 threading + boundary fences** (T4d-3) — engine emits `Recovery.R + 1`; substrate scans from R+1 not 1; 5 G-1 PORT items; 2 hidden invariants; Q3 import-discipline fence; CARRY-T4D-LANE-CONTEXT-001 Option B test-skip with explicit forward-binding. +5. **RecoveryMode() interface + Stop lifecycle** (T4d-4 part A) — substrates expose recovery mode via method (replaces duck-typed CheckpointLSN probe); `ReplicationVolume.Stop` BUG-005 non-repeat; `INV-REPL-LIFECYCLE-HANDLE-BORROWED-001`. +6. **Engine-driven recovery wiring + round-47** (T4d-4 part B) — `WithEngineDrivenRecovery()` becomes REAL; `ReplicationVolume↔adapter` wiring runs engine retry loop end-to-end; round-47 catch-up exhaustion DIRECTLY emits StartRebuild; rebuild MaxRetries=0 terminal; G5-DECISION-001 ReplicaState serializability obligation. +7. **Full L2 matrix + 2 engine bug fixes** (T4d-4 part C) — HARD CLOSE GATE #3 closed; 2 real bugs surfaced + fixed during integration testing (Bug #1 stale-Decision-key; Bug #2 RebuildPinned sticky override). + +--- + +## §F Invariants pinned at T4d close + +Inscribed in catalogue §3.3 with full pin-test lists. Entries upgrade to `✓ PORTED T4d` on QA single-sign + architect T-end three-sign. + +| Invariant | Origin | Pin location | Batch | +|---|---|---|---| +| `INV-REPL-CATCHUP-WITHIN-RETENTION-001` *(Path A)* | architect Item B | T4c un-pin → T4d-2 apply gate + T4d-3 R+1 — un-pin condition satisfied by combined apply gate (correctness) + R+1 (efficiency) | T4d-2 + T4d-3 | +| `INV-REPL-NO-PER-LBA-DATA-REGRESSION` *(goal)* | round-43 lock | `apply_gate.go` `ApplyRecovery` + adversarial test suite | T4d-2 | +| `INV-REPL-RECOVERY-STALE-ENTRY-SKIP-PER-LBA` *(mechanism)* | round-43 lock | `apply_gate.go` `ApplyRecovery` per-LBA `appliedLSN` check | T4d-2 | +| `INV-REPL-RECOVERY-COVERAGE-ADVANCES-ON-SKIP` | round-44 refinement | `apply_gate.go` `ApplyRecovery` skip-path advances `recoveryCovered` | T4d-2 | +| `INV-REPL-LIVE-LANE-STALE-FAILS-LOUD` | round-44 refinement | `apply_gate.go` `ApplyLive` returns error on stale; never advances `recoveryCovered` | T4d-2 | +| `INV-REPL-RECOVERY-COVERAGE-RESTART-SAFE` | Option C hybrid | `apply_gate.go` `initSessionLocked` queries `store.AppliedLSNs()` at session start | T4d-2 | +| `INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` | Q2 + round-46 re-emphasis | `apply_gate.go` lane-explicit methods + `replica.go` caller-side dispatch (TRANSITIONAL — CARRY-T4D-LANE-CONTEXT-001 active) | T4d-2 (gate API) + T4d-3 (transport caller) | +| `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY` | Q1+Q3 + T4d-1 strengthening | `core/engine/` zero storage imports; `core/transport/import_discipline_test.go` source-grep fence | T4d-1 (engine purity) + T4d-3 (transport fence) | +| `INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1` | T4d-3 G-1 §5 | `core/engine/apply.go` engine emits `Recovery.R + 1`; `t4d3_rplus1_test.go` | T4d-3 | +| `INV-REPL-CATCHUP-FROMLSN-FROM-ENGINE-STATE-NOT-PROBE` | T4d-3 G-1 §5 | `core/engine/apply.go` reads engine state, not raw probe payload; `t4d3_rplus1_test.go` | T4d-3 | +| `INV-REPL-LIFECYCLE-HANDLE-BORROWED-001` | T4d-4 part A | `ReplicationVolume.Stop` doesn't close substrate handle; BUG-005 non-repeat regression test | T4d-4 part A | +| `INV-REPL-CATCHUP-EXHAUSTION-ESCALATES-TO-REBUILD` | round-47 architect addition | `core/engine/apply.go:567` exhaustion emits StartRebuild + pins RebuildPinned; `recovery_test.go:301+325` | T4d-4 part B+C | +| `INV-REPL-REBUILD-FAILURE-TERMINAL` | round-47 architect addition | rebuild MaxRetries=0; first failure → terminal degraded; `recovery_test.go` | T4d-4 part B | +| **`INV-REPL-FAILED-SESSION-KIND-DRIVES-ESCALATION`** *(part C Bug #1 capture)* | T4d-4 part C bug fix | `core/engine/apply.go` exhaustion check keys on `st.Session.Kind == SessionCatchUp` (truth = what session actually failed), NOT on the mutable `Decision` field; resists stray auto-probes that re-classify Decision mid-flight; `TestT4d4_FullL2_CatchupRetryExhausted_EscalatesToRebuild` | T4d-4 part C | +| **`INV-REPL-REBUILD-ESCALATION-STICKY-UNTIL-TERMINAL`** *(part C Bug #2 capture)* | T4d-4 part C bug fix | `RecoveryTruth.RebuildPinned` sticky bool set by both WALRecycled escalation and exhaustion; `decide()` honors as override forcing `Decision=Rebuild` regardless of R/S/H; cleared by successful rebuild SessionCompleted or identity reset; ensures next probe cannot downgrade rebuild before terminal; `TestT4d4_FullL2_WALRecycled_EscalatesImmediate` | T4d-4 part C | +| `LastSentMonotonic_AcrossRetries` *(full cross-call form)* | T4c QA #8 carry | `TestT4d4_FullL2_LastSentMonotonic_AcrossRetries` (T4d-4 part C) — closes T4c Stage-2 deferred scenario | T4d-4 part C | + +**Total inscribed at T4d close: 16 invariants** (14 new + 1 promotion + 1 QA-deferred-now-closed). Of the 14 new, 2 specifically capture the part C bug fixes (`INV-REPL-FAILED-SESSION-KIND-DRIVES-ESCALATION` + `INV-REPL-REBUILD-ESCALATION-STICKY-UNTIL-TERMINAL`) per architect round-48 review. + +**Active carries (named, post-G5 hardening backlog):** +- `CARRY-T4D-LANE-CONTEXT-001` — replace transport caller-side TargetLSN==1 shim with true handler/session-context lane signal (Option B test-skip in place); bind point: **post-G5 protocol hardening (G5 hardening backlog)** +- `G5-DECISION-001` — engine recovery state behavior across primary restart (Path A persist vs Path B rebuild-from-probe; T4d-4 part B kept structure serializable) + +--- + +## §G Test evidence + +| Layer | At HEAD `e642ae8` (T4d-4 part C) | +|---|---| +| T4d-1 unit (typed kind + AppliedLSNs) | 10 new + d6b1890 follow-up 2 HARD GATE | green | +| T4d-2 unit (apply gate) | 16 new + 1 un-skipped | green | +| T4d-2 follow-up rework (`01f4ab9`) | 2 regression tests | green | +| T4d-3 unit + L2 (R+1 + boundary fences) | 11 new (3 engine + 3 transport + 1 import-discipline + 4 component) | green | +| T4d-4 part A (RecoveryMode + Stop) | per part A | green | +| T4d-4 part B (engine wiring + round-47 + G5-DECISION-001) | 9 new + 1 updated; 2 G5-DECISION-001 tests | green | +| T4d-4 part C (full L2 + bug fixes) | 3 new HARD GATE #3 tests | green | +| **Cumulative T4d-relevant tests** | **~70 new tests across T4d batches** | **all green at HEAD** | +| `core/replication` package | — | green under -race on m01 | +| `core/replication/component` package | — | green under -race on m01 | +| `core/transport` package | — | green under -race on m01 | +| `core/engine` package | — | green under -race on m01 | +| `core/storage` + `core/storage/smartwal` | — | green under -race on m01 | +| Full V3 suite under -race on m01 | **22/22 packages green** (post-`a0be6d5` test-fixture fix) | T2 NVMe `TestT2A_ConcurrentQueueStress` race resolved at `a0be6d5` during T4d close cycle; verified ×50 stress + full nvme package PASS — see §H Finding #3 | + +**Smoke runs at QA workspace HEAD + m01 -race:** + +``` +$ go test -race -count=20 -timeout 5m -run 'TestT4d4_FullL2_' ./core/replication/component/ +ok github.com/seaweedfs/seaweed-block/core/replication/component 12.523s + +$ go test -race -count=1 -timeout 10m ./core/... +21 packages green; 1 pre-existing T2A NVMe race (separate bug) + +# Post-`a0be6d5` re-run (2026-04-26): +$ CGO_ENABLED=1 go test -race -count=50 -run TestT2A_ConcurrentQueueStress ./core/frontend/nvme/ +ok github.com/seaweedfs/seaweed-block/core/frontend/nvme 7.015s +$ CGO_ENABLED=1 go test -race -count=1 ./core/frontend/nvme/ +ok github.com/seaweedfs/seaweed-block/core/frontend/nvme 22.237s + +22/22 packages now genuinely green under -race on m01. +``` + +**Production-sized substrate smoke on m01 (1 GiB extent — see §H findings):** + +| Substrate | Create 1 GiB | 20k writes (~78 MB) | ScanLBAs (1) full retention | AppliedLSNs | +|---|---|---|---|---| +| walstore (default 64 MiB WAL) | 11.5 ms | ⚠ WAL admission fires at write 14611 (~57 MiB) — see Finding #1 | n/a (test aborted) | n/a | +| smartwal (2 MiB ring) | 1.66 ms | 154.7 ms (~518 MB/s) | 211.9 ms (~94k entries/sec) | 20000 entries, max LSN=20000 | + +**Per-batch substrate smoke regression check (T4d-1 baseline `1edeb36` vs current):** +- walstore: same WAL admission boundary at write ~14611 (no regression) +- smartwal: same throughput characteristics (78 MB write in ~129 ms; ScanLBAs ~204 ms; AppliedLSNs 20000 entries) +- Substrate behavior INVARIANT across T4d-1 → T4d-4-current (substrate code stable since T4d-1) + +--- + +## §H Production-readiness findings (m01 1 GiB substrate smoke) + +These findings are **not T4d implementation defects**. They are production-readiness gaps surfaced by realistic-config testing. Recording explicitly so they don't go silent into G5. + +### Finding #1: walstore WAL admission backpressure observed under synthetic write-only workload + +**Symptom**: `walstore` 1 GiB extent + default 64 MiB WAL + 20,000 sequential synchronous writes (no intermediate Sync) → WAL admission gate fires at write 14611 (~57 MiB consumed) with `storage: WAL admission: storage: WAL region full`. + +**Honest scope**: m01 smoke ran a synthetic write-only workload with no observed checkpoint relief before the 64 MiB WAL filled. walstore HAS a background flusher (`walstore.go:189-190` `s.flusher = newFlusher(...); go s.flusher.run()`) and explicit WAL admission control (`walstore.go:335-341`); QA's earlier characterization "caller-driven, no background goroutine" was wrong (corrected per architect round-48 review). The actual question is whether the flusher's checkpoint cadence keeps up with the synthetic single-threaded loop's write rate, and what tuning policy production deployments need. + +**Production implication**: blockvolume's lifecycle interaction with the walstore flusher needs verification under realistic production write pressure (not just synthetic stress). The T4d smoke did NOT verify this end-to-end. + +**Carry**: G5 production-readiness — verify walstore flusher/checkpoint cadence under production write pressure; define tuning/operational policy. **Reword from earlier draft per architect round-48: the smoke surfaced "WAL admission pressure under synthetic workload," NOT "no background checkpoint exists."** + +### Finding #2: no `--durable-walsize` CLI flag + +**Symptom**: `cmd/blockvolume/main.go` exposes `--durable-blocks` + `--durable-blocksize` + `--durable-impl` but NOT `--durable-walsize`. walstore default 64 MiB WAL is hardcoded; smartwal default 2 MiB ring is hardcoded (`smartwal/store.go:19` `defaultWALSlots = 65536`). + +**Production implication**: operators cannot tune WAL size at deployment without code change. + +**Carry**: post-G5 / operator hardening — add `--durable-walsize` CLI flag (small change; ~10 LOC + plumb through to `newSuperblock(walSize)` opts). + +### Finding #3: pre-existing `core/frontend/nvme/TestT2A_ConcurrentQueueStress` data race — ✅ RESOLVED at `a0be6d5` + +**Original symptom**: Under `go test -race` on m01 Linux, `TestT2A_ConcurrentQueueStress` in `core/frontend/nvme` emitted `WARNING: DATA RACE` and FAILED. + +**Verified pre-existing**: same failure at `1edeb36` (T4d-1 commit). NOT introduced by T4d-4. Belonged to T2 NVMe surface. + +**Root cause** (diagnosed via -race stack traces on m01): `writeCountingBackend.lastWrite []byte` field at `core/frontend/nvme/t2_v2port_nvme_write_chunked_r2t_test.go:64` was unsynchronized; the test fixture's `Write` method assumed serial caller use ("NOT safe across concurrent writes, but the test drives them serially") but `TestT2A_ConcurrentQueueStress` violated that assumption by sharing one fixture across 8 concurrent IO queues. Stack confirmed race lives entirely in test-fixture code, NOT in production session model (which is correctly serialized: one rxLoop, one txLoop, per-Write goroutines for IO dispatch). + +**Fix**: `a0be6d5` — `lastWrite []byte` → `atomic.Pointer[[]byte]` (QA's recommended Option A); 25+/8- diff, test fixture only, zero production code touched. + +**m01 -race verification (post-fix)**: +- `count=10` round: PASS in 2.3s ✓ +- `count=50` stress round: PASS in 7.0s ✓ +- Full `core/frontend/nvme` package under -race: PASS in 22.2s ✓ + +**Status**: closed. T2A NVMe surface no longer races under -race on m01. Full V3 -race smoke now genuinely all-green (no remaining package failures). + +### Finding #4: substrate behavior invariant across T4d batches + +**Symptom**: per-batch substrate smoke shows identical results T4d-1 baseline (`1edeb36`) → current. Substrate code stable since T4d-1; T4d-2/T4d-3/T4d-4 didn't touch substrate behavior. + +**Production implication**: positive — no substrate regressions during T4d. Confirms layering discipline (T4d work above substrate didn't accidentally couple back to substrate internals). + +--- + +## §I Forward-carries surfaced during T4d + +| Carry | Owner | Bind point | Origin | +|---|---|---|---| +| `CARRY-T4D-LANE-CONTEXT-001` — replace TargetLSN==1 caller shim with true handler/session-context lane signal | sw | **post-G5 protocol hardening (G5 hardening backlog)** | T4d-2 follow-up `01f4ab9` round-46 | +| `G5-DECISION-001` — engine recovery state across primary restart (Path A persist vs Path B rebuild-from-probe) | architect at G5 | G5 collective close | T4d-4 part B round-47 | +| walstore flusher/checkpoint cadence verification under production write pressure + tuning/operational policy | sw + architect | G5 production-readiness gate | §H Finding #1 (m01 production-sized smoke) | +| `--durable-walsize` CLI flag for operator tuning | sw | post-G5 / operator hardening | §H Finding #2 | +| ~~Pre-existing `TestT2A_ConcurrentQueueStress` data race~~ | ✅ RESOLVED `a0be6d5` | (closed during T4d close cycle; m01 -race ×50 PASS) | §H Finding #3 | +| m01 hardware first-light for replicated write path (multi-replica + real network conditions) | QA + sw | G5 collective close | T4c closure §H + kickoff §2.6 | +| Multi-replica concurrent live + recovery scenarios | QA | G5 production-readiness | T4d kickoff scope; not in T4d-4 | + +--- + +## §J Sign requests at this batch close + +T4d is **the final T4 batch**. Per §8C.1 + kickoff §4 #10, T-end three-sign lands at this close. + +| Signer | Sign type | Asks | +|---|---|---| +| **QA** | §8C.2 single-sign at batch close | Confirm 14 invariants pinned per §F; confirm forward-carry checklist v0.2 all 19 active T4a/T4b/T4c invariants green at HEAD under -race on m01; confirm §H findings are honest about production-readiness gaps. **(this document is the QA single-sign artifact)** | +| **Architect** | §8C.1 T4 T-end three-sign | (1) Read review points D-1 through D-6; (2) accept §B acceptance bar fully met; (3) accept §H production-readiness findings as G5 carries (not T4d defects); (4) accept G5-DECISION-001 + CARRY-T4D-LANE-CONTEXT-001 as named carries; (5) acknowledge T4 T-end at this close (no T4e). | +| **sw** | confirmatory note (no §8C.2 sign required at batch level per kickoff §4 #10 rules) | Confirm all `INV-REPL-*` upgrades from `⏭ T4d-N` to `✓ PORTED T4d-N` in catalogue §3.3; confirm §I forward-carries with Owner=sw are recorded for post-G5/G5-readiness planning | + +--- + +## §K Notes for future readers + +- T4d is where the apply gate (lane-pure, per-LBA stale-skip + coverage-advance) and engine-driven recovery wiring (R+1 + retry loop end-to-end + rebuild path) became real. The combination closes the "production correctness boundary for replicated writes under recovery." +- **Two architectural pins surfaced + fixed in part C** (catch-up exhaustion stale-Decision-key bug; WALRecycled-no-StartRebuild-emit bug + RebuildPinned sticky override). Both surfaced because architect's round-47 ruling required full L2 matrix end-to-end testing — synthetic injection wouldn't have caught either. This validates the round-47 hard-gate decision empirically. +- **Three architectural improvements ratified during T4d** beyond the original kickoff scope: 2-enum split (T4d-1 round-46), lane-pure gate API (T4d-2 round-46), round-47 catch-up exhaustion DIRECTLY emits StartRebuild. Each was sw-proposed-or-discovered + architect-endorsed; recorded explicitly in §B so the precedent is visible. +- **Process rule inscribed**: G-1 docs land in seaweedfs FIRST; sw implementation references committed hash via `Refs G-1 sign: `. T4d-3 G-1 sign hash = `seaweedfs@80036404c`. This avoids the "pending hash" procedural gap surfaced at T4d-3. +- **Production-sized substrate smoke at m01** is the right discipline for batch close evidence — surfaces gaps that unit tests at fixture sizes cannot (walstore WAL admission boundary; CLI knob absence). Forward-carry to G5: include production-sized smoke per batch as standard evidence. +- The forward-carries in §I represent **honest production-readiness work** that didn't fit T4d's scope. G5 collective close should not narrow them silently — apply the same §B-style explicit acceptance discipline T4c established. +- **Round-48 architect review surfaced 3 blocking edits + 1 smaller fix**: (1) part C commit hash bound to `e642ae8` everywhere; (2) `CARRY-T4D-LANE-CONTEXT-001` bind point corrected from "T4e (preferred) or post-G5" to **"post-G5 protocol hardening (G5 hardening backlog)"** since "no T4e" + "T4e preferred" was contradictory if T-end signs at this close; (3) §H Finding #1 reworded to remove the false root cause about "caller-driven, no background goroutine" — walstore HAS `go s.flusher.run()` at `walstore.go:189-190`; (4) 2 named invariants added to §F capturing part C bug fixes (`INV-REPL-FAILED-SESSION-KIND-DRIVES-ESCALATION` + `INV-REPL-REBUILD-ESCALATION-STICKY-UNTIL-TERMINAL`) per architect's smaller-fix request. Total inscribed invariants 14 → 16. Recording the round-48 corrections explicitly here so the procedural diligence is visible. diff --git a/sw-block/design/v3-phase-15-t4d-mini-plan.md b/sw-block/design/v3-phase-15-t4d-mini-plan.md new file mode 100644 index 000000000..bce21545e --- /dev/null +++ b/sw-block/design/v3-phase-15-t4d-mini-plan.md @@ -0,0 +1,696 @@ +# V3 Phase 15 — T4d (Apply Gate + Engine-Driven Recovery + Substrate Hardening + Lifecycle) Mini-Plan + +**Date**: 2026-04-25 (v0.3 — addresses v0.2 engine/storage dependency blocker) +**Status**: DRAFT v0.3 — re-submitted for QA + architect sign after v0.2 review (engine failure kind is engine-owned; no engine→storage dependency) +**Owner**: sw (implementation); QA (acceptance) +**Authority sources**: +- `v3-phase-15-t4d-kickoff.md` v0.3 (architect-ratified 2026-04-25 by pingqiu) — §1–8 ratified, §9 ratified Q1/Q2/Q3 +- `v3-phase-15-t4c-closure-report.md` §I forward-carries +- Round-43 architect lock (replica recovery apply gate, lane-aware, per-LBA) +- Round-44 architect refinement (2-map split: liveTouched + recoveryCovered) +- §9 architect decisions: Q1 engine-owned retry; Q2 lane implicit from handler context; Q3 transport↔storage contract-only +**Discipline**: §8B.4 mid-T batch; §8C.2 QA single-sign per batch close +**Timeline**: ~3-4 weeks (T4d-2 apply gate is the largest batch; T4d-3 G-1 read serializes its code start) +**Predecessors**: T4c batch closed (`c910464a9`); BlockStore walHead pre-T4d hotfix landed (`f6084ee`); kickoff §1–8 + §9 RATIFIED + +--- + +## §0 Why this is 4 batches, not 3 — and where V2 stops, V3 starts + +T4d kickoff §3 ratified a 4-batch shape (was 3 in kickoff v0.1; v0.2 added substrate hardening as a separate batch after round-43 surfaced the BlockStore + walstore + smartwal vulnerability). Architect rationale (kickoff §3 sign): + +> "This ordering is correct: remove brittle contracts first, then add the correctness gate, then optimize/wire end-to-end." + +**Why 4 batches, ordered exactly this way:** + +| New | Replaces / extends | Architect's reasoning | +|---|---|---| +| **T4d-1** Substrate hardening + structured failure kind | (new — round-43 surfaced; T4c closure §I structured-kind binding lands here) | "Lowest-risk decoupled work; closes the BlockStore production vulnerability quickly even if architect chooses bundle-not-hotfix; structured RecoveryFailureKind is a contract change other batches depend on." | +| **T4d-2** Replica recovery apply gate | (new — round-43 lock + round-44 refinement) | "The apply gate is the correctness boundary. R+1 is bandwidth optimization built ON the apply gate's correctness guarantee. Building R+1 first would mean shipping unsafe recovery in production for the duration of T4d-2 if T4d-3 lands first." | +| **T4d-3** R+1 threading + engine-driven recovery wiring | (T4c §I row 1 + row 3 carries; round-43 downgraded R+1 from correctness-blocker to bandwidth-optimization) | "T4d-3 needs G-1 because it touches the V2 catch-up muscle boundary and `runCatchUpTo(replicaFlushedLSN, targetLSN)` semantics." | +| **T4d-4** Lifecycle + integration | (BUG-005 lifecycle from sketch §4 row T4d + T4c §I rows 2/5/6/7) | Bundle the remaining T4c carries with the lifecycle item; integration matrix becomes meaningfully end-to-end once T4d-3 wiring is in. | + +**§9 walshipper boundary — PORT vs V3-native (architect-ratified at kickoff Q1/Q2/Q3):** + +T4d-3 is **NOT a wholesale V2 `WALShipper` port**. Per kickoff §9.2: + +| Area | Classification | T4d-3 treatment | +|---|---|---| +| `runCatchUpTo(R, target)` start boundary | **PORT from V2** | Thread replica R through `StartCatchUp` → executor scans `R+1..target` | +| Per-call catch-up deadline | **PORT from V2** | Keep deadline scope discipline; do NOT leak to later live ships | +| Last-sent / achieved-LSN monotonicity | **PORT from V2** | "Advance only after successful frame write"; final completion via `BarrierResponse.AchievedLSN` | +| Catch-up retry budget | **SEMANTIC PORT, V3 location** | Preserve V2 `maxCatchupRetries=3` behavior via `RecoveryRuntimePolicy`; **retry bookkeeping lives in engine** (Q1) | +| WAL-recycled escalation | **SEMANTIC PORT, V3 location** | Preserve "retention miss → rebuild"; T4d-1 replaces string match with structured `RecoveryFailureKind` | +| Live ship during recovery | **V3 generalization** | V2 allowed live + recovery concurrent; V3 enforces via lineage + lane-aware apply gate | +| Unified catch-up/rebuild/LBA-rebuild protocol | **V3-native** | `StartRecovery` + `RecoveryContentKind` + runtime policy; no V2 1:1 | +| Replica stale-entry skip + coverage accounting | **V3-native** | T4d-2 apply gate; transport calls the gate; transport never queries substrate per-LBA freshness | +| Engine-driven recovery retry loop | **V3-native** (Q1) | Engine emits fresh recovery commands per policy; executor stays byte-movement-only | +| Package-boundary discipline (transport↔storage contract-only; no `core/recovery` move in T4d) | **V3-native** (Q3 / kickoff §9.5) | Inscribed via `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY`; pinned at T4d-3. Package move deferred to post-G5/T4e per kickoff §9.3 #3 | + +Architect-binding: kickoff §9.5 instructs the T4d-3 G-1 V2 read to explicitly separate PORT items (5 listed) from V3-native items (5 listed). G-1 deliverable. + +--- + +## §1 Scope + +T4d delivers **the production correctness boundary for replicated writes under recovery**: a lane-aware replica apply gate that prevents per-LBA data regression, structured failure semantics that decouple engine from substrate string-matching, V2-faithful catch-up start-boundary porting (R+1 threading), end-to-end engine-driven recovery wiring, and the lifecycle close that closes BUG-005 non-repeat. + +### §1.1 Semantic whole + +After T4d: + +- **Recovery is correct under partial-failure interrupted replay**: replica apply gate enforces per-LBA stale-skip + coverage accounting; data does not regress on any over-scan / pinLSN-below-replicaR / repeated-window / live-races-recovery scenario +- **Recovery is bandwidth-efficient**: catch-up scans from `R+1`, not genesis (V2-faithful start boundary) +- **Recovery is fully engine-driven**: `ReplicationVolume↔adapter` wired; engine retry loop runs end-to-end; `WithEngineDrivenRecovery` framework primitive becomes real; full L2 matrix gains true engine-level coverage (T4c was muscle-level only) +- **Recovery failure semantics are structured**: `RecoveryFailureKind` enum replaces substring `"WAL recycled"` match; engine switches on typed kind +- **Substrates are stale-safe in depth**: BlockStore walHead guard already landed (`f6084ee`); walstore + smartwal substrate-level stale-protection where practical +- **Lifecycle is regression-safe**: `ReplicationVolume.Stop` + cross-orchestration sessionID coordination + Windows TempDir cleanup race fixed; BUG-005 non-repeat sealed + +### §1.2 What's NOT in T4d (explicit deferrals per kickoff §2.6 + §9.3) + +- **m01 first-light for replicated write path** — G5 collective close +- **Package move `core/transport` → `core/recovery`** — post-G5/T4e refactor (§9.3 #3) +- **Lane byte on `MsgShipEntry` wire** — Q2 ratified: lane is implicit from handler context; do NOT add wire field in T4d (§9.3 #2) +- **Substrate-native per-LBA applied-LSN exposure for ALL substrates** — Option C hybrid means add where practical; remaining substrates use in-memory session map seeded by §2.5 #1 architect ratification + +### §1.3 Lane discipline (Q2 architect-ratified, kickoff §9.3 #2) + +`MsgShipEntry` does NOT carry a lane byte in T4d. Lane is derived from the receiving handler / session context: + +| Handler | Lane | Apply path | +|---|---|---| +| Live-ship handler (steady-state Ship) | live | Existing lineage/session/live-order rules; stale → reject + diagnostic; do NOT advance `recoveryCovered` | +| Catch-up / rebuild handler | recovery | Apply gate: `if entry.LSN <= appliedLSN[LBA] → skip data + recoveryCovered[LBA]=true; else apply + appliedLSN[LBA]=entry.LSN` | + +`INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` pins this at T4d-2 / T4d-3. + +### §1.4 Location + +- Production code → `seaweed_block` repo, branch `phase-15` +- Reference docs → `seaweedfs` repo, `sw-block/design/` +- Tests → `seaweed_block` alongside code; component scenarios in `core/replication/component/` +- L3 m01 — out of T4d (G5 collective) + +--- + +## §2 Tasks (4 batches; ordered; each self-contained, reviewable, mergeable) + +### T4d-1 — Substrate hardening + structured recovery failure kind end-to-end (~170 production + ~100 tests; v0.2/v0.3 revised upward from v0.1 to absorb adapter + engine event-struct contract additions) + +**Where**: +- `core/storage/recovery_contract.go` — new substrate-facing `StorageRecoveryFailureKind` enum + `RecoveryFailure` typed-error struct + wrap helpers for `ScanLBAs` errors +- `core/storage/walstore_recovery.go` — wrap `ErrWALRecycled` returns with `RecoveryFailure{Kind: WALRecycled, ...}`; add per-LBA stale-skip in `ApplyEntry` body where practical (defense-in-depth, kickoff §2.5 #3 caveat) +- `core/storage/smartwal/recovery_scan.go` — same wrap + same defense-in-depth caveat +- `core/storage/store.go` — BlockStore.ApplyEntry already gated by hotfix `f6084ee`; verify + reference here +- **`core/engine/events.go`** — define engine-owned `RecoveryFailureKind` enum and add `FailureKind RecoveryFailureKind` field to `SessionClosedFailed`; engine MUST NOT import `core/storage` +- **`core/adapter/types.go`** — add `FailureKind engine.RecoveryFailureKind` field to `SessionCloseResult` (architect HIGH v0.1 #1 + v0.2 dependency fix: end-to-end typed kind, NOT hidden in FailReason text) +- `core/engine/apply.go` — `applySessionFailed` switches on `e.FailureKind == RecoveryFailureWALRecycled` (engine-owned typed branch); `containsAny`/substring match REMOVED. `Reason` field stays as diagnostic text only +- `core/transport/catchup_sender.go` — extracts substrate-returned `storage.RecoveryFailure.Kind` via `errors.As`, maps it to `engine.RecoveryFailureKind`, and populates `SessionCloseResult.FailureKind` +- `core/replication/volume.go` (and any close-callback wiring) — propagates `FailureKind` through the close path; doesn't re-derive from text +- Adapter normalization (`OnSessionClose` → engine event): copies `FailureKind` from `SessionCloseResult` to `SessionClosedFailed` +- Existing engine test `TestSessionFailed_WALRecycled_EscalatesToRebuild` — rewritten to construct event with `FailureKind: RecoveryFailureWALRecycled` and assert engine branches on the typed field + +**Why** (kickoff §2.4 / §3 / §4 #7): T4c accepted substring matching as TEMPORARY; the binding was "T4d (preferred) or G5 final sign (latest)." Engine→storage decoupling is a contract obligation. Plus substrate hardening per kickoff §2.3 closes the round-43-surfaced defense-in-depth gap that the apply gate (T4d-2) covers as primary. + +**Typed contract end-to-end (architect HIGH v0.1 #1 + v0.2 dependency fix)** — failure kind is a typed field on every plumbing hop, NOT parsed from text. Storage may classify substrate errors, but engine owns the decision vocabulary it consumes: + +```go +// core/storage/recovery_contract.go +type StorageRecoveryFailureKind int + +const ( + StorageRecoveryFailureUnknown StorageRecoveryFailureKind = iota + StorageRecoveryFailureWALRecycled // tier-class change; adapter maps to engine rebuild escalation + StorageRecoveryFailureSubstrate // substrate IO error; retryable unless caller says otherwise +) + +type RecoveryFailure struct { + Kind StorageRecoveryFailureKind + Cause error // wrapped underlying error for diagnostics + Detail string // free-form additional context +} + +func (f *RecoveryFailure) Error() string { ... } +func (f *RecoveryFailure) Unwrap() error { return f.Cause } +// errors.Is / errors.As let transport identify the storage-side kind. + +// core/engine/events.go +type RecoveryFailureKind int + +const ( + RecoveryFailureUnknown RecoveryFailureKind = iota + RecoveryFailureWALRecycled // tier-class change; escalate to rebuild + RecoveryFailureTransport // stream/connection error; retryable + RecoveryFailureSubstrate // substrate IO error; retryable + RecoveryFailureTargetNotReached // catch-up didn't reach targetLSN; retryable + RecoveryFailureSessionInvalidated +) + +// core/adapter/types.go (modify existing SessionCloseResult) +type SessionCloseResult struct { + ReplicaID string + SessionID uint64 + Success bool + AchievedLSN uint64 + FailureKind engine.RecoveryFailureKind // NEW v0.3; engine-owned, no engine→storage dependency + FailReason string // DIAGNOSTIC TEXT ONLY — engine MUST NOT parse +} + +// core/engine/events.go (modify existing SessionClosedFailed) +type SessionClosedFailed struct { + ReplicaID string + SessionID uint64 + FailureKind RecoveryFailureKind // engine-owned; engine does not import storage + Reason string // DIAGNOSTIC TEXT ONLY — engine MUST NOT parse +} +``` + +Engine `applySessionFailed` branches on `e.FailureKind`: + +```go +// core/engine/apply.go (replace existing substring-match block) +if e.FailureKind == RecoveryFailureWALRecycled { + st.Recovery.Decision = DecisionRebuild + st.Recovery.DecisionReason = "wal_recycled" + st.Recovery.Attempts = 0 + // ... existing recycle escalation path +} +// substring `containsAny(reason, ["WAL recycled", ...])` match REMOVED +``` + +**Acceptance fence**: `TestEngine_SessionFailed_NoMoreSubstringMatch` greps `core/engine/apply.go` for `"WAL recycled"` substring usage and fails if present. The substring helper `containsAny` may stay if other engine code uses it; if not, removed too. + +**Cross-package contract note (v0.3 dependency fix)**: `core/engine` MUST NOT import `core/storage`. Storage owns substrate-side error classification; transport/adapter maps that classification into `engine.RecoveryFailureKind`. `core/adapter` already imports `core/engine`, so `adapter.SessionCloseResult.FailureKind engine.RecoveryFailureKind` does not create a new package direction. This keeps the dependency line: storage error → transport mapping → adapter close result → engine event. + +**Substrate defense-in-depth (architect §2.5 #3 caveat)**: + +| Substrate | Action | Practical-scope decision | +|---|---|---| +| BlockStore | Already done — `f6084ee` walHead gate | No further action | +| walstore | Add per-LBA stale-skip in `ApplyEntry` body BEFORE `dm.put` (read existing `dm.get(lba)`, compare LSN, skip if `incoming <= existing`) | Sw judgment at impl time per §2.5 #3 caveat. If `dm.get` exposes per-LBA LSN cleanly (it does — the dirty entry carries `lsn`), include. Else: documented carry to T4e. | +| smartwal | Add per-LBA stale-skip in `ApplyEntry` body BEFORE `writeAt` extent overwrite. smartwal's per-LBA LSN tracking lives in the ring; need a fast-path read or session map | Sw judgment per §2.5 #3 caveat. Likely needs a small `lastAppliedLSN[lba]` in-memory map maintained by `ApplyEntry` itself. Inclusion contingent on not turning T4d-1 into a substrate refactor (kickoff §2.5 #3 verbatim). | + +**Tests** (10 required): + +- `TestStorageRecoveryFailureKind_Walstore_RecycleWrappedAsKind` — walstore `ScanLBAs` ErrWALRecycled wraps as storage-side `StorageRecoveryFailureWALRecycled` kind +- `TestStorageRecoveryFailureKind_Smartwal_RecycleWrappedAsKind` — smartwal same +- `TestTransportRecoveryFailureKind_MapsStorageRecycleToEngineKind` — transport maps `StorageRecoveryFailureWALRecycled` to `engine.RecoveryFailureWALRecycled` +- `TestTransportRecoveryFailureKind_TransportError_WrappedAsEngineKind` — non-recycle stream error surfaces as `engine.RecoveryFailureTransport` +- `TestEngine_SessionFailed_StructuredKind_RecycleEscalatesToRebuild` — engine maps typed `FailureKind` field, not substring (replaces existing `TestSessionFailed_WALRecycled_EscalatesToRebuild`) +- `TestEngine_SessionFailed_StructuredKind_NonRecycleRetries` — engine retry path uses `FailureKind` match +- `TestEngine_SessionFailed_NoMoreSubstringMatch` — fence: greps `core/engine/apply.go` for `"WAL recycled"` substring; fails if present (architect HIGH v0.1 #1 — engine MUST NOT parse text) +- `TestAdapter_SessionCloseResult_TypedKindEndToEnd` — close result built with `FailureKind` set; adapter normalization to engine event preserves the typed field +- `TestEngine_SessionClosedFailed_TypedKindFieldShape` — event struct compile fence: `FailureKind` field present + non-zero kind value triggers correct engine branch without importing storage +- `TestStorageWalstore_ApplyEntry_StaleLSN_SubstrateSkips` (if walstore stale-skip included) +- `TestStorageSmartwal_ApplyEntry_StaleLSN_SubstrateSkips` (if smartwal stale-skip included) + +**Inscribed invariants**: +- Update `INV-REPL-CATCHUP-RECYCLE-ESCALATES` (catalogue §3.3) — pinning method changes from substring to structured kind + +**Preserve**: +- All T4c invariants forward-carry green (per kickoff §4 #5) +- `applySessionFailed` non-recycle behavior (retry-loop, start-timeout exception) unchanged + +**No G-1 required** (architect kickoff §3 sign #4): "T4d-1 is V3-native + small substrate edits; no V2 muscle source." + +--- + +### T4d-2 — Replica recovery apply gate (lane-aware, per-LBA, 2-map split) (~250 production + ~250 tests) + +**Where**: +- `core/replication/apply_gate.go` — new file: `ReplicaApplyGate` component with `ApplyRecovery(...)` and `ApplyLive(...)` entry methods; per-session state (`liveTouched`, `recoveryCovered`, `appliedLSN`) +- `core/replication/apply_gate_session.go` — per-session state machine (initialized at session start, cleared at session close); seeds `appliedLSN` from substrate query path (Option C hybrid per kickoff §2.5 #1) +- `core/transport/replica.go` — `MsgShipEntry` handler dispatches to gate via lane discriminator (handler context, NOT wire byte per Q2) +- Adapter wiring (whichever entry registers session start with the gate) +- Existing skipped adversarial tests in `core/replication/component/stale_entry_adversarial_test.go` — un-skip + extend per round-44 (coverage-advance + live-fails-loud) +- New component scenarios per round-44 spec +- Substrate query path for Option C hybrid: small additive method on `LogicalStorage` returning per-LBA latest-applied-LSN if known (fallback: 0 = "ask the gate's session map") + +**Why** (kickoff §2.2 round-43 lock + round-44 refinement): primary correctness boundary. Without this, walstore's per-LSN scan + replica's blind apply produces per-LBA data regression on partial-failure interrupted replay. + +**Architect-locked spec (round-43 verbatim)**: + +> Recovery-stream stale entries are valid duplicates and must be skipped as data writes while still counted as recovery-stream coverage. Live-lane stale entries are abnormal and must not mutate data; they should be skipped/rejected under lineage/order diagnostics and must not advance recovery coverage. + +**Gate behavior** (lane-aware): + +``` +ApplyRecovery(replicaID, sessionID, entry): + require: session exists + matches lineage + if entry.LSN <= appliedLSN[entry.LBA]: + recoveryCovered[entry.LBA] = true + return // skip data; coverage advances + substrate.ApplyEntry(entry.LBA, entry.Data, entry.LSN) + appliedLSN[entry.LBA] = entry.LSN + recoveryCovered[entry.LBA] = true + return + +ApplyLive(replicaID, sessionID, entry): + require: session exists + matches lineage + if entry.LSN <= appliedLSN[entry.LBA]: + log diagnostic "live stale" + reject // round-44 fail-loud + return error + substrate.ApplyEntry(entry.LBA, entry.Data, entry.LSN) + appliedLSN[entry.LBA] = entry.LSN + liveTouched[entry.LBA] = true + // do NOT touch recoveryCovered + return +``` + +**`appliedLSN` source — Option C hybrid (architect §2.5 #1 ratified)**: + +At session start, gate queries substrate for per-LBA latest applied LSN: + +```go +type LogicalStorage interface { + // ... existing methods ... + // T4d-2 addition (Option C hybrid): + // Returns per-LBA latest-applied LSN where the substrate tracks + // it. Implementations that don't track this MUST return + // (nil, ErrAppliedLSNsNotTracked). The gate then falls back to + // in-memory session-only tracking, seeded from live + recovery + // applies during the session. + AppliedLSNs() (map[uint32]uint64, error) +} + +// New sentinel in core/storage/recovery_contract.go +var ErrAppliedLSNsNotTracked = errors.New("storage: substrate does not track per-LBA applied LSN") +``` + +**Per-substrate implementation (architect MED v0.1 #2 fix — every impl explicit, none "not implemented"):** + +| Substrate | `AppliedLSNs()` impl | Source | +|---|---|---| +| **walstore** | Iterate `dm.snapshot()`; for each entry build `map[uint32]uint64{lba: entry.lsn}` | dirty map already carries per-LBA LSN | +| **smartwal** | `ring.scanValid()` then per-LBA last-writer-wins reduction (mirrors smartwal's existing recovery semantic) | Already implemented for `ScanLBAs`; reuse | +| **BlockStore** | `func (s *BlockStore) AppliedLSNs() (map[uint32]uint64, error) { return nil, ErrAppliedLSNsNotTracked }` — explicit not-tracked return; ensures interface compatibility | In-memory; no per-LBA LSN tracking | + +Gate behavior on `ErrAppliedLSNsNotTracked`: log INFO once at session start, fall back to session-only `appliedLSN` map, seed from live + recovery applies as they arrive. Restart-safety degrades for unsupported substrates (BlockStore is calibration-only, so this is acceptable). + +**Tests** (~14 required): + +Adversarial set (un-skipped + extended from T4c skip-pile): +- `TestComponent_Adversarial_StaleEntryDoesNotRegress` — un-skipped; same LBA old/new + sever before new → no regression (architect round-41 set) +- `TestComponent_RepeatedRecoveryWindow_ByteAndLSNStable` — repeated window → byte state + per-LBA LSN stable +- `TestComponent_LiveRacesRecoveryOld_LiveWins` — live write at fresher LSN beats recovery's stale entry +- `TestComponent_RecoveryStaleSkip_CoverageStillAdvances` — round-44 #1: data skipped but `recoveryCovered[LBA]` updates +- `TestComponent_LiveLaneStaleEntry_FailsLoud` — round-44 #2: live lane stale → reject + diagnostic, do NOT advance recoveryCovered + +Lane-discrimination set: +- `TestApplyGate_RecoveryHandler_DispatchesToRecoveryPath` — handler-context routing +- `TestApplyGate_LiveHandler_DispatchesToLivePath` — handler-context routing +- `TestApplyGate_LaneNeverDerivedFromWireField` — fence: no lane byte on MsgShipEntry; lane comes from handler context only + +Option C hybrid set: +- `TestApplyGate_SessionStart_SeedsFromSubstrateAppliedLSN_Walstore` — walstore exposes; gate seeds correctly +- `TestApplyGate_SessionStart_SeedsFromSubstrateAppliedLSN_Smartwal` — smartwal exposes; gate seeds correctly +- `TestApplyGate_SessionStart_BlockStoreFallback_InMemoryOnly` — BlockStore returns ErrNotTracked; gate falls back + +Restart-safety set (kickoff §4 #6 promotion): +- `TestApplyGate_RestartMidRecovery_SessionRestartReseeds` — kill session mid-recovery; new session re-queries substrate; appliedLSN consistent with substrate truth (pins `INV-REPL-RECOVERY-COVERAGE-RESTART-SAFE`) + +Coverage-completion set: +- `TestApplyGate_BarrierCompletion_BarrierAuthoritativeWithCoverageContribution` — barrier ack is the authoritative completion truth (T4c-2 `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN` preserved); the gate's `recoveryCovered` map contributes to per-LBA coverage accounting that feeds barrier judgment, but does NOT replace the barrier as completion source. (Rename per QA v0.1 review issue #1; original name `TestApplyGate_BarrierAchievedLSN_DerivesFromCoverageNotData` could misread as conflicting with T4c-2 invariant.) + +**Inscribed invariants** (catalogue §3.3 — already pre-inscribed by QA per round-44 prep work; T4d-2 promotes to PORTED): +- `INV-REPL-NO-PER-LBA-DATA-REGRESSION` (goal-level, round-43) +- `INV-REPL-RECOVERY-STALE-ENTRY-SKIP-PER-LBA` (mechanism, round-43) +- `INV-REPL-RECOVERY-COVERAGE-ADVANCES-ON-SKIP` (round-44) +- `INV-REPL-LIVE-LANE-STALE-FAILS-LOUD` (round-44) +- `INV-REPL-RECOVERY-COVERAGE-RESTART-SAFE` (Option C makes pinnable) +- `INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` (Q2) + +**Preserve**: +- T4c apply path for non-stale entries unchanged (gate forwards to substrate `ApplyEntry`) +- Existing T4c bitmap/coverage logic IF any (none — T4d-2 introduces it) +- `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN` (T4c-2): completion still derives from barrier ack; gate accounting feeds barrier judgment, doesn't replace it + +**No G-1 required** (architect kickoff §3 sign #4): "T4d-2 does not need G-1; its spec is the round-43/44 V3-native apply-gate rule." Round-43 + round-44 architect text IS the spec. + +**Estimated LOC**: ~250 production (~150 gate + state + ~50 substrate AppliedLSNs impls + ~50 wiring) + ~250 tests (high test-to-production ratio because the gate is the correctness boundary; test density matters). + +--- + +### T4d-3 — R+1 threading + engine-driven recovery wiring — **G-1 PRE-CODE REQUIRED** (~250 production + ~150 tests) + +**Where**: +- `core/transport/catchup_sender.go` — `StartCatchUp` body: `ScanLBAs(R+1, ...)` instead of `ScanLBAs(1, ...)` +- `core/transport/executor.go` — `BlockExecutor.StartCatchUp` signature gains `fromLSN uint64` parameter +- `core/adapter/executor.go` — `CommandExecutor.StartCatchUp` interface signature update; all test executor stubs updated +- `core/engine/commands.go` — `engine.StartCatchUp` command struct gains `FromLSN uint64` +- `core/engine/apply.go` — emission populates `FromLSN` from `Recovery.R`; retry-loop re-emit also populates +- `core/replication/volume.go` — wire `ReplicationVolume↔adapter` so engine retry loop runs end-to-end +- `core/replication/component/cluster.go` — `WithEngineDrivenRecovery()` becomes real (was stub at T4c) +- L2 integration matrix — gain true engine-level coverage + +**Why** (kickoff §2.4 + §3 + §9.5): +- R+1 threading: kickoff §2.4 + T4c §I row 3 — bandwidth optimization (post-round-43; was correctness-blocker pre-round-43, now the apply gate handles correctness regardless of where primary scans from). Pins `INV-REPL-CATCHUP-WITHIN-RETENTION-001` un-pin from T4c. +- Engine-driven recovery wiring: kickoff §2.4 + T4c §I row 1 — closes the integration-scope gap left at T4c (engine retry loop unit-tested only). +- Q1 architect lock: retry loop is engine-owned, not transport. T4d-3 does NOT add a self-driven retry loop in `catchup_sender.go`. + +**§9.5 architect-binding split** (T4d-3 G-1 must explicitly separate): + +PORT (5 items from V2): +1. `runCatchUpTo(replicaFlushedLSN, targetLSN)` start-boundary semantics +2. Per-call deadline scope (do not leak to live ships) +3. Last-sent monotonicity ("advance only after successful frame write") +4. Retry count semantics (`maxCatchupRetries=3` envelope; bookkeeping in engine per Q1) +5. Retention-miss escalation ("WAL recycled" → rebuild; via T4d-1 structured kind) + +V3-native (5 items, do not look for V2 source): +1. Engine-owned retry loop (Q1 — engine decides emit/escalate; transport stays byte-movement-only) +2. Unified recovery command model (`StartRecovery` / `RecoveryContentKind`) +3. Lane-aware apply gate integration (T4d-3 calls into T4d-2 gate; transport never queries substrate per-LBA freshness per Q3) +4. `BarrierResponse.AchievedLSN` as completion truth (T4c-2 `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN`) +5. Package-boundary discipline (transport↔storage contract-only per `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY`) + +**G-1 V2 read deliverable** — `sw-block/design/v3-phase-15-t4d-3-g1-v2-read.md` (sw produces; QA reviews + signs before T4d-3 code starts). + +Deliverable structure: +1. V2 source verbatim — `weed/storage/blockvol/wal_shipper.go:845` `runCatchUpTo` body + calling context +2. Per-V2-concern → V3-location map (5 PORT items + their V3 placement) +3. V3-native items list (5 items, named explicitly so reviewer doesn't expect V2 source) +4. Hidden-invariant audit (anticipated 2-3 invariants) +5. Per-substrate behavior under R+1 threading (smartwal R+1, walstore R+1, BlockStore R+1) +6. LOC heuristic + V3 vs V2 expansion budget (per `feedback_g1_pre_code_review.md` LOC asymmetry note) +7. Test parity matrix (V2 mirror + V3-only) +8. Open placement decisions (if any) +9. QA sign block + +**Tests** (~10 required): + +R+1 threading set: +- `TestT4d3_CatchUp_ScansFromReplicaR_NotGenesis` — replica at R=50, primary at H=100; sender ScanLBAs called with `fromLSN=51`, NOT `fromLSN=1` +- `TestT4d3_CatchUp_NonEmptyReplica_ShortGap_BandwidthBounded` — pins `INV-REPL-CATCHUP-WITHIN-RETENTION-001` un-pin; over-scan reduction observable via emit count (use `NewObservedScanWrap` from component framework) +- `TestT4d3_CatchUp_StartCatchUpSignature_FromLSNRequired` — sender rejects calls without explicit `fromLSN` + +Engine-driven recovery set: +- `TestT4d3_EngineRetryLoop_E2E_RetryUntilBudget` — engine retry loop end-to-end through real ReplicationVolume; budget exhaustion → escalate observable +- `TestT4d3_EngineRetryLoop_E2E_RecycleEscalatesImmediate` — ErrWALRecycled bypasses retry, escalates to rebuild via real path +- `TestComponent_WithEngineDrivenRecovery_NotStubAnymore` — framework primitive emits real engine commands + +Lane discipline (transport side): +- `TestT4d3_Catchup_TransportNeverCallsSubstratePerLBA` — `catchup_sender` does not import substrate-internal packages; pins `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY` + +L2 matrix expansion: +- 3 existing T4c L2 scenarios (ShortDisconnect, GapExceedsRetention, RecoveryModeLabelSurfaced) re-pass with engine-driven flow + +**Inscribed invariants**: +- `INV-REPL-CATCHUP-WITHIN-RETENTION-001` un-pinned at T4c → PORTED at T4d-3 +- `INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` (Q2) — receiver-side: catch-up handler routes to recovery lane +- `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY` (Q3) — sender-side: no substrate-internal imports + +**Preserve**: +- T4c `ScanLBAs(1, ...)` wasn't wrong, was wasteful; new `ScanLBAs(R+1, ...)` is V2-faithful +- T4c-2's 6 catch-up invariants (callback-return-nil-continues, lastSent-monotonic, deadline-per-call-scope, target-not-reached-vs-recycled-distinguished, recycle-escalates, completion-from-barrier-achievedLSN) all forward-carry +- T4c-pre-B `StartRecoverySession` dispatch shell unchanged (T4d-3 changes `StartCatchUp` legacy path; the unified path stays) + +**G-1 required** per kickoff §3 sign #3. + +--- + +**T4d-3 closure scope (architect-bound 2026-04-25 round-46+ Path B):** + +T4d-3 closes as **R+1 bounded catch-up PORT + boundary fences**, NOT full engine-driven recovery. Architect explicit: "do not repeat T4c's narrowing silently" — this is a **scope delta**, not a soft carry. + +**What T4d-3 (`44c60dd`) DELIVERED:** +- 5 G-1 PORT items (R+1 start boundary + per-call deadline + lastSent-monotonic + retry budget semantics + retention-miss escalation) +- 2 hidden invariants pinned (`INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1` + `INV-REPL-CATCHUP-FROMLSN-FROM-ENGINE-STATE-NOT-PROBE`) +- Boundary fence: `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY` (Q3) via import-discipline test +- CARRY-T4D-LANE-CONTEXT-001 Option B (skip-test with forward-binding) +- 11 new tests per G-1 §7 + +**What T4d-3 DEFERRED to T4d-4 (mandatory, see T4d-4 expansion below):** +- `ReplicationVolume↔adapter` wiring so engine retry loop runs end-to-end +- `WithEngineDrivenRecovery()` framework primitive becomes real (currently stub) +- Full L2 subprocess matrix gain engine→adapter→executor end-to-end coverage + +**Tests below labeled "Engine-driven recovery set"** (`TestT4d3_EngineRetryLoop_E2E_*` + `TestComponent_WithEngineDrivenRecovery_NotStubAnymore`) **DID NOT land in T4d-3** — they land in T4d-4 per Path B fold. + +**T4d cannot close** until `WithEngineDrivenRecovery()` is real and QA #8 (`LastSentMonotonic_AcrossRetries_FullForm`) can run — see updated T4d-4 acceptance criteria. + +**Estimated LOC**: ~250 production (signature change ripples through 5 files + engine wiring ~100 + framework primitive ~50) + ~150 tests. + +--- + +### T4d-4 — Lifecycle + integration + remaining T4c carries + **engine-driven recovery wiring (Path B fold)** (~300 production + ~350 tests; **REVISED v0.4** from ~150/~200) + +**Scope expansion (architect-bound 2026-04-25 round-46+ Path B):** T4d-3's mini-plan §2.3 listed engine-driven recovery wiring (`ReplicationVolume↔adapter` + `WithEngineDrivenRecovery()` real binding + full L2 matrix). T4d-3 (`44c60dd`) deferred this work; architect ratified Path B = fold into T4d-4 as **HARD CLOSE GATE** (not soft carry). T4d cannot close until this wiring is real and QA #8 can run. + +**Where (revised v0.4 — adds engine-driven wiring rows):** + +**Where**: + +*Original T4d-4 scope (lifecycle + integration + T4c carries):* +- `core/replication/volume.go` — `ReplicationVolume.Stop` lifecycle regression (BUG-005 non-repeat per `v3-phase-15-t4d-bug005-non-repeat.md`) +- `core/replication/peer.go` — cross-orchestration sessionID coordination (T4c §I row 5: live-ship + explicit catch-up share `sessionID=1`) +- `core/storage/recovery_contract.go` — substrate `RecoveryMode()` method (T4c §I row 6 — wraps forward label without duck-typed `CheckpointLSN`) +- `core/replication/component/faults.go` — wraps forward `RecoveryMode()` correctly (closes the round-40 known limitation) +- `core/transport/catchup_sender.go` — replace duck-typed `CheckpointLSN()` probe with `RecoveryMode()` call +- `core/replication/component/` — Windows TempDir cleanup race investigation/fix (T4c §I row 7); rename or split `TestT4c3_Catchup_ShortDisconnect_DeltaOnly` per kickoff §5 trailing chore +- QA `LastSentMonotonic_AcrossRetries` cross-call form (T4c §I row 2) + +***Path B fold from T4d-3 (engine-driven recovery wiring — MANDATORY HARD CLOSE GATE):*** +- **`core/replication/volume.go`** — wire `ReplicationVolume↔adapter` so engine retry loop runs end-to-end (T4d-3 deferred per Path B; mini-plan v0.2 §2.3 row) +- **`core/replication/component/cluster.go`** — `WithEngineDrivenRecovery()` becomes real (was stub at T4c; T4d-3 deferred per Path B) +- **Full L2 integration matrix** — engine→adapter→executor end-to-end coverage (closes mini-plan §4 acceptance criterion #3 that T4c narrowed and T4d-3 deferred) + +***Round-47 architect addition: rebuild path engine-driven end-to-end (MANDATORY):*** +- `core/engine/apply.go` — extend `applySessionFailed`'s rebuild escalation to actually emit a rebuild command (not just transition `Decision = Rebuild` state); engine MUST own rebuild emit per Q1 architect lock +- `core/engine/events.go` — `RebuildSessionClosed{Success,Fail}` event (or equivalent) so rebuild outcomes are observable to engine state +- `core/replication/volume.go` (or appropriate dispatch site) — wire rebuild commands through ReplicationVolume↔adapter the same way catch-up commands do +- `core/engine/apply.go` — apply rebuild-session-closed events; define rebuild failure terminal (retry budget OR immediate PublishDegraded — architect impl-time call) +- L2 integration test: real catch-up → budget-exhausted → engine emits rebuild → rebuild executes and completes (or fails terminally) → outcome observable in engine state. **Test must observe an actual rebuild session occurring**, not just `Decision = Rebuild` state transition. + +**Why** (kickoff §2.1 + §2.4 + Path B fold): close BUG-005 non-repeat (the original sketch T4d scope item); land all T4c §I forward-carries; round out integration matrix at L2 end-to-end; **complete the engine-driven recovery wiring deferred from T4d-3 (Path B fold)**. + +**Tests** (~15 required): + +Lifecycle: +- `TestReplicationVolume_Stop_LifecycleRegression_BUG005NonRepeat` — pin per `v3-phase-15-t4d-bug005-non-repeat.md` (pins `INV-REPL-LIFECYCLE-HANDLE-BORROWED-001`) +- `TestReplicationVolume_Stop_DuringCatchUp_TerminatesCleanly` +- `TestReplicationVolume_Stop_DuringRebuild_TerminatesCleanly` +- `TestReplicationVolume_Stop_Idempotent` + +Cross-orchestration sessionID: +- `TestPeer_SessionIDCoordination_LiveShipAndExplicitCatchUp_NoCollision` + +`RecoveryMode()` method: +- `TestStorageRecoveryMode_Walstore_ReportsWALReplay` — walstore exposes mode via method (not duck-typed) +- `TestStorageRecoveryMode_Smartwal_ReportsStateConvergence` +- `TestStorageRecoveryMode_BlockStore_ReportsStateConvergence` +- `TestComponent_AssertSawRecoveryMode_UnderWrap_StillWorks` — closes the round-40 wrap-vs-CheckpointLSN limitation + +Integration matrix (L2 end-to-end — Path B fold from T4d-3 — MANDATORY for T4d close): +- `TestT4d4_Integration_FullRecoveryFlow_E2E_BothSubstrates` — engine→adapter→executor→replica end-to-end +- `TestT4d4_Integration_RecoverFromMidStreamFailure_RetryAndEscalate` — partial failure → retry → escalate full chain +- `TestT4d3_EngineRetryLoop_E2E_RetryUntilBudget` (relocated from T4d-3 per Path B fold) — engine retry loop end-to-end through real ReplicationVolume; budget exhaustion → escalate observable +- `TestT4d3_EngineRetryLoop_E2E_RecycleEscalatesImmediate` (relocated from T4d-3 per Path B fold) — ErrWALRecycled bypasses retry, escalates to rebuild via real path +- `TestComponent_WithEngineDrivenRecovery_NotStubAnymore` (relocated from T4d-3 per Path B fold) — framework primitive emits real engine commands +- `TestComponent_LastSentMonotonic_AcrossRetries_FullForm` — QA Stage-1 deferred scenario (QA #8); REQUIRES `WithEngineDrivenRecovery()` real binding to be runnable; T4d cannot close until this test runs green + +Rebuild path engine-driven (round-47 architect addition — MANDATORY for T4d close): +- `TestT4d4_Integration_CatchupBudgetExhausted_EngineEmitsRebuild` — catch-up retry budget exhausted → engine emits actual rebuild command (not just state transition) +- `TestT4d4_Integration_RebuildSession_CompletesAndReportsToEngine` — rebuild session runs end-to-end; completion event reaches engine state +- `TestT4d4_Integration_RebuildFailure_TerminalDefined` — rebuild failure path; verifies terminal is either retry-budget OR PublishDegraded per architect impl-time call (test asserts whichever was chosen) +- `TestT4d4_Integration_CatchupToRebuildEscalation_RealPath` — full path: catch-up failure → escalation → rebuild session → outcome observable. **Test observes actual rebuild session occurring**, not just `Decision = Rebuild` state. + +Cleanups: +- Windows TempDir cleanup race fix verification +- `TestT4c3_Catchup_ShortDisconnect_DeltaOnly` rename or split per kickoff §5 chore + +**Inscribed invariants**: +- `INV-REPL-LIFECYCLE-HANDLE-BORROWED-001` (newly inscribed at T4d-4 per kickoff §4 #6) +- `LastSentMonotonic_AcrossRetries` full form pin + +**Preserve**: +- All prior T4d-1 / T4d-2 / T4d-3 invariants forward-carry +- All T4a/T4b/T4c invariants forward-carry per kickoff §4 #5 + +**No G-1 required** (port + tests, no new V2 muscle source). + +**Estimated LOC (revised v0.4)**: ~300 production + ~350 tests (was ~150/~200; expanded by Path B fold ~150 prod / ~150 tests for `ReplicationVolume↔adapter` wiring + `WithEngineDrivenRecovery` real binding + L2 matrix scenarios). + +**HARD CLOSE GATE for T4d (architect-bound 2026-04-25 round-46+ Path B; round-47 rebuild expansion):** +T4d cannot close until ALL of: +1. `WithEngineDrivenRecovery()` is REAL (not stub) — `core/replication/component/cluster.go` +2. `ReplicationVolume↔adapter` wiring runs engine retry loop end-to-end +3. **Full L2 matrix engine→adapter→executor→replica end-to-end** including: + - catch-up retry-until-budget end-to-end + - WAL recycled → rebuild escalation real path (NOT unit-scope only) + - `TestComponent_LastSentMonotonic_AcrossRetries_FullForm` (QA #8) runs and passes + - Mini-plan §4 acceptance criterion #3 satisfied — same bar T4c narrowed and T4d-3 deferred per Path B +4. **Rebuild path engine-driven end-to-end (round-47 architect addition; cannot be cathup-only)** — minimum: + - catch-up failure / budget exhausted → engine emits rebuild command (not just sets `Decision = Rebuild` state) + - rebuild session completion / failure → returns to engine via observable event (`RebuildSessionClosed{Success/Fail}` or equivalent) + - rebuild failure terminal defined explicitly: either rebuild has its own retry budget OR `PublishDegraded` immediately on first rebuild failure (architect call at impl time) + - L2 covers the **real path** catch-up → rebuild escalation (test must observe an actual rebuild session occurring, not just `Decision = Rebuild` state transition) +5. All other T4d-4 acceptance items (lifecycle, T4c carries, etc.) green + +**Explicitly NOT in T4d-4 hard gate (E1 — architect round-47 ruling):** +Engine state persistence across primary restart is a **G5 decision gate**, not a T4d-4 implementation gate. T4d-4 doesn't need to implement persistence; G5 must explicitly decide "persist Recovery state" vs "rebuild state from probe after restart" as a named decision (see §G5-DECISION-001 below). + +--- + +## §3 Predicates (must be true before T4d-1 starts) + +| Predicate | Source | +|---|---| +| T4c batch closed | `c910464a9` (architect 2026-04-25) | +| Round-43 architect lock | kickoff §2.2 | +| Round-44 architect refinement | kickoff §2.2 | +| Kickoff §1–8 RATIFIED | architect sign 2026-04-25 | +| Kickoff §9 (walshipper Q1/Q2/Q3) RATIFIED | architect sign 2026-04-25 | +| BlockStore walHead pre-T4d hotfix landed | `f6084ee` | +| All T4a + T4b + T4c invariants forward-carry green | last full-suite run on `f6084ee` | +| QA sign on this mini-plan | _________ pending | + +--- + +## §4 Acceptance (per kickoff §4 ratified bar) + +T4d closes when ALL of: + +1. ✅ All 4 batches (T4d-1, T4d-2, T4d-3, T4d-4) merged to `phase-15` +2. ✅ Unit tests green per task (with Windows cleanup race fixed or platform-noted) +3. ✅ Full L2 subprocess matrix green — engine→adapter→executor end-to-end (the bar T4c narrowed) +4. ⏸ L3 m01 — deferred to G5 collective per kickoff §2.6 (NOT a T4d criterion) +5. ✅ T4a/T4b/T4c invariants forward-carry verified (no regression) +6. ✅ Active invariant promotions per kickoff §4 #6 (9 invariants — see kickoff for full list) +7. ✅ Engine sentinel decoupling: substring match REMOVED from `core/engine/apply.go`; structured kind in production at T4d-1 +8. ✅ BlockStore walHead regression fixed in production — DONE (`f6084ee`) +9. ✅ QA single-sign at T4d close per §8C.2 +10. ✅ T4 T-end three-sign per §8C.1 lands at T4d close IF no T4e is created (kickoff §4 #10) +11. ✅ No §8C.3 escalation triggers fired + +--- + +## §5 Non-claims (explicit deferrals within T4d) + +- **No m01 first-light** — G5 collective close (kickoff §2.6) +- **No `core/recovery` package move** — post-G5/T4e (kickoff §9.3 #3) +- **No lane byte on `MsgShipEntry` wire** — Q2 ratified, lane is implicit from handler context (kickoff §9.3 #2) +- **No wholesale V2 `WALShipper` port** — T4d-3 ports the bounded catch-up muscle ONLY; engine-owned recovery orchestration stays V3-native (kickoff §9.2 + §0 above) +- **No substrate-native per-LBA applied LSN exposure for ALL substrates** — Option C hybrid: add where practical (walstore, smartwal); BlockStore falls back to in-memory session map (kickoff §2.5 #1) +- **No retry loop in transport** — Q1: engine-owned (kickoff §9.3 #1) +- **No engine recovery state persistence implementation** (round-47 architect ruling) — engine `Recovery{Attempts, TargetLSN, ActiveSessionID}` survives only as long as the primary process. Across primary restart, all in-flight Recovery state is lost; replicas re-probe and engine rebuilds state from observed facts. **G5-DECISION-001** below makes this explicit. + +### G5-DECISION-001 — engine recovery state behavior across primary restart (named decision record) + +**Status**: ⏳ DECISION REQUIRED at G5 collective close (architect-bound 2026-04-25 round-47). + +**Question**: when the primary process restarts mid-recovery (panic, deploy, container restart, OS kill), what happens to in-flight Recovery state (`Attempts` counter, `TargetLSN`, `ActiveSessionID`, retry budget consumed)? + +**Two paths** (G5 must explicitly choose one): +- **A — persist Recovery state**: engine state durably stored (somewhere TBD); restart resumes from persisted state. Pros: retry budget honored across restart; in-flight session can resume. Cons: substrate work; restart-during-persist atomicity; new failure mode if persistence fails. +- **B — rebuild state from probe after restart**: engine starts cold; probes replicas; reconstructs `Recovery.R` from probe responses; resets `Attempts` to 0. Pros: simpler; no new substrate. Cons: retry budget effectively unbounded (each restart resets); operator can't tell "this replica has failed 9 times" if interspersed with restarts. + +**Why this is a G5 decision, not a T4d-4 implementation**: +- T4d-4 hard gate is "engine-driven recovery wiring works in steady-state." Restart semantics is orthogonal. +- Path A vs B has architectural weight (persistence = new substrate concern; both have observable behavior differences operators must understand). +- Choosing requires production-context input (operations team's tolerance for retry-budget reset on restart, persistence cost vs simplicity). +- G5 close demands the decision recorded; the **implementation** of the chosen path may live in G5 or post-G5. + +**T4d-4 obligation**: ensure engine state structure is **decision-A-compatible** (i.e., serializable; no embedded non-serializable types). This keeps either path open. Architect call at T4d-4 close if decision A would require structural changes that should be pre-positioned now. + +**Owner of the decision**: architect at G5 collective close. + +**Risk if silent**: G5 closes assuming "recovery works in production" without anyone having decided what restart does. First production primary restart surprises operators; depending on path, retry budget is either nonsense (B + frequent restarts → infinite retries) or fails loudly with no fallback (A + persistence layer not chosen). + +--- + +## §6 Invariants to preserve (forward-carry baseline) + +Per kickoff §4 #5. Named invariants from T4a/T4b/T4c that MUST stay green: + +| Source | Invariant | T4d regression risk | Protection | +|---|---|---|---| +| T4a-2 | `INV-REPL-SHIP-TRANSPORT-MUSCLE-001` | Low — Ship not touched | T4a-2 suite via G-3 | +| T4a-4 | `INV-REPL-LSN-ORDER-FANOUT-001` | **Medium — T4d-3 changes catch-up scan boundary** | Forward-carry test in T4d-3; lock scope at PR | +| T4a-4 | `INV-REPL-PEER-REBUILD-ON-AUTHORITY-CHANGE` | Low | T4a-4 test | +| T4a-3 | `INV-REPL-FANOUT-001` | Low | T4a-6 BasicEndToEnd | +| T4b-1 | `INV-REPL-LINEAGE-BORNE-ON-BARRIER-ACK` | Low — wire untouched | T4b-1 + T4b-6 wire fence | +| T4b-3 | `INV-REPL-BARRIER-FAILURE-DEGRADES-PEER` | Low | T4b-3 + T4b-4 peer-state pin | +| T4b-4 | `INV-REPL-LOCAL-FSYNC-GATES-QUORUM` | Low | T4b-4 | +| T4b-1 | `INV-REPL-NO-ZERO-LINEAGE-FIELDS` | Low — no new wire surfaces in T4d | T4b-1 fence still applies | +| T4c-1 | `INV-REPL-LINEAGE-BORNE-ON-PROBE-PAIR` | Low — probe wire untouched | T4c-1 fence | +| T4c-1 | `INV-REPL-PROBE-NON-MUTATING-VALIDATION` | Low — probe handler untouched | T4c-1 test | +| T4c-2 | `INV-REPL-CATCHUP-CALLBACK-RETURN-NIL-CONTINUES` | **Medium — T4d-3 changes scan boundary** | Forward-carry test in T4d-3 | +| T4c-2 | `INV-REPL-CATCHUP-LASTSENT-MONOTONIC` | **Medium — T4d-3 R+1 changes start LSN** | Forward-carry test in T4d-3 | +| T4c-2 | `INV-REPL-CATCHUP-DEADLINE-PER-CALL-SCOPE` | Low — deadline discipline preserved | T4c-2 test | +| T4c-2 | `INV-REPL-CATCHUP-TARGET-NOT-REACHED-VS-RECYCLED-DISTINGUISHED` | **Medium — T4d-1 structured kind changes detection** | T4d-1 test must distinguish the two via typed kind, not substring | +| T4c-2 | `INV-REPL-CATCHUP-RECYCLE-ESCALATES` | **High — pinning method changes from substring to structured kind** | T4d-1 + T4d-2 + T4d-3 chain | +| T4c-2 | `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN` | Low — completion semantics unchanged | T4c-2 test | +| T4c-2 | `INV-REPL-PEER-STATE-CATCHINGUP` | Low | T4c-2 test | +| T4c-3 | `INV-REPL-RECOVERY-MODE-OBSERVABLE` | **Medium — T4d-4 replaces duck-typed probe with method** | T4d-4 test | +| T4c retry pin | retry-budget-per-content-kind defaults | Low | T4c-3 unit test | + +--- + +## §7 Review gates + +### §7.1 Pre-merge gates (mandatory; PR blocked until satisfied) + +**Gate G-1 — V2 diff-footprint review on muscle-path tasks.** + +Applies to **T4d-3 ONLY** in T4d. NOT required for: +- T4d-1: V3-native + small substrate edits (no V2 muscle source) +- T4d-2: V3-native; round-43 + round-44 architect text IS the spec +- T4d-4: port + tests + +T4d-3 G-1 deliverable per kickoff §3 sign #3 + §9.5 split: sw produces `sw-block/design/v3-phase-15-t4d-3-g1-v2-read.md` separating PORT items (5) from V3-native items (5). QA signs pre-code per round-11 / round-15 / round-24 / T4c-2 G-1 discipline. Sw does not write T4d-3 production code until QA signs the G-1. + +**Procedural binding** (QA v0.1 review NOTE 2): every T4d-3 PR commit message MUST reference the QA-signed G-1 doc by commit hash (the sw commit that landed the QA-sign edits). Format: `Refs G-1 sign: `. This makes the procedural chain auditable and prevents post-hoc claims of "I didn't see the G-1 finding." + +**Process rule inscribed (architect-bound 2026-04-25 round-46+ Issue 2(a) ratification):** +G-1 sign docs land in `seaweedfs/sw-block/design/` FIRST; sw implementation in `seaweed_block` references the committed G-1 hash via `Refs G-1 sign: ` per the binding above. Avoids the T4d-3 procedural lesson where `44c60dd` referenced "pending hash" because the G-1 doc was still untracked at commit time. T4d-3 G-1 doc landed at `seaweedfs@80036404c` (2026-04-25); future references to T4d-3 G-1 use this sha. + +**Gate G-2 — Three-line godoc contract on every new public method.** + +Applies to every new exported symbol in T4d-1 / T4d-2 / T4d-3 / T4d-4: +- `// Called by:` +- `// Owns:` +- `// Borrows:` + +**Gate G-3 — Cumulative suite stays green.** + +Every T4d-N commit: full `go test -count=1 ./...` green. Particular attention: +- `core/storage/` — substrate hardening + AppliedLSNs() new method +- `core/replication/` — apply gate + lifecycle changes +- `core/transport/` — catchup_sender signature change +- `core/engine/` — substring match removal + retry-loop end-to-end wiring +- `core/replication/component/` — adversarial tests un-skip + framework wraps update + +### §7.2 Risks + procedural mitigations + +| Risk | Mitigation | §8C.3 trigger? | +|---|---|---| +| T4d-2 apply gate misses lane discrimination edge case (recovery enters via wrong handler) | `INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` test + dedicated lane-routing scenarios | trigger #1 if cross-lane leak post-merge | +| T4d-2 Option C hybrid `appliedLSN` seed differs from substrate truth on race | Restart-safety test + substrate-query semantics fence | trigger #1 if seed-vs-truth divergence | +| T4d-3 R+1 threading regresses T4c L2 matrix | Forward-carry tests for `INV-REPL-CATCHUP-CALLBACK-RETURN-NIL-CONTINUES` + `INV-REPL-CATCHUP-LASTSENT-MONOTONIC` under R+1 | trigger #1 if any T4c-2 invariant fails post-merge | +| T4d-3 G-1 mis-classifies V3-native item as PORT (or vice versa) | QA G-1 review uses §9.5 split as checklist | catch at G-1 sign | +| T4d-1 substring removal misses a call site | `TestEngine_SessionFailed_NoMoreSubstringMatch` fence + grep at PR | trigger #1 if substring sneaks back | +| T4d-4 lifecycle regression mis-handles pending sessions | BUG-005 non-repeat tests; per-state cleanup pins | trigger #1 if BUG-005 reproducible | +| Substrate defense-in-depth (T4d-1) turns into refactor | §2.5 #3 caveat: "if practical" — sw judgment + documented carry to T4e if needed | acceptable carry | + +--- + +## §8 Sign table (mid-T batch, §8C.2) + +| Role | Signer | Date | Decision | +|---|---|---|---| +| QA Owner | Claude (QA agent) | _________ | ⏸ pending T4d close | + +Per §8C.2: no architect / PM sign at batch level. Architect + PM sign at T-end via closure report (kickoff §4 #10 / criterion #10) — lands at T4d close IF T4d remains the final T4 batch. + +--- + +## §9 Change log + +| Date | Change | Author | +|---|---|---| +| 2026-04-25 | Initial T4d mini-plan v0.1 drafted from kickoff v0.3 (§1–8 + §9 RATIFIED 2026-04-25). 4 batches per kickoff §3 sign: T4d-1 substrate hardening + structured failure kind (~150 LOC + ~80 tests, no G-1) → T4d-2 replica recovery apply gate (~250 LOC + ~250 tests, no G-1, V3-native spec from round-43/44) → T4d-3 R+1 + engine-driven recovery wiring (~250 LOC + ~150 tests, **G-1 required** per §9.5 split) → T4d-4 lifecycle + integration (~150 LOC + ~200 tests, no G-1). Incorporates §9 walshipper architecture: T4d-3 is bounded V2 muscle PORT (5 items) + V3-native items (5 items) per architect §9.5 binding; T4d-3 is NOT a wholesale `WALShipper` port. Q1 engine-owned retry, Q2 lane implicit from handler context, Q3 transport↔storage contract-only inscribed via `INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` + `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY`. BlockStore walHead pre-T4d hotfix already landed (`f6084ee`); §4 #8 acceptance criterion shows DONE. Predicates: T4c closed + round-43/44 + kickoff §1–9 RATIFIED + hotfix landed. T-end three-sign at T4d close conditional on no T4e per kickoff §4 #10. Total ~800 LOC production + ~680 LOC tests across 4 batches. | sw | +| 2026-04-25 (v0.2) | Architect v0.1 review absorbed. **HIGH BLOCKER FIX**: structured failure kind now end-to-end typed contract — `RecoveryFailureKind` enum + `RecoveryFailure` typed-error in storage; new `FailureKind storage.RecoveryFailureKind` field on **`adapter.SessionCloseResult`** AND **`engine.SessionClosedFailed`**; engine `applySessionFailed` branches on `e.FailureKind`, NOT on `containsAny(reason, ...)` substring; `Reason`/`FailReason` strings explicitly downgraded to "DIAGNOSTIC TEXT ONLY — engine MUST NOT parse"; new fence test `TestEngine_SessionFailed_NoMoreSubstringMatch` greps source for `"WAL recycled"` substring. Cross-package import note: typed-kind field doesn't widen dependency surface beyond existing storage-recovery-contract import (T4c-2 already established). **MED FIX**: every `LogicalStorage` impl now has explicit `AppliedLSNs()` body — walstore via `dm.snapshot()`, smartwal via `ring.scanValid()` reduction, BlockStore explicitly returns `(nil, ErrAppliedLSNsNotTracked)` (new sentinel); gate falls back to session-only map on the sentinel. **LOW FIX**: §0 PORT/V3-native table gains 5th V3-native row (package-boundary discipline / `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY`) per kickoff §9.5. **QA test rename**: `TestApplyGate_BarrierAchievedLSN_DerivesFromCoverageNotData` → `TestApplyGate_BarrierCompletion_BarrierAuthoritativeWithCoverageContribution` (avoids misread as conflicting with T4c-2 `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN`). **QA NOTE 2 procedural absorbed**: §7.1 G-1 fence now binds T4d-3 PR commits to reference signed G-1 by hash (`Refs G-1 sign: `). T4d-1 test count grows from 8 to 10 (adds typed-kind end-to-end + event-shape fence). T4d-1 LOC est revised slightly upward (~170 prod + ~100 tests) due to adapter + engine event-struct contract additions. | sw | +| 2026-04-25 (v0.3) | Architect v0.2 dependency blocker absorbed. v0.2's typed-kind direction was correct but put `storage.RecoveryFailureKind` directly on engine events, which would create a new `core/engine → core/storage` dependency. v0.3 fixes the boundary: storage owns substrate-side `StorageRecoveryFailureKind` + `RecoveryFailure`; transport extracts that with `errors.As` and maps it to **engine-owned** `engine.RecoveryFailureKind`; `adapter.SessionCloseResult.FailureKind` uses `engine.RecoveryFailureKind`; `engine.SessionClosedFailed.FailureKind` uses the local engine type. `core/engine` MUST NOT import `core/storage`; `Reason`/`FailReason` remain diagnostic-only text. Test list label corrected to 10 required and mapping tests now explicitly cover storage-kind → engine-kind translation. | architect | +| 2026-04-25 (v0.4) | Architect Path B ratification absorbed (round-46+). T4d-3 `44c60dd` deferred engine-driven recovery wiring (`ReplicationVolume↔adapter` + `WithEngineDrivenRecovery()` real binding + full L2 matrix); architect chose Path B = fold into T4d-4 as **HARD CLOSE GATE** (not soft carry). T4d-3 §2.3 closure scope clarified as "R+1 bounded catch-up PORT + boundary fences" (5 G-1 PORT items + 2 hidden invariants + Q3 fence + CARRY-T4D-LANE-CONTEXT-001 Option B); engine-driven set explicitly relocated to T4d-4. T4d-4 §2.4 scope expanded with Path B fold rows (3 production sites + 5 tests relocated/added); LOC est revised ~150/~200 → ~300/~350. T4d HARD CLOSE GATE inscribed: cannot close until `WithEngineDrivenRecovery()` is REAL (not stub), `ReplicationVolume↔adapter` wiring runs end-to-end, QA #8 (`TestComponent_LastSentMonotonic_AcrossRetries_FullForm`) passes, and acceptance criterion #3 (full L2 matrix engine→adapter→executor end-to-end) satisfied — same bar T4c narrowed and T4d-3 deferred per Path B. **Process rule** (Issue 2(a) ratification): G-1 docs land in `seaweedfs` FIRST; sw implementation references committed hash via `Refs G-1 sign: `. T4d-3 G-1 doc landed at `seaweedfs@80036404c` (this revision). Architect explicitly rejected Path C (defer to G5) — "G5 should not inherit the same full-L2 gap again." | architect | +| 2026-04-25 (v0.5) | Architect round-47 ruling absorbed (post-T4d-3 production-readiness scope conversation). **T4d-4 HARD CLOSE GATE expanded with rebuild path (4th item)**: catch-up engine-driven wiring alone is insufficient for T4d close; rebuild path must also be engine-driven end-to-end. Minimum: catch-up failure/budget exhausted → engine emits actual rebuild command (not just `Decision = Rebuild` state); rebuild session completion/failure → returns to engine; rebuild failure terminal explicitly defined; L2 covers real path (test observes actual rebuild session, not just state transition). 4 new tests added in §2.4 Tests block. **Engine recovery state persistence (E1)**: architect explicitly EXCLUDED from T4d-4 implementation gate; instead inscribed as **G5-DECISION-001** named decision record in §5. G5 must choose Path A (persist) vs Path B (rebuild from probe after restart) explicitly; T4d-4 obligation is to keep engine state structure decision-A-compatible (serializable) so either path stays open. **Lane handler-context (CARRY-T4D-LANE-CONTEXT-001)** correction: NOT promoted to G5 hard blocker (it's already explicitly carried via Option B test-skip with non-claim); stays at T4e/post-G5 per existing inscription. Scope map clarified: T4d-4 = catch-up + rebuild engine-driven wiring; G5 = multi-replica mixed states + m01 + G5-DECISION-001 + minimal metrics/backpressure assessment; post-G5/T4e = lane fix + snapshot catch-up + protocol negotiation + auth/encryption. | architect |