mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-21 22:56:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

T4d batch close artifact + mini-plan v0.5 (architect-accepted)

Browse Source 
Two artifacts landing together to close T4 batch series:

1. v3-phase-15-t4d-closure-report.md (NEW)
   QA single-sign artifact for T4d batch close per §8C.2; architect
   T-end three-sign per §8C.1 (T4d IS final T4 batch — confirmed at
   round-48 review). Round-48 + round-49 corrections incorporated:
   - Part C commit hash bound to e642ae8 throughout
   - CARRY-T4D-LANE-CONTEXT-001 bind point = post-G5 hardening
     backlog (not T4e — consistent with "T-end at this close")
   - §H Finding #1 reworded — walstore HAS background flusher
     (walstore.go:189-190); QA's earlier "caller-driven" was wrong
   - §H Finding #3 RESOLVED at a0be6d5 (T2A NVMe race fixed +
     m01 -race ×50 PASS)
   - 16 invariants pinned (added 2 named for part C bug fixes:
     INV-REPL-FAILED-SESSION-KIND-DRIVES-ESCALATION +
     INV-REPL-REBUILD-ESCALATION-STICKY-UNTIL-TERMINAL)
   - 22/22 packages green under -race on m01 (post-a0be6d5)

2. v3-phase-15-t4d-mini-plan.md (NEW — was uncommitted across
   v0.1 → v0.5 evolution)
   Final v0.5 incorporates: architect Path B fold; round-47
   rebuild path engine-driven HARD GATE expansion; G5-DECISION-001
   named decision record; 4-batch shape ratified; T4d-3 G-1 binding.

Active forward-carries (post-G5 hardening backlog):
  - CARRY-T4D-LANE-CONTEXT-001 — replace TargetLSN==1 caller shim
    with true handler/session-context lane signal
  - G5-DECISION-001 — engine recovery state behavior across
    primary restart (Path A persist vs Path B rebuild-from-probe)

G5 collective close items (NOT post-G5):
  - m01 hardware first-light for replicated write path
  - Multi-replica concurrent live + recovery scenarios
  - walstore flusher cadence verification + tuning policy
  - Minimal metrics/backpressure assessment
  - G5-DECISION-001 architect resolution

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
pingqiu
2026-04-26 00:21:20 -07:00
co-authored by Claude Opus 4.7
parent 80036404ce
commit 2ee12b2c14
2 changed files with 1048 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sw-block/design/v3-phase-15-t4d-closure-report.md
+352
View File
@@ -0,0 +1,352 @@
# V3 Phase 15 T4d — Closure Report
**Date**: 2026-04-25
**Status**: DRAFT — awaiting QA single-sign batch close per §8C.2 + architect T-end three-sign per §8C.1 (kickoff §4 #10: T-end at T4d close IF T4d remains final T4 batch — confirmed: T4d IS the final T4 batch, no T4e)
**Gate**: T4d batch (FINAL T4 batch) within G5 (replicated write path) — T4 T-end three-sign lands at this close
**Predecessor**: T4c batch QA-single-signed + architect-accepted at `c910464a9`
**Successor**: G5 collective close (multi-replica + m01 first-light + G5-DECISION-001 + production-readiness gates)
---
## §A Batch history
| Batch | Commit | Status | LOC (+/−) | Delivery |
|---|---|---|---|---|
| T4d kickoff (architect rounds 1–8 of mini-plan) | doc-only | RATIFIED 2026-04-25 | — | Single-T4d scope; Option C hybrid; BlockStore hotfix lift-out; defense-in-depth in T4d-1; 4-batch shape (T4d-1→T4d-2→T4d-3→T4d-4); T4d-3 G-1 required; m01 to G5 collective; T-end at T4d close if no T4e |
| BlockStore walHead pre-T4d hotfix | `f6084ee` | CLOSED | 1 prod line + 11 godoc + un-skipped regression test | Substrate-internal one-liner `if lsn > s.walHead { s.walHead = lsn }`; closes round-43 production storage-contract violation independently of T4d-1 |
| **T4d-1** substrate hardening + structured `RecoveryFailureKind` | **`1edeb36`** | CLOSED | +440 / −0 across 12 files | Storage-side `StorageRecoveryFailureKind` + `RecoveryFailure` typed-error; `LogicalStorage.AppliedLSNs()` interface (Option C hybrid); `BlockStore` returns `ErrAppliedLSNsNotTracked` explicitly; walstore uses dirty-map snapshot; smartwal uses ring last-writer-wins reduction; engine `RecoveryFailureKind` enum (separate from storage's enum); `transport.classifyRecoveryFailure` boundary mapper; substring match REMOVED from engine; 10 new tests |
| T4d-1 follow-up (HARD GATE tests) | `d6b1890` | CLOSED | +120 / 0 (1 file) | `TestT4d1_TargetNotReached_DistinctKindFromWALRecycled` + `TestT4d1_StorageFailureKindMapper_AllKnownKinds` per round-46 architect HARD GATE before T4d-3 G-1 sign |
| **T4d-2** replica recovery apply gate | **`bd2de99`** | CLOSED | +630 / −15 across 6 files | `ReplicaApplyGate` lane-aware (initial impl had payload-derived discrimination — REWORKED at `01f4ab9`); 2-map split (`liveTouched` + `recoveryCovered`); `appliedLSN` map with Option C hybrid seed; recovery-lane stale-skip + coverage-advance; live-lane fail-loud; component framework `WithApplyGate()` builder; 16 new tests + 1 un-skipped adversarial |
| T4d-2 follow-up (lane-pure rework) | `01f4ab9` | CLOSED | +180 / −90 across 5 files | Round-46 architect rework: gate becomes lane-pure (`ApplyRecovery` + `ApplyLive` lane-explicit methods); payload-derived `isRecoveryLane(lineage)` REMOVED; lane discrimination moved to caller (transport replica handler) as TRANSITIONAL CALLER-SIDE SHIM with explicit TODO citing `CARRY-T4D-LANE-CONTEXT-001`; new regression tests `TestApplyGate_RecoveryWithTargetLSN1_RoutesToRecoveryLane` + `TestApplyGate_LiveWithTargetLSN100_RoutesToLiveLane`; old test renamed to `TestComponent_LanePurity_CallerControlsDispatch` |
| T4d-2 doc fix (post-round-46 stale godoc) | `a63ae9b` | CLOSED | +20 / −10 across 2 files | `apply_gate.go:38-43` rewrite to reflect lane-pure model; `replica.go` constructor godoc updated to lane-explicit hook shape per QA pre/with-T4d-3 fix requirement |
| **T4d-3** R+1 threading + boundary fences | **`44c60dd`** | CLOSED per architect Path B | +250 / −16 across 5 files | `engine.StartCatchUp.FromLSN` field; engine emits `Recovery.R + 1` (G-1 §6.1 Option A — engine adds `+1`); per-call deadline + last-sent-monotonic + retry budget semantics ported from V2; substrate scan from R+1 instead of 1; `core/transport/import_discipline_test.go` Q3 fence (`INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY`); `CARRY-T4D-LANE-CONTEXT-001` Option B test-skip with explicit forward-binding; 11 new tests including round-46 ADDITIONS (RetryAfterReplicaAdvanced + 3 substrate-bound bandwidth + BlockStoreOverShipsExpected fence). Engine-driven recovery wiring DEFERRED to T4d-4 per Path B |
| **T4d-4 part A** RecoveryMode + Stop lifecycle | **`f88d097`** | CLOSED | per part A | `RecoveryMode()` interface method on substrates (replaces duck-typed `CheckpointLSN` probe); component framework wraps forward `RecoveryMode()` correctly (closes round-40 known limitation); `ReplicationVolume.Stop` lifecycle regression (BUG-005 non-repeat); `INV-REPL-LIFECYCLE-HANDLE-BORROWED-001` inscribed |
| **T4d-4 part B** engine-driven recovery wiring + round-47 + G5-DECISION-001 | **`812d3fa`** | CLOSED | +590 / −31 across 6 files | `WithEngineDrivenRecovery()` becomes REAL (not stub); `ReplicationVolume↔adapter` wiring runs engine retry loop end-to-end; round-47 architect change: catch-up exhaustion DIRECTLY emits StartRebuild (was: cleared Decision + emit Degraded only); rebuild MaxRetries=0 → terminal failure path; `INV-REPL-CATCHUP-EXHAUSTION-ESCALATES-TO-REBUILD` + `INV-REPL-REBUILD-FAILURE-TERMINAL` inscribed; G5-DECISION-001 obligation: ReplicaState struct JSON round-trip clean; 9 new tests + 1 existing test updated |
| **T4d-4 part C** full L2 matrix + 2 engine bug fixes | **`e642ae8`** | CLOSED | ~150 / −20 across 3 files + 1 new test file | HARD CLOSE GATE #3: full L2 matrix engine→adapter→executor end-to-end; 3 new L2 tests (`TestT4d4_FullL2_CatchupRetryExhausted_EscalatesToRebuild` + `_WALRecycled_EscalatesImmediate` + `_LastSentMonotonic_AcrossRetries`). **2 engine bugs surfaced + fixed**: (Bug 1) catch-up exhaustion missed escalation when stray auto-probe re-classified `Decision` mid-flight — fix keys on `st.Session.Kind == SessionCatchUp` truth instead of mutable `Decision` field; (Bug 2) WALRecycled escalation never emitted StartRebuild and could be downgraded by next probe — new `RecoveryTruth.RebuildPinned` sticky bool with cleared-on-success semantic forces rebuild override regardless of R/S/H |
All commits on `feature/sw-block` (seaweedfs) → `phase-15` (seaweed_block).
---
## §B Signed acceptance delta vs kickoff §4
The kickoff §4 acceptance bar (RATIFIED 2026-04-25) is **fully satisfied** at this close. Recording each criterion's status explicitly:
| Kickoff §4 criterion | Status | Evidence |
|---|---|---|
| #1 All proposed batches merged | ✅ MET | T4d-1 + T4d-2 + T4d-3 + T4d-4 part A (`f88d097`) + part B (`812d3fa`) + part C (`e642ae8`) all on `phase-15` |
| #2 Unit tests green per task | ✅ MET | 22 packages green at HEAD on QA Windows workspace + m01 Linux -race |
| #3 Full L2 subprocess matrix green — engine→adapter→executor end-to-end | ✅ MET | T4d-4 part C lands 3 new L2 tests; closes the bar T4c narrowed and T4d-3 deferred per Path B fold |
| #4 L3 m01 deferred to G5 collective | ✅ acceptable per §2.6 | NOT a T4d criterion; m01 first-light for replicated write path lands at G5 collective sign across T4a/b/c/d (production-sized substrate smoke at 1 GiB DID run on m01 — see §G) |
| #5 T4a/T4b/T4c invariants forward-carry verified | ✅ MET | Forward-carry checklist v0.2 all 19 active invariants green under -race on m01; 30× stress on engine-driven flow clean |
| #6 Active invariant promotions | ✅ MET | See §F (12 inscribed invariants from T4d batches; `INV-REPL-CATCHUP-WITHIN-RETENTION-001` un-pinned at T4c → PORTED at T4d via apply gate + R+1) |
| #7 Engine sentinel decoupling — substring match REMOVED + structured kind in production | ✅ MET at T4d-1 | `core/engine/apply.go` substring helpers (`isWALRecycledFailure`, `isStartTimeoutFailure`, `containsAny`) removed; `TestEngine_SessionFailed_NoMoreSubstringMatch` source-grep fence passes |
| #8 BlockStore walHead regression fixed in production | ✅ MET (pre-T4d hotfix) | `f6084ee` lifted out per architect §2.5 #2 lift-out approval |
| #9 QA single-sign at T4d close per §8C.2 | pending | This document is the QA single-sign artifact |
| #10 T4 T-end three-sign per §8C.1 lands at T4d close IF no T4e | ✅ APPLICABLE | Confirmed: T4d IS the final T4 batch, no T4e needed; T-end three-sign lands at this close |
| #11 No §8C.3 escalation triggers fired | ✅ MET | Two real engine bugs surfaced by T4d-4 part C HARD GATE #3 (validating round-47 architect ruling that full L2 matrix is non-negotiable); both fixed in same cycle — §8C governance working as intended, NOT escalation |
**Architectural improvements signed during T4d (with architect ratification):**
| Improvement | Architect-endorsed at | Status |
|---|---|---|
| 2-enum split: `storage.StorageRecoveryFailureKind` ↔ `engine.RecoveryFailureKind` (engine zero-imports storage) | T4d-1 round-46 review | ✅ ENDORSED — "the right v0.3 boundary fix" |
| Lane-pure apply gate API (`ApplyRecovery` + `ApplyLive` lane-explicit) | T4d-2 round-46 rework | ✅ RATIFIED — replaces initial payload-derived discrimination |
| Round-47 catch-up exhaustion DIRECTLY emits StartRebuild | T4d-4 part B round-47 | ✅ RATIFIED — closes "Decision flipped but no actual rebuild fired" gap |
| `RecoveryTruth.RebuildPinned` sticky override (cleared on success) | T4d-4 part C bug fix #2 | ✅ implicit ratification via test pass — surfaces the right architectural correction |
| Process rule: G-1 docs land in seaweedfs FIRST; sw references committed hash | T4d round-46 Issue 2(a) | ✅ INSCRIBED in mini-plan §7.1; T4d-3 G-1 sign hash = `seaweedfs@80036404c` |
---
## §C Changelist by file (cumulative across T4d)
### T4d-1 + d6b1890 follow-up — substrate hardening + structured kind
| File | LOC | Notes |
|---|---|---|
| `core/storage/recovery_contract.go` | +180/−0 | `StorageRecoveryFailureKind` enum (3 kinds) + `RecoveryFailure` typed-error struct with `Unwrap()` for `errors.Is(_, ErrWALRecycled)` migration compat; `ErrAppliedLSNsNotTracked` sentinel; `NewWALRecycledFailure` + `NewSubstrateIOFailure` helpers |
| `core/storage/logical_storage.go` | +30/−0 | `AppliedLSNs() (map[uint32]uint64, error)` interface method (Option C hybrid) |
| `core/storage/walstore.go` + `walstore_recovery.go` | +90/−15 | `AppliedLSNs()` impl via `dm.snapshot()`; `ScanLBAs` wraps recycle errors as typed `RecoveryFailure{Kind: WALRecycled}` |
| `core/storage/smartwal/store.go` + `recovery_scan.go` | +75/−10 | `AppliedLSNs()` impl via `ring.scanValid()` last-writer-wins reduction; same recycle-wrap pattern |
| `core/storage/store.go` | +15/−0 | BlockStore `AppliedLSNs()` returns `(nil, ErrAppliedLSNsNotTracked)` explicitly |
| `core/engine/events.go` | +10/−0 | `SessionClosedFailed.FailureKind engine.RecoveryFailureKind` field added |
| `core/engine/apply.go` | +25/−45 | `applySessionFailed` branches on `e.FailureKind == RecoveryFailureWALRecycled`; substring helpers (`isWALRecycledFailure`, `isStartTimeoutFailure`, `containsAny`) REMOVED |
| `core/engine/recovery_test.go` | +60/−15 | `TestSessionFailed_WALRecycled_EscalatesToRebuild` rewritten to use typed kind |
| `core/engine/substring_removal_fence_test.go` | +60 NEW | `TestEngine_SessionFailed_NoMoreSubstringMatch` source-grep fence with comment-stripping helper |
| `core/adapter/normalize.go` + `types.go` | +25/−5 | `SessionCloseResult.FailureKind` typed field; `NormalizeSessionClose` propagates |
| `core/adapter/typed_failure_kind_test.go` | +75 NEW | 6 subtest end-to-end + success-path sanity |
| `core/transport/recovery_session.go` | +50 NEW | `classifyRecoveryFailure` boundary mapper using `errors.As` |
| `core/transport/executor.go` | +15/−5 | `finishSession` populates typed `FailureKind` |
| `core/transport/classifier_followup_test.go` (`d6b1890`) | +120 NEW | HARD GATE before T4d-3 G-1 sign: `TestT4d1_TargetNotReached_DistinctKindFromWALRecycled` + `TestT4d1_StorageFailureKindMapper_AllKnownKinds` |
### T4d-2 + 01f4ab9 + a63ae9b — apply gate (lane-pure)
| File | LOC | Notes |
|---|---|---|
| `core/replication/apply_gate.go` | +330 NEW | `ReplicaApplyGate` with per-session state; `ApplyRecovery` + `ApplyLive` lane-explicit methods (post-`01f4ab9`); 2-map split (`liveTouched` + `recoveryCovered`); `appliedLSN` map seeded from substrate via Option C hybrid |
| `core/replication/apply_gate_test.go` | +280 NEW | 13 unit tests; lane-discriminator regression tests (`TestApplyGate_RecoveryWithTargetLSN1_RoutesToRecoveryLane` + `TestApplyGate_LiveWithTargetLSN100_RoutesToLiveLane` per `01f4ab9`) |
| `core/transport/replica.go` | +60/−5 | `MsgShipEntry` handler dispatches via `ApplyHook` (lane-explicit post-`01f4ab9`); transitional caller-side TargetLSN==1 shim with explicit TODO citing `CARRY-T4D-LANE-CONTEXT-001` |
| `core/replication/component/cluster.go` | +40/−5 | `WithApplyGate()` builder + `Cluster.ApplyGate(idx)` accessor |
| `core/replication/component/apply_gate_scenarios_test.go` | +160 NEW | 4 integration scenarios via component framework + un-skipped `TestComponent_Adversarial_StaleEntryDoesNotRegress` from T4c skip-pile |
### T4d-3 — R+1 threading + boundary fences
| File | LOC | Notes |
|---|---|---|
| `core/transport/catchup_sender.go` | +13/−5 | `ScanLBAs(fromLSN, ...)` instead of `ScanLBAs(1, ...)` per command-borne FromLSN |
| `core/transport/executor.go` | +5/−2 | `BlockExecutor.StartCatchUp` signature gains `fromLSN uint64` |
| `core/adapter/executor.go` + test stubs | +30/−15 | Interface signature update; 7 test executor stubs updated |
| `core/engine/commands.go` | +5/−0 | `engine.StartCatchUp` command struct gains `FromLSN uint64` |
| `core/engine/apply.go` | +10/−2 | Emission populates `FromLSN = Recovery.R + 1` (G-1 §6.1 Option A — engine adds +1) |
| `core/engine/t4d3_rplus1_test.go` | +135 NEW | 3 engine R+1 emit tests including hidden invariant pins |
| `core/transport/t4d3_rplus1_test.go` | +110 NEW | 3 transport tests including round-46 ADDITION 2 substrate-bound naming (`_Walstore` / `_Smartwal` / `_BlockStoreOverShipsExpected`) |
| `core/transport/import_discipline_test.go` | +80 NEW | `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY` Q3 fence: parses sources, fails on substrate-internal imports |
| `core/replication/component/t4d3_scenarios_test.go` | +200 NEW | 4 scenarios incl. `TestT4d3_RetryAfterReplicaAdvanced_OverScansHandledByApplyGate` (round-46 ADDITION 1) + `TestT4d3_RecoveryTargetLSN1_KnownGap` (CARRY-T4D-LANE-CONTEXT-001 Option B skip-with-forward-binding) |
### T4d-4 part A — RecoveryMode + Stop lifecycle
| File | LOC | Notes |
|---|---|---|
| `core/storage/recovery_contract.go` | +20/−0 | `RecoveryMode()` substrate interface method |
| `core/storage/{walstore,smartwal,store}.go` | +25/−10 | Per-substrate `RecoveryMode()` impl |
| `core/transport/catchup_sender.go` | +5/−10 | Replace duck-typed `CheckpointLSN()` probe with `RecoveryMode()` call |
| `core/replication/component/faults.go` | +15/−5 | Wraps forward `RecoveryMode()` correctly (closes round-40 known limitation) |
| `core/replication/volume.go` | +60/−10 | `ReplicationVolume.Stop` lifecycle (BUG-005 non-repeat); `INV-REPL-LIFECYCLE-HANDLE-BORROWED-001` |
### T4d-4 part B — engine-driven recovery wiring + round-47 + G5-DECISION-001
| File | LOC | Notes |
|---|---|---|
| `core/replication/component/cluster.go` | +98/−5 | `WithEngineDrivenRecovery()` REAL (was stub at T4c); adapter installed per replica; drive helpers |
| `core/adapter/adapter.go` + `adapter_test.go` | +25/−2 | Adapter wired to executor; engine emit→dispatch→executor→callback round-trip works |
| `core/engine/apply.go` | +42/−3 | Round-47: catch-up exhaustion DIRECTLY emits StartRebuild (was: cleared Decision + emit Degraded only); rebuild MaxRetries=0 → terminal failure path |
| `core/engine/recovery_test.go` | +99/−15 | 4 new round-47 tests + existing `TestSessionFailed_NonRecycled_RetriesUntilBudget` updated |
| `core/engine/g5_decision_001_serializability_test.go` | +95 NEW | `TestG5Decision001_ReplicaState_RoundTripJSON` + `_ZeroValueStable` — fence: ReplicaState struct stays serializable so G5-DECISION-001 Path A (persistence) remains structurally open |
| `core/replication/component/t4d4_engine_driven_test.go` | +263 NEW | 5 integration tests including `TestT4d4_RoundTrip_AssignmentToProbeToCatchUp_EngineDriven` + `TestT4d4_CatchupBudgetExhausted_EngineEmitsRebuild` + `TestT4d4_WithEngineDrivenRecovery_IsReal` |
### T4d-4 part C — full L2 matrix + 2 engine bug fixes (`e642ae8`)
| File | LOC | Notes |
|---|---|---|
| `core/engine/state.go` | +20/−5 | `RecoveryTruth.RebuildPinned bool` field with invariant doc |
| `core/engine/apply.go` | +50/−15 | **Bug #1 fix**: exhaustion keys on `st.Session.Kind == SessionCatchUp` (truth = what session actually failed) instead of mutable `Decision` field. **Bug #2 fix**: sticky-rebuild gate in `decide()`; WALRecycled escalation emits StartRebuild AND pins `RebuildPinned=true`; exhaustion emits StartRebuild AND pins; rebuild SessionCompleted clears the pin |
| `core/engine/recovery_test.go` | +30/−5 | Fixtures updated to set `Session.Kind: SessionCatchUp` per Bug #1 fix |
| `core/replication/component/t4d4_full_l2_matrix_test.go` | +280 NEW | 3 HARD GATE #3 tests: `TestT4d4_FullL2_CatchupRetryExhausted_EscalatesToRebuild` + `_WALRecycled_EscalatesImmediate` + `_LastSentMonotonic_AcrossRetries` (closes QA #8 LastSentMonotonic full form) |
### Cumulative T4d LOC
T4d-1 + T4d-2 + T4d-3 + T4d-4 (production + tests, including pre-T4d hotfix `f6084ee`): **~3500 / −300 across 50+ distinct files**
Component framework + QA: small additions (`WithApplyGate`, `WithEngineDrivenRecovery`, fault wraps `RecoveryMode()` forwarding)
---
## §D Review points — code spans worth architect's eye
### D-1. Engine ↔ storage 2-enum split with explicit boundary mapper — `core/engine/events.go`, `core/storage/recovery_contract.go`, `core/transport/recovery_session.go:35`
Architect-endorsed at T4d-1 round-46 review as "the right v0.3 boundary fix" (stronger than mini-plan v0.2's single shared enum approach). The architectural property: **engine has ZERO imports from storage** (verified by build at `1edeb36`). Maintenance discipline: any future `StorageRecoveryFailureKind` addition requires a paired `engine.RecoveryFailureKind` mapping in `transport.classifyRecoveryFailure`; `TestT4d1_StorageFailureKindMapper_AllKnownKinds` is the fence.
### D-2. Apply gate lane-pure API + transitional caller-side shim — `core/replication/apply_gate.go:120,142` + `core/transport/replica.go:13-18,156-165`
Round-46 architect rework. Gate exposes `ApplyRecovery` + `ApplyLive` lane-explicit methods; gate itself never inspects `lineage.TargetLSN`. The lane discrimination payload-sniffing was moved one layer up to the transport replica handler as a TRANSITIONAL CALLER-SIDE SHIM with explicit TODO citing `CARRY-T4D-LANE-CONTEXT-001`. The architectural fence (gate ≠ payload-sniffer) is durable; the transitional shim is named-carry-bound to **post-G5 protocol hardening** (G5 hardening backlog).
### D-3. Round-47 catch-up exhaustion DIRECTLY emits StartRebuild + RebuildPinned sticky override — `core/engine/apply.go:324,488,527,535,637`, `core/engine/state.go:105,120`
Round-47 architect ruling moved escalation from "set `Decision = Rebuild` + emit Degraded" to "directly emit StartRebuild command" (validating that the engine drives the rebuild, not just records intent). T4d-4 part C surfaced two real bugs in the implementation:
- **Bug #1**: exhaustion missed escalation when stray auto-probe re-classified `Decision` mid-flight → fix keys on `st.Session.Kind == SessionCatchUp` (truth = what session actually failed)
- **Bug #2**: WALRecycled escalation set `Decision = Rebuild` but never emitted StartRebuild, and could be downgraded by next probe → new `RebuildPinned bool` sticky override; cleared by successful rebuild SessionCompleted
The fixes are **structural corrections** (use the right truth source; make the decision sticky against transient probe noise), not bandaids. The fact that HARD GATE #3 surfaced them validates architect's round-47 ruling that the full L2 matrix is non-negotiable.
### D-4. Per-LBA stale-skip with coverage-advance — `core/replication/apply_gate.go` `ApplyRecovery`
Round-43/44 architect lock: recovery-stream stale entries are valid duplicates and must be skipped as data writes while still counted as recovery-stream coverage. Live-lane stale entries are abnormal and must not mutate data; they should be skipped/rejected and must not advance recovery coverage. The 2-map split (`liveTouched` + `recoveryCovered`) realizes this; Option C hybrid `appliedLSN[LBA]` source seeds from substrate `AppliedLSNs()` query.
### D-5. R+1 threading at engine — `core/engine/apply.go` (FromLSN populated as `Recovery.R + 1`)
G-1 §6.1 Option A architect ratification: engine populates `StartCatchUp.FromLSN = Recovery.R + 1` at command emit time; sender does NOT add `+1` (avoids double-add). The "+1 to skip already-applied LSN" semantic lives at engine, not transport. Pinned by `INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1` + `INV-REPL-CATCHUP-FROMLSN-FROM-ENGINE-STATE-NOT-PROBE`.
### D-6. G5-DECISION-001 obligation — engine `ReplicaState` JSON round-trip clean — `core/engine/g5_decision_001_serializability_test.go`
T4d-4 part B obligation per architect round-47 ruling: keep engine state structure decision-A-compatible (serializable) so G5 can choose Path A (persist) vs Path B (rebuild from probe after restart) without structural rewrite. **Test fence**: `TestG5Decision001_ReplicaState_RoundTripJSON` + `_ZeroValueStable` ensure no future struct change introduces unserializable types.
---
## §E Delivery summary
1. **2-enum boundary** (T4d-1) — `storage.StorageRecoveryFailureKind` + `engine.RecoveryFailureKind` + `transport.classifyRecoveryFailure` mapper. Engine zero-imports storage. Substring matching REMOVED.
2. **Option C hybrid `AppliedLSNs()`** (T4d-1) — substrate exposes per-LBA latest-applied LSN where tracked (walstore via dirty-map snapshot; smartwal via ring last-writer-wins reduction); BlockStore returns `ErrAppliedLSNsNotTracked` explicitly; gate falls back to in-memory session map.
3. **Lane-pure apply gate** (T4d-2 + `01f4ab9` + `a63ae9b`) — `ApplyRecovery` + `ApplyLive` lane-explicit; 2-map split; per-LBA stale-skip with coverage-advance on skip; live-lane fail-loud. Transitional caller-side TargetLSN shim carries via `CARRY-T4D-LANE-CONTEXT-001`.
4. **R+1 threading + boundary fences** (T4d-3) — engine emits `Recovery.R + 1`; substrate scans from R+1 not 1; 5 G-1 PORT items; 2 hidden invariants; Q3 import-discipline fence; CARRY-T4D-LANE-CONTEXT-001 Option B test-skip with explicit forward-binding.
5. **RecoveryMode() interface + Stop lifecycle** (T4d-4 part A) — substrates expose recovery mode via method (replaces duck-typed CheckpointLSN probe); `ReplicationVolume.Stop` BUG-005 non-repeat; `INV-REPL-LIFECYCLE-HANDLE-BORROWED-001`.
6. **Engine-driven recovery wiring + round-47** (T4d-4 part B) — `WithEngineDrivenRecovery()` becomes REAL; `ReplicationVolume↔adapter` wiring runs engine retry loop end-to-end; round-47 catch-up exhaustion DIRECTLY emits StartRebuild; rebuild MaxRetries=0 terminal; G5-DECISION-001 ReplicaState serializability obligation.
7. **Full L2 matrix + 2 engine bug fixes** (T4d-4 part C) — HARD CLOSE GATE #3 closed; 2 real bugs surfaced + fixed during integration testing (Bug #1 stale-Decision-key; Bug #2 RebuildPinned sticky override).
---
## §F Invariants pinned at T4d close
Inscribed in catalogue §3.3 with full pin-test lists. Entries upgrade to `✓ PORTED T4d` on QA single-sign + architect T-end three-sign.
| Invariant | Origin | Pin location | Batch |
|---|---|---|---|
| `INV-REPL-CATCHUP-WITHIN-RETENTION-001` *(Path A)* | architect Item B | T4c un-pin → T4d-2 apply gate + T4d-3 R+1 — un-pin condition satisfied by combined apply gate (correctness) + R+1 (efficiency) | T4d-2 + T4d-3 |
| `INV-REPL-NO-PER-LBA-DATA-REGRESSION` *(goal)* | round-43 lock | `apply_gate.go` `ApplyRecovery` + adversarial test suite | T4d-2 |
| `INV-REPL-RECOVERY-STALE-ENTRY-SKIP-PER-LBA` *(mechanism)* | round-43 lock | `apply_gate.go` `ApplyRecovery` per-LBA `appliedLSN` check | T4d-2 |
| `INV-REPL-RECOVERY-COVERAGE-ADVANCES-ON-SKIP` | round-44 refinement | `apply_gate.go` `ApplyRecovery` skip-path advances `recoveryCovered` | T4d-2 |
| `INV-REPL-LIVE-LANE-STALE-FAILS-LOUD` | round-44 refinement | `apply_gate.go` `ApplyLive` returns error on stale; never advances `recoveryCovered` | T4d-2 |
| `INV-REPL-RECOVERY-COVERAGE-RESTART-SAFE` | Option C hybrid | `apply_gate.go` `initSessionLocked` queries `store.AppliedLSNs()` at session start | T4d-2 |
| `INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` | Q2 + round-46 re-emphasis | `apply_gate.go` lane-explicit methods + `replica.go` caller-side dispatch (TRANSITIONAL — CARRY-T4D-LANE-CONTEXT-001 active) | T4d-2 (gate API) + T4d-3 (transport caller) |
| `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY` | Q1+Q3 + T4d-1 strengthening | `core/engine/` zero storage imports; `core/transport/import_discipline_test.go` source-grep fence | T4d-1 (engine purity) + T4d-3 (transport fence) |
| `INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1` | T4d-3 G-1 §5 | `core/engine/apply.go` engine emits `Recovery.R + 1`; `t4d3_rplus1_test.go` | T4d-3 |
| `INV-REPL-CATCHUP-FROMLSN-FROM-ENGINE-STATE-NOT-PROBE` | T4d-3 G-1 §5 | `core/engine/apply.go` reads engine state, not raw probe payload; `t4d3_rplus1_test.go` | T4d-3 |
| `INV-REPL-LIFECYCLE-HANDLE-BORROWED-001` | T4d-4 part A | `ReplicationVolume.Stop` doesn't close substrate handle; BUG-005 non-repeat regression test | T4d-4 part A |
| `INV-REPL-CATCHUP-EXHAUSTION-ESCALATES-TO-REBUILD` | round-47 architect addition | `core/engine/apply.go:567` exhaustion emits StartRebuild + pins RebuildPinned; `recovery_test.go:301+325` | T4d-4 part B+C |
| `INV-REPL-REBUILD-FAILURE-TERMINAL` | round-47 architect addition | rebuild MaxRetries=0; first failure → terminal degraded; `recovery_test.go` | T4d-4 part B |
| **`INV-REPL-FAILED-SESSION-KIND-DRIVES-ESCALATION`** *(part C Bug #1 capture)* | T4d-4 part C bug fix | `core/engine/apply.go` exhaustion check keys on `st.Session.Kind == SessionCatchUp` (truth = what session actually failed), NOT on the mutable `Decision` field; resists stray auto-probes that re-classify Decision mid-flight; `TestT4d4_FullL2_CatchupRetryExhausted_EscalatesToRebuild` | T4d-4 part C |
| **`INV-REPL-REBUILD-ESCALATION-STICKY-UNTIL-TERMINAL`** *(part C Bug #2 capture)* | T4d-4 part C bug fix | `RecoveryTruth.RebuildPinned` sticky bool set by both WALRecycled escalation and exhaustion; `decide()` honors as override forcing `Decision=Rebuild` regardless of R/S/H; cleared by successful rebuild SessionCompleted or identity reset; ensures next probe cannot downgrade rebuild before terminal; `TestT4d4_FullL2_WALRecycled_EscalatesImmediate` | T4d-4 part C |
| `LastSentMonotonic_AcrossRetries` *(full cross-call form)* | T4c QA #8 carry | `TestT4d4_FullL2_LastSentMonotonic_AcrossRetries` (T4d-4 part C) — closes T4c Stage-2 deferred scenario | T4d-4 part C |
**Total inscribed at T4d close: 16 invariants** (14 new + 1 promotion + 1 QA-deferred-now-closed). Of the 14 new, 2 specifically capture the part C bug fixes (`INV-REPL-FAILED-SESSION-KIND-DRIVES-ESCALATION` + `INV-REPL-REBUILD-ESCALATION-STICKY-UNTIL-TERMINAL`) per architect round-48 review.
**Active carries (named, post-G5 hardening backlog):**
- `CARRY-T4D-LANE-CONTEXT-001` — replace transport caller-side TargetLSN==1 shim with true handler/session-context lane signal (Option B test-skip in place); bind point: **post-G5 protocol hardening (G5 hardening backlog)**
- `G5-DECISION-001` — engine recovery state behavior across primary restart (Path A persist vs Path B rebuild-from-probe; T4d-4 part B kept structure serializable)
---
## §G Test evidence
| Layer | At HEAD `e642ae8` (T4d-4 part C) |
|---|---|
| T4d-1 unit (typed kind + AppliedLSNs) | 10 new + d6b1890 follow-up 2 HARD GATE | green |
| T4d-2 unit (apply gate) | 16 new + 1 un-skipped | green |
| T4d-2 follow-up rework (`01f4ab9`) | 2 regression tests | green |
| T4d-3 unit + L2 (R+1 + boundary fences) | 11 new (3 engine + 3 transport + 1 import-discipline + 4 component) | green |
| T4d-4 part A (RecoveryMode + Stop) | per part A | green |
| T4d-4 part B (engine wiring + round-47 + G5-DECISION-001) | 9 new + 1 updated; 2 G5-DECISION-001 tests | green |
| T4d-4 part C (full L2 + bug fixes) | 3 new HARD GATE #3 tests | green |
| **Cumulative T4d-relevant tests** | **~70 new tests across T4d batches** | **all green at HEAD** |
| `core/replication` package | — | green under -race on m01 |
| `core/replication/component` package | — | green under -race on m01 |
| `core/transport` package | — | green under -race on m01 |
| `core/engine` package | — | green under -race on m01 |
| `core/storage` + `core/storage/smartwal` | — | green under -race on m01 |
| Full V3 suite under -race on m01 | **22/22 packages green** (post-`a0be6d5` test-fixture fix) | T2 NVMe `TestT2A_ConcurrentQueueStress` race resolved at `a0be6d5` during T4d close cycle; verified ×50 stress + full nvme package PASS — see §H Finding #3 |
**Smoke runs at QA workspace HEAD + m01 -race:**
```
$ go test -race -count=20 -timeout 5m -run 'TestT4d4_FullL2_' ./core/replication/component/
ok github.com/seaweedfs/seaweed-block/core/replication/component 12.523s
$ go test -race -count=1 -timeout 10m ./core/...
21 packages green; 1 pre-existing T2A NVMe race (separate bug)
# Post-`a0be6d5` re-run (2026-04-26):
$ CGO_ENABLED=1 go test -race -count=50 -run TestT2A_ConcurrentQueueStress ./core/frontend/nvme/
ok github.com/seaweedfs/seaweed-block/core/frontend/nvme 7.015s
$ CGO_ENABLED=1 go test -race -count=1 ./core/frontend/nvme/
ok github.com/seaweedfs/seaweed-block/core/frontend/nvme 22.237s
22/22 packages now genuinely green under -race on m01.
```
**Production-sized substrate smoke on m01 (1 GiB extent — see §H findings):**
| Substrate | Create 1 GiB | 20k writes (~78 MB) | ScanLBAs (1) full retention | AppliedLSNs |
|---|---|---|---|---|
| walstore (default 64 MiB WAL) | 11.5 ms | ⚠ WAL admission fires at write 14611 (~57 MiB) — see Finding #1 | n/a (test aborted) | n/a |
| smartwal (2 MiB ring) | 1.66 ms | 154.7 ms (~518 MB/s) | 211.9 ms (~94k entries/sec) | 20000 entries, max LSN=20000 |
**Per-batch substrate smoke regression check (T4d-1 baseline `1edeb36` vs current):**
- walstore: same WAL admission boundary at write ~14611 (no regression)
- smartwal: same throughput characteristics (78 MB write in ~129 ms; ScanLBAs ~204 ms; AppliedLSNs 20000 entries)
- Substrate behavior INVARIANT across T4d-1 → T4d-4-current (substrate code stable since T4d-1)
---
## §H Production-readiness findings (m01 1 GiB substrate smoke)
These findings are **not T4d implementation defects**. They are production-readiness gaps surfaced by realistic-config testing. Recording explicitly so they don't go silent into G5.
### Finding #1: walstore WAL admission backpressure observed under synthetic write-only workload
**Symptom**: `walstore` 1 GiB extent + default 64 MiB WAL + 20,000 sequential synchronous writes (no intermediate Sync) → WAL admission gate fires at write 14611 (~57 MiB consumed) with `storage: WAL admission: storage: WAL region full`.
**Honest scope**: m01 smoke ran a synthetic write-only workload with no observed checkpoint relief before the 64 MiB WAL filled. walstore HAS a background flusher (`walstore.go:189-190` `s.flusher = newFlusher(...); go s.flusher.run()`) and explicit WAL admission control (`walstore.go:335-341`); QA's earlier characterization "caller-driven, no background goroutine" was wrong (corrected per architect round-48 review). The actual question is whether the flusher's checkpoint cadence keeps up with the synthetic single-threaded loop's write rate, and what tuning policy production deployments need.
**Production implication**: blockvolume's lifecycle interaction with the walstore flusher needs verification under realistic production write pressure (not just synthetic stress). The T4d smoke did NOT verify this end-to-end.
**Carry**: G5 production-readiness — verify walstore flusher/checkpoint cadence under production write pressure; define tuning/operational policy. **Reword from earlier draft per architect round-48: the smoke surfaced "WAL admission pressure under synthetic workload," NOT "no background checkpoint exists."**
### Finding #2: no `--durable-walsize` CLI flag
**Symptom**: `cmd/blockvolume/main.go` exposes `--durable-blocks` + `--durable-blocksize` + `--durable-impl` but NOT `--durable-walsize`. walstore default 64 MiB WAL is hardcoded; smartwal default 2 MiB ring is hardcoded (`smartwal/store.go:19` `defaultWALSlots = 65536`).
**Production implication**: operators cannot tune WAL size at deployment without code change.
**Carry**: post-G5 / operator hardening — add `--durable-walsize` CLI flag (small change; ~10 LOC + plumb through to `newSuperblock(walSize)` opts).
### Finding #3: pre-existing `core/frontend/nvme/TestT2A_ConcurrentQueueStress` data race — ✅ RESOLVED at `a0be6d5`
**Original symptom**: Under `go test -race` on m01 Linux, `TestT2A_ConcurrentQueueStress` in `core/frontend/nvme` emitted `WARNING: DATA RACE` and FAILED.
**Verified pre-existing**: same failure at `1edeb36` (T4d-1 commit). NOT introduced by T4d-4. Belonged to T2 NVMe surface.
**Root cause** (diagnosed via -race stack traces on m01): `writeCountingBackend.lastWrite []byte` field at `core/frontend/nvme/t2_v2port_nvme_write_chunked_r2t_test.go:64` was unsynchronized; the test fixture's `Write` method assumed serial caller use ("NOT safe across concurrent writes, but the test drives them serially") but `TestT2A_ConcurrentQueueStress` violated that assumption by sharing one fixture across 8 concurrent IO queues. Stack confirmed race lives entirely in test-fixture code, NOT in production session model (which is correctly serialized: one rxLoop, one txLoop, per-Write goroutines for IO dispatch).
**Fix**: `a0be6d5` — `lastWrite []byte` → `atomic.Pointer[[]byte]` (QA's recommended Option A); 25+/8- diff, test fixture only, zero production code touched.
**m01 -race verification (post-fix)**:
- `count=10` round: PASS in 2.3s ✓
- `count=50` stress round: PASS in 7.0s ✓
- Full `core/frontend/nvme` package under -race: PASS in 22.2s ✓
**Status**: closed. T2A NVMe surface no longer races under -race on m01. Full V3 -race smoke now genuinely all-green (no remaining package failures).
### Finding #4: substrate behavior invariant across T4d batches
**Symptom**: per-batch substrate smoke shows identical results T4d-1 baseline (`1edeb36`) → current. Substrate code stable since T4d-1; T4d-2/T4d-3/T4d-4 didn't touch substrate behavior.
**Production implication**: positive — no substrate regressions during T4d. Confirms layering discipline (T4d work above substrate didn't accidentally couple back to substrate internals).
---
## §I Forward-carries surfaced during T4d
| Carry | Owner | Bind point | Origin |
|---|---|---|---|
| `CARRY-T4D-LANE-CONTEXT-001` — replace TargetLSN==1 caller shim with true handler/session-context lane signal | sw | **post-G5 protocol hardening (G5 hardening backlog)** | T4d-2 follow-up `01f4ab9` round-46 |
| `G5-DECISION-001` — engine recovery state across primary restart (Path A persist vs Path B rebuild-from-probe) | architect at G5 | G5 collective close | T4d-4 part B round-47 |
| walstore flusher/checkpoint cadence verification under production write pressure + tuning/operational policy | sw + architect | G5 production-readiness gate | §H Finding #1 (m01 production-sized smoke) |
| `--durable-walsize` CLI flag for operator tuning | sw | post-G5 / operator hardening | §H Finding #2 |
| ~~Pre-existing `TestT2A_ConcurrentQueueStress` data race~~ | ✅ RESOLVED `a0be6d5` | (closed during T4d close cycle; m01 -race ×50 PASS) | §H Finding #3 |
| m01 hardware first-light for replicated write path (multi-replica + real network conditions) | QA + sw | G5 collective close | T4c closure §H + kickoff §2.6 |
| Multi-replica concurrent live + recovery scenarios | QA | G5 production-readiness | T4d kickoff scope; not in T4d-4 |
---
## §J Sign requests at this batch close
T4d is **the final T4 batch**. Per §8C.1 + kickoff §4 #10, T-end three-sign lands at this close.
| Signer | Sign type | Asks |
|---|---|---|
| **QA** | §8C.2 single-sign at batch close | Confirm 14 invariants pinned per §F; confirm forward-carry checklist v0.2 all 19 active T4a/T4b/T4c invariants green at HEAD under -race on m01; confirm §H findings are honest about production-readiness gaps. **(this document is the QA single-sign artifact)** |
| **Architect** | §8C.1 T4 T-end three-sign | (1) Read review points D-1 through D-6; (2) accept §B acceptance bar fully met; (3) accept §H production-readiness findings as G5 carries (not T4d defects); (4) accept G5-DECISION-001 + CARRY-T4D-LANE-CONTEXT-001 as named carries; (5) acknowledge T4 T-end at this close (no T4e). |
| **sw** | confirmatory note (no §8C.2 sign required at batch level per kickoff §4 #10 rules) | Confirm all `INV-REPL-*` upgrades from `⏭ T4d-N` to `✓ PORTED T4d-N` in catalogue §3.3; confirm §I forward-carries with Owner=sw are recorded for post-G5/G5-readiness planning |
---
## §K Notes for future readers
- T4d is where the apply gate (lane-pure, per-LBA stale-skip + coverage-advance) and engine-driven recovery wiring (R+1 + retry loop end-to-end + rebuild path) became real. The combination closes the "production correctness boundary for replicated writes under recovery."
- **Two architectural pins surfaced + fixed in part C** (catch-up exhaustion stale-Decision-key bug; WALRecycled-no-StartRebuild-emit bug + RebuildPinned sticky override). Both surfaced because architect's round-47 ruling required full L2 matrix end-to-end testing — synthetic injection wouldn't have caught either. This validates the round-47 hard-gate decision empirically.
- **Three architectural improvements ratified during T4d** beyond the original kickoff scope: 2-enum split (T4d-1 round-46), lane-pure gate API (T4d-2 round-46), round-47 catch-up exhaustion DIRECTLY emits StartRebuild. Each was sw-proposed-or-discovered + architect-endorsed; recorded explicitly in §B so the precedent is visible.
- **Process rule inscribed**: G-1 docs land in seaweedfs FIRST; sw implementation references committed hash via `Refs G-1 sign: <sha>`. T4d-3 G-1 sign hash = `seaweedfs@80036404c`. This avoids the "pending hash" procedural gap surfaced at T4d-3.
- **Production-sized substrate smoke at m01** is the right discipline for batch close evidence — surfaces gaps that unit tests at fixture sizes cannot (walstore WAL admission boundary; CLI knob absence). Forward-carry to G5: include production-sized smoke per batch as standard evidence.
- The forward-carries in §I represent **honest production-readiness work** that didn't fit T4d's scope. G5 collective close should not narrow them silently — apply the same §B-style explicit acceptance discipline T4c established.
- **Round-48 architect review surfaced 3 blocking edits + 1 smaller fix**: (1) part C commit hash bound to `e642ae8` everywhere; (2) `CARRY-T4D-LANE-CONTEXT-001` bind point corrected from "T4e (preferred) or post-G5" to **"post-G5 protocol hardening (G5 hardening backlog)"** since "no T4e" + "T4e preferred" was contradictory if T-end signs at this close; (3) §H Finding #1 reworded to remove the false root cause about "caller-driven, no background goroutine" — walstore HAS `go s.flusher.run()` at `walstore.go:189-190`; (4) 2 named invariants added to §F capturing part C bug fixes (`INV-REPL-FAILED-SESSION-KIND-DRIVES-ESCALATION` + `INV-REPL-REBUILD-ESCALATION-STICKY-UNTIL-TERMINAL`) per architect's smaller-fix request. Total inscribed invariants 14 → 16. Recording the round-48 corrections explicitly here so the procedural diligence is visible.
sw-block/design/v3-phase-15-t4d-mini-plan.md
+696
View File
@@ -0,0 +1,696 @@
# V3 Phase 15 — T4d (Apply Gate + Engine-Driven Recovery + Substrate Hardening + Lifecycle) Mini-Plan
**Date**: 2026-04-25 (v0.3 — addresses v0.2 engine/storage dependency blocker)
**Status**: DRAFT v0.3 — re-submitted for QA + architect sign after v0.2 review (engine failure kind is engine-owned; no engine→storage dependency)
**Owner**: sw (implementation); QA (acceptance)
**Authority sources**:
- `v3-phase-15-t4d-kickoff.md` v0.3 (architect-ratified 2026-04-25 by pingqiu) — §1–8 ratified, §9 ratified Q1/Q2/Q3
- `v3-phase-15-t4c-closure-report.md` §I forward-carries
- Round-43 architect lock (replica recovery apply gate, lane-aware, per-LBA)
- Round-44 architect refinement (2-map split: liveTouched + recoveryCovered)
- §9 architect decisions: Q1 engine-owned retry; Q2 lane implicit from handler context; Q3 transport↔storage contract-only
**Discipline**: §8B.4 mid-T batch; §8C.2 QA single-sign per batch close
**Timeline**: ~3-4 weeks (T4d-2 apply gate is the largest batch; T4d-3 G-1 read serializes its code start)
**Predecessors**: T4c batch closed (`c910464a9`); BlockStore walHead pre-T4d hotfix landed (`f6084ee`); kickoff §1–8 + §9 RATIFIED
---
## §0 Why this is 4 batches, not 3 — and where V2 stops, V3 starts
T4d kickoff §3 ratified a 4-batch shape (was 3 in kickoff v0.1; v0.2 added substrate hardening as a separate batch after round-43 surfaced the BlockStore + walstore + smartwal vulnerability). Architect rationale (kickoff §3 sign):
> "This ordering is correct: remove brittle contracts first, then add the correctness gate, then optimize/wire end-to-end."
**Why 4 batches, ordered exactly this way:**
| New | Replaces / extends | Architect's reasoning |
|---|---|---|
| **T4d-1** Substrate hardening + structured failure kind | (new — round-43 surfaced; T4c closure §I structured-kind binding lands here) | "Lowest-risk decoupled work; closes the BlockStore production vulnerability quickly even if architect chooses bundle-not-hotfix; structured RecoveryFailureKind is a contract change other batches depend on." |
| **T4d-2** Replica recovery apply gate | (new — round-43 lock + round-44 refinement) | "The apply gate is the correctness boundary. R+1 is bandwidth optimization built ON the apply gate's correctness guarantee. Building R+1 first would mean shipping unsafe recovery in production for the duration of T4d-2 if T4d-3 lands first." |
| **T4d-3** R+1 threading + engine-driven recovery wiring | (T4c §I row 1 + row 3 carries; round-43 downgraded R+1 from correctness-blocker to bandwidth-optimization) | "T4d-3 needs G-1 because it touches the V2 catch-up muscle boundary and `runCatchUpTo(replicaFlushedLSN, targetLSN)` semantics." |
| **T4d-4** Lifecycle + integration | (BUG-005 lifecycle from sketch §4 row T4d + T4c §I rows 2/5/6/7) | Bundle the remaining T4c carries with the lifecycle item; integration matrix becomes meaningfully end-to-end once T4d-3 wiring is in. |
**§9 walshipper boundary — PORT vs V3-native (architect-ratified at kickoff Q1/Q2/Q3):**
T4d-3 is **NOT a wholesale V2 `WALShipper` port**. Per kickoff §9.2:
| Area | Classification | T4d-3 treatment |
|---|---|---|
| `runCatchUpTo(R, target)` start boundary | **PORT from V2** | Thread replica R through `StartCatchUp` → executor scans `R+1..target` |
| Per-call catch-up deadline | **PORT from V2** | Keep deadline scope discipline; do NOT leak to later live ships |
| Last-sent / achieved-LSN monotonicity | **PORT from V2** | "Advance only after successful frame write"; final completion via `BarrierResponse.AchievedLSN` |
| Catch-up retry budget | **SEMANTIC PORT, V3 location** | Preserve V2 `maxCatchupRetries=3` behavior via `RecoveryRuntimePolicy`; **retry bookkeeping lives in engine** (Q1) |
| WAL-recycled escalation | **SEMANTIC PORT, V3 location** | Preserve "retention miss → rebuild"; T4d-1 replaces string match with structured `RecoveryFailureKind` |
| Live ship during recovery | **V3 generalization** | V2 allowed live + recovery concurrent; V3 enforces via lineage + lane-aware apply gate |
| Unified catch-up/rebuild/LBA-rebuild protocol | **V3-native** | `StartRecovery` + `RecoveryContentKind` + runtime policy; no V2 1:1 |
| Replica stale-entry skip + coverage accounting | **V3-native** | T4d-2 apply gate; transport calls the gate; transport never queries substrate per-LBA freshness |
| Engine-driven recovery retry loop | **V3-native** (Q1) | Engine emits fresh recovery commands per policy; executor stays byte-movement-only |
| Package-boundary discipline (transport↔storage contract-only; no `core/recovery` move in T4d) | **V3-native** (Q3 / kickoff §9.5) | Inscribed via `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY`; pinned at T4d-3. Package move deferred to post-G5/T4e per kickoff §9.3 #3 |
Architect-binding: kickoff §9.5 instructs the T4d-3 G-1 V2 read to explicitly separate PORT items (5 listed) from V3-native items (5 listed). G-1 deliverable.
---
## §1 Scope
T4d delivers **the production correctness boundary for replicated writes under recovery**: a lane-aware replica apply gate that prevents per-LBA data regression, structured failure semantics that decouple engine from substrate string-matching, V2-faithful catch-up start-boundary porting (R+1 threading), end-to-end engine-driven recovery wiring, and the lifecycle close that closes BUG-005 non-repeat.
### §1.1 Semantic whole
After T4d:
- **Recovery is correct under partial-failure interrupted replay**: replica apply gate enforces per-LBA stale-skip + coverage accounting; data does not regress on any over-scan / pinLSN-below-replicaR / repeated-window / live-races-recovery scenario
- **Recovery is bandwidth-efficient**: catch-up scans from `R+1`, not genesis (V2-faithful start boundary)
- **Recovery is fully engine-driven**: `ReplicationVolume↔adapter` wired; engine retry loop runs end-to-end; `WithEngineDrivenRecovery` framework primitive becomes real; full L2 matrix gains true engine-level coverage (T4c was muscle-level only)
- **Recovery failure semantics are structured**: `RecoveryFailureKind` enum replaces substring `"WAL recycled"` match; engine switches on typed kind
- **Substrates are stale-safe in depth**: BlockStore walHead guard already landed (`f6084ee`); walstore + smartwal substrate-level stale-protection where practical
- **Lifecycle is regression-safe**: `ReplicationVolume.Stop` + cross-orchestration sessionID coordination + Windows TempDir cleanup race fixed; BUG-005 non-repeat sealed
### §1.2 What's NOT in T4d (explicit deferrals per kickoff §2.6 + §9.3)
- **m01 first-light for replicated write path** — G5 collective close
- **Package move `core/transport` → `core/recovery`** — post-G5/T4e refactor (§9.3 #3)
- **Lane byte on `MsgShipEntry` wire** — Q2 ratified: lane is implicit from handler context; do NOT add wire field in T4d (§9.3 #2)
- **Substrate-native per-LBA applied-LSN exposure for ALL substrates** — Option C hybrid means add where practical; remaining substrates use in-memory session map seeded by §2.5 #1 architect ratification
### §1.3 Lane discipline (Q2 architect-ratified, kickoff §9.3 #2)
`MsgShipEntry` does NOT carry a lane byte in T4d. Lane is derived from the receiving handler / session context:
| Handler | Lane | Apply path |
|---|---|---|
| Live-ship handler (steady-state Ship) | live | Existing lineage/session/live-order rules; stale → reject + diagnostic; do NOT advance `recoveryCovered` |
| Catch-up / rebuild handler | recovery | Apply gate: `if entry.LSN <= appliedLSN[LBA] → skip data + recoveryCovered[LBA]=true; else apply + appliedLSN[LBA]=entry.LSN` |
`INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` pins this at T4d-2 / T4d-3.
### §1.4 Location
- Production code → `seaweed_block` repo, branch `phase-15`
- Reference docs → `seaweedfs` repo, `sw-block/design/`
- Tests → `seaweed_block` alongside code; component scenarios in `core/replication/component/`
- L3 m01 — out of T4d (G5 collective)
---
## §2 Tasks (4 batches; ordered; each self-contained, reviewable, mergeable)
### T4d-1 — Substrate hardening + structured recovery failure kind end-to-end (~170 production + ~100 tests; v0.2/v0.3 revised upward from v0.1 to absorb adapter + engine event-struct contract additions)
**Where**:
- `core/storage/recovery_contract.go` — new substrate-facing `StorageRecoveryFailureKind` enum + `RecoveryFailure` typed-error struct + wrap helpers for `ScanLBAs` errors
- `core/storage/walstore_recovery.go` — wrap `ErrWALRecycled` returns with `RecoveryFailure{Kind: WALRecycled, ...}`; add per-LBA stale-skip in `ApplyEntry` body where practical (defense-in-depth, kickoff §2.5 #3 caveat)
- `core/storage/smartwal/recovery_scan.go` — same wrap + same defense-in-depth caveat
- `core/storage/store.go` — BlockStore.ApplyEntry already gated by hotfix `f6084ee`; verify + reference here
- **`core/engine/events.go`** — define engine-owned `RecoveryFailureKind` enum and add `FailureKind RecoveryFailureKind` field to `SessionClosedFailed`; engine MUST NOT import `core/storage`
- **`core/adapter/types.go`** — add `FailureKind engine.RecoveryFailureKind` field to `SessionCloseResult` (architect HIGH v0.1 #1 + v0.2 dependency fix: end-to-end typed kind, NOT hidden in FailReason text)
- `core/engine/apply.go` — `applySessionFailed` switches on `e.FailureKind == RecoveryFailureWALRecycled` (engine-owned typed branch); `containsAny`/substring match REMOVED. `Reason` field stays as diagnostic text only
- `core/transport/catchup_sender.go` — extracts substrate-returned `storage.RecoveryFailure.Kind` via `errors.As`, maps it to `engine.RecoveryFailureKind`, and populates `SessionCloseResult.FailureKind`
- `core/replication/volume.go` (and any close-callback wiring) — propagates `FailureKind` through the close path; doesn't re-derive from text
- Adapter normalization (`OnSessionClose` → engine event): copies `FailureKind` from `SessionCloseResult` to `SessionClosedFailed`
- Existing engine test `TestSessionFailed_WALRecycled_EscalatesToRebuild` — rewritten to construct event with `FailureKind: RecoveryFailureWALRecycled` and assert engine branches on the typed field
**Why** (kickoff §2.4 / §3 / §4 #7): T4c accepted substring matching as TEMPORARY; the binding was "T4d (preferred) or G5 final sign (latest)." Engine→storage decoupling is a contract obligation. Plus substrate hardening per kickoff §2.3 closes the round-43-surfaced defense-in-depth gap that the apply gate (T4d-2) covers as primary.
**Typed contract end-to-end (architect HIGH v0.1 #1 + v0.2 dependency fix)** — failure kind is a typed field on every plumbing hop, NOT parsed from text. Storage may classify substrate errors, but engine owns the decision vocabulary it consumes:
```go
// core/storage/recovery_contract.go
type StorageRecoveryFailureKind int
const (
StorageRecoveryFailureUnknown StorageRecoveryFailureKind = iota
StorageRecoveryFailureWALRecycled // tier-class change; adapter maps to engine rebuild escalation
StorageRecoveryFailureSubstrate // substrate IO error; retryable unless caller says otherwise
)
type RecoveryFailure struct {
Kind StorageRecoveryFailureKind
Cause error // wrapped underlying error for diagnostics
Detail string // free-form additional context
}
func (f *RecoveryFailure) Error() string { ... }
func (f *RecoveryFailure) Unwrap() error { return f.Cause }
// errors.Is / errors.As let transport identify the storage-side kind.
// core/engine/events.go
type RecoveryFailureKind int
const (
RecoveryFailureUnknown RecoveryFailureKind = iota
RecoveryFailureWALRecycled // tier-class change; escalate to rebuild
RecoveryFailureTransport // stream/connection error; retryable
RecoveryFailureSubstrate // substrate IO error; retryable
RecoveryFailureTargetNotReached // catch-up didn't reach targetLSN; retryable
RecoveryFailureSessionInvalidated
)
// core/adapter/types.go (modify existing SessionCloseResult)
type SessionCloseResult struct {
ReplicaID string
SessionID uint64
Success bool
AchievedLSN uint64
FailureKind engine.RecoveryFailureKind // NEW v0.3; engine-owned, no engine→storage dependency
FailReason string // DIAGNOSTIC TEXT ONLY — engine MUST NOT parse
}
// core/engine/events.go (modify existing SessionClosedFailed)
type SessionClosedFailed struct {
ReplicaID string
SessionID uint64
FailureKind RecoveryFailureKind // engine-owned; engine does not import storage
Reason string // DIAGNOSTIC TEXT ONLY — engine MUST NOT parse
}
```
Engine `applySessionFailed` branches on `e.FailureKind`:
```go
// core/engine/apply.go (replace existing substring-match block)
if e.FailureKind == RecoveryFailureWALRecycled {
st.Recovery.Decision = DecisionRebuild
st.Recovery.DecisionReason = "wal_recycled"
st.Recovery.Attempts = 0
// ... existing recycle escalation path
}
// substring `containsAny(reason, ["WAL recycled", ...])` match REMOVED
```
**Acceptance fence**: `TestEngine_SessionFailed_NoMoreSubstringMatch` greps `core/engine/apply.go` for `"WAL recycled"` substring usage and fails if present. The substring helper `containsAny` may stay if other engine code uses it; if not, removed too.
**Cross-package contract note (v0.3 dependency fix)**: `core/engine` MUST NOT import `core/storage`. Storage owns substrate-side error classification; transport/adapter maps that classification into `engine.RecoveryFailureKind`. `core/adapter` already imports `core/engine`, so `adapter.SessionCloseResult.FailureKind engine.RecoveryFailureKind` does not create a new package direction. This keeps the dependency line: storage error → transport mapping → adapter close result → engine event.
**Substrate defense-in-depth (architect §2.5 #3 caveat)**:
| Substrate | Action | Practical-scope decision |
|---|---|---|
| BlockStore | Already done — `f6084ee` walHead gate | No further action |
| walstore | Add per-LBA stale-skip in `ApplyEntry` body BEFORE `dm.put` (read existing `dm.get(lba)`, compare LSN, skip if `incoming <= existing`) | Sw judgment at impl time per §2.5 #3 caveat. If `dm.get` exposes per-LBA LSN cleanly (it does — the dirty entry carries `lsn`), include. Else: documented carry to T4e. |
| smartwal | Add per-LBA stale-skip in `ApplyEntry` body BEFORE `writeAt` extent overwrite. smartwal's per-LBA LSN tracking lives in the ring; need a fast-path read or session map | Sw judgment per §2.5 #3 caveat. Likely needs a small `lastAppliedLSN[lba]` in-memory map maintained by `ApplyEntry` itself. Inclusion contingent on not turning T4d-1 into a substrate refactor (kickoff §2.5 #3 verbatim). |
**Tests** (10 required):
- `TestStorageRecoveryFailureKind_Walstore_RecycleWrappedAsKind` — walstore `ScanLBAs` ErrWALRecycled wraps as storage-side `StorageRecoveryFailureWALRecycled` kind
- `TestStorageRecoveryFailureKind_Smartwal_RecycleWrappedAsKind` — smartwal same
- `TestTransportRecoveryFailureKind_MapsStorageRecycleToEngineKind` — transport maps `StorageRecoveryFailureWALRecycled` to `engine.RecoveryFailureWALRecycled`
- `TestTransportRecoveryFailureKind_TransportError_WrappedAsEngineKind` — non-recycle stream error surfaces as `engine.RecoveryFailureTransport`
- `TestEngine_SessionFailed_StructuredKind_RecycleEscalatesToRebuild` — engine maps typed `FailureKind` field, not substring (replaces existing `TestSessionFailed_WALRecycled_EscalatesToRebuild`)
- `TestEngine_SessionFailed_StructuredKind_NonRecycleRetries` — engine retry path uses `FailureKind` match
- `TestEngine_SessionFailed_NoMoreSubstringMatch` — fence: greps `core/engine/apply.go` for `"WAL recycled"` substring; fails if present (architect HIGH v0.1 #1 — engine MUST NOT parse text)
- `TestAdapter_SessionCloseResult_TypedKindEndToEnd` — close result built with `FailureKind` set; adapter normalization to engine event preserves the typed field
- `TestEngine_SessionClosedFailed_TypedKindFieldShape` — event struct compile fence: `FailureKind` field present + non-zero kind value triggers correct engine branch without importing storage
- `TestStorageWalstore_ApplyEntry_StaleLSN_SubstrateSkips` (if walstore stale-skip included)
- `TestStorageSmartwal_ApplyEntry_StaleLSN_SubstrateSkips` (if smartwal stale-skip included)
**Inscribed invariants**:
- Update `INV-REPL-CATCHUP-RECYCLE-ESCALATES` (catalogue §3.3) — pinning method changes from substring to structured kind
**Preserve**:
- All T4c invariants forward-carry green (per kickoff §4 #5)
- `applySessionFailed` non-recycle behavior (retry-loop, start-timeout exception) unchanged
**No G-1 required** (architect kickoff §3 sign #4): "T4d-1 is V3-native + small substrate edits; no V2 muscle source."
---
### T4d-2 — Replica recovery apply gate (lane-aware, per-LBA, 2-map split) (~250 production + ~250 tests)
**Where**:
- `core/replication/apply_gate.go` — new file: `ReplicaApplyGate` component with `ApplyRecovery(...)` and `ApplyLive(...)` entry methods; per-session state (`liveTouched`, `recoveryCovered`, `appliedLSN`)
- `core/replication/apply_gate_session.go` — per-session state machine (initialized at session start, cleared at session close); seeds `appliedLSN` from substrate query path (Option C hybrid per kickoff §2.5 #1)
- `core/transport/replica.go` — `MsgShipEntry` handler dispatches to gate via lane discriminator (handler context, NOT wire byte per Q2)
- Adapter wiring (whichever entry registers session start with the gate)
- Existing skipped adversarial tests in `core/replication/component/stale_entry_adversarial_test.go` — un-skip + extend per round-44 (coverage-advance + live-fails-loud)
- New component scenarios per round-44 spec
- Substrate query path for Option C hybrid: small additive method on `LogicalStorage` returning per-LBA latest-applied-LSN if known (fallback: 0 = "ask the gate's session map")
**Why** (kickoff §2.2 round-43 lock + round-44 refinement): primary correctness boundary. Without this, walstore's per-LSN scan + replica's blind apply produces per-LBA data regression on partial-failure interrupted replay.
**Architect-locked spec (round-43 verbatim)**:
> Recovery-stream stale entries are valid duplicates and must be skipped as data writes while still counted as recovery-stream coverage. Live-lane stale entries are abnormal and must not mutate data; they should be skipped/rejected under lineage/order diagnostics and must not advance recovery coverage.
**Gate behavior** (lane-aware):
```
ApplyRecovery(replicaID, sessionID, entry):
require: session exists + matches lineage
if entry.LSN <= appliedLSN[entry.LBA]:
recoveryCovered[entry.LBA] = true
return // skip data; coverage advances
substrate.ApplyEntry(entry.LBA, entry.Data, entry.LSN)
appliedLSN[entry.LBA] = entry.LSN
recoveryCovered[entry.LBA] = true
return
ApplyLive(replicaID, sessionID, entry):
require: session exists + matches lineage
if entry.LSN <= appliedLSN[entry.LBA]:
log diagnostic "live stale" + reject // round-44 fail-loud
return error
substrate.ApplyEntry(entry.LBA, entry.Data, entry.LSN)
appliedLSN[entry.LBA] = entry.LSN
liveTouched[entry.LBA] = true
// do NOT touch recoveryCovered
return
```
**`appliedLSN` source — Option C hybrid (architect §2.5 #1 ratified)**:
At session start, gate queries substrate for per-LBA latest applied LSN:
```go
type LogicalStorage interface {
// ... existing methods ...
// T4d-2 addition (Option C hybrid):
// Returns per-LBA latest-applied LSN where the substrate tracks
// it. Implementations that don't track this MUST return
// (nil, ErrAppliedLSNsNotTracked). The gate then falls back to
// in-memory session-only tracking, seeded from live + recovery
// applies during the session.
AppliedLSNs() (map[uint32]uint64, error)
}
// New sentinel in core/storage/recovery_contract.go
var ErrAppliedLSNsNotTracked = errors.New("storage: substrate does not track per-LBA applied LSN")
```
**Per-substrate implementation (architect MED v0.1 #2 fix — every impl explicit, none "not implemented"):**
| Substrate | `AppliedLSNs()` impl | Source |
|---|---|---|
| **walstore** | Iterate `dm.snapshot()`; for each entry build `map[uint32]uint64{lba: entry.lsn}` | dirty map already carries per-LBA LSN |
| **smartwal** | `ring.scanValid()` then per-LBA last-writer-wins reduction (mirrors smartwal's existing recovery semantic) | Already implemented for `ScanLBAs`; reuse |
| **BlockStore** | `func (s *BlockStore) AppliedLSNs() (map[uint32]uint64, error) { return nil, ErrAppliedLSNsNotTracked }` — explicit not-tracked return; ensures interface compatibility | In-memory; no per-LBA LSN tracking |
Gate behavior on `ErrAppliedLSNsNotTracked`: log INFO once at session start, fall back to session-only `appliedLSN` map, seed from live + recovery applies as they arrive. Restart-safety degrades for unsupported substrates (BlockStore is calibration-only, so this is acceptable).
**Tests** (~14 required):
Adversarial set (un-skipped + extended from T4c skip-pile):
- `TestComponent_Adversarial_StaleEntryDoesNotRegress` — un-skipped; same LBA old/new + sever before new → no regression (architect round-41 set)
- `TestComponent_RepeatedRecoveryWindow_ByteAndLSNStable` — repeated window → byte state + per-LBA LSN stable
- `TestComponent_LiveRacesRecoveryOld_LiveWins` — live write at fresher LSN beats recovery's stale entry
- `TestComponent_RecoveryStaleSkip_CoverageStillAdvances` — round-44 #1: data skipped but `recoveryCovered[LBA]` updates
- `TestComponent_LiveLaneStaleEntry_FailsLoud` — round-44 #2: live lane stale → reject + diagnostic, do NOT advance recoveryCovered
Lane-discrimination set:
- `TestApplyGate_RecoveryHandler_DispatchesToRecoveryPath` — handler-context routing
- `TestApplyGate_LiveHandler_DispatchesToLivePath` — handler-context routing
- `TestApplyGate_LaneNeverDerivedFromWireField` — fence: no lane byte on MsgShipEntry; lane comes from handler context only
Option C hybrid set:
- `TestApplyGate_SessionStart_SeedsFromSubstrateAppliedLSN_Walstore` — walstore exposes; gate seeds correctly
- `TestApplyGate_SessionStart_SeedsFromSubstrateAppliedLSN_Smartwal` — smartwal exposes; gate seeds correctly
- `TestApplyGate_SessionStart_BlockStoreFallback_InMemoryOnly` — BlockStore returns ErrNotTracked; gate falls back
Restart-safety set (kickoff §4 #6 promotion):
- `TestApplyGate_RestartMidRecovery_SessionRestartReseeds` — kill session mid-recovery; new session re-queries substrate; appliedLSN consistent with substrate truth (pins `INV-REPL-RECOVERY-COVERAGE-RESTART-SAFE`)
Coverage-completion set:
- `TestApplyGate_BarrierCompletion_BarrierAuthoritativeWithCoverageContribution` — barrier ack is the authoritative completion truth (T4c-2 `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN` preserved); the gate's `recoveryCovered` map contributes to per-LBA coverage accounting that feeds barrier judgment, but does NOT replace the barrier as completion source. (Rename per QA v0.1 review issue #1; original name `TestApplyGate_BarrierAchievedLSN_DerivesFromCoverageNotData` could misread as conflicting with T4c-2 invariant.)
**Inscribed invariants** (catalogue §3.3 — already pre-inscribed by QA per round-44 prep work; T4d-2 promotes to PORTED):
- `INV-REPL-NO-PER-LBA-DATA-REGRESSION` (goal-level, round-43)
- `INV-REPL-RECOVERY-STALE-ENTRY-SKIP-PER-LBA` (mechanism, round-43)
- `INV-REPL-RECOVERY-COVERAGE-ADVANCES-ON-SKIP` (round-44)
- `INV-REPL-LIVE-LANE-STALE-FAILS-LOUD` (round-44)
- `INV-REPL-RECOVERY-COVERAGE-RESTART-SAFE` (Option C makes pinnable)
- `INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` (Q2)
**Preserve**:
- T4c apply path for non-stale entries unchanged (gate forwards to substrate `ApplyEntry`)
- Existing T4c bitmap/coverage logic IF any (none — T4d-2 introduces it)
- `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN` (T4c-2): completion still derives from barrier ack; gate accounting feeds barrier judgment, doesn't replace it
**No G-1 required** (architect kickoff §3 sign #4): "T4d-2 does not need G-1; its spec is the round-43/44 V3-native apply-gate rule." Round-43 + round-44 architect text IS the spec.
**Estimated LOC**: ~250 production (~150 gate + state + ~50 substrate AppliedLSNs impls + ~50 wiring) + ~250 tests (high test-to-production ratio because the gate is the correctness boundary; test density matters).
---
### T4d-3 — R+1 threading + engine-driven recovery wiring — **G-1 PRE-CODE REQUIRED** (~250 production + ~150 tests)
**Where**:
- `core/transport/catchup_sender.go` — `StartCatchUp` body: `ScanLBAs(R+1, ...)` instead of `ScanLBAs(1, ...)`
- `core/transport/executor.go` — `BlockExecutor.StartCatchUp` signature gains `fromLSN uint64` parameter
- `core/adapter/executor.go` — `CommandExecutor.StartCatchUp` interface signature update; all test executor stubs updated
- `core/engine/commands.go` — `engine.StartCatchUp` command struct gains `FromLSN uint64`
- `core/engine/apply.go` — emission populates `FromLSN` from `Recovery.R`; retry-loop re-emit also populates
- `core/replication/volume.go` — wire `ReplicationVolume↔adapter` so engine retry loop runs end-to-end
- `core/replication/component/cluster.go` — `WithEngineDrivenRecovery()` becomes real (was stub at T4c)
- L2 integration matrix — gain true engine-level coverage
**Why** (kickoff §2.4 + §3 + §9.5):
- R+1 threading: kickoff §2.4 + T4c §I row 3 — bandwidth optimization (post-round-43; was correctness-blocker pre-round-43, now the apply gate handles correctness regardless of where primary scans from). Pins `INV-REPL-CATCHUP-WITHIN-RETENTION-001` un-pin from T4c.
- Engine-driven recovery wiring: kickoff §2.4 + T4c §I row 1 — closes the integration-scope gap left at T4c (engine retry loop unit-tested only).
- Q1 architect lock: retry loop is engine-owned, not transport. T4d-3 does NOT add a self-driven retry loop in `catchup_sender.go`.
**§9.5 architect-binding split** (T4d-3 G-1 must explicitly separate):
PORT (5 items from V2):
1. `runCatchUpTo(replicaFlushedLSN, targetLSN)` start-boundary semantics
2. Per-call deadline scope (do not leak to live ships)
3. Last-sent monotonicity ("advance only after successful frame write")
4. Retry count semantics (`maxCatchupRetries=3` envelope; bookkeeping in engine per Q1)
5. Retention-miss escalation ("WAL recycled" → rebuild; via T4d-1 structured kind)
V3-native (5 items, do not look for V2 source):
1. Engine-owned retry loop (Q1 — engine decides emit/escalate; transport stays byte-movement-only)
2. Unified recovery command model (`StartRecovery` / `RecoveryContentKind`)
3. Lane-aware apply gate integration (T4d-3 calls into T4d-2 gate; transport never queries substrate per-LBA freshness per Q3)
4. `BarrierResponse.AchievedLSN` as completion truth (T4c-2 `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN`)
5. Package-boundary discipline (transport↔storage contract-only per `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY`)
**G-1 V2 read deliverable** — `sw-block/design/v3-phase-15-t4d-3-g1-v2-read.md` (sw produces; QA reviews + signs before T4d-3 code starts).
Deliverable structure:
1. V2 source verbatim — `weed/storage/blockvol/wal_shipper.go:845` `runCatchUpTo` body + calling context
2. Per-V2-concern → V3-location map (5 PORT items + their V3 placement)
3. V3-native items list (5 items, named explicitly so reviewer doesn't expect V2 source)
4. Hidden-invariant audit (anticipated 2-3 invariants)
5. Per-substrate behavior under R+1 threading (smartwal R+1, walstore R+1, BlockStore R+1)
6. LOC heuristic + V3 vs V2 expansion budget (per `feedback_g1_pre_code_review.md` LOC asymmetry note)
7. Test parity matrix (V2 mirror + V3-only)
8. Open placement decisions (if any)
9. QA sign block
**Tests** (~10 required):
R+1 threading set:
- `TestT4d3_CatchUp_ScansFromReplicaR_NotGenesis` — replica at R=50, primary at H=100; sender ScanLBAs called with `fromLSN=51`, NOT `fromLSN=1`
- `TestT4d3_CatchUp_NonEmptyReplica_ShortGap_BandwidthBounded` — pins `INV-REPL-CATCHUP-WITHIN-RETENTION-001` un-pin; over-scan reduction observable via emit count (use `NewObservedScanWrap` from component framework)
- `TestT4d3_CatchUp_StartCatchUpSignature_FromLSNRequired` — sender rejects calls without explicit `fromLSN`
Engine-driven recovery set:
- `TestT4d3_EngineRetryLoop_E2E_RetryUntilBudget` — engine retry loop end-to-end through real ReplicationVolume; budget exhaustion → escalate observable
- `TestT4d3_EngineRetryLoop_E2E_RecycleEscalatesImmediate` — ErrWALRecycled bypasses retry, escalates to rebuild via real path
- `TestComponent_WithEngineDrivenRecovery_NotStubAnymore` — framework primitive emits real engine commands
Lane discipline (transport side):
- `TestT4d3_Catchup_TransportNeverCallsSubstratePerLBA` — `catchup_sender` does not import substrate-internal packages; pins `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY`
L2 matrix expansion:
- 3 existing T4c L2 scenarios (ShortDisconnect, GapExceedsRetention, RecoveryModeLabelSurfaced) re-pass with engine-driven flow
**Inscribed invariants**:
- `INV-REPL-CATCHUP-WITHIN-RETENTION-001` un-pinned at T4c → PORTED at T4d-3
- `INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` (Q2) — receiver-side: catch-up handler routes to recovery lane
- `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY` (Q3) — sender-side: no substrate-internal imports
**Preserve**:
- T4c `ScanLBAs(1, ...)` wasn't wrong, was wasteful; new `ScanLBAs(R+1, ...)` is V2-faithful
- T4c-2's 6 catch-up invariants (callback-return-nil-continues, lastSent-monotonic, deadline-per-call-scope, target-not-reached-vs-recycled-distinguished, recycle-escalates, completion-from-barrier-achievedLSN) all forward-carry
- T4c-pre-B `StartRecoverySession` dispatch shell unchanged (T4d-3 changes `StartCatchUp` legacy path; the unified path stays)
**G-1 required** per kickoff §3 sign #3.
---
**T4d-3 closure scope (architect-bound 2026-04-25 round-46+ Path B):**
T4d-3 closes as **R+1 bounded catch-up PORT + boundary fences**, NOT full engine-driven recovery. Architect explicit: "do not repeat T4c's narrowing silently" — this is a **scope delta**, not a soft carry.
**What T4d-3 (`44c60dd`) DELIVERED:**
- 5 G-1 PORT items (R+1 start boundary + per-call deadline + lastSent-monotonic + retry budget semantics + retention-miss escalation)
- 2 hidden invariants pinned (`INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1` + `INV-REPL-CATCHUP-FROMLSN-FROM-ENGINE-STATE-NOT-PROBE`)
- Boundary fence: `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY` (Q3) via import-discipline test
- CARRY-T4D-LANE-CONTEXT-001 Option B (skip-test with forward-binding)
- 11 new tests per G-1 §7
**What T4d-3 DEFERRED to T4d-4 (mandatory, see T4d-4 expansion below):**
- `ReplicationVolume↔adapter` wiring so engine retry loop runs end-to-end
- `WithEngineDrivenRecovery()` framework primitive becomes real (currently stub)
- Full L2 subprocess matrix gain engine→adapter→executor end-to-end coverage
**Tests below labeled "Engine-driven recovery set"** (`TestT4d3_EngineRetryLoop_E2E_*` + `TestComponent_WithEngineDrivenRecovery_NotStubAnymore`) **DID NOT land in T4d-3** — they land in T4d-4 per Path B fold.
**T4d cannot close** until `WithEngineDrivenRecovery()` is real and QA #8 (`LastSentMonotonic_AcrossRetries_FullForm`) can run — see updated T4d-4 acceptance criteria.
**Estimated LOC**: ~250 production (signature change ripples through 5 files + engine wiring ~100 + framework primitive ~50) + ~150 tests.
---
### T4d-4 — Lifecycle + integration + remaining T4c carries + **engine-driven recovery wiring (Path B fold)** (~300 production + ~350 tests; **REVISED v0.4** from ~150/~200)
**Scope expansion (architect-bound 2026-04-25 round-46+ Path B):** T4d-3's mini-plan §2.3 listed engine-driven recovery wiring (`ReplicationVolume↔adapter` + `WithEngineDrivenRecovery()` real binding + full L2 matrix). T4d-3 (`44c60dd`) deferred this work; architect ratified Path B = fold into T4d-4 as **HARD CLOSE GATE** (not soft carry). T4d cannot close until this wiring is real and QA #8 can run.
**Where (revised v0.4 — adds engine-driven wiring rows):**
**Where**:
*Original T4d-4 scope (lifecycle + integration + T4c carries):*
- `core/replication/volume.go` — `ReplicationVolume.Stop` lifecycle regression (BUG-005 non-repeat per `v3-phase-15-t4d-bug005-non-repeat.md`)
- `core/replication/peer.go` — cross-orchestration sessionID coordination (T4c §I row 5: live-ship + explicit catch-up share `sessionID=1`)
- `core/storage/recovery_contract.go` — substrate `RecoveryMode()` method (T4c §I row 6 — wraps forward label without duck-typed `CheckpointLSN`)
- `core/replication/component/faults.go` — wraps forward `RecoveryMode()` correctly (closes the round-40 known limitation)
- `core/transport/catchup_sender.go` — replace duck-typed `CheckpointLSN()` probe with `RecoveryMode()` call
- `core/replication/component/` — Windows TempDir cleanup race investigation/fix (T4c §I row 7); rename or split `TestT4c3_Catchup_ShortDisconnect_DeltaOnly` per kickoff §5 trailing chore
- QA `LastSentMonotonic_AcrossRetries` cross-call form (T4c §I row 2)
***Path B fold from T4d-3 (engine-driven recovery wiring — MANDATORY HARD CLOSE GATE):***
- **`core/replication/volume.go`** — wire `ReplicationVolume↔adapter` so engine retry loop runs end-to-end (T4d-3 deferred per Path B; mini-plan v0.2 §2.3 row)
- **`core/replication/component/cluster.go`** — `WithEngineDrivenRecovery()` becomes real (was stub at T4c; T4d-3 deferred per Path B)
- **Full L2 integration matrix** — engine→adapter→executor end-to-end coverage (closes mini-plan §4 acceptance criterion #3 that T4c narrowed and T4d-3 deferred)
***Round-47 architect addition: rebuild path engine-driven end-to-end (MANDATORY):***
- `core/engine/apply.go` — extend `applySessionFailed`'s rebuild escalation to actually emit a rebuild command (not just transition `Decision = Rebuild` state); engine MUST own rebuild emit per Q1 architect lock
- `core/engine/events.go` — `RebuildSessionClosed{Success,Fail}` event (or equivalent) so rebuild outcomes are observable to engine state
- `core/replication/volume.go` (or appropriate dispatch site) — wire rebuild commands through ReplicationVolume↔adapter the same way catch-up commands do
- `core/engine/apply.go` — apply rebuild-session-closed events; define rebuild failure terminal (retry budget OR immediate PublishDegraded — architect impl-time call)
- L2 integration test: real catch-up → budget-exhausted → engine emits rebuild → rebuild executes and completes (or fails terminally) → outcome observable in engine state. **Test must observe an actual rebuild session occurring**, not just `Decision = Rebuild` state transition.
**Why** (kickoff §2.1 + §2.4 + Path B fold): close BUG-005 non-repeat (the original sketch T4d scope item); land all T4c §I forward-carries; round out integration matrix at L2 end-to-end; **complete the engine-driven recovery wiring deferred from T4d-3 (Path B fold)**.
**Tests** (~15 required):
Lifecycle:
- `TestReplicationVolume_Stop_LifecycleRegression_BUG005NonRepeat` — pin per `v3-phase-15-t4d-bug005-non-repeat.md` (pins `INV-REPL-LIFECYCLE-HANDLE-BORROWED-001`)
- `TestReplicationVolume_Stop_DuringCatchUp_TerminatesCleanly`
- `TestReplicationVolume_Stop_DuringRebuild_TerminatesCleanly`
- `TestReplicationVolume_Stop_Idempotent`
Cross-orchestration sessionID:
- `TestPeer_SessionIDCoordination_LiveShipAndExplicitCatchUp_NoCollision`
`RecoveryMode()` method:
- `TestStorageRecoveryMode_Walstore_ReportsWALReplay` — walstore exposes mode via method (not duck-typed)
- `TestStorageRecoveryMode_Smartwal_ReportsStateConvergence`
- `TestStorageRecoveryMode_BlockStore_ReportsStateConvergence`
- `TestComponent_AssertSawRecoveryMode_UnderWrap_StillWorks` — closes the round-40 wrap-vs-CheckpointLSN limitation
Integration matrix (L2 end-to-end — Path B fold from T4d-3 — MANDATORY for T4d close):
- `TestT4d4_Integration_FullRecoveryFlow_E2E_BothSubstrates` — engine→adapter→executor→replica end-to-end
- `TestT4d4_Integration_RecoverFromMidStreamFailure_RetryAndEscalate` — partial failure → retry → escalate full chain
- `TestT4d3_EngineRetryLoop_E2E_RetryUntilBudget` (relocated from T4d-3 per Path B fold) — engine retry loop end-to-end through real ReplicationVolume; budget exhaustion → escalate observable
- `TestT4d3_EngineRetryLoop_E2E_RecycleEscalatesImmediate` (relocated from T4d-3 per Path B fold) — ErrWALRecycled bypasses retry, escalates to rebuild via real path
- `TestComponent_WithEngineDrivenRecovery_NotStubAnymore` (relocated from T4d-3 per Path B fold) — framework primitive emits real engine commands
- `TestComponent_LastSentMonotonic_AcrossRetries_FullForm` — QA Stage-1 deferred scenario (QA #8); REQUIRES `WithEngineDrivenRecovery()` real binding to be runnable; T4d cannot close until this test runs green
Rebuild path engine-driven (round-47 architect addition — MANDATORY for T4d close):
- `TestT4d4_Integration_CatchupBudgetExhausted_EngineEmitsRebuild` — catch-up retry budget exhausted → engine emits actual rebuild command (not just state transition)
- `TestT4d4_Integration_RebuildSession_CompletesAndReportsToEngine` — rebuild session runs end-to-end; completion event reaches engine state
- `TestT4d4_Integration_RebuildFailure_TerminalDefined` — rebuild failure path; verifies terminal is either retry-budget OR PublishDegraded per architect impl-time call (test asserts whichever was chosen)
- `TestT4d4_Integration_CatchupToRebuildEscalation_RealPath` — full path: catch-up failure → escalation → rebuild session → outcome observable. **Test observes actual rebuild session occurring**, not just `Decision = Rebuild` state.
Cleanups:
- Windows TempDir cleanup race fix verification
- `TestT4c3_Catchup_ShortDisconnect_DeltaOnly` rename or split per kickoff §5 chore
**Inscribed invariants**:
- `INV-REPL-LIFECYCLE-HANDLE-BORROWED-001` (newly inscribed at T4d-4 per kickoff §4 #6)
- `LastSentMonotonic_AcrossRetries` full form pin
**Preserve**:
- All prior T4d-1 / T4d-2 / T4d-3 invariants forward-carry
- All T4a/T4b/T4c invariants forward-carry per kickoff §4 #5
**No G-1 required** (port + tests, no new V2 muscle source).
**Estimated LOC (revised v0.4)**: ~300 production + ~350 tests (was ~150/~200; expanded by Path B fold ~150 prod / ~150 tests for `ReplicationVolume↔adapter` wiring + `WithEngineDrivenRecovery` real binding + L2 matrix scenarios).
**HARD CLOSE GATE for T4d (architect-bound 2026-04-25 round-46+ Path B; round-47 rebuild expansion):**
T4d cannot close until ALL of:
1. `WithEngineDrivenRecovery()` is REAL (not stub) — `core/replication/component/cluster.go`
2. `ReplicationVolume↔adapter` wiring runs engine retry loop end-to-end
3. **Full L2 matrix engine→adapter→executor→replica end-to-end** including:
- catch-up retry-until-budget end-to-end
- WAL recycled → rebuild escalation real path (NOT unit-scope only)
- `TestComponent_LastSentMonotonic_AcrossRetries_FullForm` (QA #8) runs and passes
- Mini-plan §4 acceptance criterion #3 satisfied — same bar T4c narrowed and T4d-3 deferred per Path B
4. **Rebuild path engine-driven end-to-end (round-47 architect addition; cannot be cathup-only)** — minimum:
- catch-up failure / budget exhausted → engine emits rebuild command (not just sets `Decision = Rebuild` state)
- rebuild session completion / failure → returns to engine via observable event (`RebuildSessionClosed{Success/Fail}` or equivalent)
- rebuild failure terminal defined explicitly: either rebuild has its own retry budget OR `PublishDegraded` immediately on first rebuild failure (architect call at impl time)
- L2 covers the **real path** catch-up → rebuild escalation (test must observe an actual rebuild session occurring, not just `Decision = Rebuild` state transition)
5. All other T4d-4 acceptance items (lifecycle, T4c carries, etc.) green
**Explicitly NOT in T4d-4 hard gate (E1 — architect round-47 ruling):**
Engine state persistence across primary restart is a **G5 decision gate**, not a T4d-4 implementation gate. T4d-4 doesn't need to implement persistence; G5 must explicitly decide "persist Recovery state" vs "rebuild state from probe after restart" as a named decision (see §G5-DECISION-001 below).
---
## §3 Predicates (must be true before T4d-1 starts)
| Predicate | Source |
|---|---|
| T4c batch closed | `c910464a9` (architect 2026-04-25) |
| Round-43 architect lock | kickoff §2.2 |
| Round-44 architect refinement | kickoff §2.2 |
| Kickoff §1–8 RATIFIED | architect sign 2026-04-25 |
| Kickoff §9 (walshipper Q1/Q2/Q3) RATIFIED | architect sign 2026-04-25 |
| BlockStore walHead pre-T4d hotfix landed | `f6084ee` |
| All T4a + T4b + T4c invariants forward-carry green | last full-suite run on `f6084ee` |
| QA sign on this mini-plan | _________ pending |
---
## §4 Acceptance (per kickoff §4 ratified bar)
T4d closes when ALL of:
1. ✅ All 4 batches (T4d-1, T4d-2, T4d-3, T4d-4) merged to `phase-15`
2. ✅ Unit tests green per task (with Windows cleanup race fixed or platform-noted)
3. ✅ Full L2 subprocess matrix green — engine→adapter→executor end-to-end (the bar T4c narrowed)
4. ⏸ L3 m01 — deferred to G5 collective per kickoff §2.6 (NOT a T4d criterion)
5. ✅ T4a/T4b/T4c invariants forward-carry verified (no regression)
6. ✅ Active invariant promotions per kickoff §4 #6 (9 invariants — see kickoff for full list)
7. ✅ Engine sentinel decoupling: substring match REMOVED from `core/engine/apply.go`; structured kind in production at T4d-1
8. ✅ BlockStore walHead regression fixed in production — DONE (`f6084ee`)
9. ✅ QA single-sign at T4d close per §8C.2
10. ✅ T4 T-end three-sign per §8C.1 lands at T4d close IF no T4e is created (kickoff §4 #10)
11. ✅ No §8C.3 escalation triggers fired
---
## §5 Non-claims (explicit deferrals within T4d)
- **No m01 first-light** — G5 collective close (kickoff §2.6)
- **No `core/recovery` package move** — post-G5/T4e (kickoff §9.3 #3)
- **No lane byte on `MsgShipEntry` wire** — Q2 ratified, lane is implicit from handler context (kickoff §9.3 #2)
- **No wholesale V2 `WALShipper` port** — T4d-3 ports the bounded catch-up muscle ONLY; engine-owned recovery orchestration stays V3-native (kickoff §9.2 + §0 above)
- **No substrate-native per-LBA applied LSN exposure for ALL substrates** — Option C hybrid: add where practical (walstore, smartwal); BlockStore falls back to in-memory session map (kickoff §2.5 #1)
- **No retry loop in transport** — Q1: engine-owned (kickoff §9.3 #1)
- **No engine recovery state persistence implementation** (round-47 architect ruling) — engine `Recovery{Attempts, TargetLSN, ActiveSessionID}` survives only as long as the primary process. Across primary restart, all in-flight Recovery state is lost; replicas re-probe and engine rebuilds state from observed facts. **G5-DECISION-001** below makes this explicit.
### G5-DECISION-001 — engine recovery state behavior across primary restart (named decision record)
**Status**: ⏳ DECISION REQUIRED at G5 collective close (architect-bound 2026-04-25 round-47).
**Question**: when the primary process restarts mid-recovery (panic, deploy, container restart, OS kill), what happens to in-flight Recovery state (`Attempts` counter, `TargetLSN`, `ActiveSessionID`, retry budget consumed)?
**Two paths** (G5 must explicitly choose one):
- **A — persist Recovery state**: engine state durably stored (somewhere TBD); restart resumes from persisted state. Pros: retry budget honored across restart; in-flight session can resume. Cons: substrate work; restart-during-persist atomicity; new failure mode if persistence fails.
- **B — rebuild state from probe after restart**: engine starts cold; probes replicas; reconstructs `Recovery.R` from probe responses; resets `Attempts` to 0. Pros: simpler; no new substrate. Cons: retry budget effectively unbounded (each restart resets); operator can't tell "this replica has failed 9 times" if interspersed with restarts.
**Why this is a G5 decision, not a T4d-4 implementation**:
- T4d-4 hard gate is "engine-driven recovery wiring works in steady-state." Restart semantics is orthogonal.
- Path A vs B has architectural weight (persistence = new substrate concern; both have observable behavior differences operators must understand).
- Choosing requires production-context input (operations team's tolerance for retry-budget reset on restart, persistence cost vs simplicity).
- G5 close demands the decision recorded; the **implementation** of the chosen path may live in G5 or post-G5.
**T4d-4 obligation**: ensure engine state structure is **decision-A-compatible** (i.e., serializable; no embedded non-serializable types). This keeps either path open. Architect call at T4d-4 close if decision A would require structural changes that should be pre-positioned now.
**Owner of the decision**: architect at G5 collective close.
**Risk if silent**: G5 closes assuming "recovery works in production" without anyone having decided what restart does. First production primary restart surprises operators; depending on path, retry budget is either nonsense (B + frequent restarts → infinite retries) or fails loudly with no fallback (A + persistence layer not chosen).
---
## §6 Invariants to preserve (forward-carry baseline)
Per kickoff §4 #5. Named invariants from T4a/T4b/T4c that MUST stay green:
| Source | Invariant | T4d regression risk | Protection |
|---|---|---|---|
| T4a-2 | `INV-REPL-SHIP-TRANSPORT-MUSCLE-001` | Low — Ship not touched | T4a-2 suite via G-3 |
| T4a-4 | `INV-REPL-LSN-ORDER-FANOUT-001` | **Medium — T4d-3 changes catch-up scan boundary** | Forward-carry test in T4d-3; lock scope at PR |
| T4a-4 | `INV-REPL-PEER-REBUILD-ON-AUTHORITY-CHANGE` | Low | T4a-4 test |
| T4a-3 | `INV-REPL-FANOUT-001` | Low | T4a-6 BasicEndToEnd |
| T4b-1 | `INV-REPL-LINEAGE-BORNE-ON-BARRIER-ACK` | Low — wire untouched | T4b-1 + T4b-6 wire fence |
| T4b-3 | `INV-REPL-BARRIER-FAILURE-DEGRADES-PEER` | Low | T4b-3 + T4b-4 peer-state pin |
| T4b-4 | `INV-REPL-LOCAL-FSYNC-GATES-QUORUM` | Low | T4b-4 |
| T4b-1 | `INV-REPL-NO-ZERO-LINEAGE-FIELDS` | Low — no new wire surfaces in T4d | T4b-1 fence still applies |
| T4c-1 | `INV-REPL-LINEAGE-BORNE-ON-PROBE-PAIR` | Low — probe wire untouched | T4c-1 fence |
| T4c-1 | `INV-REPL-PROBE-NON-MUTATING-VALIDATION` | Low — probe handler untouched | T4c-1 test |
| T4c-2 | `INV-REPL-CATCHUP-CALLBACK-RETURN-NIL-CONTINUES` | **Medium — T4d-3 changes scan boundary** | Forward-carry test in T4d-3 |
| T4c-2 | `INV-REPL-CATCHUP-LASTSENT-MONOTONIC` | **Medium — T4d-3 R+1 changes start LSN** | Forward-carry test in T4d-3 |
| T4c-2 | `INV-REPL-CATCHUP-DEADLINE-PER-CALL-SCOPE` | Low — deadline discipline preserved | T4c-2 test |
| T4c-2 | `INV-REPL-CATCHUP-TARGET-NOT-REACHED-VS-RECYCLED-DISTINGUISHED` | **Medium — T4d-1 structured kind changes detection** | T4d-1 test must distinguish the two via typed kind, not substring |
| T4c-2 | `INV-REPL-CATCHUP-RECYCLE-ESCALATES` | **High — pinning method changes from substring to structured kind** | T4d-1 + T4d-2 + T4d-3 chain |
| T4c-2 | `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN` | Low — completion semantics unchanged | T4c-2 test |
| T4c-2 | `INV-REPL-PEER-STATE-CATCHINGUP` | Low | T4c-2 test |
| T4c-3 | `INV-REPL-RECOVERY-MODE-OBSERVABLE` | **Medium — T4d-4 replaces duck-typed probe with method** | T4d-4 test |
| T4c retry pin | retry-budget-per-content-kind defaults | Low | T4c-3 unit test |
---
## §7 Review gates
### §7.1 Pre-merge gates (mandatory; PR blocked until satisfied)
**Gate G-1 — V2 diff-footprint review on muscle-path tasks.**
Applies to **T4d-3 ONLY** in T4d. NOT required for:
- T4d-1: V3-native + small substrate edits (no V2 muscle source)
- T4d-2: V3-native; round-43 + round-44 architect text IS the spec
- T4d-4: port + tests
T4d-3 G-1 deliverable per kickoff §3 sign #3 + §9.5 split: sw produces `sw-block/design/v3-phase-15-t4d-3-g1-v2-read.md` separating PORT items (5) from V3-native items (5). QA signs pre-code per round-11 / round-15 / round-24 / T4c-2 G-1 discipline. Sw does not write T4d-3 production code until QA signs the G-1.
**Procedural binding** (QA v0.1 review NOTE 2): every T4d-3 PR commit message MUST reference the QA-signed G-1 doc by commit hash (the sw commit that landed the QA-sign edits). Format: `Refs G-1 sign: <sha>`. This makes the procedural chain auditable and prevents post-hoc claims of "I didn't see the G-1 finding."
**Process rule inscribed (architect-bound 2026-04-25 round-46+ Issue 2(a) ratification):**
G-1 sign docs land in `seaweedfs/sw-block/design/` FIRST; sw implementation in `seaweed_block` references the committed G-1 hash via `Refs G-1 sign: <sha>` per the binding above. Avoids the T4d-3 procedural lesson where `44c60dd` referenced "pending hash" because the G-1 doc was still untracked at commit time. T4d-3 G-1 doc landed at `seaweedfs@80036404c` (2026-04-25); future references to T4d-3 G-1 use this sha.
**Gate G-2 — Three-line godoc contract on every new public method.**
Applies to every new exported symbol in T4d-1 / T4d-2 / T4d-3 / T4d-4:
- `// Called by:`
- `// Owns:`
- `// Borrows:`
**Gate G-3 — Cumulative suite stays green.**
Every T4d-N commit: full `go test -count=1 ./...` green. Particular attention:
- `core/storage/` — substrate hardening + AppliedLSNs() new method
- `core/replication/` — apply gate + lifecycle changes
- `core/transport/` — catchup_sender signature change
- `core/engine/` — substring match removal + retry-loop end-to-end wiring
- `core/replication/component/` — adversarial tests un-skip + framework wraps update
### §7.2 Risks + procedural mitigations
| Risk | Mitigation | §8C.3 trigger? |
|---|---|---|
| T4d-2 apply gate misses lane discrimination edge case (recovery enters via wrong handler) | `INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` test + dedicated lane-routing scenarios | trigger #1 if cross-lane leak post-merge |
| T4d-2 Option C hybrid `appliedLSN` seed differs from substrate truth on race | Restart-safety test + substrate-query semantics fence | trigger #1 if seed-vs-truth divergence |
| T4d-3 R+1 threading regresses T4c L2 matrix | Forward-carry tests for `INV-REPL-CATCHUP-CALLBACK-RETURN-NIL-CONTINUES` + `INV-REPL-CATCHUP-LASTSENT-MONOTONIC` under R+1 | trigger #1 if any T4c-2 invariant fails post-merge |
| T4d-3 G-1 mis-classifies V3-native item as PORT (or vice versa) | QA G-1 review uses §9.5 split as checklist | catch at G-1 sign |
| T4d-1 substring removal misses a call site | `TestEngine_SessionFailed_NoMoreSubstringMatch` fence + grep at PR | trigger #1 if substring sneaks back |
| T4d-4 lifecycle regression mis-handles pending sessions | BUG-005 non-repeat tests; per-state cleanup pins | trigger #1 if BUG-005 reproducible |
| Substrate defense-in-depth (T4d-1) turns into refactor | §2.5 #3 caveat: "if practical" — sw judgment + documented carry to T4e if needed | acceptable carry |
---
## §8 Sign table (mid-T batch, §8C.2)
| Role | Signer | Date | Decision |
|---|---|---|---|
| QA Owner | Claude (QA agent) | _________ | ⏸ pending T4d close |
Per §8C.2: no architect / PM sign at batch level. Architect + PM sign at T-end via closure report (kickoff §4 #10 / criterion #10) — lands at T4d close IF T4d remains the final T4 batch.
---
## §9 Change log
| Date | Change | Author |
|---|---|---|
| 2026-04-25 | Initial T4d mini-plan v0.1 drafted from kickoff v0.3 (§1–8 + §9 RATIFIED 2026-04-25). 4 batches per kickoff §3 sign: T4d-1 substrate hardening + structured failure kind (~150 LOC + ~80 tests, no G-1) → T4d-2 replica recovery apply gate (~250 LOC + ~250 tests, no G-1, V3-native spec from round-43/44) → T4d-3 R+1 + engine-driven recovery wiring (~250 LOC + ~150 tests, **G-1 required** per §9.5 split) → T4d-4 lifecycle + integration (~150 LOC + ~200 tests, no G-1). Incorporates §9 walshipper architecture: T4d-3 is bounded V2 muscle PORT (5 items) + V3-native items (5 items) per architect §9.5 binding; T4d-3 is NOT a wholesale `WALShipper` port. Q1 engine-owned retry, Q2 lane implicit from handler context, Q3 transport↔storage contract-only inscribed via `INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` + `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY`. BlockStore walHead pre-T4d hotfix already landed (`f6084ee`); §4 #8 acceptance criterion shows DONE. Predicates: T4c closed + round-43/44 + kickoff §1–9 RATIFIED + hotfix landed. T-end three-sign at T4d close conditional on no T4e per kickoff §4 #10. Total ~800 LOC production + ~680 LOC tests across 4 batches. | sw |
| 2026-04-25 (v0.2) | Architect v0.1 review absorbed. **HIGH BLOCKER FIX**: structured failure kind now end-to-end typed contract — `RecoveryFailureKind` enum + `RecoveryFailure` typed-error in storage; new `FailureKind storage.RecoveryFailureKind` field on **`adapter.SessionCloseResult`** AND **`engine.SessionClosedFailed`**; engine `applySessionFailed` branches on `e.FailureKind`, NOT on `containsAny(reason, ...)` substring; `Reason`/`FailReason` strings explicitly downgraded to "DIAGNOSTIC TEXT ONLY — engine MUST NOT parse"; new fence test `TestEngine_SessionFailed_NoMoreSubstringMatch` greps source for `"WAL recycled"` substring. Cross-package import note: typed-kind field doesn't widen dependency surface beyond existing storage-recovery-contract import (T4c-2 already established). **MED FIX**: every `LogicalStorage` impl now has explicit `AppliedLSNs()` body — walstore via `dm.snapshot()`, smartwal via `ring.scanValid()` reduction, BlockStore explicitly returns `(nil, ErrAppliedLSNsNotTracked)` (new sentinel); gate falls back to session-only map on the sentinel. **LOW FIX**: §0 PORT/V3-native table gains 5th V3-native row (package-boundary discipline / `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY`) per kickoff §9.5. **QA test rename**: `TestApplyGate_BarrierAchievedLSN_DerivesFromCoverageNotData` → `TestApplyGate_BarrierCompletion_BarrierAuthoritativeWithCoverageContribution` (avoids misread as conflicting with T4c-2 `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN`). **QA NOTE 2 procedural absorbed**: §7.1 G-1 fence now binds T4d-3 PR commits to reference signed G-1 by hash (`Refs G-1 sign: <sha>`). T4d-1 test count grows from 8 to 10 (adds typed-kind end-to-end + event-shape fence). T4d-1 LOC est revised slightly upward (~170 prod + ~100 tests) due to adapter + engine event-struct contract additions. | sw |
| 2026-04-25 (v0.3) | Architect v0.2 dependency blocker absorbed. v0.2's typed-kind direction was correct but put `storage.RecoveryFailureKind` directly on engine events, which would create a new `core/engine → core/storage` dependency. v0.3 fixes the boundary: storage owns substrate-side `StorageRecoveryFailureKind` + `RecoveryFailure`; transport extracts that with `errors.As` and maps it to **engine-owned** `engine.RecoveryFailureKind`; `adapter.SessionCloseResult.FailureKind` uses `engine.RecoveryFailureKind`; `engine.SessionClosedFailed.FailureKind` uses the local engine type. `core/engine` MUST NOT import `core/storage`; `Reason`/`FailReason` remain diagnostic-only text. Test list label corrected to 10 required and mapping tests now explicitly cover storage-kind → engine-kind translation. | architect |
| 2026-04-25 (v0.4) | Architect Path B ratification absorbed (round-46+). T4d-3 `44c60dd` deferred engine-driven recovery wiring (`ReplicationVolume↔adapter` + `WithEngineDrivenRecovery()` real binding + full L2 matrix); architect chose Path B = fold into T4d-4 as **HARD CLOSE GATE** (not soft carry). T4d-3 §2.3 closure scope clarified as "R+1 bounded catch-up PORT + boundary fences" (5 G-1 PORT items + 2 hidden invariants + Q3 fence + CARRY-T4D-LANE-CONTEXT-001 Option B); engine-driven set explicitly relocated to T4d-4. T4d-4 §2.4 scope expanded with Path B fold rows (3 production sites + 5 tests relocated/added); LOC est revised ~150/~200 → ~300/~350. T4d HARD CLOSE GATE inscribed: cannot close until `WithEngineDrivenRecovery()` is REAL (not stub), `ReplicationVolume↔adapter` wiring runs end-to-end, QA #8 (`TestComponent_LastSentMonotonic_AcrossRetries_FullForm`) passes, and acceptance criterion #3 (full L2 matrix engine→adapter→executor end-to-end) satisfied — same bar T4c narrowed and T4d-3 deferred per Path B. **Process rule** (Issue 2(a) ratification): G-1 docs land in `seaweedfs` FIRST; sw implementation references committed hash via `Refs G-1 sign: <sha>`. T4d-3 G-1 doc landed at `seaweedfs@80036404c` (this revision). Architect explicitly rejected Path C (defer to G5) — "G5 should not inherit the same full-L2 gap again." | architect |
| 2026-04-25 (v0.5) | Architect round-47 ruling absorbed (post-T4d-3 production-readiness scope conversation). **T4d-4 HARD CLOSE GATE expanded with rebuild path (4th item)**: catch-up engine-driven wiring alone is insufficient for T4d close; rebuild path must also be engine-driven end-to-end. Minimum: catch-up failure/budget exhausted → engine emits actual rebuild command (not just `Decision = Rebuild` state); rebuild session completion/failure → returns to engine; rebuild failure terminal explicitly defined; L2 covers real path (test observes actual rebuild session, not just state transition). 4 new tests added in §2.4 Tests block. **Engine recovery state persistence (E1)**: architect explicitly EXCLUDED from T4d-4 implementation gate; instead inscribed as **G5-DECISION-001** named decision record in §5. G5 must choose Path A (persist) vs Path B (rebuild from probe after restart) explicitly; T4d-4 obligation is to keep engine state structure decision-A-compatible (serializable) so either path stays open. **Lane handler-context (CARRY-T4D-LANE-CONTEXT-001)** correction: NOT promoted to G5 hard blocker (it's already explicitly carried via Option B test-skip with non-claim); stays at T4e/post-G5 per existing inscription. Scope map clarified: T4d-4 = catch-up + rebuild engine-driven wiring; G5 = multi-replica mixed states + m01 + G5-DECISION-001 + minimal metrics/backpressure assessment; post-G5/T4e = lane fix + snapshot catch-up + protocol negotiation + auth/encryption. | architect |