From 80036404ceea26a0d776a0a048adb6037c543fd2 Mon Sep 17 00:00:00 2001 From: pingqiu Date: Sat, 25 Apr 2026 22:26:21 -0700 Subject: [PATCH] T4d planning + G-1 doc landing (architect Path B + Issue 2(a) ratification) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lands four T4d planning artifacts together: 1. v3-phase-15-t4d-3-g1-v2-read.md (NEW) T4d-3 G-1 V2 read v0.2, QA-signed in conversation 2026-04-25. Per architect Issue 2(a) ratification: G-1 docs land first; implementation references the committed hash. Future T4d-3 commits should reference this commit's sha via: Refs G-1 sign: 2. v3-phase-15-t4d-forward-carry-checklist.md (NEW) v0.2 — 19 active T4a/T4b/T4c invariants with risk grades and per-batch focus rows. T4d-3 close gate inscribed (CARRY-T4D-LANE-CONTEXT-001 option A or B); pre/with-T4d-3 doc fixes recorded. 3. v3-phase-15-t4d-qa-scenario-catalogue.md (NEW) v0.1 — 9 QA component-scope scenarios mirroring T4c QA Stage-1 discipline. 10 framework primitives surfaced for sw's batch PRs. 4. v2-v3-contract-bridge-catalogue.md (UPDATED) §3.3 inscriptions for T4d-locked invariants: - INV-REPL-NO-PER-LBA-DATA-REGRESSION (round-43) - INV-REPL-RECOVERY-STALE-ENTRY-SKIP-PER-LBA (round-43) - INV-REPL-RECOVERY-COVERAGE-ADVANCES-ON-SKIP (round-44) - INV-REPL-LIVE-LANE-STALE-FAILS-LOUD (round-44) - INV-REPL-RECOVERY-COVERAGE-RESTART-SAFE (Option C) - INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT (Q2 + round-46) - INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY (Q1+Q3 + T4d-1 strengthening) - INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1 (T4d-3 G-1 §5) - INV-REPL-CATCHUP-FROMLSN-FROM-ENGINE-STATE-NOT-PROBE (T4d-3 G-1 §5) - CARRY-T4D-LANE-CONTEXT-001 (named carry, T4e/post-G5) INV-REPL-CATCHUP-WITHIN-RETENTION-001 status updated: T4c downgrade → T4d-2+T4d-3 un-pin path. Process rule inscribed (architect 2026-04-25): G-1 sign docs land in seaweedfs FIRST; sw implementation in seaweed_block references the committed G-1 hash via "Refs G-1 sign: " per mini-plan §7.1 procedural binding. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../design/v2-v3-contract-bridge-catalogue.md | 344 +++++++++++++- .../design/v3-phase-15-t4d-3-g1-v2-read.md | 429 ++++++++++++++++++ ...v3-phase-15-t4d-forward-carry-checklist.md | 144 ++++++ .../v3-phase-15-t4d-qa-scenario-catalogue.md | 370 +++++++++++++++ 4 files changed, 1271 insertions(+), 16 deletions(-) create mode 100644 sw-block/design/v3-phase-15-t4d-3-g1-v2-read.md create mode 100644 sw-block/design/v3-phase-15-t4d-forward-carry-checklist.md create mode 100644 sw-block/design/v3-phase-15-t4d-qa-scenario-catalogue.md diff --git a/sw-block/design/v2-v3-contract-bridge-catalogue.md b/sw-block/design/v2-v3-contract-bridge-catalogue.md index 8353e6abe..26bc9213e 100644 --- a/sw-block/design/v2-v3-contract-bridge-catalogue.md +++ b/sw-block/design/v2-v3-contract-bridge-catalogue.md @@ -424,30 +424,342 @@ If the entity is `new` (§1.0), contracts are fresh and subject to §8C.5 semant --- -## §3 Replication (T4 / G5) — MANDATORY BEFORE T4 T-START THREE-SIGN +## §3 Replication (T4 / G5) -### §3.1 Entity bridge map (V2 → V3) — TO BE FILLED pre-T4 +**Status**: LIVING — T4 rounds 1–11 consolidated 2026-04-23; architect re-read pending before T4 T-start three-sign; architect-line items A/B/C at sketch §6.1 still open for sign. -Expected V2 entities (per `v2-test-db.md` + memory references): +**Scope**: T4 delivers primary-writes-to-N-replicas with explicit durability ack and short-disconnect tolerance. Rebuild beyond WAL retention → T5. Failover → T6. -- `wal_shipper` (WAL ship protocol state machine) — session-scope -- `shipper_group` (multi-replica fan-out) — volume-scope -- `replica_apply` (receiver side) — volume-scope -- `replica_barrier` (write ack barrier) — volume-scope -- Replication session lifecycle (per-connection between primary + replica) -- Durable replicated LSN (state on primary + replica) +### §3.0 Framing -Expected V3 topology: TBD — sw + QA T4 sketch must enumerate. 1:1 / split / merge / new for each. +This section consolidates T4 port judgments from the six T4-specific docs into the canonical catalogue format: -### §3.2 Contract catalogue (per entity) — TO BE FILLED +- `v3-phase-15-t4-l1-survey.md` — V2 entity inventory (§2.1–§2.10) +- `v3-phase-15-t4-v3-entity-skeleton.md` — V3 entity reality (layers A–E) + skeleton signatures +- `v3-phase-15-t4-function-level-audit.md` — 24-method action classification + batch implications +- `v3-phase-15-t4-sketch.md` — T4 scope/gate/batches +- `v3-phase-15-t4a-mini-plan.md` — T4a task list + Gates +- `v3-phase-15-t4a-2-g1-v2-read.md` — V2 Ship source read + Option B shape -### §3.3 V3 model shifts relevant to replication +#### §3.0.1 Architect-LOCKed framings (round 4) -Pre-fill hints (subject to T4 sketch authoring): +- **H5 LOCK**: Replication wire frames carry authority facts explicitly; no primary-side cache is correctness source. Two invariants inscribed across §3 rows: `INV-REPL-ACK-FRAME-IS-AUTHORITY` + `INV-REPL-CACHE-ADVISORY-ONLY`. +- **Q5 LOCK**: V3 replication wire is clean-break from V2; versioned envelope `{magic, protocol_version, message_type, flags_or_reserved}` from day one. No V2 replica interop in T4. Envelope **landed at T4a-1** (commit `seaweed_block@56ad349`): 12B preamble = 4B magic `SWRP` + 1B protocol_version=1 + 1B message_type + 1B flags + 1B reserved + 4B length. Architect acceptance: the landed shape matches the Q5 LOCK recommendation as drafted in sketch §6.1 Item C.1; any future revision requires a §8C.3 Discovery Bridge (not a plain sketch revision). Sketch §6.1 Item C remaining open surface is narrowed to C.2 (BarrierResponse epoch echo) + C.3 (ProbeResponse epoch echo) only — the envelope layout question is no longer an architect-line decision surface. +- **Item B pending**: T4c catch-up semantics — Path A (retained-WAL delta, preserves T5 self-definition) vs Path B (full-transfer, defers retention-window + T5 rebuild boundary to P10). Architect decides at T4 T-start three-sign. -- *Event model*: V2 replication callbacks (barrier fn, degradation fn) → V3 likely typed error return + ctx cancellation -- *Authority model*: V2 primary advanced epoch on promotion locally → V3 master authority publishes; replica storage is passive recipient -- *Concurrency model*: V2 shipper had bounded in-flight state → V3 likely similar but with clean context cancellation +#### §3.0.2 Substrate discipline (inherited from T3) + +- **smartwal** = sole product-signing substrate for G5; walstore is non-gating secondary-matrix coverage until BUG-007 closes. T4 does NOT silently re-open BUG-007 as a hidden blocker. + +#### §3.0.3 V3 model shifts relevant to replication + +- **Authority model**: V2 primary advanced epoch locally on promotion; V3 master authority publishes, replica storage is passive recipient. Per-frame epoch on wire is correctness source (H5 LOCK). +- **Event model**: V2 replication callbacks (barrier fn, degradation fn) → V3 typed error return + ctx cancellation + `transport.OnSessionClose/OnFenceComplete` hooks. +- **Storage model**: V2's local `BlockVol` was monolith (engine + replication + host-session); V3 decomposes — `DurableProvider` owns engine lifecycle, `ReplicationVolume` owns replication lifecycle, `StorageBackend` owns host-session I/O. BUG-005 lesson drives "borrowed vs owned" contract on every new public method (Gate G-2). +- **Lifecycle model**: V2 shippers/receivers had internal dial/reconnect logic; V3 `core/transport/` is sole `net.Dial` + frame-I/O owner (all peer execution goes through `*BlockExecutor` public methods). +- **Concurrency model**: V2 shipper had bounded in-flight state with per-peer mutex; V3 preserves bounded in-flight but uses typed ctx cancellation and `core/transport/` session/conn attach/detach helpers. + +#### §3.0.4 Status legend (for the per-entity rows below) + +- **✓ PORTED** — V3 code exists and merged on `phase-15` branch +- **⊙ DECIDED-NOT-CODED** — T4 plan names concrete shape + action; body to be written per T4a/b/c mini-plans +- **⏭ DEFERRED-T5** — explicit non-claim in T4; rebuild-lane or post-G5 +- **~ PARTIAL** — some parts PORTED, others DECIDED-NOT-CODED + +--- + +### §3.1 Entity bridge map (V2 → V3) + +Ten V2 stateful entities (per L1 survey §2) + four V3-native entities (per skeleton §2.E). Consolidated verdicts — see §3.2 per-entity rows for detail. + +| L1 ref | V2 entity | V2 scope | Bridge shape | V3 home | T4 status | +|---|---|---|---|---|---| +| §2.1 | WALShipper | per-replica | **split + partial-port** | `transport.BlockExecutor.Ship` (new method, T4a-2) + `ReplicaPeer` state (T4a-3) | ~ PARTIAL | +| §2.2 | ShipperGroup | per-volume | **merge into new V3 entity** | `ReplicationVolume` (new, T4a-4) | ⊙ DECIDED | +| §2.3 | ReplicaReceiver | per-volume (replica-side) | **split — wire/frame + storage-apply already V3-covered; state-machine to T4c** | `transport.ReplicaListener` (V3-covered) + `storage.LogicalStorage.ApplyEntry` (V3-covered) + `ReplicaPeer` state (T4c adds full state machine) | ~ PARTIAL | +| §2.4 | ReplicaBarrier FSM | per-request closure + per-volume queue | **merge into new V3 entity** | `DurabilityCoordinator.SyncLocalAndReplicas` + per-peer barrier calls (T4b) | ⊙ DECIDED | +| §2.5 | DistGroupCommit | per-write closure | **1:1-shift into new V3 entity** (closure → entity; same math) | `DurabilityCoordinator.EvaluateBarrierAcks` (T4b) | ⊙ DECIDED | +| §2.6 | RebuildSession | per-session volatile | **merge into new T5 entity** | `RebuildCoordinator` (T5) | ⏭ DEFERRED-T5 | +| §2.7 | RebuildServer | per-primary listener | **split — listener already V3-covered via `BlockExecutor.StartRebuild`; policy to T5** | `transport.BlockExecutor.StartRebuild` (V3-covered, semantic-gap per §3.2.7) + `RebuildCoordinator` policy (T5) | ~ PARTIAL | +| §2.8 | RebuildTransportServer | per-session | **already V3-covered** | `transport.rebuild_sender.go` internals (private) | ✓ PORTED | +| §2.9 | RebuildTransportClient | per-session replica-side | **already V3-covered** | `transport.ReplicaListener` receive path (V3-covered) | ✓ PORTED | +| §2.10 | RebuildBitmap | per-session volatile | **move to T5 entity** | `RebuildCoordinator` internal state (T5) | ⏭ DEFERRED-T5 | +| — | (new) ReplicationVolume | per-volume | **V3-native, no V2 analog** (closes gap left when ShipperGroup merged) | `core/replication/volume.go` (new package, T4a-4) | ⊙ DECIDED | +| — | (new) ReplicaPeer | per-replica | **V3-native, no V2 analog** (collects per-peer state the V2 WALShipper carried implicitly) | `core/replication/peer.go` (T4a-3, T4b extends, T4c state machine) | ⊙ DECIDED | +| — | (new) DurabilityCoordinator | per-volume | **V3-native, no V2 analog** (replaces V2's `vol.writeSync` closure binding) | `core/replication/durability.go` (T4b) | ⊙ DECIDED | +| — | (new) RebuildCoordinator | per-volume | **V3-native, no V2 analog** | `core/replication/rebuild.go` (T5) | ⏭ DEFERRED-T5 | + +--- + +### §3.2 Per-entity detail + +Each row carries V2 attributes (from L1 survey) + V3 home + per-contract verdict + test anchor. Contracts marked with LOC and status. + +#### §3.2.1 WALShipper (L1 §2.1) — split + partial-port + +- **V2 file**: `weed/storage/blockvol/wal_shipper.go` (lines 57–912) +- **V2 scope**: per-replica; one goroutine set per replica in fan-out +- **Bridge shape**: split — V2's `Ship()` kernel (98 LOC per G-1 V2 read) decomposes into multiple V3 homes; state machine lifts to `ReplicaPeer` +- **T4 status**: ~ PARTIAL — T4a-2 adopts **Option B** (architect round 11) per §3.4 drift event; sw coding in progress on `phase-15` branch; round-11 Gate G-1 re-signed + +**Contracts**: + +| ID | Statement | V3 verdict | Rationale + impl | Test anchor | Status | +|---|---|---|---|---|---| +| C1-SHIP-EPOCH-EQ-SILENT-DROP | Stale-epoch entry dropped silently (`return nil`); caller treats as success; V2 line 217 | **PRESERVE** verbatim | `BlockExecutor.Ship` at `ship_sender.go:89-95` checks `lineage.Epoch != session.lineage.Epoch`; logs + returns nil. Source of epoch: frame-borne lineage (H5 LOCK compliant), replacing V2's `epochFn()` callback | `TestExecutor_Ship_StaleEpoch_SilentDrop` (landed `seaweed_block@043b9f7`) | ✓ PORTED T4a-2 | +| C2-SHIP-3S-WRITE-DEADLINE | `conn.SetWriteDeadline(time.Now().Add(3*time.Second))` around `WriteMsg`; V2 line 240 | **PRESERVE** byte-identical | `ship_sender.go:143` via `shipWriteDeadline` const; set before write, cleared after | `TestExecutor_Ship_WriteDeadline_Fires` landed `seaweed_block@8fed2a8`; uses `net.Pipe()` to bypass kernel-buffer auto-tuning; three-way assertion (err≠nil + elapsed≥2.5s + elapsed≤10s) proves deadline IS the unblocking mechanism. Runs 3.00s exactly on Windows loopback | ✓ PORTED T4a-2 | +| C3-SHIP-LAZY-DIAL-ON-REGISTERED-SESSION | `ensureDataConn` — if the **already-registered** session's `conn` is nil, `net.DialTimeout` + attach; V2 line 246. **Scope boundary**: Ship does NOT bootstrap session creation — if the session is not registered, Ship returns error and does not auto-register. Session registration is done by `Probe` / `StartCatchUp` / `StartRebuild` / future steady-state-attach path | **PRESERVE** per Option B (round 11) | `BlockExecutor.Ship` body at `ship_sender.go:87-90` returns error on unknown session; at `ship_sender.go:107-135` lazy-dials only when `session.conn == nil` for an already-registered session. Was NEARLY dropped at round 10 (Option C moved dial to ReplicaPeer layer); architect corrected at round 11 — "transport = sole net.Dial owner" layering principle applies, composite stability requires lazy-dial to stay in muscle | `TestExecutor_Ship_LazyDial_OnRegisteredSessionWithoutConn` + `TestExecutor_Ship_LazyDial_DialFailure_ReturnsError` + `TestExecutor_Ship_NoSession` (T4a-2, landed `seaweed_block@043b9f7`) | ✓ PORTED T4a-2 | +| C4-SHIP-NO-HARD-STOP | Ship failure returns error without panicking upstream caller; caller degrades peer and continues | **PRESERVE** — composite with C5 (handoff to peer-layer degradation) | Architect round-11 clarified: V2 `Ship()` does NOT retry in-call; it returns + caller (V2 shipper state machine) calls `markDegraded`. V3 Ship returns error; `ReplicaPeer.ShipEntry` (T4a-3) catches + calls `Invalidate`. **Behavior delta noted**: V2 returned nil after markDegraded (no error bubbled); V3 returns error because BlockExecutor does not own ReplicaState. Upstream composite-behavior preserved because ReplicaPeer layer absorbs the error (forward-carry CARRY-1) | `TestExecutor_Ship_ConnFailure` (T4a-2 `043b9f7`); `TestReplicaPeer_ShipEntry_ConnFailure_MarksDegraded` (T4a-3 `99c4e1d`) proves Healthy→Degraded via Invalidate | ✓ PORTED T4a-3 (CARRY-1 closed round 13) | +| C5-SHIP-REPLICA-FLUSHED-LSN-MONOTONIC | `replicaFlushedLSN` advances via `CompareAndSwap` forward-only; V2 line 414 | **DEFER to T4b** (barrier mode work; not T4a) | `ReplicaPeer.durableLSN` field; CAS forward-only on barrier ack | T4b barrier test (TBD) | ⏭ T4b | +| C6-SHIP-BARRIER-OK-WITHOUT-FLUSHED-LSN-NOT-SYNCALL | `BarrierOK` without `FlushedLSN` MUST NOT count toward `sync_all`; V2 legacy-replica safety | **DEFER to T4b** | Decision encoded in `DurabilityCoordinator.EvaluateBarrierAcks` quorum math | T4b barrier test | ⏭ T4b | +| C7-SHIP-MAX-CATCHUP-RETRIES-3 | After 3rd catch-up fail, escalate to `ReplicaNeedsRebuild`; V2 line 286 | **DEFER to T4c** | T4c recovery policy owns this (via `RebuildCoordinator.Decide` call path) | T4c recovery test | ⏭ T4c | +| C8-SHIP-ACTIVE-REBUILD-SESSION-FLAG | `activeRebuildSession` atomic prevents stale state transitions from opening live lane; V2 line 143 | **DEFER to T5** (coupled to RebuildSession) | Rebuild-lane concern | T5 rebuild tests | ⏭ T5 | + +**Partial-port audit (round 11)**: V2 Ship's 5 concerns — (1) stale-epoch drop → PRESERVE in C1; (2) 3s deadline → PRESERVE in C2; (3) lazy-dial → PRESERVE in C3 (after architect round 11); (4) ReplicaState machine → `ReplicaPeer.state` one layer up (same gate); (5) liveShippingPolicy gate → T4b. LOC estimate ~50 (was ~30 at round-10 Option C; revised to ~50 after Option B adoption). + +#### §3.2.2 ShipperGroup (L1 §2.2) — merge into ReplicationVolume + +- **V2 file**: `weed/storage/blockvol/shipper_group.go` (lines 19–408) +- **V2 scope**: per-volume (one group per primary); spans N configured replicas +- **V2 external dep**: RF = replica count — comes from master assignment via `BlockVol.SetReplicaAddrs([]string)`; group resizing is an assignment-op +- **Bridge shape**: merge — ShipperGroup's responsibilities decompose into `ReplicationVolume` (T4a-4; fan-out owner) + `DurabilityCoordinator` (T4b; quorum math) +- **T4 status**: ⊙ DECIDED — T4a-4 ships `ReplicationVolume` MVP + +**Contracts**: + +| ID | Statement | V3 verdict | Rationale + impl | Test anchor | Status | +|---|---|---|---|---|---| +| C1-SHIPPERGROUP-RF-EXTERNAL | N = RF supplied from master assignment via `UpdateReplicaSet`; group resizing = teardown+rebuild, never internal | **PRESERVE** — cross-entity contract: master assignment ↔ `ReplicationVolume.peers` map size ↔ `ReplicaListener` expected-connection-count ↔ `DurabilityCoordinator` quorum arithmetic | `ReplicationVolume.UpdateReplicaSet(targets []ReplicaTarget)` called from Host authority-callback path | `TestReplicationVolume_UpdateReplicaSet_AddPeer` + `_RemovePeer_ExecutorTornDown` (T4a-4) | ⊙ T4a-4 | +| C2-SHIPPERGROUP-BARRIER-ALL-PARALLEL | N goroutines in parallel via `WaitGroup`; accessors use `RWMutex.RLock` | **PRESERVE** — V3 `DurabilityCoordinator.SyncLocalAndReplicas` spawns N goroutines | T4b `DurabilityCoordinator` impl | T4b barrier parallel test | ⏭ T4b | +| C3-SHIPPERGROUP-MIN-REPLICA-FLUSHED-LSN-ALL | `MinReplicaFlushedLSNAll` returns (0, false) unless EVERY shipper reports progress | **PRESERVE** — quorum-arithmetic input | T4b `DurabilityCoordinator` | T4b quorum test | ⏭ T4b | +| C4-SHIPPERGROUP-RETENTION-BUDGET-TRANSITIONS-UNDER-RLOCK | `EvaluateRetentionBudgets` can flip shipper state to `ReplicaNeedsRebuild` under `RWMutex.RLock` | **DEFER to T4c** — retention-window behavior tied to Item B decision | `ReplicationVolume.EvaluatePeerProgress` (T4c) | T4c peer-state test | ⏭ T4c | +| C5-SHIPPERGROUP-DOUBLE-WATERMARK | `MinShippedLSN` (diagnostic, Ceph retention) vs `MinReplicaFlushedLSNAll` (authoritative durability) — two consumers, two semantics | **PRESERVE** — L1 §2.2 invariant #3 | Shipped LSN not tracked in T4a; both watermarks exposed in T4b via `DurabilityCoordinator` query methods | T4b watermark test | ⏭ T4b | +| C6-SHIPPERGROUP-ANY-HAS-FLUSHED-PROGRESS-SEED | Seeded on reassignment to detect cold-start shippers | **PRESERVE** | `ReplicationVolume.UpdateReplicaSet` initializes new `ReplicaPeer` with `lastProbe.ReplicaFlushedLSN = 0, ProgressSeen = false` | T4c reassignment test | ⏭ T4c | + +#### §3.2.3 ReplicaReceiver (L1 §2.3) — split; wire + apply V3-covered; full state machine T4c + +- **V2 files**: `weed/storage/blockvol/replica_apply.go` (lines 20–425) + `replica_barrier.go` (lines 10–204) +- **V2 scope**: per-volume replica-side singleton +- **Bridge shape**: split — network/frame-handling is covered by `transport.ReplicaListener`; storage-apply is covered by `LogicalStorage.ApplyEntry`; state machine (`receivedLSN`/`flushedLSN` tracking + barrier cond + `ioMu.RLock` nesting) distributes across T4b (barrier) + T4c (state machine) +- **T4 status**: ~ PARTIAL — wire/apply PORTED; full state-machine DECIDED (T4c) with bits in T4a + +**Contracts**: + +| ID | Statement | V3 verdict | Rationale + impl | Test anchor | Status | +|---|---|---|---|---|---| +| C1-REPLICARECV-STATE-ON-ENGINE-NOT-SINGLETON | V2 kept `receivedLSN`/`flushedLSN` on a per-volume `ReplicaReceiver` singleton that outlived promote/demote (`blockvol.go:1515`). V3 has no equivalent singleton: replica-side LSN counters are carried by `LogicalStorage` (one per `DurableProvider` cached volume handle) and surfaced via `LogicalStorage.Boundaries()` | **REBUILD** — container shape changed; contract surface changed | V3 impl: `LogicalStorage.Boundaries()` returns `(R, S, H)` tuple; `transport.ReplicaListener` reads/advances via `ApplyEntry`/`AdvanceFrontier`. **Residual property**: "LSN state survives across an assignment churn that keeps the same volume cached in Provider" follows from `DurableProvider.volumes` map — BUG-005 regression indirectly fences a related lifetime property, but does NOT prove the specific demote→promote→read-LSN-unchanged sequence for replica-side state | **Proof coverage is narrow today**: `TestT3_Durable_RecoverThenServe` proves *process-restart* durability only, not *in-process demote/promote* lifetime. A dedicated proof — e.g. `TestT4c_ReplicaLSN_SurvivesDemotePromote` — is scoped for **T4c** when the full state machine lands | ⏭ T4c (proof); ✓ engine-side surface is PORTED via T3 | +| C2-REPLICARECV-CONTIGUOUS-LSN | `entry.LSN == receivedLSN+1` fence; gaps → `ErrDuplicateLSN`; V2 line 306 | **PRESERVE** | `LogicalStorage.ApplyEntry` enforces contiguous LSN | `walstore` / `smartwal` apply tests (V3-covered) | ✓ PORTED | +| C3-REPLICARECV-EPOCH-FENCE | `entry.Epoch` vs `vol.epoch.Load()`; mismatch → `ErrStaleEpoch` silent drop; V2 line 294 | **PRESERVE** — same silent drop as C1-SHIP | Replica-side epoch source: receiver's most-recent observed lineage from `RecoveryLineage` frame (H5 LOCK compliant) | T4c apply test covering epoch mismatch on replica | ⊙ T4c | +| C4-REPLICARECV-BARRIER-3-PHASE | Wait-LSN → `fd.Sync()` → atomically advance `flushedLSN`; V2 `replica_barrier.go:147-204` | **PRESERVE** across `DurabilityCoordinator` + `LogicalStorage.Sync` | Barrier arrives over wire; `ReplicaListener` routes to Sync; `BarrierResponse` includes epoch echo (H5 LOCK; §3.3 C.2) | T4b barrier test | ⊙ T4b | +| C5-REPLICARECV-IOMU-RLOCK-NESTING | `ioMu.RLock` held for entire apply path; released inside `replicaAppendWithRetry` during WAL-full wait; V2 line 287 | **DEFER to T4c** — if needed; V3's `LogicalStorage.ApplyEntry` may already satisfy via internal flusher synchronization (verify at T4c) | L1 §2.3 invariant #5; verify V3 equivalent during T4c | T4c concurrent apply test | ⏭ T4c | +| C6-REPLICARECV-WAL-FULL-RETRY-RELEASES-MU | `replicaAppendWithRetry` releases `mu` on full → re-acquires after flusher notify | **DEFER to T4c** (if needed; V3 flusher may have different pressure-release seam) | L1 §2.3 invariant #4 | T4c WAL-full test | ⏭ T4c | +| C7-REPLICARECV-REBUILD-SESSION-ROUTING | Entries during `RebuildPhaseRunning` route to session WAL lane; else normal apply | **DEFER to T5** | Rebuild-lane | T5 | ⏭ T5 | + +#### §3.2.4 ReplicaBarrier FSM (L1 §2.4) — merge into DurabilityCoordinator + +- **V2 file**: `weed/storage/blockvol/replica_barrier.go` (lines 147–204) +- **V2 scope**: per-request call-closure, BUT queue-state shared per-volume via `cond.Wait()` (L1 §2.4 clarified scope) +- **Bridge shape**: merge — barrier FSM responsibilities absorb into `DurabilityCoordinator.SyncLocalAndReplicas` closure + per-peer wire round-trip +- **T4 status**: ⊙ DECIDED — T4b + +**Contracts**: + +| ID | Statement | V3 verdict | Rationale + impl | Test anchor | Status | +|---|---|---|---|---|---| +| C1-BARRIER-3-PHASE | Wait-LSN-receipt → fsync → advance `flushedLSN` | **PRESERVE** (from §3.2.3 C4) | `DurabilityCoordinator.SyncLocalAndReplicas` drives; `LogicalStorage.Sync` performs fsync | T4b barrier 3-phase test | ⊙ T4b | +| C2-BARRIER-TIMEOUT-5S-HARDCODED | `barrierTimeout=5s` — V2 line 18 hardcoded | **REBUILD** — V3 elevates to config (context-cancellable; default 5s) | `ctx` arg on `SyncLocalAndReplicas` | T4b timeout test | ⊙ T4b | +| C3-BARRIER-COND-MULTI-WATCHER | Two barriers in quick succession share cond; first fsync satisfies both LSN requirements | **REBUILD** — V3 uses ctx + goroutine per barrier (simpler, no cond needed) | Per-call goroutine in `DurabilityCoordinator` | T4b concurrent-barrier test | ⊙ T4b | +| C4-BARRIER-LEGACY-1B-RESPONSE | V2 supports 1-byte response from old replica decoder at `repl_proto.go:64` | **BREAK** per Q5 LOCK (clean-break wire) | V3 `BarrierResponse` structured + versioned | — (no V2 interop claimed) | ✓ decided | +| C5-BARRIER-RESP-FULL-LINEAGE-ECHO | **Architect round-21 decision text (verbatim; drops into T4b mini-plan §2 T4b-1)**: "`BarrierResponse` adopts **full `RecoveryLineage` echo** and extends its payload to **`[32B lineage][8B achievedLSN]`**, carried inside the existing `MsgBarrierResp` body; the 12B envelope preamble remains unchanged. Encode/decode is strict and fail-closed: payloads shorter than 40 bytes, zero-valued or malformed lineage fields, or decode layouts that do not exactly match the field order `SessionID, Epoch, EndpointVersion, TargetLSN, AchievedLSN` MUST be rejected as invalid barrier acknowledgements and MUST NOT contribute to correctness decisions. Receiver-side filtering MUST treat the echoed lineage as the authority identity of the ack: a barrier ack only counts if its full lineage matches the request/session the caller is awaiting; mismatched, stale, or partially-zeroed lineage is advisory for logging only and MUST NOT be accepted. `T4b-2` may initially key its coordinator logic primarily on epoch when evaluating durability progress, but it MUST preserve the stronger rule that only full-lineage-valid acks are eligible for counting; no future optimization may weaken this to epoch-only acceptance. Diagnostics should log peer ID plus the full expected/actual lineage tuple on mismatch so asymmetric authority bugs are observable without relying on cached primary-side assumptions." | **NEW-explicit** per H5 LOCK + architect round-21 sign + uniform rule | See verbatim decision text above. Rule applies uniformly across all barrier-ack-like surfaces per `INV-REPL-LINEAGE-BORNE-ON-BARRIER-ACK` uniform clause | `TestDurabilityCoordinator_BarrierAck_ShortPayload_Rejected` + `_ZeroedLineage_Rejected` + `_MismatchedLineage_NotCounted` + `_DiagnosticLogFormat` (T4b-1) | ⊙ T4b-1 | + +#### §3.2.5 DistGroupCommit (L1 §2.5) — 1:1-shift to DurabilityCoordinator + +- **V2 file**: `weed/storage/blockvol/dist_group_commit.go` (lines 15–83) +- **V2 scope**: per-write-operation ephemeral closure bound once to `vol.writeSync` at volume init +- **Bridge shape**: 1:1-shift — closure becomes entity (`DurabilityCoordinator`); math preserved verbatim; mode selectable at call time instead of bound at init +- **T4 status**: ⊙ DECIDED — T4b + +**Contracts**: + +| ID | Statement | V3 verdict | Rationale + impl | Test anchor | Status | +|---|---|---|---|---|---| +| C1-DGC-QUORUM-ARITHMETIC | `rf = group.Len() + 1`; `quorum = rf/2 + 1`; primary counts as 1 durable node | **PRESERVE** pure math | `DurabilityCoordinator.EvaluateBarrierAcks(mode, rf, acks)` | `TestDurabilityCoordinator_Quorum_RF3` + `_RF5` (T4b) | ⊙ T4b | +| C2-DGC-SYNC-ALL-FAIL-ON-ANY-BARRIER-FAILURE | Any peer barrier failure → write fails; no silent swallow | **PRESERVE** | `EvaluateBarrierAcks` returns error on first ack.Success=false under `DurabilitySyncAll` | T4b sync_all failure test | ⊙ T4b | +| C3-DGC-SYNC-QUORUM-FAIL-ON-INSUFFICIENT | `durableNodes < quorum` → `ErrDurabilityQuorumLost` | **PRESERVE** | `EvaluateBarrierAcks` under `DurabilitySyncQuorum` | T4b quorum-loss test | ⊙ T4b | +| C4-DGC-BEST-EFFORT-SILENCE-ON-FAIL | Peer barrier failures logged + `degradeReplica` called; write NOT failed | **PRESERVE** | `EvaluateBarrierAcks` under `DurabilityBestEffort`; calls `ReplicaPeer.Invalidate` | T4b best_effort test | ⊙ T4b | +| C5-DGC-EPOCH-ACK-VALIDITY | Stale-epoch ack MUST NOT count toward quorum (H5 LOCK) | **PRESERVE** + **NEW INV** per H5 LOCK | `EvaluateBarrierAcks` filters `ack.Epoch != primary.Epoch.Load()` from quorum count; frame-borne epoch is source (`INV-REPL-ACK-FRAME-IS-AUTHORITY`) | `TestDurabilityCoordinator_StaleEpochAck_NotCounted` (T4b) | ⊙ T4b | +| C6-DGC-PARALLEL-LOCAL-AND-REMOTE | Parallel local `walSync` + `group.BarrierAll` via `WaitGroup` | **PRESERVE** | `SyncLocalAndReplicas` spawns local + per-peer goroutines | T4b parallel-sync test | ⊙ T4b | + +#### §3.2.6 RebuildSession (L1 §2.6) — deferred T5 + +- **V2 file**: `weed/storage/blockvol/rebuild_session.go` (lines 41–327) +- **V2 scope**: per-replica-session; volatile (non-durable across crash) +- **Bridge shape**: merge into T5 `RebuildCoordinator` +- **T4 status**: ⏭ DEFERRED-T5 — non-durable across crash per L1 §2.6 invariant (sw round-3 verified `rebuild_bitmap.go` has zero file I/O) +- **Contracts**: all deferred to T5. Notable: phase FSM (Idle→Accepted→Running→BaseComplete→Completed|Failed); bitmap WAL-wins rule; hydration guard fails closed if local checkpoint > baseLSN; bitmap-set-precedes-ack. + +#### §3.2.7 RebuildServer (L1 §2.7) — split; listener V3-covered, policy T5 + +- **V2 file**: `weed/storage/blockvol/rebuild.go` (lines 21–307) +- **V2 scope**: per-primary singleton TCP listener +- **Bridge shape**: split — TCP listener + per-request dispatch is absorbed by `transport.BlockExecutor.StartRebuild` (V3-covered, wire-identical frame types) + `transport.ReplicaListener`; rebuild policy + 2-phase coordination + snapshot-export → T5 `RebuildCoordinator` +- **T4 status**: ~ PARTIAL — wire PORTED; semantic gap noted; policy DEFERRED-T5 + +**Semantic gap** (from audit §6.3): V3 `BlockExecutor.StartCatchUp` AND `StartRebuild` both ship `primaryStore.AllBlocks()` (every LBA), not V2's retained-WAL-delta (`StartCatchUp` per V2 `StreamEntries`) or extent-bitmap (`StartRebuild` per V2 `handleFullExtent`). File comment at `catchup_sender.go:13-18` acknowledges: *"today catch-up ships every block in the primary store, not just the WAL window. Bounded incremental WAL streaming belongs to later execution-lifecycle work (P10)."* This is sketch §6.1 Item B — **architect decides at T4 T-start three-sign**: + +- **Path A**: T4c upgrades `StartCatchUp` to retained-WAL delta + `ErrWALRecycled` escalation; preserves V2 parity; T5 rebuild boundary cleanly definable (`catch-up exhausted → rebuild`). +- **Path B**: T4c uses current full-store `StartCatchUp`; retention-window property deferred to P10; **T5 rebuild boundary also gated on P10** because "WAL exhausted" has no retention-window definition yet. + +**Contracts**: + +| ID | Statement | V3 verdict | Rationale + impl | Test anchor | Status | +|---|---|---|---|---|---| +| C1-REBUILDSVR-WIRE-FRAMES | `MsgRebuildBlock` / `MsgRebuildDone` / `MsgRebuildError` wire types | **PRESERVE** (semantic equivalent via V3 shape) | `transport.protocol.go` MsgRebuildBlock/MsgRebuildDone — Q5-envelope-compliant versions exist | `core/transport/transport_test.go` StartRebuild tests (V3-covered) | ✓ PORTED | +| C2-REBUILDSVR-EPOCH-FENCE | Request-time epoch validation; mismatch → `MsgRebuildError("EPOCH_MISMATCH")` | **PRESERVE** | `BlockExecutor.StartRebuild` carries lineage; replica-side validates | T5 rebuild-epoch test | ⏭ T5 | +| C3-REBUILDSVR-TWO-PHASE | WAL catch-up → extent → second catch-up (client-driven) | **REBUILD per Item B** — Path A preserves; Path B collapses to single-shot full-transfer | `RebuildCoordinator` (T5) decides phase sequence based on Item B outcome | T5 rebuild phase test | ⏭ T5 | +| C4-REBUILDSVR-FLUSH-BEFORE-STREAMING | ForceFlush before extent stream | **PRESERVE** | V3 `StartRebuild` calls `LogicalStorage.Sync` pre-stream (verify during T5 wire-up) | T5 rebuild-flush test | ⏭ T5 | +| C5-REBUILDSVR-EXTENT-BYPASS-DIRTY-MAP | V2 `readBlockFromExtent` bypasses dirty map (serves only flushed) | **REBUILD** — V3 `AllBlocks()` reads through dirty map per §2.3 V3 observation 14 | Semantic divergence called out in audit §3.14; decision at T5 via Path A/B outcome | T5 rebuild vs concurrent-write test | ⏭ T5 | +| C6-REBUILDSVR-SNAPSHOT-EXPORT | Snapshot manifest + image stream (CSI feature) | **DEFER** — not T4/T5; CSI snapshot is G10-era | — | — | ⏭ G10 | + +#### §3.2.8 RebuildTransportServer (L1 §2.8) — V3-covered + +- **V2 file**: `weed/storage/blockvol/rebuild_transport.go` (lines 152–246) +- **V2 scope**: per-session (one per base-lane connection) +- **Bridge shape**: already V3-covered — `transport.rebuild_sender.go` internals (private; not exposed) +- **T4 status**: ✓ PORTED (wire); policy ⏭ T5 +- **Contracts**: all wire-level PRESERVE via `RecoveryLineage` + `MsgRebuildBlock` frames (Q5-compliant). Achieved-LSN semantic preserved (replica-side tracks). Base-stream-NOT-point-in-time property + two-line bitmap convergence → T5. + +#### §3.2.9 RebuildTransportClient (L1 §2.9) — V3-covered + +- **V2 file**: `weed/storage/blockvol/rebuild_transport.go` (lines 252–321) +- **V2 scope**: per-session replica-side +- **Bridge shape**: already V3-covered — `transport.ReplicaListener` receive path (routes `MsgRebuildBlock` to `LogicalStorage.ApplyEntry`) +- **T4 status**: ✓ PORTED (wire); session/policy → T5 +- **Contracts**: frame-parsing PRESERVE; bitmap-conflict-skip → T5; graceful disconnect → T5. + +#### §3.2.10 RebuildBitmap (L1 §2.10) — deferred T5 + +- **V2 file**: `weed/storage/blockvol/rebuild_bitmap.go` (~84 LOC; purely in-memory per L1 §2.10 corrected invariant) +- **V2 scope**: per-rebuild-session; volatile +- **Bridge shape**: move into T5 `RebuildCoordinator` internal state +- **T4 status**: ⏭ DEFERRED-T5 +- **Contracts**: WAL-wins-over-base PRESERVE (T5 core invariant); hydration guard PRESERVE (T5); bit-set-precedes-ack PRESERVE (T5). + +#### §3.2.11 (new V3) ReplicationVolume — no V2 analog + +- **V3 file**: `core/replication/volume.go` (new package; T4a-4) +- **V3 scope**: per-volume coordination owner +- **Why new**: V2 `BlockVol` was monolith; V3 decomposes. `ReplicationVolume` closes the layer-E gap between layer A (authority consumption) + layer C (local engine) + layer D (per-peer transport). Nothing in V2 maps 1:1 — closest are fragments of `BlockVol.HandleAssignment` + `ShipperGroup` + `DistGroupCommit` binding. +- **T4 status**: ⊙ DECIDED — MVP lands T4a-4 (UpdateReplicaSet + OnLocalWrite best_effort); Sync/ProbeAll/EvaluatePeerProgress/ExecuteRecoveryPlan land T4b/T4c +- **Key contracts**: + - **INV-REPL-LIFECYCLE-HANDLE-BORROWED-001** — BUG-005 non-repeat: `ReplicationVolume` borrows `LogicalStorage`, NEVER calls `store.Close()`. Enforced via Gate G-2 three-line godoc contract. + - **INV-REPL-FANOUT-001** — Every acked primary write either ships to every eligible peer or reports ship failure; no silent drop. + - **INV-REPL-PEER-LIFECYCLE-BY-UPDATE** — `UpdateReplicaSet` is sole entry point for peer add/remove; tears down executor on remove (Opt-3 explicit test `TestReplicationVolume_UpdateReplicaSet_RemovePeer_ExecutorTornDown`). + +#### §3.2.12 (new V3) ReplicaPeer — no V2 analog + +- **V3 file**: `core/replication/peer.go` (T4a-3 leaf; T4b adds Fence; T4c extends state machine) +- **V3 scope**: per-remote-replica runtime +- **Why new**: V2 `WALShipper` tangled per-peer state with transport + state machine + retention-budget + reconnect; V3 decomposes — transport in layer D, retention in `ReplicationVolume.EvaluatePeerProgress`, state-machine here. V2 approximate analog is `WALShipper` shorn of its non-peer-state concerns. +- **T4 status**: ⊙ DECIDED — T4a-3 leaf (Healthy/Degraded); T4b Fence; T4c full state machine (+CatchingUp, Rebuilding, NeedsRebuild) +- **Key contracts**: + - ShipEntry carries full `RecoveryLineage` (`INV-REPL-LINEAGE-BORNE-ON-SHIPENTRY`) + - Fence carries full `{sessionID, epoch, endpointVersion}` (`INV-REPL-LINEAGE-BORNE-ON-FENCE`) + - ShipEntry error → `Invalidate` → Degraded (round-11 forward-carry CARRY-1) + - Close is idempotent; tears down `*BlockExecutor` cleanly + +#### §3.2.13 (new V3) DurabilityCoordinator — no V2 analog + +- **V3 file**: `core/replication/durability.go` (T4b) +- **V3 scope**: per-volume quorum/durability closure +- **Why new**: replaces V2's `vol.writeSync` closure-bound-at-init pattern with call-time dispatch entity; same math (C1-DGC-QUORUM-ARITHMETIC PRESERVED verbatim). +- **T4 status**: ⊙ DECIDED — T4b +- **Key contracts**: all §3.2.5 C1–C6 install into this entity. BarrierResponse wire extension (§3.3 C.2) lands alongside to enable `INV-REPL-ACK-FRAME-IS-AUTHORITY`. + +#### §3.2.14 (new V3) RebuildCoordinator — deferred T5 + +- **V3 file**: `core/replication/rebuild.go` (T5) +- **V3 scope**: per-volume rebuild-path decision owner +- **T4 status**: ⏭ DEFERRED-T5 — landing depends on Path A vs Path B resolution (sketch §6.1 Item B) +- **Key contracts**: deferred. Notable: Decide (policy); Begin/Complete (session bookkeeping); InvalidateOlderSessions (`activeRebuildSession` semantic lifted up). + +--- + +### §3.3 Cross-entity invariants (architect-LOCKED + T4-defined) + +| ID | Statement | Source | T4 status | +|---|---|---|---| +| INV-REPL-ACK-FRAME-IS-AUTHORITY | Replica ack validity is determined by fields carried in the frame, not by primary-side inferred replica state | H5 LOCK (round 4) | ⊙ enforced T4b via §3.3 C.2 BarrierResponse epoch | +| INV-REPL-CACHE-ADVISORY-ONLY | Any primary-side per-replica cache is advisory only and must never upgrade an otherwise-invalid ack into a quorum-contributing ack | H5 LOCK (round 4) | ⊙ enforced T4b | +| INV-REPL-WIRE-CLEAN-BREAK | V3 replication wire is clean-break from V2; versioned envelope `{magic, protocol_version, message_type, flags_or_reserved}` from day one; no V2 replica interop | Q5 LOCK (round 4) | ✓ PORTED T4a-1 (commit `seaweed_block@56ad349`, 12B preamble SWRP + version 1) | +| INV-REPL-LINEAGE-BORNE-ON-SHIPENTRY | ShipEntry frames carry explicit `RecoveryLineage` sourced from the peer's registered live-ship session; session is fresh per authority era (peer is torn down + recreated by `ReplicationVolume.UpdateReplicaSet` on target `{Epoch,EndpointVersion}` change, **not** in-place updated) | Sketch §8 + T4a-3 design | ✓ PORTED T4a-2 (wire) + T4a-3 (peer-session ownership); peer-rebuild-on-authority-change binding on T4a-4 | +| INV-REPL-PEER-REBUILD-ON-AUTHORITY-CHANGE | **Statement**: Any change to the authority-bearing peer target (`Epoch`, `EndpointVersion`, or any field carried on replication frames as correctness-bearing lineage) MUST tear down the existing `*ReplicaPeer` and create a new one; in-place lineage mutation is forbidden. **Lifecycle ordering**: old peer sessions/attachments MUST be invalidated (conn closed, transport session marked invalid, old peer rejects all ship/fence/etc. after the signal) BEFORE the new peer becomes active / visible to callers. Replaces V2's per-Ship `epochFn()` callback with V3's "each peer instance = one authority-snapshot-era" ownership model | T4a-3 round-13 design call + round-14 user co-sign strengthening | ✓ PORTED T4a-4 (`seaweed_block@84ec9bf`) — pinned by `TestReplicationVolume_UpdateReplicaSet_LineageBump_RecreatesPeer` | +| INV-REPL-LSN-ORDER-FANOUT-001 | **Statement**: `ReplicationVolume.OnLocalWrite` serializes fan-out in LSN order for a given volume. Caller order is not trusted as the correctness mechanism. **Lock-scope binding**: the per-volume mutex MUST guard the full per-peer Ship dispatch loop — not `LSN-read + peer-map-snapshot + unlock + ship`. Unlocking before ship loses the serialization property. **Accepted trade-off**: T4a is correctness-first, not throughput-first — `OnLocalWrite` may block behind lazy dial + 3s write deadline + slow/dead peer timeouts. Async-queue decoupling (Option Z from T4a-4 G-1) is deferred optimization, NOT forgotten design debt. **Why it matters**: V2 kept this atomic via `shipMu` spanning `LSN allocate + WAL append + ShipAll`; V3 splits these across LogicalStorage (allocate + append) and ReplicationVolume (fan-out); without this invariant, two goroutines with LSN=1 and LSN=2 can hit Ship(2)-before-Ship(1), and `ReplicaListener.acceptMutationLineage` does NOT re-order on receipt. Direct structural port of V2 safety property; skipping it = BUG-001 class | T4a-4 G-1 V2 read (round 15) + architect co-sign | ✓ PORTED T4a-4 (`seaweed_block@84ec9bf`) — pinned by `TestReplicationVolume_OnLocalWrite_ConcurrentLSNs_OrderedAtReplica` (40 concurrent goroutines, LBA/LSN deliberately interleaved so any reorder is byte-detectable at replica's `LogicalStorage.AllBlocks()`); runs deterministically (10 consecutive runs clean) | +| INV-REPL-LINEAGE-BORNE-ON-FENCE | Fence calls carry explicit `{sessionID, epoch, endpointVersion}` | Sketch §8 | ✓ layer D already has this shape at `executor.go:75` | +| INV-REPL-LOCAL-FSYNC-GATES-QUORUM | **Statement**: Local-fsync result gates mode-arithmetic evaluation. If `localSync` returns error, `DurabilityCoordinator.SyncLocalAndReplicas` returns that error immediately — mode-dependent quorum/sync_all arithmetic is NOT evaluated. Rationale: primary is the "1" in `durableNodes = 1 + successful_barriers`; if primary itself isn't durable, the arithmetic is meaningless (would count primary as durable when it isn't). V2 precedent: `dist_group_commit.go` lines 46-48 return `localErr` before running mode arithmetic. **Why it matters**: silently running quorum math over a failed local fsync would let a fake "success" leak when primary lost its own local durability | T4b-3 G-1 V2 read concern #7 | ⊙ T4b-3 test `_LocalSyncFails_AllModesError` pins | +| INV-REPL-BARRIER-FAILURE-DEGRADES-PEER | **Statement**: Every per-peer barrier failure inside `DurabilityCoordinator.SyncLocalAndReplicas` MUST trigger `peer.Invalidate(reason)` on that peer BEFORE the coordinator returns. This holds in ALL three durability modes (sync_all / sync_quorum / best_effort) — even when the mode's arithmetic itself doesn't fail the Sync call (best_effort especially). **Why it matters**: V2 implemented this inside `WALShipper.Barrier` via `failBarrier → markDegraded`; V3's `BlockExecutor.Barrier` (T4b-2) intentionally does NOT own ReplicaState, so `DurabilityCoordinator` MUST translate barrier failure → peer degrade explicitly. Silent barrier failure without peer degrade = BUG-001 class: future writes to an effectively-dead peer would continue through best_effort without surfacing the problem | T4b-3 G-1 V2 read concern #8 | ⊙ T4b-3 test `_BestEffort_Silences_Failures` asserts peer becomes Degraded | +| INV-REPL-ZERO-PEER-NO-SPAWN | **Statement**: `DurabilityCoordinator.SyncLocalAndReplicas` with zero peers (standalone / RF=1) MUST short-circuit: run local sync only, no parallel goroutine spawn. All three modes return success if local sync succeeds (primary = 1 durable ≥ 1 quorum for RF=1). V2 precedent: `dist_group_commit.go` lines 19-22. **Why it matters**: consistent with T4a-5 Q3 empty peer set semantics (standalone is a real production case); goroutine spawn for zero peers is wasteful and obscures the arithmetic | T4b-3 G-1 V2 read concern #9 | ⊙ T4b-3 test (to add: `_ZeroPeers_StandaloneFastPath`) | +| INV-REPL-NO-ZERO-LINEAGE-FIELDS | **Statement**: No RecoveryLineage construction anywhere in `core/transport/` or `core/replication/` may intentionally use zero values for `SessionID`, `Epoch`, `EndpointVersion`, or `TargetLSN` as semantic sentinels. The strict-decode rule (reject-on-zero) makes such conventions observable as bugs. Use non-zero sentinel constants (e.g. `fenceSentinelTargetLSN = 1` in `core/transport/executor.go:98`) when a field is not meaningful for a given op. **Why it matters**: landed during T4b-1 self-test (`seaweed_block@0a43eff`) when `executor.Fence`'s `TargetLSN=0` convention collided with new strict decode. Same pattern will fire at T4c catch-up-done-ack and T5 rebuild-done-ack extensions; scan every existing `RecoveryLineage{...}` construction when those tasks start | T4b-1 self-test round 23 | ✓ enforced at transport decode layer (per T4b-1 tests); sw to scan for new instances at T4c/T5 wire extensions | +| INV-REPL-LINEAGE-BORNE-ON-BARRIER-ACK | **Statement**: `BarrierResponse` carries full `RecoveryLineage` echo (SessionID + Epoch + EndpointVersion + TargetLSN), NOT epoch-only, alongside the durability result payload. **Symmetry rule**: request carries authority tuple; response proves which authority tuple it is acknowledging. **UNIFORM APPLICATION (architect round 21 meta-call)**: this rule applies uniformly across ALL barrier-ack-like surfaces in the replication system — **durability barriers** (T4b), **fence acks** (layer-D already has this shape), **catch-up done acks** (T4c), **rebuild done acks** (T5). Epoch-only echo is NOT accepted on ANY of these surfaces. **Why full echo (architect round 21)**: (1) Barrier is an authority-bearing ack surface under H5 LOCK, not just a durability bit — epoch-only recreates the asymmetric-invariant smell T4 worked to remove; (2) avoids a second wire bump in T4c when `sessionID`/`endpointVersion` turn out to be needed; (3) lets late/mismatched acks be rejected against exact lineage for better callback filtering + diagnostics, not accepted on a too-weak epoch match; (4) uniform mental model across frames — matches `ShipEntry` full-lineage shape. **T4b logic latitude**: `DurabilityCoordinator.EvaluateBarrierAcks` MAY initially key correctness primarily on `epoch` if that is the immediate need, BUT MUST preserve the stronger rule that only full-lineage-valid acks are eligible for counting. No future optimization may weaken this to epoch-only acceptance. **Eligibility rule**: a barrier ack only counts if its full lineage matches the request/session the caller is awaiting; mismatched, stale, or partially-zeroed lineage is advisory for logging only and MUST NOT be accepted | Sketch §6.1 C.2 — architect sign round 21 (full echo + uniform rule) | ✓ LOCKED (architect round 21); ⊙ T4b-1 coding; binding on T4c (catch-up done ack) + T5 (rebuild done ack) | +| INV-REPL-LINEAGE-BORNE-ON-PROBE-PAIR | **Statement**: probe is a symmetric authority-bearing pair, not a one-direction echo. **`ProbeReq` carries full `RecoveryLineage`** (32B preamble before any payload); **`ProbeResponse` carries the full echoed `RecoveryLineage` + R/S/H tuple** (32B + 24B = 56B body). Replica validates incoming `ProbeReq` lineage AND echoes it; primary validates echoed lineage before accepting R/S/H facts; mismatched/stale/partial-zeroed lineage on either direction is diagnostic-log only and MUST NOT enter recovery decisions. Strict decode inherits from `BarrierResponse` rules: <56B / zero-valued lineage / field-order malformed → fail closed. **No zero-TargetLSN exception**: probe uses the standard `RecoveryLineage` from the registered session even though probe is non-mutating; carving out a separate non-mutating-probe lineage type is explicitly rejected by architect for this batch | Sketch §6.1 C.3 — architect sign round 26 (full echo + symmetric rule) | ✓ PORTED T4c-1 (`seaweed_block@4dfe582`); pinned by `TestProbeResp_FullLineageEcho_RoundTrip` + `_ShortPayload_Rejected` + `_ZeroedLineage_Rejected` + `_MalformedFieldOrder_Rejected` + `_ByteLayoutFence` + `TestReplica_ProbeReq_AcceptedLineage_EchoesAndReturnsRSH` + `_StaleLineage_Rejected` + `_ZeroedLineage_Rejected` + `TestExecutor_Probe_LineageMismatch_Rejected` + `ErrProbeLineageMismatch` sentinel test | +| INV-REPL-PROBE-NON-MUTATING-VALIDATION | **Statement**: probe lineage validation MUST NOT advance `activeLineage` on the replica side; probe is a non-mutating observation operation. The replica uses `validateProbeLineage` (gate-without-advance) NOT `acceptMutationLineage` (gate-with-advance). **Why it matters**: T4c-1 architectural pin — using `acceptMutationLineage` for probes caused C5 calibration scenario to fail because probe's monotonic sessionID raced ahead of an in-flight rebuild's lower sessionID, then rebuild frames at the lower sessionID got rejected as stale. **Discovered at PR review time during T4c-1 implementation** (no G-1 pre-code on T4c-1 because wire+validation is V3-native design); fixed in `seaweed_block@4dfe582` with new `validateProbeLineage` function + comment block at the call site `replica.go:122`. Generalizes: any future non-mutating frame consumer (status query, health check, etc.) should use gate-without-advance discipline | T4c-1 round-37 architectural pin (sw-discovered + fixed in same commit) | ✓ PORTED T4c-1 (`seaweed_block@4dfe582`); covered by C5 calibration scenario continuing to pass + dedicated probe-non-mutation test surface | +| INV-REPL-FANOUT-001 | Every acked local primary write is either shipped to every eligible peer or reported as a ship failure; no silent drop | Sketch §8 | ✓ PORTED T4a-4 (`seaweed_block@84ec9bf`) — `ReplicationVolume.OnLocalWrite` iterates every peer; per-peer ship errors degrade the peer (best_effort) but do not silently skip | +| INV-REPL-SHIP-TRANSPORT-MUSCLE-001 | `Ship` preserves V2 composite stability (lazy dial + silent drop + 3s deadline + no-hard-stop-on-failure + degrade-handoff-to-caller) | Architect round 11 (Option B) | ⊙ T4a-2 coding | +| INV-REPL-LIFECYCLE-HANDLE-BORROWED-001 | `ReplicationVolume` borrows `LogicalStorage`; never closes it (BUG-005 non-repeat) | Gate G-2 | ✓ PORTED T4a-4 (`seaweed_block@84ec9bf`) — pinned by `TestReplicationVolume_Constructor_DoesNotOwnStore` regression fence | +| INV-REPL-DURABILITY-SYNC-ALL-001 | Under `sync_all`, any peer barrier failure fails the write | §3.2.5 C2 | ⊙ T4b | +| INV-REPL-DURABILITY-QUORUM-001 | Under `sync_quorum`, `durableNodes >= rf/2+1` required; primary = 1 durable; stale-epoch acks excluded | §3.2.5 C1+C3+C5 | ⊙ T4b | +| INV-REPL-DURABILITY-BEST-EFFORT-001 | Under `best_effort`, peer barrier failures are logged + peer degraded, do not fail the write | §3.2.5 C4 | ⊙ T4b | +| INV-REPL-FENCE-EPOCH-EQ | Replica drops WAL entries with `entry.Epoch != currentEpoch` silently | §3.2.1 C1 + §3.2.3 C3 | ⊙ coding T4a-2 + T4c | +| INV-REPL-CATCHUP-WITHIN-RETENTION-001 *(Path A only)* | Reconnect within retained WAL window catches up via delta; `ErrWALRecycled` escalates to rebuild | Sketch §6.1 B Path A | **⏭ T4d (downgraded from T4c per closure §B delta #4)** — un-pinned at T4c because catch-up sender hardcodes `ScanLBAs(1)` not R+1; spurious-rebuild risk under checkpoint-past-1 + replica-within-retention. T4d-2 (apply gate) + T4d-3 (R+1 threading) co-pin | +| INV-REPL-CATCHUP-FULL-TRANSFER-001 *(Path B only)* | Reconnect after any disconnect catches up via full-state transfer of `AllBlocks()`; retention-window deferred to P10 | Sketch §6.1 B Path B | NOT ACTIVE — Path A signed by architect | +| **INV-REPL-NO-PER-LBA-DATA-REGRESSION** *(round-43 lock, GOAL-LEVEL)* | Per-LBA data on a replica MUST NOT regress under any apply path (live, recovery, retry, replica restart mid-recovery). Frontier monotonicity (walHead/nextLSN/R/H) is necessary but not sufficient — `BlockStore.ApplyEntry` regressed walHead AND blindly overwrote LBA bytes on older-LSN apply pre-`f6084ee`. The mechanism invariants below (STALE-ENTRY-SKIP-PER-LBA + COVERAGE-ADVANCES-ON-SKIP + LIVE-LANE-STALE-FAILS-LOUD + RESTART-SAFE) collectively make this hold | Round-43 architect lock | ⏭ T4d-2 | +| **INV-REPL-RECOVERY-STALE-ENTRY-SKIP-PER-LBA** *(round-43 lock, MECHANISM)* | Replica must skip recovery-stream entry data writes when `entry.LSN <= appliedLSN[LBA]`. Apply only if strictly newer; on apply, update `appliedLSN[LBA] = entry.LSN`. Substrate fixes are defense-in-depth; the replica recovery apply gate (T4d-2) is authoritative | Round-43 architect lock | ⏭ T4d-2 | +| **INV-REPL-RECOVERY-COVERAGE-ADVANCES-ON-SKIP** *(round-44 refinement)* | Skipped recovery entries (data-skip per stale-LSN check) MUST still update `recoveryCovered[LBA]=true`. Skip data ≠ ignore frame: completion accounting depends on every LBA in the recovery window being marked processed. Architect text: "Recovery-stream stale entries are valid duplicates and must be skipped as data writes while still counted as recovery-stream coverage." | Round-44 architect refinement | ⏭ T4d-2 | +| **INV-REPL-LIVE-LANE-STALE-FAILS-LOUD** *(round-44 refinement)* | Live-lane stale entries (received `MsgShipEntry` on the live handler with `LSN <= appliedLSN[LBA]`) are abnormal — repeat packet, stale session, lineage error, or retry on wrong lane. They MUST NOT mutate data, MUST be skipped/rejected with diagnostic surfacing, and MUST NOT advance `recoveryCovered[LBA]`. Distinct from the recovery-lane stale-skip, which is normal | Round-44 architect refinement | ⏭ T4d-2 | +| **INV-REPL-RECOVERY-COVERAGE-RESTART-SAFE** *(Option C hybrid lock)* | Replica restart mid-recovery MUST NOT regress per-LBA data. Apply gate seeds `appliedLSN[LBA]` from substrate query at session start (Option C hybrid: substrate-native truth where available, in-memory session map updated thereafter). Re-shipped recovery window after restart correctly skips entries already applied pre-restart | Architect §2.5 #1 ratification (Option C) | ⏭ T4d-2 | +| **INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT** *(§9 architect Q2 lock; round-46 architect re-emphasis)* | `MsgShipEntry` lane is derived from the **accepting handler/session context** in T4d. **NOT from payload-derived signals like `lineage.TargetLSN` sentinels.** Live handlers MUST NOT execute recovery stale-skip/coverage logic; recovery handlers MUST NOT bypass it. Apply gate hook MUST expose lane-explicit methods (e.g. `ApplyLive(...)` + `ApplyRecovery(...)`, or `Apply(..., lane ApplyLane)`) — caller supplies lane from connection/session context. A test must fail if a recovery entry reaches the live path or a live entry reaches the recovery accounting path. Regression `TestApplyGate_RecoveryWithTargetLSN1_RoutesToRecoveryLane` pins the edge case where TargetLSN=1 (sentinel-collision-with-live) MUST still route via handler-context to recovery. Explicit lane wire-tag deferred to post-G5 protocol-hardening if handler-context discipline becomes fragile. **Round-46 history**: T4d-2 `bd2de99` initial implementation drifted to payload-derived discrimination via `lineage.TargetLSN > 1` sentinel; QA T4d-2 review surfaced the edge case (recovery with H=1 misroutes); architect reaffirmed handler-context as the correct rule + bound T4d-2 follow-up rework as HARD GATE before T4d-3 | §9 architect Q2 + round-46 architect re-emphasis | ⏭ T4d-2 follow-up (REWORK GATE) → T4d-3 | +| **INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY** *(§9 architect Q1+Q3 lock; T4d-1 strengthened)* | `core/transport` may depend on `core/storage` recovery-contract symbols only (`LogicalStorage`, `RecoveryEntry`, `ErrWALRecycled` / structured replacement, `RecoveryMode`), never substrate internals (`core/storage/walstore`, `core/storage/smartwal`). Engine retry loop lives in `core/engine` (Q1 Option α); package move to `core/recovery` deferred per Q3. Lane-aware apply decisions go through the T4d-2 apply gate interface, not via direct substrate queries from transport. **T4d-1 strengthening (architect-endorsed 2026-04-25)**: 2-enum split inscribed — `storage.StorageRecoveryFailureKind` is substrate classification; `engine.RecoveryFailureKind` is engine decision vocabulary; `transport.classifyRecoveryFailure` is the explicit boundary mapper. `core/engine` MUST NOT import `core/storage` at all (stricter than Q1+Q3 baseline; verified at `1edeb36` build). | §9 architect Q1+Q3 + T4d-1 architect endorsement | ⏭ T4d-3 (transport fence); ⊙ T4d-1 (engine-purity verified) | +| **INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1** *(T4d-3 G-1 §5 hidden invariant)* | Engine emits `StartCatchUp.FromLSN = Recovery.R + 1`. Sender does NOT add `+1` (avoids double-add). The "+1 to skip already-applied LSN" semantic lives at the engine command-emit site, not transport. Surfaced at G-1 §6.1 placement decision; QA-ratified Option A (engine adds +1) | T4d-3 G-1 §5 (architect Q1 derivation) | ⏭ T4d-3 | +| **INV-REPL-CATCHUP-FROMLSN-FROM-ENGINE-STATE-NOT-PROBE** *(T4d-3 G-1 §5 hidden invariant)* | Engine populates `StartCatchUp.FromLSN` from its **own** `Recovery.R` state (the single source of truth). Probe results are ingested as facts that update engine state; the command emit path reads the updated state, never the raw probe payload directly. Pins the race window: probe → recovery decision → command emit always goes through engine state, no shortcut. Positive-form per QA nit; predecessor name `INV-REPL-CATCHUP-FROMLSN-NOT-FROM-PROBE-DIRECTLY` retained in G-1 history only | T4d-3 G-1 §5 (architect Q1 derivation) | ⏭ T4d-3 | +| **CARRY-T4D-LANE-CONTEXT-001** *(named carry — round-46+ architect)* | **NOT AN INVARIANT — A NAMED CARRY.** T4d-2 follow-up `01f4ab9` made the apply gate lane-pure (correct architectural fence: gate ≠ payload-sniffer) but moved the lane-discrimination payload sniffing one layer up to `core/transport/replica.go:13-18,156-165` as a TRANSITIONAL CALLER-SIDE SHIM — replica handler reads `lineage.TargetLSN==liveShipTargetLSN(=1)` to dispatch to gate's `ApplyLive` vs `ApplyRecovery`. Edge case: recovery session with H=1 (primary has exactly 1 entry, replica empty) ships with `TargetLSN=1` and misroutes to live-lane via the shim → fail-loud fires incorrectly. **Bound resolution**: replace the TargetLSN==1 caller shim with true handler/session context lane signal (per-conn lane tag at handshake / separate handlers / distinct ports — implementation choice). **Owner**: sw. **Bind point**: T4e (preferred) or post-G5 protocol-hardening (latest). **T4d-3 close gate**: T4d-3 MUST do exactly one of (A) land the fix and remove the shim, or (B) add explicit failing/skip-marked L2 test `TestT4d3_RecoveryTargetLSN1_KnownGap` documenting H=1 as known gap with godoc citing this carry id. Architect explicit prohibition: NO engine precondition `TargetLSN >= 2`. | T4d-2 follow-up review round-46+ architect direction | ⏳ CARRY (active) — T4d-3 close evidence cites; resolution at T4e or post-G5 | + +Exactly one of `INV-REPL-CATCHUP-{WITHIN-RETENTION,FULL-TRANSFER}` queues ACTIVE per architect Item B choice (Path A signed; WITHIN-RETENTION active, FULL-TRANSFER not). + +--- + +### §3.4 Drift events (T4 port judgment trail) + +| Round | Event | Entity | Root cause | Corrected to | +|---|---|---|---|---| +| Round 6 (sw) | `core/transport/` package entirely missed in initial V3 pre-scan; §3.11 H6 + §3.12 H7 + §3.13 hazards reasoned against incomplete V3 picture | Layer D reality | QA V3 pre-scan limited to `core/frontend/durable/` + `core/storage/` | Entity skeleton §2.D added; §3.11–§3.14 retired/reframed | +| Round 7 (sw) | `BlockExecutor.Ship` + `.Barrier` don't exist as public API (claim in audit §6 was wrong); `BlockExecutor.StartCatchUp` ships AllBlocks, not WAL-window delta | Layer D reality | QA conflated test-call helpers (`exec.StartCatchUp` in test files) with real public API + assumed semantic | Audit §7 counts recalibrated; sketch §5 layer-D addendum gains wire-shape work; Item B elevated to architect-line | +| Round 8 (architect) | walstore substrate scope ambiguity; lineage row over-broad; Path B weakening hidden in §6.1 | T4 scope discipline | QA missing T3 walstore inheritance; failed to split lineage per frame family | Sketch §1/§2/§7 walstore discipline; §8 lineage rows split by frame family; §3.2 Path B weakening surfaced in main gate text | +| Round 11 (architect) | V2 Ship characterized as "transparent in-call retry"; V3 Option C (retry in `ReplicaPeer`) accepted prematurely | §3.2.1 WALShipper | QA mis-read V2 composite stability as single-call behavior; would have additionally forced exposing `registerSession`/`attachConn` public seams violating transport-ownership principle | §3.2.1 C3-C4 Option B adopted; INV-REPL-SHIP-TRANSPORT-MUSCLE-001 inscribed; partial-port discipline surfaced (QA is 2-for-2 on V3-existing-shape under-scanning within T4) | + +**Pattern**: QA-solo sign has recurring failure mode on partial-port + V3-existing-shape. sw and architect caught both mis-signs before code landed. Round-11 proposal (sketch of `§8C.2.a partial-port architect co-sign rule`) deferred — architect to decide discipline shape after full re-read of this §3 + backing T4 docs. + +--- + +### §3.5 Change log + +| Date | Change | Author | +|---|---|---| +| 2026-04-23 | §3 filled from T4 rounds 1–11 consolidated. 10 V2 entities + 4 new V3 entities; 14 per-entity sections in §3.2; 16 cross-entity invariants in §3.3 (3 architect-LOCKed, 4 sketch-§6.1-conditional); 4-round drift trail in §3.4. T4a-2 rows reflect Option B adoption (architect round 11); sw coding in progress. Item B catch-up semantics still pending architect sign at T4 T-start | QA Owner | +| 2026-04-23 | **T4b BATCH CLOSE — QA single-sign per §8C.2.** `seaweed_block@21e9e83` landed T4b-6 L2 integration matrix: 8 scenario subtests (`{SyncAll_TwoPeers_AllHealthy, SyncAll_OnePeer_FailsOnBarrierError, SyncQuorum_RF3_TolerantOfOneFailure, BestEffort_AllPeersFail_StillSucceeds} × {smartwal, walstore}`) + 1 wire-shape fence (`BarrierWire_FullLineageEcho_StillEnforced`) all green. Both impls passed at L2 per round-22 honesty correction (smartwal sign-bearing; walstore non-gating only for BUG-007-class which L2 does not exercise). T4b cumulative: 6 commits, 34+3 sanity pins, ~590 LOC production + ~1200 LOC tests landed against `phase-15` branch. **8 invariants pinned** at T4b close: INV-REPL-LINEAGE-BORNE-ON-BARRIER-ACK (wire+executor layers), INV-REPL-BARRIER-FAILURE-DEGRADES-PEER (peer+coordinator layers — round-24 §0-B two-layer coverage), INV-REPL-DURABILITY-{SYNC-ALL,QUORUM,BEST-EFFORT}-001 (coord+L2 matrix), INV-REPL-LOCAL-FSYNC-GATES-QUORUM (G-1 concern #7), INV-REPL-ZERO-PEER-NO-SPAWN (G-1 concern #9), INV-REPL-NO-ZERO-LINEAGE-FIELDS (T4b-1 self-test surfacing). **Forward-carry from T4a all verified green**: T4a-2 Ship invariants + T4a-3 peer state + T4a-4 INV-REPL-LSN-ORDER-FANOUT-001 (under T4b-5 concurrent Sync+Write) + T4a-6 BasicEndToEnd + DisconnectThenReassign. 21-package repo suite green at every commit. QA ✓ for T4b batch close. **Architect + PM sign at T4 T-end** per §8C.1 (after T4c + T4d optional) | QA Owner | +| 2026-04-23 | T4b-5 closure: `seaweed_block@907e368` landed `ReplicationVolume.Sync` + `WriteObserver.Sync` seam (~30 LOC core + interface extension). Lock-scope binding honored: v.mu held across full Sync call (architect round-15 Condition A discipline forward-carried). Per-volume durability mode (set via `SetDurabilityMode`); per-Sync override deferred per mini-plan §5. `StorageBackend.Sync` branches: observer==nil → pre-T4b local-only behavior preserved; observer!=nil → delegates full sync (local + barriers) to `ReplicationVolume.Sync` → `DurabilityCoordinator`. 2 pins green: `_Sync_BestEffort_E2E` (full wire write→ship→sync→barrier→primary R reaches H) + `_Sync_PreservesLSNOrderUnderConcurrency` (40 goroutines, caller-serialized LSN, Sync interleaved every 4th iteration, inverse-LBA-order plan; all markers land correctly). **Forward-carry fence verified**: T4a-4 adversarial pin `TestReplicationVolume_OnLocalWrite_ConcurrentLSNs_OrderedAtReplica` continues to pass — INV-REPL-LSN-ORDER-FANOUT-001 not regressed by Sync path. 21-package suite green | QA Owner | +| 2026-04-23 | T4b-4 closure: `seaweed_block@921521d` landed `DurabilityCoordinator` (~260 LOC prod + ~580 LOC tests). V2 `MakeDistributedSync` ported to V3 `core/replication/durability.go` per G-1 signed outline. §0-B discipline verified — zero `peer.executor.Barrier` refs; all 7 call sites use `peer.Barrier` wrapper. 9 pins green including `_ZeroPeers_StandaloneFastPath` (3 mode subtests; goroutine-ID check for INV-REPL-ZERO-PEER-NO-SPAWN), `_LocalSyncFails_AllModesError` (3 mode subtests for INV-REPL-LOCAL-FSYNC-GATES-QUORUM), `_BestEffort_Silences_Failures` (INV-REPL-BARRIER-FAILURE-DEGRADES-PEER coordinator-layer pin via peer.Barrier), `_ParallelExecution` (0.10s actual vs 150ms budget), `_StaleLineageAck_NotCounted` (H5 LOCK). T4b-3 pin + T4b-4 pin together cover INV-REPL-BARRIER-FAILURE-DEGRADES-PEER at both peer and coordinator layers (architect round-24 two-layer coverage). T4b-3 `seaweed_block@90e6c64` also landed in sequence (peer wrappers — leaf shipped first per Opt-4 pattern). T4b catalogue §3 entity rows can upgrade ⊙ → ✓ at T4b batch close after T4b-5 + T4b-6 | QA Owner | +| 2026-04-23 | T4b round-24 §0-B-correct reorder absorbed. QA realized prior-turn Decision #1 sign conflated two questions (peer.Invalidate locality vs peer.Barrier-vs-peer.executor.Barrier routing); sw had correctly flagged the §0-B question separately. Signed §0-B-correct reading: T4b-3 scope expanded from "Fence-only thin wrap" to "ReplicaPeer.Barrier + ReplicaPeer.Fence wraps" leaf (mirrors T4a-3 leaf-first pattern). T4b-4 becomes DurabilityCoordinator (was T4b-3); T4b-5 becomes ReplicationVolume.Sync (was T4b-4); test count unchanged at 9 for DurabilityCoordinator. G-1 artifact renamed `v3-phase-15-t4b-3-g1-v2-read.md` → `v3-phase-15-t4b-4-g1-v2-read.md` with 7 revision points absorbed (call-site `peer.Barrier` not `peer.executor.Barrier`; INV-REPL-BARRIER-FAILURE-DEGRADES-PEER pinned at TWO layers — peer.Barrier wrapper + DurabilityCoordinator best-effort test; "always attempt BarrierAll MUST NOT gate on Degraded" binding relocated to T4b-3 wrapper). Short-cut alternative explicitly rejected per §0-B discipline. G-1 re-signed after revision. sw codes T4b-3 first, then T4b-4 against peer.Barrier surface | QA Owner | +| 2026-04-23 | T4b-3 Gate G-1 SIGNED. sw posted G-1 V2 read at `v3-phase-15-t4b-3-g1-v2-read.md`; LOC 130-160 V3 vs 45 V2 muscle (3×, not shrinkage — justified by `EvaluateBarrierAcks` split + explicit per-peer lineage construction). **5 hidden invariants surfaced beyond mini-plan's enumerated 6**: #7 local-fsync-short-circuits-quorum-arithmetic; #8 barrier-failure-degrades-peer (the big one — V2's peer-state mutation lives inside `WALShipper.Barrier` not in `dist_group_commit`; V3 `BlockExecutor.Barrier` intentionally doesn't own ReplicaState per T4b-2 design, so `DurabilityCoordinator` MUST explicitly call `peer.Invalidate` to preserve V2 semantic); #9 zero-peer-fast-path; #10 targetLSN derivation responsibility shift (doc-level); #11 WaitGroup flat vs nested layout (pure refactor). Three QA decisions signed: (1) peer-state mutation lives in `DurabilityCoordinator` directly, not T4b-5 wrapper (same-commit validation); (2) metrics DEFER to T4-end pass (V2 was log-only anyway); (3) flat `wg.Add(1 + len(peers))`. Three new invariants inscribed: `INV-REPL-LOCAL-FSYNC-GATES-QUORUM` (#7), `INV-REPL-BARRIER-FAILURE-DEGRADES-PEER` (#8), `INV-REPL-ZERO-PEER-NO-SPAWN` (#9). Test count increased from 8 → 9 (new `_ZeroPeers_StandaloneFastPath` for #9). Clarified meta-question for user: T4b-3 is NOT "adding V2-absent logic" — it's relocating V2's degrade-on-barrier-fail mechanism from `WALShipper` layer (where V2 had it) to V3 `DurabilityCoordinator` layer (the equivalent same-level owner), consistent with T4b-2's explicit decision that `BlockExecutor` does not own ReplicaState. §0 PRESERVE baseline intact: mechanism ported, container relocated | QA Owner | +| 2026-04-25 | T4c-2 closure: `seaweed_block@ff731dd` landed catch-up muscle port + state machine + ScanLBAs interface seam per G-1 V2 read signed at round 36-37. **Storage interface seam** (G-1 §3 row #4): unified `RecoveryEntry` + `ErrWALRecycled` + `RecoveryMode` types in new `core/storage/recovery_contract.go`; `LogicalStorage.ScanLBAs` interface method; walstore + smartwal POC code promoted to production; `BlockStore` adapter implements ScanLBAs. **Catch-up sender muscle port** (G-1 §3 narrow scope): `catchup_sender.go` rewritten — AllBlocks walk → ScanLBAs callback. **All 6 invariants honored** (5 from G-1 §5 + 1 from §4.2 binding): callback-return-nil-continues, lastSent-monotonic, deadline-per-call-scope, target-not-reached-vs-recycled-distinguished, completion-from-barrier-achieved-lsn, done-marker-always-emitted. `recovery_mode` label emitted in success log per memo §5.1 / §13.0a observability requirement. **Engine wiring** (G-1 §4.1 + §4.3): `RecoveryRuntimePolicy.MaxRetries` per-kind defaults (3 / 0 / 1); `applySessionFailed` detects `ErrWALRecycled` sentinel and escalates `recovery.Decision = Rebuild` (engine→storage decoupling preserved via string match, not import). **Peer state machine** (memo §2.2): `ReplicaCatchingUp` + `ReplicaNeedsRebuild` states + `replicaStateTransitionAllowed` table-driven gate; NeedsRebuild terminal in T4c. **15 new tests** (8 catch-up sender + 4 peer state machine + 3 engine pins). 21-package suite green. **LOC actual ~120 vs G-1 estimated 60-70**: structure-driven (ScanLBAs callback adds boilerplate vs V2 inline `StreamEntries(fn)+WriteFrame(MsgCatchupDone)` pattern + mode-label probe + per-error-class wrapping); LOC heuristic check is V3<40% V2 = red flag for SIMPLIFICATION drift; 120>67 is the opposite direction so PRESERVE concern N/A. **§4.1 retry-loop wiring deferred to T4c-3**: MaxRetries field present + per-kind defaults pinned at T4c-2 (3 engine tests pin); engine SessionFailed→re-emit-with-counter loop is observable behavior testable at L2/L3 in T4c-3 integration matrix, NOT at unit scope. **Forward-carry binding to T4c-3**: integration matrix MUST exercise retry path end-to-end with budget exhaustion + escalation; without this, T4c retry mechanism has no live exercise. INV-REPL-CATCHUP-* invariants queue ACTIVE at T4c batch close after T4c-3 | QA Owner | +| 2026-04-25 | T4c-1 closure: `seaweed_block@4dfe582` landed ProbeReq + ProbeResponse symmetric full-lineage wire + validation per architect round-26 Item C.3. **Wire**: ProbeReq 32B body (full RecoveryLineage); ProbeResponse 56B body (32B echoed lineage + 24B R/S/H tuple); strict decode mirrored from BarrierResp (short / zeroed / field-order rejected). **Validation**: replica `validateProbeLineage` gates incoming ProbeReq; primary validates echoed lineage in response via new `ErrProbeLineageMismatch` sentinel. **Adapter**: `prepareQueuedCommands` mints transient probe sessionID for ProbeReplica (parallel to FenceAtEpoch); 7 test executor stubs updated with new sessionID param. **Architectural pin discovered at PR time + fixed in same commit**: `acceptMutationLineage` advances `activeLineage`, which broke C5 calibration when probe's monotonic sessionID raced ahead of in-flight rebuild's lower sessionID. Fix: new `validateProbeLineage` gates without advancing (probe is non-mutating). Inscribed as new invariant `INV-REPL-PROBE-NON-MUTATING-VALIDATION` in §3.3 — generalizes to any future non-mutating frame consumer. **21 new tests + 1 rewritten** (12 wire + 7 validation + 2 transient sessionID); 21-package suite green. `INV-REPL-LINEAGE-BORNE-ON-PROBE-PAIR` upgraded ⊙ → ✓ PORTED with full pin-test list | QA Owner | +| 2026-04-23 | T4b-2 closure: `seaweed_block@a55eacd` landed `BlockExecutor.Barrier` public method (83 LOC) + `executor.doFence` round-22 validation upgrade. 5 pins green: `Barrier_Happy` / `_TimeoutFires` (5.00s exact — deadline IS the mechanism, mirrors T4a-2 pattern) / `_LineageMismatch_Rejected` (new `ErrBarrierLineageMismatch` sentinel) / `Fence_LineageMismatch_MarksFailure` (round-22 pin — no more silent-accept at first existing consumer) / `Fence_ShortOrZeroedLineage_MarksFailure`. `_ = resp` silent-accept at `executor.go:111-121` replaced with `resp.Lineage != lineage` validation + log with expected+actual tuple per round-21 diagnostic shape. Full-lineage uniform rule now enforced at two ack-consuming surfaces: new `Barrier` (T4b-2) + upgraded `doFence` (T4b-2 round-22). Remaining surfaces covered at T4c catch-up-done + T5 rebuild-done per mini-plan preserve clause. 21-package suite green | QA Owner | +| 2026-04-23 | T4b-1 closure: `seaweed_block@0a43eff` landed BarrierResponse full-lineage wire extension. Four pins green: `TestBarrierResp_FullLineageEcho_RoundTrip` (3 sub-cases) / `_ShortPayload_Rejected` (exhaustive 0..39-byte loop) / `_ZeroedLineage_Rejected` (5 sub-cases: all-zero + 4 per-field-zero) / `_MalformedFieldOrder_Rejected`. §3.2.4 C5-BARRIER-RESP-FULL-LINEAGE-ECHO row can upgrade ⊙ → ✓ at T4b batch close (holding until then per ledger discipline). **Non-trivial regression caught in self-test**: `executor.Fence` previously used `TargetLSN=0` as a local "fence has no recovery target" convention; new strict decode (rule: zero lineage field → reject) rejected fence acks. Fix landed same commit: `fenceSentinelTargetLSN = 1` constant + comment block explaining why. Calibration + sparrow suites flagged it immediately. This is exactly the class of latent drift the strict-decode fail-closed rule was designed to surface — validates the architect round-21 decision text's "zero-valued fields MUST be rejected" clause. Replica echo sites (`replica.go` MsgBarrierReq handler + MsgRebuildDone handler) now echo request's full lineage via `EncodeBarrierResp`; catch-up + rebuild decode paths still work, their lineage validation deferred to T4c/T5 per T4b-2 Preserve clause. Full 21-package suite green | QA Owner | +| 2026-04-23 | **Round-21 architect sign: Item C.2 LOCKED on full `RecoveryLineage` echo** (not epoch-only) in `BarrierResponse`. Four rationale points: (1) Barrier is authority-bearing ack under H5 LOCK, not just durability bit — epoch-only recreates asymmetric-invariant smell; (2) avoids second wire bump at T4c; (3) late/mismatched acks rejected against exact lineage (diagnostics + callback filtering); (4) uniform mental model — matches `ShipEntry` full-lineage shape. T4b logic latitude: may initially key correctness primarily on `epoch`, but wire contract is full-lineage from day one. Catalogue §3.3 `INV-REPL-LINEAGE-BORNE-ON-BARRIER-ACK` updated with signed shape + rationale; new contract row §3.2.4 `C5-BARRIER-RESP-FULL-LINEAGE-ECHO` added. Sketch §6.1 Item C.2 row marked ✓ LOCKED; remaining architect-line items narrowed to Item A (Ship approve — Ship already landed), Item B (Path A/B catch-up — T4c), Item C.3 (ProbeResponse echo shape — T4c). **T4b is now unblocked on architect side**; QA to draft T4b mini-plan (architect offered to draft exact T4b-1 wire-shape decision text — QA accepting) | QA Owner | +| 2026-04-23 | **T4a BATCH CLOSE** (QA single-sign per §8C.2): `seaweed_block@e373753` landed L2 subprocess integration test sealing T4a. 4 matrix rows green: `BasicEndToEnd × {smartwal_gating, walstore_nongating}` + `BestEffort_DisconnectThenReassign × {smartwal_gating, walstore_nongating}`. Scope-leak fence inscribed: disconnect-test asserts second-batch LBAs do NOT appear on fresh replica — catches any future refactor that silently pulls catch-up (T4c) into T4a. In-scope addition: `StorageBackend.Write` → `ReplicationVolume.OnLocalWrite` hook via narrow `WriteObserver` interface (prevents `core/frontend/durable` importing `core/replication`; optional-pointer preserves backward compat with all T3 callers). Cumulative T4a: 6 commits across `core/transport/` + `core/replication/` + host wire + integration test. All 5 invariants pinned with adversarial tests: INV-REPL-LSN-ORDER-FANOUT-001, INV-REPL-SHIP-TRANSPORT-MUSCLE-001, INV-REPL-PEER-REBUILD-ON-AUTHORITY-CHANGE, INV-REPL-LIFECYCLE-HANDLE-BORROWED-001, Opt-3 executor-teardown. QA ✓ for T4a batch close; architect + PM sign at T4 T-end per §8C.1 (after T4b + T4c + T4d optional) | QA Owner | +| 2026-04-23 | T4a-5 closure: `seaweed_block@5711b19` landed Host authority-callback → `ReplicationVolume.UpdateReplicaSet` wire per P-refined design. AssignmentFact wire extended with master-minted peer set + monotonic generation (Epoch<<32 \| EndpointVersion). Separate `decodeReplicaTargets` path preserves AST fence `TestNoOtherAssignmentInfoConstruction`. Four P-guardrails verified: AST fence green; adapter + engine zero changes; peer set master-minted only (`collectPeers` reads `Publisher.LastPublished`); generation monotonic. Three Q-bindings honored: Q1 nil-return + `replayedGens atomic.Uint64` counter (test #9); Q2 peer-ID-set delta log format; Q3 empty peer set flows through shared teardown (test #11 three assertions). 380 LOC net delta (ex-protobuf). Closes §8C.3 Discovery Bridge opened at T4a-5.0. Spot-checked: `TestReplicationVolume_UpdateReplicaSet_Generation_MonotonicGuard` + `_GenerationZero_DoesNotAdvanceGuard` + `_EmptyPeerSet_AppliesWithTeardown` + `TestNoOtherAssignmentInfoConstruction` all pass | QA Owner | +| 2026-04-23 | T4a-3 closure: `seaweed_block@99c4e1d` landed `ReplicaPeer` leaf in new `core/replication/` package. CARRY-1 closed via `TestReplicaPeer_ShipEntry_ConnFailure_MarksDegraded`. §3.2.1 C4 upgraded ~ PARTIAL → ✓ PORTED. §3.3 INV-REPL-LINEAGE-BORNE-ON-SHIPENTRY expanded to document peer-owned-session mechanism; new invariant `INV-REPL-PEER-REBUILD-ON-AUTHORITY-CHANGE` added as T4a-4 binding — peer tear-down+recreate on target epoch/endpointVersion change (NOT in-place mutation) is the V3 mechanism preserving the V2 "lineage reflects current authority" property. One sw-flagged design call (ShipEntry accepts lineage param but silently ignores) deferred to round-13 follow-up commit: QA recommends drop the param for signature honesty + BUG-001-class anti-silent-contract discipline. Sw decides final shape; either is a small diff | QA Owner | +| 2026-04-23 | T4a-2 closure: `seaweed_block@8fed2a8` landed `TestExecutor_Ship_WriteDeadline_Fires` pin test (net.Pipe-based; asserts elapsed ≥ 2.5s AND ≤ 10s + Timeout() chain check). Passes 3.00s exactly. §3.2.1 C2 row updated from `~ PARTIAL` to `✓ PORTED T4a-2`. All four `INV-REPL-SHIP-TRANSPORT-MUSCLE-001` composite components (lazy dial / silent drop / 3s deadline / no-hard-stop) now have explicit pin tests | QA Owner | +| 2026-04-23 | Round-12 architect review absorbed: three truthfulness fixes. (H) Q5 envelope state alignment — §3.0.1 Q5 LOCK row expanded with concrete landed shape (12B `SWRP` + v1 + type + flags + reserved + length); explicit note that C.1 is no longer an architect-line decision surface (any future rev = §8C.3). Sketch §6.1 Item C narrowed to C.2+C.3; §5 layer-D addendum C.1 row marked LANDED; §9 sign-table hint updated. (M) C1-REPLICARECV row rewritten — renamed to `C1-REPLICARECV-STATE-ON-ENGINE-NOT-SINGLETON`; acknowledges V3 has no per-volume receiver singleton; test anchor split between PORTED engine-side surface (T3 `TestT3_Durable_RecoverThenServe`) and PENDING demote/promote lifetime proof (T4c `TestT4c_ReplicaLSN_SurvivesDemotePromote`); status downgraded to `⏭ T4c (proof); ✓ engine-side surface`. (L) C3-SHIP-LAZY-DIAL renamed to `C3-SHIP-LAZY-DIAL-ON-REGISTERED-SESSION`; scope boundary "Ship does NOT bootstrap session creation" made explicit; cites actual `ship_sender.go:87-90` unknown-session-returns-error code path to prevent Option-C-shape re-reading. Also updated T4a-2 Ship C1/C2/C4 rows from `⊙ coding` to PORTED / ~ PARTIAL reflecting `seaweed_block@043b9f7` landed state + QA round-12 validation noting `TestExecutor_Ship_WriteDeadline_Fires` pin test pending | QA Owner | --- diff --git a/sw-block/design/v3-phase-15-t4d-3-g1-v2-read.md b/sw-block/design/v3-phase-15-t4d-3-g1-v2-read.md new file mode 100644 index 000000000..508c212fe --- /dev/null +++ b/sw-block/design/v3-phase-15-t4d-3-g1-v2-read.md @@ -0,0 +1,429 @@ +# T4d-3 — G-1 V2 Read (pre-code diff-footprint review with explicit V2/V3 split) + +**Date**: 2026-04-25 (v0.2 — addresses QA v0.1 ADDITIONS 1+2 + nit) +**Owner**: sw (porter) +**Reviewer**: QA (Gate G-1 sign) +**Mini-plan ref**: `v3-phase-15-t4d-mini-plan.md` v0.3 §2.3 T4d-3 / §7.1 Gate G-1 +**Discipline**: pre-code (§0-B Stability-Locality rule); same as T4a-2 / T4a-4 / T4b-4 / T4c-2 G-1 +**Scope binding** (kickoff §9.5): explicit PORT vs V3-native split required. T4d-3 ports the bounded catch-up muscle, NOT the V2 `WALShipper` object model +**Predecessors**: T4c batch closed (`c910464a9`); T4d kickoff §1–9 RATIFIED; T4d mini-plan v0.3 (architect-revised); BlockStore walHead pre-T4d hotfix (`f6084ee`); T4d-1 + T4d-2 may run in parallel — this G-1 gates T4d-3 code only + +--- + +## §1 Scope + +T4d-3 has TWO production deliverables that share one G-1: + +1. **R+1 threading** — `StartCatchUp` signature carries `fromLSN`; executor scans `R+1..target`, not `1..target` +2. **Engine-driven recovery wiring** — `ReplicationVolume↔adapter` plumbed end-to-end; engine retry loop runs through real production code path; `WithEngineDrivenRecovery` framework primitive becomes real + +The G-1 is required because (1) touches the V2 catch-up muscle boundary (`runCatchUpTo(replicaFlushedLSN, targetLSN)`). (2) is V3-native end-to-end (no V2 source). + +| Code | In G-1? | Reason | +|---|---|---| +| `core/transport/catchup_sender.go` `doCatchUp` body — replace `ScanLBAs(1, ...)` with `ScanLBAs(fromLSN, ...)` | **YES** | direct V2-faithful start-boundary port; §0-B Stability-Locality binds | +| `core/transport/executor.go` `BlockExecutor.StartCatchUp` signature gains `fromLSN uint64` | **YES** (signature change boundary) | propagates V2's two-arg shape into V3 | +| `core/adapter/executor.go` `CommandExecutor.StartCatchUp` interface signature update | no | mechanical signature ripple | +| `core/engine/commands.go` `engine.StartCatchUp` command struct gains `FromLSN uint64` | no | mechanical signature ripple | +| `core/engine/apply.go` populates `FromLSN` from `Recovery.R` at emit + retry-loop re-emit | no | V3-native engine-emit; spec is "thread R from probe state" | +| `core/replication/volume.go` `ReplicationVolume↔adapter` wiring | no | V3-native; engine retry loop already exists at unit scope | +| `core/replication/component/cluster.go` `WithEngineDrivenRecovery()` becomes real | no | framework primitive; V3-native | + +**Anchor**: T4c-2 already ported the inner-loop muscle (callback-return-nil-continues, lastSent-monotonic, deadline-per-call-scope, target-not-reached-vs-recycled-distinguished, completion-from-barrier-achievedLSN). T4d-3 only changes the **start LSN** of that muscle from `1` to `R+1`. The 6 T4c-2 invariants must forward-carry green. + +--- + +## §2 V2 reference excerpts — verbatim + +### §2.1 `runCatchUpTo` body — `wal_shipper.go:845-911` + +```go +func (s *WALShipper) runCatchUpTo(fromLSN uint64, targetLSN uint64) (uint64, error) { + s.state.Store(uint32(ReplicaCatchingUp)) + + // Set a deadline for the entire catch-up operation. + s.mu.Lock() + if s.dataConn != nil { + s.dataConn.SetDeadline(time.Now().Add(catchupTimeout)) + } + conn := s.dataConn + s.mu.Unlock() + + if conn == nil { + return 0, fmt.Errorf("catch-up: no data connection") + } + + // Stream entries from WAL. + var lastSent uint64 + err := s.wal.StreamEntries(fromLSN+1, func(entry *WALEntry) error { + if targetLSN > 0 && entry.LSN > targetLSN { + return nil + } + encoded, encErr := entry.Encode() + if encErr != nil { + return encErr + } + if wErr := WriteFrame(conn, MsgWALEntry, encoded); wErr != nil { + return wErr + } + lastSent = entry.LSN + return nil + }) + + if err != nil { + if errors.Is(err, ErrWALRecycled) { + s.state.Store(uint32(ReplicaNeedsRebuild)) + return lastSent, fmt.Errorf("catch-up: WAL recycled: %w", err) + } + return lastSent, fmt.Errorf("catch-up: stream error: %w", err) + } + + // Send CatchupDone marker. + doneLSN := lastSent + if doneLSN == 0 { + doneLSN = fromLSN + } + if err := WriteFrame(conn, MsgCatchupDone, EncodeCatchupDone(doneLSN)); err != nil { + return lastSent, fmt.Errorf("catch-up: send done: %w", err) + } + + // Clear deadline. + s.mu.Lock() + if s.dataConn != nil { + s.dataConn.SetDeadline(time.Time{}) + } + s.mu.Unlock() + + effectiveLast := lastSent + if effectiveLast == 0 { + effectiveLast = fromLSN + } + if targetLSN > 0 && effectiveLast < targetLSN { + return lastSent, fmt.Errorf("catch-up: target %d not reached (last=%d)", targetLSN, lastSent) + } + log.Printf("wal_shipper: catch-up complete %s: from=%d target=%d last=%d", + s.dataAddr, fromLSN+1, targetLSN, effectiveLast) + return lastSent, nil +} +``` + +### §2.2 Calling frame — `wal_shipper.go:258-298` `CatchUpTo` + +```go +func (s *WALShipper) CatchUpTo(targetLSN uint64) (uint64, error) { + // ... handshake to obtain replicaFlushedLSN ... + case ReplicaCatchingUp: + achievedLSN, catchErr := s.runCatchUpTo(replicaFlushedLSN, targetLSN) + if catchErr != nil { + s.catchupFailures++ + if s.catchupFailures >= maxCatchupRetries { // line 152: const = 3 + s.state.Store(uint32(ReplicaNeedsRebuild)) + return achievedLSN, fmt.Errorf("catch-up failed %d times: %w", s.catchupFailures, catchErr) + } + s.markDegraded() + return achievedLSN, ErrReplicaDegraded + } + // ... +``` + +### §2.3 Pre-bounded-catch-up call site — `wal_shipper.go:204-210` + +```go +// Fresh or late-attached replicas must consume the retained backlog before +// receiving a live-tail entry. This closes the LSN-gap path where the first +// post-attach live write would otherwise arrive before the retained prefix. +if st == ReplicaDisconnected && s.wal != nil && entry.LSN > 1 { + if _, err := s.CatchUpTo(entry.LSN - 1); err != nil { + log.Printf("wal_shipper: bounded catch-up before live ship failed ...") + return nil + } +} +``` + +This is V2's late-attach pattern: fresh replica needing backlog before live tail. V3 covers the same case differently — the engine emits `StartCatchUp` from probe results; this V2 inline call is **NOT** ported. Document explicitly so reviewer doesn't expect to find it. + +--- + +## §3 PORT items (5 — kickoff §9.5) + +Each row: V2 concern → V3 location → diff against current V3 (T4c-2 baseline) → invariant pin. + +### Item 1 — `runCatchUpTo(replicaFlushedLSN, targetLSN)` start boundary + +| Aspect | V2 | V3 today (T4c-2) | V3 after T4d-3 | +|---|---|---|---| +| Scan start arg | `fromLSN+1` (line 862) where `fromLSN = replicaFlushedLSN` | `ScanLBAs(1, ...)` hardcoded (`catchup_sender.go` post-T4c-3 minor fix) | `ScanLBAs(fromLSN, ...)` where `fromLSN = command.FromLSN` | +| Source of `replicaFlushedLSN` | V2 `reconnectWithHandshake` returns it | V3: probe handshake reports `Recovery.R`; engine populates `StartCatchUp.FromLSN = Recovery.R + 1` at command emit | same as v0.3 spec | + +**V2/V3 diff (explicit):** +- V2 derives `fromLSN` from a reconnect-handshake call **inside** `CatchUpTo` (`replicaFlushedLSN, err := s.reconnectWithHandshake()`). The handshake is shipper-internal. +- V3 derives `FromLSN` from `Recovery.R` **inside the engine** at command emit time. Probe (T4c-1) is the discovery mechanism; engine state is the source of truth. **Reason for the relocation**: engine-owned recovery decisions per Q1 (kickoff §9.3 #1). Transport doesn't re-discover R; engine tells transport what to ship. +- Off-by-one: V2 calls `StreamEntries(fromLSN+1, ...)` (skip applied LSN); V3 must do the same — engine should populate `FromLSN = Recovery.R + 1` directly, OR sender does the `+1` adjustment locally. **Decision needed in §6** below. + +**Pin**: `INV-REPL-CATCHUP-WITHIN-RETENTION-001` (un-pinned at T4c, ported at T4d-3). + +### Item 2 — Per-call catch-up deadline + +| Aspect | V2 | V3 today (T4c-2) | V3 after T4d-3 | +|---|---|---|---| +| Deadline source | `catchupTimeout` const (V2 package-level) | `recoveryConnTimeout` const (T4c-2 sender) | unchanged in T4d-3; T4c-pre-B `RecoveryRuntimePolicy.Timeout` migration is T4e/G5 work | +| Deadline scope | per-call: set at line 851, cleared at line 897 | per-call: SetDeadline at entry, deferred SetDeadline(zero) at exit | preserved; T4d-3 must NOT regress | + +**V2/V3 diff (explicit):** +- V2 sets the deadline ONLY when there's an existing `s.dataConn`; if conn is nil, returns `"no data connection"` error immediately. V3 doesn't have the same nil-check shape — V3 dials lazily inside `doCatchUp`. Both are correct; V2's nil-check is implementation-specific to its conn-pooled `WALShipper`. **NOT ported** (different conn-lifecycle model). +- V2 clears deadline AFTER `MsgCatchupDone` write, BEFORE final log line. V3's deferred `SetDeadline(time.Time{})` clears at function exit. Functionally equivalent; V3 deferred is cleaner. + +**Pin**: `INV-REPL-CATCHUP-DEADLINE-PER-CALL-SCOPE` (T4c-2; forward-carry). + +### Item 3 — Last-sent / achieved-LSN monotonicity + +| Aspect | V2 | V3 today (T4c-2) | V3 after T4d-3 | +|---|---|---|---| +| `lastSent` advance | line 873 — AFTER successful `WriteFrame` | T4c-2: `lastSent = entry.LSN` AFTER successful WriteMsg | preserved | +| Final completion source | V2: returns `lastSent` from `runCatchUpTo`; CALLER (`CatchUpTo`) returns it | V3: returns `BarrierResponse.AchievedLSN` from barrier round-trip (T4c-2 §4.2 binding) | preserved (V3 collapses V2's `MsgCatchupDone` into the existing barrier; INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN) | + +**V2/V3 diff (explicit):** +- V2 emits `MsgCatchupDone(doneLSN)` as a separate wire frame (line 890). V3 collapsed this into the existing barrier per T4c-2 §4.2 architect Option B (`INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN`). **NOT a regression** — completion information content is preserved (`BarrierResponse.AchievedLSN`); the wire is simpler. +- V2's `effectiveLast = lastSent; if zero, fromLSN` fallback (lines 901-904) is for the empty-window case (replica was already caught up). V3's barrier returns `AchievedLSN` from replica-side `Sync()` which already encodes that semantic — **NOT explicitly ported**, but the equivalent state is reached. + +**Pin**: `INV-REPL-CATCHUP-LASTSENT-MONOTONIC` (T4c-2; forward-carry under R+1 — test must verify lastSent advances normally when starting from R+1, not 1). + +### Item 4 — Catch-up retry budget (semantic port, V3 location) + +| Aspect | V2 | V3 today | V3 after T4d-3 | +|---|---|---|---| +| Retry budget value | `maxCatchupRetries = 3` (V2 line 152) | `RecoveryRuntimePolicy.MaxRetries = 3` for `wal_delta` (T4c-pre-B) | preserved value | +| Retry counter location | `s.catchupFailures` on `WALShipper` instance (line 285) | `RecoveryTruth.Attempts` on engine state (T4c-3) | unchanged | +| Retry escalation | `if catchupFailures >= max → state = NeedsRebuild` (line 287) | engine `applySessionFailed` increments Attempts; if exhausted → escalate via `recovery.Decision = Unknown` + `PublishDegraded` (T4c-3) | wired end-to-end via T4d-3 ReplicationVolume↔adapter plumbing | + +**V2/V3 diff (explicit):** +- V2 puts retry counter on the `WALShipper` per-instance (transport-side bookkeeping). V3 puts it on engine state per Q1 (kickoff §9.3 #1). **Architect-locked relocation** — engine owns recovery decisions including retry. +- V2 ESCALATION: `s.state.Store(NeedsRebuild)` directly. V3: engine sets `Decision = Rebuild` via the structured-kind path (T4d-1 work — `RecoveryFailureWALRecycled` triggers it). **Different mechanism, equivalent semantic.** +- V2's catch-up failure on non-recycle errors: returns `ErrReplicaDegraded` (line 291) — caller handles. V3: engine retry loop re-emits StartCatchUp until budget exhausted, then publishes Degraded. **More structured.** +- V2 `start_timeout` analog: V2 has no equivalent of V3's watchdog start-timeout; V3 added it for adapter-level executor stalls (T4c-3). Bypasses retry per T4c-3 design. **NOT a V2 port; V3-native robustness add.** + +**Pin**: V2 `maxCatchupRetries = 3` value preserved; mechanism is V3-native engine retry loop. + +### Item 5 — Retention-miss escalation (semantic port, V3 location) + +| Aspect | V2 | V3 today (T4c-2) | V3 after T4d-3 | +|---|---|---|---| +| Substrate signal | `errors.Is(err, ErrWALRecycled)` (line 878) | `storage.ErrWALRecycled` sentinel via `ScanLBAs` (T4c-2) | T4d-1 wraps as `storage.RecoveryFailure{Kind: StorageRecoveryFailureWALRecycled}` | +| Sender response | wraps `"catch-up: WAL recycled: %w"` (line 880) + sets `NeedsRebuild` | wraps `"catch-up: WAL recycled: %w"` + returns to caller; engine maps via substring (T4c-2) | sender extracts kind via `errors.As`, maps to `engine.RecoveryFailureWALRecycled`, populates `SessionCloseResult.FailureKind` | +| Engine response | (V2: state set in sender) | engine `applySessionFailed` substring-match → `Decision = Rebuild` (T4c-2) | T4d-1 typed branch: `if e.FailureKind == RecoveryFailureWALRecycled → Decision = Rebuild`; substring removed | + +**V2/V3 diff (explicit):** +- V2 sets state DIRECTLY in sender (`s.state.Store(NeedsRebuild)` line 879). V3 separates: sender reports failure kind, engine decides state transition. **Q1 architect-locked separation.** +- V2 leaks substrate sentinel name into transport error message ("WAL recycled"). V3 v0.3 contract: storage owns `StorageRecoveryFailureKind`; transport maps to engine kind; engine never imports storage. **Cleaner boundary; v0.3 dependency lock.** +- V2 retention boundary IS the WAL writer's `checkpointLSN`. V3 retention boundary is the substrate's own (walstore: `checkpointLSN`; smartwal: `oldestPreserved = head - capacity`). **Same semantic, substrate-specific implementation** — and T4d-3 doesn't add any new substrate work; T4c-2 already covers. + +**Pin**: `INV-REPL-CATCHUP-RECYCLE-ESCALATES` (T4c-2; pinning method changes from substring to typed `FailureKind` at T4d-1). + +--- + +## §4 V3-NATIVE items (5 — kickoff §9.5) + +These items have NO V2 source. The G-1 reviewer must NOT expect a V2 mapping for them. Each row is described to make scope explicit. + +### Item A — Engine-owned retry loop (Q1) + +**V3-native because**: V2 owned retry on the WALShipper instance (transport-side). V3 architect-locked at Q1: retry is engine-owned. Transport stays byte-movement-only. T4c-3 already implemented this at engine unit scope. T4d-3 wires it through ReplicationVolume so it runs end-to-end at integration scope. + +**Implementation today (T4c-3)**: `engine/apply.go applySessionFailed` increments `Recovery.Attempts`; while ≤ `MaxRetries`, re-emits `StartCatchUp`/`StartRebuild`; on exhaustion, resets + emits `PublishDegraded`. + +**T4d-3 work**: wire `ReplicationVolume↔adapter` so the engine's re-emitted commands actually drive a fresh executor session (not just unit-test scope). Component framework's `WithEngineDrivenRecovery()` swaps from stub to real. + +**No V2 source.** Do NOT look for one. + +### Item B — Unified recovery command model (`StartRecovery` / `RecoveryContentKind` / runtime policy) + +**V3-native because**: V2 has separate `CatchUpTo` and rebuild paths (different code, different commands). V3 unified at T4c-pre-B per memo §7a (architect rounds 29-30): one `StartRecovery` command + `RecoveryContentKind` + per-kind `RuntimePolicy`. V2 has no analog. + +**T4d-3 work**: nothing direct — T4c-pre-B already shipped the unified command. T4d-3 just continues to dispatch through the existing `StartRecoverySession` bridge. The legacy `StartCatchUp` path retained for engine emit; engine emit migration to `StartRecovery` is post-G5. + +**No V2 source.** + +### Item C — Lane-aware apply gate integration + +**V3-native because**: T4d-2 builds the gate from round-43/44 architect text; no V2 source. T4d-3's transport call into the gate is the integration point. + +**T4d-3 work**: catch-up sender's per-entry callback no longer calls `substrate.ApplyEntry` directly on the replica (it doesn't today either — it ships via wire and the replica handler applies). The change is at the **replica handler side** in T4d-2: `replica.go MsgShipEntry` handler routes via apply gate based on handler context (recovery vs live). T4d-3 ensures the catch-up handler route is wired correctly post-R+1 threading. Sender does NOT query substrate per-LBA freshness (`INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY` — Q3). + +**No V2 source.** V2 has no gate; replica blindly applies. + +### Item D — `BarrierResponse.AchievedLSN` as completion truth + +**V3-native because**: V2 uses `MsgCatchupDone(doneLSN)` as a separate wire frame. V3 collapsed at T4c-2 §4.2 (architect Option B) — `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN`. T4d-3 does NOT add MsgCatchupDone back. + +**T4d-3 work**: nothing direct — T4c-2 already established. Forward-carry test must verify barrier still completes correctly when scan starts at R+1 (not 1). + +**No V2 source for the collapse.** Original wire shape IS V2; the collapse is V3-native. + +### Item E — Package-boundary discipline (`INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY`) + +**V3-native because**: V2 monolithic `weed/storage/blockvol/` package — no equivalent layered separation. V3 layered architecture has clean boundaries (Q3 architect-locked at kickoff §9.3 #3): `core/transport` may depend on `core/storage` recovery contracts only, never substrate internals; `core/recovery` package move deferred post-G5/T4e. + +**T4d-3 work**: ensure the new R+1 threading + adapter wiring does NOT introduce new substrate-internal imports. Sender-side import audit in PR. **V3-native discipline; no V2 source for the rule.** + +**Pin**: `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY` inscribed at T4d-3. + +--- + +## §5 Hidden-invariant audit + +T4d-3 is largely a delta-on-T4c-2 (start LSN change + adapter wiring); most invariants are forward-carries from T4c-2. Anticipated 2 NEW hidden invariants: + +| # | Name | Statement | Inscribed at | +|---|---|---|---| +| 1 | `INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1` | Engine emits `StartCatchUp.FromLSN = Recovery.R + 1`. Sender does NOT add `+1` (avoids double-add). The "+1 to skip already-applied LSN" semantic lives at the engine, not transport. | T4d-3 | +| 2 | `INV-REPL-CATCHUP-FROMLSN-FROM-ENGINE-STATE-NOT-PROBE` | Engine populates `StartCatchUp.FromLSN` from its **own** `Recovery.R` state (the single source of truth). Probe results are ingested as facts that update engine state; the command emit path reads the updated state, never the raw probe payload directly. This pins the race window: probe → recovery decision → command emit always goes through engine state, no shortcut. (v0.2: positive-form rename per QA nit; was `INV-REPL-CATCHUP-FROMLSN-NOT-FROM-PROBE-DIRECTLY` in v0.1.) | T4d-3 | + +These pin the V3-native engine-emit policy decisions. Both relate to the V2/V3 diff Item 1 (start boundary derivation is V3-native; V2's reconnect-handshake-inside-shipper has no V3 analog). + +--- + +## §6 Open placement decisions (QA must sign before code) + +### §6.1 `+1` location: engine vs sender + +V2 calls `StreamEntries(fromLSN+1, ...)` — the `+1` lives at the consumer (V2 sender). + +V3 has two places for `+1`: + +| Option | Where | Pros | Cons | +|---|---|---|---| +| **A** — engine adds `+1` | `apply.go` populates `StartCatchUp.FromLSN = Recovery.R + 1` | engine owns the policy ("skip applied LSN"); sender stays substrate-mechanical; matches Q1 engine-owned-decisions philosophy | engine has to encode the off-by-one rule | +| **B** — sender adds `+1` | `catchup_sender.go` calls `ScanLBAs(command.FromLSN + 1, ...)` | sender semantics matches V2 (`StreamEntries(fromLSN+1, ...)`) | engine emits a "raw R" that the sender modifies; spreads the off-by-one rule | + +**sw recommendation**: **Option A** (engine adds `+1`). Reasons: +1. Engine owns recovery decisions per Q1; "skip applied LSN" is a recovery decision +2. Sender stays mechanical: scan from whatever the command says, no policy +3. Pinning becomes simpler: `INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1` lives at one site (engine emit) +4. V2's "+1 in sender" was an artifact of V2's monolithic shipper — V3's separation lets the policy move to its right home + +QA decision required pre-code. + +### §6.2 `Recovery.R` source for retry re-emit + +When engine retries (T4c-3 retry loop), it re-emits `StartCatchUp` with a fresh sessionID. Question: does the re-emit use `Recovery.R + 1` (the value from the original probe) or re-query somehow? + +V2 has no analog (V2 reconnects + handshakes per attempt — gets fresh `replicaFlushedLSN`). + +V3 options: + +| Option | Behavior | Notes | +|---|---|---| +| **A** — re-emit uses original `Recovery.R + 1` | Engine reuses the probe-derived value across retries within the same Decision window | Simpler; if previous attempt advanced replica's frontier, the next attempt will harmlessly re-ship the gap (replica's apply-gate handles via per-LBA stale-skip per T4d-2) | +| **B** — engine re-probes between retries | Inserts a probe between failure and re-emit | More accurate fromLSN; more wire traffic; engine state machine more complex | + +**sw recommendation**: **Option A** for T4d-3. Reasons: +1. Apply gate (T4d-2) is the correctness boundary: re-shipping the gap is safe +2. V2's per-attempt reconnect-handshake was for connection establishment, not specifically for fresh-fromLSN +3. Adding a probe between retries complicates the engine retry loop without correctness benefit +4. If the apply gate finds replica advanced, scan-time waste is limited (replica's per-LBA tracking causes immediate skip) + +QA decision required pre-code. + +### §6.3 Test rename for T4c-3 trailing chore + +T4d kickoff §5 carries: rename or split `TestT4c3_Catchup_ShortDisconnect_DeltaOnly` per architect's PM/lease round-1 finding ("test name overclaims today; once R+1 lands, either rename to reflect the now-correct delta semantic or split"). + +| Option | Action | +|---|---| +| **A** — rename | `TestT4d3_Catchup_RPlus1ShortGap_DeltaOnly` reflects the now-actually-delta semantic | +| **B** — split | Keep the original as a "ship-from-genesis" test; add `TestT4d3_Catchup_RPlus1ShortGap_DeltaOnly` as the R+1-specific test | + +**sw recommendation**: **Option B** (split). Both shapes are useful — the genesis-ship variant exercises the wrap-pattern (no replica state); the R+1 variant exercises the actual production path. Splitting documents the difference in the test names. + +QA decision required pre-code. + +--- + +## §7 Test parity matrix + +V2 test references: V2 has no formal R+1 unit test (the behavior is exercised through `TestRebuildWALCatchUpToReplica` and full integration scenarios). V3 will pin explicitly. + +| Test | Purpose | Pins which invariant | +|---|---|---| +| `TestT4d3_CatchUp_ScansFromReplicaR_NotGenesis` | Replica at R=50, primary at H=100; ScanLBAs called with `fromLSN=51`, NOT `1` | `INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1` | +| `TestT4d3_CatchUp_NonEmptyReplica_ShortGap_BandwidthBounded_Walstore` | Walstore substrate; use `NewObservedScanWrap`; emit count reflects R+1..target only, not 1..target. (v0.2 ADDITION 2: substrate-bound name per QA — walstore honors fromLSN; BlockStore synthesis is fromLSN-agnostic per §9 caveat) | `INV-REPL-CATCHUP-WITHIN-RETENTION-001` (T4c un-pin → T4d-3 PORTED) | +| `TestT4d3_CatchUp_NonEmptyReplica_ShortGap_BandwidthBounded_Smartwal` | Smartwal substrate; same shape as walstore variant; both honor fromLSN-driven retention | same | +| `TestT4d3_CatchUp_NonEmptyReplica_BlockStoreOverShipsExpected` | BlockStore variant — asserts the OPPOSITE: emit count = ALL stored LBAs regardless of fromLSN. Documents the test-scaffolding limitation per §9; surfaces if BlockStore synthesis ever changes (which would silently break the bandwidth narrative for production substrates) | (regression fence on §9 caveat) | +| `TestT4d3_RetryAfterReplicaAdvanced_OverScansHandledByApplyGate` | (v0.2 ADDITION 1 — pins §6.2 Option A integration assumption explicitly) Setup: catch-up attempt #1 ships entries `[R+1..K]` partially, then fails non-recycle; replica's actual R has advanced from R to K via the partial. Engine retry re-emits with **original** `Recovery.R+1` (not K+1). Attempt #2 over-ships `[R+1..K]` again. Apply gate (T4d-2) MUST per-LBA stale-skip the duplicates while still advancing `recoveryCovered`. Assert: (a) replica converges byte-exact, (b) per-LBA data NOT regressed at any LBA in `[R+1..K]`, (c) `recoveryCovered` includes `[R+1..K] ∪ [K+1..target]`. **Without this test, §6.2 Option A's "apply gate handles re-shipped gap" is asserted but not verified.** | `INV-REPL-NO-PER-LBA-DATA-REGRESSION` (T4d-2) + `INV-REPL-RECOVERY-COVERAGE-ADVANCES-ON-SKIP` (T4d-2) under retry path | +| `TestT4d3_CatchUp_Engine_RPlus1_FromOwnState_NotProbeDirectly` | Probe reports R=50; engine state updated; second probe reports R=60; engine emit uses CURRENT state R=60+1, not stale 50+1 | `INV-REPL-CATCHUP-FROMLSN-FROM-ENGINE-STATE-NOT-PROBE` | +| `TestT4d3_CatchUp_StartCatchUpSignature_FromLSNRequired` | Sender rejects calls without explicit `fromLSN` (compile fence) | signature contract | +| `TestT4d3_EngineRetryLoop_E2E_RetryUntilBudget` | Engine retry loop end-to-end through real ReplicationVolume; budget exhaustion → escalate observable | engine retry-loop wiring | +| `TestT4d3_EngineRetryLoop_E2E_RecycleEscalatesImmediate` | ErrWALRecycled bypasses retry, escalates to rebuild via real path | T4d-1 + T4d-3 chain | +| `TestComponent_WithEngineDrivenRecovery_NotStubAnymore` | Framework primitive emits real engine commands | framework primitive | +| `TestT4d3_Catchup_TransportNeverImportsSubstrateInternals` | grep `core/transport/catchup_sender.go` for `core/storage/walstore` or `core/storage/smartwal` imports; fails if present | `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY` (Q3) | +| (forward-carry from T4c-2) `TestCatchUpSender_HappyPath_WALReplay` | re-runs against R+1; must still pass | `INV-REPL-CATCHUP-CALLBACK-RETURN-NIL-CONTINUES`, lastSent-monotonic, deadline-per-call-scope | +| L2 matrix re-pass | 3 existing T4c L2 scenarios (ShortDisconnect, GapExceedsRetention, RecoveryModeLabelSurfaced) re-pass with engine-driven flow | T4c forward-carries | + +--- + +## §8 LOC heuristic + V3 vs V2 expansion budget + +Per `feedback_g1_pre_code_review.md` LOC asymmetry note: only V3 < 40-60% V2 is the red-flag direction. + +| Surface | V2 LOC | V3 expected LOC (T4d-3) | Notes | +|---|---|---|---| +| Inner-loop body (`runCatchUpTo` ~67 lines) | 67 | T4c-2 already at ~120 (callback shape adds boilerplate; QA-accepted at round-37) | T4d-3 changes ~3 lines (start LSN); no body rewrite | +| Calling frame (`CatchUpTo` ~40 lines retry+escalation) | 40 | engine apply.go applySessionFailed retry block (~30 lines today) + new `+1` site at emit (~3 lines) | engine handles retry; transport doesn't grow | +| Signature ripple | n/a (V2 method on shipper) | ~5 file touches: `engine/commands.go` + `apply.go` + `adapter/executor.go` + `transport/executor.go` + test stubs (mockExecutor / closure / policy / restart / Healthy / noop / ops) | ~50 LOC mechanical | +| ReplicationVolume↔adapter wiring | n/a (V2 monolithic) | ~80-100 LOC in `volume.go` to bind adapter into the live volume's command path | V3-native; no V2 source | +| `WithEngineDrivenRecovery()` real binding | n/a | ~50 LOC in component framework | V3-native | + +**Estimated total**: ~250 production + ~150 tests (matches mini-plan §2.3). No red-flag direction expected. + +--- + +## §9 Per-substrate behavior under R+1 + +T4d-3 only changes the start LSN of `ScanLBAs`. Each substrate's `ScanLBAs` already exists from T4c-2. Verify no substrate-specific edge case opens up: + +| Substrate | Behavior at `ScanLBAs(R+1, ...)` where R+1 > 1 | Concern? | +|---|---|---| +| **walstore** | Walks WAL entries in LSN order, filters `entry.LSN >= R+1`. Same code path as `ScanLBAs(1, ...)`; just emits fewer entries | None | +| **smartwal** | `oldestPreserved = head - capacity` if head > capacity, else 1. If `R+1 < oldestPreserved` → ErrWALRecycled (correct: gap exceeds retention). If `R+1 >= oldestPreserved` → scans normally | None — substrate handles cleanly | +| **BlockStore** | In-memory; no WAL. T4c-2 synthesized with `walHead` as scan-time LSN. Behavior at `ScanLBAs(51, ...)` when walHead=100: emits all stored LBAs at LSN=100. fromLSN=51 doesn't filter (synthesis emits at frontier-LSN, not per-LBA write-LSN) | T4d-3-specific concern: BlockStore's synthesis ALWAYS emits all LBAs regardless of fromLSN. **Test scenarios that use BlockStore + R+1 will not see bandwidth reduction** — they'll over-ship. This is a test-scaffolding limitation; production uses walstore/smartwal which honor fromLSN correctly. Document in test comments | + +**Action** (v0.2 — explicit per QA ADDITION 2): +- `TestT4d3_CatchUp_NonEmptyReplica_ShortGap_BandwidthBounded_Walstore` — asserts emit count is bounded +- `TestT4d3_CatchUp_NonEmptyReplica_ShortGap_BandwidthBounded_Smartwal` — asserts emit count is bounded +- `TestT4d3_CatchUp_NonEmptyReplica_BlockStoreOverShipsExpected` — asserts emit count is UNBOUNDED on BlockStore (regression fence: documents the synthesis limitation; surfaces immediately if BlockStore behavior ever changes) + +The substrate-bound naming prevents the silent-failure mode where a future "make tests run faster" refactor swaps walstore→BlockStore in the matrix and the bandwidth assertion silently passes a no-op test. + +--- + +## §10 Predicates (must be true before sw codes T4d-3) + +| Predicate | Source / receipt | +|---|---| +| T4c batch closed | `c910464a9` (architect 2026-04-25) | +| T4d kickoff §1–9 RATIFIED | architect sign 2026-04-25 | +| T4d mini-plan v0.3 ratified | architect 2026-04-25 | +| T4d-1 typed `RecoveryFailureKind` contract LANDED | (gates T4d-3 #5 retention-miss escalation port) — sw owns sequencing | +| T4d-2 apply gate LANDED | (gates T4d-3 Item C — sender-into-gate integration) — sw owns sequencing | +| QA single-sign on this G-1 | _________ pending | + +**Sequencing**: T4d-3 cannot land before T4d-1 + T4d-2 because items #5 (typed kind) and Item C (apply gate) depend on them. This G-1 documents the dependency; sw schedules accordingly. + +--- + +## §11 Sign + +| Role | Signer | Date | Decision | +|---|---|---|---| +| QA Owner | Claude (QA agent) | _________ | ⏸ pending review of §3 PORT items + §4 V3-native classification + §6 placement decisions | + +sw will not write T4d-3 production code until QA signs this G-1. Per mini-plan v0.3 §7.1 procedural binding: every T4d-3 PR commit references this G-1 commit hash via `Refs G-1 sign: `. + +--- + +## §12 Change log + +| Date | Change | Author | +|---|---|---| +| 2026-04-25 | Initial G-1 V2 read v0.1 for T4d-3 (R+1 threading + engine-driven recovery wiring). Explicit V2/V3 diff per kickoff §9.5 binding: 5 PORT items (start boundary, deadline, lastSent-monotonic, retry budget, retention-miss escalation) each with V2/V3-today/V3-after table + explicit diff narration; 5 V3-native items (engine-owned retry, unified command model, lane-aware apply gate integration, BarrierResponse.AchievedLSN as completion truth, package-boundary discipline) each with "no V2 source — do NOT look for one" disclaimer. 2 NEW hidden invariants surfaced for inscription at T4d-3 close: `INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1` + `INV-REPL-CATCHUP-FROMLSN-NOT-FROM-PROBE-DIRECTLY` (renamed in v0.2 per nit). 3 placement decisions raised for QA pre-code sign: §6.1 `+1` location (engine vs sender; sw recommends engine), §6.2 `Recovery.R` source for retry re-emit (original vs re-probe; sw recommends original), §6.3 T4c trailing test rename/split (sw recommends split). LOC heuristic: ~250 prod expected, no red-flag direction. Test parity matrix: 8 R+1 + engine wiring tests, 1 import-discipline fence (Q3), forward-carry of T4c-2 muscle invariants under R+1. Per-substrate audit: walstore + smartwal honor fromLSN cleanly; BlockStore synthesis is fromLSN-agnostic (test scaffolding limitation, documented). Predicates: T4d-1 + T4d-2 must land before T4d-3 (dependency ordering). | sw | +| 2026-04-25 (v0.2) | QA v0.1 review absorbed: ⏸ APPROVE WITH 2 ADDITIONS + 1 nit. **§6 placement decisions all RATIFIED** (Option A engine adds +1; Option A reuse original Recovery.R+1 conditional on Addition 1; Option B split T4c trailing test). **ADDITION 1 (medium)**: §6.2 Option A's "apply gate handles re-shipped gap via per-LBA stale-skip" was asserted-not-verified. Added `TestT4d3_RetryAfterReplicaAdvanced_OverScansHandledByApplyGate` to §7 — explicit setup: catch-up #1 ships `[R+1..K]` partially then fails non-recycle; replica's actual R advanced to K; engine retry re-emits with **original** `Recovery.R+1`; attempt #2 over-ships; apply gate per-LBA stale-skips while advancing recoveryCovered; assertions cover (a) byte-exact convergence, (b) no per-LBA data regression, (c) coverage union completeness. Pins `INV-REPL-NO-PER-LBA-DATA-REGRESSION` + `INV-REPL-RECOVERY-COVERAGE-ADVANCES-ON-SKIP` under retry path. **ADDITION 2 (low)**: §7 bandwidth test split into substrate-bound names: `_Walstore` + `_Smartwal` (assert bounded) + `_BlockStoreOverShipsExpected` (regression fence asserting unbounded — documents §9 synthesis limitation; surfaces if BlockStore behavior changes). Substrate-bound naming prevents the silent-failure mode where a future test refactor swaps walstore→BlockStore and the bandwidth assertion becomes a no-op. **Nit**: `INV-REPL-CATCHUP-FROMLSN-NOT-FROM-PROBE-DIRECTLY` → `INV-REPL-CATCHUP-FROMLSN-FROM-ENGINE-STATE-NOT-PROBE` (positive form per catalogue convention; same semantic). §9 action list updated with the 3 substrate-bound test names explicitly. Test count grows from 8 to 11. | sw | diff --git a/sw-block/design/v3-phase-15-t4d-forward-carry-checklist.md b/sw-block/design/v3-phase-15-t4d-forward-carry-checklist.md new file mode 100644 index 000000000..54653affd --- /dev/null +++ b/sw-block/design/v3-phase-15-t4d-forward-carry-checklist.md @@ -0,0 +1,144 @@ +# T4d Forward-Carry Verification Checklist (T4a + T4b + T4c → T4d) + +**Date**: 2026-04-25 (v0.2 — aligned with mini-plan v0.2 §6 superset) +**Status**: ✅ ACTIVE — gates T4d batch close per kickoff §4 #5 +**Owner**: QA +**Used by**: every T4d batch (T4d-1, T4d-2, T4d-3, T4d-4) at PR review + T4d batch close evidence + +**Revision history:** +- **v0.1** (2026-04-25 morning) — initial draft from T4c closure §I (6 T4c + 2 T4b + 1 T4a invariants) +- **v0.2** (2026-04-25 afternoon) — aligned with mini-plan v0.2 §6 superset; adds 3 T4a + 4 T4b + 4 T4c invariants I'd missed; corrects "Engine retry-budget escalation (no formal INV id)" to its actual catalogue name `INV-REPL-CATCHUP-RECYCLE-ESCALATES`; risk grades match mini-plan §6 column + +--- + +## Purpose + +Per kickoff §4 acceptance criterion #5: *"T4a/T4b/T4c invariants forward-carry verified (no regression)."* + +T4d touches three high-risk surfaces: **substrate `ApplyEntry`** (T4d-1), **replica apply gate above the substrate** (T4d-2), and **`StartCatchUp` signature + engine retry wire** (T4d-3). Plus lifecycle close (T4d-4). Each could regress an active T4a/T4b/T4c invariant if the T4d change accidentally bypasses an existing pin point. This checklist is the gate that catches it. + +Risk grading mirrors mini-plan v0.2 §6 column: **High** = T4d directly changes the pinning method; **Medium** = T4d changes adjacent code that touches the invariant's enforcement path; **Low** = T4d code does not touch this invariant's enforcement (smoke-test only). + +--- + +## Active T4a invariants (forward-carry) + +| Invariant | Pin location at T4a | T4d regression risk | Verification | +|---|---|---|---| +| `INV-REPL-SHIP-TRANSPORT-MUSCLE-001` | T4a-2 Ship transport muscle | **Low** — Ship code path not touched | Smoke via T4a-2 suite under G-3 | +| `INV-REPL-FANOUT-001` | T4a-3 ReplicaPeer fan-out | **Low** | Smoke via T4a-6 BasicEndToEnd | +| `INV-REPL-LSN-ORDER-FANOUT-001` | T4a-4 ReplicationVolume Option-X LSN-order fan-out | **Medium — T4d-3 changes catch-up scan boundary; engine-driven recovery wires through ReplicationVolume** | Re-run T4a-4 fan-out tests; assert LSN-order fan-out across replicas preserved when engine drives recovery (no out-of-order under retry) | +| `INV-REPL-PEER-REBUILD-ON-AUTHORITY-CHANGE` | T4a-4 peer rebuild on authority change | **Low** | T4a-4 test | + +--- + +## Active T4b invariants (forward-carry) + +| Invariant | Pin location at T4b | T4d regression risk | Verification | +|---|---|---|---| +| `INV-REPL-LINEAGE-BORNE-ON-BARRIER-ACK` | T4b-1 BarrierResponse 32B lineage echo | **Low — wire untouched** | T4b-1 + T4b-6 wire fence; assert BarrierResponse still carries echoed RecoveryLineage on R+1-aware path | +| `INV-REPL-NO-ZERO-LINEAGE-FIELDS` | T4b-1 strict-decode no-zero | **Low — no new wire surfaces in T4d** | T4b-1 fence still applies | +| `INV-REPL-BARRIER-FAILURE-DEGRADES-PEER` | T4b-3 peer-state on barrier failure | **Low** | T4b-3 + T4b-4 peer-state pin | +| `INV-REPL-LOCAL-FSYNC-GATES-QUORUM` | T4b-4 DurabilityCoordinator | **Low** | T4b-4 | +| `INV-REPL-DURABILITY-COORDINATOR-OWNS-SYNC` | T4b-4 + T4b-5 `ReplicationVolume.Sync` | **Medium — T4d-4 lifecycle close interacts with coordinator state** | Re-run T4b-4 + T4b-5 durability tests during T4d-4 lifecycle work; assert coordinator state cleanup doesn't bypass durability invariants | + +--- + +## Active T4c invariants (forward-carry) + +| Invariant | Pin location at T4c | T4d regression risk | Verification | +|---|---|---|---| +| `INV-REPL-LINEAGE-BORNE-ON-PROBE-PAIR` | T4c-1 ProbeReq + ProbeResponse symmetric pair | **Low — probe wire untouched** | T4c-1 fence | +| `INV-REPL-PROBE-NON-MUTATING-VALIDATION` | `core/transport/replica.go:238` `validateProbeLineage` + T4c-1 unit + QA #1 | **Low — probe handler untouched** | T4c-1 test; assert `validateProbeLineage` still called for `MsgProbeReq` (NOT replaced by `acceptMutationLineage`) | +| `INV-REPL-CATCHUP-CALLBACK-RETURN-NIL-CONTINUES` | T4c-2 per-entry target cap | **Medium — T4d-2 apply gate inserted between scan callback and substrate; T4d-3 changes scan boundary** | Forward-carry test in T4d-2 (apply gate forwards nil correctly) + T4d-3 (R+1 path); assert returning nil continues scan | +| `INV-REPL-CATCHUP-LASTSENT-MONOTONIC` | T4c-2 `doCatchUp` lastSent loop + T4c-2 unit + QA #6 | **Medium — T4d-3 R+1 changes start LSN** | Forward-carry test in T4d-3; assert `lastSent` advances only-forward across the new R+1 + retry-loop path | +| `INV-REPL-CATCHUP-DEADLINE-PER-CALL-SCOPE` | T4c-2 per-call deadline `defer SetDeadline(time.Time{})` + T4c-2 unit + QA #5 | **Low — deadline discipline preserved** | T4c-2 test + QA #5 DeadlinePerCallScope_NoSpilling; assert each engine-driven retry sets + clears its own deadline | +| `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN` | T4c-2 barrier-as-terminator + T4c-2 unit + QA #4 | **Low — completion semantics unchanged** (barrier remains authoritative; T4d-2 coverage map contributes to barrier judgment, doesn't replace) | T4c-2 test + QA #4 BarrierAchievedLSN_PartialProgress + new T4d-2 `TestApplyGate_BarrierCompletion_BarrierAuthoritativeWithCoverageContribution` | +| `INV-REPL-CATCHUP-TARGET-NOT-REACHED-VS-RECYCLED-DISTINGUISHED` | T4c-2 distinguishing failure modes | **Medium — T4d-1 structured-kind switch changes detection** | T4d-1 test must distinguish the two via typed kind (`RecoveryFailureKind`), not substring | +| `INV-REPL-CATCHUP-RECYCLE-ESCALATES` *(corrected from v0.1 "Engine retry-budget escalation; no formal INV id")* | T4c-2 recycle escalation + T4c-3 L2 + QA #2 | **High — T4d-1 changes pinning method from substring match to structured `FailureKind` field** | T4d-1 test chain: substrate wraps as `RecoveryFailure{Kind: WALRecycled}` → sender extracts via `errors.As` → engine branches on typed `e.FailureKind` → escalates `Decision = Rebuild`. Substring `"WAL recycled"` MUST be removed from `core/engine/apply.go` per criterion #7. Fence test `TestEngine_SessionFailed_NoMoreSubstringMatch` greps source. | +| `INV-REPL-PEER-STATE-CATCHINGUP` | T4c-2 peer state machine | **Low** | T4c-2 test | +| `INV-REPL-RECOVERY-MODE-OBSERVABLE` | T4c-2 mode label emit + T4c-2 unit + QA #3 | **Medium — T4d-4 replaces duck-typed `CheckpointLSN` probe with `RecoveryMode()` method** | T4d-4 test + QA #3 ModeLabelObservability; assert success log line still carries `recovery_mode=wal_replay` or `recovery_mode=state_convergence` per substrate after method replacement; closes round-40 wrap-vs-CheckpointLSN limitation | +| Engine retry-budget per-content-kind defaults *(no formal INV id; T4c-3 forward-carry)* | T4c-3 `RecoveryRuntimePolicy.MaxRetries` (3 / 0 / 1) | **Low** | T4c-3 unit test | + +--- + +## Verification protocol (per T4d batch PR) + +Each T4d batch PR must run this checklist as part of QA review: + +1. **Smoke-test the full active invariant suite**: + ```bash + go test -count=1 ./core/replication/... ./core/transport/... ./core/engine/... ./core/storage/... + ``` + All packages must remain green. + +2. **Spot-test the High + Medium rows above** for THIS batch's regression-risk column — re-run the specific named tests, not just package-level smoke. Catch silent semantic drift that compiles + passes generic tests but breaks the specific pin. + +3. **Catalogue cross-check**: confirm no `✓ PORTED T4a/T4b/T4c` row in catalogue §3.3 quietly downgrades to `⊙` or `⏭` in the diff. If a T4d change requires downgrade, that's a §B-style scope-delta requiring architect acceptance, not a silent change. + +4. **Document deltas**: any forward-carry that fails or weakens MUST appear in T4d batch's PR description as an explicit delta entry (mirroring T4c closure §B's discipline). + +--- + +## Per-batch focus (high+medium risk only — efficient PR review) + +**T4d-1 (substrate hardening + structured kind)** — verify these specifically: +- High: `INV-REPL-CATCHUP-RECYCLE-ESCALATES` (substring → typed kind switch) — ✅ landed `1edeb36` with `TestEngine_SessionFailed_NoMoreSubstringMatch` fence + comment-stripping helper +- Medium: `INV-REPL-CATCHUP-TARGET-NOT-REACHED-VS-RECYCLED-DISTINGUISHED` (typed kind detection) + - **REQUIRED FOLLOW-UP before T4d-3 G-1 sign** (architect-bound 2026-04-25): + - `TestT4d1_TargetNotReached_DistinctKindFromWALRecycled` — asserts `classifyRecoveryFailure(...)` maps target-not-reached error to `engine.RecoveryFailureTargetNotReached` AND maps WAL recycled to `engine.RecoveryFailureWALRecycled` (distinct values) + - `TestT4d1_StorageFailureKindMapper_AllKnownKinds` — fence: enumerate all `storage.StorageRecoveryFailureKind` values; assert `classifyRecoveryFailure` maps each to a non-`Unknown` `engine.RecoveryFailureKind`. Catches future storage enum additions not mapped in transport. + - Where: either small T4d-1 follow-up patch OR rolled into T4d-2's first commit — sw discretion + - Hard gate: T4d-3 G-1 sign requires both tests green + +**T4d-2 (replica apply gate)** — verify these specifically: +- Medium: `INV-REPL-CATCHUP-CALLBACK-RETURN-NIL-CONTINUES` (apply gate inserted between callback and substrate) — ✅ landed `bd2de99` +- Low (but high-impact regression if broken): `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN` (barrier remains authoritative; coverage feeds judgment) — ✅ forward-carry green +- **REQUIRED FOLLOW-UP — HARD GATE before T4d-3** (architect-bound 2026-04-25 round-46 review): + - **Lane discriminator rework**: T4d-2's `bd2de99` implementation uses `lineage.TargetLSN > liveShipTargetLSN(=1)` as discriminator — payload-derived, not handler-context-derived. Architect: "I would not solve it by adding an engine precondition TargetLSN >= 2; that changes recovery semantics to protect an implementation shortcut. Choose Option 2: rework to true handler-context lane." + - Concrete direction: + - Change apply gate hook from single `Apply(lineage, ...)` to lane-explicit: either `ApplyLive(...)` + `ApplyRecovery(...)` separate methods OR `Apply(..., lane ApplyLane)` with explicit param + - Caller must supply `lane` from connection/session handler context — NOT inferred from `lineage.TargetLSN` + - Keep no wire byte (Q2 still stands) + - Add regression test `TestApplyGate_RecoveryWithTargetLSN1_RoutesToRecoveryLane` — pins the edge case (recovery to H=1 must NOT misroute to live lane) + - Rename/remove existing T4d-2 tests with "lane derived from TargetLSN" phrasing — that mental model must go away from the codebase + - Where: T4d-2 follow-up patch BEFORE T4d-3 starts + - Hard gate: T4d-3 cannot start until the rework lands (architect: "T4d-3 will thread real TargetLSN/FromLSN and could make this edge case less theoretical. T4d-3 should not build more wiring on a payload-sentinel lane rule.") + - Rationale: `INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` catalogue inscription says "accepting handler/session context" — sw's implementation drifted from the inscription text; the rework realigns implementation with the architect-locked invariant statement + +**T4d-3 (R+1 + engine-driven recovery wiring)** — verify these specifically: +- Medium: `INV-REPL-LSN-ORDER-FANOUT-001` (engine drives recovery through ReplicationVolume) +- Medium: `INV-REPL-CATCHUP-CALLBACK-RETURN-NIL-CONTINUES` (R+1 scan boundary) +- Medium: `INV-REPL-CATCHUP-LASTSENT-MONOTONIC` (R+1 changes start LSN) +- **HARD CLOSE CONDITION** (architect-bound 2026-04-25 round-46+): the caller-side `TargetLSN==1` lane-discrimination shim in `core/transport/replica.go:13-18,156-165` (introduced T4d-2 follow-up `01f4ab9` as TRANSITIONAL) **cannot survive as an unbounded known bug into T4d-3 close**. T4d-3 must do EXACTLY ONE of: + - **(option A)** Land true handler/session-context lane signal that REMOVES the TargetLSN==1 shim entirely (per-conn lane tag at handshake / separate handlers / distinct ports — sw picks) + - **(option B)** Add an explicit failing-or-skip-marked L2 test `TestT4d3_RecoveryTargetLSN1_KnownGap` that documents the H=1 edge case as a known gap, with godoc citing `CARRY-T4D-LANE-CONTEXT-001`. This test MUST exercise the H=1 scenario (primary writes 1 entry, replica empty, catch-up emitted with TargetLSN=1) and either fail-loud or skip with explicit reference to the carry id. + - Architect explicit prohibition: do NOT add engine precondition `TargetLSN >= 2` ("changes recovery semantics to protect an implementation shortcut") +- **PRE/WITH-T4d-3 doc fixes** (architect-bound 2026-04-25): + - `core/replication/apply_gate.go:38-43` — type-level godoc still describes OLD payload-derived discrimination ("the gate reads the lineage's TargetLSN signal"). Rewrite to reflect lane-pure model with caller-supplied lane. + - `core/transport/replica.go` — `NewReplicaListener*` constructor godoc that still references `hook.Apply(...)` / "lane discrimination" wording must update to reflect lane-explicit `ApplyRecovery` / `ApplyLive` hook shape. + - Where: T4d-3 first commit OR small standalone PR before T4d-3 +- **CARRY-T4D-LANE-CONTEXT-001** (named carry inscribed in catalogue §3.3 + below): replace TargetLSN==1 caller shim with true handler/session context lane signal. Owner: sw. Bind point: T4e (preferred) or post-G5 protocol-hardening. T4d-3 close evidence MUST cite this carry id if option B taken. + +**T4d-4 (lifecycle + integration)** — verify these specifically: +- Medium: `INV-REPL-DURABILITY-COORDINATOR-OWNS-SYNC` (lifecycle close interacts with coordinator) +- Medium: `INV-REPL-RECOVERY-MODE-OBSERVABLE` (`RecoveryMode()` method replaces duck-typed probe) + +--- + +## Closure evidence at T4d batch close + +QA single-sign at T4d close per §8C.2 includes a §-formatted block in the closure report: + +``` +| Carry-forward | Status | Evidence | +|---|---|---| +| INV-REPL-PROBE-NON-MUTATING-VALIDATION | ✅ green | T4c-1 + QA #1 re-run at HEAD | +| INV-REPL-CATCHUP-RECYCLE-ESCALATES | ✅ green; pinning method changed substring→typed kind | T4d-1 chain re-run; fence TestEngine_SessionFailed_NoMoreSubstringMatch green | +| INV-REPL-LSN-ORDER-FANOUT-001 | ✅ green; engine-driven retry preserves order | T4a-4 + T4d-3 retry path | +| INV-REPL-DURABILITY-COORDINATOR-OWNS-SYNC | ✅ green; lifecycle cleanup doesn't bypass | T4b-4 + T4b-5 + T4d-4 lifecycle | +| INV-REPL-RECOVERY-MODE-OBSERVABLE | ✅ green; method replaces duck-typed probe; wraps forward correctly | T4c-2 + QA #3 + T4d-4 RecoveryMode method tests | +| ... (all 19 active T4a/T4b/T4c invariants) | ... | ... | +``` + +Filling this in is QA's last action before signing the T4d closure report. diff --git a/sw-block/design/v3-phase-15-t4d-qa-scenario-catalogue.md b/sw-block/design/v3-phase-15-t4d-qa-scenario-catalogue.md new file mode 100644 index 000000000..66eb4c429 --- /dev/null +++ b/sw-block/design/v3-phase-15-t4d-qa-scenario-catalogue.md @@ -0,0 +1,370 @@ +# T4d QA Scenario Catalogue (component-scope) + +**Date**: 2026-04-25 +**Status**: ✅ ACTIVE — catalogue ready for conversion to test code as each T4d-N batch lands +**Owner**: QA +**Mirrors**: T4c QA Stage-1 (`core/replication/component/qa_t4c_scenarios_test.go`) — same framework + same authoring discipline +**Used by**: QA at each T4d-N batch close; converted to actual test code in `core/replication/component/qa_t4d_scenarios_test.go` (or split per batch) + +--- + +## §1 Purpose + +Component-scope QA scenarios provide a **second independent perspective** alongside sw's unit + L2 tests. Distinct discipline: + +| Sw tests | QA component scenarios | +|---|---| +| Unit-level: tight scope, single-replica, mock peers | Cluster-level: `WithReplicas(N)`, real wire, real apply path | +| Test the implementation pieces | Test the invariants through the orchestration layer | +| Single substrate (or matrix at unit-scope) | `component.RunMatrix` — both substrates per scenario by default | +| Pre-built sw test setup helpers | Component framework primitives (`WithPrimaryStorageWrap`, `NewSeverDuringScanWrap`, `NewObservedScanWrap`, `WithLiveShip`, `PrimaryWriteViaBackend`) | + +QA scenarios MUST NOT duplicate sw's tests at a different file location — they must exercise **different combinations** that surface integration-layer behavior sw's unit tests don't naturally cover. + +--- + +## §2 Authoring discipline (per T4c QA Stage-1 precedent) + +- Each scenario pins ≥1 invariant from catalogue §3.3 +- `RunMatrix` when invariant applies to both substrates; `RunSubstrate(...,Smartwal/Walstore,...)` when substrate-mode specific +- Don't reach into framework internals — surface a primitive gap to sw if missing (T4c precedent: `WithPrimaryStorageWrap` etc.) +- Component-scope ONLY (not L2 subprocess, not L3 hardware) — those are sw + future m01 deliverables +- Scenarios stay ~30-50 lines each; bigger means the framework is missing a primitive + +--- + +## §3 T4d-2 scenarios (apply gate) + +T4d-2 is where most new invariants land. Sw's mini-plan §2.4 lists 14 unit-scope tests. QA scenarios add multi-replica + matrix + composition coverage. + +### QA #1 — `TestT4d_QA_NoPerLBADataRegression_RetryAfterReplicaAdvanced` + +**Pins**: `INV-REPL-NO-PER-LBA-DATA-REGRESSION` + `INV-REPL-RECOVERY-STALE-ENTRY-SKIP-PER-LBA` (composite — goal-level invariant via mechanism invariant under retry path) + +**Distinct from sw**: sw's `TestComponent_Adversarial_StaleEntryDoesNotRegress` is single-replica + single-attempt. This composes multi-replica + sever-mid-stream + retry — verifies the §6.2 Option A safety claim from a higher orchestration angle than the G-1 §7 test pins. + +**Substrate**: matrix + +**Setup outline**: +```go +component.RunMatrix(t, func(t, c) { + c.WithReplicas(2). + WithPrimaryStorageWrap(component.NewSeverDuringScanWrap(5, nil)). + Start() + c.PrimaryWriteN(20) + c.PrimarySync() + // attempt #1 ships [R+1..5], severs + r1 := c.CatchUpReplica(0); assert !r1.Success + // primary writes more; replica 0's R has advanced via partial + c.PrimaryWriteN(10) + // attempt #2 retries with original Recovery.R+1; over-ships [R+1..5] before reaching new entries + r2 := c.CatchUpReplica(0); assert r2.Success + // assert byte-exact + no per-LBA regression at any LBA + c.AssertReplicaConverged(0) + c.AssertNoPerLBARegression(0) // new framework primitive needed +}) +``` + +**Framework primitive needed**: `Cluster.AssertNoPerLBARegression(idx)` — surface to sw if not already in T4d-2 framework additions. + +--- + +### QA #2 — `TestT4d_QA_LaneDiscipline_LiveAndRecoveryConcurrent_BothCorrect` + +**Pins**: `INV-REPL-LANE-DERIVED-FROM-HANDLER-CONTEXT` (Q2) — composition test + +**Distinct from sw**: sw's `TestApplyGate_LaneNeverDerivedFromWireField` is a fence test (asserts no wire byte). This exercises the actual concurrent-lane behavior — live ships + catch-up running simultaneously, verify each lane's apply path executes correctly without cross-contamination. + +**Substrate**: matrix + +**Setup outline**: +```go +component.RunMatrix(t, func(t, c) { + c.WithReplicas(1).WithLiveShip().Start() + // concurrent: live-ship some LBAs while catch-up is mid-stream + c.PrimaryWriteViaBackendN(5) // live lane + go c.CatchUpReplica(0) // recovery lane + c.PrimaryWriteViaBackendN(5) // more live during catch-up + c.WaitForConverge(2 * time.Second) + // assert: liveTouched contains live's LBAs; recoveryCovered contains catch-up's LBAs; + // no overlap-routing errors (recovery LBA didn't go through live path or vice versa) + c.AssertLaneIntegrity(0) // new framework primitive needed +}) +``` + +**Framework primitive needed**: `Cluster.AssertLaneIntegrity(idx)` — verifies internal handler-routing correctness from the Cluster orchestration angle. + +--- + +### QA #3 — `TestT4d_QA_CoverageAdvancesOnSkip_BarrierStillCompletes` + +**Pins**: `INV-REPL-RECOVERY-COVERAGE-ADVANCES-ON-SKIP` + `INV-REPL-CATCHUP-COMPLETION-FROM-BARRIER-ACHIEVED-LSN` together (forward-carry under coverage path) + +**Distinct from sw**: sw's `TestComponent_RecoveryStaleSkip_CoverageStillAdvances` tests single-LBA case. This tests the all-stale case: replica is FULLY caught up (no actual data writes needed), recovery stream ships everything, every entry stale-skips, barrier MUST still complete. Stresses the "skip data ≠ ignore frame" architectural decision. + +**Substrate**: matrix + +**Setup outline**: +```go +component.RunMatrix(t, func(t, c) { + c.WithReplicas(1).Start() + // pre-seed replica fully via ReplicaApply (matches primary) + for i := 0; i < 10; i++ { + data := makeData(i) + lsn := c.PrimaryWrite(uint32(i), data) + c.ReplicaApply(0, uint32(i), data, lsn) + } + c.PrimarySync() + // catch-up will ship entries; replica will skip all (already applied at higher LSN) + result := c.CatchUpReplica(0) + assert result.Success // barrier completes + assert result.AchievedLSN == 10 // achieved despite zero data writes + c.AssertReplicaConverged(0) +}) +``` + +--- + +### QA #4 — `TestT4d_QA_RecoveryRestart_SecondAttemptSeedsFromSubstrate` + +**Pins**: `INV-REPL-RECOVERY-COVERAGE-RESTART-SAFE` (Option C hybrid) + +**Distinct from sw**: sw's `TestApplyGate_RestartMidRecovery_SessionRestartReseeds` tests the gate's seed mechanism in isolation. This tests the cross-substrate Option C hybrid behavior at the orchestration layer: walstore + smartwal must seed from `AppliedLSNs()`; BlockStore variant must fall back to in-memory + log INFO once. + +**Substrate**: matrix + dedicated BlockStore variant + +**Setup outline (matrix walstore+smartwal)**: +```go +component.RunMatrix(t, func(t, c) { + c.WithReplicas(1).Start() + c.PrimaryWriteN(10) + c.PrimarySync() + c.CatchUpReplica(0) // replica caught up via session #1 + c.RestartReplica(0) // simulate restart; new framework primitive + c.PrimaryWriteN(5) + c.PrimarySync() + c.CatchUpReplica(0) // session #2 re-seeds appliedLSN from substrate + c.AssertReplicaConverged(0) + c.AssertSessionStartReseededFromSubstrate(0) // new framework primitive +}) +``` + +Plus dedicated `TestT4d_QA_RecoveryRestart_BlockStoreFallback_LogsOnce` smartwal variant verifying the INFO log. + +**Framework primitives needed**: `Cluster.RestartReplica(idx)` + `Cluster.AssertSessionStartReseededFromSubstrate(idx)`. + +--- + +## §4 T4d-3 scenarios (R+1 + engine-driven recovery) + +T4d-3 unblocks `INV-REPL-CATCHUP-WITHIN-RETENTION-001` (un-pinned at T4c). Sw's mini-plan §2.5 lists 11 R+1 + engine-wiring tests. QA adds matrix-multi-replica composition. + +### QA #5 — `TestT4d_QA_CatchUpFromRPlus1_BandwidthBounded_MultiReplica` + +**Pins**: `INV-REPL-CATCHUP-WITHIN-RETENTION-001` (T4c un-pin → T4d-3 PORTED) — at multi-replica scope + +**Distinct from sw**: sw's `TestT4d3_CatchUp_NonEmptyReplica_ShortGap_BandwidthBounded_{Walstore,Smartwal}` is single-replica. This composes 2 replicas at different R values — primary's catch-up to each must scan from each replica's individual R+1, not a shared start. + +**Substrate**: matrix walstore + smartwal (BlockStore variant separate per G-1 §9 caveat) + +**Setup outline**: +```go +component.RunSubstrate(t, "walstore", component.Walstore, func(t, c) { + c.WithReplicas(2).Start() + c.PrimaryWriteN(50) + c.PrimarySync() + c.CatchUpReplica(0) // replica 0 fully caught up: R=50 + c.PrimaryWriteN(20) // primary advances to H=70 + c.PrimarySync() + counter1 := new(atomic.Int32) + counter0 := new(atomic.Int32) + // engine emits StartCatchUp.FromLSN=51 for replica 0 (R+1=51); ships 20 entries + // replica 1 still empty: FromLSN=1; ships ~70 entries + c.CatchUpReplicaWithEmitCounter(0, counter0) // new primitive + c.CatchUpReplicaWithEmitCounter(1, counter1) + assert counter0.Load() == 20 + assert counter1.Load() >= 70 +}) +``` + +**Framework primitive needed**: `Cluster.CatchUpReplicaWithEmitCounter(idx, *atomic.Int32)` — convenience wrapper around existing `NewObservedScanWrap` + `CatchUpReplica`. + +--- + +### QA #6 — `TestT4d_QA_FromLSNFromEngineState_NotStaleProbeDirectly` + +**Pins**: `INV-REPL-CATCHUP-FROMLSN-FROM-ENGINE-STATE-NOT-PROBE` + +**Distinct from sw**: sw's `TestT4d3_CatchUp_Engine_RPlus1_FromOwnState_NotProbeDirectly` tests at engine unit scope. This drives the same scenario through the Cluster — two probes interleave; engine state must reflect the LATEST probe; emit must use updated state. + +**Substrate**: matrix + +**Setup outline**: +```go +component.RunMatrix(t, func(t, c) { + c.WithReplicas(1).Start() + c.PrimaryWriteN(50); c.PrimarySync() + c.ProbeReplica(0) // probe #1 reports R=0; engine state R=0 + c.CatchUpReplica(0) // catches up; replica R now 50 + c.ProbeReplica(0) // probe #2 reports R=50; engine state updated R=50 + c.PrimaryWriteN(20); c.PrimarySync() + counter := new(atomic.Int32) + c.CatchUpReplicaWithEmitCounter(0, counter) + // engine emit uses current R=50 → FromLSN=51 → ships 20 entries + // NOT stale probe #1's R=0 → FromLSN=1 → would ship 70 + assert counter.Load() == 20 +}) +``` + +--- + +### QA #7 — `TestT4d_QA_TransportImportFence_NoSubstrateInternals` + +**Pins**: `INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY` (Q3) — compile-time fence + +**Distinct from sw**: sw's `TestT4d3_Catchup_TransportNeverImportsSubstrateInternals` is the same fence; QA's variant is a parallel grep test that runs in CI alongside sw's, with QA-perspective error message ("transport package leaked substrate dependency — check core/transport imports against contract surface allow-list"). Belt-and-suspenders for an architecturally-critical invariant. + +**Substrate**: N/A (linter-style; no Cluster setup needed) + +**Setup outline**: +```go +func TestT4d_QA_TransportImportFence_NoSubstrateInternals(t *testing.T) { + forbidden := []string{ + "github.com/seaweedfs/seaweed-block/core/storage/walstore", + "github.com/seaweedfs/seaweed-block/core/storage/smartwal", + } + transportFiles, _ := filepath.Glob("../../transport/*.go") + for _, file := range transportFiles { + content, _ := os.ReadFile(file) + for _, forbid := range forbidden { + if strings.Contains(string(content), forbid) { + t.Errorf("transport package leaked substrate dependency: %s imports %s\n"+ + " Q3 architect lock: transport may only depend on core/storage contract surface, not substrate internals.\n"+ + " See INV-REPL-TRANSPORT-STORAGE-CONTRACT-ONLY", + file, forbid) + } + } + } +} +``` + +--- + +## §5 T4d-4 scenarios (lifecycle + remaining T4c carries) + +### QA #8 — `TestT4d_QA_LastSentMonotonic_AcrossRetries_FullForm` + +**Pins**: `LastSentMonotonic_AcrossRetries` full cross-call form (deferred T4c Stage-2 scenario, now testable) + +**Distinct from sw**: this IS the full form of T4c QA #6 (`TestT4c_QA_LastSentMonotonic_WithinCall`). T4c could only test within-call because `WithEngineDrivenRecovery` was a stub; T4d-3 binds it for real. T4d-4 lands the cross-call form QA-side. + +**Substrate**: matrix + +**Setup outline**: +```go +component.RunMatrix(t, func(t, c) { + c.WithReplicas(1). + WithEngineDrivenRecovery(). // now real, not stub + WithPrimaryStorageWrap(component.NewSeverDuringScanWrap(3, nil)). + Start() + c.PrimaryWriteN(20) + c.PrimarySync() + // engine drives multiple retry attempts (sever happens each call) + // assert: lastSent across attempts is monotonic (each attempt's lastSent ≥ prior achieved) + observed := c.WaitForEngineExhaustion(0, 10*time.Second) // new primitive + c.AssertLastSentMonotonicAcrossAttempts(observed) // new primitive +}) +``` + +**Framework primitives needed**: `Cluster.WaitForEngineExhaustion(idx, timeout)` + `Cluster.AssertLastSentMonotonicAcrossAttempts([]SessionResult)`. + +--- + +### QA #9 — `TestT4d_QA_LifecycleStop_DuringActiveCatchUp_NoHandleLeak` + +**Pins**: `INV-REPL-LIFECYCLE-HANDLE-BORROWED-001` — adversarial timing + +**Distinct from sw**: sw's `TestReplicationVolume_Stop_DuringCatchUp_TerminatesCleanly` tests that Stop completes. This adds the BUG-005 non-repeat assertion: substrate handle was BORROWED (not owned) by ReplicationVolume; Stop MUST NOT close it; subsequent ProvOpen must reuse the handle without "already closed" errors. + +**Substrate**: matrix + +**Setup outline**: +```go +component.RunMatrix(t, func(t, c) { + c.WithReplicas(1).Start() + c.PrimaryWriteN(50) + c.PrimarySync() + done := make(chan struct{}) + go func() { + c.CatchUpReplica(0) + close(done) + }() + time.Sleep(10*time.Millisecond) // let catch-up begin + c.StopReplicationVolume(0) // Stop mid-catchup + <-done + // assert substrate handle still open (borrowed-not-owned) + c.AssertSubstrateHandleStillOpen(0) + // assert can ProvOpen + reuse without error + c.ReopenReplicationVolume(0) + c.AssertReplicaConverged(0) +}) +``` + +**Framework primitives needed**: `Cluster.StopReplicationVolume(idx)`, `Cluster.AssertSubstrateHandleStillOpen(idx)`, `Cluster.ReopenReplicationVolume(idx)`. + +--- + +## §6 Cumulative scenario count + +| Batch | QA scenarios | Sw tests in mini-plan | Total invariant pin coverage | +|---|---|---|---| +| T4d-2 | 4 (QA #1-#4) | 14 | All 6 T4d-2 invariants pinned by both sides | +| T4d-3 | 3 (QA #5-#7) | 11 | All 4 T4d-3 invariants pinned by both sides | +| T4d-4 | 2 (QA #8-#9) | 15 | T4c §I carries closed | +| **Total T4d QA** | **9 scenarios** | 40 sw tests | 19 forward-carry invariants verified by checklist | + +T4c QA Stage-1 had 6 scenarios; T4d expands to 9 reflecting the bigger T4d scope. + +--- + +## §7 Framework primitives requested from sw (forward-carry to T4d-2/T4d-3/T4d-4 framework PRs) + +Per T4c precedent, QA surfaces missing primitives early so sw can land them in the same batch as the underlying production code: + +| Primitive | Used by QA scenario | Lands in batch | +|---|---|---| +| `Cluster.AssertNoPerLBARegression(idx)` | QA #1 | T4d-2 (apply gate is the source of truth for per-LBA tracking) | +| `Cluster.AssertLaneIntegrity(idx)` | QA #2 | T4d-2 | +| `Cluster.RestartReplica(idx)` | QA #4 | T4d-2 | +| `Cluster.AssertSessionStartReseededFromSubstrate(idx)` | QA #4 | T4d-2 | +| `Cluster.CatchUpReplicaWithEmitCounter(idx, *atomic.Int32)` | QA #5, #6 | T4d-3 (convenience wrapper around `NewObservedScanWrap` + `CatchUpReplica`) | +| `Cluster.WaitForEngineExhaustion(idx, timeout)` | QA #8 | T4d-3 (alongside `WithEngineDrivenRecovery` real binding) | +| `Cluster.AssertLastSentMonotonicAcrossAttempts([]SessionResult)` | QA #8 | T4d-3 | +| `Cluster.StopReplicationVolume(idx)` | QA #9 | T4d-4 | +| `Cluster.AssertSubstrateHandleStillOpen(idx)` | QA #9 | T4d-4 | +| `Cluster.ReopenReplicationVolume(idx)` | QA #9 | T4d-4 | + +Surfacing now (pre-batch) avoids the round-39/40 T4c experience where QA had to wait for sw to add `WithPrimaryStorageWrap` + `WithLiveShip` before authoring deferred scenarios. + +--- + +## §8 Authoring sequence + +QA scenarios convert from catalogue to actual test code per batch: + +| When | Action | +|---|---| +| After T4d-2 lands | Author QA #1-#4 in `core/replication/component/qa_t4d2_scenarios_test.go` (or merge into single `qa_t4d_scenarios_test.go`) | +| After T4d-3 lands | Author QA #5-#7 | +| After T4d-4 lands | Author QA #8-#9 | +| At T4d batch close | Verify all 9 scenarios green at HEAD; cite in QA single-sign artifact | + +If a scenario can't be authored due to missing framework primitive (per §7), surface to sw immediately — same discipline as T4c QA round-39 (`WithPrimaryStorageWrap` etc.). + +--- + +## §9 Change log + +| Date | Change | Author | +|---|---|---| +| 2026-04-25 | Initial v0.1 catalogue from T4d kickoff v0.3 + mini-plan v0.2 + G-1 v0.2. 9 component-scope scenarios across T4d-2 (4) / T4d-3 (3) / T4d-4 (2). Each pins specific invariant + setup outline + framework primitive requirements. 10 framework primitives surfaced for sw's batch PRs (avoids round-40-style "QA blocked on framework gap" deferral). Discipline mirrors T4c QA Stage-1 (`qa_t4c_scenarios_test.go`). Authoring sequence: convert catalogue → test code per batch as sw's production code lands. | QA |