From 21ccf06ef3027ab143886855fc4c4ce76acc740a Mon Sep 17 00:00:00 2001 From: pingqiu Date: Fri, 3 Apr 2026 08:48:05 -0700 Subject: [PATCH] docs: Phase 13 CP13-1..CP13-7 technical packs, acceptance status, design updates - phase-13.md: CP13-1 through CP13-6 accepted, CP13-7 active - phase-13-log.md: full technical + delivery packs for CP13-2..CP13-7 - phase-13-cp4-state-eligibility.md: refined barrier behavior table (Disconnected/Degraded as recovery entry points, not eligibility) - phase-12.md: minor cross-reference updates - Older phase docs: minor wording alignment - Design docs: V2 development plan and completion overview updated Co-Authored-By: Claude Opus 4.6 (1M context) --- sw-block/.private/phase/phase-12.md | 28 +- .../phase/phase-13-cp4-state-eligibility.md | 38 +- sw-block/.private/phase/phase-13-log.md | 1038 +++++++++++++++++ sw-block/.private/phase/phase-13.md | 608 ++++++++++ sw-block/design/v2-phase-development-plan.md | 68 +- .../design/v2-product-completion-overview.md | 61 +- 6 files changed, 1793 insertions(+), 48 deletions(-) create mode 100644 sw-block/.private/phase/phase-13.md diff --git a/sw-block/.private/phase/phase-12.md b/sw-block/.private/phase/phase-12.md index d7799bdba..525fe7bad 100644 --- a/sw-block/.private/phase/phase-12.md +++ b/sw-block/.private/phase/phase-12.md @@ -1,7 +1,7 @@ # Phase 12 Date: 2026-04-02 -Status: active +Status: accepted Purpose: move the accepted chosen-path implementation from candidate-safe product closure toward production-safe behavior under restart, disturbance, and operational reality ## Why This Phase Exists @@ -423,7 +423,7 @@ Reject if: Status: -- active +- accepted Carry-forward from `P3`: @@ -431,23 +431,13 @@ Carry-forward from `P3`: 2. `P4` now asks whether that accepted path has an explicit measured floor and an explicit first-launch envelope 3. later work after `Phase 12` should be a productionization program, not another hidden hardening slice -## Assignment For `sw` +## Phase Close-Out Note -Current next tasks: +`Phase 12` is now accepted as bounded production hardening on the chosen path: -1. deliver `Phase 12 P4` as bounded performance-floor / rollout-gate hardening -2. keep the acceptance object fixed on one measured workload envelope plus one explicit launch-envelope / gate artifact -3. keep earlier accepted `Phase 09` / `Phase 10` / `Phase 11` / `Phase 12 P1` / `Phase 12 P2` / `Phase 12 P3` semantics stable unless `P4` exposes a real bug or measurement-surface gap -4. do not let `P4` turn into broad optimization, topology expansion, or generic launch marketing +1. `P1` accepted disturbance correctness +2. `P2` accepted bounded soak / long-run stability +3. `P3` accepted diagnosability / blocker accounting / runbook hardening +4. `P4` accepted bounded performance floor / rollout-gate hardening -## Assignment For `tester` - -Current next tasks: - -1. validate `P4` as real bounded performance-floor / rollout-gate hardening rather than “benchmark numbers exist” prose -2. require explicit validation targets for: - - one named workload envelope with measured floor values - - one explicit launch-envelope / rollout-gate artifact - - no-overclaim around broad production readiness beyond the named envelope -3. keep no-overclaim active around accepted `Phase 09` / `Phase 10` / `Phase 11` / `Phase 12 P1` / `Phase 12 P2` / `Phase 12 P3` closure -4. keep `P4` bounded rather than letting it absorb post-Phase-12 productionization work +Next work should open a new phase or program rather than silently continuing inside `Phase 12`. diff --git a/sw-block/.private/phase/phase-13-cp4-state-eligibility.md b/sw-block/.private/phase/phase-13-cp4-state-eligibility.md index 44423f8be..0b48995e9 100644 --- a/sw-block/.private/phase/phase-13-cp4-state-eligibility.md +++ b/sw-block/.private/phase/phase-13-cp4-state-eligibility.md @@ -7,14 +7,14 @@ Code change: one new test (`TestBarrier_NonEligibleStates_FailClosed`) The replication path uses a bounded 6-state set (`wal_shipper.go:25-30`): -| State | Value | Meaning | Barrier eligible? | -|-------|-------|---------|-------------------| -| `Disconnected` | 0 | No session (initial state) | No — attempts bootstrap/reconnect, fails if no progress | -| `Connecting` | 1 | Socket open, handshake pending | No — immediate `ErrReplicaDegraded` | -| `CatchingUp` | 2 | Connected, replaying missed WAL | No — immediate `ErrReplicaDegraded` | -| `InSync` | 3 | Eligible for sync_all barriers | **Yes** — only state that proceeds to barrier request | -| `Degraded` | 4 | Transient failure, retry allowed | No — attempts reconnect, fails if reconnect fails | -| `NeedsRebuild` | 5 | WAL gap too large, rebuild required | No — immediate `ErrReplicaDegraded` | +| State | Value | Meaning | Barrier behavior | +|-------|-------|---------|-----------------| +| `Disconnected` | 0 | No session (initial state) | Attempts bootstrap/reconnect inside Barrier(); fails if no progress or reconnect fails | +| `Connecting` | 1 | Socket open, handshake pending | Immediate `ErrReplicaDegraded` | +| `CatchingUp` | 2 | Connected, replaying missed WAL | Immediate `ErrReplicaDegraded` | +| `InSync` | 3 | Eligible for sync_all barriers | **Proceeds to barrier request** — only state that can complete barrier successfully | +| `Degraded` | 4 | Transient failure, retry allowed | Attempts reconnect inside Barrier(); fails if reconnect fails | +| `NeedsRebuild` | 5 | WAL gap too large, rebuild required | Immediate `ErrReplicaDegraded` | ## Barrier State Gate @@ -33,16 +33,28 @@ default: } ``` -**Only `InSync` enters the barrier request path.** All other states either fail immediately or attempt reconnect (which must succeed and transition to `InSync` before reaching the barrier). +**Contract (precise):** + +- **Only `InSync` can complete barrier successfully.** It is the only state that proceeds + directly to the barrier request (ensureCtrlConn → MsgBarrierReq → wait for BarrierOK). +- **`Disconnected` and `Degraded` use Barrier() as a recovery entry point.** They attempt + bootstrap/reconnect inside the Barrier() call. If recovery succeeds and transitions to + InSync, the barrier request proceeds. If recovery fails, the barrier fails. +- **`Connecting`, `CatchingUp`, `NeedsRebuild` are rejected immediately** with `ErrReplicaDegraded`. + +The key distinction: Barrier() can be *invoked* from Disconnected/Degraded (as a recovery +trigger), but only InSync can *satisfy* barrier success. The Disconnected/Degraded paths +are recovery attempts, not barrier eligibility. ## sync_all Gate `dist_group_commit.go:59-66`: sync_all counts barrier failures. Any shipper that returns an error from `Barrier()` increments `failCount`. If `failCount > 0`, sync_all returns `ErrDurabilityBarrierFailed`. Combined with the CP13-3 fix (FlushedLSN=0 rejected), the full chain is: -1. Only `InSync` shippers reach the barrier request -2. Only `BarrierOK` with `FlushedLSN > 0` counts as success -3. sync_all fails if any barrier fails +1. Only `InSync` shippers proceed to the barrier request +2. Disconnected/Degraded may recover inside Barrier(), transitioning to InSync before requesting +3. Only `BarrierOK` with `FlushedLSN > 0` counts as success +4. sync_all fails if any barrier fails ## Proof Promotion @@ -50,7 +62,7 @@ Combined with the CP13-3 fix (FlushedLSN=0 rejected), the full chain is: | Test | What it proves for CP13-4 | |------|--------------------------| -| `TestBarrier_NonEligibleStates_FailClosed` | Connecting, CatchingUp, NeedsRebuild all rejected immediately; Disconnected fails on dead address; only InSync enters barrier path | +| `TestBarrier_NonEligibleStates_FailClosed` | 5 sub-cases: Connecting/CatchingUp/NeedsRebuild rejected immediately; Disconnected fails (no recovery on dead addr); InSync enters barrier path (verified by MsgBarrierReq receipt on fake server) | | `TestBarrier_RejectsReplicaNotInSync` | SyncCache fails when replica is not InSync (end-to-end) | | `TestBarrier_DuringCatchup_Rejected` | Barrier rejected while replica is CatchingUp | | `TestDistSync_SyncAll_AllDegraded_Fails` | sync_all fails when all replicas degraded | diff --git a/sw-block/.private/phase/phase-13-log.md b/sw-block/.private/phase/phase-13-log.md index 782e6793f..2f248c22b 100644 --- a/sw-block/.private/phase/phase-13-log.md +++ b/sw-block/.private/phase/phase-13-log.md @@ -246,3 +246,1041 @@ These are explicitly reserved for CP13-2 through CP13-7. The baseline must expos 2. the failing baseline maps cleanly to later checkpoints 3. already-correct behavior is distinguished from true gaps 4. no implementation overclaim sneaks into the checkpoint + +--- + +### `CP13-2` Technical Pack + +Date: 2026-04-03 +Goal: close the canonical replica addressing gap so replication endpoint truth is always exported as routable authoritative `ip:port` rather than wildcard listener strings or incomplete address forms + +#### Layer 1: Semantic Core + +##### Problem statement + +`CP13-1` froze the current replication baseline and left one explicit `PASS*` gap for canonical addressing: + +1. `TestBug3_ReplicaAddr_MustBeIPPort_WildcardBind` + +This means current code has evidence that the wildcard-bind address path is still only partially closed. + +`CP13-2` therefore accepts only one bounded thing: + +1. canonical endpoint truth for replica addresses used by the replication path + +It does not accept: + +1. durable-progress truth +2. reconnect handshake or catch-up +3. retention policy or rebuild fallback +4. broad networking redesign + +##### State / contract + +`CP13-2` must make these truths explicit: + +1. exported replica addresses used by replication are canonical routable `ip:port` +2. wildcard listener strings and bare `:port` are not valid published replication truth +3. any canonicalization occurs at the production truth surface, not only in tests or log formatting +4. loopback stays loopback only when explicitly intended rather than by accidental leakage + +##### Reject shapes + +Reject before implementation if the checkpoint: + +1. fixes only one string formatting path while another exported address path still leaks wildcard/bare-port truth +2. relies on test-side normalization rather than production-side canonicalization +3. mixes endpoint canonicalization with reconnect/rebuild protocol work + +#### Layer 2: Execution Core + +##### Current gap `CP13-2` must close + +1. replica receiver / registration code may still export `:port` or wildcard-bind forms instead of routable `ip:port` +2. authoritative registry / heartbeat truth may diverge from local listener truth if canonicalization is partial + +##### Suggested file targets + +1. `weed/storage/blockvol/replica_receiver.go` +2. `weed/storage/blockvol/replica_meta.go` +3. nearby address-canonicalization helpers +4. `weed/server/master_block_registry.go` +5. heartbeat / registration path carrying replica endpoints, only if alignment is required + +##### Validation focus + +Required proofs: + +1. wildcard-bind canonicalization proof + - exported replication endpoint becomes canonical `ip:port` +2. no-leak proof + - wildcard/bare-port listener strings do not escape into replication truth +3. boundedness proof + - checkpoint remains about endpoint truth, not later protocol behavior + +Reject if: + +1. `TestBug3_ReplicaAddr_MustBeIPPort_WildcardBind` passes only because the test normalizes the string itself +2. one endpoint path is fixed while another published path still leaks stale/non-canonical truth +3. delivery claims imply reconnect/durability closure + +##### Suggested first cut + +1. identify the authoritative source of replica `DataAddr` / `CtrlAddr` +2. canonicalize there to routable `ip:port` +3. verify the same canonical truth is what registration / heartbeat / registry consumers observe +4. flip the `PASS*` address witness into a real proof + +##### Assignment For `sw` + +1. Goal + - deliver bounded canonical replica addressing on the production truth surface +2. Required outputs + - one implementation update at the real endpoint-truth surface + - one focused proof package centered on wildcard-bind/exported-address correctness + - one delivery note explaining: + - files updated in place + - why the chosen canonicalization point is authoritative + - what later checkpoints remain untouched +3. Hard rules + - do not touch reconnect handshake / catch-up logic + - do not touch retention / rebuild policy + - do not count test-side normalization as a real fix + +##### Assignment For `tester` + +1. Goal + - validate that `CP13-2` closes canonical endpoint truth and nothing broader +2. Validate + - wildcard-bind export becomes canonical `ip:port` + - no wildcard/bare-port leak remains on the published replication path + - no-overclaim around `CP13-3+` +3. Reject if + - the proof stays witness-only + - canonicalization is partial or inconsistent across exported surfaces + - the delivery claims reconnect or durability semantics changed + +#### Short judgment + +`CP13-2` is acceptable when: + +1. the wildcard-bind address witness becomes a real endpoint-truth proof +2. exported replica addresses are canonical `ip:port` +3. no wildcard/bare-port leak remains on the published replication path +4. the checkpoint stays clearly separate from `CP13-3+` + +--- + +### `CP13-2` Delivery Pack + +Bounded contract: + +1. `CP13-2` accepts canonical replica addressing only +2. it does not accept durable-progress truth, reconnect/catch-up, retention, rebuild, or rollout claims + +What `sw` should deliver: + +1. one production-side canonicalization fix for replica endpoint truth +2. one focused proof package showing exported replication endpoints are canonical `ip:port` +3. one delivery note with: + - changed files + - proof shape + - no-overclaim statement + +Recommended delivery shape: + +1. code: + - update the authoritative endpoint-truth surface in place +2. tests: + - strengthen `TestBug3_ReplicaAddr_MustBeIPPort_WildcardBind` from witness to proof + - add one adjacent no-leak assertion only if needed to prove the production path rather than one helper +3. note: + - explain why the fix closes canonical endpoint truth + - explain why `CP13-3+` remains untouched + +Review checklist: + +1. does the fix happen at the real truth surface? +2. does exported replication truth now always use canonical `ip:port`? +3. are wildcard/bare-port leaks actually eliminated? +4. is the checkpoint still bounded to address truth? + +--- + +### `CP13-3` Technical Pack + +Date: 2026-04-03 +Goal: make durable replication progress explicit and authoritative so sync correctness is tied to replica flushed durability rather than sender-side send progress or informal success heuristics + +#### Layer 1: Semantic Core + +##### Problem statement + +`CP13-1` baseline shows many durable-progress tests already pass on current code, but baseline evidence alone does not close the checkpoint. + +`CP13-3` therefore accepts only one bounded thing: + +1. durable progress truth for the replication path + +It does not accept: + +1. reconnect/catch-up protocol +2. retention policy +3. rebuild fallback +4. broad replication state-machine closure + +##### State / contract + +`CP13-3` must make these truths explicit: + +1. `replicaFlushedLSN` means replica-side WAL durability, not sender transmission progress +2. barrier success for sync correctness depends on durable progress truth +3. sender-side sent/shipped LSN remains diagnostic and must not authorize sync success +4. barrier/control surfaces expose durable progress explicitly enough to prove correctness + +##### Reject shapes + +Reject before implementation or review if the checkpoint: + +1. treats passing baseline tests as automatic closure without restating the durable-progress contract +2. leaves sender-side send progress able to masquerade as durable authority +3. mixes durable-progress work with reconnect/rebuild protocol changes + +#### Layer 2: Execution Core + +##### Current gap `CP13-3` must close + +1. durable progress truth may still be only implicitly inferred rather than explicitly owned and reviewed +2. baseline evidence suggests current code may already satisfy much of the contract, but that must be confirmed in a bounded proof package + +##### Suggested file targets + +1. `weed/storage/blockvol/replica_apply.go` +2. `weed/storage/blockvol/wal_shipper.go` +3. `weed/storage/blockvol/dist_group_commit.go` +4. protocol message definitions used by the barrier/control path + +##### Validation focus + +Required proofs: + +1. barrier-uses-flushed proof + - barrier success is grounded in `replicaFlushedLSN` +2. monotonicity proof + - flushed progress is monotonic within epoch +3. no-false-authority proof + - receive/sent/shipped progress alone does not count as durability authority +4. boundedness proof + - checkpoint remains about durable-progress truth, not `CP13-4+` + +Reject if: + +1. a passing test still depends on sender-side send progress rather than replica durability +2. the proof package cannot explain why receive progress and flushed progress are distinct +3. delivery wording implies reconnect or retention semantics were closed + +##### Suggested first cut + +1. restate the durable-progress contract explicitly against current code +2. review the existing PASS baseline tests and identify which are: + - real proof of the contract + - adjacent support evidence + - out of scope for `CP13-3` +3. make only the minimum code/test adjustments needed to close any remaining contract gap +4. produce one delivery note that explains why durable-progress truth is now explicit and bounded + +##### Assignment For `sw` + +1. Goal + - deliver bounded durable-progress truth on the production replication path +2. Required outputs + - one explicit durable-progress contract summary + - one focused proof package centered on: + - `replicaFlushedLSN` + - barrier response truth + - sender-side progress remaining diagnostic only + - one delivery note explaining: + - files updated in place + - which baseline PASS tests now count as real `CP13-3` proof + - what later checkpoints remain untouched +3. Hard rules + - do not broaden into reconnect handshake / catch-up + - do not broaden into retention or rebuild policy + - do not treat “tests already pass” as sufficient without contract-level review + +##### Assignment For `tester` + +1. Goal + - validate that `CP13-3` closes durable-progress truth and nothing broader +2. Validate + - barrier success is tied to flushed durability + - flushed progress is monotonic and not advanced on mere receive + - sender-side shipped/sent progress remains diagnostic only + - no-overclaim around `CP13-4+` +3. Reject if + - durable-progress authority is still implicit or mixed + - evidence relies on transport progress rather than replica durability + - the delivery claims reconnect/retention/rebuild closure + +#### Short judgment + +`CP13-3` is acceptable when: + +1. durable progress truth is explicit and authoritative +2. barrier correctness is tied to replica flushed durability +3. sender-side send progress is clearly non-authoritative +4. the checkpoint stays clearly separate from `CP13-4+` + +--- + +### `CP13-3` Delivery Pack + +Bounded contract: + +1. `CP13-3` accepts durable-progress truth only +2. it does not accept reconnect/catch-up, retention, rebuild, or rollout claims + +What `sw` should deliver: + +1. one focused contract review of `replicaFlushedLSN` / barrier durable-progress truth +2. one bounded code/test package only if needed to close the contract +3. one delivery note with: + - changed files + - proof shape + - which baseline PASS tests are promoted into real `CP13-3` evidence + - no-overclaim statement + +Recommended delivery shape: + +1. contract: + - define why `replicaFlushedLSN` is the authority for sync correctness +2. code/tests: + - keep updates minimal and local to durable-progress surfaces + - strengthen or narrow existing tests only where needed to prove the contract cleanly +3. note: + - explain which proof cases are primary versus support evidence + - explain why `CP13-4+` remains untouched + +Review checklist: + +1. is durable progress explicitly defined as replica durability? +2. does barrier success rely on flushed truth rather than send progress? +3. are sent/shipped progress variables clearly diagnostic only? +4. is the checkpoint still bounded to durable-progress truth? + +--- + +### `CP13-4` Technical Pack + +Date: 2026-04-03 +Goal: make replica state and barrier eligibility explicit so only `InSync` replicas can satisfy sync durability while non-eligible states fail closed instead of drifting into accidental success + +#### Layer 1: Semantic Core + +##### Problem statement + +`CP13-1` baseline and later review show state/eligibility behavior is partly present in current code, but the checkpoint is not closed until the state contract is made explicit and reviewed as a bounded object. + +`CP13-4` therefore accepts only one bounded thing: + +1. replica state and barrier eligibility truth + +It does not accept: + +1. reconnect/catch-up protocol +2. retention policy +3. rebuild fallback +4. broad rollout or performance claims + +##### State / contract + +`CP13-4` must make these truths explicit: + +1. the replication path uses a bounded state set: + - `Disconnected` + - `Connecting` + - `CatchingUp` + - `InSync` + - `Degraded` + - `NeedsRebuild` +2. only `InSync` replicas count toward sync durability +3. non-eligible states must reject or fail closed rather than silently participating in barrier success +4. state eligibility is separate from later reconnect/rebuild protocol closure + +##### Reject shapes + +Reject before implementation or review if the checkpoint: + +1. treats passing baseline tests as automatic closure without restating the state/eligibility contract +2. leaves one or more non-eligible states able to satisfy sync durability implicitly +3. mixes state/eligibility work with reconnect, retention, or rebuild implementation + +#### Layer 2: Execution Core + +##### Current gap `CP13-4` must close + +1. state names and barrier eligibility rules may still be implicit or scattered +2. baseline evidence suggests some fail-closed behavior already exists, but that must be confirmed in a bounded proof package + +##### Suggested file targets + +1. `weed/storage/blockvol/wal_shipper.go` +2. `weed/storage/blockvol/dist_group_commit.go` +3. `weed/storage/blockvol/shipper_group.go` +4. focused protocol/adversarial tests for state eligibility + +##### Validation focus + +Required proofs: + +1. only-in-sync proof + - only `InSync` replicas satisfy barrier eligibility for sync durability +2. fail-closed proof + - degraded/non-eligible replicas do not silently count toward `sync_all` +3. state-boundary proof + - barrier rejects or excludes disallowed states explicitly +4. boundedness proof + - checkpoint remains about state/eligibility truth, not `CP13-5+` + +Reject if: + +1. a passing test still allows a non-eligible state to count toward durable success +2. the proof package cannot explain how `CatchingUp`, `Degraded`, or `NeedsRebuild` are excluded +3. delivery wording implies reconnect/retention/rebuild closure + +##### Suggested first cut + +1. restate the replica-state eligibility contract explicitly against current code +2. review the existing PASS baseline tests and identify which are: + - primary proof of the eligibility contract + - adjacent support evidence + - out of scope for `CP13-4` +3. make only the minimum code/test adjustments needed to close any remaining eligibility gap +4. produce one delivery note that explains why state/eligibility truth is now explicit and bounded + +##### Assignment For `sw` + +1. Goal + - deliver bounded replica state / barrier eligibility truth on the production replication path +2. Required outputs + - one explicit state/eligibility contract summary + - one focused proof package centered on: + - only `InSync` counts + - non-eligible states fail closed + - barrier admission/exclusion rules + - one delivery note explaining: + - files updated in place + - which baseline PASS tests now count as real `CP13-4` proof + - what later checkpoints remain untouched +3. Hard rules + - do not broaden into reconnect handshake / catch-up + - do not broaden into retention or rebuild policy + - do not treat “tests already pass” as sufficient without contract-level review + +##### Assignment For `tester` + +1. Goal + - validate that `CP13-4` closes replica state / barrier eligibility and nothing broader +2. Validate + - only `InSync` replicas count toward sync durability + - degraded/non-eligible replicas fail closed + - state-boundary rejection/exclusion is explicit + - no-overclaim around `CP13-5+` +3. Reject if + - non-eligible states still leak into durable success + - evidence relies on reconnect or rebuild behavior not in scope + - the delivery claims reconnect/retention/rebuild closure + +#### Short judgment + +`CP13-4` is acceptable when: + +1. the state set and eligibility rules are explicit +2. only `InSync` replicas count toward sync durability +3. non-eligible states fail closed rather than drifting into false success +4. the checkpoint stays clearly separate from `CP13-5+` + +--- + +### `CP13-4` Delivery Pack + +Bounded contract: + +1. `CP13-4` accepts replica state / barrier eligibility only +2. it does not accept reconnect/catch-up, retention, rebuild, or rollout claims + +What `sw` should deliver: + +1. one focused contract review of replica state and barrier eligibility +2. one bounded code/test package only if needed to close the contract +3. one delivery note with: + - changed files + - proof shape + - which baseline PASS tests are promoted into real `CP13-4` evidence + - no-overclaim statement + +Recommended delivery shape: + +1. contract: + - define the bounded state set and why only `InSync` is eligible +2. code/tests: + - keep updates minimal and local to state/eligibility surfaces + - strengthen or narrow existing tests only where needed to prove the contract cleanly +3. note: + - explain which proof cases are primary versus support evidence + - explain why `CP13-5+` remains untouched + +Review checklist: + +1. is the replica state set explicit? +2. does `sync_all` count only `InSync` replicas? +3. do non-eligible states fail closed? +4. is the checkpoint still bounded to state/eligibility truth? + +--- + +### `CP13-5` Technical Pack + +Date: 2026-04-03 +Goal: make reconnect after replica disturbance explicit and correct so a replica with prior durable progress resumes through handshake truth and retained-WAL catch-up rather than unsafe bootstrap or barrier-level guesswork + +#### Layer 1: Semantic Core + +##### Problem statement + +`CP13-4` accepted the replica state / barrier-eligibility contract. +That closed who may satisfy sync durability, but it did not close how a previously-synced replica returns to `InSync` after disturbance. + +`CP13-1` baseline leaves the remaining reconnect cluster explicit: + +1. `TestAdversarial_ReconnectUsesHandshakeNotBootstrap` fails +2. `TestAdversarial_CatchupMultipleDisconnects` fails +3. `TestAdversarial_CatchupDoesNotOverwriteNewerData` fails because catch-up does not complete cleanly enough to exercise the safety invariant +4. `TestReconnect_GapBeyondRetainedWal_NeedsRebuild` is still only `PASS*` because it proves bounded failure, not full rebuild-state closure + +`CP13-5` therefore accepts only one bounded thing: + +1. reconnect handshake + WAL catch-up truth for recoverable retained-WAL gaps + +It does not accept: + +1. replica-aware WAL retention policy +2. full `NeedsRebuild` lifecycle closure +3. rebuild execution/orchestration +4. broad rollout or performance claims + +##### State / contract + +`CP13-5` must make these truths explicit: + +1. fresh replicas with no prior durable progress may bootstrap +2. replicas with prior durable progress must reconnect using explicit resume truth, not bare bootstrap +3. if the retained WAL covers the replica gap, catch-up must replay the missing WAL and only then allow barrier success +4. if the gap cannot be recovered within retained WAL, reconnect must fail closed in this checkpoint and leave full rebuild closure to `CP13-7` +5. repeated disconnect/reconnect cycles must remain bounded and recoverable rather than drifting into permanent degraded failure + +##### Reject shapes + +Reject before implementation or review if the checkpoint: + +1. allows a previously-synced replica to succeed without using reconnect handshake truth +2. allows barrier success before catch-up has restored `InSync` +3. mixes reconnect/catch-up work with retention or rebuild lifecycle closure +4. claims full `NeedsRebuild` semantics as if `CP13-7` were already closed + +#### Layer 2: Execution Core + +##### Current gap `CP13-5` must close + +1. degraded shippers with prior durable progress still fail to reconnect and catch up cleanly before barrier +2. repeated disconnect/reconnect cycles do not recover robustly +3. catch-up safety cases are blocked because reconnect/catch-up does not complete cleanly enough to exercise the intended invariant +4. gap-beyond-retained-WAL behavior is only bounded-failure evidence today, not a fully-reviewed reconnect contract + +##### Suggested file targets + +1. `weed/storage/blockvol/wal_shipper.go` +2. reconnect / catch-up helpers adjacent to `wal_shipper` +3. `weed/storage/blockvol/sync_all_protocol_test.go` +4. `weed/storage/blockvol/sync_all_adversarial_test.go` +5. nearby protocol message helpers only if the reconnect contract requires them + +##### Validation focus + +Required proofs: + +1. reconnect-discriminator proof + - fresh shipper bootstrap remains separate from reconnect-with-progress +2. catch-up proof + - recoverable retained-WAL gap replays missing WAL and returns the shipper to `InSync` +3. repeated-recovery proof + - multiple disconnect/reconnect cycles recover without hanging or silently bypassing catch-up +4. bounded-failure proof + - unrecoverable gaps fail closed and do not claim reconnect success +5. boundedness proof + - checkpoint remains about reconnect/catch-up truth, not `CP13-6+` + +Reject if: + +1. reconnect success is inferred only from a final passing `SyncCache` without proving the handshake/catch-up path +2. a proof relies on retention tuning or rebuild execution not accepted in this checkpoint +3. delivery wording collapses bounded failure on unrecoverable gaps into full `NeedsRebuild` lifecycle closure + +##### Suggested first cut + +1. restate the reconnect decision matrix explicitly against current code: + - no prior durable progress -> bootstrap + - prior durable progress + recoverable gap -> handshake + catch-up + - prior durable progress + unrecoverable gap -> fail closed here, rebuild later +2. close the failing adversarial reconnect tests first +3. then tighten one safety proof that catch-up replays missing WAL without overwriting newer replica data +4. keep full rebuild-state closure and retention-policy broadening out of this checkpoint + +##### Assignment For `sw` + +1. Goal + - deliver bounded reconnect handshake + WAL catch-up truth on the production replication path +2. Required outputs + - one explicit reconnect/catch-up contract summary + - one focused code/test package centered on: + - reconnect discriminator + - retained-WAL catch-up replay + - repeated disconnect/reconnect recovery + - bounded failure on unrecoverable gaps + - one delivery note explaining: + - files updated in place + - which failing baseline tests are now closed by `CP13-5` + - which `PASS` / `PASS*` baseline tests are promoted into real `CP13-5` evidence + - what later checkpoints remain untouched +3. Hard rules + - do not broaden into replica-aware retention policy + - do not claim full `NeedsRebuild` lifecycle closure + - do not use “final SyncCache passed” as the only reconnect proof + +##### Assignment For `tester` + +1. Goal + - validate that `CP13-5` closes reconnect handshake / WAL catch-up and nothing broader +2. Validate + - previously-synced replicas reconnect through resume truth rather than bootstrap + - recoverable retained-WAL gaps replay and re-enter `InSync` + - repeated disconnect/reconnect cycles recover + - unrecoverable gaps fail closed without overclaiming `CP13-7` + - no-overclaim around `CP13-6+` +3. Reject if + - reconnect proof is indirect or only inferred from a final success result + - evidence depends on retention/rebuild logic not actually accepted here + - the delivery claims full rebuild or retention closure + +#### Short judgment + +`CP13-5` is acceptable when: + +1. reconnect path selection is explicit and correct +2. retained-WAL catch-up closes the recoverable-gap path back to `InSync` +3. repeated disconnect/reconnect recovery is bounded and test-backed +4. unrecoverable gaps fail closed without pretending `CP13-7` is done +5. the checkpoint stays clearly separate from `CP13-6+` + +--- + +### `CP13-5` Delivery Pack + +Bounded contract: + +1. `CP13-5` accepts reconnect handshake + WAL catch-up only +2. it does not accept replica-aware WAL retention, full rebuild fallback, or rollout claims + +What `sw` should deliver: + +1. one focused contract review of reconnect discriminator, resume truth, and catch-up admission +2. one bounded code/test package only where current code still fails the reconnect/catch-up contract +3. one delivery note with: + - changed files + - proof shape + - which failing baseline tests are now closed + - which baseline `PASS` / `PASS*` tests are promoted into real `CP13-5` evidence + - no-overclaim statement + +Recommended delivery shape: + +1. contract: + - define bootstrap vs reconnect and recoverable vs unrecoverable gap outcomes +2. code/tests: + - keep updates local to reconnect/catch-up surfaces + - make the handshake/catch-up path directly observable in proofs +3. note: + - distinguish primary proof from support evidence + - explain why `CP13-6+` remains untouched + +Review checklist: + +1. does prior durable progress force reconnect rather than bootstrap? +2. does recoverable gap replay happen before barrier success? +3. do repeated disconnect/reconnect cycles recover cleanly? +4. do unrecoverable gaps fail closed without overclaiming rebuild closure? +5. is the checkpoint still bounded to reconnect/catch-up truth? + +--- + +### `CP13-6` Technical Pack + +Date: 2026-04-03 +Goal: make WAL retention explicit and replica-aware so reclaim preserves the retained-WAL window needed by recoverable replicas while bounded budgets still escalate safely when WAL cannot be held forever + +#### Layer 1: Semantic Core + +##### Problem statement + +`CP13-5` accepted reconnect handshake + catch-up truth for recoverable gaps. +That closes how a recoverable replica returns to `InSync`, but it does not close how long the primary must preserve WAL so that recoverable path remains valid. + +`CP13-1` baseline leaves the retention cluster explicit: + +1. `TestWalRetention_RequiredReplicaBlocksReclaim` already passes and suggests the core hold-back behavior may exist +2. `TestWalRetention_TimeoutTriggersNeedsRebuild` already passes and suggests bounded timeout escalation may exist +3. `TestWalRetention_MaxBytesTriggersNeedsRebuild` is still `PASS*` because the code logs that max-bytes triggering is not fully implemented and the shipper stays degraded + +`CP13-6` therefore accepts only one bounded thing: + +1. replica-aware WAL retention truth and bounded retention-budget escalation + +It does not accept: + +1. full `NeedsRebuild` lifecycle closure +2. rebuild execution/orchestration +3. broad rollout or performance claims + +##### State / contract + +`CP13-6` must make these truths explicit: + +1. recoverable replicas with prior durable progress may hold WAL needed for catch-up +2. retention inputs must be based on replica-aware durable progress, not sender-side guesses or primary-local convenience +3. reclaim may proceed only when a replica no longer needs the retained WAL window or when bounded budgets explicitly escalate the situation +4. timeout and max-bytes budgets must have real fail-closed consequences, not comment-only or log-only placeholders +5. full rebuild closure after escalation remains `CP13-7` + +##### Reject shapes + +Reject before implementation or review if the checkpoint: + +1. reclaims WAL still needed by a recoverable replica +2. treats timeout/max-bytes behavior as accepted without a real state effect +3. mixes retention truth with full rebuild lifecycle closure +4. reopens `CP13-5` reconnect semantics instead of preserving them + +#### Layer 2: Execution Core + +##### Current gap `CP13-6` must close + +1. max-bytes retention behavior is still only partial evidence (`PASS*`), not accepted truth +2. retention budgeting must be reviewed as an explicit contract rather than inferred from scattered passing tests +3. reclaim and escalation rules must remain aligned with the recoverable catch-up contract from `CP13-5` + +##### Suggested file targets + +1. `weed/storage/blockvol/flusher.go` +2. `weed/storage/blockvol/shipper_group.go` +3. `weed/storage/blockvol/wal_shipper.go` only if retention accounting truly depends on replica-state inputs there +4. `weed/storage/blockvol/sync_all_protocol_test.go` +5. nearby WAL-retention helpers and bounded component tests only if needed + +##### Validation focus + +Required proofs: + +1. hold-back proof + - required WAL is not reclaimed while a recoverable replica still needs it +2. timeout-budget proof + - bounded timeout escalation works and releases the WAL hold safely +3. max-bytes-budget proof + - bounded max-bytes escalation has a real state effect and is not just logged +4. compatibility proof + - retention truth remains compatible with `CP13-5` recoverable catch-up +5. boundedness proof + - checkpoint remains about retention truth, not `CP13-7+` + +Reject if: + +1. a passing test still relies on log text rather than observable state for max-bytes behavior +2. retention correctness is inferred only from local flusher progress rather than replica-aware durable progress +3. delivery wording collapses bounded escalation into full rebuild lifecycle closure + +##### Suggested first cut + +1. restate the retention decision matrix explicitly against current code: + - recoverable replica still needs WAL -> hold reclaim + - recoverable replica exceeds timeout budget -> bounded escalation, release hold + - recoverable replica exceeds max-bytes budget -> bounded escalation, release hold +2. convert the existing max-bytes `PASS*` into a real proof first +3. then confirm the already-green hold-back and timeout tests are truly primary proof rather than witness coverage +4. keep full `NeedsRebuild` lifecycle and rebuild execution out of this checkpoint + +##### Assignment For `sw` + +1. Goal + - deliver bounded replica-aware WAL retention truth on the production replication path +2. Required outputs + - one explicit retention contract summary + - one focused code/test package centered on: + - recoverable-replica WAL hold-back + - timeout budget + - max-bytes budget + - compatibility with reconnect/catch-up + - one delivery note explaining: + - files updated in place + - which baseline `PASS` / `PASS*` tests are promoted into real `CP13-6` evidence + - what later checkpoints remain untouched +3. Hard rules + - do not claim full `NeedsRebuild` lifecycle closure + - do not broaden into rebuild execution/orchestration + - do not accept max-bytes behavior based only on log text or comments + +##### Assignment For `tester` + +1. Goal + - validate that `CP13-6` closes replica-aware WAL retention and nothing broader +2. Validate + - recoverable replicas hold WAL as needed for catch-up + - timeout and max-bytes budgets trigger observable bounded escalation + - retention remains aligned with `CP13-5` catch-up truth + - no-overclaim around `CP13-7+` +3. Reject if + - max-bytes proof remains witness-only + - reclaim can still discard catch-up-critical WAL + - the delivery claims rebuild lifecycle closure + +#### Short judgment + +`CP13-6` is acceptable when: + +1. WAL retention is explicitly tied to recoverable replica progress +2. recoverable replicas block reclaim of required WAL +3. timeout and max-bytes budgets trigger real bounded escalation +4. the checkpoint stays clearly separate from `CP13-7+` + +--- + +### `CP13-6` Delivery Pack + +Bounded contract: + +1. `CP13-6` accepts replica-aware WAL retention only +2. it does not accept full rebuild fallback, rebuild execution, or rollout claims + +What `sw` should deliver: + +1. one focused contract review of retention inputs, reclaim hold-back, and bounded budget escalation +2. one bounded code/test package only where current code still fails the retention contract +3. one delivery note with: + - changed files + - proof shape + - which baseline `PASS` / `PASS*` tests are promoted into real `CP13-6` evidence + - no-overclaim statement + +Recommended delivery shape: + +1. contract: + - define who holds WAL, when reclaim may proceed, and what timeout/max-bytes escalation means +2. code/tests: + - keep updates local to retention/reclaim surfaces + - make timeout/max-bytes effects directly observable in proofs +3. note: + - distinguish primary proof from support evidence + - explain why `CP13-7+` remains untouched + +Review checklist: + +1. is retention tied to recoverable replica progress? +2. does reclaim stay blocked while catch-up-critical WAL is still needed? +3. do timeout and max-bytes budgets have real observable effects? +4. does retention remain compatible with `CP13-5` catch-up? +5. is the checkpoint still bounded to retention truth? + +--- + +### `CP13-7` Technical Pack + +Date: 2026-04-03 +Goal: make `NeedsRebuild` a real fail-closed fallback so unrecoverable replicas stop participating in normal replication, expose rebuild intent clearly, and re-enter only through bounded rebuild handoff + +#### Layer 1: Semantic Core + +##### Problem statement + +`CP13-6` accepted replica-aware WAL retention and bounded escalation into `NeedsRebuild`. +That closes when the system decides retained-WAL recovery is no longer valid, but it does not close what the system must do once `NeedsRebuild` is reached. + +The remaining rebuild-fallback cluster is explicit in current evidence: + +1. `TestAdversarial_NeedsRebuildBlocksAllPaths` still fails and is the main open blocker +2. `TestReconnect_GapBeyondRetainedWal_NeedsRebuild` is still only bounded evidence because it proves failure on an unrecoverable gap, not the full `NeedsRebuild` lifecycle +3. existing rebuild tests already suggest strong support evidence for heartbeat visibility, rebuild completion, epoch aborts, and post-rebuild progress initialization + +`CP13-7` therefore accepts only one bounded thing: + +1. `NeedsRebuild` fallback and bounded rebuild handoff truth + +It does not accept: + +1. broad rollout or production-workload validation +2. new protocol discovery outside rebuild fallback +3. broad performance claims + +##### State / contract + +`CP13-7` must make these truths explicit: + +1. unrecoverable replicas transition to `NeedsRebuild`, not indefinite `Degraded` +2. a `NeedsRebuild` shipper is fail-closed: + - normal ship/barrier participation must not continue as if it were recoverable +3. rebuild handoff is explicit: + - visible in heartbeat / state surfaces + - start/abort/complete paths are bounded and epoch-safe +4. post-rebuild progress/state is initialized from checkpoint truth so the replica can re-enter from a bounded baseline rather than stale or zeroed progress +5. full real-workload proof remains `CP13-8` + +##### Reject shapes + +Reject before implementation or review if the checkpoint: + +1. leaves `NeedsRebuild` as a label without blocking normal replication paths +2. proves only “SyncCache failed” without proving actual `NeedsRebuild` state ownership +3. mixes rebuild fallback with real-workload validation or broad rollout claims +4. leaves post-rebuild progress/state ambiguous or stale + +#### Layer 2: Execution Core + +##### Current gap `CP13-7` must close + +1. unrecoverable gap detection exists in parts of the system, but the main adversarial fail-closed lifecycle is not yet closed +2. `TestReconnect_GapBeyondRetainedWal_NeedsRebuild` still needs to be promoted from bounded failure evidence to real rebuild-fallback proof +3. `NeedsRebuild` must be shown to block ship/barrier paths and then hand off cleanly into rebuild/re-entry + +##### Suggested file targets + +1. `weed/storage/blockvol/wal_shipper.go` +2. `weed/storage/blockvol/shipper_group.go` +3. `weed/storage/blockvol/rebuild.go` +4. `weed/storage/blockvol/sync_all_adversarial_test.go` +5. `weed/storage/blockvol/sync_all_protocol_test.go` +6. `weed/storage/blockvol/rebuild_v1_test.go` + +##### Validation focus + +Required proofs: + +1. transition proof + - unrecoverable gap transitions to `NeedsRebuild` +2. fail-closed proof + - `NeedsRebuild` blocks normal ship/barrier participation +3. surface proof + - heartbeat / status surfaces expose `NeedsRebuild` clearly enough for rebuild orchestration +4. rebuild-handoff proof + - rebuild complete / abort behavior is bounded and epoch-safe +5. post-rebuild-progress proof + - rebuilt replica starts from checkpoint-based progress truth +6. boundedness proof + - checkpoint remains about rebuild fallback, not `CP13-8+` + +Reject if: + +1. a test proves only degraded failure without proving `NeedsRebuild` +2. rebuild completion is accepted without checking post-rebuild progress/state initialization +3. delivery wording implies real-workload launch readiness + +##### Suggested first cut + +1. close `TestAdversarial_NeedsRebuildBlocksAllPaths` first +2. promote `TestReconnect_GapBeyondRetainedWal_NeedsRebuild` from bounded failure witness to real transition proof +3. then classify existing rebuild tests into: + - primary proof + - support evidence + - out of scope for `CP13-7` +4. keep `CP13-8` workload claims out of the checkpoint + +##### Assignment For `sw` + +1. Goal + - deliver bounded rebuild fallback truth on the production replication path +2. Required outputs + - one explicit `NeedsRebuild` fallback contract summary + - one focused code/test package centered on: + - unrecoverable-gap transition + - fail-closed blocking semantics + - rebuild start/abort/complete handoff + - post-rebuild progress/state initialization + - one delivery note explaining: + - files updated in place + - which remaining `FAIL` / `PASS*` evidence is now closed by `CP13-7` + - which existing rebuild tests are promoted into real `CP13-7` evidence + - what later checkpoints remain untouched +3. Hard rules + - do not broaden into real-workload benchmarking/validation + - do not claim generic rollout readiness + - do not treat “rebuild exists” as sufficient without fail-closed lifecycle proof + +##### Assignment For `tester` + +1. Goal + - validate that `CP13-7` closes rebuild fallback and nothing broader +2. Validate + - unrecoverable replicas become `NeedsRebuild` + - `NeedsRebuild` blocks normal replication paths + - heartbeat/status surfaces expose rebuild intent + - rebuild handoff is bounded and epoch-safe + - post-rebuild progress/state is initialized correctly + - no-overclaim around `CP13-8+` +3. Reject if + - the main adversarial `NeedsRebuild` fail-closed test is still open + - proof depends on workload/rollout claims outside scope + - post-rebuild progress is still stale, zeroed, or implicit + +#### Short judgment + +`CP13-7` is acceptable when: + +1. unrecoverable gaps transition cleanly to `NeedsRebuild` +2. `NeedsRebuild` is genuinely fail-closed for normal replication paths +3. rebuild handoff/re-entry is bounded and checkpoint-based +4. the checkpoint stays clearly separate from `CP13-8+` + +--- + +### `CP13-7` Delivery Pack + +Bounded contract: + +1. `CP13-7` accepts rebuild fallback only +2. it does not accept workload validation, rollout claims, or broad performance positioning + +What `sw` should deliver: + +1. one focused contract review of `NeedsRebuild` transition, blocking semantics, and rebuild handoff +2. one bounded code/test package only where current code still fails the rebuild-fallback contract +3. one delivery note with: + - changed files + - proof shape + - which remaining `FAIL` / `PASS*` tests are now closed + - which existing rebuild tests are promoted into real `CP13-7` evidence + - no-overclaim statement + +Recommended delivery shape: + +1. contract: + - define when `NeedsRebuild` is entered, what it blocks, and how rebuild hands off back to a bounded re-entry state +2. code/tests: + - keep updates local to rebuild fallback surfaces + - make fail-closed blocking and post-rebuild progress directly observable in proofs +3. note: + - distinguish primary proof from support evidence + - explain why `CP13-8+` remains untouched + +Review checklist: + +1. do unrecoverable gaps really transition to `NeedsRebuild`? +2. does `NeedsRebuild` block normal ship/barrier paths? +3. is rebuild handoff bounded and epoch-safe? +4. is post-rebuild progress initialized from checkpoint truth? +5. is the checkpoint still bounded to rebuild fallback? diff --git a/sw-block/.private/phase/phase-13.md b/sw-block/.private/phase/phase-13.md new file mode 100644 index 000000000..b58b73536 --- /dev/null +++ b/sw-block/.private/phase/phase-13.md @@ -0,0 +1,608 @@ +# Phase 13 + +Date: 2026-04-02 +Status: active +Purpose: carry one explicit engineering gap beyond accepted `Phase 12` hardening into a bounded implementation phase so `RF=2 sync_all` becomes a correct, test-backed replicated durability mode under real reconnect, catch-up, retention, and rebuild conditions + +## Why This Phase Exists + +`Phase 09` accepted chosen-path execution closure. +`Phase 10` accepted bounded control-plane closure. +`Phase 11` accepted bounded product-surface rebinding. +`Phase 12` accepted bounded hardening, diagnosability, and first-launch envelope evidence. + +What still remains is not broad protocol discovery. +It is one concrete engineering problem: + +1. `sync_all` still needs a cleaner replicated-durability contract under cross-machine reconnect and replica recovery reality +2. that contract must be expressed in code and tests so later feature work can reuse it rather than reopen replication semantics repeatedly + +## Phase Goal + +Turn `RF=2 sync_all` from a bounded chosen-path mode with accepted launch-hardening evidence into a correct, reusable replicated-durability model for reconnect, catch-up, retention, and rebuild on real workloads. + +Execution note: + +1. use `phase-13-log.md` as the technical pack for: + - checkpoint breakdown + - acceptance objects + - reject shapes + - assignment text for `sw` and `tester` +2. prefer test-first baseline plus checkpointed implementation +3. keep the goal narrow: replication correctness first, not broad optimization or new transport work + +## Scope + +### In scope + +1. canonical replica address truth +2. authoritative per-replica durable-progress tracking +3. reconnect handshake and WAL catch-up +4. replica-aware WAL retention / truncation +5. rebuild fallback when catch-up is impossible +6. real ext4 / PostgreSQL validation on real block devices for cross-machine `sync_all` +7. mode normalization work that depends directly on the corrected replication model + +### Out of scope + +1. broad new protocol discovery outside the replication path +2. new transport projects such as `SPDK`, `io_uring`, or striped-layout redesign +3. generic benchmark positioning beyond correctness-backed validation +4. unrelated control-plane or product-surface expansion +5. reopening accepted `Phase 09` / `Phase 10` / `Phase 11` / `Phase 12` semantics unless this phase exposes a real bug + +## Phase 13 Items + +### `CP13-1`: Test-First Baseline + +Goal: + +- freeze a failing/passing baseline that exposes the current replication gaps before protocol work begins + +Acceptance object: + +1. the focused sync-replication gap tests exist +2. they are run on current code before major implementation work +3. the fail/pass split is captured explicitly so later checkpoint claims are grounded + +Status: + +- accepted + +Carry-forward: + +1. the baseline report is frozen in `phase-13-cp1-baseline.md` +2. no protocol code was changed in `CP13-1` +3. `CP13-2` and later checkpoints must treat the baseline as the starting truth, not redefine it after implementation + +### `CP13-2`: Canonical Replica Addressing + +Goal: + +- make replica endpoint truth canonical and routable so cross-machine replication never depends on wildcard listener strings, incomplete `:port` forms, or other non-authoritative address leakage + +Acceptance object: + +1. `CP13-2` accepts canonical replica address truth for the replication path +2. it does not accept durable-progress truth, reconnect protocol, WAL retention, or rebuild fallback by implication +3. it does not accept broad networking redesign beyond endpoint canonicalization + +Execution steps: + +1. Step 1: address truth contract freeze + - define the canonical replica endpoint form for replication surfaces as routable `host:port` + - define which forms are invalid for exported/registered truth: + - bare `:port` + - wildcard listener strings such as `[::]:port` + - accidental loopback when cross-machine routing is intended +2. Step 2: implementation hardening + - canonicalize replica listener addresses at the source where receiver/registration surfaces expose them + - keep authoritative endpoint truth aligned across local listener state, registration/heartbeat publication, and any registry copies +3. Step 3: proof package + - prove canonical `host:port` truth is emitted under wildcard-bind cases + - prove no wildcard or incomplete address string leaks into exported replication truth + - prove no-overclaim around reconnect, retention, or rebuild semantics + +Required scope: + +1. replica receiver endpoint truth +2. registration / heartbeat / registry path carrying replica endpoints +3. one focused wildcard-bind proof plus bounded cross-machine truth checks +4. explicit distinction between address canonicalization and later reconnect protocol work + +Must prove: + +1. cross-machine replica addresses exported for replication are canonical routable `host:port` +2. wildcard bind strings do not escape into replication truth +3. local canonicalization does not silently rewrite intentionally loopback-only cases into incorrect external truth +4. acceptance wording stays bounded to endpoint truth rather than later replication recovery semantics + +Reuse discipline: + +1. `weed/storage/blockvol/replica_receiver.go`, `replica_meta.go`, and nearby address helpers may be updated in place as the primary endpoint-truth surface +2. `weed/server/master_block_registry.go` and heartbeat/registration paths may be updated in place only if needed to keep authoritative endpoint truth aligned +3. focused unit/protocol tests should carry the main proof burden; component tests are support-only unless they prove an otherwise unreachable leak +4. no checkpoint work may silently introduce reconnect protocol, retention policy, or rebuild logic + +Verification mechanism: + +1. one focused wildcard-bind canonicalization proof +2. explicit checks that exported/registered replica endpoints are routable `host:port` +3. no-overclaim review so `CP13-2` does not absorb `CP13-3+` + +Hard indicators: + +1. one accepted canonical-endpoint proof: + - wildcard-bind listener state resolves to canonical exported `host:port` +2. one accepted no-leak proof: + - bare `:port` / wildcard listener strings no longer escape into replication truth +3. one accepted boundedness proof: + - `CP13-2` claims endpoint truth only, not reconnect or durability semantics + +Reject if: + +1. the checkpoint fixes only one test string shape but leaves other exported endpoint paths unchanged +2. canonicalization happens only in tests rather than at the production truth surface +3. the checkpoint quietly broadens into reconnect, retention, or rebuild protocol work + +Status: + +- accepted + +Carry-forward: + +1. `localServerID` remains stable control identity and may be opaque +2. `advertisedHost` is now the transport-facing canonicalization input for wildcard-bind replica endpoints +3. `CP13-3` and later checkpoints must not reopen identity-vs-transport separation unless a new concrete bug is exposed + +### `CP13-3`: Durable Progress Truth + +Goal: + +- make durable replication progress explicit and authoritative so sync correctness is grounded in replica flushed durability rather than sender-side send progress or loosely inferred health + +Acceptance object: + +1. `CP13-3` accepts durable progress truth for the replication path +2. it does not accept reconnect/catch-up protocol, retention policy, rebuild fallback, or broader state-machine closure by implication +3. it does not accept generic “tests pass” reasoning without an explicit durable-progress contract review + +Execution steps: + +1. Step 1: durable-progress contract freeze + - define `replicaFlushedLSN` as replica-side WAL durability confirmed at barrier time + - define sender-side shipped/sent progress as diagnostic only, not authority for sync correctness + - define what barrier responses must expose as explicit durable progress truth +2. Step 2: implementation hardening or proof confirmation + - update the durable-progress path only where current code fails to meet the contract + - if current code already satisfies the contract, keep changes minimal and make the proof package explicit instead of broadening scope +3. Step 3: proof package + - prove barrier success is grounded in replica flushed durability + - prove flushed progress is monotonic within epoch and not updated on mere receive + - prove no-overclaim around `CP13-4+` + +Required scope: + +1. replica receiver durable-progress state +2. barrier request/response path +3. sender/group tracking of replica durable progress +4. explicit separation between durable-progress truth and later reconnect / retention semantics + +Must prove: + +1. `replicaFlushedLSN` means replica durability, not sender transmission progress +2. barrier responses expose durable progress explicitly enough for sync correctness decisions +3. sender-side progress such as shipped/sent LSN is diagnostic only and cannot authorize sync success +4. acceptance wording stays bounded to durable-progress truth rather than broader recovery/state-machine closure + +Reuse discipline: + +1. `weed/storage/blockvol/replica_apply.go`, `wal_shipper.go`, `dist_group_commit.go`, and related protocol message code may be updated in place as the primary durable-progress surfaces +2. focused unit/protocol tests should carry the main proof burden +3. `weed/server/*` should remain reference only unless durable-progress truth requires an exposed wiring change +4. no checkpoint work may silently introduce reconnect protocol, retention policy, rebuild policy, or broader transport redesign + +Verification mechanism: + +1. one focused proof set around barrier/flushed progress truth +2. explicit checks that receive progress alone does not advance durable authority +3. no-overclaim review so `CP13-3` does not absorb `CP13-4+` + +Hard indicators: + +1. one accepted barrier-truth proof: + - barrier success is tied to replica flushed durability +2. one accepted monotonicity proof: + - `replicaFlushedLSN` is monotonic within epoch +3. one accepted no-false-authority proof: + - sender-side shipped/sent progress is diagnostic only +4. one accepted boundedness proof: + - `CP13-3` claims durable-progress truth only + +Reject if: + +1. the checkpoint treats passing baseline tests as automatic closure without reviewing the durable-progress contract +2. durable-progress truth is still mixed with sender-side transmission progress +3. the checkpoint quietly broadens into reconnect, retention, rebuild, or general replication redesign + +Status: + +- accepted + +Carry-forward: + +1. `replicaFlushedLSN` is now the authoritative durable-progress variable for `sync_all` +2. legacy `BarrierOK` responses without `FlushedLSN` are rejected and cannot count as durable authority +3. `CP13-4` and later checkpoints must treat sender-side send progress as diagnostic only, not as sync-correctness authority + +### `CP13-4`: Replica State Machine / Barrier Eligibility + +Goal: + +- make replica state and barrier eligibility explicit so only `InSync` replicas can satisfy sync durability while non-eligible states fail closed instead of drifting into accidental success + +Acceptance object: + +1. `CP13-4` accepts the replica state machine and barrier-eligibility contract +2. it does not accept reconnect/catch-up protocol, retention policy, rebuild fallback, or broader rollout claims by implication +3. it does not accept vague “state seems fine” reasoning without an explicit eligibility contract + +Execution steps: + +1. Step 1: state contract freeze + - define the bounded state set used by the replication path: + - `Disconnected` + - `Connecting` + - `CatchingUp` + - `InSync` + - `Degraded` + - `NeedsRebuild` + - define barrier eligibility: + - only `InSync` replicas count toward sync durability + - non-eligible states must pre-reject or fail closed +2. Step 2: implementation hardening or proof confirmation + - update the state/eligibility path only where current code fails the contract + - if current code already satisfies much of the contract, keep code changes minimal and make the proof package explicit +3. Step 3: proof package + - prove barrier rejects replicas not eligible for sync durability + - prove degraded or catching-up replicas do not silently count toward `sync_all` + - prove no-overclaim around `CP13-5+` + +Required scope: + +1. replica shipper state transitions and eligibility checks +2. barrier admission path +3. `sync_all` failure semantics when replicas are non-eligible +4. explicit separation between state eligibility and later reconnect/rebuild protocol work + +Must prove: + +1. only `InSync` replicas count toward sync durability +2. `Disconnected`, `Connecting`, `CatchingUp`, `Degraded`, and `NeedsRebuild` do not silently satisfy barrier eligibility +3. degraded/non-eligible replicas fail closed for `sync_all` rather than producing false durability success +4. acceptance wording stays bounded to state/eligibility truth rather than reconnect, retention, or rebuild closure + +Reuse discipline: + +1. `weed/storage/blockvol/wal_shipper.go`, `dist_group_commit.go`, `shipper_group.go`, and nearby replication coordination code may be updated in place as the primary state/eligibility surfaces +2. focused unit/protocol/adversarial tests should carry the main proof burden +3. `weed/server/*` should remain reference only unless state eligibility requires a surfaced wiring correction +4. no checkpoint work may silently introduce reconnect handshake, retention policy, rebuild flow, or broader transport redesign + +Verification mechanism: + +1. one focused proof set around replica state and barrier eligibility +2. explicit checks that non-`InSync` states cannot satisfy `sync_all` +3. no-overclaim review so `CP13-4` does not absorb `CP13-5+` + +Hard indicators: + +1. one accepted eligibility proof: + - only `InSync` replicas count toward sync durability +2. one accepted fail-closed proof: + - non-eligible replicas cause bounded failure rather than false success +3. one accepted state-boundary proof: + - barrier rejects or excludes disallowed states explicitly +4. one accepted boundedness proof: + - `CP13-4` claims state/eligibility truth only + +Reject if: + +1. the checkpoint treats passing baseline tests as automatic closure without restating the state/eligibility contract +2. non-eligible replica states can still satisfy sync durability +3. the checkpoint quietly broadens into reconnect, retention, rebuild, or general replication redesign + +Status: + +- accepted + +Carry-forward: + +1. the replica state set and barrier-eligibility contract are now explicit +2. only `InSync` may satisfy sync durability; `Disconnected`/`Degraded` may invoke `Barrier()` only as bounded recovery entry paths +3. `CP13-5` and later checkpoints must preserve this eligibility boundary rather than reopening it implicitly + +### `CP13-5`: Reconnect Handshake + WAL Catch-up + +Goal: + +- make reconnect after replica disturbance explicit and correct so a replica with known durable progress can resume from retained WAL, catch up, and re-enter `InSync` without false bootstrap success or barrier hangs + +Acceptance object: + +1. `CP13-5` accepts the reconnect handshake and WAL catch-up contract for recoverable gaps on the replication path +2. it does not accept replica-aware WAL retention policy, full rebuild fallback lifecycle, or broader rollout claims by implication +3. it does not accept vague “reconnect seems to work” reasoning without an explicit resume/catch-up contract + +Execution steps: + +1. Step 1: reconnect contract freeze + - define when a replica must use bootstrap versus reconnect: + - fresh replica with no prior durable progress may bootstrap + - replica with prior flushed progress must reconnect via explicit resume truth + - define reconnect decision outcomes: + - already caught up + - recoverable gap within retained WAL + - unrecoverable gap that must fail closed and defer full rebuild handling to `CP13-7` +2. Step 2: implementation hardening + - update the reconnect path only where current code still fails the resume/catch-up contract + - ensure catch-up replays retained WAL before barrier success is allowed + - ensure repeated disconnect/reconnect cycles remain bounded and do not silently fall back to unsafe bootstrap +3. Step 3: proof package + - prove degraded replicas with prior durable progress use handshake/reconnect rather than bootstrap + - prove retained-WAL catch-up completes and re-enters `InSync` on recoverable gaps + - prove reconnect fails closed on unrecoverable or incomplete recovery cases + - prove no-overclaim around `CP13-6+` + +Required scope: + +1. `wal_shipper` reconnect discriminator and resume handshake +2. retained-WAL catch-up replay path +3. repeated disconnect/reconnect recovery behavior +4. bounded failure semantics for gaps that cannot be recovered within this checkpoint +5. explicit separation between reconnect/catch-up closure and later retention/rebuild policy work + +Must prove: + +1. fresh shippers bootstrap, but previously-synced shippers reconnect using resume truth +2. barrier success after disturbance is allowed only after reconnect/catch-up has re-established `InSync` +3. repeated disconnect/reconnect cycles do not strand the replica in false degraded recovery +4. recoverable gaps replay retained WAL correctly without overwriting newer replica data +5. acceptance wording stays bounded to reconnect/catch-up truth rather than retention or rebuild closure + +Reuse discipline: + +1. `weed/storage/blockvol/wal_shipper.go`, reconnect/catch-up helpers, and nearby replication protocol code may be updated in place as the primary reconnect surface +2. focused protocol/adversarial tests should carry the main proof burden; component tests are support-only unless a protocol gap is otherwise unreachable +3. `weed/server/*` should remain reference only unless reconnect correctness requires surfaced wiring changes +4. no checkpoint work may silently broaden into retention policy, rebuild orchestration, or performance tuning + +Verification mechanism: + +1. one focused proof set around reconnect discriminator, catch-up replay, and post-reconnect barrier behavior +2. explicit checks for repeated disconnect/reconnect recovery +3. explicit checks that recoverable gaps replay retained WAL before sync success +4. no-overclaim review so `CP13-5` does not absorb `CP13-6+` + +Hard indicators: + +1. one accepted reconnect-discriminator proof: + - prior durable progress uses handshake/reconnect rather than bootstrap +2. one accepted catch-up proof: + - recoverable retained-WAL gap replays and returns to `InSync` +3. one accepted repeated-recovery proof: + - multiple disconnect/reconnect cycles recover without hanging or drifting +4. one accepted fail-closed proof: + - reconnect does not falsely succeed when recovery is incomplete or impossible within retained WAL +5. one accepted boundedness proof: + - `CP13-5` claims reconnect/catch-up truth only + +Reject if: + +1. a previously-synced replica can still skip resume truth and succeed via unsafe bootstrap +2. barrier success can occur before reconnect/catch-up has restored `InSync` +3. repeated reconnect cycles still hang, strand, or silently degrade correctness +4. the checkpoint quietly broadens into retention, explicit `NeedsRebuild` lifecycle closure, rebuild execution, or general replication redesign + +Status: + +- accepted + +Carry-forward: + +1. replacement shippers now preserve prior durable-progress intent across `SetReplicaAddrs` +2. previously-synced replicas must reconnect through resume truth and retained-WAL catch-up rather than unsafe bootstrap +3. `CP13-6` and later checkpoints must preserve the reconnect/catch-up contract rather than weakening it through reclaim or rebuild shortcuts + +### `CP13-6`: Replica-Aware WAL Retention + +Goal: + +- make WAL retention explicit and replica-aware so reclaim is gated by recoverable replica progress and bounded retention budgets rather than silently discarding catch-up-critical WAL + +Acceptance object: + +1. `CP13-6` accepts replica-aware WAL retention and retention-budget truth on the replication path +2. it does not accept full rebuild fallback lifecycle, rebuild execution, or broader rollout claims by implication +3. it does not accept vague “reclaim seems safe” reasoning without an explicit retention contract + +Execution steps: + +1. Step 1: retention contract freeze + - define which replica progress is authoritative for WAL retention: + - only replicas with prior durable progress and still recoverable state may hold WAL + - define bounded retention outcomes: + - reclaim blocked while a recoverable replica still needs retained WAL + - timeout / max-bytes budgets may escalate boundedly and release the WAL hold + - full rebuild handling after escalation remains `CP13-7` +2. Step 2: implementation hardening + - update the retention path only where current code still fails the bounded retention contract + - ensure retention decisions use replica-aware progress rather than primary-local heuristics alone + - ensure budget-triggered escalation is explicit and fail-closed rather than silent reclaim +3. Step 3: proof package + - prove recoverable replicas block reclaim of needed WAL + - prove timeout / max-bytes budgets trigger bounded escalation instead of indefinite WAL growth + - prove retention remains aligned with `CP13-5` reconnect/catch-up truth + - prove no-overclaim around `CP13-7+` + +Required scope: + +1. WAL retention/reclaim gates +2. shipper-group retention inputs derived from recoverable replica progress +3. bounded timeout / max-bytes escalation behavior +4. explicit separation between retention truth and full rebuild lifecycle closure + +Must prove: + +1. reclaim does not drop WAL still required by a recoverable replica +2. retention inputs come from replica-aware durable progress, not sender-side guesses +3. timeout / max-bytes budgets trigger bounded escalation when WAL cannot be held indefinitely +4. acceptance wording stays bounded to retention truth rather than full rebuild closure + +Reuse discipline: + +1. `weed/storage/blockvol` WAL-retention, flusher, shipper-group, and adjacent replication coordination code may be updated in place as the primary retention surface +2. focused unit/protocol tests should carry the main proof burden; component tests are support-only unless a retention gap is otherwise unreachable +3. `weed/server/*` should remain reference only unless retention truth requires surfaced reporting changes +4. no checkpoint work may silently broaden into rebuild execution, broad control-plane redesign, or performance tuning + +Verification mechanism: + +1. one focused proof set around retention hold, reclaim gating, and budget-triggered escalation +2. explicit checks that max-bytes and timeout paths are real production behaviors, not just comments/logs +3. explicit checks that retention stays compatible with `CP13-5` recoverable catch-up +4. no-overclaim review so `CP13-6` does not absorb `CP13-7+` + +Hard indicators: + +1. one accepted hold-back proof: + - recoverable replicas block reclaim of required WAL +2. one accepted timeout-budget proof: + - timeout can escalate a stalled recoverable replica into bounded fail-closed behavior +3. one accepted max-bytes-budget proof: + - max-bytes pressure triggers explicit bounded escalation rather than silent reclaim or TODO-only behavior +4. one accepted boundedness proof: + - `CP13-6` claims retention truth only + +Reject if: + +1. reclaim can still silently discard WAL needed for a recoverable replica +2. max-bytes behavior is still only log text / placeholder behavior without real state effect +3. the checkpoint quietly broadens into full `NeedsRebuild` lifecycle closure, rebuild execution, or general replication redesign + +Status: + +- accepted + +Carry-forward: + +1. retention inputs and bounded retention budgets are now replica-aware +2. timeout and max-bytes escalation can move a stalled recoverable replica into `NeedsRebuild` +3. `CP13-7` must turn that escalation into a real fail-closed rebuild lifecycle rather than leaving `NeedsRebuild` as a partially-signaled state + +### `CP13-7`: Rebuild Fallback + +Goal: + +- make `NeedsRebuild` a real fail-closed recovery state so unrecoverable replicas stop participating in normal replication paths, surface rebuild intent clearly, and re-enter the replication contract only through bounded rebuild handoff + +Acceptance object: + +1. `CP13-7` accepts the `NeedsRebuild` fallback and bounded rebuild handoff lifecycle on the replication path +2. it does not accept broad rollout claims or real-workload validation by implication +3. it does not accept vague “rebuild eventually works” reasoning without an explicit fail-closed lifecycle contract + +Execution steps: + +1. Step 1: rebuild-fallback contract freeze + - define what `NeedsRebuild` means: + - unrecoverable via retained WAL catch-up + - excluded from normal ship/barrier success + - visible to rebuild orchestration and observability surfaces + - define lifecycle boundaries: + - detection/escalation into `NeedsRebuild` + - fail-closed behavior while in `NeedsRebuild` + - bounded rebuild handoff and post-rebuild re-entry +2. Step 2: implementation hardening + - update the rebuild-fallback path only where current code still leaves `NeedsRebuild` partial, leaky, or inconsistent + - ensure ship/barrier paths block correctly while `NeedsRebuild` + - ensure successful rebuild resets progress/state in a way compatible with later re-entry +3. Step 3: proof package + - prove unrecoverable gaps transition to `NeedsRebuild` + - prove `NeedsRebuild` blocks normal replication participation + - prove rebuild handoff can re-establish a bounded healthy starting point + - prove no-overclaim around `CP13-8+` + +Required scope: + +1. `NeedsRebuild` detection and state ownership on the primary shipper side +2. fail-closed behavior for ship/barrier and related replication paths while `NeedsRebuild` +3. rebuild start/abort/complete handoff boundaries +4. post-rebuild progress/state initialization needed for safe re-entry +5. explicit separation between rebuild fallback closure and later real-workload validation + +Must prove: + +1. unrecoverable gaps do not remain merely degraded; they transition to `NeedsRebuild` +2. a shipper in `NeedsRebuild` cannot silently participate in ship/barrier success +3. rebuild completion restores a bounded re-entry point without faking immediate `InSync` +4. acceptance wording stays bounded to rebuild fallback truth rather than `CP13-8` rollout/workload claims + +Reuse discipline: + +1. `weed/storage/blockvol` rebuild, shipper-group, wal-shipper, and adjacent replication coordination code may be updated in place as the primary rebuild-fallback surface +2. focused unit/protocol/adversarial tests should carry the main proof burden; component tests are support-only unless a rebuild gap is otherwise unreachable +3. `weed/server/*` should remain reference only unless rebuild fallback requires surfaced status/reporting changes +4. no checkpoint work may silently broaden into real-workload benchmarking, performance tuning, or new protocol discovery + +Verification mechanism: + +1. one focused proof set around `NeedsRebuild` transition, blocking semantics, and rebuild re-entry +2. explicit checks that `NeedsRebuild` blocks normal replication paths rather than merely logging/marking degraded +3. explicit checks that post-rebuild progress initializes from bounded truth such as checkpoint state +4. no-overclaim review so `CP13-7` does not absorb `CP13-8+` + +Hard indicators: + +1. one accepted transition proof: + - unrecoverable retained-WAL gap transitions to `NeedsRebuild` +2. one accepted fail-closed proof: + - `NeedsRebuild` blocks ship/barrier participation +3. one accepted rebuild-handoff proof: + - rebuild start/complete path restores a bounded re-entry state +4. one accepted post-rebuild-progress proof: + - replica progress after rebuild is initialized from checkpoint truth, not stale/zeroed state +5. one accepted boundedness proof: + - `CP13-7` claims rebuild fallback only + +Reject if: + +1. an unrecoverable gap can still linger in `Degraded` without escalating to `NeedsRebuild` +2. a `NeedsRebuild` shipper can still satisfy normal ship/barrier paths +3. rebuild completion jumps directly to misleading healthy semantics without bounded re-entry proof +4. the checkpoint quietly broadens into `CP13-8` real-workload validation or general replication redesign + +Status: + +- active + +### Later checkpoints inside `Phase 13` + +1. `CP13-8`: real-workload validation +2. `CP13-9`: mode normalization + +## Reuse Discipline + +1. `weed/storage/blockvol/*` is the primary implementation surface and may be updated in place +2. focused unit/component/adversarial tests should carry the main proof burden +3. real-node / real-device validation belongs in testrunner or bounded component scenarios, not chat prose +4. `weed/server/*` may be updated only when replication correctness requires registry / assignment / heartbeat truth to change +5. no checkpoint may silently broaden into performance-optimization or broad rollout work + +## Expected Outcome + +If `Phase 13` succeeds: + +1. reconnect / catch-up / rebuild semantics become explicit and test-backed +2. `sync_all` correctness no longer depends on partial or implicit sender-state assumptions +3. later feature work can reuse a clearer replication contract instead of re-deriving durability semantics each time diff --git a/sw-block/design/v2-phase-development-plan.md b/sw-block/design/v2-phase-development-plan.md index 47d42f931..988962c44 100644 --- a/sw-block/design/v2-phase-development-plan.md +++ b/sw-block/design/v2-phase-development-plan.md @@ -301,6 +301,60 @@ After `Phase 12`: 3. run a limited internal pilot with incident-driven hardening 4. perform controlled rollout only after explicit launch-gate review +### Post-`Phase 12`: Productionization Program + +Goal: + +1. turn the accepted `Phase 12` chosen path into a bounded first-launch product envelope without reopening protocol discovery + +Program slices: + +1. Program `P0`: launch-envelope freeze + - freeze the first supported launch envelope from accepted `P1`-`P4` evidence + - lock: + - supported topology / transport matrix + - explicit exclusions + - launch-blocking vs post-launch blockers + - reject if any launch claim outruns the measured matrix or accepted blockers/gates +2. Program `P1`: internal pilot pack + - convert the frozen launch envelope into a limited internal pilot package + - define: + - pilot environment and topology + - preflight checklist + - success criteria + - stop / rollback conditions + - incident intake template tied to accepted diagnosability surfaces + - reject if pilot success depends on tribal knowledge or undefined operator judgment +3. Program `P2`: incident-driven hardening loop + - route pilot findings into explicit buckets: + - config / environment issue + - known exclusion + - true product bug + - keep one bounded incident ledger and one bounded fix queue + - reject if incidents accumulate as vague notes or exclusions are silently redefined +4. Program `P3`: controlled rollout review + - decide whether to: + - stay in pilot + - widen within the same launch envelope + - block expansion + - require explicit mapping from any expansion decision back to: + - accepted `Phase 12` evidence + - pilot outcomes + - incident dispositions + - reject if rollout broadens beyond the named envelope or reuses pilot success as generic production proof + +Cross-cutting rules: + +1. do not invent a `Phase 12 P5`; this is a separate productionization program +2. keep the accepted chosen path fixed unless incidents expose a real bug +3. treat known missing evidence as explicit constraints until cleared, especially: + - failover-under-load performance + - hours/days soak under load + - `RF>2` + - broad transport matrix + - full gRPC-stream integration evidence +4. keep V2 post-`Phase 12` direction aligned with the learn-tree close/gate work so the roadmap does not split into contradictory stories + ## Module Status Map @@ -347,18 +401,26 @@ If the goal is to maximize product completion efficiently, the recommended order 2. keep `Phase 10` closed and do not reopen accepted bounded control-plane closure casually 3. move next to `Phase 11` product surface rebinding 4. then `Phase 12` production hardening +5. then the post-`Phase 12` productionization program: + - freeze launch envelope + - run limited internal pilot + - harden from incidents + - review controlled rollout -The most important near-term engineering weight should now go to `Phase 12`. +The most important near-term engineering weight should now go to: + +1. finishing `Phase 12` +2. then the bounded productionization program immediately after it ## Short Summary The V2 line now has accepted execution closure on one bounded chosen path. -The next development plan should treat later work as control/product completion phases, not more protocol discovery. +The next development plan should treat later work as bounded hardening plus productionization, not more protocol discovery. The main heavy engineering work still ahead is: 1. stronger end-to-end control-plane closure 2. later product-surface rebinding 3. production hardening -4. bounded cleanup of residual operational rough edges without reopening accepted semantics +4. post-`Phase 12` launch-envelope freeze, internal pilot, and controlled rollout review diff --git a/sw-block/design/v2-product-completion-overview.md b/sw-block/design/v2-product-completion-overview.md index 751c2af83..dbb5112b1 100644 --- a/sw-block/design/v2-product-completion-overview.md +++ b/sw-block/design/v2-product-completion-overview.md @@ -228,7 +228,7 @@ Target: Status: -1. next active stage +1. accepted and closed on the bounded chosen path Main work: @@ -237,6 +237,37 @@ Main work: 3. performance floor and cost characterization 4. explicit production blockers / rollout gates +### Stage 6: Post-`Phase 12` Productionization Program + +Target: + +1. turn the accepted `Phase 12` chosen path into a bounded first-launch product envelope without reopening protocol discovery + +Status: + +1. next active stage after `Phase 12` + +Main work: + +1. freeze the first supported launch envelope from accepted `P1`-`P4` evidence +2. define a limited internal pilot package with explicit preflight, success, and stop conditions +3. run incident-driven hardening with explicit classification: + - config / environment issue + - known exclusion + - true product bug +4. perform controlled rollout review only within the named supported envelope + +Rules: + +1. this is not `Phase 12 P5` +2. pilot success is not generic production proof +3. missing evidence remains an explicit launch constraint until cleared, especially: + - failover-under-load performance + - hours/days soak under load + - `RF>2` + - broad transport matrix + - full gRPC-stream integration evidence + ## Completion Gates The most important gates from here are: @@ -251,31 +282,35 @@ The most important gates from here are: - front-end surfaces should only rebind after backend correctness is strong enough 5. production-hardening gate - restart, soak, diagnosis, and repeated disturbance must be acceptable +6. productionization gate + - first launch envelope, pilot discipline, incident routing, and controlled rollout review must be explicit ## Near-Term Planning Guidance If the goal is to maximize product completion efficiently: -1. do not make `CSI`, `NVMe`, or broad snapshot productization the immediate next heavy phase -2. first close production execution gaps in the backend path -3. then strengthen control-plane closure -4. then rebind product surfaces +1. do not reopen accepted execution, control-plane, or product-surface semantics casually +2. finish `Phase 12` hardening cleanly +3. then freeze the first supported launch envelope +4. then run a limited internal pilot +5. then widen only through explicit incident review and rollout-gate review In short: -1. backend truth and execution first -2. product surfaces second -3. production hardening last +1. chosen-path closure first +2. production hardening second +3. bounded productionization third ## Short Summary The V2 line is already beyond "algorithm only". -It has an accepted bounded chosen path through backend, control-plane, and selected product surfaces. +It has an accepted bounded chosen path through backend, control-plane, selected product surfaces, and `Phase 12` hardening. But the remaining work is still substantial, and it is mostly engineering work: -1. production hardening under restart / disturbance -2. long-run stability and diagnosability -3. performance floor and rollout gating +1. freeze the first supported launch envelope from accepted evidence +2. run a limited internal pilot with explicit stop conditions +3. harden from incidents without silently broadening scope +4. perform controlled rollout review inside a bounded launch envelope -That is the practical path from the current candidate-safe engine to a production-ready block product. +That is the practical path from the current production-safe chosen path to a bounded first-launch block product.