From 270615e0058c4eb0528eb0e2f35a15fb4f6373de Mon Sep 17 00:00:00 2001 From: pingqiu Date: Sun, 26 Apr 2026 20:15:44 -0700 Subject: [PATCH] P15 doc cleanup pass 1: G9A placement gate + roadmap entry doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 3 changes for clearer dev roadmap: 1. v3-phase-15-mvp-scope-gates.md — added G9A Placement Controller MVP per architect direction 2026-04-26. Sits between G9 lifecycle and G10 snapshot. P0 priority. Source rationale: production block storage needs V2-like operational ergonomics (operator asks for intent → system computes placement → master mints assignment) but V3 authority discipline must be preserved (no heartbeat-as- authority, no V2 promote/demote). G9A bridges the two: - flat-topology RF placement (NO rack/AZ awareness in P15) - durable desired topology generation - explainable candidate filtering (why selected, why rejected) - replacement-on-drain/disk-loss - master mints ONLY from desired topology Explicit non-scope (defer to G20 / P16): rack-aware, hot rebalance, automatic load movement, multi-master HA, V2 promote/demote. Updated P0 table, dependency graph §4.5, closure rule §5 #13. 2. v3-dev-roadmap.md (NEW) — 1-page entry point for "where are we, what's next." Lists 22 P15 gates with status emoji, current batch state, naming decoder, source-of-truth pointers, recently closed batches, prediction for after-G5. QA owns; updates at every gate-close. 3. v3-phase-development-model.md — added §0 header note clarifying this is methodology-only, NOT current state. Points to v3-dev-roadmap.md as current-state entry. Methodology sections (§1-§6, §8-§14) remain canonical. Doc layer architecture now: Methodology: v3-phase-development-model.md (stable) Roadmap: v3-dev-roadmap.md (entry point; updated per gate-close) Canonical: v3-phase-15-mvp-scope-gates.md (22 gates + closure) Rationale: v3-product-placement-authority-rationale.md (why G9A) Co-Authored-By: Claude Opus 4.7 (1M context) --- sw-block/design/v3-dev-roadmap.md | 159 ++ .../design/v3-phase-15-mvp-scope-gates.md | 892 ++++++++++ sw-block/design/v3-phase-development-model.md | 1573 +++++++++++++++++ 3 files changed, 2624 insertions(+) create mode 100644 sw-block/design/v3-dev-roadmap.md create mode 100644 sw-block/design/v3-phase-15-mvp-scope-gates.md create mode 100644 sw-block/design/v3-phase-development-model.md diff --git a/sw-block/design/v3-dev-roadmap.md b/sw-block/design/v3-dev-roadmap.md new file mode 100644 index 000000000..96fb4e513 --- /dev/null +++ b/sw-block/design/v3-dev-roadmap.md @@ -0,0 +1,159 @@ +# V3 Dev Roadmap (Entry Point) + +**Date**: 2026-04-26 (kept current; update on every gate close) +**Status**: ACTIVE — single navigation point for "where are we, what's next" +**Purpose**: 1-page roadmap for new readers + returning collaborators. Points to canonical docs for detail. + +--- + +## 1. The 3 doc layers + +| Layer | Doc | Purpose | +|---|---|---| +| **Methodology** | [`v3-phase-development-model.md`](./v3-phase-development-model.md) | How V3 phases work (closed-loop discipline, target/proof/closure). Apr 2026 draft, methodology-stable. | +| **Roadmap (this doc)** | `v3-dev-roadmap.md` | What's done, what's next. Entry point. | +| **Canonical scope** | [`v3-phase-15-mvp-scope-gates.md`](./v3-phase-15-mvp-scope-gates.md) | THE 22 gates of P15 + dependency graph + closure rules + P0/P1/P2 priority | + +If you only read one doc beyond this one, read `v3-phase-15-mvp-scope-gates.md`. + +--- + +## 2. Current state + +| Phase | State | +|---|---| +| **P14** | ✅ CLOSED (authority + topology layer) | +| **P15** | 🟡 IN PROGRESS — closing gate **G5** of 22 | +| **P16** | ⏳ Not designed yet — only hinted as "in-place V2→V3 migration" | + +P15 = the MVP. 22 gates G0–G22. Currently at gate 5/22. Long way to G22 final cluster validation. + +--- + +## 3. P15 gates — visual progress + +``` +G0 Product hosting ✅ closed (T0 implementation) +G1 Master-volume RPC ✅ closed (T0/T1 implementation) +G2 Frontend smoke ✅ closed (T1/T2) +G3 Real frontends iSCSI+NVMe ✅ closed (T2) +G4 Local durable data path ✅ closed (T3) +G5 Replicated write path 🟡 closing (T4 done; G5-1..6 close work; G5-4 just closed) +G6 Incremental WAL catch-up ⏳ next (T4c muscle done, mostly verification) +G7 Rebuild / replica re-creation ⏳ (T4d-4 part B/C scaffolding done) +G8 Failover data continuity ⏳ +G9 Volume lifecycle ⏳ +G9A Placement Controller MVP ⏳ NEW (added 2026-04-26 per architect; flat-topology only, no rebalance) +G10 Snapshot ⏳ +G11 Resize ⏳ +G12 Disk failure handling ⏳ +G13 Node lifecycle ⏳ +G14 External API ⏳ +G15 CSI lifecycle ⏳ +G16 Security / Auth ⏳ +G17 Diagnostics / Metrics / Logs ⏳ +G18 Configuration / Deployment ⏳ +G19 Migration / Coexistence ⏳ (V2→V3 in-place is P16+) +G20 QoS / rack / operator / GC ⏳ (P2 — defer-allowed) +G21 Performance SLO ⏳ +G22 Final cluster validation ⏳ (release gate; P15 closes here) +``` + +--- + +## 4. Where we actually are right now (granular) + +| Item | State | +|---|---| +| T4d (replicated write path implementation) | ✅ CLOSED at `seaweedfs@2ee12b2c1` (closure report) | +| G5 collective close (production-readiness) | 🟡 IN PROGRESS — 6 sub-batches: | +| ↳ G5-1 multi-replica QA scenarios | 🟡 2 landed; rest on hold | +| ↳ G5-2 walstore cadence smoke | ⏳ sw cleared, not started | +| ↳ G5-3 metrics/backpressure | ⏳ sw cleared, not started | +| ↳ G5-4 binary T4 wiring | ✅ **CLOSED** at `seaweedfs@daafc8e25` (mini-plan v0.5) | +| ↳ G5-5 m01 hardware first-light | ▶️ unblocked by G5-4; QA + sw next | +| ↳ G5-6 G5-DECISION-001 + closure report | ⏳ at G5 collective close | + +--- + +## 5. Naming convention (decoder) + +| Term | Meaning | +|---|---| +| **P** | Phase. P14, P15, P16. P15 = current MVP. | +| **G** | Gate. 22 gates within P15. Product-level milestones. | +| **T** | Implementation track. T0–T4 historical (each closed one or more gates). T5+ may appear for gates that need substantial new code. | +| **G_x_-N** | Sub-batch within a gate-close (e.g., G5-1..G5-6). Used when most implementation is done; close work is verification + missing pieces. | + +**T-tracks vs G-N batches**: contextual choice per gate. T-tracks for substantial new implementation; G-N for gate-close verification when implementation is mostly done. Architect picks at each kickoff. + +--- + +## 6. Source of truth pointers + +For any specific question, go to the canonical doc: + +| Question | Doc | +|---|---| +| What does P15 promise? | `v3-phase-15-mvp-scope-gates.md` (22 gates + closure rules) | +| What's the product goal + execution model? | `v3-phase-15-product-plan.md` | +| Why pre-declared topology + authority discipline (not V2-style heartbeat-as-authority)? | `v3-product-placement-authority-rationale.md` | +| What's the methodology / closed-loop discipline? | `v3-phase-development-model.md` | +| What invariants does V3 promise + which tests prove them? | `v3-invariant-ledger.md` | +| What's the V2→V3 contract bridge state? | `v2-v3-contract-bridge-catalogue.md` | +| What's the QA system (G-1, G-2, G-3 gates, kickoff/mini-plan/G-1 cadence)? | `v3-phase-15-qa-system.md` + `v3-quality-system.md` | + +--- + +## 7. Recently closed batches (for context window) + +| Batch | Closed | Commit | Highlights | +|---|---|---|---| +| BlockStore walHead hotfix | 2026-04-25 | `seaweed_block@f6084ee` | substrate-internal one-liner; closes round-43 storage-contract violation | +| T4d-1 typed RecoveryFailureKind | 2026-04-25 | `1edeb36` + `d6b1890` | 2-enum split; engine zero-imports storage | +| T4d-2 lane-pure apply gate | 2026-04-25 | `bd2de99` + `01f4ab9` + `a63ae9b` | round-43/44 architectural locks; per-LBA stale-skip | +| T4d-3 R+1 + boundary fences | 2026-04-25 | `44c60dd` | engine emit R+1; CARRY-T4D-LANE-CONTEXT-001 Option B | +| T4d-4 part A RecoveryMode + Stop | 2026-04-25 | `f88d097` | substrate method; BUG-005 non-repeat | +| T4d-4 part B engine wiring + round-47 | 2026-04-25 | `812d3fa` | WithEngineDrivenRecovery REAL; rebuild engine-driven | +| T4d-4 part C full L2 + 2 bug fixes | 2026-04-25 | `e642ae8` | 2 engine bugs surfaced + fixed during HARD GATE #3 | +| T4d closure report | 2026-04-25 | `seaweedfs@2ee12b2c1` | T4 batch series CLOSED | +| T2A NVMe race fix | 2026-04-25 | `seaweed_block@a0be6d5` | atomic.Pointer test fixture; -race ×50 PASS | +| G5-4 binary T4 wiring | 2026-04-26 | `seaweed_block@c820e17` + ledger `seaweedfs@36ba7b44e` + close-lock `daafc8e25` | binary now wires T4 stack; criteria 3+4 relocated to G5-5 | + +--- + +## 8. After G5 closes + +Per `v3-phase-15-mvp-scope-gates.md` §4.5 dependency graph: +- G6 Incremental Catch-Up (T4c muscle done; mostly verification) +- G7 Rebuild (T4d-4 part B/C scaffolding done) +- G8 Failover continuity (substantial new work likely) +- G9 Volume lifecycle (new code) +- **G9A Placement Controller MVP** (new addition 2026-04-26 — flat-topology only, sits between G9 and G10) +- G10–G22 (mix of new code + verification + integration) + +P15 closes at **G22 final cluster validation**. After P15 → P16 (in-place migration is the only hinted scope). + +--- + +## 9. Doc hygiene — outdated/historical docs + +The `sw-block/design/` directory has accumulated 60+ `v3-phase-15-*` docs. Most are historical artifacts from closed T-tracks. To find current work: + +- **Active**: any doc dated 2026-04-25 or later, OR referenced from this roadmap +- **Historical**: T0/T1/T2/T3 assignments + sketches + closure reports — kept for reference, not part of active work + +For a full doc archive plan see §10 of this roadmap (when committed); meanwhile, use this roadmap's pointers + commit dates to navigate. + +--- + +## 10. Update protocol + +This doc gets updated at every gate-close: +1. Move closed gate from "🟡 closing" → "✅ closed" in §3 +2. Move next gate to "🟡 closing" or "▶️ next" +3. Update §4 "where we actually are" with the new active batch +4. Append the gate-close commit to §7 recent-closed table +5. Re-check §8 "after G_x_ closes" prediction against actual scope + +QA owns this doc; sw + architect review at each gate-close report sign. diff --git a/sw-block/design/v3-phase-15-mvp-scope-gates.md b/sw-block/design/v3-phase-15-mvp-scope-gates.md new file mode 100644 index 000000000..85e58bd36 --- /dev/null +++ b/sw-block/design/v3-phase-15-mvp-scope-gates.md @@ -0,0 +1,892 @@ +# V3 Phase 15 Product Gates Canonical + +Date: 2026-04-20 +Status: canonical acceptance gate candidate - ready to drive P15 product-plan refactor +Purpose: define the full single-active-master V3 production MVP scope with pass/fail gates, V2 port guidance, explicit deferrals, and no silent protocol-only closure + +## 1. Rule + +`P15` is not closed by an API shell, a CSI skeleton, a memback frontend, or local unit tests. + +`P15` closes only when the accepted single-active-master MVP can run as a product: + +1. real product-process hosting (pinned P15 beta default: `cmd/blockmaster` / `cmd/blockvolume` transitional V3 daemons; future track may fold into `weed master` / `weed volume` — see §3 G0 "Product host shape") +2. real master-volume control RPC +3. real frontend/client I/O +4. real local durable data path +5. real replicated write path +6. bounded catch-up and rebuild +7. failover with data continuity +8. lifecycle, diagnostics, security, deployment, and cluster validation + +Every item below has an explicit **Pass / Fail** gate. If a gate is deferred, the document must say **Deferred** and name the user-visible consequence. + +## 2. Baseline Assumptions + +### A1. Master Availability + +Decision required before P15 implementation continues: + +1. **Option A - inherit SeaweedFS master HA / Raft hosting**: V3 authority is hosted under the existing master HA envelope, but still has exactly one active authority owner for a volume at a time. +2. **Option B - single-node master for beta**: V3 beta is single master only; master outage pauses control-plane changes, while already-attached frontend behavior is whatever the data path can safely support. + +**Pass gate**: `v3-phase-15-product-plan.md` names one option and every cluster test uses the same assumption. + +**Fail gate**: docs or tests use "single-active-master" sometimes to mean "one primary per volume" and sometimes to mean "one master process" without clarification. + +### A2. Multi-Master + +Multi-master leader election, distributed authority store, and split-brain prevention across multiple active masters are out of P15 scope. + +**Pass gate**: no P15 track claims multi-master HA. + +**Fail gate**: any P15 test or doc implies distributed authority safety without a dedicated phase. + +### A3. V2 Port Rule + +Port V2 product muscle aggressively, but do not port V2 authority semantics. + +**Port**: + +1. frontend protocols +2. local storage / WAL / rebuild machinery +3. CSI / API / operator shape +4. testrunner / scenarios / reports +5. monitoring and deployment assets + +**Do not port**: + +1. `promotion.go` +2. `HandleAssignment` +3. `promote` / `demote` +4. heartbeat-as-authority +5. local-role-as-authority +6. volume-local assignment mutation + +**Pass gate**: each P15 track names V2 files to port and V2 semantics rejected. + +**Fail gate**: "ported from V2" is used without naming the rejected authority paths. + +## 2.5 Relation To QA System + +This document defines **what must pass**. The QA system defines **how each claim is tracked and audited**. + +Every gate below must create or update: + +1. `v3-invariant-ledger.md` rows for durable claims (`INV-*`, `EXT-*`, `PCDD-*`, or `V2SCN-*`) +2. `v3-test-matrix.md` coverage rows for the same IDs +3. `v3-phase-15-qa-system.md` gate status when the gate enters implementation +4. a Protocol Evolution Proposal if the gate changes an accepted semantic invariant + +Minimum gate-to-QA rule: + +1. every P0 gate needs at least one matrix row with `Component` and `Scenario` coverage planned +2. every safety or data-continuity gate needs a ledger row before implementation starts +3. every deferred gate needs a matrix row marked `DEFERRED` plus customer implication +4. every Final Gate claim must be traceable to a ledger row and a concrete scenario artifact + +If a gate cannot be represented in the ledger/matrix, the gate definition is incomplete. + +## 3. P15 Hard Gates + +### G0. Master / VolumeServer Product Hosting + +**Target**: P14's `ObservationHost`, `TopologyController`, `Publisher`, durable authority store, and evidence surfaces run inside real product processes, not only `cmd/sparrow` tests. + +**Product host shape (round-2 T0-sketch sync)**: P15 beta product host may be EITHER: + +1. Direct integration of the P14 authority components into `weed master` / `weed volume`, OR +2. Transitional V3-native product daemons `cmd/blockmaster` / `cmd/blockvolume` in the `seaweed-block` module, provided they meet the product-daemon contract (long-lived, operator-managed, stable advertised flags, graceful shutdown, no hidden smoke/test fallbacks, structured logs). + +**Option 2 is the P15 beta default**; Option 1 is a later integration track. Both satisfy G0's pass gate — the gate cares about real daemons, not which source tree they live in. + +**Must ship** (whichever option chosen): + +1. Product master process hosts durable authority and controller. +2. Product volume process reports local replica facts. +3. Startup order is explicit: durable authority reload before publication. +4. Process restart preserves authority and supportability semantics. +5. Explicit decision on `cmd/sparrow` after P15: dev tool, conformance runner, or retired. + +**Port from V2**: + +1. `weed/server/volume_server_block.go` +2. `weed/server/volume_grpc_block.go` +3. `weed/server/master_grpc_server_block.go` +4. `weed/server/block_heartbeat_loop.go` +5. lifecycle/wiring patterns from `weed/server/volume_server_block.go` + +**Make new**: + +1. V3 hosting adapters from product process to `core/authority`. +2. product config for authority store path and topology input. +3. process lifecycle tests using real `weed` / `sparrow` binaries as appropriate. + +**Pass gate**: L2 process test starts product master + product volume process, reloads durable authority, ingests volume facts, and the volume's adapter projection reflects the assigned lineage (`Epoch` / `EndpointVersion` advance to published values) — without `StaticDirective` or direct `AssignmentInfo`. **Note**: G0 pass gate does NOT require `ModeHealthy`; Mode / fence / probe / session evidence belongs to G3 / G4+ (see T0 sketch §2.1). + +**Fail gate**: route still depends on `cmd/sparrow` hidden smoke mode or in-process test helper as the only non-unit evidence. + +### G1. Master-Volume Control RPC + +**Target**: master and volume servers communicate over a real control route. + +**Must ship**: + +1. heartbeat / observation report route +2. assignment publication / subscription route +3. status / evidence query route +4. backpressure / timeout behavior for dead peers + +**Port from V2**: + +1. `weed/server/master_grpc_server_block.go` +2. `weed/server/volume_grpc_block.go` +3. `weed/server/master_server_handlers_block.go` +4. `weed/storage/blockvol/block_heartbeat.go` +5. `weed/storage/blockvol/block_heartbeat_proto.go` + +**Make new**: + +1. V3-safe RPC messages that carry observation and product status, not authority mutations. +2. assignment subscription route backed by `Publisher`, not by volume-local state. + +**Pass gate**: L2 test sends heartbeat over RPC, master synthesizes `ClusterSnapshot`, publisher mints assignment, volume process receives assignment through subscription, and stale RPC messages are rejected or ignored. + +**Fail gate**: tests still mutate authority directly or stuff `AssignmentInfo`. + +### G2. Frontend Contract Smoke + +**Target**: first frontend backend contract proves lineage gating and stale-primary rejection. Memback is allowed only here. + +**Must ship**: + +1. frontend backend interface +2. readiness from V3 adapter projection +3. stale primary read/write rejection +4. real-process heartbeat ingress for smoke + +**Port from V2**: + +1. frontend adapter shape from `weed/storage/blockvol/adapter.go` +2. test patterns from `weed/storage/blockvol/test/component/publish_healthy_test.go` + +**Make new**: + +1. `core/frontend` contract +2. lineage-gated memback provider +3. L1/L2 route smoke + +**Pass gate**: old backend returns `ErrStalePrimary` after authority moves; new backend can write/read its own post-failover bytes. + +**Fail gate**: claims replicated data continuity from memback. + +### G3. First Real Frontends (iSCSI **and** NVMe/TCP — both required per CEO pin 2026-04-21) + +**Target**: **both** iSCSI and NVMe/TCP are usable real client-facing block protocols served from product host. Single-protocol delivery does **not** satisfy G3 in P15. + +**Scope change history**: G3 originally read as "at least one real frontend (choice of iSCSI, NVMe/TCP, or approved minimal)". Effective 2026-04-21 (CEO pin, captured in `v3-phase-15-t2-sketch.md` §2), G3 is widened to mandate **both** iSCSI and NVMe/TCP as P15 product requirement. The minimal block API fallback is retired as a G3 option; CSI alone still does not satisfy G3. + +**Execution structure** (reflected in T2 assignment): + +1. **T2A** — iSCSI path to L0/L1/L2 green +2. **T2B** — NVMe/TCP path to L0/L1/L2 green +3. **T2 closure** — requires both T2A and T2B complete; partial (one-protocol-only) closure is not permitted + +T2A and T2B may be implemented in separate commits, but the G3 gate does not pass until both protocol paths have L2 product-process evidence. + +**Port from V2** (both required): + +1. `weed/storage/blockvol/iscsi/` (pdu / params / login / session / scsi / dataio / target) +2. `weed/storage/blockvol/nvme/` (protocol / wire / fabric / admin / controller / io / server / adapter / write_retry) +3. Frontend tests from `weed/storage/blockvol/test/` + +**Make new** (both protocols): + +1. V3 projection-to-frontend readiness bridge **shared** between iSCSI and NVMe/TCP (must not duplicate readiness logic per protocol) +2. V3 stale-primary fence at protocol boundary for each protocol +3. Protocol-specific integration tests against V3 authority, including per-protocol stale-path rejection + +**Stale-path rule (applies to both protocols, both directions)**: + +After an authority move (RefreshEndpoint EV advance is acceptable; cross-replica failover is G8/T6), the old frontend path must reject **both directions**: +- **Stale WRITE**: must not ACK success to the initiator. Protocol-appropriate failure (SCSI CHECK CONDITION / NVMe status) required. +- **Stale READ**: must not return data success. Stale read can silently poison upper layers (especially after catch-up/rebuild lands in later tracks), so "write-only fence" is insufficient. + +Both protocols must have L0 + L1 + L2 evidence for **both stale WRITE and stale READ** rejection. + +**Pass gate** (all required; single protocol or write-only insufficient): + +1. **iSCSI stale-path proven**: + a. L2 product-process test attaches through iSCSI, writes+reads (round-trip), triggers authority move, asserts old iSCSI **WRITE** rejects + b. Same L2 (or companion L2) asserts old iSCSI **READ** rejects after the same move +2. **NVMe/TCP stale-path proven**: same two-direction shape (a) + (b) as above +3. V2 port audit covers both protocol trees with per-function verdict +4. Ledger rows `INV-FRONTEND-ISCSI-001` + `INV-FRONTEND-NVME-001` both ACTIVE at Integration layer, and `INV-FRONTEND-PROTOCOL-002` (stale-path fail-closed) ACTIVE at Integration for **both** directions on **both** protocols (4 scenario intersections) +5. CSI presence does not count toward G3 (belongs to G9 / T7) +6. L2 evidence conforms to the initiator-evidence rules below + +**Fail gate** (any one fails the gate): + +1. Only one protocol has L2 evidence; the other is Unit/Component-only or deferred +2. Stale WRITE proven but stale READ not proven (or vice versa) on either protocol — "write-only fence" is insufficient +3. "Frontend ready" means only internal Go interface or memback contract smoke +4. Minimal block API is used as an escape hatch in place of iSCSI or NVMe/TCP +5. Readiness bridge is duplicated across iSCSI and NVMe/TCP (indicates both protocols re-deriving readiness instead of consuming T1 contract) +6. Either protocol's stale-WRITE or stale-READ on L2 is silently acknowledged as success +7. T2 closes using only L2-harness evidence; OS-initiator evidence absent entirely (see initiator-evidence rules) + +**Initiator-evidence rules (what counts as L2)**: + +L2 evidence is tiered. Two classes are distinguished so intermediate states are honest: + +| Evidence class | What it is | Counts for | +|----------------|-----------|------------| +| **L2-OS** | Real OS initiator (Linux `iscsiadm` / `nvme connect`, or equivalent native initiator on a declared target environment) attaches to `cmd/blockvolume` | Required for **T2-CLOSED**. Proves kernel-side behavior: logout/reconnect, device path, timeout handling, protocol error propagation. | +| **L2-harness** | V2 Go initiator harness (or equivalent in-repo client) attaches to real `cmd/blockvolume` process over TCP | Counts for **T2A-/T2B-provisional** intermediate status. Does NOT close T2 alone. Proves protocol wire correctness but not kernel/user-space behavior. | + +**Rule**: T2-CLOSED requires L2-OS evidence for at least the stale-WRITE + stale-READ pass-gate assertions on **both protocols**. L2-harness evidence may substitute for other L2 assertions (e.g., authorities-assign round-trip) but the stale-path critical assertions require L2-OS. + +If the primary CI environment lacks OS initiators, QA may declare a target environment (e.g., a specific Linux test host) and mark CI-blocked L2-OS assertions as environment-gated; "no environment available" is not acceptable for T2-CLOSED. + +### G4. Local Durable Data Path + +**Target**: V3 frontend I/O reaches real local block storage with durable write/recovery semantics. + +**Port from V2**: + +1. `weed/storage/blockvol/logical_storage*.go` +2. `weed/storage/blockvol/smartwal*.go` +3. `weed/storage/blockvol/wal_*.go` +4. `weed/storage/blockvol/dirty_map.go` +5. `weed/storage/blockvol/group_commit.go` +6. `weed/storage/blockvol/write_gate.go` +7. `weed/storage/blockvol/superblock.go` + +**Make new**: + +1. V3 local storage adapter with epoch/session/fence awareness. +2. recovery integration with V3 publication state. + +**Pass gate**: crash/restart test writes acknowledged data through the real frontend, kills/restarts the local process, and reads the acknowledged data back. + +**Fail gate**: only in-memory storage or mock storage backs the frontend. + +### G5. Replicated Write Path + +**Target**: primary writes are replicated to at least one replica with explicit ack/durability semantics. + +**Port from V2**: + +1. `weed/storage/blockvol/wal_shipper.go` +2. `weed/storage/blockvol/shipper_group.go` +3. `weed/storage/blockvol/repl_proto.go` +4. `weed/storage/blockvol/replica_apply.go` +5. `weed/storage/blockvol/replica_barrier.go` +6. V2 replication tests and QA cases + +**Make new**: + +1. V3 lineage-carrying replication route. +2. write acknowledgment policy for beta. +3. stale replica apply rejection tied to V3 epoch / endpointVersion. + +**Pass gate**: L2/L3 test writes through primary, confirms replica has the same committed data, kills primary, and verifies the candidate has the acknowledged write. + +**Fail gate**: primary-only write path is called replicated. + +### G6. Incremental WAL Catch-Up + +**Target**: lagging replicas inside the WAL retention window catch up using WAL delta, not full block shipping. + +**Port from V2**: + +1. WAL retention and shipper mechanics from `wal_shipper.go` +2. `shipper_group.go` +3. component tests such as `fast_rejoin_catchup_test.go` +4. relevant testrunner scenarios around fast reconnect + +**Make new**: + +1. V3 `StartCatchUp` execution that streams only the required WAL delta. +2. achieved-frontier feedback to adapter/engine session close. +3. no rebuild escalation on transient transport loss. + +**Pass gate**: test creates a short lag, restarts replica, verifies catch-up transfers only delta range and reaches byte-for-byte match. + +**Fail gate**: catch-up still ships every primary block while claiming production catch-up. + +### G7. Rebuild / Replica Re-Creation + +**Target**: replicas outside WAL window, lost replicas, and re-created replicas can be rebuilt safely. + +**Must distinguish**: + +1. rebuild from existing stale local data +2. replica re-creation after disk loss or permanent local state loss + +**Port from V2**: + +1. `weed/storage/blockvol/rebuild*.go` +2. `weed/storage/blockvol/rebuild_bitmap.go` +3. `weed/storage/blockvol/rebuild_transport.go` +4. `weed/storage/blockvol/rebuild_session.go` +5. component tests under `weed/storage/blockvol/test/component/rebuild_*` + +**Make new**: + +1. V3 rebuild coordinator that consumes P14 authority but does not own authority. +2. admission control / backoff / bandwidth policy. +3. disk-lost replica re-creation path. + +**Pass gate**: L3 test loses or invalidates a replica, creates/rebuilds a replacement, reaches byte-for-byte match, and records bounded evidence for every step. + +**Fail gate**: rebuild assumes local data still exists when the failure was a disk loss. + +### G8. Failover Data Continuity + +**Target**: after primary failure, the new primary can serve previously acknowledged data and the old primary cannot corrupt future state. + +**Port from V2**: + +1. `ha-io-continuity.yaml` +2. `ha-failover.yaml` +3. `ha-full-lifecycle.yaml` +4. HA component tests under `weed/storage/blockvol/test/` + +**Make new**: + +1. V3 authority-to-frontend failover integration. +2. client reconnect / retry behavior. +3. data verification across failover. + +**Pass gate**: cluster test writes known data, kills primary, waits for V3 reassignment, reattaches or reconnects, and verifies the new primary reads the exact acknowledged data. + +**Ledger anchors**: + +1. `INV-CONT-001` Failover Data Continuity - composite claim +2. `INV-AUTH-001` authority line monotonicity +3. `INV-FENCE-001` stale primary fencing +4. `INV-REPL-001` acknowledged write reaches eligible replica +5. `INV-CATCHUP-001` WAL-window catch-up preserves data +6. `INV-REBUILD-001` rebuild/re-create reaches byte-for-byte match + +**Matrix coverage required**: `INV-CONT-001` must have Scenario and Soak coverage before P15 closes. + +**Fail gate**: failover test only checks authority moved or frontend stale rejection, without data verification. + +### G9. Volume Lifecycle: Create / Delete / Attach / Detach / Publish / Unpublish + +**Target**: users and orchestrators can manage volume lifecycle without manual authority stuffing. + +**Port from V2**: + +1. `weed/storage/blockvol/csi/` +2. `weed/storage/blockvol/blockapi/` +3. `weed/server/master_grpc_server_block.go` +4. V2 lifecycle tests + +**Make new**: + +1. V3 desired-volume model. +2. safe lifecycle bridge into P14 controller. +3. idempotent lifecycle operations. + +**Pass gate**: API or CSI test creates volume, attaches, writes/reads through real frontend, detaches, deletes, and verifies no orphan authority line or data path remains. + +**Fail gate**: lifecycle green status can be returned before data path is usable. + +### G9A. Desired Topology / Placement Controller MVP + +**Target**: users can create and repair replicated volumes without hand-authoring every slot, while authority remains publisher-minted (no V2-style heartbeat-as-authority). + +**Source rationale**: see [`v3-product-placement-authority-rationale.md`](./v3-product-placement-authority-rationale.md). Production block storage requires V2-like operational ergonomics (`operator asks for intent → system computes placement → system drives authority + replication`), but V3's authority discipline must be preserved (no heartbeat-as-authority, no local-role-as-authority). G9A bridges the two: minimal master placement feature on top of V3 desired-topology + authority model. + +**Port from V2** (selectively): + +1. `weed/topology/` placement candidate selection — flat-topology subset only +2. V2 master placement loop — concept, NOT direct code (V3 reframes to desired-topology generation) + +**Make new**: + +1. Desired topology generation: durable, master-owned, versioned (replaces V2's "master invents placement on the fly") +2. Flat-topology RF placement for create-volume requests (no rack/AZ awareness in P15) +3. Eligibility filter: pick nodes/slots from observed capacity + supportability — but observation is *evidence*, not authority +4. Replacement placement on disk-loss / drain +5. Plan/explain output: why selected, why rejected (operator-facing) +6. Master/publisher mints assignment ONLY from desired topology, never directly from heartbeat observation + +**Must ship**: + +1. Flat-topology RF placement for create volume +2. Durable desired topology generation +3. Explainable candidate filtering and rejection reasons +4. Replacement placement for drain/disk-loss workflows +5. Assignment minted only from desired topology, never directly from heartbeat observation + +**Explicit non-scope for P15** (defer to G20 / P16): + +- Rack/AZ-aware placement +- Hot-volume rebalance +- Automatic load-based movement +- Multi-master HA / replicated authority store +- Advanced scheduler scoring +- V2 `promote/demote` semantics + +**Pass gate**: API/CSI or admin test creates RF=2/RF=3 volume from intent, observes desired topology generation, master publishes assignment, volumes bind roles, and no product path stuffs `AssignmentInfo` directly. Replacement-on-drain test passes. + +**Fail gate**: P15 docs imply V2-like dynamic assignment while operators still manually edit topology, OR heartbeat/register directly changes authority, OR `AssignmentInfo` constructed outside `authority.Publisher`. + +### G10. Snapshot + +**Target**: decide whether P15 beta supports volume snapshots. + +**Option A - implement minimal snapshot**: + +1. CSI `VolumeSnapshot` / `VolumeSnapshotClass` if CSI is shipped. +2. point-in-time read-only snapshot with documented consistency boundary. + +**Option B - explicit defer**: + +1. P15 beta does not support native snapshots. +2. users must use external backup or frontend/filesystem-level backup. + +**Port from V2**: + +1. `weed/storage/blockvol/snapshot*.go` +2. `weed/storage/blockvol/snapshot_export*.go` +3. CSI snapshot id helpers and tests + +**Pass gate**: either a snapshot create/restore/read-only validation test passes, or the beta docs explicitly state "no native snapshot" and name the backup implication. + +**Fail gate**: CSI is claimed production-ready while snapshot behavior is silent. + +### G11. Resize / Expand + +**Target**: decide whether P15 beta supports volume expansion. + +**Option A - implement resize**: + +1. CSI `ControllerExpandVolume` +2. CSI `NodeExpandVolume` +3. local storage and frontend capacity update + +**Option B - explicit defer**: + +1. P15 beta volumes are fixed-size. +2. users must create a larger volume and migrate data externally. + +**Port from V2**: + +1. `weed/storage/blockvol/expand_test.go` +2. `weed/storage/blockvol/resize_test.go` +3. CSI resize helpers/tests + +**Pass gate**: either online/offline expansion test passes through selected frontend, or beta docs explicitly mark resize unsupported with user workaround. + +**Fail gate**: PVC resize is silently unsupported. + +### G12. Disk Failure Handling + +**Target**: physical/local storage failure is handled differently from process restart. + +**Must ship or explicitly defer**: + +1. detect bad disk / unreadable extent / corrupt local state +2. mark node/replica unsuitable +3. evict or recreate replica +4. prevent rebuild from assuming local data still exists + +**Port from V2**: + +1. WAL corruption and disk-full tests +2. `fault-disk-full.yaml` +3. rebuild and re-creation test patterns + +**Make new**: + +1. V3 disk-health observation fact. +2. policy path from bad disk to unsupported / recreate. +3. operator evidence. + +**Pass gate**: L3 test simulates disk loss/corruption, V3 stops using the bad replica, creates/rebuilds a replacement, and maintains data correctness if redundancy exists. + +**Fail gate**: disk loss is treated only as volume-server process restart. + +### G13. Node Lifecycle: Join / Drain / Decommission + +**Target**: production operators can add, maintain, and remove nodes. + +**Must ship**: + +1. node join +2. node drain +3. node decommission +4. no new assignment to draining nodes +5. safe migration/rebuild of affected replicas or explicit unsupported state +6. admin CLI/gRPC verbs for drain and decommission; Kubernetes operator coordination is separate and may be deferred under G20. + +**Port from V2**: + +1. operator workflow patterns +2. testrunner action vocabulary +3. master-side placement and evidence code as reference + +**Make new**: + +1. V3 node state model. +2. placement exclusion rules. +3. drain/decommission API and runbook. + +**Pass gate**: test drains a node with hosted replicas, verifies no new assignments land there, data remains available or unsupported evidence is explicit, and decommission removes it from topology. + +**Fail gate**: planned maintenance requires manual authority edits. + +### G14. External API + +**Target**: product verbs are safe and usable by tools/tests without exposing authority internals. + +**Must ship**: + +1. create/delete/get/list volume +2. attach/detach/status +3. read-only topology/authority/convergence +4. safe mutating intent only + +**Port from V2**: + +1. `weed/storage/blockvol/blockapi/` +2. `weed/server/master_grpc_server_block.go` +3. `weed/server/master_server_handlers_block.go` + +**Make new**: + +1. V3-safe product verbs. +2. client library used by tests. + +**Pass gate**: external client drives create -> attach -> write/read -> status -> delete without constructing `AssignmentInfo` or selecting epoch. + +**Fail gate**: API test only validates request/response without route effect. + +### G15. CSI Lifecycle + +**Target**: if CSI is in beta, Kubernetes can create, publish, mount/use, unpublish, and delete V3 volumes. + +**Port from V2**: + +1. `weed/storage/blockvol/csi/` +2. `weed/storage/blockvol/csi/deploy/` +3. `op-csi-lifecycle.yaml` + +**Make new**: + +1. V3 CSI backend. +2. lifecycle mapping to V3 product API. +3. idempotency/retry tests. + +**Pass gate**: K8s or local CSI integration test provisions a volume, mounts/uses it through real frontend, and deletes it cleanly. + +**Fail gate**: CSI reports ready while selected frontend/data path is not usable. + +### G16. Security / Auth + +**Target**: production-facing paths are protected. + +**Must ship**: + +1. authn/authz for mutating APIs +2. TLS/mTLS posture for control endpoints +3. frontend auth where applicable +4. audit records + +**Port from V2**: + +1. `weed/storage/blockvol/iscsi/auth.go` +2. `weed/storage/blockvol/iscsi/auth_test.go` +3. CSI RBAC manifests +4. operator RBAC patterns + +**Explicit defer**: + +1. data-at-rest encryption is not P15 unless product owner changes this gate. +2. beta users needing encryption must use LUKS/dm-crypt or cloud-disk encryption under the block device. + +**Pass gate**: unauthorized clients cannot mutate lifecycle, authority-related desired state, or frontend attachment; authorized clients can, with audit. + +**Fail gate**: unauthenticated mutating endpoint is shipped. + +### G17. Diagnostics, Metrics, Structured Logs, Documentation + +**Target**: humans and machines can understand and monitor the product. + +**Must ship**: + +1. human explainability: topology, authority, convergence, unsupported, stuck, frontend readiness +2. Prometheus metrics endpoint +3. structured log schema +4. user docs: install, configure, create/use volume, failover/recovery, backup limitation, known unsupported features +5. support artifact bundle +6. `/healthz` and `/readyz` or equivalent liveness/readiness endpoints for deployment systems + +**Port from V2**: + +1. `weed/server/master_block_observability.go` +2. `weed/server/master_block_evidence.go` +3. `weed/storage/blockvol/monitoring/` +4. `weed/storage/blockvol/testrunner/metrics.go` +5. `learn/test/evidence-convention.md` + +**Make new**: + +1. V3 status contracts. +2. V3 metric names. +3. V3 structured log fields. +4. V3 beta docs. + +**Pass gate**: every required cluster-validation failure class produces structured evidence, metrics, logs, and a documented operator interpretation. + +**Fail gate**: diagnostics require reading engine internals or grep-only logs. + +### G18. Configuration / Deployment / Upgrade + +**Target**: product is deployable, configurable, restartable, and upgradable. + +**Must ship**: + +1. configuration model: config file / flags / env / K8s ConfigMap rules +2. master address and volume-server registration config +3. replication factor and accepted topology config +4. storage paths and durable store path +5. timeout and retry knobs +6. systemd/container/K8s deployment guide +7. upgrade/rollback checklist + +**Port from V2**: + +1. CSI deploy manifests +2. operator config manifests if operator is accepted +3. testrunner deployment patterns + +**Make new**: + +1. V3 config schema. +2. config validation. +3. deployment examples. + +**Pass gate**: fresh machine or test cluster can be deployed from documented config and passes smoke validation without manual code/test harness steps. + +**Fail gate**: production setup depends on undocumented flags or hidden smoke modes. + +### G19. Migration / Coexistence + +**Target**: existing V2 users know whether and how to move to V3. + +**Must ship or explicitly defer**: + +1. compatibility matrix +2. coexistence isolation rules +3. migration dry-run +4. rollback plan + +**Port from V2**: + +1. `weed/storage/blockvol/v2bridge/` +2. `op-upgrade-rollback.yaml` +3. V2 status and blockapi mapping patterns + +**Pass gate**: the product owner selects one migration story before beta: + +1. **Option A - supported migration**: V3 beta supports a named migration path with dry-run, data compatibility check, and rollback. +2. **Option B - explicit non-migration beta**: V3 beta does not support in-place migration; V2 and V3 are independent clusters; migration is P16+. + +Both options require a public-facing statement. Option A requires an L3/L4 dry-run artifact; Option B requires documentation and release-note wording. + +**Fail gate**: beta users infer in-place upgrade is safe without an explicit path. + +### G20. QoS, Rack Awareness, Operator CRD, GC Policy + +These are not all mandatory for beta, but cannot be silent. + +**QoS / rate limit**: + +1. default P15 beta: deferred unless explicitly accepted. +2. client implication: no per-volume noisy-neighbor isolation. + +**Rack-aware placement**: + +1. default P15 beta: deferred; flat topology only. +2. client implication: do not rely on rack/AZ failure-domain placement. + +**Dedicated K8s operator / CRDs**: + +1. default P15 beta: deferred unless product owner accepts operator scope. +2. client implication: use CSI manifests / Helm/raw manifests if provided. + +**Compaction / GC**: + +1. must choose automatic, manual, or not supported. +2. client implication must be documented because no GC can exhaust disk. + +**Pass gate**: each item is either implemented with a test or explicitly deferred with customer implication. + +**Fail gate**: docs imply production support while behavior is undefined. + +### G21. Performance SLO And Regression + +**Target**: performance is measurable and compared to V2 baseline. + +**Must ship**: + +1. latency SLO target +2. throughput baseline on named hardware +3. failover time SLO +4. catch-up/rebuild time baseline +5. V2 regression comparison where comparable + +**Port from V2**: + +1. `cp103-perf-baseline.yaml` +2. `cp85-perf-baseline.yaml` +3. benchmark scripts under `learn/test` +4. testrunner benchmark actions + +**Make new**: + +1. V3 performance report template. +2. V3 baseline runs. + +**Pass gate**: final gate includes performance report with hardware profile, V2 comparison where available, and explicit pass/fail against SLO. + +**Fail gate**: "perf tested" means an ad hoc benchmark log with no SLO. + +### G22. Final Cluster Validation + +**Target**: independent test agent validates the product as a cluster. + +**Must run**: + +1. multi-process / multi-node bring-up +2. lifecycle create/use/delete +3. heartbeat -> topology -> assignment +4. replicated write/read +5. primary crash / failover / data continuity +6. replica restart / catch-up +7. disk failure / replica re-creation or explicit unsupported +8. rebuild +9. master restart under chosen master availability model +10. node join/drain/decommission +11. unsupported topology +12. security negative tests +13. metrics/log/artifact validation +14. performance and soak + +**Port from V2**: + +1. `weed/storage/blockvol/testrunner/` +2. scenario YAMLs under `weed/storage/blockvol/testrunner/scenarios/` +3. artifact/report machinery +4. `learn/test` evidence convention + +**Make new**: + +1. V3 scenario pack. +2. V3 claim-to-test matrix export. +3. V3 release evidence bundle. + +**Pass gate**: final gate emits `manifest.json`, `result.json`, `result.xml`, logs, metrics, and claim matrix; all P0/P1 gates pass or have product-owner-approved residuals. + +**Fail gate**: P15 closes with only `go test ./...` or local process smoke. + +## 4. P0 / P1 / P2 Classification + +In this document, `P0` means **must decide before beta**. A P0 item may close by implementation or by explicit product-owner-approved deferral only when the gate text allows deferral and names customer implication. + +| Priority | Gate | Default disposition | +|---|---|---| +| P0 | G0 Product hosting | Must implement | +| P0 | G1 Master-volume RPC | Must implement | +| P0 | G3 Real frontends (iSCSI **and** NVMe/TCP) | Must implement **both** per CEO pin 2026-04-21; single-protocol delivery fails the gate | +| P0 | G4 Local durable data path | Must implement | +| P0 | G5 Replicated write path | Must implement | +| P0 | G6 Incremental catch-up | Must implement | +| P0 | G7 Rebuild / replica re-creation | Must implement | +| P0 | G8 Failover data continuity | Must implement | +| P0 | G9 Lifecycle | Must implement core verbs | +| P0 | G9A Placement Controller MVP | Must implement flat-topology placement OR explicitly mark P15 beta as declared-topology/manual-placement only | +| P0 | G10 Snapshot | Implement or explicit beta defer | +| P0 | G11 Resize | Implement or explicit beta defer | +| P0 | G12 Disk failure | Must implement or explicit unsupported evidence path | +| P0 | G13 Node lifecycle | Must implement join/drain/decommission or explicit beta limitation | +| P0 | A1 Master availability | Must decide before P15 implementation continues | +| P1 | G14 External API | Strongly recommended; required if CSI is not complete | +| P1 | G15 CSI | Required if Kubernetes beta is claimed | +| P1 | G16 Security/Auth | Required for non-local beta | +| P1 | G17 Diagnostics/Metrics/Logs/Docs | Required for beta supportability | +| P1 | G18 Configuration/Deployment/Upgrade | Required for beta deployment | +| P1 | G19 Migration/Coexistence | Implement or explicit beta defer | +| P1 | G21 Performance SLO | Required before beta sign-off | +| P2 | G20 QoS / rack-aware / operator / GC | Implement or explicit defer with implication | +| Release | G22 Final cluster validation | Required to close P15 | + +## 4.5 Gate Dependency Sketch + +The default scheduling spine is: + +```text +G0 Product hosting + -> G1 Master-volume RPC + -> G4 Local durable data path + -> G5 Replicated write path + -> G6 Incremental catch-up + -> G7 Rebuild / replica re-creation + -> G8 Failover data continuity + +G2 Frontend contract smoke + -> G3 First real frontends (iSCSI + NVMe/TCP, both required) + -> G9 Lifecycle + -> G9A Placement Controller MVP (depends on G9 lifecycle + G8 failover + G13 node lifecycle observation) + -> G10 Snapshot / G11 Resize / G12 Disk failure / G13 Node lifecycle + -> G14 External API / G15 CSI / G16 Security / G17 Diagnostics + -> G18 Deployment / G19 Migration / G20 Deferred-product decisions / G21 Performance + -> G22 Final cluster validation +``` + +Parallelism is allowed only when the producer/consumer contract between gates is already pinned in writing and the downstream gate does not fake an upstream product capability. + +## 5. Product Closure Rule + +P15 cannot close if any of these are true: + +1. master/volume hosting is still test-only +2. frontend is only memback +3. writes are primary-only but called replicated +4. catch-up still full-ships every block while called incremental +5. rebuild cannot handle disk loss / replica re-creation or explicitly reject it +6. failover test does not verify data continuity +7. CSI/API says ready while data path is not usable +8. snapshot/resize/disk failure/node lifecycle are silent +9. security is absent on non-local mutating endpoints +10. metrics/logs/docs are absent +11. performance has no SLO +12. final validation is only local unit tests +13. **placement is silently manual while docs imply V2-like dynamic assignment** — G9A must either ship flat-topology placement OR P15 beta must be explicitly marked as "declared-topology / manual-placement only" + +## 6. Immediate Action + +Before further T1 coding: + +1. update `v3-phase-15-product-plan.md` to reference this gate map as the canonical P15 closure definition +2. revise `v3-phase-15-t1-assignment.md` so T1 is clearly **contract smoke only**, not product data continuity +3. create `P15 T0 Master / VolumeServer Hosting` assignment +4. require every subsequent assignment to include a `Gx` gate list and V2 port/reject list + +## 7. After P15 Closure + +After G22 passes, ongoing maintenance follows `v3-quality-system.md`: + +1. every release stamps `v3-invariant-ledger.md` +2. every protocol-affecting change uses the PEP process under `protocol-evolutions/` +3. every production bug fix uses the route-closure standard: bad-state family, minimum repro, owner layer, closure point, regression +4. Final Gate safety-invariant scenarios are re-run for data-loss, stale-primary, authority, recovery, and migration-risk fixes +5. annual review refreshes `v3-protocol-truths.md`, `v3-semantic-constraint-checklist.md`, and the active test matrix + +P15 closure is a release gate, not the end of verification discipline. diff --git a/sw-block/design/v3-phase-development-model.md b/sw-block/design/v3-phase-development-model.md new file mode 100644 index 000000000..1ac2d3aa8 --- /dev/null +++ b/sw-block/design/v3-phase-development-model.md @@ -0,0 +1,1573 @@ +# V3 Phase Development Model + +Date: 2026-04-11 (methodology stable; **NOT current-state — see §0 below**) +Status: methodology reference; **for current state see [`v3-dev-roadmap.md`](./v3-dev-roadmap.md)** +Purpose: define how `V3` phases should be run so every phase has a clear target, a closed loop, and a visible path toward production scope + +## §0 Document scope (added 2026-04-26) + +**This doc is the methodology, NOT the current state.** Sections describe the closed-loop discipline, target/proof/closure rules, and suggested phase shapes. Some illustrative examples (especially §7 "Suggested V3 Phases" and §15 "Production Roadmap Layer") were drafted Apr 2026 and may not match current reality. + +**For current state, go to:** + +| Question | Doc | +|---|---| +| What's done, what's next, where are we? | [`v3-dev-roadmap.md`](./v3-dev-roadmap.md) | +| What does P15 promise (22 gates)? | [`v3-phase-15-mvp-scope-gates.md`](./v3-phase-15-mvp-scope-gates.md) | +| Why pre-declared topology + authority discipline? | [`v3-product-placement-authority-rationale.md`](./v3-product-placement-authority-rationale.md) | + +**This doc remains canonical for**: (§1-§6) closed-loop discipline rules; (§8) anti-pattern closure mapping; (§9) phase-gates-must-stay-narrow rule; (§10-§14) repo bootstrap + naming conventions. These are methodology-stable. + +--- + + + +## 1. Short Answer + +Yes, `V3` should use phase-based development. + +In fact, `V3` needs phase discipline even more than `V2`, because: + +1. `V3` is trying to protect semantic cleanliness +2. it is likely to reuse execution muscles that do not fully align +3. it may later support multiple runtimes such as Go and Rust +4. it must not quietly accumulate workaround semantics while "moving fast" + +So the point of phases is not bureaucracy. + +The point is: + +1. each phase must have one clear target +2. each phase must define what it will not do +3. each phase must define what counts as proof +4. each phase must close its loop before the next phase expands scope + +## 2. What A Closed Loop Means In V3 + +For `V3`, a phase is not closed when code exists. + +A phase is closed only when all four are true: + +1. the semantic target is explicit +2. `sw` implemented only that target +3. `tester` validated the intended proof level +4. the result was recorded in a durable phase decision/progress artifact + +That means the loop is: + +```text +target + -> implementation + -> evidence + -> review + -> accepted closure +``` + +If one of these is missing, the phase is not closed. + +## 3. Why V3 Needs Stronger Phase Rules Than V2 + +`V2` had to discover many truths while implementation was already moving. + +`V3` has a different job: + +1. preserve the semantic lessons already learned +2. prevent old route pollution from re-entering +3. convert design package into a portable semantic platform + +That means `V3` phases should be stricter about: + +1. semantic scope +2. non-goals +3. evidence gates +4. reuse boundaries + +## 4. Phase Shape + +Each `V3` phase should contain three documents: + +1. `phase-xx.md` +2. `phase-xx-log.md` +3. `phase-xx-decisions.md` + +This follows the earlier project process, but the content emphasis changes for `V3`. + +### 4.1 `phase-xx.md` + +Use for: + +1. current phase target +2. current scope +3. explicit non-goals +4. accepted guardrails +5. handoff expectations for `sw` and `tester` + +It should stay short. + +### 4.2 `phase-xx-log.md` + +Use for: + +1. design evolution +2. review corrections +3. carry-forward items +4. reuse notes from `V2` +5. why a boundary moved or stayed fixed + +It can be longer. + +### 4.3 `phase-xx-decisions.md` + +Use for: + +1. durable decisions +2. gate decisions +3. accepted compromises +4. explicit "do not re-open without new evidence" items + +This keeps `V3` from re-arguing the same architecture boundary every week. + +## 5. Required Fields For Every V3 Phase + +Every phase should explicitly answer these questions. + +### 5.1 Target + +What single thing is this phase trying to establish? + +Good examples: + +1. mini engine can execute deterministic conformance +2. adapter-backed route can drive one real recovery family path +3. selected `V2` scenarios calibrate correctly under `V3` + +Bad examples: + +1. improve architecture +2. make `V3` better +3. support more stuff + +### 5.2 Scope + +What is included in this phase? + +### 5.3 Non-goals + +What is explicitly excluded in this phase? + +This is critical for `V3`, because otherwise: + +1. storage extensions leak in early +2. scheduler policy leaks into semantic core +3. mismatched `V2` muscles get imported "temporarily" + +### 5.4 Reuse boundary + +What may be reused from `V2`, and in what form? + +Examples: + +1. reference only +2. wrapper allowed +3. copy allowed +4. must rewrite in native `V3` form + +### 5.5 Required proof + +What evidence level is required? + +Examples: + +1. schema validation +2. conformance runner +3. adapter-backed calibration +4. selected runner scenarios + +### 5.6 Reject conditions + +What makes this phase fail, even if code exists? + +Examples: + +1. duplicate semantic route appears +2. projection is reused as control input +3. terminal success has multiple authorities +4. adapter silently decides policy + +### 5.7 Closure Checklist + +Every phase should also answer three closure questions explicitly. + +These are not optional review style points. + +They are part of whether the phase is actually closed. + +#### A. Semantic closure + +Question: + +Has the semantic route evolved far enough for this phase, with the required +constraints complete and no hidden anti-pattern dependence? + +Must check: + +1. the engine / adapter / accepted command boundary now covers the semantic target of this phase +2. stale, reorder, overlap, timeout, cancel, and handoff behavior are fail-closed at the proof level required by this phase +3. target meaning, lineage meaning, and terminal truth authority are still explicit and fixed +4. no new surface quietly turns progress, projection, transport state, or policy hints into semantic truth +5. relevant anti-patterns from `protocol-anti-patterns.md` are either: + - already closed + - explicitly bounded in this phase + - or explicitly deferred to a named later phase + +Short form: + +**the semantic contract needed by this phase is complete enough, and it does not +quietly depend on timing luck or execution residue.** + +#### B. Functional closure + +Question: + +Does the new function introduced in this phase, including any `V2` muscle +migration, work cleanly and honestly at the intended boundary? + +Must check: + +1. new behavior is inside the declared scope of the phase +2. any migrated `V2` execution muscle names what old semantic coupling was removed +3. code path, docs, scope statement, and test/evidence story all describe the same capability +4. integration behavior exists at the intended level, not only isolated local unit success +5. the claimed feature has enough evidence to prevent "implemented but not actually closed" drift + +Short form: + +**new function and migrated execution both land cleanly, and the evidence matches +the real product/runtime claim.** + +#### C. Phase-responsibility closure + +Question: + +Is this work closed in the correct phase, rather than quietly borrowing authority +from a later phase or leaving a current-phase obligation unresolved? + +Must check: + +1. the work belongs to this phase's declared responsibility line +2. this phase does not silently pull policy, topology authority, or product meaning backward from a later phase +3. anything not closed here is explicitly assigned to a later named phase +4. if a task or PR spans more than one responsibility line, the split or the phase definition should be re-reviewed + +Short form: + +**the work is not only correct; it is closed at the correct layer and in the +correct phase.** + +## 6. The V3 Phase Loop + +Each phase should run through the same loop. + +### Step 1: Phase definition + +Owner: + +1. manager / architect + +Output: + +1. target +2. scope +3. non-goals +4. gate + +### Step 2: Expectation definition + +Owner: + +1. tester + +Output: + +1. must-pass expectations +2. failure-class checklist +3. required test level +4. reject conditions + +### Step 3: Implementation + +Owner: + +1. sw + +Output: + +1. code +2. delivery template +3. trace hooks +4. reuse note + +### Step 4: Technical review + +Owner: + +1. architect + +Output: + +1. correctness review +2. semantic-boundary review +3. fail-closed review + +### Step 5: Evidence closure + +Owner: + +1. tester + +Output: + +1. runner/conformance evidence +2. calibration closure +3. reject-or-accept statement + +### Step 6: Durable closure + +Owner: + +1. manager / architect + +Output: + +1. phase decision +2. carry-forward list +3. next-phase handoff + +## 7. Suggested V3 Phases + +The exact numbering can change, but the semantic order should not. + +### Phase 0: Constitution Freeze + +Target: + +1. freeze the initial `V3` semantic package enough to begin implementation + +Close when: + +1. truth domains are accepted +2. mini engine unit is accepted +3. metadata boundary is accepted +4. growth plan and calibration plan exist + +### Phase 1: Initial Core + +Target: + +1. executable deterministic mini engine + +Close when: + +1. schema loads +2. engine applies events deterministically +3. conformance examples run +4. stale rejection and projection derivation are reviewable + +### Phase 2: Reference Runtime + +Target: + +1. a small replay/conformance runtime exists + +Close when: + +1. conformance runner works +2. traces are readable +3. semantic outputs are replayable + +### Phase 3: First Adapter Route + +Target: + +1. one real adapter-backed route exists from runtime facts to session close + +Close when: + +1. no duplicate trigger path exists +2. no duplicate terminal path exists +3. adapter does not silently own policy + +### Phase 4: Runnable Block Sparrow + +Target: + +1. one complete but narrow block slice can run end-to-end through the new core + +Close when: + +1. one basic block route can be started and exercised end-to-end +2. the route still uses one semantic decision path and one terminal-close path +3. the slice is narrow, but it is complete enough to act as the first public runnable check-in +4. phase output can honestly say "this repo runs a basic block slice" + +### Phase 5: Bootstrap And Validation Stabilization + +Target: + +1. the runnable block sparrow becomes easy to start, inspect, and validate repeatedly + +Close when: + +1. the repo has a minimal bootstrap and read-only inspection surface for the runnable slice +2. the repo has a minimal test interface for validating the runnable slice +3. first-time users can start, inspect, and troubleshoot the slice without reading deep design docs +4. progress remains visible and honest in the standalone repo +5. no real operator CLI is introduced before a later single-node demo stage + +### Phase 6: Calibration MVP + +Target: + +1. selected `V2` scenarios are calibrated through the runnable `V3` route + +Close when: + +1. chosen calibration scenarios pass +2. semantic route matches observed runtime route +3. tester closes the calibration evidence + +### Phase 7: Persistent Single-Node Slice + +Target: + +1. admit one persistence-backed single-node block slice cleanly + +Close when: + +1. one persistence-backed single-node slice survives stop/restart with data still present +2. the persistence seam does not pollute the semantic core +3. the result stays honest about proving local persistence rather than broader durability + +### Phase 8: Local Data Process + +Target: + +1. admit one coherent local data process around the accepted persistence seam + +Close when: + +1. local read/write/flush/checkpoint/recover responsibilities are explicit +2. bounded abrupt-termination proof exists for the accepted crash model +3. the local data process does not pollute the semantic core + +### Phase 9: Data Sync Process + +Target: + +1. admit one coherent data-sync institution for catch-up and rebuild data movement + +Close when: + +1. data movement, barrier, and achieved-frontier feedback responsibilities are explicit +2. replication/rebuild data transfer no longer lives as scattered transport muscle +3. the data-sync process does not decide recovery class or terminal truth + +This phase is where the remaining data-plane closure work belongs: + +1. data-sync execution must be one named institution rather than scattered transport helpers +2. byte movement, barrier, achieved-frontier, and restart/retry behavior must be explicit +3. recovery traffic must stay lineage-bound and fail closed under stale or overlapping execution +4. this phase closes data movement correctness, not topology governance + +Practical sequencing rule: + +1. implementation of `Phase 9` and `Phase 10` may overlap +2. declared closure must still treat `Phase 9` as earlier, because `Phase 10` + timeout/cancel/lifecycle claims depend on `Phase 9` wire correctness and + lineage-safe feedback behavior + +### Phase 10: Recovery Execution Process + +Target: + +1. admit one coherent recovery execution institution around engine-issued commands + +Close when: + +1. catch-up and rebuild execution lifecycles are explicit +2. engine-issued targets remain fixed through execution +3. the execution process does not reinterpret command intent or publish terminal semantic truth by itself + +This phase is where the remaining execution-side handoff closure belongs: + +1. command-to-execution-to-close lifecycle must be explicit for catch-up and rebuild +2. old execution must become semantically dead immediately after newer lineage exists +3. delayed callback, retry, timeout, cancellation, and overlap rules must be explicit and tested +4. this phase closes recovery execution semantics, not promotion or failover policy + +### Phase 11: Single-Node Product Surface + +Target: + +1. make the accepted single-node route operable as a bounded product surface + +Close when: + +1. one operator-usable single-node workflow exists +2. observability and safe inspection are strong enough for repeatable local operation +3. the product surface still reads truth from accepted lower institutions rather than inventing a new authority path + +This phase is where bounded single-node governance surfaces belong: + +1. local operator visibility, inspection, and safe action surfaces become explicit +2. the repo should state clearly that current closure is recovery semantics on one node, not cluster topology governance +3. any "promote" or "failover" wording must remain out of scope unless backed by later replicated-phase evidence +4. `Phase 11` surface must not expose cluster-shaped APIs that are likely to require breaking meaning changes when later topology phases land + +### Phase 12: Replicated Durable Slice + +Target: + +1. admit one honest replicated durable slice around the accepted route + +Close when: + +1. the replicated route supports bounded catch-up and rebuild with honest failover/rejoin behavior +2. durability and fencing claims are backed by explicit evidence +3. replicated execution still preserves the accepted semantic ownership split + +This phase is the first phase that should close bounded failover behavior: + +1. reassignment, failover, rejoin, and fencing behavior become explicit replicated claims here +2. the product may claim one bounded failover/rejoin contract only after explicit evidence exists here +3. this phase still consumes topology facts from above; it does not yet own broad topology arbitration policy + +Non-goals that must stay explicit: + +1. `Phase 12` may consume `epoch` from an authority, but it must not mint or + govern `epoch` itself +2. `Phase 12` may use a trivial test/manual authority as a placeholder +3. `Phase 12` does not choose which node becomes the new primary +4. `Phase 12` does not decide what condition should trigger a failover + +Practical bounded-contract checklist: + +1. given `(old primary, new primary, epoch+1)`, rejoin must converge without stale execution pollution +2. acknowledged or rejected stale callbacks during epoch turnover must not contaminate current truth +3. bounded failover/rejoin claims must name what is accepted +4. bounded failover/rejoin claims must also name what remains outside this phase + +### Phase 13: Production Hardening And Release Gate + +Target: + +1. close the mainline product hardening gate for the accepted replicated slice + +Close when: + +1. qualification, observability, and release criteria are explicit +2. the repo can state a bounded production-ready claim honestly +3. the mainline can hand off to later scale and ecosystem phases without reopening core authority boundaries + +This phase is where bounded first-launch governance closure belongs: + +1. bounded failover/publication/runbook criteria should be explicit +2. operational disturbance handling must be clear enough for a bounded production-ready statement +3. this is still not the phase for broad topology expansion or broad placement policy + +### Phase 14: Scale And Topology Expansion + +Target: + +1. widen the accepted product into larger-scale and richer-topology deployment shapes + +Close when: + +1. placement, rebalance, and failover behavior remain correct under the accepted expanded topology set +2. scale/topology work reuses the accepted semantic and execution institutions rather than replacing them +3. the resulting topology claims are backed by explicit evidence + +This phase is where topology governance and richer failover policy belong: + +1. topology authority, placement, rebalance, and failover policy become explicit institutions here +2. eligibility, selection, and topology-wide convergence policy should be closed here rather than hidden inside recovery execution +3. only from this phase onward should the product claim richer topology behavior beyond the bounded replicated slice + +Mainline product meaning: + +1. `Phase 14` is not a side expansion line; it is the mainline stage where fuller topology/governance product closure should land +2. if the repo wants to aim at a real full product rather than a bounded replicated slice, that aim must pass through `Phase 14` + +### Phase 15: Operations And Ecosystem Expansion + +Target: + +1. complete the mainline product loop through operations and ecosystem integration + +Close when: + +1. operator workflows, ecosystem entry points, and lifecycle operations are production-reviewable +2. the ecosystem surface still preserves accepted truth boundaries +3. the mainline product and operations story is complete enough to shift later work into expansion lines + +This phase is where cluster-facing operator policy becomes consumable: + +1. operator workflows for failover, recovery supervision, and lifecycle control become explicit product surfaces here +2. ecosystem entry points may expose topology/failover operations only after the lower topology-policy phases are already closed +3. this phase makes the accepted governance and topology behavior operable; it does not invent new truth + +Mainline product meaning: + +1. `Phase 15` is where the mainline product loop becomes consumable through operations and ecosystem surfaces +2. `Phase 14` + `Phase 15` together should be read as the intended full-product completion target for the mainline + +## 8. Anti-Pattern Closure Mapping + +`V3` phases should explicitly track which anti-patterns are: + +1. already avoided in the semantic core +2. only partially closed +3. deferred until adapter/runtime phases + +Reference: + +1. `protocol-anti-patterns.md` + +### 8.1 A1: Heartbeat Timing Defines Recovery Semantics + +Status: + +1. **not fully closed in Phase 1-2** + +Reason: + +1. the semantic core already decides from bounded facts rather than timer values +2. but the real closure requires a live adapter/runtime path where heartbeat, probe, and refresh can arrive in different timings + +Closure phase: + +1. primary closure target: **Phase 3** +2. calibration proof target: **Phase 6** + +Required proof: + +1. heartbeat timing changes latency only +2. it does not directly choose recovery class +3. selected reconnect/rejoin scenarios prove this through the adapter-backed route + +### 8.2 A4: Event Ordering Determines Semantics + +Status: + +1. **partially closed in Phase 1-2** + +Reason: + +1. the mini engine already rejects stale epoch/endpoint/session and uses bounded facts for decisions +2. but full closure requires proof that runtime/adapter event ordering still converges to the same semantic result + +Closure phase: + +1. semantic-core partial closure: **Phase 1** +2. replay/runtime strengthening: **Phase 2** +3. adapter/runtime closure: **Phase 3** +4. calibration proof: **Phase 6** + +Required proof: + +1. same facts produce the same commands/projection regardless of arrival ordering +2. replay and adapter-backed scenario evidence agree + +### 8.3 A7: Transport Mechanics Leak Into the Semantic Engine + +Status: + +1. **avoided in Phase 1-2 engine/runtime core** +2. **not yet fully proven at integration boundary** + +Reason: + +1. the semantic reducer and reference runtime are intentionally transport-free +2. but real closure requires attaching execution muscles without reintroducing transport state into semantic logic + +Closure phase: + +1. engine/runtime avoidance: **Phase 1-2** +2. integration closure: **Phase 3** +3. calibration proof: **Phase 6** + +Required proof: + +1. transport lifecycle stays in adapter/runtime +2. engine only receives normalized facts +3. no semantic rule depends on TCP/session plumbing details + +### 8.4 Practical Rule + +If a phase touches runtime integration and does not say how it preserves closure +against `A1`, `A4`, and `A7`, then the phase definition is incomplete. + +## 9. Phase Gates Must Stay Narrow + +A common failure is letting one phase carry too much ambition. + +For `V3`, phase gates should be narrow enough that failure is diagnosable. + +Bad phase gate: + +1. "V3 MVP should support failover, rebuild, RF3, SmartWAL, and Rust readiness" + +Good phase gate: + +1. "one adapter-backed route handles short-gap and long-gap recovery with one terminal-close authority" + +Narrow gates are important because they make closure real. + +## 10. Repo Bootstrap After Phase 4 + +`Phase 4` should be treated as the first complete runnable sparrow, but a small +repo bootstrap step should immediately follow its closure. + +Purpose: + +1. turn the first runnable slice into a credible first public check-in +2. freeze public-facing paths early so git history is not dominated by later renames +3. make the repo operable by readers who did not follow the internal design history + +Recommended immediate actions after `Phase 4` closes: + +1. freeze the primary semantic path under `core/` +2. add a minimal runnable entry path +3. add minimal repo status and progress files +4. prepare the bootstrap and validation stabilization work for `Phase 5` + +Practical rule: + +1. `Phase 4` proves the sparrow can run +2. the immediate bootstrap step makes that sparrow visible and usable as the first stable repo shape + +## 11. Testrunner Inside The Phase Model + +The runner should be attached to phases explicitly. + +### Phases 1-2 + +Use: + +1. conformance runner + +### Phases 3-4 + +Use: + +1. calibration scenarios +2. selected `V2` scenario reuse + +### Phases 6-8 + +Use: + +1. broader SW system scenarios +2. chaos / soak / workload validation as appropriate + +This keeps the runner aligned with phase goals instead of becoming a giant undifferentiated pass/fail bucket. + +## 12. Progress Exposure In The Future V3 Repo + +Each phase should be visible in the standalone repo through: + +1. `docs/progress/current-status.md` +2. `docs/progress/phase-xx.md` +3. `docs/progress/phase-xx-log.md` +4. `docs/progress/phase-xx-decisions.md` +5. `docs/conformance/status.md` + +For each phase, report: + +1. target +2. current status +3. evidence level reached +4. open risks +5. next gate + +This progress surface is still necessary even when a broader production roadmap +also exists. + +## 13. Naming And Repo Structure Rule + +For the future standalone repo, public-facing directory names should describe +system role, not internal generation history. + +Recommended top-level implementation names: + +1. `core/` +2. `runtime/` +3. `adapter/` +4. `conformance/` +5. `schema/` + +Reason: + +1. the standalone repo is the first public implementation, not a public "`V3` repo" +2. outside readers should see stable subsystem names rather than internal rewrite history +3. the semantic core is expected to grow into a fuller system rather than be replaced by another generation directory later + +Current transition rule: + +1. after the runnable-sparrow closeout, the preferred public semantic-center path is `core/` +2. any remaining `v3mini/` references should be treated as historical or transitional, not as the preferred long-term public package name +3. phase plans should refer to the semantic center as `core` when describing the intended standalone repo structure +4. documents may still mention "`v3mini`" when referring to the already-built Phase 01-03 code or historical implementation path + +Practical reading rule: + +1. `core` = the public-facing semantic center name +2. `v3mini` = the current internal implementation name for that same semantic center until the repo is renamed + +## 14. Practical Rule + +If a phase cannot be summarized in one sentence of the form: + +**"This phase proves X, and it is closed when Y evidence exists."** + +then the phase is probably too vague. + +## 15. Production Roadmap Layer + +The phase model above is the execution model. + +It should not be the only planning view. + +`V3` also needs a top-down product roadmap so the team can see how narrow phase +closures accumulate into a production-capable block system. + +The correct planning shape is: + +1. one broad production roadmap describing the major capability gates +2. one narrow execution-phase ladder describing the next concrete closure step +3. a clear mapping from the current phase to the broader roadmap position + +### 15.1 Production Target + +The production target is not: + +1. a runnable demo +2. a narrow persistence slice +3. a calibration-only repo + +The production target is: + +1. a semantically correct block system +2. with real persistence and restart safety +3. with credible crash-safety boundaries +4. with operator-usable surfaces +5. with replicated recovery and fencing behavior that are production-reviewable +6. with observability, validation, and release-hardening strong enough for real deployment + +### 15.2 Top-Down Roadmap + +The broad roadmap should be read as a capability stack: + +1. semantic core correctness +2. runnable adapter-backed block slice +3. calibration against selected `V2` reality +4. persistent single-node slice +5. local data process around the accepted persistence seam +6. data-sync process for catch-up and rebuild movement +7. recovery execution process around engine-issued commands +8. operator-usable single-node product surface +9. replicated durable slice with honest failover/fencing behavior +10. production hardening, qualification, and release gate +11. scale and topology expansion +12. operations and ecosystem expansion + +Each layer should inherit the truth already closed below it. + +No layer should be claimed early just because a lower layer demo exists. + +### 15.3 Current Position + +The current execution package should be read as: + +1. Phase 04: runnable narrow slice +2. Phase 05: bootstrap and inspection stabilization +3. Phase 06: first-pass calibration +4. Phase 07: first persistence-backed single-node slice + +This means the project is currently moving from: + +1. "runnable and calibrated" + +to: + +2. "persisted on one node" + +It is NOT yet at: + +1. full crash-safety qualification +2. operator-ready product surface +3. replicated production rollout +4. release hardening / GA gate +5. scale/topology expansion +6. operations/ecosystem completion + +Practical reading rule: + +1. the current delivered closure is recovery semantics and its lower execution boundaries +2. topology authority, failover selection/promotion policy, and broader cluster governance are explicitly assigned to later phases rather than implied as already done + +### 15.5 Responsibility Table + +For scope checking, the later mainline should be read in this short form: + +| Phase | One-line responsibility | +|---|---| +| `P9` | move bytes correctly under lineage | +| `P10` | execute recovery lifecycle correctly under start/cancel/complete/timeout | +| `P11` | ship bounded single-node product surface with no cluster-shape leak | +| `P12` | close bounded replicated handoff/rejoin mechanism as mechanism, not topology policy | +| `P13` | harden the bounded replicated slice on persistent storage for release criteria | +| `P14` | close topology authority and failover policy for who/when/where | +| `P15` | expose accepted governance through operator and ecosystem surfaces without creating new authority | + +Practical rule: + +1. if one task or PR cannot be assigned cleanly to one row, either the task is + over-scoped or the phase boundary is still unclear + +Delivery-weight note: + +1. `P13-P15` should be read as a substantial finalization program, not light + finishing work +2. the expected cost center shifts from semantic-engine invention to + control-plane, frontend, and ecosystem integration +3. the mainline is expected to rely heavily on selected `V2` product-muscle + porting rather than broad greenfield rewrite +4. use `v3-p13-p15-work-estimate.md` as the planning baseline for workload, + sequencing, and expected new-code versus port mix + +### 15.4 Planning Rule + +Future phase definitions should always answer two questions: + +1. what narrow loop does this phase close +2. which production-roadmap layer does that closure advance + +If a phase closes a local loop but does not clearly move one roadmap layer +forward, the phase is probably too internal or too detached from product +direction. + +## 16. Execution Institution Model + +`V3` should not invent lower execution institutions independently from the +semantic model. + +Instead, they should be derived from: + +1. the model truth domains +2. the accepted event types +3. the accepted command boundaries +4. the visibility contract +5. the crash/recover contract + +This means execution-layer interfaces are not just engineering convenience. + +They are the controlled lower institutions that receive delegated authority +from the already-accepted `V3` semantic route. + +### 16.1 Core Rule + +Execution institutions may have: + +1. internal data structures +2. asynchronous scheduling +3. batching, retry, and queueing +4. local progress tracking +5. narrow local decisions required to execute accepted commands + +Execution institutions may NOT have: + +1. authority to redefine semantic meaning +2. authority to reinterpret accepted commands +3. authority to publish terminal truth on their own +4. authority to widen or narrow semantic targets such as `targetLSN` +5. authority to turn local progress into system truth without returning through the accepted route + +The short rule is: + +**inside freedom is allowed; outside contract is fixed.** + +### 16.2 Knowledge And Authority Split + +The semantic route should continue to own: + +1. recovery classification +2. command intent +3. target meaning +4. terminal truth +5. external visibility rules + +Execution institutions may own only: + +1. local execution state +2. local persistence state +3. transport/data-transfer state +4. progress and error detail +5. local recovery mechanics that do not redefine semantic meaning + +### 16.3 Typical Execution Institutions + +Examples of correct lower institutions: + +1. `LogicalStorage` +2. `DataCommunicator` +3. `RecoveryExecutor` +4. `ProgressFeed` or equivalent trace/progress surface + +Expected role boundaries: + +1. `LogicalStorage` owns local persistence execution, local reopen, and local data recovery details +2. `DataCommunicator` owns data transfer, barrier execution, progress reporting, and transport-local retry behavior +3. `RecoveryExecutor` owns orchestration of already-decided commands by calling storage and communication institutions +4. trace/progress surfaces expose evidence but do not become semantic control input + +### 16.3A One Possible Further Split + +The list above is not the only acceptable cut. + +One possible future split, if the system needs stronger named institutions, is: + +1. `IdentityManager` +2. `DataManager` +3. `CommunicationManager` +4. `LBAMap` or `RecoveryIndex` +5. `RecoverProcess` + +This is a possible partition, not a claim that all five already exist as +cleanly separated interfaces today. + +Suggested responsibilities: + +1. `IdentityManager` owns self identity, peer identity, assignment, epoch, endpoint version, and topology context +2. `DataManager` owns local read, local write, `Write -> lsn`, sync, flush, checkpoint, recover, and local durability boundaries +3. `CommunicationManager` owns replica connection state, ship/receive, barrier, backpressure, and remote progress +4. `LBAMap` or `RecoveryIndex` owns recovery-oriented mapping knowledge such as which `lsn` touched which `lba` +5. `RecoverProcess` owns orchestration of engine-issued recovery intent across the other institutions + +Suggested prohibitions: + +1. `IdentityManager` should not own bytes truth or semantic recovery class +2. `DataManager` should not own topology, connection policy, or shipping policy +3. `CommunicationManager` should not decide recovery class or terminal semantic success +4. `LBAMap` should not own identity, connection lifecycle, or semantic authority +5. `RecoverProcess` should not invent recovery meaning or rewrite engine-issued targets + +One useful consequence of this split is: + +1. read APIs can go directly through `DataManager` +2. writes can return `lsn` directly from `DataManager` +3. upper layers can maintain `LBAMap` knowledge for smarter recovery without pushing that planning logic down into local storage +4. `CommunicationManager` can consume `lsn`-ordered change facts explicitly rather than being hidden behind local write side effects + +### 16.3B Manager Split Rule + +The institutions above do not need to become perfectly synchronized at every +instant. + +Local execution lag is acceptable. + +Semantic lag is not. + +Allowed: + +1. `IdentityManager` may accept a newer assignment, epoch, or topology fact before older data-movement goroutines are physically gone +2. `CommunicationManager` may still hold an old connection, sender, or queue while cleanup drains +3. `DataManager` may still retain local state associated with an older replica path while invalidation is being processed + +Not allowed: + +1. an old shipper or old connection continuing to advance current semantic truth after `IdentityManager` has already made it stale +2. stale progress, ack, barrier, or "caught up" results being accepted into the live semantic route +3. cleanup timing differences turning into authority differences + +The intended rule is: + +1. execution lag is allowed +2. authority lag is not + +Practical consequence: + +1. once a newer assignment / epoch / endpoint version / session context exists, older execution state must become semantically dead immediately +2. older goroutines may still run for cleanup, but their outputs must be ignored or rejected at the acceptance boundary +3. every manager split should be reviewed by asking whether the split changes only cleanup timing, or whether it also changes who still gets to affect system truth + +Short form: + +**old process may still run, but it must already be semantically dead.** + +### 16.3C Deliver-Lineage Rule + +When a `V3` recovery command crosses from semantic truth into execution, +it must no longer be lineage-free. + +The accepted deliver-side rule is: + +1. the engine chooses recovery class and freezes `targetLSN` +2. the adapter or execution boundary binds that recovery work to a + fresh `sessionID` +3. the execution route carries at least `sessionID + epoch + + endpointVersion + targetLSN` +4. stale or superseded lineage must be rejected before it can mutate + current semantic truth or current replica bytes + +This keeps the semantic reducer small without allowing the runtime to +become ambiguous during handoff. + +Short form: + +**partial semantic command is acceptable; lineage-free execution is not.** + +### 16.4 V2 Muscle Migration Rule + +`V3` is not a blank-slate rewrite. + +The working model is: + +1. `V3` defines semantic truth, authority, and boundaries first +2. compatible `V2` execution muscles may then be migrated behind clean `V3` seams +3. migration is filtered by semantic compatibility, not by code reuse convenience alone + +This means: + +1. semantic modules from `V2` are usually reference-only or rewritten in native `V3` form +2. execution modules from `V2` may often be migrated as whole muscles after old semantic coupling is stripped +3. the phase claim may stay narrow even when the migrated execution muscle is substantial + +The purpose is to avoid a renamed `V2` that still quietly inherits older truth +structures. + +## 17. Interface Review Checklist + +When `sw` proposes a new execution-layer interface or implementation, review it +with these questions first. + +### 17.1 Knowledge Boundary + +1. what facts does this interface know +2. are those facts local execution facts or semantic truth +3. does it hold any knowledge that should remain in the semantic route instead + +### 17.2 Authority Boundary + +1. what local decisions is this institution allowed to make +2. does it only execute an accepted command, or does it reinterpret the command +3. can it accidentally choose recovery class, target meaning, or terminal truth +4. does it add the required execution lineage before work leaves the semantic route + +### 17.3 Visibility Boundary + +1. what does it expose as trace, progress, or status +2. can any exposed local state be mistaken for semantic truth +3. does externally visible success still require the accepted route rather than local completion alone + +### 17.4 Crash And Recover Boundary + +1. what local state must survive restart +2. what local state may be discarded after crash +3. after restart, which facts must return through the accepted event/command route before they become system truth + +### 17.5 Migration Boundary + +1. is this a semantic module, adapter/glue module, or execution muscle from `V2` +2. if migrated from `V2`, what old semantic coupling was explicitly removed +3. is the migration saving throwaway work without re-importing old truth ownership + +### 17.6 Reject Signals + +Reject the interface or implementation if any of these appear: + +1. it changes the meaning of an accepted command +2. it silently widens or narrows `targetLSN` +3. it treats local progress as terminal semantic success +4. it converts trace/projection output into control truth +5. it carries hidden `V2` policy that the `V3` route did not explicitly accept +6. it allows stale callback, stale probe result, or stale mutation traffic to pass without lineage rejection + +## 18. Summary + +Yes, `V3` should use phases. + +But the important rule is: + +1. each phase has one semantic target +2. each phase has explicit non-goals +3. each phase has a clear proof gate +4. each phase is not closed until `sw` + `tester` + review all complete the loop +5. those narrow phase closures should be explicitly mapped to a broader production roadmap +6. lower execution institutions should be derived from `V3` semantic boundaries and reviewed for knowledge/authority correctness +7. compatible `V2` execution muscles may be migrated, but only after semantic filtering and boundary cleanup + +That is the safest way to grow `V3` without turning it into another open-ended architecture drift. + +## 19. Canonical P14B To Production Plan + +This section is the current canonical reading for the remaining mainline work +from late `P14` to production. + +It exists because the earlier coarse reading of `P14` and `P15` was too small +on internal control-plane closure and too optimistic about how much of the +production gap could be postponed to operator-facing phases. + +Short form: + +1. `P14B` must close the internal control-plane truth loop +2. `P14A` must verify each new mixed route created by that closure +3. `P15` must expose and productize the already-closed internal loop +4. multi-master HA is not silently included in this bounded mainline plan + +### 19.1 Stable Anchors That Must Not Drift + +The following documents remain the stable review anchors while `P14B` and `P15` +move quickly: + +1. `v3-protocol-truths.md` +2. `v3-protocol-claim-and-evidence.md` +3. `v3-semantic-constraint-checklist.md` +4. the active phase package and `14A` sidecar + +Planning may move faster than before. +These anchors must not. + +Practical rule: + +1. do not widen engine truth just because control-plane work grows +2. do not import old `V2` policy ownership with reused code +3. do not let heartbeat, transport mood, or projection convenience become semantic authority +4. do not claim broader product closure than the currently proved route + +### 19.2 Global Target + +The remaining mainline target is not just "more topology" or "more operator +surface". + +The target is: + +1. one bounded single-active-master deployment shape +2. one bounded topology/control-plane truth loop +3. one bounded host-consumable block product loop + +The control-plane truth loop that must become real is: + +`heartbeat / observation` +-> `inventory merge and freshness` +-> `stable ClusterSnapshot synthesis` +-> `policy/controller decision` +-> `assignment publication` +-> `adapter / engine convergence` +-> `observed confirmation` +-> `restart recovery of current truth` + +If this loop is not closed, `P15` surfaces have nothing stable to expose. + +### 19.3 Phase Split From Here + +The remaining mainline should be read as: + +1. `P14B`: internal control-plane closure +2. `P14A`: verification sidecar over new `P14B` mixed routes +3. `P15`: external control surface, operator/product surface, and frontend/productization + +That means: + +1. `P14B` is still part of `P14`, not a separate product phase +2. `P15` must not absorb missing internal truth closure +3. `14A` remains proof and regression pressure, not institution ownership + +### 19.4 P14B Overall Responsibility + +`P14B` is closed only when the bounded accepted topology set has: + +1. real observation ingestion +2. real snapshot synthesis +3. real durable authority source +4. real convergence and confirmation rules +5. real restart-safe current-truth recovery +6. proof through the real adapter/engine route, not only publisher-local or recording-consumer proof + +Bounded deployment assumption for this plan: + +1. one active master / control-plane owner +2. multiple volumes +3. per volume, three bounded replica slots on distinct servers +4. one current primary and two bounded candidates +5. no multi-master leader election or distributed authority ownership in this plan + +### 19.5 P14B-1 Observation Institution + +Target: + +1. turn heartbeat and inventory into a stable `ClusterSnapshot` producer rather than a test input + +Must close: + +1. heartbeat ingestion +2. freshness / expiry rules +3. partial and conflicting observation handling +4. bounded unsupported evidence for incomplete or inconsistent topology input +5. stable `ClusterSnapshot` synthesis for the accepted topology set + +Engine work: + +1. none by default +2. engine remains a consumer of already-published identity truth +3. no new engine truth or projection field unless a later proof shows a real missing read-only output + +Primary new-code areas: + +1. `core/authority/` for normalized cluster snapshot contract and observation institution boundary +2. `weed/server/` for heartbeat collector and raw master-side wiring +3. `weed/storage/blockvol/v2bridge/` only for bounded source-format adaptation when needed + +Port now from `V2` (mechanism only): + +1. `weed/server/block_heartbeat_loop.go` +2. `weed/storage/blockvol/block_heartbeat.go` +3. `weed/storage/blockvol/block_heartbeat_proto.go` +4. selected source-format adaptation ideas from `weed/storage/blockvol/v2bridge/control.go` + +Do not port directly: + +1. any heartbeat-to-policy shortcut that directly decides recovery/failover from timing +2. any volume-local authority mutation path +3. any old projection/status reuse as control truth + +Required proof: + +1. incomplete inventory becomes unsupported evidence, not failover input +2. stale observation cannot mint fresh authority +3. one bad or unsupported volume does not block unrelated healthy volume progress + +### 19.6 P14B-2 Durable Authority Institution + +Target: + +1. make the current authority line durable and restart-recoverable under the bounded single-owner deployment + +Must close: + +1. durable source for current per-volume authority line +2. restart recovery for controller / publisher current truth +3. explicit bounded single-owner rule +4. epoch / endpointVersion continuity after restart +5. bounded recovery when observed state lags durable authority + +Engine work: + +1. none by default +2. no semantic ownership moves into engine +3. only read-only evidence additions are acceptable, and only if later proof forces them + +Primary new-code areas: + +1. `core/authority/` for durable registry / current-line persistence / replay into controller state +2. `weed/server/` for master-side hosting and lifecycle +3. possibly `weed/storage/blockvol/` or adjacent persistence helpers only as storage muscle, not authority owner + +Port now from `V2` (mechanism only): + +1. `weed/server/master_block_registry.go` +2. `weed/server/master_block_assignment_queue.go` +3. lifecycle/wiring patterns from `weed/server/volume_server_block.go` + +Do not port directly: + +1. old master or volume code that mutates assignment truth from local convenience state +2. promote/demote ownership hidden inside volume-local paths +3. any old registry state treated as semantic authority without explicit filtering + +Required proof: + +1. restart does not lose current authority line +2. stale pre-restart state cannot revive newer authority +3. bounded single-owner truth remains intact without multi-master assumptions + +### 19.7 P14B-3 Convergence Institution + +Target: + +1. make `publish-until-observed` a real bounded convergence loop instead of a local planning idea + +Must close: + +1. desired-state pending / observed / superseded rules +2. retry / suppression / backoff / dedupe rules +3. explicit confirmation source for assignment and endpoint moves +4. stale observation handling that does not thrash authority +5. bounded authority transition honesty while new identity is converging + +Engine work: + +1. preserve the existing ack-gated fence and publication contract +2. do not add a generic controller loop to adapter/runtime +3. only tighten bounded-fate handling if a real mixed-route hole appears under `14A` review + +Primary new-code areas: + +1. `core/authority/` for convergence state, confirmation rules, and bounded desired-state lifecycle +2. `weed/server/` for observation inputs and hosting +3. `core/adapter/` only if a new bounded-fate or stale-rejection proof forces a narrow integration fix + +Port now from `V2` (mechanism only): + +1. selected planning/failover plumbing from `weed/server/master_block_plan.go` +2. selected selection/evidence mechanics from `weed/server/master_block_failover.go` +3. selected evidence patterns from `weed/server/master_block_evidence.go` + +Do not port directly: + +1. old failover trigger meaning as-is +2. old policy ownership hidden in master convenience logic +3. any route where transport/heartbeat timing directly becomes failover truth + +Required proof: + +1. published desired state remains until observed or superseded +2. stale observation cannot churn the current line +3. authority transition does not overclaim healthy publication +4. failover and rebalance remain bounded and diagnosable under the accepted topology set + +### 19.8 P14B-4 Full P14 Close + +Target: + +1. close one bounded topology/governance product shape on the full accepted topology set + +Must close: + +1. multi-volume topology authority under the accepted three-slot pattern +2. failover and rebalance through real adapter/engine convergence +3. restart/catch-up/fence/publication behavior across the full bounded route +4. one final bounded supported-topology statement +5. one explicit unsupported list + +Engine work: + +1. no new policy ownership +2. no broad semantic rewrite +3. only bounded fixes that `14A` proves are necessary on the mixed routes + +Primary proof expectation: + +1. controller-driven route must reach real `VolumeReplicaAdapter`, not only recording consumers +2. controller-driven failover/rebalance must remain honest at the publication surface during transition +3. crash/restart mixed-route proofs must exist for the bounded deployment + +### 19.9 P14A Role During P14B + +`14A` remains the verification sidecar. +It should reopen only when a new `P14B` workstream creates real mixed-route pressure. + +Expected `14A` reopen themes by workstream: + +1. after Observation Institution: + - stale / delayed heartbeat + - partial inventory + - conflicting observation + - unsupported evidence honesty +2. after Durable Authority Institution: + - stale authority after restart + - old truth revival + - bounded single-owner correctness +3. after Convergence Institution: + - publish-but-not-observed loops + - timeout / suppress / dedupe correctness + - transition publication honesty + - new silent liveness holes +4. before Full P14 Close: + - mixed-route sequence pass across observation, authority, convergence, demotion, and failover overlap + +`14A` must not: + +1. invent new product institutions +2. widen engine truth just because testing is hard +3. claim global engine stability + +### 19.10 P15 Responsibility After P14B + +`P15` starts only after the internal control-plane loop is closed enough to expose. + +`P15` owns: + +1. external control APIs +2. operator-facing diagnostics and explanation surfaces +3. runbook and lifecycle productization +4. frontend/export protocols such as `iSCSI`, `NVMe/TCP`, and `CSI` +5. operator packaging and ecosystem integration + +`P15` does NOT own: + +1. heartbeat truth formation +2. durable current authority truth +3. convergence semantics +4. the internal policy/controller truth loop itself + +### 19.11 V2 Port Matrix For The Mainline Plan + +#### Port now into `P14B` + +These are the highest-value muscles for the reopened `P14` mainline: + +1. `weed/server/block_heartbeat_loop.go` +2. `weed/storage/blockvol/block_heartbeat.go` +3. `weed/storage/blockvol/block_heartbeat_proto.go` +4. `weed/server/master_block_registry.go` +5. `weed/server/master_block_assignment_queue.go` +6. selected mechanism-shaped parts of `weed/server/master_block_plan.go` +7. selected mechanism-shaped parts of `weed/server/master_block_failover.go` +8. selected evidence patterns from `weed/server/master_block_evidence.go` +9. bounded master/volume hosting patterns from `weed/server/volume_server_block.go` + +#### Port later into `P15` + +1. `weed/storage/blockvol/iscsi/` +2. `weed/storage/blockvol/nvme/` +3. `weed/storage/blockvol/csi/` +4. `weed/storage/blockvol/operator/` +5. `weed/storage/blockvol/monitoring/` +6. larger `testrunner` and scenario-product muscles + +#### Reference only or do-not-port directly + +1. `weed/storage/blockvol/promotion.go` +2. old `HandleAssignment` / `promote` / `demote` ownership paths +3. old engine/orchestrator/registry semantic owners as current truth owners +4. any `V2` route that turns heartbeat timing, local status, or transport convenience directly into authority +5. any code that would make volume-local runtime mutate assignment truth on its own + +### 19.12 Practical Implementation Order + +Use this order unless new evidence forces a re-cut: + +1. `P14B-1` Observation Institution +2. `14A` targeted reopen on observation routes +3. `P14B-2` Durable Authority Institution +4. `14A` targeted reopen on restart and durable-truth routes +5. `P14B-3` Convergence Institution +6. `14A` targeted reopen on convergence and publication-honesty routes +7. `P14B-4` Full P14 close package +8. `P15` external/operator/frontend productization + +This is intentionally a big-step plan. +It is not a license to blur the engine. + +The speed rule is: + +1. move fast by porting `V2` muscles aggressively +2. keep engine truth, authority ownership, and semantic constraints stable +3. prefer one big bounded institution per step over many plumbing-only micro-steps + +### 19.13 One-Sentence Summary + +From here to production, the mainline should be read as: + +**`P14B` closes the internal control-plane truth loop using semantic-filtered `V2` muscles, `14A` verifies the new mixed routes, and `P15` exposes the already-closed loop through operator and ecosystem surfaces.**