mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-21 22:56:55 +00:00
docs: Phase 13 CP13-1..CP13-7 technical packs, acceptance status, design updates
- phase-13.md: CP13-1 through CP13-6 accepted, CP13-7 active - phase-13-log.md: full technical + delivery packs for CP13-2..CP13-7 - phase-13-cp4-state-eligibility.md: refined barrier behavior table (Disconnected/Degraded as recovery entry points, not eligibility) - phase-12.md: minor cross-reference updates - Older phase docs: minor wording alignment - Design docs: V2 development plan and completion overview updated Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
pingqiu
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
1d3fb1f119
commit
21ccf06ef3
@@ -1,7 +1,7 @@
|
||||
# Phase 12
|
||||
|
||||
Date: 2026-04-02
|
||||
Status: active
|
||||
Status: accepted
|
||||
Purpose: move the accepted chosen-path implementation from candidate-safe product closure toward production-safe behavior under restart, disturbance, and operational reality
|
||||
|
||||
## Why This Phase Exists
|
||||
@@ -423,7 +423,7 @@ Reject if:
|
||||
|
||||
Status:
|
||||
|
||||
- active
|
||||
- accepted
|
||||
|
||||
Carry-forward from `P3`:
|
||||
|
||||
@@ -431,23 +431,13 @@ Carry-forward from `P3`:
|
||||
2. `P4` now asks whether that accepted path has an explicit measured floor and an explicit first-launch envelope
|
||||
3. later work after `Phase 12` should be a productionization program, not another hidden hardening slice
|
||||
|
||||
## Assignment For `sw`
|
||||
## Phase Close-Out Note
|
||||
|
||||
Current next tasks:
|
||||
`Phase 12` is now accepted as bounded production hardening on the chosen path:
|
||||
|
||||
1. deliver `Phase 12 P4` as bounded performance-floor / rollout-gate hardening
|
||||
2. keep the acceptance object fixed on one measured workload envelope plus one explicit launch-envelope / gate artifact
|
||||
3. keep earlier accepted `Phase 09` / `Phase 10` / `Phase 11` / `Phase 12 P1` / `Phase 12 P2` / `Phase 12 P3` semantics stable unless `P4` exposes a real bug or measurement-surface gap
|
||||
4. do not let `P4` turn into broad optimization, topology expansion, or generic launch marketing
|
||||
1. `P1` accepted disturbance correctness
|
||||
2. `P2` accepted bounded soak / long-run stability
|
||||
3. `P3` accepted diagnosability / blocker accounting / runbook hardening
|
||||
4. `P4` accepted bounded performance floor / rollout-gate hardening
|
||||
|
||||
## Assignment For `tester`
|
||||
|
||||
Current next tasks:
|
||||
|
||||
1. validate `P4` as real bounded performance-floor / rollout-gate hardening rather than “benchmark numbers exist” prose
|
||||
2. require explicit validation targets for:
|
||||
- one named workload envelope with measured floor values
|
||||
- one explicit launch-envelope / rollout-gate artifact
|
||||
- no-overclaim around broad production readiness beyond the named envelope
|
||||
3. keep no-overclaim active around accepted `Phase 09` / `Phase 10` / `Phase 11` / `Phase 12 P1` / `Phase 12 P2` / `Phase 12 P3` closure
|
||||
4. keep `P4` bounded rather than letting it absorb post-Phase-12 productionization work
|
||||
Next work should open a new phase or program rather than silently continuing inside `Phase 12`.
|
||||
|
||||
@@ -7,14 +7,14 @@ Code change: one new test (`TestBarrier_NonEligibleStates_FailClosed`)
|
||||
|
||||
The replication path uses a bounded 6-state set (`wal_shipper.go:25-30`):
|
||||
|
||||
| State | Value | Meaning | Barrier eligible? |
|
||||
|-------|-------|---------|-------------------|
|
||||
| `Disconnected` | 0 | No session (initial state) | No — attempts bootstrap/reconnect, fails if no progress |
|
||||
| `Connecting` | 1 | Socket open, handshake pending | No — immediate `ErrReplicaDegraded` |
|
||||
| `CatchingUp` | 2 | Connected, replaying missed WAL | No — immediate `ErrReplicaDegraded` |
|
||||
| `InSync` | 3 | Eligible for sync_all barriers | **Yes** — only state that proceeds to barrier request |
|
||||
| `Degraded` | 4 | Transient failure, retry allowed | No — attempts reconnect, fails if reconnect fails |
|
||||
| `NeedsRebuild` | 5 | WAL gap too large, rebuild required | No — immediate `ErrReplicaDegraded` |
|
||||
| State | Value | Meaning | Barrier behavior |
|
||||
|-------|-------|---------|-----------------|
|
||||
| `Disconnected` | 0 | No session (initial state) | Attempts bootstrap/reconnect inside Barrier(); fails if no progress or reconnect fails |
|
||||
| `Connecting` | 1 | Socket open, handshake pending | Immediate `ErrReplicaDegraded` |
|
||||
| `CatchingUp` | 2 | Connected, replaying missed WAL | Immediate `ErrReplicaDegraded` |
|
||||
| `InSync` | 3 | Eligible for sync_all barriers | **Proceeds to barrier request** — only state that can complete barrier successfully |
|
||||
| `Degraded` | 4 | Transient failure, retry allowed | Attempts reconnect inside Barrier(); fails if reconnect fails |
|
||||
| `NeedsRebuild` | 5 | WAL gap too large, rebuild required | Immediate `ErrReplicaDegraded` |
|
||||
|
||||
## Barrier State Gate
|
||||
|
||||
@@ -33,16 +33,28 @@ default:
|
||||
}
|
||||
```
|
||||
|
||||
**Only `InSync` enters the barrier request path.** All other states either fail immediately or attempt reconnect (which must succeed and transition to `InSync` before reaching the barrier).
|
||||
**Contract (precise):**
|
||||
|
||||
- **Only `InSync` can complete barrier successfully.** It is the only state that proceeds
|
||||
directly to the barrier request (ensureCtrlConn → MsgBarrierReq → wait for BarrierOK).
|
||||
- **`Disconnected` and `Degraded` use Barrier() as a recovery entry point.** They attempt
|
||||
bootstrap/reconnect inside the Barrier() call. If recovery succeeds and transitions to
|
||||
InSync, the barrier request proceeds. If recovery fails, the barrier fails.
|
||||
- **`Connecting`, `CatchingUp`, `NeedsRebuild` are rejected immediately** with `ErrReplicaDegraded`.
|
||||
|
||||
The key distinction: Barrier() can be *invoked* from Disconnected/Degraded (as a recovery
|
||||
trigger), but only InSync can *satisfy* barrier success. The Disconnected/Degraded paths
|
||||
are recovery attempts, not barrier eligibility.
|
||||
|
||||
## sync_all Gate
|
||||
|
||||
`dist_group_commit.go:59-66`: sync_all counts barrier failures. Any shipper that returns an error from `Barrier()` increments `failCount`. If `failCount > 0`, sync_all returns `ErrDurabilityBarrierFailed`.
|
||||
|
||||
Combined with the CP13-3 fix (FlushedLSN=0 rejected), the full chain is:
|
||||
1. Only `InSync` shippers reach the barrier request
|
||||
2. Only `BarrierOK` with `FlushedLSN > 0` counts as success
|
||||
3. sync_all fails if any barrier fails
|
||||
1. Only `InSync` shippers proceed to the barrier request
|
||||
2. Disconnected/Degraded may recover inside Barrier(), transitioning to InSync before requesting
|
||||
3. Only `BarrierOK` with `FlushedLSN > 0` counts as success
|
||||
4. sync_all fails if any barrier fails
|
||||
|
||||
## Proof Promotion
|
||||
|
||||
@@ -50,7 +62,7 @@ Combined with the CP13-3 fix (FlushedLSN=0 rejected), the full chain is:
|
||||
|
||||
| Test | What it proves for CP13-4 |
|
||||
|------|--------------------------|
|
||||
| `TestBarrier_NonEligibleStates_FailClosed` | Connecting, CatchingUp, NeedsRebuild all rejected immediately; Disconnected fails on dead address; only InSync enters barrier path |
|
||||
| `TestBarrier_NonEligibleStates_FailClosed` | 5 sub-cases: Connecting/CatchingUp/NeedsRebuild rejected immediately; Disconnected fails (no recovery on dead addr); InSync enters barrier path (verified by MsgBarrierReq receipt on fake server) |
|
||||
| `TestBarrier_RejectsReplicaNotInSync` | SyncCache fails when replica is not InSync (end-to-end) |
|
||||
| `TestBarrier_DuringCatchup_Rejected` | Barrier rejected while replica is CatchingUp |
|
||||
| `TestDistSync_SyncAll_AllDegraded_Fails` | sync_all fails when all replicas degraded |
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,608 @@
|
||||
# Phase 13
|
||||
|
||||
Date: 2026-04-02
|
||||
Status: active
|
||||
Purpose: carry one explicit engineering gap beyond accepted `Phase 12` hardening into a bounded implementation phase so `RF=2 sync_all` becomes a correct, test-backed replicated durability mode under real reconnect, catch-up, retention, and rebuild conditions
|
||||
|
||||
## Why This Phase Exists
|
||||
|
||||
`Phase 09` accepted chosen-path execution closure.
|
||||
`Phase 10` accepted bounded control-plane closure.
|
||||
`Phase 11` accepted bounded product-surface rebinding.
|
||||
`Phase 12` accepted bounded hardening, diagnosability, and first-launch envelope evidence.
|
||||
|
||||
What still remains is not broad protocol discovery.
|
||||
It is one concrete engineering problem:
|
||||
|
||||
1. `sync_all` still needs a cleaner replicated-durability contract under cross-machine reconnect and replica recovery reality
|
||||
2. that contract must be expressed in code and tests so later feature work can reuse it rather than reopen replication semantics repeatedly
|
||||
|
||||
## Phase Goal
|
||||
|
||||
Turn `RF=2 sync_all` from a bounded chosen-path mode with accepted launch-hardening evidence into a correct, reusable replicated-durability model for reconnect, catch-up, retention, and rebuild on real workloads.
|
||||
|
||||
Execution note:
|
||||
|
||||
1. use `phase-13-log.md` as the technical pack for:
|
||||
- checkpoint breakdown
|
||||
- acceptance objects
|
||||
- reject shapes
|
||||
- assignment text for `sw` and `tester`
|
||||
2. prefer test-first baseline plus checkpointed implementation
|
||||
3. keep the goal narrow: replication correctness first, not broad optimization or new transport work
|
||||
|
||||
## Scope
|
||||
|
||||
### In scope
|
||||
|
||||
1. canonical replica address truth
|
||||
2. authoritative per-replica durable-progress tracking
|
||||
3. reconnect handshake and WAL catch-up
|
||||
4. replica-aware WAL retention / truncation
|
||||
5. rebuild fallback when catch-up is impossible
|
||||
6. real ext4 / PostgreSQL validation on real block devices for cross-machine `sync_all`
|
||||
7. mode normalization work that depends directly on the corrected replication model
|
||||
|
||||
### Out of scope
|
||||
|
||||
1. broad new protocol discovery outside the replication path
|
||||
2. new transport projects such as `SPDK`, `io_uring`, or striped-layout redesign
|
||||
3. generic benchmark positioning beyond correctness-backed validation
|
||||
4. unrelated control-plane or product-surface expansion
|
||||
5. reopening accepted `Phase 09` / `Phase 10` / `Phase 11` / `Phase 12` semantics unless this phase exposes a real bug
|
||||
|
||||
## Phase 13 Items
|
||||
|
||||
### `CP13-1`: Test-First Baseline
|
||||
|
||||
Goal:
|
||||
|
||||
- freeze a failing/passing baseline that exposes the current replication gaps before protocol work begins
|
||||
|
||||
Acceptance object:
|
||||
|
||||
1. the focused sync-replication gap tests exist
|
||||
2. they are run on current code before major implementation work
|
||||
3. the fail/pass split is captured explicitly so later checkpoint claims are grounded
|
||||
|
||||
Status:
|
||||
|
||||
- accepted
|
||||
|
||||
Carry-forward:
|
||||
|
||||
1. the baseline report is frozen in `phase-13-cp1-baseline.md`
|
||||
2. no protocol code was changed in `CP13-1`
|
||||
3. `CP13-2` and later checkpoints must treat the baseline as the starting truth, not redefine it after implementation
|
||||
|
||||
### `CP13-2`: Canonical Replica Addressing
|
||||
|
||||
Goal:
|
||||
|
||||
- make replica endpoint truth canonical and routable so cross-machine replication never depends on wildcard listener strings, incomplete `:port` forms, or other non-authoritative address leakage
|
||||
|
||||
Acceptance object:
|
||||
|
||||
1. `CP13-2` accepts canonical replica address truth for the replication path
|
||||
2. it does not accept durable-progress truth, reconnect protocol, WAL retention, or rebuild fallback by implication
|
||||
3. it does not accept broad networking redesign beyond endpoint canonicalization
|
||||
|
||||
Execution steps:
|
||||
|
||||
1. Step 1: address truth contract freeze
|
||||
- define the canonical replica endpoint form for replication surfaces as routable `host:port`
|
||||
- define which forms are invalid for exported/registered truth:
|
||||
- bare `:port`
|
||||
- wildcard listener strings such as `[::]:port`
|
||||
- accidental loopback when cross-machine routing is intended
|
||||
2. Step 2: implementation hardening
|
||||
- canonicalize replica listener addresses at the source where receiver/registration surfaces expose them
|
||||
- keep authoritative endpoint truth aligned across local listener state, registration/heartbeat publication, and any registry copies
|
||||
3. Step 3: proof package
|
||||
- prove canonical `host:port` truth is emitted under wildcard-bind cases
|
||||
- prove no wildcard or incomplete address string leaks into exported replication truth
|
||||
- prove no-overclaim around reconnect, retention, or rebuild semantics
|
||||
|
||||
Required scope:
|
||||
|
||||
1. replica receiver endpoint truth
|
||||
2. registration / heartbeat / registry path carrying replica endpoints
|
||||
3. one focused wildcard-bind proof plus bounded cross-machine truth checks
|
||||
4. explicit distinction between address canonicalization and later reconnect protocol work
|
||||
|
||||
Must prove:
|
||||
|
||||
1. cross-machine replica addresses exported for replication are canonical routable `host:port`
|
||||
2. wildcard bind strings do not escape into replication truth
|
||||
3. local canonicalization does not silently rewrite intentionally loopback-only cases into incorrect external truth
|
||||
4. acceptance wording stays bounded to endpoint truth rather than later replication recovery semantics
|
||||
|
||||
Reuse discipline:
|
||||
|
||||
1. `weed/storage/blockvol/replica_receiver.go`, `replica_meta.go`, and nearby address helpers may be updated in place as the primary endpoint-truth surface
|
||||
2. `weed/server/master_block_registry.go` and heartbeat/registration paths may be updated in place only if needed to keep authoritative endpoint truth aligned
|
||||
3. focused unit/protocol tests should carry the main proof burden; component tests are support-only unless they prove an otherwise unreachable leak
|
||||
4. no checkpoint work may silently introduce reconnect protocol, retention policy, or rebuild logic
|
||||
|
||||
Verification mechanism:
|
||||
|
||||
1. one focused wildcard-bind canonicalization proof
|
||||
2. explicit checks that exported/registered replica endpoints are routable `host:port`
|
||||
3. no-overclaim review so `CP13-2` does not absorb `CP13-3+`
|
||||
|
||||
Hard indicators:
|
||||
|
||||
1. one accepted canonical-endpoint proof:
|
||||
- wildcard-bind listener state resolves to canonical exported `host:port`
|
||||
2. one accepted no-leak proof:
|
||||
- bare `:port` / wildcard listener strings no longer escape into replication truth
|
||||
3. one accepted boundedness proof:
|
||||
- `CP13-2` claims endpoint truth only, not reconnect or durability semantics
|
||||
|
||||
Reject if:
|
||||
|
||||
1. the checkpoint fixes only one test string shape but leaves other exported endpoint paths unchanged
|
||||
2. canonicalization happens only in tests rather than at the production truth surface
|
||||
3. the checkpoint quietly broadens into reconnect, retention, or rebuild protocol work
|
||||
|
||||
Status:
|
||||
|
||||
- accepted
|
||||
|
||||
Carry-forward:
|
||||
|
||||
1. `localServerID` remains stable control identity and may be opaque
|
||||
2. `advertisedHost` is now the transport-facing canonicalization input for wildcard-bind replica endpoints
|
||||
3. `CP13-3` and later checkpoints must not reopen identity-vs-transport separation unless a new concrete bug is exposed
|
||||
|
||||
### `CP13-3`: Durable Progress Truth
|
||||
|
||||
Goal:
|
||||
|
||||
- make durable replication progress explicit and authoritative so sync correctness is grounded in replica flushed durability rather than sender-side send progress or loosely inferred health
|
||||
|
||||
Acceptance object:
|
||||
|
||||
1. `CP13-3` accepts durable progress truth for the replication path
|
||||
2. it does not accept reconnect/catch-up protocol, retention policy, rebuild fallback, or broader state-machine closure by implication
|
||||
3. it does not accept generic “tests pass” reasoning without an explicit durable-progress contract review
|
||||
|
||||
Execution steps:
|
||||
|
||||
1. Step 1: durable-progress contract freeze
|
||||
- define `replicaFlushedLSN` as replica-side WAL durability confirmed at barrier time
|
||||
- define sender-side shipped/sent progress as diagnostic only, not authority for sync correctness
|
||||
- define what barrier responses must expose as explicit durable progress truth
|
||||
2. Step 2: implementation hardening or proof confirmation
|
||||
- update the durable-progress path only where current code fails to meet the contract
|
||||
- if current code already satisfies the contract, keep changes minimal and make the proof package explicit instead of broadening scope
|
||||
3. Step 3: proof package
|
||||
- prove barrier success is grounded in replica flushed durability
|
||||
- prove flushed progress is monotonic within epoch and not updated on mere receive
|
||||
- prove no-overclaim around `CP13-4+`
|
||||
|
||||
Required scope:
|
||||
|
||||
1. replica receiver durable-progress state
|
||||
2. barrier request/response path
|
||||
3. sender/group tracking of replica durable progress
|
||||
4. explicit separation between durable-progress truth and later reconnect / retention semantics
|
||||
|
||||
Must prove:
|
||||
|
||||
1. `replicaFlushedLSN` means replica durability, not sender transmission progress
|
||||
2. barrier responses expose durable progress explicitly enough for sync correctness decisions
|
||||
3. sender-side progress such as shipped/sent LSN is diagnostic only and cannot authorize sync success
|
||||
4. acceptance wording stays bounded to durable-progress truth rather than broader recovery/state-machine closure
|
||||
|
||||
Reuse discipline:
|
||||
|
||||
1. `weed/storage/blockvol/replica_apply.go`, `wal_shipper.go`, `dist_group_commit.go`, and related protocol message code may be updated in place as the primary durable-progress surfaces
|
||||
2. focused unit/protocol tests should carry the main proof burden
|
||||
3. `weed/server/*` should remain reference only unless durable-progress truth requires an exposed wiring change
|
||||
4. no checkpoint work may silently introduce reconnect protocol, retention policy, rebuild policy, or broader transport redesign
|
||||
|
||||
Verification mechanism:
|
||||
|
||||
1. one focused proof set around barrier/flushed progress truth
|
||||
2. explicit checks that receive progress alone does not advance durable authority
|
||||
3. no-overclaim review so `CP13-3` does not absorb `CP13-4+`
|
||||
|
||||
Hard indicators:
|
||||
|
||||
1. one accepted barrier-truth proof:
|
||||
- barrier success is tied to replica flushed durability
|
||||
2. one accepted monotonicity proof:
|
||||
- `replicaFlushedLSN` is monotonic within epoch
|
||||
3. one accepted no-false-authority proof:
|
||||
- sender-side shipped/sent progress is diagnostic only
|
||||
4. one accepted boundedness proof:
|
||||
- `CP13-3` claims durable-progress truth only
|
||||
|
||||
Reject if:
|
||||
|
||||
1. the checkpoint treats passing baseline tests as automatic closure without reviewing the durable-progress contract
|
||||
2. durable-progress truth is still mixed with sender-side transmission progress
|
||||
3. the checkpoint quietly broadens into reconnect, retention, rebuild, or general replication redesign
|
||||
|
||||
Status:
|
||||
|
||||
- accepted
|
||||
|
||||
Carry-forward:
|
||||
|
||||
1. `replicaFlushedLSN` is now the authoritative durable-progress variable for `sync_all`
|
||||
2. legacy `BarrierOK` responses without `FlushedLSN` are rejected and cannot count as durable authority
|
||||
3. `CP13-4` and later checkpoints must treat sender-side send progress as diagnostic only, not as sync-correctness authority
|
||||
|
||||
### `CP13-4`: Replica State Machine / Barrier Eligibility
|
||||
|
||||
Goal:
|
||||
|
||||
- make replica state and barrier eligibility explicit so only `InSync` replicas can satisfy sync durability while non-eligible states fail closed instead of drifting into accidental success
|
||||
|
||||
Acceptance object:
|
||||
|
||||
1. `CP13-4` accepts the replica state machine and barrier-eligibility contract
|
||||
2. it does not accept reconnect/catch-up protocol, retention policy, rebuild fallback, or broader rollout claims by implication
|
||||
3. it does not accept vague “state seems fine” reasoning without an explicit eligibility contract
|
||||
|
||||
Execution steps:
|
||||
|
||||
1. Step 1: state contract freeze
|
||||
- define the bounded state set used by the replication path:
|
||||
- `Disconnected`
|
||||
- `Connecting`
|
||||
- `CatchingUp`
|
||||
- `InSync`
|
||||
- `Degraded`
|
||||
- `NeedsRebuild`
|
||||
- define barrier eligibility:
|
||||
- only `InSync` replicas count toward sync durability
|
||||
- non-eligible states must pre-reject or fail closed
|
||||
2. Step 2: implementation hardening or proof confirmation
|
||||
- update the state/eligibility path only where current code fails the contract
|
||||
- if current code already satisfies much of the contract, keep code changes minimal and make the proof package explicit
|
||||
3. Step 3: proof package
|
||||
- prove barrier rejects replicas not eligible for sync durability
|
||||
- prove degraded or catching-up replicas do not silently count toward `sync_all`
|
||||
- prove no-overclaim around `CP13-5+`
|
||||
|
||||
Required scope:
|
||||
|
||||
1. replica shipper state transitions and eligibility checks
|
||||
2. barrier admission path
|
||||
3. `sync_all` failure semantics when replicas are non-eligible
|
||||
4. explicit separation between state eligibility and later reconnect/rebuild protocol work
|
||||
|
||||
Must prove:
|
||||
|
||||
1. only `InSync` replicas count toward sync durability
|
||||
2. `Disconnected`, `Connecting`, `CatchingUp`, `Degraded`, and `NeedsRebuild` do not silently satisfy barrier eligibility
|
||||
3. degraded/non-eligible replicas fail closed for `sync_all` rather than producing false durability success
|
||||
4. acceptance wording stays bounded to state/eligibility truth rather than reconnect, retention, or rebuild closure
|
||||
|
||||
Reuse discipline:
|
||||
|
||||
1. `weed/storage/blockvol/wal_shipper.go`, `dist_group_commit.go`, `shipper_group.go`, and nearby replication coordination code may be updated in place as the primary state/eligibility surfaces
|
||||
2. focused unit/protocol/adversarial tests should carry the main proof burden
|
||||
3. `weed/server/*` should remain reference only unless state eligibility requires a surfaced wiring correction
|
||||
4. no checkpoint work may silently introduce reconnect handshake, retention policy, rebuild flow, or broader transport redesign
|
||||
|
||||
Verification mechanism:
|
||||
|
||||
1. one focused proof set around replica state and barrier eligibility
|
||||
2. explicit checks that non-`InSync` states cannot satisfy `sync_all`
|
||||
3. no-overclaim review so `CP13-4` does not absorb `CP13-5+`
|
||||
|
||||
Hard indicators:
|
||||
|
||||
1. one accepted eligibility proof:
|
||||
- only `InSync` replicas count toward sync durability
|
||||
2. one accepted fail-closed proof:
|
||||
- non-eligible replicas cause bounded failure rather than false success
|
||||
3. one accepted state-boundary proof:
|
||||
- barrier rejects or excludes disallowed states explicitly
|
||||
4. one accepted boundedness proof:
|
||||
- `CP13-4` claims state/eligibility truth only
|
||||
|
||||
Reject if:
|
||||
|
||||
1. the checkpoint treats passing baseline tests as automatic closure without restating the state/eligibility contract
|
||||
2. non-eligible replica states can still satisfy sync durability
|
||||
3. the checkpoint quietly broadens into reconnect, retention, rebuild, or general replication redesign
|
||||
|
||||
Status:
|
||||
|
||||
- accepted
|
||||
|
||||
Carry-forward:
|
||||
|
||||
1. the replica state set and barrier-eligibility contract are now explicit
|
||||
2. only `InSync` may satisfy sync durability; `Disconnected`/`Degraded` may invoke `Barrier()` only as bounded recovery entry paths
|
||||
3. `CP13-5` and later checkpoints must preserve this eligibility boundary rather than reopening it implicitly
|
||||
|
||||
### `CP13-5`: Reconnect Handshake + WAL Catch-up
|
||||
|
||||
Goal:
|
||||
|
||||
- make reconnect after replica disturbance explicit and correct so a replica with known durable progress can resume from retained WAL, catch up, and re-enter `InSync` without false bootstrap success or barrier hangs
|
||||
|
||||
Acceptance object:
|
||||
|
||||
1. `CP13-5` accepts the reconnect handshake and WAL catch-up contract for recoverable gaps on the replication path
|
||||
2. it does not accept replica-aware WAL retention policy, full rebuild fallback lifecycle, or broader rollout claims by implication
|
||||
3. it does not accept vague “reconnect seems to work” reasoning without an explicit resume/catch-up contract
|
||||
|
||||
Execution steps:
|
||||
|
||||
1. Step 1: reconnect contract freeze
|
||||
- define when a replica must use bootstrap versus reconnect:
|
||||
- fresh replica with no prior durable progress may bootstrap
|
||||
- replica with prior flushed progress must reconnect via explicit resume truth
|
||||
- define reconnect decision outcomes:
|
||||
- already caught up
|
||||
- recoverable gap within retained WAL
|
||||
- unrecoverable gap that must fail closed and defer full rebuild handling to `CP13-7`
|
||||
2. Step 2: implementation hardening
|
||||
- update the reconnect path only where current code still fails the resume/catch-up contract
|
||||
- ensure catch-up replays retained WAL before barrier success is allowed
|
||||
- ensure repeated disconnect/reconnect cycles remain bounded and do not silently fall back to unsafe bootstrap
|
||||
3. Step 3: proof package
|
||||
- prove degraded replicas with prior durable progress use handshake/reconnect rather than bootstrap
|
||||
- prove retained-WAL catch-up completes and re-enters `InSync` on recoverable gaps
|
||||
- prove reconnect fails closed on unrecoverable or incomplete recovery cases
|
||||
- prove no-overclaim around `CP13-6+`
|
||||
|
||||
Required scope:
|
||||
|
||||
1. `wal_shipper` reconnect discriminator and resume handshake
|
||||
2. retained-WAL catch-up replay path
|
||||
3. repeated disconnect/reconnect recovery behavior
|
||||
4. bounded failure semantics for gaps that cannot be recovered within this checkpoint
|
||||
5. explicit separation between reconnect/catch-up closure and later retention/rebuild policy work
|
||||
|
||||
Must prove:
|
||||
|
||||
1. fresh shippers bootstrap, but previously-synced shippers reconnect using resume truth
|
||||
2. barrier success after disturbance is allowed only after reconnect/catch-up has re-established `InSync`
|
||||
3. repeated disconnect/reconnect cycles do not strand the replica in false degraded recovery
|
||||
4. recoverable gaps replay retained WAL correctly without overwriting newer replica data
|
||||
5. acceptance wording stays bounded to reconnect/catch-up truth rather than retention or rebuild closure
|
||||
|
||||
Reuse discipline:
|
||||
|
||||
1. `weed/storage/blockvol/wal_shipper.go`, reconnect/catch-up helpers, and nearby replication protocol code may be updated in place as the primary reconnect surface
|
||||
2. focused protocol/adversarial tests should carry the main proof burden; component tests are support-only unless a protocol gap is otherwise unreachable
|
||||
3. `weed/server/*` should remain reference only unless reconnect correctness requires surfaced wiring changes
|
||||
4. no checkpoint work may silently broaden into retention policy, rebuild orchestration, or performance tuning
|
||||
|
||||
Verification mechanism:
|
||||
|
||||
1. one focused proof set around reconnect discriminator, catch-up replay, and post-reconnect barrier behavior
|
||||
2. explicit checks for repeated disconnect/reconnect recovery
|
||||
3. explicit checks that recoverable gaps replay retained WAL before sync success
|
||||
4. no-overclaim review so `CP13-5` does not absorb `CP13-6+`
|
||||
|
||||
Hard indicators:
|
||||
|
||||
1. one accepted reconnect-discriminator proof:
|
||||
- prior durable progress uses handshake/reconnect rather than bootstrap
|
||||
2. one accepted catch-up proof:
|
||||
- recoverable retained-WAL gap replays and returns to `InSync`
|
||||
3. one accepted repeated-recovery proof:
|
||||
- multiple disconnect/reconnect cycles recover without hanging or drifting
|
||||
4. one accepted fail-closed proof:
|
||||
- reconnect does not falsely succeed when recovery is incomplete or impossible within retained WAL
|
||||
5. one accepted boundedness proof:
|
||||
- `CP13-5` claims reconnect/catch-up truth only
|
||||
|
||||
Reject if:
|
||||
|
||||
1. a previously-synced replica can still skip resume truth and succeed via unsafe bootstrap
|
||||
2. barrier success can occur before reconnect/catch-up has restored `InSync`
|
||||
3. repeated reconnect cycles still hang, strand, or silently degrade correctness
|
||||
4. the checkpoint quietly broadens into retention, explicit `NeedsRebuild` lifecycle closure, rebuild execution, or general replication redesign
|
||||
|
||||
Status:
|
||||
|
||||
- accepted
|
||||
|
||||
Carry-forward:
|
||||
|
||||
1. replacement shippers now preserve prior durable-progress intent across `SetReplicaAddrs`
|
||||
2. previously-synced replicas must reconnect through resume truth and retained-WAL catch-up rather than unsafe bootstrap
|
||||
3. `CP13-6` and later checkpoints must preserve the reconnect/catch-up contract rather than weakening it through reclaim or rebuild shortcuts
|
||||
|
||||
### `CP13-6`: Replica-Aware WAL Retention
|
||||
|
||||
Goal:
|
||||
|
||||
- make WAL retention explicit and replica-aware so reclaim is gated by recoverable replica progress and bounded retention budgets rather than silently discarding catch-up-critical WAL
|
||||
|
||||
Acceptance object:
|
||||
|
||||
1. `CP13-6` accepts replica-aware WAL retention and retention-budget truth on the replication path
|
||||
2. it does not accept full rebuild fallback lifecycle, rebuild execution, or broader rollout claims by implication
|
||||
3. it does not accept vague “reclaim seems safe” reasoning without an explicit retention contract
|
||||
|
||||
Execution steps:
|
||||
|
||||
1. Step 1: retention contract freeze
|
||||
- define which replica progress is authoritative for WAL retention:
|
||||
- only replicas with prior durable progress and still recoverable state may hold WAL
|
||||
- define bounded retention outcomes:
|
||||
- reclaim blocked while a recoverable replica still needs retained WAL
|
||||
- timeout / max-bytes budgets may escalate boundedly and release the WAL hold
|
||||
- full rebuild handling after escalation remains `CP13-7`
|
||||
2. Step 2: implementation hardening
|
||||
- update the retention path only where current code still fails the bounded retention contract
|
||||
- ensure retention decisions use replica-aware progress rather than primary-local heuristics alone
|
||||
- ensure budget-triggered escalation is explicit and fail-closed rather than silent reclaim
|
||||
3. Step 3: proof package
|
||||
- prove recoverable replicas block reclaim of needed WAL
|
||||
- prove timeout / max-bytes budgets trigger bounded escalation instead of indefinite WAL growth
|
||||
- prove retention remains aligned with `CP13-5` reconnect/catch-up truth
|
||||
- prove no-overclaim around `CP13-7+`
|
||||
|
||||
Required scope:
|
||||
|
||||
1. WAL retention/reclaim gates
|
||||
2. shipper-group retention inputs derived from recoverable replica progress
|
||||
3. bounded timeout / max-bytes escalation behavior
|
||||
4. explicit separation between retention truth and full rebuild lifecycle closure
|
||||
|
||||
Must prove:
|
||||
|
||||
1. reclaim does not drop WAL still required by a recoverable replica
|
||||
2. retention inputs come from replica-aware durable progress, not sender-side guesses
|
||||
3. timeout / max-bytes budgets trigger bounded escalation when WAL cannot be held indefinitely
|
||||
4. acceptance wording stays bounded to retention truth rather than full rebuild closure
|
||||
|
||||
Reuse discipline:
|
||||
|
||||
1. `weed/storage/blockvol` WAL-retention, flusher, shipper-group, and adjacent replication coordination code may be updated in place as the primary retention surface
|
||||
2. focused unit/protocol tests should carry the main proof burden; component tests are support-only unless a retention gap is otherwise unreachable
|
||||
3. `weed/server/*` should remain reference only unless retention truth requires surfaced reporting changes
|
||||
4. no checkpoint work may silently broaden into rebuild execution, broad control-plane redesign, or performance tuning
|
||||
|
||||
Verification mechanism:
|
||||
|
||||
1. one focused proof set around retention hold, reclaim gating, and budget-triggered escalation
|
||||
2. explicit checks that max-bytes and timeout paths are real production behaviors, not just comments/logs
|
||||
3. explicit checks that retention stays compatible with `CP13-5` recoverable catch-up
|
||||
4. no-overclaim review so `CP13-6` does not absorb `CP13-7+`
|
||||
|
||||
Hard indicators:
|
||||
|
||||
1. one accepted hold-back proof:
|
||||
- recoverable replicas block reclaim of required WAL
|
||||
2. one accepted timeout-budget proof:
|
||||
- timeout can escalate a stalled recoverable replica into bounded fail-closed behavior
|
||||
3. one accepted max-bytes-budget proof:
|
||||
- max-bytes pressure triggers explicit bounded escalation rather than silent reclaim or TODO-only behavior
|
||||
4. one accepted boundedness proof:
|
||||
- `CP13-6` claims retention truth only
|
||||
|
||||
Reject if:
|
||||
|
||||
1. reclaim can still silently discard WAL needed for a recoverable replica
|
||||
2. max-bytes behavior is still only log text / placeholder behavior without real state effect
|
||||
3. the checkpoint quietly broadens into full `NeedsRebuild` lifecycle closure, rebuild execution, or general replication redesign
|
||||
|
||||
Status:
|
||||
|
||||
- accepted
|
||||
|
||||
Carry-forward:
|
||||
|
||||
1. retention inputs and bounded retention budgets are now replica-aware
|
||||
2. timeout and max-bytes escalation can move a stalled recoverable replica into `NeedsRebuild`
|
||||
3. `CP13-7` must turn that escalation into a real fail-closed rebuild lifecycle rather than leaving `NeedsRebuild` as a partially-signaled state
|
||||
|
||||
### `CP13-7`: Rebuild Fallback
|
||||
|
||||
Goal:
|
||||
|
||||
- make `NeedsRebuild` a real fail-closed recovery state so unrecoverable replicas stop participating in normal replication paths, surface rebuild intent clearly, and re-enter the replication contract only through bounded rebuild handoff
|
||||
|
||||
Acceptance object:
|
||||
|
||||
1. `CP13-7` accepts the `NeedsRebuild` fallback and bounded rebuild handoff lifecycle on the replication path
|
||||
2. it does not accept broad rollout claims or real-workload validation by implication
|
||||
3. it does not accept vague “rebuild eventually works” reasoning without an explicit fail-closed lifecycle contract
|
||||
|
||||
Execution steps:
|
||||
|
||||
1. Step 1: rebuild-fallback contract freeze
|
||||
- define what `NeedsRebuild` means:
|
||||
- unrecoverable via retained WAL catch-up
|
||||
- excluded from normal ship/barrier success
|
||||
- visible to rebuild orchestration and observability surfaces
|
||||
- define lifecycle boundaries:
|
||||
- detection/escalation into `NeedsRebuild`
|
||||
- fail-closed behavior while in `NeedsRebuild`
|
||||
- bounded rebuild handoff and post-rebuild re-entry
|
||||
2. Step 2: implementation hardening
|
||||
- update the rebuild-fallback path only where current code still leaves `NeedsRebuild` partial, leaky, or inconsistent
|
||||
- ensure ship/barrier paths block correctly while `NeedsRebuild`
|
||||
- ensure successful rebuild resets progress/state in a way compatible with later re-entry
|
||||
3. Step 3: proof package
|
||||
- prove unrecoverable gaps transition to `NeedsRebuild`
|
||||
- prove `NeedsRebuild` blocks normal replication participation
|
||||
- prove rebuild handoff can re-establish a bounded healthy starting point
|
||||
- prove no-overclaim around `CP13-8+`
|
||||
|
||||
Required scope:
|
||||
|
||||
1. `NeedsRebuild` detection and state ownership on the primary shipper side
|
||||
2. fail-closed behavior for ship/barrier and related replication paths while `NeedsRebuild`
|
||||
3. rebuild start/abort/complete handoff boundaries
|
||||
4. post-rebuild progress/state initialization needed for safe re-entry
|
||||
5. explicit separation between rebuild fallback closure and later real-workload validation
|
||||
|
||||
Must prove:
|
||||
|
||||
1. unrecoverable gaps do not remain merely degraded; they transition to `NeedsRebuild`
|
||||
2. a shipper in `NeedsRebuild` cannot silently participate in ship/barrier success
|
||||
3. rebuild completion restores a bounded re-entry point without faking immediate `InSync`
|
||||
4. acceptance wording stays bounded to rebuild fallback truth rather than `CP13-8` rollout/workload claims
|
||||
|
||||
Reuse discipline:
|
||||
|
||||
1. `weed/storage/blockvol` rebuild, shipper-group, wal-shipper, and adjacent replication coordination code may be updated in place as the primary rebuild-fallback surface
|
||||
2. focused unit/protocol/adversarial tests should carry the main proof burden; component tests are support-only unless a rebuild gap is otherwise unreachable
|
||||
3. `weed/server/*` should remain reference only unless rebuild fallback requires surfaced status/reporting changes
|
||||
4. no checkpoint work may silently broaden into real-workload benchmarking, performance tuning, or new protocol discovery
|
||||
|
||||
Verification mechanism:
|
||||
|
||||
1. one focused proof set around `NeedsRebuild` transition, blocking semantics, and rebuild re-entry
|
||||
2. explicit checks that `NeedsRebuild` blocks normal replication paths rather than merely logging/marking degraded
|
||||
3. explicit checks that post-rebuild progress initializes from bounded truth such as checkpoint state
|
||||
4. no-overclaim review so `CP13-7` does not absorb `CP13-8+`
|
||||
|
||||
Hard indicators:
|
||||
|
||||
1. one accepted transition proof:
|
||||
- unrecoverable retained-WAL gap transitions to `NeedsRebuild`
|
||||
2. one accepted fail-closed proof:
|
||||
- `NeedsRebuild` blocks ship/barrier participation
|
||||
3. one accepted rebuild-handoff proof:
|
||||
- rebuild start/complete path restores a bounded re-entry state
|
||||
4. one accepted post-rebuild-progress proof:
|
||||
- replica progress after rebuild is initialized from checkpoint truth, not stale/zeroed state
|
||||
5. one accepted boundedness proof:
|
||||
- `CP13-7` claims rebuild fallback only
|
||||
|
||||
Reject if:
|
||||
|
||||
1. an unrecoverable gap can still linger in `Degraded` without escalating to `NeedsRebuild`
|
||||
2. a `NeedsRebuild` shipper can still satisfy normal ship/barrier paths
|
||||
3. rebuild completion jumps directly to misleading healthy semantics without bounded re-entry proof
|
||||
4. the checkpoint quietly broadens into `CP13-8` real-workload validation or general replication redesign
|
||||
|
||||
Status:
|
||||
|
||||
- active
|
||||
|
||||
### Later checkpoints inside `Phase 13`
|
||||
|
||||
1. `CP13-8`: real-workload validation
|
||||
2. `CP13-9`: mode normalization
|
||||
|
||||
## Reuse Discipline
|
||||
|
||||
1. `weed/storage/blockvol/*` is the primary implementation surface and may be updated in place
|
||||
2. focused unit/component/adversarial tests should carry the main proof burden
|
||||
3. real-node / real-device validation belongs in testrunner or bounded component scenarios, not chat prose
|
||||
4. `weed/server/*` may be updated only when replication correctness requires registry / assignment / heartbeat truth to change
|
||||
5. no checkpoint may silently broaden into performance-optimization or broad rollout work
|
||||
|
||||
## Expected Outcome
|
||||
|
||||
If `Phase 13` succeeds:
|
||||
|
||||
1. reconnect / catch-up / rebuild semantics become explicit and test-backed
|
||||
2. `sync_all` correctness no longer depends on partial or implicit sender-state assumptions
|
||||
3. later feature work can reuse a clearer replication contract instead of re-deriving durability semantics each time
|
||||
@@ -301,6 +301,60 @@ After `Phase 12`:
|
||||
3. run a limited internal pilot with incident-driven hardening
|
||||
4. perform controlled rollout only after explicit launch-gate review
|
||||
|
||||
### Post-`Phase 12`: Productionization Program
|
||||
|
||||
Goal:
|
||||
|
||||
1. turn the accepted `Phase 12` chosen path into a bounded first-launch product envelope without reopening protocol discovery
|
||||
|
||||
Program slices:
|
||||
|
||||
1. Program `P0`: launch-envelope freeze
|
||||
- freeze the first supported launch envelope from accepted `P1`-`P4` evidence
|
||||
- lock:
|
||||
- supported topology / transport matrix
|
||||
- explicit exclusions
|
||||
- launch-blocking vs post-launch blockers
|
||||
- reject if any launch claim outruns the measured matrix or accepted blockers/gates
|
||||
2. Program `P1`: internal pilot pack
|
||||
- convert the frozen launch envelope into a limited internal pilot package
|
||||
- define:
|
||||
- pilot environment and topology
|
||||
- preflight checklist
|
||||
- success criteria
|
||||
- stop / rollback conditions
|
||||
- incident intake template tied to accepted diagnosability surfaces
|
||||
- reject if pilot success depends on tribal knowledge or undefined operator judgment
|
||||
3. Program `P2`: incident-driven hardening loop
|
||||
- route pilot findings into explicit buckets:
|
||||
- config / environment issue
|
||||
- known exclusion
|
||||
- true product bug
|
||||
- keep one bounded incident ledger and one bounded fix queue
|
||||
- reject if incidents accumulate as vague notes or exclusions are silently redefined
|
||||
4. Program `P3`: controlled rollout review
|
||||
- decide whether to:
|
||||
- stay in pilot
|
||||
- widen within the same launch envelope
|
||||
- block expansion
|
||||
- require explicit mapping from any expansion decision back to:
|
||||
- accepted `Phase 12` evidence
|
||||
- pilot outcomes
|
||||
- incident dispositions
|
||||
- reject if rollout broadens beyond the named envelope or reuses pilot success as generic production proof
|
||||
|
||||
Cross-cutting rules:
|
||||
|
||||
1. do not invent a `Phase 12 P5`; this is a separate productionization program
|
||||
2. keep the accepted chosen path fixed unless incidents expose a real bug
|
||||
3. treat known missing evidence as explicit constraints until cleared, especially:
|
||||
- failover-under-load performance
|
||||
- hours/days soak under load
|
||||
- `RF>2`
|
||||
- broad transport matrix
|
||||
- full gRPC-stream integration evidence
|
||||
4. keep V2 post-`Phase 12` direction aligned with the learn-tree close/gate work so the roadmap does not split into contradictory stories
|
||||
|
||||
## Module Status Map
|
||||
|
||||
|
||||
@@ -347,18 +401,26 @@ If the goal is to maximize product completion efficiently, the recommended order
|
||||
2. keep `Phase 10` closed and do not reopen accepted bounded control-plane closure casually
|
||||
3. move next to `Phase 11` product surface rebinding
|
||||
4. then `Phase 12` production hardening
|
||||
5. then the post-`Phase 12` productionization program:
|
||||
- freeze launch envelope
|
||||
- run limited internal pilot
|
||||
- harden from incidents
|
||||
- review controlled rollout
|
||||
|
||||
The most important near-term engineering weight should now go to `Phase 12`.
|
||||
The most important near-term engineering weight should now go to:
|
||||
|
||||
1. finishing `Phase 12`
|
||||
2. then the bounded productionization program immediately after it
|
||||
|
||||
## Short Summary
|
||||
|
||||
The V2 line now has accepted execution closure on one bounded chosen path.
|
||||
The next development plan should treat later work as control/product completion phases, not more protocol discovery.
|
||||
The next development plan should treat later work as bounded hardening plus productionization, not more protocol discovery.
|
||||
|
||||
The main heavy engineering work still ahead is:
|
||||
|
||||
1. stronger end-to-end control-plane closure
|
||||
2. later product-surface rebinding
|
||||
3. production hardening
|
||||
4. bounded cleanup of residual operational rough edges without reopening accepted semantics
|
||||
4. post-`Phase 12` launch-envelope freeze, internal pilot, and controlled rollout review
|
||||
|
||||
|
||||
@@ -228,7 +228,7 @@ Target:
|
||||
|
||||
Status:
|
||||
|
||||
1. next active stage
|
||||
1. accepted and closed on the bounded chosen path
|
||||
|
||||
Main work:
|
||||
|
||||
@@ -237,6 +237,37 @@ Main work:
|
||||
3. performance floor and cost characterization
|
||||
4. explicit production blockers / rollout gates
|
||||
|
||||
### Stage 6: Post-`Phase 12` Productionization Program
|
||||
|
||||
Target:
|
||||
|
||||
1. turn the accepted `Phase 12` chosen path into a bounded first-launch product envelope without reopening protocol discovery
|
||||
|
||||
Status:
|
||||
|
||||
1. next active stage after `Phase 12`
|
||||
|
||||
Main work:
|
||||
|
||||
1. freeze the first supported launch envelope from accepted `P1`-`P4` evidence
|
||||
2. define a limited internal pilot package with explicit preflight, success, and stop conditions
|
||||
3. run incident-driven hardening with explicit classification:
|
||||
- config / environment issue
|
||||
- known exclusion
|
||||
- true product bug
|
||||
4. perform controlled rollout review only within the named supported envelope
|
||||
|
||||
Rules:
|
||||
|
||||
1. this is not `Phase 12 P5`
|
||||
2. pilot success is not generic production proof
|
||||
3. missing evidence remains an explicit launch constraint until cleared, especially:
|
||||
- failover-under-load performance
|
||||
- hours/days soak under load
|
||||
- `RF>2`
|
||||
- broad transport matrix
|
||||
- full gRPC-stream integration evidence
|
||||
|
||||
## Completion Gates
|
||||
|
||||
The most important gates from here are:
|
||||
@@ -251,31 +282,35 @@ The most important gates from here are:
|
||||
- front-end surfaces should only rebind after backend correctness is strong enough
|
||||
5. production-hardening gate
|
||||
- restart, soak, diagnosis, and repeated disturbance must be acceptable
|
||||
6. productionization gate
|
||||
- first launch envelope, pilot discipline, incident routing, and controlled rollout review must be explicit
|
||||
|
||||
## Near-Term Planning Guidance
|
||||
|
||||
If the goal is to maximize product completion efficiently:
|
||||
|
||||
1. do not make `CSI`, `NVMe`, or broad snapshot productization the immediate next heavy phase
|
||||
2. first close production execution gaps in the backend path
|
||||
3. then strengthen control-plane closure
|
||||
4. then rebind product surfaces
|
||||
1. do not reopen accepted execution, control-plane, or product-surface semantics casually
|
||||
2. finish `Phase 12` hardening cleanly
|
||||
3. then freeze the first supported launch envelope
|
||||
4. then run a limited internal pilot
|
||||
5. then widen only through explicit incident review and rollout-gate review
|
||||
|
||||
In short:
|
||||
|
||||
1. backend truth and execution first
|
||||
2. product surfaces second
|
||||
3. production hardening last
|
||||
1. chosen-path closure first
|
||||
2. production hardening second
|
||||
3. bounded productionization third
|
||||
|
||||
## Short Summary
|
||||
|
||||
The V2 line is already beyond "algorithm only".
|
||||
It has an accepted bounded chosen path through backend, control-plane, and selected product surfaces.
|
||||
It has an accepted bounded chosen path through backend, control-plane, selected product surfaces, and `Phase 12` hardening.
|
||||
|
||||
But the remaining work is still substantial, and it is mostly engineering work:
|
||||
|
||||
1. production hardening under restart / disturbance
|
||||
2. long-run stability and diagnosability
|
||||
3. performance floor and rollout gating
|
||||
1. freeze the first supported launch envelope from accepted evidence
|
||||
2. run a limited internal pilot with explicit stop conditions
|
||||
3. harden from incidents without silently broadening scope
|
||||
4. perform controlled rollout review inside a bounded launch envelope
|
||||
|
||||
That is the practical path from the current candidate-safe engine to a production-ready block product.
|
||||
That is the practical path from the current production-safe chosen path to a bounded first-launch block product.
|
||||
|
||||
Reference in New Issue
Block a user