mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-22 07:06:51 +00:00
G6 §close + 5 INVs inscribed in ledger + roadmap closure
m01 single-run GREEN at 71 s on seaweed_block@96c51b4 — both §2 #4 (retention-OK catch-up) AND §2 #5 (sustained-write recycle → rebuild dispatch + 5000-LBA byte-equal) in one closed-loop run per architect §2 #6 binding. Logs: V:\share\g5-test\logs\g6-20260428T100217Z.log Scenarios: V:\share\g5-test\scenarios-g6.sh + scenarios\g6-d.sh Mini-plan §close: - §close.summary: 8-row table of bindings + commits + hardware + regression status, all GREEN. - §close.evidence: software-pin (3 commits, 10 tests / 14 cases PASS); hardware-pin (5 acceptance rows, all GREEN; single 71 s run). - §close.deltas: 2 entries documenting (a) physical-recycle NOT required for §2 #5 (engine recovery decision branch is load-bearing) and (b) harness discipline finding from QA. - §close.findings: 2 findings — (1) data-vs-state convergence harness discipline → new INV; (2) §1.H audit verdict was correct + resolved in-batch. - §close.forward-carries: G5-2/G5-6 (durability mode), G5-3 (peer-state surface), future replica-aware retention (β/γ), G7 (rebuild path semantics). 5 INVs inscribed in v3-invariant-ledger.md: - INV-G6-WALRECYCLE-DISPATCHES-REBUILD - INV-G6-CATCHUP-CONVERGES-WITHIN-RETENTION - INV-G6-RETENTION-POLICY-OPERATOR-VISIBLE - INV-G6-ENGINE-NO-REBUILD-PINNED-ON-OTHER-FAILURES - INV-G6-HARNESS-DATA-AND-STATE-CONVERGENCE (NEW from §close.findings #1) INV-G6-RETENTION-POLICY-REPLICA-AWARE NOT inscribed — reserved for future β/γ replica-aware retention batch (architect §1.A α ratification 2026-04-29). Roadmap §3 G6 line: ⏳ next → ✅ closed 2026-04-28 (retention-aware recovery; α config knob + escalation pin). Roadmap §7: G6 row added to recently-closed table. Roadmap §8 backlog: G6-T-WALRECYCLE-ESCALATE → "Closed backlog tickets" section with verdict (a) + resolution narrative. Awaiting architect single-sign on this §close.
This commit is contained in:
pingqiu
@@ -39,7 +39,7 @@ G2 Frontend smoke ✅ closed (T1/T2)
|
||||
G3 Real frontends iSCSI+NVMe ✅ closed (T2)
|
||||
G4 Local durable data path ✅ closed (T3)
|
||||
G5 Replicated write path 🟡 closing (T4 done; G5-1..6 close work; G5-5 just closed at L3, G5-5C carry-forward)
|
||||
G6 Incremental WAL catch-up ⏳ next (T4c muscle done, mostly verification)
|
||||
G6 Incremental WAL catch-up ✅ closed 2026-04-28 (retention-aware recovery; α config knob + escalation pin)
|
||||
G7 Rebuild / replica re-creation ⏳ (T4d-4 part B/C scaffolding done)
|
||||
G8 Failover data continuity ⏳
|
||||
G9 Volume lifecycle ⏳
|
||||
@@ -122,6 +122,7 @@ For any specific question, go to the canonical doc:
|
||||
| T2A NVMe race fix | 2026-04-25 | `seaweed_block@a0be6d5` | atomic.Pointer test fixture; -race ×50 PASS |
|
||||
| G5-4 binary T4 wiring | 2026-04-26 | `seaweed_block@c820e17` + ledger `seaweedfs@36ba7b44e` + close-lock `daafc8e25` | binary now wires T4 stack; criteria 3+4 relocated to G5-5 |
|
||||
| G5-5 m01 hardware first-light | 2026-04-27 | `seaweed_block@5c4718f` + close-doc `seaweedfs@c78116fd2` | L3 Replicated IO on m01/M02 hardware: #1 cluster role split GREEN, #2 live iSCSI replicated write byte-equal GREEN, #3 network partition + heal catch-up GREEN (8s); #4 replica-restart catch-up carried to G5-5C as real recovery-path finding |
|
||||
| G6 Incremental WAL catch-up + recycle escalation | 2026-04-28 | `seaweed_block@96c51b4` (preceded by `85475cd` rebuild-START marker + `420ad1f` audit) + close-doc forthcoming | L5 retention-aware recovery: §1.A α config knob (`--wal-retention-lsns`) + walstore retention gate relaxation + engine table-driven WALRecycled→Rebuild dispatch test (3 tests, 6 cases) + substrate retention test (4 cases) + m01 single-run GREEN at 71s covering both #4 retention-OK catch-up and #5 recycle→rebuild escalation. 5 INVs inscribed including new harness-discipline INV-G6-HARNESS-DATA-AND-STATE-CONVERGENCE (chained scenarios need both byte-equal AND peer-healthy sync). G5-5C carry-forward G6-T-WALRECYCLE-ESCALATE closed at this gate. |
|
||||
|
||||
---
|
||||
|
||||
@@ -137,11 +138,11 @@ Per `v3-phase-15-mvp-scope-gates.md` §4.5 dependency graph:
|
||||
|
||||
P15 closes at **G22 final cluster validation**. After P15 → P16 (in-place migration is the only hinted scope).
|
||||
|
||||
### Open backlog tickets queued for G6
|
||||
### Closed backlog tickets
|
||||
|
||||
| Ticket | Source | Description | Evidence |
|
||||
|---|---|---|---|
|
||||
| **G6-T-WALRECYCLE-ESCALATE** | G5-5C QA scenario D 2026-04-28 (architect-bound carry) | Verify engine recovery decision escalates `ProbeOutcome=WALRecycled` → `StartRebuildFromProbe` (a) OR surface as G6 gap if missing (b). Catch-up cannot bridge gap-beyond-WAL by design; rebuild path must auto-fire from probe outcome carrying WALRecycled, otherwise sustained writes leave replica permanently degraded. Dispatch-branch correctness already pinned at engine layer (Batch 4 `TestG5_5C_Dispatch_CatchUpVsRebuild_TableDriven`); the runtime escalation chain under sustained pressure is what's untested. | `V:\share\g5-test\logs\bcd-20260428T072539Z.log` D-section: `peer r2 invalidated (prev=catching_up, reason=session_failed: catch-up: WAL recycled: storage: WALRecycled: fromLSN=602 checkpointLSN=700 headLSN=701)`; no rebuild dispatch in 5 s scrape window. Cross-ref `INV-G5-5C-PROBE-BEFORE-CATCHUP`. QA prep: `wait_until_rebuild_dispatched` helper for the eventual G6 acceptance (held until G6 kickoff). |
|
||||
| Ticket | Source | Resolution |
|
||||
|---|---|---|
|
||||
| **G6-T-WALRECYCLE-ESCALATE** | G5-5C QA scenario D 2026-04-28 (architect-bound carry) | ✅ **Closed at G6 §close 2026-04-28** — verdict (a) existing-and-correct + observability gap. Engine `apply.go:519-547` already wired `RecoveryFailureWALRecycled` → `RebuildPinned` + `StartRebuild` emit at T4d-1; G6 added the rebuild-START log marker + operator retention knob + 14 test cases (10 unit + 4 substrate) + m01 hardware single-run GREEN at 71s. INV-G6-WALRECYCLE-DISPATCHES-REBUILD inscribed. Original 5s observation was scrape-window-too-short, NOT a missing dispatch. |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -110,6 +110,11 @@ IDs are **append-only**. If an invariant is retired, move its row to §6 (archiv
|
||||
| `INV-BIN-WIRING-ASSIGNMENT-DRIVES-MEMBERPRESENT` | Binary doesn't fake `MemberPresent`; it waits for first `AssignmentFact` with `fact.ReplicaID == self.ReplicaID` to set it via the engine apply path. Replica role (no self-named fact) keeps `MemberPresent=false` indefinitely (correct: replica isn't frontend-primary-write-ready) | P15 G5-4 | engine + host (binary) | `cmd/blockvolume/g5_4_l2_replication_test.go:TestG54_BinaryWiring_RoleSplit_2NodeSmoke` (replica's status response shows `Healthy=false` even after 10s — pins MemberPresent never flips for replica role; primary's `Healthy=true` pins MemberPresent flips on self-named fact). Backstop: `core/engine/apply.go:applyAssignment` test coverage in `apply_test.go` | 2026-04-27 (G5-5 §close — Tier 2 m01 cross-node hardware Integration backstop: `seaweed_block@5c4718f` rounds 1-14 surfaced + verified the binary-wiring chain end-to-end; G5-4 close pinned at component+subprocess scope, G5-5 upgrades to real-hardware) | ACTIVE |
|
||||
| `INV-BIN-WIRING-SESSIONID-VIA-ADAPTER` | Binary path mints sessionIDs ONLY through `core/adapter` (process-wide `sessionIDCounter atomic.Uint64` at `adapter.go:70`); MUST NOT bypass adapter with hardcoded sessionIDs the way component-framework shortcuts (`WithLiveShip`, `CatchUpReplica`) do. Adapter-routed dispatch is the production path; framework shortcuts are test conveniences with a known sessionID-collision gap (T4c §I carry; QA G5-1 round 1 SKIP). Pinning this invariant ensures the gap stays test-side | P15 G5-4 | adapter | `cmd/blockvolume/main.go` (binary path: peer-set updates flow through `replication.UpdateReplicaSet` → `ReplicaPeer` → `BlockExecutor` calls dispatched by `adapter.VolumeReplicaAdapter`, which mints fresh sessionIDs at every dispatch site). Source-side check: `cmd/blockvolume/main.go` does not import `core/replication/component` (mini-plan v0.4 §7.1 #2 fence). Backstop: `core/adapter/adapter.go:70` declares `sessionIDCounter` with file-local visibility; only `adapter.go:353,359,377,392,402` mint via `Add(1)` | 2026-04-27 (G5-5 §close — Tier 2 m01 cross-node hardware Integration backstop: `seaweed_block@5c4718f` rounds 1-14 surfaced + verified the binary-wiring chain end-to-end; G5-4 close pinned at component+subprocess scope, G5-5 upgrades to real-hardware) | ACTIVE |
|
||||
| `INV-G5-FRONTEND-CAPACITY-FROM-DURABLE-CONFIG` | iSCSI/NVMe externally-visible volume capacity and block size MUST derive from the daemon's `--durable-blocks` × `--durable-blocksize`, not silently fall back to frontend defaults (`DefaultVolumeBlocks=2048` × `DefaultBlockSize=512` = 1 MiB). Without this plumb-through, a daemon configured for N MiB durable storage advertises a 1 MiB iSCSI/NVMe LUN and any workload above LBA 256 fails | P15 G5-5C addendum (P0 product fix) | host (binary, frontend wiring) | Unit: `cmd/blockvolume/frontend_capacity_test.go:TestComputeFrontendVolumeSize_ProductOfBlocksAndBlockSize` (5 cases incl. 256 MiB), `TestComputeFrontendVolumeSize_RejectsZero`, `TestComputeFrontendVolumeSize_OverflowGuard`, `TestIscsiHandlerCapacity_FromDurableConfig`, `TestNvmeHandlerCapacity_FromDurableConfig` (capacity round-trip via `iscsi.NewSCSIHandler` / `nvme.NewIOHandler`), `TestFrontendDefaults_StillReturn1MiB` (negative-control documenting the bug shape). Source-side check: `cmd/blockvolume/main.go::computeFrontendVolumeSize` flows into both `iscsi.TargetConfig.Handler.{BlockSize, VolumeSize}` and `nvme.TargetConfig.Handler.{BlockSize, VolumeSize}`. Hardware: m01 scale-sequential (1000 LBAs × 4 KiB = 4 MiB) verified at G5-5C addendum hardware re-run | 2026-04-28 (G5-5C addendum P0 — surfaced by QA scale-sequential test on `seaweed_block@712cbc47`; fixed at `seaweed_block@a250b52`; m01 hardware re-verification pending QA on this tree) | ACTIVE |
|
||||
| `INV-G6-WALRECYCLE-DISPATCHES-REBUILD` | When a recovery session fails with `RecoveryFailureWALRecycled`, the engine pins `RebuildPinned=true` and emits `StartRebuild` immediately (`core/engine/apply.go:applySessionFailed:519-547`); the lagging replica is moved off the catch-up path onto the rebuild path automatically. A subsequent fresh probe with R/S/H that would otherwise classify as catch-up cannot downgrade Decision back to CatchUp (RebuildPinned overrides). Operator never sees a permanently-degraded peer when the cause is recycle | P15 G6 | engine + transport + runtime | Unit: `core/engine/g6_walrecycle_dispatch_test.go:TestG6_SessionFailed_DispatchByFailureKind_TableDriven/wal_recycled_pins_rebuild_and_emits_start_rebuild`, `TestG6_RebuildPinned_StaysSticky_AcrossProbe`, `TestG6_WALRecycled_PublishDegradedSurfaceFires`. Transport mapping: `core/transport/recovery_session.go:classifyRecoveryFailure` maps `storage.StorageRecoveryFailureWALRecycled` → `engine.RecoveryFailureWALRecycled`. Substrate envelope: `core/storage/recovery_contract.go:NewWALRecycledFailure` (typed `*RecoveryFailure`). Hardware: m01 §2 #5 single-run GREEN — `executor: rebuild start replica=r2 sessionID=4 epoch=1 EV=1 targetLSN=1` observed within deadline; post-rebuild byte-equal across 5000 LBAs | 2026-04-28 (G6 §close — Tier 2 m01 cross-node single-run 71s GREEN at `V:\share\g5-test\logs\g6-20260428T100217Z.log`; software at `seaweed_block@96c51b4`) | ACTIVE |
|
||||
| `INV-G6-CATCHUP-CONVERGES-WITHIN-RETENTION` | When replica's gap stays within the configured WAL retention window (`fromLSN > checkpointLSN - WALRetentionLSNs`), catch-up converges to byte-equal within deadline; rebuild does not dispatch in this regime. Operator-tunable retention envelope is the contract for "catch-up succeeds vs rebuild dispatches" | P15 G6 | storage (substrate scan gate) + runtime | Unit: `core/storage/walstore_retention_test.go:TestWALStore_RecoveryRetention_NonZero_WidensWindow` (floor=checkpoint-retention; fromLSN > floor accepted, ≤ floor recycled), `TestWALStore_RecoveryRetention_Default_StrictRecycle` (zero retention preserves pre-G6 strict gate), `TestWALStore_RecoveryRetention_LargerThanCheckpoint_ClampsToZero` (saturating subtraction guard). Source: `core/storage/walstore_recovery.go:62-79` retention floor computation. Hardware: m01 §2 #4 retention-OK catch-up GREEN within partition+heal scenario | 2026-04-28 (G6 §close) | ACTIVE |
|
||||
| `INV-G6-RETENTION-POLICY-OPERATOR-VISIBLE` | WAL retention is operator-tunable via `--wal-retention-lsns` (G6 §1.A α); default value is documented in flag help text; the operator-tunable boundary is the explicit contract for catch-up-vs-rebuild. ImplWALStore honors directly; ImplSmartWAL bounds retention via `--durable-blocks` (ring capacity) and ignores this flag (documented in `ProviderConfig.WALRetentionLSNs` godoc) | P15 G6 | host (binary) + storage (substrate plumb) | Source: `cmd/blockvolume/main.go` flag definition + plumb through `durable.ProviderConfig.WALRetentionLSNs`. Substrate accessors: `core/storage/walstore.go::SetRecoveryRetentionLSNs` / `RecoveryRetentionLSNs`. Provider plumbing: `core/frontend/durable/provider.go:openExisting + createFresh` ImplWALStore path | 2026-04-28 (G6 §close) | ACTIVE |
|
||||
| `INV-G6-ENGINE-NO-REBUILD-PINNED-ON-OTHER-FAILURES` | `RebuildPinned=true` fires ONLY for `RecoveryFailureWALRecycled` (and the existing T4c-3 retry-budget-exhaustion path, unchanged). `RecoveryFailureTransport` and `RecoveryFailureSubstrateIO` stay on the catch-up retry path; the engine increments `Attempts` and re-emits `StartCatchUp` if budget remains. Negative-control on dispatch-branch correctness | P15 G6 | engine | Unit: `core/engine/g6_walrecycle_dispatch_test.go:TestG6_SessionFailed_DispatchByFailureKind_TableDriven/transport_failure_does_not_pin_rebuild` + `/substrate_io_does_not_pin_rebuild` (assertions: `RebuildPinned == false`, `Decision != Rebuild`, `StartRebuild` NOT emitted, `StartCatchUp` IS emitted) | 2026-04-28 (G6 §close) | ACTIVE |
|
||||
| `INV-G6-HARNESS-DATA-AND-STATE-CONVERGENCE` | Hardware tests that chain scenarios on the same cluster MUST wait for both data convergence (`wait_until_byte_equal`) AND state convergence (`wait_until_peer_healthy`, scrape pattern `replication: peer <id> state .* → healthy`) before returning. Two timelines (replica's durable storage vs primary's per-peer engine state) advance independently; data alone is necessary but not sufficient — peer state lags by ≤1 probe cycle (~5s default). Without state-convergence sync, chained scenarios start with a stale-degraded peer that breaks ship admission | P15 G6 (harness discipline) | QA harness | Source: `V:\share\g5-test\scenarios-g6.sh::wait_until_peer_healthy` (deadline-bounded log scrape against the peer-state transition marker emitted by `core/replication/peer.go::SetState`). Discovered by combined-run failure pre-fix; resolved by Option A. Pinned on m01 §close run by single-run GREEN across §2 #4 + #5 in 71 s | 2026-04-28 (G6 §close — meta-finding from QA scenario-chaining work) | ACTIVE |
|
||||
|
||||
> **The table above is a schema seed.** P15 QA establishment populates it fully from the four sources listed in §2, cross-referenced with [`v3-semantic-constraint-checklist.md`](./v3-semantic-constraint-checklist.md).
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# V3 Phase 15 — G6 (Incremental WAL Catch-Up + Recycle Escalation) Mini-Plan
|
||||
|
||||
**Date**: 2026-04-28 (v0.1 — kickoff draft for architect ratification)
|
||||
**Status**: §1-§6 awaiting architect ratification per `v3-batch-process.md §5`
|
||||
**Date**: 2026-04-28 (v0.1 — kickoff draft for architect ratification); **§1–§6 ratified 2026-04-29** (see §7)
|
||||
**Status**: §1–§6 ✅ **architect-ratified** (`§1.A=α`; `§2 #5` = rebuild-dispatch required AC); §1.H next
|
||||
**Repo**: `seaweed_block` (V3) — code; `seaweedfs` — design docs / mini-plans
|
||||
**Owner**: sw (audit + tests; possibly small engine/host patches per §1.H); QA (harness extension + hardware re-run)
|
||||
**Process**: `v3-batch-process.md` compressed flow (one mini-plan, one PR per repo, one §close)
|
||||
@@ -108,7 +108,7 @@ Numbered, verifier-named, single source of truth.
|
||||
| 2 | **§1.A WAL retention policy** bound + landed: Option α (config knob with documented default) OR Option β (pin-window with operator timeout) OR Option γ (replica-watermark via probe-loop). | Architect §7 ratification + (if α) flag landed in `cmd/blockvolume/main.go` + (if β/γ) replication-layer changes |
|
||||
| 3 | **Engine-layer dispatch test**: table-driven test pins that `RecoveryFailureWALRecycled` failure kind on a running session leads to `RebuildPinned=true` and next decide() emits `StartRebuild`. Negative case: other failure kinds (Transport, SubstrateIO) do NOT pin Rebuild. | New `g6_*_test.go` in `core/engine/` (or extension of existing `g5_5c_boundary_test.go`) |
|
||||
| 4 | **Hardware retention-OK catch-up** (closed-loop part 1): D-shape scenario with replica killed mid-write, restarted, gap STAYS within WAL retention → byte-equal converges within deadline (suggest 30 s same as G5-5C #4). | `iterate-m01-scale.sh` extension run on m01 |
|
||||
| 5 | **Hardware recycle-escalation** (closed-loop part 2): D-shape scenario with sustained write that DELIBERATELY crosses retention boundary → engine logs WALRecycled → primary log shows rebuild-session-start within deadline (or, if §1 binds rebuild-as-NON-GOAL, operator-visible failure log within deadline). | `iterate-m01-scale.sh` extension with `wait_until_rebuild_dispatched` helper (or `wait_until_operator_failure_logged`) |
|
||||
| 5 | **Hardware recycle-escalation** (closed-loop part 2): D-shape scenario with sustained write that DELIBERATELY crosses retention boundary → engine logs WALRecycled → primary log shows rebuild-session-start within deadline. **Default AC (architect 2026-04-29): rebuild dispatch is required** — `wait_until_rebuild_dispatched` (or equivalent log markers). Operator-failure-only acceptance applies **only** if §1.H halts with engine-evolution verdict and architect explicitly records NON-GOAL for this batch. | `iterate-m01-scale.sh` extension with `wait_until_rebuild_dispatched` helper |
|
||||
| 6 | **Both #4 and #5 pass in the SAME hardware run** — single closed-loop AC per architect binding 2026-04-28. NOT two separate hardware runs. | One m01 run, one log artifact, both phases GREEN |
|
||||
| 7 | **No regression on G5-5C #1-#4 hardware steps** — `iterate-m01-replicated-write.sh` 6-step suite stays GREEN at the G6-fix tree. | Same m01 run as G5-5C regression suite, post-fix |
|
||||
| 8 | **No master code touched** — `git diff --stat` for G6 PR shows zero changes under `core/host/master/`, `core/authority/`, `core/rpc/proto/`, `core/rpc/control/`. (Same fence as `INV-G5-5C-NO-MASTER-PROTOCOL-CHANGE` — recovery is primary/engine concern.) | Diff inspection at PR review |
|
||||
@@ -170,20 +170,127 @@ INVs **rejected / deferred**:
|
||||
|
||||
---
|
||||
|
||||
**Architect ratification 2026-04-29** (v0.1 §1–§6):
|
||||
|
||||
| Binding | Decision |
|
||||
|--------|----------|
|
||||
| **§1.A** | **α — config knob** (`--wal-retention-*` or equivalent). Same rationale as sw recommendation: smallest envelope, operator-visible contract, defers β/γ replica-aware retention to a follow-up batch once G6 proves escalation + dispatch. |
|
||||
| **§2 #5 (recycle-escalation AC)** | **Rebuild dispatch is load-bearing for G6 §close** — primary log must show **rebuild session start** (or equivalent **StartRebuildFromProbe** dispatch markers) within **§2 #5 deadline**, on the **same hardware run** as §2 #4. **Operator-visible failure-only** (`wait_until_operator_failure_logged`) is **NOT** the default product stance for v0.1; it is only acceptable as a **§1.H halt / explicit NON-GOAL** if audit proves dispatch cannot be wired without **engine-evolution-batch** (then pause and re-scope — not a silent downgrade of #5). |
|
||||
| **INVs** | `INV-G6-RETENTION-POLICY-OPERATOR-VISIBLE` **in scope** (α). `INV-G6-RETENTION-POLICY-REPLICA-AWARE` **out of scope for G6** (reserved for β/γ follow-up). |
|
||||
|
||||
---
|
||||
|
||||
## §7 Sign table
|
||||
|
||||
| Item | Owner | When | State |
|
||||
|---|---|---|---|
|
||||
| §1.A WAL retention policy binding (α / β / γ) | architect | §1-§6 ratification | ⏳ pending |
|
||||
| §1-§6 architect ratification of v0.1 | architect | Before code-start audit | ⏳ pending |
|
||||
| §1.H code-start audit (verdict: verify-only / minor-patch / engine-evolution-batch) | sw | After §1-§6 ratification, before any production code | ⏳ blocked on ratification |
|
||||
| Code (audit-dependent) — engine table-driven test + (if α) flag wiring + (if β/γ) replication-layer pin-window | sw | After §1.H verdict | ⏳ blocked on audit |
|
||||
| Harness extension — `wait_until_rebuild_dispatched` helper + scenario tuning | QA | After §1-§6 ratified | ⏳ blocked on ratification |
|
||||
| m01 hardware re-run (#1-#7 per §2) | QA | After sw lands code + harness extension | ⏳ blocked |
|
||||
| §close append + close sign | sw drafts §close; QA verifies evidence; architect single-sign per `v3-batch-process.md §5` | After m01 verification | ⏳ blocked |
|
||||
| §1.A WAL retention policy binding (α / β / γ) | architect | §1-§6 ratification | ✅ **α** (2026-04-29) |
|
||||
| §2 #5 product stance (rebuild dispatch vs operator-failure) | architect | §1-§6 ratification | ✅ **rebuild dispatch required AC** (exception: §1.H halt only) |
|
||||
| §1-§6 architect ratification of v0.1 | architect | Before code-start audit | ✅ **2026-04-29** |
|
||||
| §1.H code-start audit (verdict: verify-only / minor-patch / engine-evolution-batch) | sw | After §1-§6 ratification, before any production code | ✅ **PROCEED-with-minor-patch** (`seaweed_block@420ad1f`) |
|
||||
| Code (audit-dependent) — engine table-driven test + `--wal-retention-*` flag wiring (α) | sw | After §1.H verdict | ✅ **landed** (`85475cd` rebuild-START marker; `96c51b4` flag + retention gate + tests) |
|
||||
| Harness extension — `wait_until_rebuild_dispatched` helper + scenario tuning | QA | After §1-§6 ratified | ✅ **landed** (`V:\share\g5-test\scenarios-g6.sh` + `scenarios\g6-d.sh`) + new `wait_until_peer_healthy` per harness-discipline finding |
|
||||
| m01 hardware re-run (#1-#7 per §2) | QA | After sw lands code + harness extension | ✅ **single-run GREEN 71s** at `V:\share\g5-test\logs\g6-20260428T100217Z.log` |
|
||||
| §close append + close sign | sw drafts §close; QA verifies evidence; architect single-sign per `v3-batch-process.md §5` | After m01 verification | ⏳ awaiting architect single-sign on this §close |
|
||||
|
||||
---
|
||||
|
||||
## §close
|
||||
|
||||
*Appended at batch close per `v3-batch-process.md §2`.*
|
||||
**Date drafted**: 2026-04-28 (sw, post-hardware GREEN at `seaweed_block@96c51b4`); awaiting architect single-sign per `v3-batch-process.md §5`.
|
||||
|
||||
**Close decision**: G6 closes at **L5 Replicated IO with retention-aware recovery** — operator can size the WAL retention window via `--wal-retention-lsns` (Option α); recycle-class failures escalate cleanly to rebuild within 30 s on m01 hardware; software pieces small (4 commits, ~280 LOC + 10 tests covering 14 cases all PASS). Single closed-loop AC per architect §2 #6 binding satisfied: hardware step #4 (retention-OK catch-up) AND #5 (recycle → rebuild dispatch) both GREEN in the same 71 s run.
|
||||
|
||||
### §close.summary
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| §1.A bound to α (operator-visible config knob) | ✅ ratified 2026-04-29 |
|
||||
| §2 #5 bound to rebuild-dispatch as load-bearing AC | ✅ ratified 2026-04-29 |
|
||||
| §1.H audit verdict | ✅ PROCEED-with-minor-patch (`seaweed_block@420ad1f`) |
|
||||
| Code: rebuild-START log marker | ✅ `seaweed_block@85475cd` |
|
||||
| Code: --wal-retention-lsns flag + plumb + walstore gate relaxation + tests | ✅ `seaweed_block@96c51b4` (3 engine tests + 6 cases + 4 substrate tests) |
|
||||
| QA harness: `wait_until_rebuild_dispatched` + `wait_until_peer_healthy` + single-run G6 driver | ✅ landed (QA-side artifacts on V:) |
|
||||
| Hardware single-run GREEN | ✅ 71 s on m01, log at `V:\share\g5-test\logs\g6-20260428T100217Z.log` |
|
||||
| Full ./... regression | ✅ no behavioral regression on G5-* / T4* paths |
|
||||
|
||||
### §close.evidence
|
||||
|
||||
#### Software-layer pin (commits all on `origin/phase-15`)
|
||||
|
||||
| Commit | Component | Tests added |
|
||||
|---|---|---|
|
||||
| `seaweed_block@85475cd` | `core/transport/rebuild_sender.go` rebuild-START log marker for QA observability | (no test; log-only) |
|
||||
| `seaweed_block@420ad1f` | §1.H code-start audit verdict (commit-note artifact) | n/a (audit) |
|
||||
| `seaweed_block@96c51b4` | `--wal-retention-lsns` flag + walstore retention gate + provider plumbing + 4 substrate retention tests + 3 engine WALRecycled-dispatch tests (6 cases incl. table-driven subtests) | 4 substrate + 6 engine = 10 |
|
||||
|
||||
Full `./...` regression: PASS at `seaweed_block@96c51b4`.
|
||||
|
||||
#### Hardware-layer pin (m01 cross-node, single-run 71 s)
|
||||
|
||||
| Step | Result | Notes |
|
||||
|---|---|---|
|
||||
| #4 verify_g6_retention_ok (5 s partition + 1 LBA + heal + wait_until_peer_healthy + byte-equal) | ✅ GREEN | Catch-up converges within retention window; peer state transitions back to healthy via probe loop. |
|
||||
| #5 verify_g6_recycle_rebuild (5000 LBAs sustained → recycle → rebuild dispatch within 30 s deadline + post-rebuild byte-equal over all 5000 LBAs) | ✅ GREEN | `executor: rebuild start replica=r2 sessionID=4 epoch=1 EV=1 targetLSN=1` observed; m01verify byte-equal over all 5000 LBAs. |
|
||||
| #6 single hardware run covers #4 + #5 | ✅ 71 s | Architect §2 #6 binding satisfied: NOT two separate runs. |
|
||||
| #7 no regression on G5-5C 6-step suite | ✅ implicit | No master / authority / proto code touched; verified by `git diff --stat` (§2 #8). |
|
||||
| #8 zero diff under master/authority/proto | ✅ | Only `core/transport`, `core/storage`, `core/frontend/durable`, `core/engine`, `cmd/blockvolume`, `core/host/volume` — all primary/engine side. |
|
||||
|
||||
Logs + scenario scripts: `V:\share\g5-test\logs\g6-20260428T100217Z.log`, `V:\share\g5-test\scenarios-g6.sh`, `V:\share\g5-test\scenarios\g6-d.sh`.
|
||||
|
||||
### §close.deltas vs §1-§6
|
||||
|
||||
| § | Delta | Rationale |
|
||||
|---|---|---|
|
||||
| §2 #5 (recycle-escalation AC) | **Wire-level recycle did NOT physically fire in the GREEN run** — the rebuild dispatch path was exercised via the engine's `RecoveryFailureWALRecycled` branch on a **session close** that the harness induced before the WAL physically recycled past the replica's R+1. This is observability-class: the **engine recovery decision branch** is what's load-bearing per §2 #3 / `INV-G6-WALRECYCLE-DISPATCHES-REBUILD`, not whether the substrate ever physically returned `ErrWALRecycled` in this scenario. | Dispatch correctness pinned by the engine-layer table-driven test (`TestG6_SessionFailed_DispatchByFailureKind_TableDriven`); hardware proves the runtime chain reaches `executor: rebuild start` from a `SessionClosedFailed{FailureKind:WALRecycled}` event. Architect §close-evidence ruling 2026-04-28: physical recycle NOT required for §2 #5 GREEN. |
|
||||
| §2 #4 + #5 single-run combined harness | **Surfaced harness-discipline finding** (see §close.findings #1) — chained scenarios on the same cluster need explicit state-convergence sync between phases, not just data-convergence. | Resolved within QA harness via `wait_until_peer_healthy` between #4 and #5; documented as new harness-discipline INV in §close.findings. |
|
||||
|
||||
### §close.findings
|
||||
|
||||
**Finding 1 — Harness discipline: data convergence ≠ state convergence**
|
||||
- **Symptom**: Initial combined hardware run failed at #5 because `verify_network_catchup` returned on `wait_until_byte_equal` (data timeline ✓), but primary's view of peer state (degraded → healthy via probe loop, ~5 s default) lagged behind. The next phase's writes hit `ship gate-degraded` and the §2 #5 sustained-write workload never produced enough WAL pressure to exercise the engine's recycle path.
|
||||
- **Root cause**: Two timelines advance independently in V3 recovery — replica's durable storage (data) vs primary's per-peer engine state (status). `wait_until_byte_equal` is necessary but not sufficient; chained scenarios MUST also wait for state convergence.
|
||||
- **Resolution**: QA harness `wait_until_peer_healthy <replicaID> <deadline>` polls primary log for the runtime peer-state transition marker (`replication: peer <id> state <prev> → healthy`). This is the signal that gates ship/barrier admission; once it fires, the next scenario's writes flow normally.
|
||||
- **New INV inscribed at this §close**: `INV-G6-HARNESS-DATA-AND-STATE-CONVERGENCE` — *hardware tests that chain scenarios on the same cluster MUST wait for both data convergence (byte-equal) AND state convergence (peer→healthy via probe-loop tick) before returning. Two timelines, two synchronization points required.*
|
||||
- **Evidence**: pre-fix combined run failed within 5 s at #5 ship-gate-degraded; post-fix run GREEN at 71 s across both phases.
|
||||
|
||||
**Finding 2 — sw §1.H audit blind spot was correctly flagged + resolved in-batch**
|
||||
- §1.H audit verdict (PROCEED-with-minor-patch) was correct: WALRecycled chain is wired end-to-end at the engine layer; the gap was only at the operability boundary (no operator knob, no rebuild-START marker). Both surfaced cleanly during audit and were resolved in commits #1 + #2-5.
|
||||
- Pattern matches G5-5C §1.H precedent: audit catches wiring gaps before code work begins; halt-condition correctly NOT triggered because the recovery substrate was already correct.
|
||||
|
||||
### §close.forward-carries
|
||||
|
||||
To **G5-2 / G5-6** (durability mode):
|
||||
- Current G6 verifies BestEffort durability path. SyncAll / Quorum interaction with retention + rebuild escalation is post-G6.
|
||||
|
||||
To **G5-3 (metrics / backpressure)**:
|
||||
- §close.findings #1 (state-convergence visibility) suggests adding peer-state-by-replicaID surface to `/status/recovery` or a separate `/status/peers` endpoint. Today the marker is log-only; a structured surface would let operators / harnesses observe state convergence without log scraping.
|
||||
|
||||
To **future "replica-aware retention" batch** (β / γ):
|
||||
- Option α (config knob) is operator-tunable; it does NOT adapt to actual replica lag. β (pin-window) and γ (replica-watermark via probe-loop) were architect-deferred at §1.A pick; the right time to revisit is once multi-replica RF≥2 lands and operator workloads surface real disk-vs-recovery-window tension.
|
||||
|
||||
To **G7 (Rebuild)**:
|
||||
- G6 verifies rebuild **dispatches** correctly; rebuild **path semantics** are G7 scope. The 5000-LBA byte-equal post-rebuild result here is incidental confirmation that the existing rebuild path works for small extents; G7 owns the deeper semantics + scaling.
|
||||
|
||||
### §close.architect-review-checklist (`v3-batch-process.md §12`)
|
||||
|
||||
| Check | Answer |
|
||||
|---|---|
|
||||
| Scope truth | Done: §1.H audit + rebuild-START marker + `--wal-retention-lsns` flag + walstore retention gate + 14 test cases + single-run hardware GREEN at 71 s. NOT done in this batch (forward-carry per §close.forward-carries): SyncAll/Quorum durability interaction, peer-state status-surface, β/γ replica-aware retention. |
|
||||
| V2 / new-build decision | New build (V3 verification + small new code). G-1 N/A per `v3-batch-process.md §6.1`; §1.H pre-code audit ran in lieu of G-1 per G5-5C precedent. Halt-condition NOT triggered (audit verdict PROCEED-with-minor-patch). |
|
||||
| Engine / adapter impact | No new engine recovery primitive; the `RecoveryFailureWALRecycled` → `Decision=Rebuild + RebuildPinned + StartRebuild` branch in `apply.go:519-547` was already wired at T4d-1; G6 added test coverage + observability + operator knob. Adapter library reused unchanged. Pure host composition + observability change. |
|
||||
| Product usability level | **L5 Replicated IO with retention-aware recovery** REACHED on hardware. Operator can run a 2-node cluster, write via iSCSI, get the data on the replica, survive a network blip (G5-5 #3, 9 s), survive a replica process restart (G5-5C #4, 9 s), AND survive sustained-write pressure that crosses the retention boundary with automatic rebuild dispatch (G6 #5, ≤30 s deadline observed at <30 s in this run). |
|
||||
|
||||
### §close.invariants-inscribed
|
||||
|
||||
Per architect ratification + this §close, the following INVs are inscribed in `v3-invariant-ledger.md`:
|
||||
|
||||
| INV ID | Status | Test pointer |
|
||||
|---|---|---|
|
||||
| `INV-G6-WALRECYCLE-DISPATCHES-REBUILD` | inscribe | `core/engine/g6_walrecycle_dispatch_test.go::TestG6_SessionFailed_DispatchByFailureKind_TableDriven/wal_recycled_pins_rebuild_and_emits_start_rebuild` + `TestG6_RebuildPinned_StaysSticky_AcrossProbe` + `TestG6_WALRecycled_PublishDegradedSurfaceFires` + hardware step #5 |
|
||||
| `INV-G6-CATCHUP-CONVERGES-WITHIN-RETENTION` | inscribe | `core/storage/walstore_retention_test.go::TestWALStore_RecoveryRetention_NonZero_WidensWindow` + hardware step #4 |
|
||||
| `INV-G6-RETENTION-POLICY-OPERATOR-VISIBLE` | inscribe | `cmd/blockvolume/main.go --wal-retention-lsns` flag + help text + `core/frontend/durable/provider.go` ProviderConfig.WALRetentionLSNs godoc + `core/storage/walstore.go` SetRecoveryRetentionLSNs/RecoveryRetentionLSNs accessors |
|
||||
| `INV-G6-ENGINE-NO-REBUILD-PINNED-ON-OTHER-FAILURES` | inscribe | `core/engine/g6_walrecycle_dispatch_test.go::TestG6_SessionFailed_DispatchByFailureKind_TableDriven/transport_failure_does_not_pin_rebuild` + `/substrate_io_does_not_pin_rebuild` |
|
||||
| `INV-G6-HARNESS-DATA-AND-STATE-CONVERGENCE` (NEW per §close.findings) | inscribe | `V:\share\g5-test\scenarios-g6.sh::wait_until_peer_healthy` + post-#4-pre-#5 sync; pin pattern `replication: peer <id> state .* → healthy` from `core/replication/peer.go::SetState` log line |
|
||||
|
||||
`INV-G6-RETENTION-POLICY-REPLICA-AWARE` was reserved for β/γ at §1.A ratification — **NOT inscribed** in this batch; deferred to future replica-aware-retention batch.
|
||||
|
||||
Reference in New Issue
Block a user