From ead22edcd5f7fd3bcbfb971b8f8f7c7867190a4a Mon Sep 17 00:00:00 2001 From: pingqiu Date: Sun, 26 Apr 2026 11:38:31 -0700 Subject: [PATCH] G5: surface binary T4-wiring as new G5-4 batch (was implicit; now explicit) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hand-off doc v0.3 + G5 kickoff v0.2: m01+M02 bring-up smoke surfaced that cmd/blockvolume binary lacks T4 replication wiring entirely. Sw-confirmed root cause: - --t1-readiness HealthyPathExecutor is primary-only by design - volume.Config.ReplicationVolume slot exists (host.go:73) with godoc "T4a-5 production wiring sets this" — but T4a-5 only added the field; the wiring NEVER landed - T4d-4 part B wired WithEngineDrivenRecovery() for component test framework (cluster.go:357-369), NOT for the binary - Result: V3 components compose end-to-end (proven by T4d HARD GATE #3); the production binary still constructs a primary-only data plane Sw confirmed this is real implementation work (150-300 LOC + design), not a 50-LOC quick patch. Four design decisions needed: 1. Role inference (assignment vs CLI flag vs topology) 2. Peer discovery (from AssignmentFact.Peers) 3. Listener lifecycle (--data-addr reuse + Stop) 4. Engine instantiation (one engine per volume) G5 kickoff revised to v0.2: - 5 batches → 6 batches (binary wiring promoted to G5-4) - G5-1/2/3 are NOT blocked by G5-4 (component framework already binds T4d-4 part B; QA scenarios + walstore cadence at component/primary-only scope can run in parallel) - G5-4 binary wiring: needs full governance loop (kickoff → architect ratify → mini-plan → architect ratify → G-1 → code). G-1 source: T4d-4 part B component framework as V3-native PORT - G5-5 m01 hardware first-light DEPENDS on G5-4 (script can't drive replica scenarios until binary supports replicas) - G5-6 G5-DECISION-001 resolution at G5 close (was G5-5 in v0.1) Co-Authored-By: Claude Opus 4.7 (1M context) --- sw-block/design/v3-phase-15-g5-kickoff.md | 34 ++++++++------ .../v3-phase-15-g5-m0102-bringup-handoff.md | 44 +++++++++++++++++-- 2 files changed, 61 insertions(+), 17 deletions(-) diff --git a/sw-block/design/v3-phase-15-g5-kickoff.md b/sw-block/design/v3-phase-15-g5-kickoff.md index e7c8ba094..45bda6a79 100644 --- a/sw-block/design/v3-phase-15-g5-kickoff.md +++ b/sw-block/design/v3-phase-15-g5-kickoff.md @@ -1,7 +1,11 @@ # G5 — Replicated Write Path Collective Close — Kick-Off (PROPOSAL) -**Date**: 2026-04-26 (v0.1 — QA-authored proposal post-T4 close) -**Status**: ⏸ DRAFT — awaiting architect ratification on §2 scope + §3 batch shape + §4 acceptance bar + §5 G5-DECISION-001 path choice +**Date**: 2026-04-26 (v0.2 — QA-authored revision post m01+M02 bring-up debug; new G5-4 batch surfaced) +**Status**: ⏸ DRAFT v0.2 — awaiting architect ratification on §2 scope + §3 REVISED batch shape (now 6 batches, was 5; new G5-4 binary T4 wiring promoted from "implicit assumption" to "explicit batch") + §4 acceptance bar + §5 G5-DECISION-001 path choice + +**Revision history:** +- v0.1 (2026-04-26 morning) — initial proposal post-T4d close +- v0.2 (2026-04-26 afternoon) — m01+M02 bring-up smoke surfaced binary T4 replication wiring gap (`cmd/blockvolume/main.go` has zero ReplicationVolume references; `host.go:73 ReplicationVolume` slot was added at T4a-5 but never wired). Promoted binary wiring to its own batch (G5-4) ahead of m01 hardware first-light (now G5-5). Sw confirmed 150-300 LOC + design work — see `v3-phase-15-g5-m0102-bringup-handoff.md` v0.3 §00 for full diagnosis. **Predicates met**: T4 batch series closed (T4a + T4b + T4c + T4d batches all signed; T4 T-end three-sign at T4d closure report `seaweedfs@2ee12b2c1`) **Reference**: `v3-phase-15-t4d-closure-report.md` §I forward-carries; `v3-phase-15-t4d-mini-plan.md` §5 G5-DECISION-001 named decision record; `v3-phase-15-t4c-closure-report.md` §H m01 deferral @@ -56,22 +60,26 @@ G5 closes when these collectively pass + the architect signs off on production-r ## §3 Proposed batch shape — architect ratification ask -QA proposal for G5 batch decomposition (smaller than T4 batches since G5 is integration / verification heavy, not new-feature heavy): +QA proposal v0.2 for G5 batch decomposition (6 batches; binary T4 wiring promoted to own batch): | Batch | Scope | LOC est | Owner | |---|---|---|---| -| **G5-1** Multi-replica concurrent live + recovery scenarios at component scope | Author the QA scenarios catalogued at T4d (#1–#9) as actual test code; add multi-replica RF=3 mixed-state scenarios; component framework primitives needed (`AssertNoPerLBARegression`, `AssertLaneIntegrity`, `RestartReplica`, etc. from T4d QA scenario catalogue §7) | ~50 prod (framework primitives) + ~600 tests (9 scenarios + RF=3 mixed-state + new framework primitives) | QA + sw (framework only) | -| **G5-2** walstore flusher cadence verification + tuning policy | Production write pressure smoke (sustained 10+ minute write load); identify flusher cadence under pressure; document operational tuning policy (knobs to expose? defaults? what operator should monitor?) | ~80 prod (instrumentation if needed) + ~150 tests (sustained-write + observability) | sw + architect | -| **G5-3** Minimal metrics/backpressure assessment | Document current observability surface (logs, status endpoint, etc.); identify minimal Prometheus-style metrics needed for production (catch-up progress, retry counters, error rates per kind, session duration); decide what's G5 must-have vs post-G5 | ~50 prod (metrics) + ~50 tests (assertion fences) | sw + architect | -| **G5-4** m01 hardware first-light + L3 integration | Author `iterate-m01-replicated-write.sh` script (mirroring `iterate-m01-nvme.sh` from T2); drive real 2-node primary↔replica via blockmaster + assignment; iptables-based mid-stream disconnect; verify catch-up + rebuild paths under real network | ~200 (bash script + Go test driver) | QA + sw | -| **G5-5** G5-DECISION-001 resolution + closure | Architect resolves Path A (persist Recovery state) vs Path B (rebuild from probe after restart); IF Path A: persistence implementation + tests; IF Path B: documentation of retry-budget-resets-on-restart semantic + operator awareness; G5 closure report drafted | varies by path | architect + sw + QA | +| **G5-1** Multi-replica concurrent live + recovery scenarios at component scope | Author the QA scenarios catalogued at T4d (#1–#9) as actual test code; add multi-replica RF=3 mixed-state scenarios; component framework primitives needed (`AssertNoPerLBARegression`, `AssertLaneIntegrity`, `RestartReplica`, etc. from T4d QA scenario catalogue §7). **Component-scope only — does not depend on G5-4 binary wiring** since component framework already binds T4d-4 part B `WithEngineDrivenRecovery()` directly. | ~50 prod (framework primitives) + ~600 tests | QA + sw (framework only) | +| **G5-2** walstore flusher cadence verification + tuning policy | Production write pressure smoke (sustained 10+ minute write load); identify flusher cadence under pressure; document operational tuning policy. **Primary-only path** — does not depend on G5-4 binary wiring. | ~80 prod (instrumentation if needed) + ~150 tests | sw + architect | +| **G5-3** Minimal metrics/backpressure assessment | Document current observability surface (logs, status endpoint, etc.); identify minimal Prometheus-style metrics needed for production; decide what's G5 must-have vs post-G5 | ~50 prod (metrics) + ~50 tests (assertion fences) | sw + architect | +| **G5-4** **NEW (v0.2): Binary T4 replication wiring** | Mirror `cmd/blockvolume/main.go` to T4d-4 part B's `WithEngineDrivenRecovery()` framework binding. Construct `ReplicationVolume + ReplicaPeer + ReplicaListener`, pass via `volume.Config{ReplicationVolume: ...}`. **Design decisions** (per sw round 2026-04-26): (a) role inference (assignment vs CLI flag vs topology); (b) peer discovery (from `AssignmentFact.Peers`); (c) listener lifecycle (`--data-addr` reuse + Stop); (d) engine instantiation (one engine per volume). **G-1 source**: T4d-4 part B component framework (`cluster.go:357-369`) is V3-native PORT reference. **Surfaced by m01+M02 bring-up smoke** — see `v3-phase-15-g5-m0102-bringup-handoff.md` v0.3 §00. | 150-300 prod + ~150 tests | sw | +| **G5-5** m01 hardware first-light + L3 integration **(was G5-4 in v0.1)** | Author `iterate-m01-replicated-write.sh` script; drive real 2-node primary↔replica via blockmaster + assignment; iptables-based mid-stream disconnect; verify catch-up + rebuild paths under real network. **DEPENDS on G5-4 binary wiring** — script can't drive replica scenarios until binary supports replicas. | ~200 (bash script + Go test driver) | QA + sw | +| **G5-6** G5-DECISION-001 resolution + closure **(was G5-5 in v0.1)** | Architect resolves Path A (persist) vs Path B (rebuild from probe); G5 closure report drafted. | varies by path | architect + sw + QA | -**Architect decision asks for §3:** -1. Confirm 5-batch shape vs alternatives (3-batch dense / 7-batch with mixed-state and rebuild-edge as separate)? -2. Confirm batch ordering — component-multi-replica → walstore-cadence → metrics → m01 → G5-DECISION-001 close? -3. Confirm G5-5 ordering — does G5-DECISION-001 resolution happen at the START of G5 (architect call upfront) or at G5 CLOSE (after evidence from G5-1/2/3/4)? +**Architect decision asks for §3 (v0.2 REVISED):** +1. Confirm 6-batch shape (was 5; G5-4 binary wiring promoted from "implicit assumption" to explicit batch). +2. Confirm batch ordering — G5-1/2/3 are all primary-only-OK and parallel-eligible; G5-4 must precede G5-5; G5-5 precedes G5-6. +3. Confirm G5-4 governance loop — same as T4 (kickoff → architect ratify → mini-plan → architect ratify → G-1 → code) given 150-300 LOC + design decisions. +4. Confirm G5-6 ordering — does G5-DECISION-001 resolution happen at START of G5 (architect call upfront) or at G5 CLOSE (after evidence from G5-1/2/3/4/5)? -QA recommendation: G5-5 at close, because evidence from G5-1/2/3/4 (especially walstore cadence + production write pressure findings) helps inform Path A vs Path B decision. +QA recommendation: G5-6 at close, because evidence from G5-1/2/3/4/5 (especially walstore cadence + m01 production write pressure findings) helps inform Path A vs Path B decision. + +**Key architectural insight (v0.2)**: G5-1/2/3 are NOT blocked by G5-4. The component framework already binds T4d-4 part B; QA scenarios can author + run at component scope without binary wiring. This means G5 work can start in parallel as soon as architect ratifies — sw works G5-4 binary wiring while QA authors G5-1 scenarios + sw measures G5-2 walstore cadence at component-or-primary-only scope. --- diff --git a/sw-block/design/v3-phase-15-g5-m0102-bringup-handoff.md b/sw-block/design/v3-phase-15-g5-m0102-bringup-handoff.md index 0e2555612..c6a502d37 100644 --- a/sw-block/design/v3-phase-15-g5-m0102-bringup-handoff.md +++ b/sw-block/design/v3-phase-15-g5-m0102-bringup-handoff.md @@ -1,13 +1,49 @@ # G5-4 m01+M02 Cluster Bring-Up — Hand-off to sw -**Date**: 2026-04-26 (v0.2 — RESOLVED via local single-node debug round) -**Status**: ✅ RESOLVED — root cause was missing `--expected-slots-per-volume 2` flag on blockmaster (default is 3); also documented secondary finding about non-primary replica not becoming "Healthy" via T1 path -**From**: QA (round 2026-04-26 cross-node smoke attempt → pivoted to local Windows debug) -**To**: sw (G5-4 framework owner) +**Date**: 2026-04-26 (v0.3 — bring-up resolved; binary T4-wiring gap surfaced as G5-4 implementation work) +**Status**: ✅ Bring-up gate RESOLVED via sw's `--expected-slots-per-volume` flag (`seaweed_block@f5de7c5`) + `seaweedfs@e21c68693` answer doc. **NEW finding**: cmd/blockvolume binary lacks T4a-T4d replication wiring — the `host.go:73 ReplicationVolume` slot was added at T4a-5 but **never wired in main.go**. This is real G5-4 implementation work (150-300 LOC + design), not a config patch. +**From**: QA (rounds 2026-04-26 — local debug → m01+M02 verification → binary-wiring gap discovery) +**To**: sw + architect (G5-4 implementation kickoff input) **Context**: G5-4 m01 hardware first-light per [g5-kickoff §3 batch G5-4](v3-phase-15-g5-kickoff.md). Skeleton script committed at `seaweed_block@eabafe8` (`scripts/iterate-m01-replicated-write.sh`). --- +## §00 v0.3 finding — binary T4 replication wiring is the actual G5-4 work + +After bring-up resolved (§0 below), QA verified the cross-node 2-node cluster on m01+M02 (`192.168.1.181` + `192.168.1.184`): + +``` +m01 primary status: {"VolumeID":"v1","ReplicaID":"r1","Epoch":1,"EndpointVersion":1,"Healthy":true} +M02 replica log: blockvolume: volume v1 authority is now r1@1 (not this replica r2); + recording supersede, not applying to adapter + blockvolume: durable open: frontend: volume not ready +``` + +Primary side fully Healthy; replica side stuck because **`cmd/blockvolume/main.go` has zero references to `ReplicationVolume / ReplicaListener / ReplicaPeer`**. + +**Sw-confirmed root** (round 2026-04-26): +- `--t1-readiness` is primary-only by design (`core/host/volume/healthy_executor.go:10-28` godoc) +- `volume.Config.ReplicationVolume` slot exists (`host.go:73`) with godoc *"T4a-5 production wiring sets this"* — but **T4a-5 only added the field; the wiring never landed** +- T4d-4 part B wired `WithEngineDrivenRecovery()` for the **component test framework** (`cluster.go:357-369`), NOT for the binary +- Result: V3 components compose end-to-end (proven by T4d HARD GATE #3); the production binary still constructs a primary-only data plane + +**Why this is real implementation work, not a quick patch** (per sw round 2026-04-26): + +| Design decision | Options | +|---|---| +| Role inference (primary vs replica) | (a) From `AssignmentInfo.PrimaryReplicaID == self.ReplicaID`; (b) CLI flag operator-declared; (c) topology lookup deterministic | +| Peer discovery | From `AssignmentFact.Peers` (already exists per T4a-5 P-refined); wire peer-set updater on assignment events | +| Listener lifecycle | `ReplicaListener` bind addr (probably existing `--data-addr`); Stop on shutdown | +| Engine instantiation | One engine per volume, fed by assignment subscription | + +**Estimate** (sw): 150-300 LOC + tests. **Real design work, not 50-LOC**. Same governance loop T4 used: kickoff → architect ratify → mini-plan → architect ratify → G-1 (vs T4d-4 part B component-framework wiring as V3-native PORT source) → code. + +**G5-4 kickoff revision needed** — see `v3-phase-15-g5-kickoff.md` revision (binary-wiring promoted to its own batch ahead of m01 hardware first-light, since the script requires the binary). + +**Acceptable interim**: primary-only smoke per sw's option C (G5 scenarios that don't need replica writes — e.g., walstore cadence under sustained primary load) can proceed without the binary wiring. Anything involving real cross-node replication waits. + +--- + ## §0 Resolution (added 2026-04-26 v0.2) **Root cause** (found via local Windows reproduction — m01/M02 was unnecessary for this debug):