Pins the architecture principle that V3 egress components (per-peer
shipper / session pump / flusher / barrier driver) must be modeled
as a single decision core with single-queue / single serializable
worker shape. Monotonic pointers (cursor, applied LSN, emit profile,
bound conn) advance only via the core's internal transitions; external
callers deliver commands/events; direct external mutation is a
design-debt side door.
Grounds the principle in three hardware-validated incidents on m01/M02
during 2026-05-02, all of which turn out to be the same shape:
§3.1 executor.Ship overwrites the WalShipper's emit context mid-
session (g7 #5: 498 LBA mismatches at concurrent-write range)
§3.2 PrimaryBridge onStart/onClose dropped ReplicaID; engine and
runtime peer state diverged (g7 #5/#6 dispatch never fires)
§3.3 A-class sender→coord RecordBarrierWalLegOk side-write
(reverted 2026-05-02 working tree per this principle)
Companion to v3-rebuild-from-lsn-pin-clarification.md. Anchors in
consensus: §I P1, §I P7, §6.8, INV-SINGLE.
No new wire field, predicate, or invariant; clarifies a rule that
earlier docs imply but don't articulate. Provides judgment criterion
for upcoming Ship/PushLiveWrite collapse and peer-state ownership
work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pins the rebuild path's `fromLSN=0` sentinel semantic for the current
`StartRebuild` signature. Closes the gap §I P7 calls out ("transport
silently overwriting `fromLSN := 0` violates parity") in the absence of
an engine-published `fromLSN` for rebuild.
Hardware-validated by seaweed_block@bc4286e g7-dual-lane on m01/M02:
G7-#2 PASS (dispatch=1s, complete=1s, total=2s, 1000 LBAs byte-equal).
Sentinel rule:
- Caller passes 0 ⇒ "rebuild — primary picks the pin"
- Transport translates: sessionFromLSN := targetLSN
- Receiver-visible fromLSN = targetLSN
- Future catch-up (engine surfaces fromLSN := replicaLSN): passthrough
Three transport mechanics that satisfy the rule without violating any
consensus invariant: sentinel translation in startRebuildDualLane,
cursor-caught-up shortcut in WalShipper.DrainBacklog (preserves the
recycle gate's <= strictness verbatim), SeedWalApplied at SessionStart
so base-only rebuilds satisfy the A-class TryComplete conjunct.
Anti-discipline: no new wire fields, predicates, or invariants. Memo
clarifies the meaning of an existing transport-layer constant.
Anchors: v3-recovery-algorithm-consensus.md §I P7 / §6.9 / §6.10;
recover-semantics-adjustment-plan.md §1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the WalShipper implementation mini-plan that bridges
v3-recovery-wal-shipper-spec.md to the seaweed_block layout (phased
PR rollout P0..P4, INV ↔ test mapping, reviewer checklist).
§10 P2d decision request — the architect-gated handoff:
P2c is closed (slice A / B-1 / B-2 merged on g7-redo/wal-shipper-impl).
The bridging senderBacklogSink owns the live-write buffer + flushAndSeal
under sinkMu; Sender.Run barriers as soon as sink.DrainBacklog returns;
Close/closeCh/liveQueue/drainAndSeal are deleted from Sender. Atomic-seal
contract migrates intact (capture-vs-reject from queueMu → sinkMu).
P2d is gated on a three-axis decision the architect must make before a
real transport.WalShipper sink can replace the bridging path:
1. Body format on the dual-lane port:
(A) MsgShipEntry payload (unify on legacy steady encoding), OR
(B) frameWALEntry payloads (teach WalShipper.Emit to encode), OR
(C) documented third (e.g. envelope byte).
2. Single applier owner:
recovery.Receiver vs transport replica handler.
3. Replay source of truth:
which encoding the on-disk WAL playback decoder reads.
§10 also lists pre-decision deliverables that can land in parallel:
adapter scaffolding (transport-side struct satisfying recovery.WalShipperSink
by duck typing) + integration tests for architect rules 1+2 (emit context
before StartSession; restore steady lineage after EndSession).
V2 wire-compat is gated separately per feedback_porting_discipline.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per repository policy: dev/design docs live in
seaweedfs/sw-block/design/, not in seaweed_block/docs/. Formal
product docs come later. This commit relocates the 7 recovery
design markdown docs (4 trunk-merged in seaweed_block phase-15;
3 in-flight on g7-redo branches) plus the 1 hardware canonical
YAML to sw-block/design/ with v3-recovery-* prefix to match the
existing naming pattern (v3-recovery-live-line-backlog-spec.md).
Companion cleanup: a follow-on PR on seaweed_block removes the
docs from docs/ (and the YAML from testrunner/scenarios/) — that
PR is the seaweed_block side of the relocation.
Files added:
v3-recovery-pin-floor-wire.md — was docs/recovery-pin-floor-wire.md
on seaweed_block phase-15 (PR #11+#16)
v3-recovery-wiring-plan.md — was docs/recovery-wiring-plan.md
(PR #13)
v3-recovery-execution-institution.md — was docs/recovery-execution-institution.md
v3-recovery-inv-test-map.md — was docs/recovery-inv-test-map.md
(PR #11/#14/#15)
v3-recovery-unified-wal-stream-kickoff.md — was docs/recovery-unified-wal-stream-kickoff.md
g7-redo/unified-wal-kickoff (v0.3)
v3-recovery-unified-wal-stream-mini-plan.md — was docs/recovery-unified-wal-stream-mini-plan.md
g7-redo/unified-wal-mini-plan (v0.2)
v3-recovery-dual-lane-canonical-runbook.md — was docs/recovery-dual-lane-canonical-runbook.md
g7-redo/hardware-canonical-paper
v3-recovery-dual-lane-canonical.yaml — was testrunner/scenarios/recovery-dual-lane-canonical.yaml
g7-redo/hardware-canonical-paper
Internal cross-references updated in-place via sed:
- docs/recovery-inv-test-map.md → v3-recovery-inv-test-map.md
- docs/recovery-pin-floor-wire.md → v3-recovery-pin-floor-wire.md
- docs/recovery-wiring-plan.md → v3-recovery-wiring-plan.md
- testrunner/scenarios/recovery-dual-lane-canonical.yaml →
v3-recovery-dual-lane-canonical.yaml
Hand-edits:
- runbook §1 companion-YAML link: was
"../v3-recovery-dual-lane-canonical.yaml" (parent dir from
seaweed_block/docs); now same-directory link in design/.
- runbook §8 §3.2 #3 reference: was relative to seaweed_block
memory file (../../.claude/...); rewritten to point to
v3-recovery-unified-wal-stream-kickoff.md §4 directly.
- mini-plan Q15: docs/archive/ wording updated to
sw-block/design/archive/.
Stages-of-evidence still readable from the docs themselves
(kickoff §11, mini-plan §10 resolution logs, inv-test-map row
versions). Original seaweed_block branches preserve git
history for the in-flight content; the cleanup PR closes them
once this lands.
NOTE: this commit does NOT include the user's unrelated
ongoing edits in feature/sw-block (M v3-batch-process.md,
M v3-dev-roadmap.md, M v3-phase-15-g6-mini-plan.md, etc.).
Those stay uncommitted for the user to handle separately.
QA's G7 pre-work surfaced a discrepancy: the v0.1 §harness-notes
pointed at `exec_rebuild_started` / `exec_rebuild_completed` as
the harness markers. Those are RecoveryLog event names (internal
Orchestrator.Log ring buffer, process-local) — NOT visible in
primary.log on hardware. Hardware harnesses can't scrape them
without a /recovery-log HTTP surface (G5-3 forward-carry).
Hardware-visible markers (corrected):
- START: `executor: rebuild start replica=<id> sessionID=<n>
epoch=<n> EV=<n> targetLSN=<n>`
from core/transport/rebuild_sender.go:41
(added at G6 #1, seaweed_block@85475cd)
- COMPLETE: `executor: rebuild complete, sent <n> blocks
(targetLSN=<n>)`
from core/transport/rebuild_sender.go:120
(pre-existing T4d-4 part B / earlier)
Both produced via log.Printf in rebuild_sender.go and routed to
the daemon's stdout/stderr stream (which iterate harness captures
to ${REMOTE_RUN_DIR}/logs/primary.log). Both are sessionID-
correlatable for chained-scenario filtering. The G6 hardware run
already proved the START marker pattern; COMPLETE follows the
same shape.
Files corrected:
- §2 #7 acceptance row (harness helper text)
- §2 entry-marker table row
- §3 risks "Ambiguous rebuild done vs peer healthy" row
- §harness-notes (full rewrite with v0.1 correction note +
marker table + RecoveryLog clarification + recommended helper
shape with sessionID filter)
Negative-references to RecoveryLog event names retained in
explanatory context (so future readers don't re-introduce the
mistake by reading the engine code in isolation).
QA pre-work artifact V:\share\g5-test\scenarios\g7-helpers.sh is
already written against the corrected literals; this commit
brings the §harness-notes source-of-truth into alignment.
Standing by for architect §1.A ratification (Q1 topology / Q2
fold-G6 / Q3 deadline / etc.) before §1.H code-start audit.
Per architect ruling 2026-04-28 + sw §close.appendix: D's WALRecycled
boundary finding is G6 territory, not a G5-5C reopener. Adding the
backlog ticket here so it doesn't get lost between G5-5C close and
G6 kickoff.
Ticket text + evidence pointer + cross-references all preserved
from the §close.appendix; this is the dev-roadmap-side mirror so
the ticket surfaces when planning G6 scope.
Standing by for architect final §close single-sign on G5-5C.
Per architect ruling 2026-04-28 on QA's expanded scenario report:
- A (capacity): 🐛 → ✅ already-fixed at seaweed_block@a250b52, INV inscribed.
- B (500 random LBAs over 65536-LBA volume): ✅ GREEN. Confidence
bump on dirty-map skew + ship order under random write pattern.
- C (kill replica mid-write-storm + restart + 200 LBAs converge):
✅ GREEN. Highest-signal recovery scenario in the expansion;
validates G5-5C peer-recovery trigger under load.
- D (5000-LBA sustained write → WALRecycled past replica LSN):
🐛 boundary finding. Architect: G6 territory, not G5-5C reopener.
Catch-up requires WAL retention; rebuild path is for gap-beyond-
WAL. Engine has dispatch-branch tests (Batch 4); runtime
escalation path under sustained pressure is G6 acceptance scope.
Doc updates:
- New §close.appendix table with all 4 scenario rows + dispositions.
- Semantic clarification on D — catch-up vs WAL recycle vs rebuild.
- §close.forward-carries gets a NEW G6 entry with backlog ticket
text, evidence pointer, cross-reference to INV-G5-5C-PROBE-BEFORE-
CATCHUP, and explicit non-reopener rationale.
- Logs + scenario script paths recorded for QA continuity.
§close substance unchanged: G5-5C gate (verify_restart_catchup
GREEN within 30 s) was met on the canonical case at
seaweed_block@712cbc47 + capacity addendum at a250b52. B/C are
strengthening, not gating; D is forward-carry.
Awaiting architect final §close single-sign on this tree.
Per architect ruling 2026-04-28 + sw addendum landing at
seaweed_block@a250b52: inscribe new INV in the ledger.
Statement: iSCSI/NVMe externally-visible volume capacity and block
size MUST derive from --durable-blocks × --durable-blocksize when
--durable-root is set, not silently fall back to frontend defaults
(DefaultVolumeBlocks=2048 × DefaultBlockSize=512 = 1 MiB). Without
this plumb-through, a daemon configured for N MiB durable storage
advertises a 1 MiB iSCSI/NVMe LUN and any workload above LBA 256
fails.
Test pointers: cmd/blockvolume/frontend_capacity_test.go (6 tests:
ProductOfBlocksAndBlockSize, RejectsZero, OverflowGuard,
IscsiHandlerCapacity, NvmeHandlerCapacity, FrontendDefaults_
StillReturn1MiB). Source-side: cmd/blockvolume/main.go::
computeFrontendVolumeSize flows into both iscsi.TargetConfig and
nvme.TargetConfig handler.
First introduced: P15 G5-5C addendum (P0 product fix).
Owner layer: host (binary, frontend wiring).
Last verified: 2026-04-28 (G5-5C addendum P0; m01 hardware re-
verification pending QA).
Status: ACTIVE.
Awaiting m01 hardware re-run for full §close ledger update.
Architect approved Option B 2026-04-27: absorb the hardware-revealed
gap into G5-5C as Batch #7 instead of carrying to G5-5D.
§1.I scope:
- core/host/volume/peer_command_executor.go (NEW, ~120 LOC)
- core/host/volume/peer_adapter_registry.go (NEW, ~100 LOC)
- core/replication/volume.go ConfigurePeerLifecycleHook (~30 LOC)
- core/host/volume/probe_loop_wiring.go router signature (~20 net)
- cmd/blockvolume/main.go registry wire-up (~20 net)
- ~10 new tests, ~250 LOC test code
INV INV-G5-5C-PER-PEER-ADAPTER-PER-PEER-ENGINE absorbed back
in-batch (was previously deferred to G5-5D in pre-architect-ruling
draft).
Pass criterion unchanged: m01 verify_restart_catchup GREEN within
30s deadline; #1-#3 regression GREEN in the same run.
§close updated: ceremony waits for Batch #7 land + hardware re-run;
G5-5C closes at full L4 in one shot.
m01 hardware run 3 at seaweed_block@ac9392d:
- #1 verify_cluster_ready ✅ GREEN
- #2 verify_byte_equal ✅ GREEN
- #3 verify_network_catchup ✅ GREEN (9s)
- #4 verify_restart_catchup ❌ RED (30s timeout)
Root cause (verified in code + log):
Primary log shows probe loop fired correctly post-restart and the
wire probe SUCCEEDED twice (R=2 S=1 H=3), but no StartCatchUp ever
dispatched. Engine apply.go:117-128 checkReplicaID drops events
whose ReplicaID doesn't match the adapter's tracked Identity —
cmd/blockvolume's host adapter tracks the PRIMARY'S OWN slot
(ReplicaID=r1), not peer r2. Probe results for r2 are correctly
dropped as wrong_replica.
Component test (Batch #6) passed because cluster.go's
WithEngineDrivenRecovery constructs c.primary.adapters[] — one
per peer. cmd/blockvolume only constructs ONE adapter for the
host's own slot. The component test exercised a different
(architecturally-correct) wiring than production has.
§1.H audit verdict was correct on engine SEMANTICS; it did not
extend to whether the production binary CONSTRUCTS per-peer engine
state. That layer was assumed; hardware revealed the assumption.
§close decision:
- G5-5C software pieces all sound, stay landed (50 unit + integ
tests PASS; full ./... regression PASS).
- Hardware finding carries to G5-5D — Per-peer adapter wiring for
primary-side recovery dispatch.
- G5-5D pass criterion = exact verify_restart_catchup case from
this run; seed evidence = sw-block/design/g5-artifacts/primary-fail.log.
- New INV to inscribe at G5-5D close:
INV-G5-5D-PER-PEER-ADAPTER-PER-PEER-ENGINE.
Doc updates:
- §close.evidence: hardware-pin row table filled with run 3 results.
- §close.deltas: 3 implicit assumptions surfaced.
- §close.findings: 2 findings (#1 per-peer adapter gap; #2 script
port-release race already fixed).
- §close.forward-carries: G5-5D added as named carry.
- architect-review-checklist: scope/audit/engine-impact/product
level all updated to reflect actual reached state (L3+, not L4).
Awaiting architect ratification of G5-5D scope at single-sign or
earlier; sw drafts G5-5D mini-plan once architect rules.
Per v3-batch-process.md §2: §close drafted as soon as software is
ready. Hardware row table left as TBD; sw fills evidence pointers
once iterate-m01-replicated-write.sh completes. Forward-carries +
deferred ledger pointers + architect-review-checklist all populated
based on G5-5C scope already in-batch.
Awaiting:
1. m01 hardware run completion → fill #1-#4 evidence rows
2. QA evidence verification → §close.deltas / findings if needed
3. architect single-sign per v3-batch-process.md §5 + §8C.2
Per v0.5 §1.H step 3, sw publishes audit findings as a commit note
before any G5-5C production code change.
AUDIT METHOD: greped seaweed_block/core/{engine,replication,adapter}
for the structural backing of each in-scope INV; cited apply.go +
state.go + replication/volume.go + adapter/adapter.go line numbers
as evidence.
PER-INV FINDINGS:
[1] INV-G5-5C-PRIMARY-RECOVERY-AUTHORITY-BOUNDED
Owner: core/replication/volume.go (ReplicationVolume.peers map)
Status: ✅ PASS. peers map is sole probe target collection;
UpdateReplicaSet is sole mutator and is master-fact-driven only.
Halt-cond cleared.
[2] INV-G5-5C-GENERATION-FENCE
Owner: core/engine/apply.go:132-166 (stale event rejection) +
state.go:24-32 (IdentityTruth.{Epoch, EndpointVersion} carrier)
Status: ✅ PASS. Engine rejects events with epoch < Identity.Epoch
or (epoch == AND ev < Identity.EndpointVersion). identityChanged
triggers wholesale Recovery reset (line 166-169). Fence is
carried on engine state, not re-derived per call site.
Halt-cond cleared.
[3] INV-G5-5C-SINGLE-INFLIGHT-PER-PEER
Owner: core/engine/state.go:144-151 (SessionTruth single-slot) +
apply.go phase-guards at 183/236/364/417/442/455/472/507/536
Status: ✅ PASS. ReplicaState.Session is one slot per peer.
Engine FSM handlers explicitly skip / reject when Phase is
PhaseStarting or PhaseRunning. apply.go:536 "Skip if a rebuild
session already exists" pinned. In-flight is engine-explicit,
not implicit. Halt-cond cleared.
[4] INV-G5-5C-PROBE-BEFORE-CATCHUP
Owner: core/engine/state.go:84-121 (RecoveryTruth) +
decide() probe-driven decision path
Status: ✅ PASS. RecoveryTruth.Decision is derived from R/S/H
(boundaries from probe), NOT from transport reachability.
Engine's RebuildPinned guard prevents stale auto-probe from
downgrading Rebuild back to CatchUp mid-flight (line 105-120).
Halt-cond cleared.
[5] INV-G5-5C-RECOVERY-BACKOFF
Owner: engine retry budget (state.go:91-103
RecoveryTruth.Attempts + RuntimePolicy.MaxRetries from T4c-3) +
NEW G5-5C runtime cooldown (5s base → 10s → 20s → 40s → 60s cap;
reset on success)
Status: ⚠ PARTIAL — engine has retry budget but no exponential
cooldown. G5-5C adds the cooldown as a primary-runtime policy on
top of engine retry budget. NOT an engine FSM change. Acceptable
under §1.H "minimum evolution" criterion. Halt-cond cleared.
[6] INV-G5-5C-STALE-ACK-NO-HEALTH-PROMOTION
Owner: core/engine/apply.go:766-789 (Healthy gate)
Status: ✅ PASS. Healthy = true requires three conjuncts:
(a) Recovery.Decision == DecisionNone, (b) Reachability.Status
== ProbeReachable, (c) Identity.Epoch <= Reachability.FencedEpoch.
A barrier ack with AchievedLSN < TargetLSN does not transition
SessionTruth, decide() does not flip Decision to None on
insufficient achieved LSN — Healthy stays false. Halt-cond
cleared.
OVERALL VERDICT: PROCEED.
All six in-scope INVs have their backing infrastructure in engine
(state.go + apply.go) or replication (volume.go). G5-5C is a runtime
wiring batch + small policy extension (backoff). No engine FSM
rewrite needed. No halt-condition fires; no engine-evolution
mini-plan required.
NEXT STEP: implement primary-side probe loop +
ReplicaPeer.ProbeIfDegraded() + lifecycle/cooldown/dispatch tests +
component test, all under core/replication/. Probe loop owned by
ReplicationVolume lifecycle per architect binding. Test method
names to be concretized as code-start commit-note addendum to §2.
This audit commit fulfills §1.H step 3 (audit findings published) +
§2 #15 (audit commit note before production code).
Architect single-signed §1-§6 at seaweedfs@ba7bd0ba4 2026-04-27 with:
- Option B trigger source (primary-side degraded-peer probe loop)
- Probe loop placement = core/replication/ owned by ReplicationVolume
- Master protocol unchanged
- §1.H code-start audit gate before code
This commit:
1. Records the single-sign in the doc header.
2. Adds a §1 scope-rule one-liner near the top so future readers find
the architect-bound boundary without re-reading the v0.1→v0.5 trail:
"master owns identity/topology; primary+engine own data recovery;
the protocol aligns the two via (PeerSetGeneration, epoch,
EndpointVersion) fences."
§1.A already bound Option B in v0.4; no flip needed there. No design
change. §1.H audit is the next sw step before any production code.
Architect framing 2026-04-27: enumerate ten protocol boundary rules
and address engine-evolution question.
Engine vs primary runtime vs master split:
- Engine owns: recovery FSM, single in-flight per peer,
generation/epoch fence, probe→decision, backoff/cooldown policy,
stale-ack-cannot-promote-health rule, recovery reason / projection
- Primary runtime/adapter owns: timer / degraded-peer loop, transport
probe execution, feeding probe result into engine, executing
engine-emitted commands, ReplicationVolume / ReplicaPeer connection
lifecycle
- Master owns: identity / topology / assignment / health observation
ONLY. No runtime recovery. No epoch bumps for short up/down.
Six in-scope boundary rules (#1, #2, #3, #4, #7, #8):
- #1 Admitted Peer Rule — already INV-G5-5C-PRIMARY-RECOVERY-AUTHORITY-BOUNDED
- #2 Generation Fence — NEW INV-G5-5C-GENERATION-FENCE
- #3 Single In-Flight Per Peer — NEW INV-G5-5C-SINGLE-INFLIGHT-PER-PEER
- #4 Probe Before Catch-Up — NEW INV-G5-5C-PROBE-BEFORE-CATCHUP
- #7 Backoff/Cooldown — NEW INV-G5-5C-RECOVERY-BACKOFF (extends v0.4
fixed-5s into 5s→10s→20s→40s→60s cap, reset on success)
- #8 Stale Ack Guard — NEW INV-G5-5C-STALE-ACK-NO-HEALTH-PROMOTION
(cross-refs G5-5 round-14 gate-degraded artifact)
Three forward-carries OUT of G5-5C (per §5):
- #5 Durability Mode Explicit → G5-2 / G5-6
- #6 RF Health Reporting Separate From Recovery → future master
observability batch
- #10 Status Surface (recovery reason, effective RF, last probe) →
G5-3 metrics/backpressure
One citation (#9 Replica-side lineage check): already enforced by T4
acceptMutationLineage gate; G5-5C cites, no new code.
§1.H code-start audit gate: sw audits per-INV current owner location
BEFORE writing any code. Halt-condition: if recovery FSM is embedded
in ReplicationVolume, fence is re-derived per call site, in-flight is
implicit, or stale-ack guard is missing — sw stops and re-scopes as
engine-evolution batch instead of layering ifs in core/replication/.
Audit findings published as commit note pre-code; PR includes
audit-summary.
§2 acceptance criteria: add #13 (stale-ack guard), #14 (backoff
progression), #15 (code-start audit). Acceptance count now 15
covering 7 INVs (6 new + reconnect orthogonality from v0.4.3).
Standing by for architect single-sign of v0.4.4.
Architect framing 2026-04-27 (sharpening v0.4.2): reconnect splits
along two orthogonal dimensions — connection recovery vs identity /
lineage change. Each axis has different protocol semantics; G5-5C
must handle both correctly.
Architect's protocol judgment points:
1. PeerSetGeneration only changes for identity / address / lineage
change. Brief disconnects / restarts / freshness flapping do NOT
bump generation.
2. Primary's degraded-peer loop only acts on currently-admitted peers
(§1.E reaffirmed).
3. After reconnect, primary still probes R/S/H — reconnect alone is
not assumed sufficient.
4. If a higher PeerSetGeneration arrives during reconnect / probe,
the in-flight recovery must stop or invalidate.
Changes:
- New §1.F with two cases:
Case 1 (identity unchanged): primary retries existing peer
descriptor; new sessionID minted (sessions are session-scoped, not
peer-scoped); probe R/S/H; catch-up / rebuild as needed; no master
re-emit needed. This is G5-5C's core path.
Case 2 (identity changed): existing UpdateReplicaSet T4a-5 path
(volume.go:229-246) tears down + recreates; in-flight aborts via
Close(); new peer with new lineage takes over.
- Misread guards documented: "primary keeps retrying old address
forever" rejected by Case 2 + §1.E (c); "master must bump on every
blip" rejected by Case 1 + §1.D.
- New INV-G5-5C-RECONNECT-ORTHOGONAL-AXES in §3.
- New §2 #11 (reconnect Case 1 — identity unchanged, no re-emit) and
§2 #12 (reconnect Case 2 — lineage bump mid-flight).
This is structural reaffirmation: the V3 code already does Case 2
correctly (T4a-5 teardown). Case 1 is what the probe loop adds. The
new tests pin both axes against future drift.
Standing by for architect single-sign of v0.4.3.
Architect framing 2026-04-27 (sharpening v0.4.1): §1.D ordering-
independence must NOT be misread as "primary may self-discover and
connect to any replica it sees on the network." Tighten with a
second protocol invariant.
Rule (architect verbatim): "Primary recovery loop may retry only peers
that were previously admitted by a master-issued assignment fact for
the current authority lineage."
Layering: master establishes identity once; primary owns retry /
recovery for that admitted peer until master revokes or changes the
assignment.
This is structurally true in V3 today (probe loop reads
ReplicationVolume.peers, which UpdateReplicaSet populates from master
facts) but v0.4.2 promotes it from implementation detail to protocol
invariant so future contributors don't widen the probe surface.
Changes:
- New §1.E with three scenarios:
(a) first-time replica join — disallowed without master fact
(b) brief outage + recovery (G5-5C core case) — allowed without
master re-emit
(c) epoch / assignment change — probe must stop; in-flight aborts
- Implementation requirement made explicit: ReplicaPeer.Close() must
abort in-flight probe synchronously.
- Authority alignment surface table: replicaID/epoch/EV → identity;
AssignmentFact.Peers → only legal probe targets;
PeerSetGeneration → existing lastAppliedGeneration guard preserved.
- New INV-G5-5C-PRIMARY-RECOVERY-AUTHORITY-BOUNDED in §3.
- New §2 #9 (authority-bounded targets test) and §2 #10 (lineage-
change-during-probe test).
- §1.A bound-shape Master-interaction row references §1.E.
- §1 Files peer.go row notes Close() must abort in-flight probe.
Standing by for architect single-sign of v0.4.2.
Architect framing 2026-04-27: when a replica goes down or recovers,
both the control-plane identity/health loop and the data-plane
governance loop receive feedback. Protocol must treat them as two
independent loops with no ordering dependency, alignment via durable
identity facts (replicaID/epoch/EV/peer address), and idempotency on
primary-side dispatch absorbing duplicate triggers.
This is a sharpening of v0.4, not a re-bind. Design unchanged:
Option B primary-side probe loop, no master protocol change.
Changes:
- New §1.D: explicit two-loop table, five ordering scenarios all
ending safe, anti-requirements (master re-emit NOT prerequisite,
primary recovery NOT blocked on master), idempotency guarantees,
future RF-health observability noted as different-batch scope.
- New INV-G5-5C-TWO-LOOPS-ORDERING-INDEPENDENT in §3 with test
pointer (peer_test.go simultaneous-fire test).
- New §2 #8 acceptance criterion: unit test exercising the
"simultaneous-fire" case (concurrent fact replay + concurrent
ProbeIfDegraded on same degraded peer; idempotent absorption).
Standing by for architect single-sign of v0.4.1.
Architect REVISE ruling 2026-04-27: bind trigger source to Option A
with both halves in scope (no split into G5-5B). Reject B and C.
QA review v0.1 flagged: master-side scope must be explicit; pin §5
evidence path.
Changes:
- §1.A: collapse three-option proposal to bound Option A. Make A1
(master-side observation-driven re-emission) and A2 (primary-side
recovery dispatch) explicit as two halves of one causal chain.
Record B/C rejection rationale for future reference.
- §1 Files: revise table with Side column (master/primary). Add
master-side rows (A1 re-emit logic + ObservationStore freshness
helper). Total estimate ~360 prod + ~150 test, split master ~90 /
primary ~120 / tests ~150.
- §2: rewrite criteria #1-#5 around bound Option A (drop per-Option
deadline language). Split #2/#3 into A1 master-side + A2
primary-side criteria. Hardware deadline at #5 stays 30s.
- §2 verifier note: file paths + test names pinned at code-start
(acceptable for v0.2 per QA review).
- §5: pin G5-5 seed evidence to actual artifact path
V:\share\g5-test\logs\artifacts-20260427T092858Z\primary-fail.log
(no future task — fact-pointer).
- §7: trigger-source binding row marked done (architect REVISE);
single-sign of v0.2 still pending.
- Header: v0.1 → v0.2 status note updated.
Standing by for architect single-sign of v0.2.
No code starts until single-sign.
Per architect single-sign of G5-5 §close (`seaweedfs@c78116fd2`):
(a) v3-dev-roadmap.md
- §3: G5 line note now mentions G5-5 closed at L3 + G5-5C carry-forward
- §4: G5-5 row → CLOSED (link to seaweedfs@c78116fd2); G5-5C row added
as next active gate with bound pass criterion
- §7: G5-5 close commit appended to recently-closed table
(seaweed_block@5c4718f + seaweedfs@c78116fd2, L3 reached, #4 carry)
(b) v3-phase-15-g5-5c-mini-plan.md (new) v0.1 kickoff
- §1 scope: peer recovery trigger after replica restart; reuse T4d-4
primitives (architect binding); no engine logic change
- §1.A: three trigger source options (A master observation, B periodic
probe, C transport reconnect) with tradeoffs; sw recommends A; final
pick deferred to architect ratification
- §2: 6 acceptance criteria, hardware step is exactly G5-5 #4
(verify_restart_catchup → GREEN with no harness changes)
- §3: 2 new INVs proposed (REPL-PEER-RECOVERY-TRIGGER-001 +
-NO-RETRIGGER-LOOP) + 2 deferred ledger pointers from G5-5 close
- §4: G-1 N/A (new build, no V2 PORT)
- §5: forward-carries from G5-5 §close all addressed
- §6: 5 risks tabled
- §7: sign table awaiting architect §1-§6 ratification including
trigger source pick
Standing by for architect ratification of trigger source binding.
No code starts until §1-§6 signed.
Architect's round-15 hygiene callout: §7 sign table still had
three pre-code 'blocked' rows after the real close-state rows
landed in the prior doc-fix commit. Pure leftover from before
the close-state update overwrote earlier rows but didn't delete
the trailing pre-code rows.
Removed:
- 'Code start (script + Go helper) ... blocked on ratification'
- 'm01 hardware verification run ... blocked'
- '§close append + close sign ... blocked'
Sign table now ends cleanly at the §close architect single-sign
pending row. Ready for sign.
pingqiu
