FailoverDiagnostic now carries V2PromotionEnabled and V2PromotionReady
fields. MasterServer.FailoverDiagnosticSnapshot() enriches the failover
state diagnostic with rollout gate visibility so operators can confirm
whether the master is on V1, V2, or V2-fail-closed-placeholder mode.
Update phase-20.md: document default=false rollout policy (safe default
until proto regen enables evidence RPC, then flip to default true).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wire V2 promotion into production binary:
- Add --block.v2Promotion CLI flag on weed master (default false)
- MasterOption.BlockV2Promotion → NewMasterServer wires flag + querier
- defaultBlockVSQueryEvidence placeholder (returns explicit error until
proto regen on M01 enables gRPC evidence RPC)
Fix three fail-closed violations found by tester:
1. blockV2Promotion=true + nil querier now fails closed with explicit
log instead of silently falling back to V1
2. Partial evidence (any candidate query failed) now fails closed —
unreachable candidate may be the most durable, promoting from
incomplete evidence violates durability-first ordering
3. Clear EngineProjectionMode in applyPromotionLocked (already in
previous commit, verified in tests here)
2 new tests: NilQuerier_FailsClosed, PartialEvidenceFailure_FailsClosed.
Total T3 tests: 7, all pass. Existing V1 failover tests unaffected.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wire V2 promotion into the real master failover decision path:
promoteReplica() now dispatches to promoteReplicaV2() when
blockV2Promotion flag is true. V2 path queries each candidate for
fresh evidence via pluggable BlockPromotionEvidenceQuerier, selects
by CommittedLSN (durability-first), and fail-closes when no eligible
candidate exists. No silent fallback to V1.
Feature flag: blockV2Promotion bool on MasterServer. When false,
existing promoteReplicaV1() (health-score-first) is used unchanged.
Flag is explicit and observable, not a hidden rescue path.
Registry: add PromoteReplicaByServer() for V2 path where master
already knows the winner. Clear stale EngineProjectionMode in
applyPromotionLocked (complements T1 turnover fix).
T2 fix: fail-closed when V2 core projection is absent —
Eligible=false with reason "missing_engine_projection". CommittedLSN
from core used unconditionally (no WALHeadLSN overstatement).
5 T3 integration tests: higher CommittedLSN wins, all-ineligible
fail-closed, evidence-failure fail-closed, flag-off uses legacy,
epoch bump + assignment enqueue only after selection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
VS-side evidence handler (QueryBlockPromotionEvidence) reads live
blockvol.Status() + V2 core projection at call time. Fail-closed:
no core projection → ineligible with reason "missing_engine_projection".
Engine CommittedLSN used unconditionally when core present (no WALHeadLSN
overstatement). Eligibility owned by local V2 engine, not master.
Master-side selection (selectDurabilityFirstCandidate): durability-first
ordering by CommittedLSN, tie-break WALHeadLSN then HealthScore. All
ineligible → fail-closed, no promotion. Pluggable querier
(BlockPromotionEvidenceQuerier) for T3 wiring.
Proto messages added to volume_server.proto. gRPC transport binding
pending proto regen on M01 — this commit delivers evidence semantics
and selection substrate, not full end-to-end RPC closure.
Phase 20 doc updated with T2-T5 reviewer packs and cross-task guardrails.
13 tests: live facts, core projection mode, fail-closed no-core, 4 gated
modes, missing volume, epoch mismatch, CommittedLSN ordering, WALHeadLSN
tie-break, HealthScore tie-break, all-ineligible, mixed collection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add engine_projection_mode as a distinct proto/wire/registry field
that carries pure V2 engine-derived local projection mode from VS
to master. Reads ONLY from CoreProjection — no ad-hoc fallback.
Separate from existing VolumeMode: EngineProjectionMode is VS-local
V2 engine truth, VolumeMode is the existing field that conflates V2
and V1 paths. Both exist during transition; only EngineProjectionMode
is V2-authoritative.
Clears stale value on primary turnover: when a newly promoted primary
heartbeats without the field, the old primary's projection is not
preserved (prevents synthetic master-side truth).
5 focused tests: propagation, distinctness (hard assertion), backward
compat preservation, turnover-clears, turnover-with-field.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Live HTTP evidence transport, continuous Loop2 service, bounded auto
failover trigger, runtime-managed frontend export, bounded replica
repair, end-to-end RF2 handoff with continued I/O on new primary,
bounded operator HTTP surface, and CSI V2 runtime backend adapter.
11 new proof tests covering the full M6-M10 chain plus CSI create/
lookup/publish through the V2 runtime path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Freeze the first bounded pilot/preflight/stop/rollout-review artifact set and sync the global product ledgers so productionization can start from an explicit chosen-envelope discipline instead of ad hoc rollout judgment.
Made-with: Cursor
Freeze the first Phase 17 branch/contract/policy/envelope package, add review and supported-matrix artifacts, and sync the product-completion and claim-evidence ledgers to the new bounded post-Phase-16 checkpoint.
Made-with: Cursor
Bind non-authoritative inventory, restart primary-truth rebasing, and sparse replica readiness retention into the heartbeat/master seam, and package the bounded finish-line checkpoint with explicit claims, non-claims, and proof commands.
Made-with: Cursor
Carry explicit volume_mode_reason across the heartbeat/master/API seam so outward surfaces retain the bounded core-owned explanation behind mode transitions.
Made-with: Cursor
Use ReplicaEligible instead of PublishHealthy in the heartbeat collector test now that publish health is rebound to publication truth rather than receiver readiness.
Made-with: Cursor
Make the heartbeat/master boundary preserve explicit volume_mode truth so master consume no longer reconstructs outward mode only from secondary heartbeat signals. Keep backward compatibility by falling back to the previous reconstruction when older heartbeats do not send the field.
Made-with: Cursor
Make the heartbeat/master boundary preserve explicit publish_healthy truth so master consume no longer reconstructs healthy publication only from secondary readiness and degraded heuristics. Keep backward compatibility by falling back to the previous reconstruction when older heartbeats do not send the field.
Made-with: Cursor
Make the heartbeat/master boundary preserve explicit needs_rebuild truth so primary heartbeat consume no longer collapses that stronger mode into a generic degraded signal. Keep backward compatibility by falling back to the previous heuristic when older heartbeats do not send the field.
Made-with: Cursor
Make the heartbeat/master boundary carry explicit replica readiness truth so the registry no longer depends only on replica transport-address presence as a readiness proxy. Keep backward compatibility by falling back to the old address heuristic when older heartbeats do not send the field.
Made-with: Cursor
Move removed-replica drain and replica-scoped invalidation onto explicit core-command paths so the widened multi-replica runtime no longer depends on coarse host-side recovery handling.
Made-with: Cursor
Emit one core-owned start_recovery_task per primary catch-up replica so the bounded multi-replica startup path no longer depends on a single-replica assumption.
Made-with: Cursor
Track catch-up observations per replica so the volume-level recovery view stays in catching_up until all bounded replicas complete. This preserves the current bounded semantics while removing an overclaim that would block later multi-replica startup ownership work.
Made-with: Cursor
Carry replica-scoped addressing through bounded recovery planning and completion events so the core no longer depends on a volume-only observation seam. This preserves the current single-replica catch-up and rebuilding behavior while aligning the observation side with the replica-scoped command path.
Made-with: Cursor
Replace the remaining volume-scoped recovery command and pending slot
with replica-scoped addressing on the bounded core-present path. This
preserves the current single-replica catch-up and rebuilding behavior
while removing the structural blocker for later multi-replica startup
ownership.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move dispatcher-facing host effects out of volume_server_block.go into
blockcmd while keeping server-owned cache/state semantics in weed/server.
Document Batch 10 delivery and Batch 11 stop-line review so the
separation line closes without over-extracting readiness-state mutation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move BlockVol-backed command bindings into v2bridge and move non-BlockVol
command operations into weed/server/blockcmd. This keeps dispatch and host
effects in weed/server, keeps backend binding in v2bridge, and further
shrinks volume_server_block.go toward a host shell while preserving
current command-driven proofs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
resolveRecoveryContext now also derives rebuildAddr from assignments,
so the full host-side recovery context is resolved in one call:
- volPath (from replicaID)
- rebuildAddr (from assignments via deriveRebuildAddr)
- recovery bindings (driver + executor via BuildRecoveryBundle)
- replicaFlushedLSN (from sender session)
startTask/runRecovery/runCatchUp/runRebuild now pass assignments
instead of rebuildAddr. No separate rebuildAddr resolution remains
outside the resolver.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New recoveryContext type + resolveRecoveryContext method consolidates:
- volumePathForReplica (volPath from replicaID)
- v2bridge.BuildRecoveryBundle (driver + executor from BlockVol)
- sender/session lookup (replicaFlushedLSN for catch-up start)
runCatchUp and runRebuild now read as:
resolve → plan → branch (legacy or core-present)
Removed buildRecoveryBundle (inlined into resolveRecoveryContext).
block_recovery.go no longer has any inline context assembly —
it is now a pure orchestration shell.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New v2bridge.BuildRecoveryBundle(vol, rebuildAddr) assembles all
recovery bindings (Reader + Pinner + StorageAdapter + Executor) from
a real BlockVol instance in one call.
block_recovery.go changes:
- Removed local recoveryBundle type
- buildRecoveryBundle now delegates to v2bridge.BuildRecoveryBundle
inside WithVolume, returns (driver, executor, err)
- Removed direct v2bridge.NewReader/NewPinner/NewExecutor construction
- Removed bridge import (no longer needed)
- runCatchUp/runRebuild use (driver, executor, err) directly
block_recovery.go no longer knows how to construct Reader, Pinner,
StorageAdapter, or Executor. It only knows: resolve volPath, ask the
factory for bindings, plan, branch to legacy or core-present path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Step 2: Rebuild completion status port
- New runtime.RebuildCompletionStatus + DeriveRebuildCommitted:
reusable shaping logic for post-rebuild snapshot → RebuildCommitted event
- block_recovery.go OnRebuildCompleted: delegates to DeriveRebuildCommitted,
host only reads raw snapshot via readRebuildStatus (thin binding)
- Removed 15 lines of inline flushedLSN/checkpointLSN/achievedLSN computation
Step 3: Recovery bundle factory
- New buildRecoveryBundle: shared host-side setup for both catch-up and rebuild
(creates Reader + Pinner + StorageAdapter + Executor + RecoveryDriver)
- runCatchUp and runRebuild both use buildRecoveryBundle instead of
duplicating the WithVolume → NewReader → NewPinner → NewStorageAdapter →
NewExecutor → RecoveryDriver chain
- runCatchUp/runRebuild are now thin host-shell methods
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace interface{} fields in runtime.PendingExecution with typed handles:
- Driver: *engine.RecoveryDriver (was interface{})
- Plan: *engine.RecoveryPlan (was interface{})
- CatchUpIO: engine.CatchUpIO (was interface{})
- RebuildIO: engine.RebuildIO (was interface{})
block_recovery.go:
- ExecutePendingCatchUp/Rebuild: direct field access (pe.Driver, pe.Plan)
instead of type assertions (pe.Driver.(*engine.RecoveryDriver))
- CancelFunc: pe.Driver.CancelPlan(pe.Plan, reason) — no casts
- 6 type assertions removed from production path
Test files: remove Plan type assertions — fields are typed end-to-end.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
H wiring: block_recovery.go now uses runtime.PendingCoordinator
- Removed local pendingRecoveryExecution type + store/take/peek/has/cancel
- ExecutePendingCatchUp/Rebuild delegate to coord.TakeCatchUp/TakeRebuild
- Shutdown uses coord.CancelAll
- Added CancelAll to PendingCoordinator
I wiring: executeCatchUpPlan/executeRebuildPlan replaced
- ExecutePendingCatchUp now calls rt.ExecuteCatchUpPlan with RecoveryManager
as RecoveryCallbacks (OnCatchUpCompleted/OnRebuildCompleted)
- ExecutePendingRebuild follows same pattern
- Local executeCatchUpPlan/executeRebuildPlan methods removed
J structural: legacy no-core branches extracted
- executeLegacyCatchUp: wraps rt.ExecuteCatchUpPlan for v2Core==nil path
- executeLegacyRebuild: wraps rt.ExecuteRebuildPlan for v2Core==nil path
- Clear "LEGACY NO-CORE COMPATIBILITY" section with structural separation
- runCatchUp/runRebuild now branch cleanly: legacy helper vs core coordinator
Test updates: pendingRecoveryExecution → rt.PendingExecution, field casing,
Plan type assertions.
Validation: all P4, P16B, and ApplyAssignments tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add explicit "LEGACY NO-CORE COMPATIBILITY" section header in
block_recovery.go marking HandleAssignmentResult and
HandleRemovedAssignments as compatibility-only entry points.
The comment block explicitly states:
- These are for pre-Phase-16 no-core paths and older tests
- Core-present paths use StartRecoveryTask + ExecutePending*
- These should NOT be strengthened into semantic-authority proofs
No behavioral change — structural labeling only. All validation passes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New reusable pending-execution coordinator with fail-closed command matching:
- Store/TakeCatchUp/TakeRebuild/Cancel/Has/Peek
- TakeCatchUp: fail-closed on target LSN mismatch (cancel + return nil)
- TakeRebuild: same fail-closed semantics
- Cancel callback invoked on mismatch or explicit cancellation
9 tests prove boundary behavior:
- match succeeds, mismatch cancels, explicit cancel, noop on empty,
peek non-destructive, store replaces, take from empty
No weed/ imports. Pure coordination logic reusable by any adapter shell.
weed/server/block_recovery.go rebinding deferred to Task I.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Task F (Pinner):
- block_recovery.go: removed pinnerShimForRecovery (11 lines of pure
pass-through). v2bridge.Pinner structurally satisfies bridge.BlockVolPinner
(same method signatures), so it's passed directly.
Task G (Executor):
- Already clean. v2bridge.Executor is used directly without any shim —
structurally satisfies engine.CatchUpIO and engine.RebuildIO.
No code changes needed.
After Task E+F+G: zero shim types remain in block_recovery.go.
v2bridge Reader/Pinner/Executor all satisfy sw-block contracts directly.
Validation:
- go test ./weed/storage/blockvol/v2bridge/ -run "TestPinner_|TestExecutor_|TestBridge_" → PASS
- go test ./weed/server/ -run "TestP4_|TestP16B_" → PASS (8 tests)
- go test ./sw-block/bridge/blockvol/... → PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reader backend-binding extraction:
- v2bridge/reader.go: Reader.ReadState() now returns bridge.BlockVolState
directly instead of a local v2bridge.BlockVolState mirror type.
Removed the local BlockVolState type entirely.
- block_recovery.go: removed readerShimForRecovery (12 lines of 1:1
field copying). Reader is now passed directly as bridge.BlockVolReader.
Before: v2bridge.Reader → v2bridge.BlockVolState → readerShim → bridge.BlockVolState
After: v2bridge.Reader → bridge.BlockVolState (direct)
v2bridge now imports sw-block/bridge/blockvol for the contract type
(control.go already did this, reader.go now follows the same pattern).
Validation:
- go test ./sw-block/bridge/blockvol/... → PASS
- go test ./weed/storage/blockvol/v2bridge/ -run "TestReader_" → PASS
- go test ./weed/server/ -run "TestP4_|TestP16B_" → PASS (8 tests)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove direct fmt.Sprintf identity construction from v2bridge/control.go.
Both convertReplicaAssignment and convertRebuildAssignment now use:
- bridge.ReplicaAssignmentForServer (canonical ReplicaID derivation)
- bridge.RecoveryTargetForRole (canonical role → SessionKind mapping)
Before: 3 call sites with inline fmt.Sprintf("%s/%s", vol, server)
After: 0 — all identity construction goes through sw-block canonical helpers
volume_server_block.go already used bridge helpers (no change needed).
Validation:
- go test ./sw-block/bridge/blockvol/... → PASS (10 tests)
- go test ./weed/storage/blockvol/v2bridge/ -run "TestControl_|TestBridge_" → PASS (7 tests)
- go test ./weed/server/ -run "TestBlockService_ApplyAssignments_RebuildingRole_" → PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
16B widened from catch-up-only to catch-up + rebuild:
- StartRebuildCommand: core emits rebuild command, adapter executes
- Fail-closed: pending rebuild does not run without fresh command
- Recovery observations close back into core projection
New proofs:
- StartRebuildCommand_ConsumesPendingPlanAndUpdatesProjection
- RunRebuild_FailClosedWithoutFreshStartRebuildCommand
Review docs:
- phase-16-rev3-review.md: widened 16B review object
- phase-16-rev3-manager-rereview.md: manager challenge response
- phase-16-checkpoint-review.md: updated
Non-claims: not full recovery-loop closure, not end-to-end
failover/publication, not launch readiness.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Make the first V2 core owner explicit in sw-block by freezing Phase 14 docs, mode/readiness/publication semantics, and bounded command emission rules. This turns accepted Phase 13 constraints into executable core behavior without overclaiming live runtime cutover.
Made-with: Cursor
Add computed VolumeMode to BlockVolumeEntry with 5 normalized modes:
- allocated_only: RF=1, no replicas (standalone)
- bootstrap_pending: RF>1 but replicas not yet ready (first-write pending)
- publish_healthy: all replicas ready, no transport degradation
- degraded: replication impaired but recoverable
- needs_rebuild: unrecoverable gap, rebuild required
Code changes:
- master_block_registry.go: computeVolumeMode() called from
recomputeReplicaState(), VolumeMode field on BlockVolumeEntry
- master_server_handlers_block.go: VolumeMode exposed in REST API
- blockapi/types.go: VolumeMode field in VolumeInfo
- testrunner types: VolumeMode for scenario assertions
7 tests prove mode normalization:
- AllocatedOnly, BootstrapPending (2 cases), PublishHealthy,
Degraded, NeedsRebuild, SurfaceConsistency (transition proof)
Interpretation rule: current integrated tests validate V1 runtime
under V2 constraints, not a completed V2 runtime (Phase 14 scope).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bug: After failover promotes a replica to primary, the old primary
re-registers via heartbeat as a replica (lower epoch). But the master
never sent an updated Primary assignment to the new primary with the
re-registered replica's addresses. The new primary had 0 shippers →
replication dead. sync_all barrier passed vacuously.
Root cause: upsertServerAsReplica (heartbeat reconciliation) added the
re-registered server to Replicas[] but didn't (a) populate DataAddr/
CtrlAddr from heartbeat info, or (b) trigger a primary assignment
refresh.
Fix:
- master_block_registry.go: upsertServerAsReplica now copies DataAddr/
CtrlAddr from heartbeat info and sets NeedsPrimaryRefresh flag.
UpdateFullHeartbeat returns HeartbeatResult with PrimaryRefreshNeeded
entries. DrainPrimaryRefreshNeeded collects and clears the flag.
- master_block_failover.go: add enqueuePrimaryRefresh — builds a
Primary assignment with all current replica addresses and enqueues it.
- master_grpc_server.go: heartbeat handler processes PrimaryRefreshNeeded
entries after UpdateFullHeartbeat.
Gate test: TestPromote_AssignmentHasReplicaAddrs now PASSES —
after promote + re-register, the new primary gets an assignment with
replicaDataAddr=vs1:14260 and replicaAddrs=1.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
P0 bug on real hardware: assignments are re-delivered every heartbeat
cycle (5s). First setupReplicaReceiver succeeds (receiver starts on
deterministic port). Second call fails with "bind: address already in
use" because the listener is already bound. The volume stays permanently
degraded, blocking all RF=2 sync_all replication.
Fix: skip StartReplicaReceiver if v.replRecv is already set. The
receiver only needs to start once per volume lifetime.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Investigation result:
- Dual-BlockVol hypothesis: DISPROVEN (one instance per path, correct wiring)
- Root cause: adapter wiring bug in test allocator
soak_test.go blockVSAllocate returned ReplicaDataAddr = "vs2:9333:14260"
(server + ":port" where server already has a port → three colons, invalid)
This caused setupReplicaReceiver to fail silently → no data replicated
Root cause classification: adapter/test-harness bug
- NOT a backend data visibility bug
- NOT a core-rule gap
- The engine read path works correctly (TestSyncAll_FullRoundTrip passes)
Code changes:
- qa_block_soak_test.go: fix allocator to use host:port (not server:port),
use deterministic FNV-hashed ports matching production ReplicationPorts
- qa_block_cp13_8a_test.go: 2 new integration tests proving replica reads
work through both ReadLBA and adapter.ReadAt, before and after promotion
Remaining contradiction for CP13-8 scenario on real hardware:
- The production weed cluster uses ReplicationPorts (deterministic) which
should not have this bug. If CP13-8 still fails on m01/M02, the cause
is different from this test-harness issue and needs a separate investigation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
pingqiuandClaude Opus 4.6
pingqiu