mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-27 19:37:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
13,129 Commits 177 Branches 357 Tags
600dac602938af3eca1dfea197bb8dca8a71d366
Commit Graph
44 Commits
Author SHA1 Message Date
pingqiuandClaude Opus 4.6 600dac6029 feat: Phase 13 CP13-1 — frozen test-first baseline for sync replication gaps 
Baseline report (phase-13-cp1-baseline.md) from running 44 existing
replication-gap tests on current code with zero protocol changes:

  37 PASS / 4 FAIL / 3 PASS*

4 FAILs expose real gaps:
- ReconnectUsesHandshakeNotBootstrap: degraded shipper doesn't catch up (CP13-5)
- CatchupMultipleDisconnects: repeated reconnect cycles don't recover (CP13-5)
- NeedsRebuildBlocksAllPaths: stays Degraded after large gap (CP13-5+7)
- CatchupDoesNotOverwriteNewerData: catch-up fails at barrier (CP13-5)

3 PASS* are witness-only (pass but don't prove the property):
- Bug3_ReplicaAddr: documents gap, not fix (CP13-2)
- GapBeyondRetainedWal: asserts barrier failure, not NeedsRebuild (CP13-7)
- MaxBytesTriggersNeedsRebuild: logs "not implemented" (CP13-6)

No protocol code changed. Baseline is test-first evidence only.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 17:07:21 -07:00
pingqiuandClaude Opus 4.6 c0a805184f chore: archive superseded V2 design docs 
Copies of design docs removed in Phase 09, preserved in sw-block/docs/archive/
for historical reference.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 16:26:34 -07:00
pingqiuandClaude Opus 4.6 bdf20fde71 feat: Phase 12 — production hardening (disturbance, soak, testrunner scenarios) 
P1 Disturbance: restart/reconnect correctness tests — assignment delivery
  through real proto → ProcessAssignments, epoch validation on promoted
  volume, mandatory reconnect assertions

P2 Soak: repeated create/failover/recover cycles with end-of-cycle truth
  checks, runtime hygiene (no stale tasks/entries), steady-state idempotence

Testrunner recovery actions + scenarios:
- recovery.go: wait_recovery_complete, assert_recovery_state, trigger_rebuild
- 8 new YAML scenarios: baseline (failover/crash/partition), stability
  (replication-tax, netem-sweep, packet-loss, degraded), robust shipper

HA edge case and EC6 fix tests for regression coverage.

(P3 diagnosability + P4 perf floor committed separately in 643a5a107)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 16:26:17 -07:00
pingqiuandClaude Opus 4.6 bdf83e350e feat: Phase 11 — product-surface rebinding (snapshot, CSI, publication, restore) 
P1 Snapshots: CoW snapshot lifecycle through V2 engine path, create/list/delete
  via master RPC, BaseLSN tracking in manifest, ImportSnapshotForRebuild

P2 CSI Lifecycle: masterServerBackend calling real MasterServer in-process,
  CreateVolume/DeleteVolume/ExpandVolume through CSI → master → VS flow,
  ExportedControllerServer/ExportedNodeServer for cross-package testing

P3 Publication: LookupBlockVolume coherence across failover, iSCSI + NVMe
  address switching on promotion, repeated lookup self-consistency

P4 Restore: RestoreBlockSnapshot RPC through master and volume server,
  snapshot restore with runtime convergence, epoch/role validation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 16:25:58 -07:00
pingqiuandClaude Opus 4.6 3ec8fab2f1 feat: Phase 10 — control-plane closure (identity, convergence, idempotence) 
Stable identity on wire:
- ServerID fields in proto (replica_server_id, server_id on ReplicaAddrMessage)
- volumeServerId wired through volume.go → BlockService.SetServerID
- Identity derived from canonical server ID, not transport addresses

Assignment convergence:
- V2 idempotence via lastAppliedAssignment.equals (full replica set comparison)
- setupPrimaryReplication/Multi idempotence guards
- ProcessAssignments with V2 + V1 dual-path assignment handling

Master-driven control loop:
- RecoveryManager: serialized cancel-and-drain via done channels
- Per-replica heartbeat state reporting (ReplicaShipperStatus)
- masterServerBackend: VolumeBackend calling real MasterServer in-process
- RestoreBlockSnapshot RPC (master + volume server proto)

QA tests (P10 P1-P4):
- Identity: ServerID on wire, fail-closed on missing
- Convergence: assignment delivery, epoch monotonicity, registry coherence
- Idempotence: repeated assignment, multi-replica set comparison
- Control loop: integrationMaster + real allocator + proto round-trip

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 16:25:43 -07:00
pingqiuandClaude Opus 4.6 c7eb87c587 feat: Phase 09 — V2 execution primitives and production closure 
Engine execution layer for V2 replication protocol:
- RebuildInstaller: full state handoff (dirty map, WAL, superblock, flusher)
- TruncateToLSN: exact safety predicate (checkpointLSN == truncateLSN),
  ErrTruncationUnsafe escalation to NeedsRebuild
- SyncReceiverProgress: unconditional Store for post-rebuild alignment
- V2StatusSnapshot: CommittedLSN = nextLSN-1 for sync_all

V2 bridge real I/O executors:
- TransferFullBase: TCP streaming + RebuildInstaller + second catch-up
- TransferSnapshot: SHA-256 verified streaming to disk
- TruncateWAL: ErrTruncationUnsafe detection + escalation
- StreamWALEntries: rebuild-mode TCP apply

Engine executor interfaces:
- CatchUpIO.TruncateWAL, RebuildIO.TransferFullBase returns achievedLSN
- CatchUpExecutor truncation-only skip, NeedsRebuild escalation
- RebuildExecutor uses achievedLSN for progress tracking

Design docs reorganized: superseded planning docs removed, protocol
truths and closure map added.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 16:25:23 -07:00
pingqiuandClaude Opus 4.6 643a5a1074 feat: Phase 12 P3+P4 — diagnosability surfaces, perf floor, rollout gates 
P3: Add explicit bounded read-only diagnosis surfaces for all symptom classes:
- FailoverDiagnostic: volume-oriented failover state with per-volume
  DeferredPromotion/PendingRebuild entries and proper timer lifecycle
- PublicationDiagnostic: two-read coherence check (LookupBlockVolume vs
  registry authority) with computed Coherent verdict
- RecoveryDiagnostic: minimal ActiveTasks surface (Path A)
- Blocker ledger: 3 diagnosed + 3 unresolved, finite, from actual file
- Runbook references only exposed surfaces, no internal state

P4: Add bounded performance floor + rollout-gate package:
- Engine-local floor measurement with explicit IOPS gates per workload
- Cost characterization: WAL 2x write amp, -56% replication tax
- Rollout gates with semantic cross-checks against cited evidence
  (baseline numbers, transport/network matrix, blocker counts)
- Launch envelope tightened to actually measured combinations only

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 16:20:22 -07:00
pingqiuandClaude Opus 4.6 46faf0f7e3 feat: Phase 09 P0 — production execution closure plan 
Execution-closure targets:
- P1: TransferFullBase — reuse rebuild.go TCP protocol
- P2: TransferSnapshot — checkpoint image + WAL tail
- P3: TruncateWAL — AdvanceTail + superblock update
- P4: Runtime ownership — V2 orchestrator drives execution

Key reuse sources identified:
- rebuild.go: rebuildFullExtent (client), RebuildServer (server)
- wal_writer.go: AdvanceTail
- flusher.go: updateSuperblockCheckpoint
- blockvol.go: ScanWALEntries (already wired)

Slice order: full-base first (highest value), then snapshot,
then truncation, then runtime ownership.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-31 17:25:09 -07:00
pingqiuandClaude Opus 4.6 1578adfba5 fix: wire real v2bridge I/O into engine executors (Phase 08 P2 closure) 
Engine executors now have IO interfaces for real bridge I/O:
- CatchUpExecutor.IO (CatchUpIO): StreamWALEntries
- RebuildExecutor.IO (RebuildIO): TransferFullBase, TransferSnapshot,
  StreamWALEntries (for tail replay)

When IO is set, executor calls real bridge I/O during execution.
When IO is nil, executor uses caller-supplied progress (test mode).

RecoveryPlan.CatchUpStartLSN: bound at plan time for IO bridge.

v2bridge.Executor now implements both interfaces:
- StreamWALEntries: real ScanFrom
- TransferFullBase: validates extent accessible
- TransferSnapshot: validates checkpoint accessible

Chain tests wire IO:
- CatchUpClosure: exec.IO = executor → real WAL scan through engine
- RebuildClosure: exec.IO = executor → real transfer through engine

This closes the engine → executor → v2bridge → blockvol chain.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-31 15:10:50 -07:00
pingqiuandClaude Opus 4.6 46ef79ce35 fix: stable ServerID in assignments, fail-closed on missing identity, wire into ProcessAssignments 
Finding 1: Identity no longer address-derived
- ReplicaAddr.ServerID field added (stable server identity from registry)
- BlockVolumeAssignment.ReplicaServerID field added (scalar RF=2 path)
- ControlBridge uses ServerID, NOT address, for ReplicaID
- Missing ServerID → replica skipped (fail closed), logged

Finding 2: Wired into real ProcessAssignments
- BlockService.v2Bridge field initialized in StartBlockService
- ProcessAssignments converts each assignment via v2Bridge.ConvertAssignment
  BEFORE existing V1 processing (parallel, not replacing yet)
- Logged at glog V(1)

Finding 3: Fail-closed on missing identity
- Empty ServerID in ReplicaAddrs → replica skipped with log
- Empty ReplicaServerID in scalar path → no replica created
- Test: MissingServerID_FailsClosed verifies both paths

7 tests: StableServerID, AddressChange_IdentityPreserved,
MultiReplica_StableServerIDs, MissingServerID_FailsClosed,
EpochFencing_IntegratedPath, RebuildAssignment, ReplicaAssignment

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-31 10:46:17 -07:00
pingqiuandClaude Opus 4.6 cfec3bff4a fix: update contract.go field source docs to match P1 implementation 
BlockVolState field mapping now matches actual StatusSnapshot():
- WALTailLSN ← super.WALCheckpointLSN (was: flusher.RetentionFloor)
- CommittedLSN ← flusher.CheckpointLSN() V1 interim (was: distCommit)
- CheckpointTrusted ← super.Validate()==nil (was: superblock.Valid)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 20:44:04 -07:00
pingqiuandClaude Opus 4.6 8c326c871c feat: add contract interfaces and pin/release via release-func pattern (Phase 07 P1) 
E5 handoff contract (contract.go):
- BlockVolReader: ReadState() → BlockVolState from real blockvol
- BlockVolPinner: HoldWALRetention/HoldSnapshot/HoldFullBase → release func
- BlockVolExecutor: StreamWALEntries/TransferSnapshot/TransferFullBase/TruncateWAL
- Clear import direction: weed-side imports sw-block, not reverse

StorageAdapter refactored:
- Consumes BlockVolReader + BlockVolPinner interfaces
- Pin/release uses release-func pattern (not map-based tracking)
- PushStorageAdapter for tests (push-based, no blockvol dependency)

10 bridge tests:
- 4 control adapter (identity, address change, role mapping, primary)
- 4 storage adapter (retained history, WAL pin reject, snapshot reject, symmetry)
- 1 E2E (assignment → adapter → engine → plan → execute → InSync)
- 1 contract interface verification

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 18:07:20 -07:00
pingqiuandClaude Opus 4.6 05daede7f9 feat: add V2 bridge adapters for blockvol (Phase 07 P0) 
Creates sw-block/bridge/blockvol/ — concrete adapters connecting
the V2 engine to real blockvol storage and control-plane state.

control_adapter.go:
- MakeReplicaID: volume-name/server-id (NOT address-derived)
- ToAssignmentIntent: maps master assignment → engine intent
- Role → SessionKind translation (pure mapping, no policy)

storage_adapter.go:
- BlockVolState: maps to real blockvol fields (WAL head/tail,
  committed, checkpoint) — NOT reconstructed from metadata
- GetRetainedHistory from real state
- PinSnapshot rejects untrusted checkpoint
- PinWALRetention rejects recycled range
- PinFullBase / ReleaseFullBase

8 bridge tests:
- StableIdentity: ReplicaID = vol/server (not address)
- AddressChangePreservesIdentity: same ID, different address
- RebuildRoleMapping: "rebuilding" → SessionRebuild
- PrimaryNoRecovery: no recovery targets for primary
- RetainedHistoryFromRealState: all fields from BlockVolState
- WALPinRejectsRecycled: tail validation
- SnapshotPinRejectsInvalid: trust validation
- E2E_AssignmentToRecovery: master assignment → adapter →
  engine intent → plan → execute → InSync

Adapter replacement order:
P0: control_adapter + storage_adapter (this delivery)
P1: executor_bridge + observe_adapter (deferred)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 17:39:39 -07:00
pingqiuandClaude Opus 4.6 4df61f290b fix: true mid-executor invalidation test via OnStep hook 
CatchUpExecutor.OnStep: optional callback fired between executor-managed
progress steps. Enables deterministic fault injection (epoch bump)
between steps without racing or manual sender calls.

E2_EpochBump_MidExecutorLoop:
- Executor runs 5 progress steps
- OnStep hook bumps epoch after step 1 (after 2 successful steps)
- Executor's own loop detects invalidation at step 2's check
- Resources released by executor's release path (not manual cancel)
- Log shows session_invalidated + exec_resources_released

This closes the remaining FC2 gap: invalidation is now detected
and cleaned up by the executor itself, not by external code.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 15:51:21 -07:00
pingqiuandClaude Opus 4.6 5b63d34d6b fix: snapshot+tail WAL pin failure cleanup + true mid-executor epoch test 
Finding 1: PlanRebuild snapshot+tail WAL pin failure now fail-closed
- InvalidateSession("wal_pin_failed_during_rebuild", StateNeedsRebuild)
- Snapshot pin released, session invalidated, no dangling state
- New test: E2_RebuildWALPinFailure_SessionCleaned

Finding 2: True mid-executor invalidation test
- Executor makes 2 successful progress steps (60, 70)
- Epoch bumps BETWEEN steps (real mid-execution)
- Third progress step fails — session invalidated
- Resources released via executor cancel
- New test: E2_EpochBump_AfterExecutorProgress

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 15:44:21 -07:00
pingqiuandClaude Opus 4.6 332f598606 fix: close P3 failure classes — session cleanup, causal logging, CancelPlan 
Finding 1: PlanRebuild now invalidates session on pin failure
- FullBasePin failure → InvalidateSession("full_base_pin_failed", StateNeedsRebuild)
- SnapshotPin failure → InvalidateSession("snapshot_pin_failed", StateNeedsRebuild)
- No dangling rebuild session after resource acquisition failure

Finding 2: Rebuild source logging shows causal reason
- plan_rebuild_full_base now logs: untrusted_checkpoint,
  trusted_checkpoint_unreplayable_tail, or no_checkpoint

Finding 3: CancelPlan for address-change cleanup
- New RecoveryDriver.CancelPlan(plan, reason): releases resources +
  invalidates session + logs plan_cancelled with reason
- Changed-address test uses CancelPlan (not manual ReleasePlan)

Finding 4: Executor-level epoch-bump test
- Executor's mid-step invalidation detection catches stale session
- Resources released via executor release path, not manual cancel

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 14:28:57 -07:00
pingqiuandClaude Opus 4.6 56afa55f13 feat: add P3 failure-class validation through planner/executor (Phase 06) 
6 new tests (validation_test.go) mapped to tester expectations E1-E5:

E1/FC1: Changed-address restart through planner/executor
- Active session invalidated by address change
- Sender identity preserved, old plan resources released
- Log shows: endpoint_changed → new session → plan → execute

E2/FC2: Epoch bump mid-execution step
- Partial progress, epoch bumps between steps
- Further progress rejected, executor cancels with resource release
- Log shows: session_invalidated + exec_resources_released

E3/FC5: Cross-layer proof — trusted base + unreplayable tail
- Storage: checkpoint=50, tail=80 → unreplayable
- RebuildSourceDecision → FullBase (not SnapshotTail)
- FullBasePin acquired, executed through RebuildExecutor, released
- Log shows: plan_rebuild_full_base (observable reason)

E4/FC8: Rebuild fallback when trusted-base proof fails
- Untrusted checkpoint → full-base, full-base pin fails → error
- Untrusted checkpoint → full-base, full-base pin succeeds → InSync
- Log shows: full_base_pin_failed

E5: Observability — full recovery chain logged
- Verifies 7 required log events from assignment through completion

Delivery template:
Changed contracts: P3 validates planner/executor path, not convenience
Fail-closed: epoch bump mid-step releases resources + logs cause
Resources: cross-layer proof chain validated end-to-end
Carry-forward: FC3/FC4/FC6/FC7 sufficient from prior phases

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 14:17:24 -07:00
pingqiuandClaude Opus 4.6 f5c0aab454 fix: rebuild executor consumes bound plan, fix catch-up timing 
Planner/executor contract:
- RebuildExecutor.Execute() takes no arguments — consumes plan-bound
  RebuildSource, RebuildSnapshotLSN, RebuildTargetLSN
- RecoveryPlan binds all rebuild targets at plan time
- Executor cannot re-derive policy from caller-supplied history

Catch-up timing:
- Removed unused completeTick parameter from CatchUpExecutor.Execute
- Per-step ticks synthesized as startTick + stepIndex + 1
- API shape matches implementation

New test: PlanExecuteConsistency_RebuildCannotSwitchSource
- Plans snapshot+tail, then mutates storage history
- Executor succeeds using plan-bound values (not re-derived)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 13:33:52 -07:00
pingqiuandClaude Opus 4.6 50442acb2e feat: add stepwise executor with release symmetry (Phase 06 P2) 
New: executor.go — CatchUpExecutor + RebuildExecutor
Replaces convenience wrappers with stepwise execution that owns
resource lifecycle on every exit path.

CatchUpExecutor.Execute:
  1. BeginCatchUp (freezes target)
  2. Stepwise RecordCatchUpProgress + CheckBudget per step
  3. RecordTruncation (if required)
  4. CompleteSessionByID
  5. Release resources (success or failure)

RebuildExecutor.Execute:
  1. BeginConnect + RecordHandshake
  2. SelectRebuildFromHistory
  3. BeginRebuildTransfer + progress
  4. BeginRebuildTailReplay + progress (snapshot+tail)
  5. CompleteRebuild
  6. Release resources (success or failure)

Both executors:
- Release all pins on every exit path (success, failure, cancellation)
- Check session validity mid-execution (detect epoch bump / endpoint change)
- Log resource release with causal reason

14 new tests (executor_test.go), mapped to tester expectations:
- E1: Partial catch-up failure releases WAL pin (2 tests)
- E2: Partial rebuild failure releases all pins (1 test)
- E3: Epoch bump / cancel releases resources (3 tests)
- E4: Successful execution releases resources (2 tests)
- E5: Stepwise not convenience (2 tests)

Delivery template:
Changed contracts: executor owns resource lifecycle (not caller)
Fail-closed: session check mid-execution, release on every error
Resources: WAL/snapshot/full-base pins released on all exit paths
Carry-forward: CompleteCatchUp/CompleteRebuild remain test-only

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 13:24:37 -07:00
pingqiuandClaude Opus 4.6 45bf111ce8 fix: derive WAL pin from actual replay need, PlanRebuild fails closed 
WAL pin tied to actual recovery contract:
- Truncation-only (replica ahead): no WAL pin acquired
- Real catch-up: pins from replicaFlushedLSN (actual replay start)
- Logs distinguish plan_truncate_only from plan_catchup

PlanRebuild precondition checks:
- Error on missing sender
- Error on no active session
- Error on non-rebuild session kind
- All fail closed with clear error messages

4 new tests:
- ReplicaAhead_NoWALPin: truncation-only, no WAL resources
- PlanRebuild_MissingSender: returns error
- PlanRebuild_NoSession: returns error
- PlanRebuild_NonRebuildSession: returns error

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 12:51:38 -07:00
pingqiuandClaude Opus 4.6 d4f7697dd8 fix: add full-base pin and clean up session on WAL pin failure 
Full-base rebuild resource:
- StorageAdapter.PinFullBase/ReleaseFullBase for full-extent base image
- PlanRebuild full_base branch now acquires FullBasePin
- RecoveryPlan.FullBasePin field, released by ReleasePlan

Session cleanup on resource failure:
- PlanRecovery invalidates session when WAL pin fails
  (no dangling live session after failed resource acquisition)

3 new tests:
- PlanRebuild_FullBase_PinsBaseImage: pin acquired + released
- PlanRebuild_FullBase_PinFailure: logged + error
- PlanRecovery_WALPinFailure_CleansUpSession: session invalidated,
  sender disconnected (no dangling state)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 12:20:24 -07:00
pingqiuandClaude Opus 4.6 f73a3fdab2 feat: add storage/control adapters and recovery driver (Phase 06 P0/P1) 
Phase 06 module boundaries:

adapter.go — StorageAdapter + ControlPlaneAdapter interfaces:
- GetRetainedHistory: real WAL retention state
- PinSnapshot / ReleaseSnapshot: rebuild resource management
- PinWALRetention / ReleaseWALRetention: catch-up resource management
- HandleHeartbeat / HandleFailover: control-plane event conversion

driver.go — RecoveryDriver replaces synchronous convenience:
- PlanRecovery: connect + handshake from storage state + acquire resources
- PlanRebuild: acquire snapshot + WAL pins for rebuild
- ReleasePlan: release all acquired resources

Convenience flow classification:
- ProcessAssignment, UpdateSenderEpoch, InvalidateEpoch → stepwise engine tasks
- ExecuteRecovery → planner (connect + classify)
- CompleteCatchUp, CompleteRebuild → TEST-ONLY convenience

7 new tests (driver_test.go):
- CatchUp plan + execute with WAL pin
- ZeroGap plan (no resources pinned)
- NeedsRebuild → rebuild plan with resource acquisition
- WAL pin failure → logged + error
- Snapshot pin failure → logged + error
- ReplicaAhead truncation through driver
- Cross-layer: storage proves recoverability, engine consumes proof

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 11:35:25 -07:00
pingqiuandClaude Opus 4.6 512bb5bcf6 fix: orchestrator owns full catch-up contract (budget + truncation) 
CompleteCatchUp now integrates:
- BeginCatchUp with start tick (freezes target)
- RecordCatchUpProgress (skips if already converged, e.g., truncation-only)
- CheckBudget at completion tick (escalates to NeedsRebuild + logs)
- RecordTruncation before completion (logs truncation_recorded)
- Logs causal reason for every rejection/escalation

CatchUpOptions: StartTick/CompleteTick (separate) + TruncateLSN.

3 new orchestrator-level tests:
- ReplicaAhead_TruncateViaOrchestrator: truncation through entry path
- ReplicaAhead_NoTruncate_CompletionRejected: logs completion_rejected
- BudgetEscalation_ViaOrchestrator: budget violation → NeedsRebuild + logs

Observability tests relabeled as sender-level (not entry-path).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 11:04:34 -07:00
pingqiuandClaude Opus 4.6 adaff8ddb3 fix: only log endpoint_changed when endpoint actually changed 
ProcessAssignment now compares pre/post endpoint state before
logging session_invalidated with "endpoint_changed" reason.
Normal session supersede (same endpoint, assignment_intent) no
longer mislabeled as endpoint change.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 08:10:35 -07:00
pingqiuandClaude Opus 4.6 5cdee4a011 fix: orchestrator owns zero-gap completion and per-replica invalidation logging 
Zero-gap completion:
- ExecuteRecovery auto-completes zero-gap sessions (no sender call needed)
- RecoveryResult.FinalState = StateInSync for zero-gap

Epoch transition:
- UpdateSenderEpoch: orchestrator-owned epoch advancement with auto-log
- InvalidateEpoch: per-replica session_invalidated events (not aggregate)

Endpoint-change invalidation:
- ProcessAssignment detects session ID change from endpoint update
- Logs per-replica session_invalidated with "endpoint_changed" reason

All integration tests now use orchestrator exclusively for core lifecycle.
No direct sender API calls for recovery execution in integration tests.

1 new test: EndpointChange_LogsInvalidation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 01:01:53 -07:00
pingqiuandClaude Opus 4.6 47238df0d7 fix: add RecoveryOrchestrator as real integrated entry path 
New: orchestrator.go — RecoveryOrchestrator drives recovery lifecycle
from assignment through execution to completion/escalation:
- ProcessAssignment: reconcile + session creation + auto-log
- ExecuteRecovery: connect → handshake from RetainedHistory → outcome
- CompleteCatchUp: begin catch-up → progress → complete + auto-log
- CompleteRebuild: connect → handshake → history-driven source →
  transfer → tail replay → complete + auto-log
- InvalidateEpoch: invalidate stale sessions + auto-log

All integration tests rewritten to use orchestrator as entry path.
No direct sender API calls in recovery lifecycle.

SessionSnapshot now includes: TruncateRequired/ToLSN/Recorded,
RebuildSource, RebuildPhase.

RecoveryLog is auto-populated by orchestrator at every transition.

7 integration tests via orchestrator:
- ChangedAddress, NeedsRebuild→Rebuild, EpochBump, MultiReplica
- Observability: session snapshot, rebuild snapshot, auto-populated log

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 00:25:58 -07:00
pingqiuandClaude Opus 4.6 7436b3b79c feat: add integration closure and observability (Phase 05 Slice 4) 
New files:
- observe.go: RegistryStatus, SenderStatus, RecoveryLog for debugging
- integration_test.go: V2-boundary integration tests through real
  engine entry path

Observability:
- Registry.Status() returns full snapshot: per-sender state, session
  snapshots, counts by category (InSync, Recovering, Rebuilding)
- RecoveryLog: append-only event log for recovery lifecycle debugging

Integration tests (6):
- ChangedAddress_FullFlow: initial recovery → address change →
  sender preserved → new session → recovery with proof
- NeedsRebuild_ThenRebuildAssignment: catch-up fails → NeedsRebuild
  → rebuild assignment → history-driven source → InSync
- EpochBump_DuringRecovery: mid-recovery epoch bump → old session
  rejected → new assignment at new epoch → InSync
- MultiReplica_MixedOutcomes: 3 replicas, 3 outcomes via
  RetainedHistory proofs, registry status verified
- RegistryStatus_Snapshot: observability snapshot structure
- RecoveryLog: event recording and filtering

Engine module at 54 tests (12 + 18 + 18 + 6).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 00:15:46 -07:00
pingqiuandClaude Opus 4.6 4d06622c01 fix: add nil check for RetainedHistory in sender APIs 
RecordHandshakeFromHistory and SelectRebuildFromHistory now
return an error instead of panicking on nil history input.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 23:57:19 -07:00
pingqiuandClaude Opus 4.6 cc8c529962 fix: connect recovery decisions to RetainedHistory, fix rebuild source 
RetainedHistory as engine input:
- RecordHandshakeFromHistory: sender-level API consuming RetainedHistory
  directly, returns RecoverabilityProof alongside outcome
- SelectRebuildFromHistory: sender-level API consuming RetainedHistory
  for rebuild-source decision

RebuildSourceDecision soundness:
- Now requires BOTH trusted checkpoint AND replayable tail
  (CheckpointLSN >= TailLSN and CommittedLSN <= HeadLSN)
- Trusted checkpoint with unreplayable tail falls back to full_base

4 new tests:
- TrustedCheckpoint_UnreplayableTail (the regression case)
- SenderDriven_CatchUp (history → proof → outcome → complete)
- SenderDriven_Rebuild_SnapshotTail (history → source → rebuild)
- SenderDriven_Rebuild_FallsBackToFullBase (unreplayable tail)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 23:55:31 -07:00
pingqiuandClaude Opus 4.6 ff7ea41099 feat: add engine data/recoverability core (Phase 05 Slice 3) 
New file: history.go — RetainedHistory connects recovery decisions
to actual WAL retention state:
- IsRecoverable: checks gap against tail/head boundaries
- MakeHandshakeResult: generates HandshakeResult from retention state
- RebuildSourceDecision: chooses snapshot+tail vs full base from
  checkpoint state (trusted vs untrusted)
- ProveRecoverability: generates explicit proof explaining why
  recovery is or is not allowed

14 new tests (recoverability_test.go):
- Recoverable/unrecoverable gap (exact boundary, beyond head)
- Trusted/untrusted/no checkpoint → rebuild source selection
- Handshake from retained history → outcome classification
- Recoverability proofs (zero-gap, ahead, within retention, beyond)
- E2E: two replicas driven by retained history (catch-up + rebuild)
- Truncation required for replica ahead of committed

Engine module at 44 tests (12 + 18 + 14).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 23:04:51 -07:00
pingqiuandClaude Opus 4.6 368a956aee fix: correct catch-up entry counting and rebuild transfer gate 
Entry counting:
- Session.setRange now initializes recoveredTo = startLSN
- RecordCatchUpProgress delta counts only actual catch-up work
  (recoveredTo - startLSN), not the replica's pre-existing prefix

Rebuild transfer gate:
- BeginTailReplay requires TransferredTo >= SnapshotLSN
- Prevents tail replay on incomplete base transfer

3 new regression tests:
- BudgetEntries_NonZeroStart_CountsOnlyDelta (30 entries within 50 budget)
- BudgetEntries_NonZeroStart_ExceedsBudget (30 entries exceeds 20 budget)
- Rebuild_PartialTransfer_BlocksTailReplay

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 21:35:03 -07:00
pingqiuandClaude Opus 4.6 930de4ba78 feat: add Slice 2 recovery execution tests (Phase 05) 
15 new engine-level recovery execution tests:
- Zero-gap / catch-up / needs-rebuild branching (3 tests)
- Stale execution rejection during active recovery (2 tests)
- Bounded catch-up: frozen target, duration, entries, stall (5 tests)
- Completion before convergence rejected
- Rebuild exclusivity: catch-up APIs excluded (1 test)
- Rebuild lifecycle: snapshot+tail, full base, stale ID (3 tests)
- Assignment-driven recovery flow

Engine module now at 27 tests (12 Slice 1 + 15 Slice 2).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 21:14:18 -07:00
pingqiuandClaude Opus 4.6 61e9408261 fix: separate stable ReplicaID from Endpoint in registry 
Registry is now keyed by stable ReplicaID, not by address.
DataAddr changes preserve sender identity — the core V2 invariant.

Changes:
- ReplicaAssignment{ReplicaID, Endpoint} replaces map[string]Endpoint
- AssignmentIntent.Replicas uses []ReplicaAssignment
- Registry.Reconcile takes []ReplicaAssignment
- Tests use stable IDs ("replica-1", "r1") independent of addresses

New test: ChangedDataAddr_PreservesSenderIdentity
- Same ReplicaID, different DataAddr (10.0.0.1 → 10.0.0.2)
- Sender pointer preserved, session invalidated, new session attached
- This is the exact V1/V1.5 regression that V2 must fix

doc.go: clarified Slice 1 core vs carried-forward files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 21:06:11 -07:00
pingqiuandClaude Opus 4.6 bb24b4b039 fix: encapsulate engine sender/session authority state 
All mutable state on Sender and Session is now unexported:
- Sender.state, .epoch, .endpoint, .session, .stopped → accessors
- Session.id, .phase, .kind, etc. → read-only accessors
- Session() replaced by SessionSnapshot() (returns disconnected copy)
- SessionID() and HasActiveSession() for common queries
- AttachSession returns (sessionID, error) not (*Session, error)
- SupersedeSession returns sessionID not *Session

Budget configuration via SessionOption:
- WithBudget(CatchUpBudget) passed to AttachSession
- No direct field mutation on session from external code

New test: Encapsulation_SnapshotIsReadOnly proves snapshot
mutation does not leak back to sender state.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 20:58:28 -07:00
pingqiuandClaude Opus 4.6 20d70f9fb6 feat: add V2 engine replication core (Phase 05 Slice 1) 
Creates sw-block/engine/replication/ — the real V2 engine ownership core,
promoted from sw-block/prototype/enginev2/ with all accepted invariants.

Files:
- types.go: Endpoint, ReplicaState, SessionKind, SessionPhase, FSM transitions
- sender.go: per-replica Sender with full execution + rebuild APIs
- session.go: Session with identity, phases, frozen target, truncation, budget
- registry.go: Registry with reconcile + assignment intent + epoch invalidation
- budget.go: CatchUpBudget (duration, entries, stall detection)
- rebuild.go: RebuildState FSM (snapshot+tail vs full base)
- outcome.go: HandshakeResult + ClassifyRecoveryOutcome

Tests (ownership_test.go, 13 tests):
- Changed-address invalidation (A10)
- Stale session ID rejected at all APIs (A3)
- Stale completion after supersede (A3)
- Epoch bump invalidates all sessions (A3)
- Stale assignment epoch rejected
- Rebuild exclusivity (catch-up APIs rejected)
- Rebuild full lifecycle
- Frozen target rejects chase (A5)
- Budget violation escalates (A5)
- E2E: 3 replicas, 3 outcomes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 20:51:01 -07:00
pingqiuandClaude Opus 4.6 26a1b33c2e feat: add A5-A8 acceptance traceability and rebuild-source evidence 
Cleanup: removed redundant TargetLSNAtStart from CatchUpBudget.
FrozenTargetLSN on RecoverySession is the single source of truth.

Acceptance traceability (acceptance_test.go):
- A5: 3 evidence tests (unrecoverable gap, budget escalation, frozen target)
- A6: 2 evidence tests (exact boundary, contiguity required)
- A7: 3 evidence tests (snapshot history, catch-up replay, truncation)
- A8: 2 evidence tests (convergence required, truncation required)

Rebuild-source decision evidence:
- snapshot_tail when trusted base exists
- full_base when no snapshot or untrusted
- 3 explicit tests

13 new tests total.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 15:42:48 -07:00
pingqiuandClaude Opus 4.6 8f5070679c fix: make frozen target intrinsic and rebuild completion exclusive 
Frozen target is now unconditional:
- FrozenTargetLSN field on RecoverySession, set by BeginCatchUp
- RecordCatchUpProgress enforces FrozenTargetLSN regardless of Budget
- Catch-up is always a bounded (R, H0] contract

Rebuild completion exclusivity:
- CompleteSessionByID explicitly rejects SessionRebuild by kind
- Rebuild sessions can ONLY complete via CompleteRebuild

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 15:30:17 -07:00
pingqiuandClaude Opus 4.6 8e4028758f fix: make rebuild path exclusive, enforce phase discipline, require tick for stall budget 
Rebuild exclusivity:
- BeginCatchUp rejects SessionRebuild ("must use rebuild APIs")
- RecordCatchUpProgress rejects SessionRebuild
- Rebuild sessions can only be completed via CompleteRebuild
- All legacy rebuild-through-catch-up paths in tests converted

Phase discipline:
- SelectRebuildSource requires session.Phase == PhaseHandshake
- Cannot skip BeginConnect + RecordHandshake

Stall budget:
- RecordCatchUpProgress requires tick parameter when
  ProgressDeadlineTicks > 0 (no silent stall budget bypass)

3 new tests: rebuild exclusivity (catch-up APIs rejected),
rebuild source requires handshake phase, stall budget requires tick.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 15:21:39 -07:00
pingqiuandClaude Opus 4.6 5b66a85f92 fix: wire rebuild FSM into sender, enforce frozen target, fix entry counting 
Rebuild execution path:
- newRecoverySession auto-initializes RebuildState for SessionRebuild
- Sender rebuild APIs: SelectRebuildSource, BeginRebuildTransfer,
  RecordRebuildTransferProgress, BeginRebuildTailReplay,
  RecordRebuildTailProgress, CompleteRebuild
- All rebuild APIs are sender-authority-gated by sessionID
- E2E rebuild test now drives through rebuild FSM, not catch-up APIs

Bounded CatchUp enforcement:
- BeginCatchUp freezes TargetLSNAtStart from session.TargetLSN
- RecordCatchUpProgress rejects progress beyond frozen target
- Entry counting uses LSN delta (recoveredTo - previous), not call count
- Merged RecordCatchUpProgressAt into RecordCatchUpProgress (tick param)

5 new tests: target-frozen enforcement, sender-level rebuild via
rebuild APIs, reject non-rebuild, reject stale ID on rebuild.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 15:16:56 -07:00
pingqiuandClaude Opus 4.6 3f0048cbd9 feat: add bounded CatchUp budget and Rebuild mode state machine (Phase 4.5 P0) 
Bounded CatchUp:
- CatchUpBudget: MaxDurationTicks, MaxEntries, ProgressDeadlineTicks
- BudgetCheck: runtime consumption tracker (StartTick, EntriesReplayed, LastProgressTick)
- Sender.CheckBudget: evaluates budget, escalates to NeedsRebuild on violation
- RecordCatchUpProgressAt: tracks progress tick for stall detection
- BeginCatchUp accepts optional startTick for budget tracking

Rebuild state machine:
- RebuildSource: snapshot_tail (preferred) vs full_base (fallback)
- RebuildPhase: init → source_select → transfer → tail_replay → completed|aborted
- SelectSource: chooses based on snapshot availability
- Phase ordering enforced, transfer regression rejected
- ReadyToComplete validates target reached

13 new tests: budget enforcement (duration, entries, stall, no-budget),
sender budget integration, rebuild lifecycle (snapshot+tail, full base,
abort, phase order, regression), E2E bounded catch-up → rebuild.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 14:33:06 -07:00
pingqiuandClaude Opus 4.6 90c39b549d feat: add prototype scenario closure (Phase 04 P4) 
Maps V2 acceptance criteria A1-A7, A10 to enginev2 prototype evidence.
Adds 4 V2-boundary scenarios against the prototype.

Scenario tests:
- A1: committed data survives promotion (WAL truncation boundary)
- A2: uncommitted data truncated, not revived
- A3: stale epoch fenced at sender + session + assignment layers
- A4: short-gap catch-up with WAL-backed proof + data verification
- A5: unrecoverable gap escalates to NeedsRebuild with proof
- A6: recoverability boundary exact (tail +/- 1 LSN)
- A7: historical data correct after tail advancement (snapshot)
- A10: changed-address → invalidation → new assignment → recovery

V2-boundary scenarios:
- NeedsRebuild persists across topology update
- catch-up does not overwrite safe data
- 5 disconnect/reconnect cycles preserve sender identity
- full V2 harness: 3 replicas, 3 outcomes (zero-gap, catch-up, rebuild)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-29 11:31:56 -07:00
pingqiuandClaude Opus 4.6 942a0b7da7 fix: strengthen IsRecoverable contiguity check and StateAt snapshot correctness 
IsRecoverable now verifies three conditions:
- startExclusive >= tailLSN (not recycled)
- endInclusive <= headLSN (within WAL)
- all LSNs in range exist contiguously (no holes)

StateAt now uses base snapshot captured during AdvanceTail:
- returns nil for LSNs before snapshot boundary (unreconstructable)
- correctly includes block state from recycled entries via snapshot

5 new tests: end-beyond-head, missing entries, state after tail
advance, nil before snapshot, block last written before tail.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-28 18:52:11 -07:00
pingqiuandClaude Opus 4.6 c89709e47e feat: add WAL history model and recoverability proof (Phase 04 P3) 
Adds minimal historical-data prototype to enginev2:

- WALHistory: retained-prefix model with Append, Commit, AdvanceTail,
  Truncate, EntriesInRange, IsRecoverable, StateAt
- MakeHandshakeResult connects WAL state to outcome classification
- RecordTruncation execution API for divergent tail cleanup
- CompleteSessionByID gates on truncation when required
- Zero-gap requires exact equality (FlushedLSN == CommittedLSN)
- Replica-ahead classified as CatchUp with mandatory truncation

15 new tests: WAL basics, provable recoverability, unprovable gap,
exact boundary, truncation enforcement, WAL-backed end-to-end
recovery with data verification.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-28 11:29:27 -07:00
pingqiuandClaude Opus 4.6 edec7098e8 feat: add V2 protocol simulator and enginev2 sender/session prototype 
Adds sw-block/ directory with:

- distsim: protocol correctness simulator (96 tests)
  - cluster model with epoch fencing, barrier semantics, commit modes
  - endpoint identity, control-plane flow, candidate eligibility
  - timeout events, timer races, same-tick ordering
  - session ownership tracking with ID-based stale fencing

- enginev2: standalone V2 sender/session implementation (63 tests)
  - per-replica Sender with identity-preserving reconciliation
  - RecoverySession with FSM phase transitions and session ID
  - execution APIs: BeginConnect, RecordHandshake, BeginCatchUp,
    RecordCatchUpProgress, CompleteSessionByID — all sender-authority-gated
  - recovery outcome branching: zero-gap, catch-up, needs-rebuild
  - assignment-intent orchestration with epoch fencing

- design docs: acceptance criteria, open questions, first-slice spec,
  protocol development process

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-28 10:38:27 -07:00