mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-19 22:44:18 +00:00
Code Issues Packages Projects Releases Wiki Activity
13,263 Commits 181 Branches 360 Tags
fbcfe89e2461d42158cdc5e678d64e281ec7d2c8
Commit Graph
8 Commits
Author SHA1 Message Date
pingqiuandClaude Opus 4.6 59a36013d4 feat: rebuild hardening A1-A5 + session-controlled execution path 
A1 Engine kind-routing fix:
  SessionProgressObserved/Completed/Failed now respect active session
  Kind. Rebuild progress no longer leaks into catch-up aggregate.
  sessionKindMismatch guard + observeRebuildProgress helper.
  2 regression tests lock kind isolation.

A2 Retention pin:
  Rebuild session ack drives progress-based WAL retention floor.
  Pin installed at base_lsn on accepted, advances with wal_applied_lsn,
  released on completed/failed/cancelled. rebuildProgressPinFloor
  returns min across all active replicas.
  Retention pin test: 100 blocks fill WAL, 5 flusher cycles with
  20 pinned rebuild entries — all verified correct.

A3 Progress ack emission:
  Automatic sessionAck(running/base_complete/completed/failed) emitted
  from rebuild session lifecycle transitions. sessionAckLocked builds
  ack under session lock. emitRebuildSessionAck callback wired through
  SetOnRebuildSessionAck on BlockVol.
  ObserveReplicaRebuildSessionAck maps acks to core engine events.
  WireLocalReplicaRebuildSessionAcks bridges local callback to server.
  5 server tests proving ack→core, pin advance, pin cleanup.

A4 Deadline/timeout:
  rebuildAckWatch watchdog: armed on accepted/running/base_complete,
  refreshed on each ack, cleared on completed/failed. Timeout
  cancels local session + clears pin + fail-closes.
  2 tests: timeout→fail-close, progress→refresh.

A5 Session-controlled execution path:
  v2bridge.Executor.TransferFullBase now uses session-controlled loop:
  beginControlledFullBase → real sessionControl over TCP →
  transferExtentToSession via RebuildTransportClient →
  PrepareFullBaseRebuild → TryCompleteRebuildSession.
  ReplicaReceiver control channel handles MsgSessionControl alongside
  MsgBarrierReq. Session acks written back on same TCP connection.
  RebuildSessionBase request type separates new per-block stream from
  legacy raw extent stream. Full-base cleanup deferred until success.
  Deadlock fix: ApplyBaseBlock releases session lock before ioMu.
  Hydration skip for full-base sessions.

23 rebuild component tests (all pass):
  11 kernel correctness, 8 transport/runtime, 3 scenario-scale,
  including 1GB primary-initiated with CRC validation.

29 files changed, ~2500 insertions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-08 14:39:11 -07:00
pingqiuandClaude Opus 4.6 44103a1bd7 feat: Phase 20 acceptance fixes + sw-test-runner suite mode 
Acceptance rows closed:
- WriteLBA/SyncCache contract: code comments document write-back vs
  durability fence semantics
- RF=2 stable identity: v2bridge always uses SetReplicaAddrs (preserves
  ServerID); blockcmd dispatcher also fixed to use setupPrimaryReplicationMulti;
  test asserts exact expected ReplicaID="vs-2" (not just non-empty)
- Tests treating WriteLBA as commit: replica_read_test rewritten with
  SyncCache as durability fence
- publish_healthy contract: 3 gate tests with hard assertions including
  gate 3 (PrimaryShipperConnected)
- SetReplicaAddr deprecation warning added
- WALShipper.ReplicaID() getter added for identity verification

Test runner enhancements:
- sw-test-runner suite command: build → deploy → run N scenarios in one
  invocation with --skip-deploy support
- Suite YAML definitions for T6 Stage 0 and Stage 1
- deploy action: kill stale processes, clean dirs, cross-compile, upload
- run-phase20-t6.ps1 PowerShell script (deprecated by suite command)

Engine/runtime fixes:
- Recovery executor nil-safety improvements
- Recovery bundle BuildRecoveryBundle defensive checks
- ShipperGroup MinReplicaFlushedLSNAll surface

Docs: acceptance checklist refined, test matrix updated, T6 runbook,
engine maintainer tutorial, design README updated.

26 files changed, ~1600 insertions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-06 11:30:54 -07:00
pingqiuandClaude Opus 4.6 c7eb87c587 feat: Phase 09 — V2 execution primitives and production closure 
Engine execution layer for V2 replication protocol:
- RebuildInstaller: full state handoff (dirty map, WAL, superblock, flusher)
- TruncateToLSN: exact safety predicate (checkpointLSN == truncateLSN),
  ErrTruncationUnsafe escalation to NeedsRebuild
- SyncReceiverProgress: unconditional Store for post-rebuild alignment
- V2StatusSnapshot: CommittedLSN = nextLSN-1 for sync_all

V2 bridge real I/O executors:
- TransferFullBase: TCP streaming + RebuildInstaller + second catch-up
- TransferSnapshot: SHA-256 verified streaming to disk
- TruncateWAL: ErrTruncationUnsafe detection + escalation
- StreamWALEntries: rebuild-mode TCP apply

Engine executor interfaces:
- CatchUpIO.TruncateWAL, RebuildIO.TransferFullBase returns achievedLSN
- CatchUpExecutor truncation-only skip, NeedsRebuild escalation
- RebuildExecutor uses achievedLSN for progress tracking

Design docs reorganized: superseded planning docs removed, protocol
truths and closure map added.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 16:25:23 -07:00
pingqiuandClaude Opus 4.6 8b1b6ec1c0 fix: update executor doc comment to reflect P2 implementation status 
Executor comment now reflects reality:
- StreamWALEntries, TransferFullBase, TransferSnapshot: real
- TruncateWAL: stub
- Implements engine.CatchUpIO and engine.RebuildIO interfaces

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-31 15:14:34 -07:00
pingqiuandClaude Opus 4.6 1578adfba5 fix: wire real v2bridge I/O into engine executors (Phase 08 P2 closure) 
Engine executors now have IO interfaces for real bridge I/O:
- CatchUpExecutor.IO (CatchUpIO): StreamWALEntries
- RebuildExecutor.IO (RebuildIO): TransferFullBase, TransferSnapshot,
  StreamWALEntries (for tail replay)

When IO is set, executor calls real bridge I/O during execution.
When IO is nil, executor uses caller-supplied progress (test mode).

RecoveryPlan.CatchUpStartLSN: bound at plan time for IO bridge.

v2bridge.Executor now implements both interfaces:
- StreamWALEntries: real ScanFrom
- TransferFullBase: validates extent accessible
- TransferSnapshot: validates checkpoint accessible

Chain tests wire IO:
- CatchUpClosure: exec.IO = executor → real WAL scan through engine
- RebuildClosure: exec.IO = executor → real transfer through engine

This closes the engine → executor → v2bridge → blockvol chain.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-31 15:10:50 -07:00
pingqiuandClaude Opus 4.6 c9671c4e47 feat: integrated execution chain — catch-up + rebuild + cleanup (Phase 08 P2) 
Live catch-up chain:
- Assignment → engine plan → v2bridge WAL scan → blockvol ScanFrom
- StreamWALEntries transfers real entries (transferred=5)
- V1 interim: engine classifies ZeroGap (committed=0), but WAL scan
  chain proven mechanically (executor→v2bridge→blockvol→progress)

Live rebuild chain (full-base):
- ForceFlush advances checkpoint → NeedsRebuild detected
- TransferFullBase now real: validates extent accessible at committed LSN
- Engine rebuild session: connect → handshake → source select →
  transfer → complete → InSync

Execution cleanup:
- CancelPlan releases resources + invalidates session
- Log shows plan_cancelled with reason

Observability:
- sender_added + escalated events explain execution causality
- Escalation includes proof reason from RetainedHistory

4 new execution chain tests + TransferFullBase implementation.

Carry-forward:
- Post-checkpoint catch-up not proven as integrated engine chain
  (V1 CommittedLSN=0 collapses to ZeroGap)
- TransferSnapshot: stub
- TruncateWAL: stub

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-31 14:22:27 -07:00
pingqiuandClaude Opus 4.6 785a7d7efd feat: wire real pinner into flusher retention + real WAL scan executor (Phase 07 P1) 
Pinner wired to real retention:
- NewPinner calls vol.SetV2RetentionFloor(p.MinWALRetentionFloor)
- Flusher.RetentionFloorFn() / SetRetentionFloorFn() exposed
- SetV2RetentionFloor chains with existing shipper retention floor
- Holds actually prevent WAL reclaim (not just tracked state)

Executor uses real WAL scan:
- BlockVol.ScanWALEntries(fromLSN, callback) wraps wal.ScanFrom
  with real fd, walOffset, checkpointLSN
- Executor.StreamWALEntries uses ScanWALEntries (not stub)
- Reads real WAL entries, tracks highest LSN scanned

CommittedLSN mapping:
- Explicitly documented as interim V1 model (committed = checkpointed)
- Will diverge when V2 distributed commit separates from local flush

Carry-forward:
- TransferSnapshot/TransferFullBase/TruncateWAL: stubs (need extent I/O)
- Control intent from confirmed failover: deferred

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 20:01:46 -07:00
pingqiuandClaude Opus 4.6 c00c9e3e3d feat: add real BlockVolPinner + BlockVolExecutor in v2bridge (Phase 07 P1) 
Pinner (pinner.go):
- HoldWALRetention: validates startLSN >= current tail, tracks hold
- HoldSnapshot: validates checkpoint exists + trusted
- HoldFullBase: tracks hold by ID
- MinWALRetentionFloor: returns minimum held position across all
  WAL/snapshot holds — designed for flusher RetentionFloorFn hookup
- Release functions remove holds from tracking map

Executor (executor.go):
- StreamWALEntries: validates range against real WAL tail/head
  (actual ScanFrom integration deferred to network-layer wiring)
- TransferSnapshot/TransferFullBase/TruncateWAL: stubs for P1

Key integration points:
- Pinner reads real StatusSnapshot for validation
- Pinner.MinWALRetentionFloor can wire into flusher.RetentionFloorFn
- Executor validates WAL range availability from real state

Carry-forward:
- Real ScanFrom wiring needs WAL fd + offset (network layer)
- TransferSnapshot/TransferFullBase need extent I/O
- Control intent from confirmed failover (master-side)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 19:54:24 -07:00