Commit Graph
1137 Commits
Author SHA1 Message Date
pingqiuandClaude Opus 4.6 f90ccf5bfd fix: proactive shipper reconnect on rejoin (Bug 5)
After rejoin, the shipper is configured but no I/O triggers Ship(),
so the shipper stays Disconnected and the core stays at
awaiting_shipper_connected indefinitely.

Fix: observePrimaryShipperConnectivity now calls TryReconnectShippers
when ShipperConfigured=true but ShipperConnected=false. This triggers
the full reconnect protocol (dial + handshake + bounded catch-up)
proactively, bringing the replica current without waiting for I/O.

Option B approach: uses the same reconnect path as Barrier() — not a
fake write or bare dial probe. CatchUpTo(headLSN) replays any retained
WAL entries, bringing the replica fully current.

New methods:
- WALShipper.TryReconnect(): full reconnect without foreground I/O
- ShipperGroup.TryReconnectAll(): probes all disconnected shippers
- BlockVol.TryReconnectShippers(): volume-level entry point

Also fix pre-existing test expectation: engine now emits
start_recovery_task on primary assignment with replicas.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 00:14:46 -07:00
pingqiuandClaude Opus 4.6 53246d2780 fix: recover TOCTOU + WAL pressure edge case tests
Fix recover path TOCTOU: re-Lookup after AddReplica so the primary
refresh assignment includes the freshly added replica addresses.
Previously, Lookup (copy) was called before AddReplica modified the
registry, so entry.Replicas was empty → primary got replicas=0 →
shipper never configured.

Add 2 WAL pressure edge case tests:
- ShipperCatchUpOrEscalate: 64KB WAL, 200 writes, aggressive flusher.
  Proves no hang/deadlock/corruption. Shipper either keeps up or
  correctly escalates to NeedsRebuild.
- RebuildWithPinWhilePrimaryWrites: rebuild session active while
  primary writes 7600+ blocks in 2s. Proves primary never freezes
  — rebuild pin is on replica only, primary WAL recycles freely.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 23:56:26 -07:00
pingqiuandClaude Opus 4.6 e0116fc631 fix: three hardware blockers — WAL retention + registry race + shutdown beat
All 43 actions pass on m01/m02 hardware. Auto-failover PASS.
dd_write: 30s → 123ms. Post-failover write: 33,621 IOPS.

1. WAL retention: remove keepup retention floor (MinShippedLSN).
   WAL cannot be pinned during sustained async writes — any pin
   strategy either fills WAL (blocking writes) or over-recycles
   (breaking catch-up). Flusher recycles freely. Future LBA map
   will provide catch-up without WAL retention.
   MinShippedLSN on ShipperGroup retained as diagnostic surface.

2. Registry stale-cleanup race: add RegisteredAt grace period.
   Race: master registers volume → next VS heartbeat arrives before
   VS discovers the volume → stale cleanup deletes the entry →
   failover finds 0 entries. Fix: skip stale cleanup for entries
   registered within 30s (> 2 heartbeat intervals).
   2 new tests: grace protects new entry, old entry still cleaned.

3. Shutdown heartbeat: VS disconnect heartbeat no longer claims
   block inventory authority. Previously, the shutdown beat's
   empty inventory triggered stale cleanup, deleting the entry
   before failover could use it.

Scenario fix: recovery-baseline-failover.yaml now kills the
correct node (discovered primary, not hardcoded), connects to
the correct new primary for post-failover verification.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 22:59:46 -07:00
pingqiuandClaude Opus 4.6 39f1232fe2 feat: validation matrix closure — Rebuild Ready 12/12, Restore Ready 10/10
Close all Rebuild Ready and Restore Ready matrix gaps. V2 Ready at 10/14
(2 partial, 2 missing — honest assessment).

New tests (tester-written):
- R1: syncAck-driven trigger via protocol engine decision
- R3: stale replica restart beyond WAL → rebuild converges
- R5: connection drop mid-base → cancel → fresh rebuild converges
- R10: failover-rejoin with forced WAL recycling, strict rebuild assert
- R11: divergent replica full overwrite convergence
- R12: crash mid-rebuild → fresh session converges (not resume)
- S2: corrupt WAL entry + corrupt base block both rejected
- S5: snapshot-tail rebuild (base + WAL tail replay)
- S7: crash between base install and tail replay
- S8: snapshot under concurrent writes
- V5: rebuild complete without DurableLSN blocks publish_healthy
- V9: mixed replica health aggregate projection
- V14: negative fail-closed matrix (epoch, kind, stale)

Bug fix: StartRebuildSession now clears stale dirty map + resets WAL +
updates checkpoint AFTER safety check but BEFORE session.Start(). Fixes
stale extent data shadowing rebuild base blocks on reopened replicas.

Cleanup: remove 14 obsolete design docs (migration batches, old WAL-v2
specs, simulator goals) — all superseded by current protocol docs.

34 component tests + 8 protocol engine tests + server tests all pass.
1GB CRC validation passes in 19s.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 16:31:55 -07:00
pingqiuandClaude Opus 4.6 59a36013d4 feat: rebuild hardening A1-A5 + session-controlled execution path
A1 Engine kind-routing fix:
  SessionProgressObserved/Completed/Failed now respect active session
  Kind. Rebuild progress no longer leaks into catch-up aggregate.
  sessionKindMismatch guard + observeRebuildProgress helper.
  2 regression tests lock kind isolation.

A2 Retention pin:
  Rebuild session ack drives progress-based WAL retention floor.
  Pin installed at base_lsn on accepted, advances with wal_applied_lsn,
  released on completed/failed/cancelled. rebuildProgressPinFloor
  returns min across all active replicas.
  Retention pin test: 100 blocks fill WAL, 5 flusher cycles with
  20 pinned rebuild entries — all verified correct.

A3 Progress ack emission:
  Automatic sessionAck(running/base_complete/completed/failed) emitted
  from rebuild session lifecycle transitions. sessionAckLocked builds
  ack under session lock. emitRebuildSessionAck callback wired through
  SetOnRebuildSessionAck on BlockVol.
  ObserveReplicaRebuildSessionAck maps acks to core engine events.
  WireLocalReplicaRebuildSessionAcks bridges local callback to server.
  5 server tests proving ack→core, pin advance, pin cleanup.

A4 Deadline/timeout:
  rebuildAckWatch watchdog: armed on accepted/running/base_complete,
  refreshed on each ack, cleared on completed/failed. Timeout
  cancels local session + clears pin + fail-closes.
  2 tests: timeout→fail-close, progress→refresh.

A5 Session-controlled execution path:
  v2bridge.Executor.TransferFullBase now uses session-controlled loop:
  beginControlledFullBase → real sessionControl over TCP →
  transferExtentToSession via RebuildTransportClient →
  PrepareFullBaseRebuild → TryCompleteRebuildSession.
  ReplicaReceiver control channel handles MsgSessionControl alongside
  MsgBarrierReq. Session acks written back on same TCP connection.
  RebuildSessionBase request type separates new per-block stream from
  legacy raw extent stream. Full-base cleanup deferred until success.
  Deadlock fix: ApplyBaseBlock releases session lock before ioMu.
  Hydration skip for full-base sessions.

23 rebuild component tests (all pass):
  11 kernel correctness, 8 transport/runtime, 3 scenario-scale,
  including 1GB primary-initiated with CRC validation.

29 files changed, ~2500 insertions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 14:39:11 -07:00
pingqiuandClaude Opus 4.6 342f8baa69 feat: rebuild transport wiring — session control + base block streaming
Wire protocol messages and transport handlers for the rebuild MVP:

Protocol messages (rebuild_transport.go):
- SessionControlMsg: epoch, sessionID, command, baseLSN, targetLSN,
  snapshotID. Encode/Decode with fixed 37-byte wire format.
- SessionAckMsg: epoch, sessionID, phase, walAppliedLSN, baseComplete,
  achievedLSN. Encode/Decode with fixed 34-byte wire format.
- MsgSessionControl (0x10) and MsgSessionAck (0x11) on control channel.
- SendSessionControl/SendSessionAck convenience functions.

Transport handlers:
- RebuildTransportServer: primary-side, streams all extent blocks as
  MsgRebuildExtent frames (reusing existing rebuild message type),
  ends with MsgRebuildDone.
- RebuildTransportClient: replica-side, receives base blocks and
  routes through vol.ApplyRebuildSessionBaseBlock, marks base
  complete on MsgRebuildDone.

4 transport tests:
- SessionControl wire round-trip
- SessionAck wire round-trip
- BaseBlockStreaming: full TCP loop, 1024 blocks streamed and verified
- SessionControlOverTCP: real TCP send/receive with accepted ack

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 14:57:43 -07:00
pingqiuandClaude Opus 4.6 d2d57851b0 feat: rebuild MVP — dual-lane session with bitmap protection
Rebuild session protocol implementation for v2-rebuild-mvp-session-protocol.md.

New files:
- rebuild_bitmap.go: RebuildBitmap — session-scoped dense bitset for
  WAL-applied LBA tracking. MarkApplied on local WAL write (not receive).
  ShouldApplyBase returns false for WAL-covered LBAs (WAL always wins).

- rebuild_session.go: RebuildSession — replica-side two-line rebuild.
  WAL lane (ApplyWALEntry) + base lane (ApplyBaseBlock) with bitmap
  conflict resolution. TryComplete requires BOTH base_complete AND
  wal_applied_lsn >= target_lsn. Volume-level control surface:
  StartRebuildSession, ApplyRebuildSessionWALEntry/BaseBlock,
  MarkRebuildSessionBaseComplete, TryCompleteRebuildSession,
  CancelRebuildSession, ActiveRebuildSession.

- rebuild_mvp_test.go: 4 correctness tests — base+WAL converge,
  WAL-applied never overwritten by base, bitmap set on applied not
  received, control surface start/supersede/complete.

- rebuild_transport_test.go: 2 transport-level tests — two-line with
  real WAL shipping, live writes during base copy with bitmap conflict.

Design docs:
- v2-rebuild-mvp-session-protocol.md: MVP spec with message set, apply
  rules, completion/failure/crash rules, test matrix
- v2-sync-recovery-protocol.md: full protocol context (keepup/catchup/
  rebuild unified design, primary decision logic, two-line model)
- v2-session-protocol-shape.md: protocol shape overview

Protocol engine (reference, not production):
- sw-block/protocol/: 7-event engine with ~300 lines, 13 tests

6 rebuild tests pass, all existing component tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 14:30:34 -07:00
pingqiuandClaude Opus 4.6 55013e103b feat: Phase 20 Stage 0+1 closure — bootstrap + sustained workload on hardware
Stage 0 (bootstrap closure): PASS on m01/M02
  - create RF=2 sync_all → 10s shipper wait → 4k fsync → publish_healthy
  - Proves: BarrierAccepted observation, ShipperConnected, DurableLSN > 0

Stage 1 (sustained workload): 32/33 actions PASS
  - bootstrap → fio 10s randwrite → dd_write 1M×2 fsync → data checksum
  - Remaining: auto-failover promotion (separate issue)

Key fixes:
  - BarrierAccepted callback: SyncCache success → core DurableLSN update
  - BarrierRejected callback: barrier failures surface to core with reason
  - Shipper state callback for new volumes (not just startup volumes)
  - CatchUpTo ctrl conn reset: prevents stale control channel after recovery
  - CP13-6 max-bytes budget suspended: uses replicaFlushedLSN which can't
    advance without barrier; kills healthy shippers during async writes.
    Will be replaced by v2 negotiated sync/recovery protocol.
  - Barrier diagnostic logging: start/fail/success with reason and LSN
  - Scenario restructured: Stage 0 (bootstrap-closure) + Stage 1 (failover)
  - dd_write: sync_mode param + real stderr capture
  - sw-test-runner suite command: deploy once, run N scenarios
  - WAL size plumbing: proto + API + handler (forward-compatible)

Known: 6 blockvol/server test failures from Barrier() path change
(bounded catch-up in Barrier). Need test updates to match new semantics.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 19:55:12 -07:00
pingqiuandClaude Opus 4.6 44103a1bd7 feat: Phase 20 acceptance fixes + sw-test-runner suite mode
Acceptance rows closed:
- WriteLBA/SyncCache contract: code comments document write-back vs
  durability fence semantics
- RF=2 stable identity: v2bridge always uses SetReplicaAddrs (preserves
  ServerID); blockcmd dispatcher also fixed to use setupPrimaryReplicationMulti;
  test asserts exact expected ReplicaID="vs-2" (not just non-empty)
- Tests treating WriteLBA as commit: replica_read_test rewritten with
  SyncCache as durability fence
- publish_healthy contract: 3 gate tests with hard assertions including
  gate 3 (PrimaryShipperConnected)
- SetReplicaAddr deprecation warning added
- WALShipper.ReplicaID() getter added for identity verification

Test runner enhancements:
- sw-test-runner suite command: build → deploy → run N scenarios in one
  invocation with --skip-deploy support
- Suite YAML definitions for T6 Stage 0 and Stage 1
- deploy action: kill stale processes, clean dirs, cross-compile, upload
- run-phase20-t6.ps1 PowerShell script (deprecated by suite command)

Engine/runtime fixes:
- Recovery executor nil-safety improvements
- Recovery bundle BuildRecoveryBundle defensive checks
- ShipperGroup MinReplicaFlushedLSNAll surface

Docs: acceptance checklist refined, test matrix updated, T6 runbook,
engine maintainer tutorial, design README updated.

26 files changed, ~1600 insertions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 11:30:54 -07:00
pingqiuandClaude Opus 4.6 d1a16fac03 feat: protocol-aware execution wave — phase gate for live WAL shipping
Add host-side protocol state seam that derives per-replica execution
state from V2 sender/session snapshots and blocks live-tail WAL
shipping while an active recovery session is in progress.

New file: weed/server/block_protocol_state.go
  - replicaProtocolExecutionState derived from engine snapshots
  - LiveEligible=false during active catch-up/rebuild sessions
  - bindProtocolExecutionPolicy wires policy into BlockVol
  - syncProtocolExecutionState called after assignments + core events

Data plane changes:
  - WALShipper.Ship() checks liveShippingPolicy before dial/send
  - BlockVol.SetLiveShippingPolicy persists across shipper group rebuilds
  - ShipperGroup propagates policy to all shippers

Design contract: sw-block/design/v2-protocol-aware-execution.md

Scope: WAL-first rollout only. Prevents illegal live-tail delivery
during active recovery. Does not change snapshot/build behavior or
move backlog. Next wave: bounded WAL catch-up under same contract.

Tests: 4 unit/component tests for phase gate behavior, plus bootstrap
seam tests that confirmed the two pre-existing bugs locally.

13 files changed, 900 insertions, 69 deletions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 23:47:07 -07:00
pingqiuandClaude Opus 4.6 6bf9a6c283 test: Phase 20 Tier 1 component tests — wiring proof for CI/CD
6 new component tests closing gaps identified in the test matrix audit:

P20-T4-C3: Missing projection with active V2 core fails closed
  - v2Core != nil, no projection cached → gate with "missing_engine_projection"

P20-T4-C6: Gate actually removes iSCSI target (enforcement)
  - real TargetServer → HasTarget(iqn)==true before gate
  - gate → HasTarget(iqn)==false (DisconnectVolume called)
  - ungate → HasTarget(iqn)==true (AddVolume restores)

P20-T5-C3: FailoverDiagnosticSnapshot carries both mode fields
  - register volume with EngineProjectionMode + ClusterReplicationMode
  - trigger pending rebuild → volume appears in diagnostic
  - diagnostic entry carries both modes from registry lookup

P20-T3-C5: V2PromotionMode diagnostic tri-state
  - disabled / placeholder_fail_closed / transport_ready
  - all three configurations produce correct diagnostic value

P20-T1-C3: EngineProjectionMode proto round-trip
  - set value survives InfoMessageToProto → InfoMessageFromProto
  - empty value produces nil proto field (presence semantics)

P20-T4-C8: ActivationGated proto round-trip
  - gated=true + reason survives round-trip
  - not-gated produces no spurious reason

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 19:39:00 -07:00
pingqiuandClaude Opus 4.6 ceb68cc66b fix: Phase 20 T5 — RF2 missing replica degraded + transport signal + API surface
Fix three tester findings on T5:

1. RF2 with missing replicas now reports "degraded" instead of
   "no_replicas". Only RF=1 with no replicas returns "no_replicas".
   Missing replica in an RF2 set is a degraded cluster state.

2. TransportDegraded signal now incorporated: if master-observed
   transport is degraded, ClusterReplicationMode is at least
   "degraded" regardless of individual replica health.

3. API surface exposure: EngineProjectionMode and
   ClusterReplicationMode now appear on blockapi.VolumeInfo and are
   populated in entryToVolumeInfo(). Operators can consume both
   through GET /block/volume/{name} with distinct JSON field names.

12 tests: keepup, catching_up, stale degraded, LSN gap needs_rebuild,
rebuilding role, RF1 no_replicas, RF2 missing degraded, transport
degraded, distinctness, heartbeat update, worst dominates, API
surface distinct naming.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 18:49:37 -07:00
pingqiuandClaude Opus 4.6 46f72572c5 fix: Phase 20 T4 — real serving enforcement + wire propagation + runtime ungate
Fix three tester findings on T4 activation gate:

1. Real serving enforcement: evaluateActivationGate now calls
   gateServing() → DisconnectVolume(iqn) on gate (terminates active
   iSCSI sessions, removes volume from target). ungateServing() →
   AddVolume(iqn, adapter) on clear (re-registers volume). This is
   actual serving enforcement, not just bookkeeping.

2. Wire propagation: add activation_gated (field 25) and
   activation_gate_reason (field 26) to proto BlockVolumeInfoMessage.
   Add generated Go fields + getters. Add proto conversion in
   InfoMessageToProto/InfoMessageFromProto. Gate state now rides the
   real VS→master heartbeat wire.

3. Runtime ungate: evaluateActivationGate() now also runs in
   applyCoreEvent() (the observation-driven path), not just
   applyCoreAssignmentEvent(). Recovery/catch-up completion that
   transitions the projection to publish_healthy/replica_ready now
   clears the gate and re-registers the volume automatically.

ClearActivationGate() remains as an explicit override for edge cases
but is no longer the primary ungate path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 18:22:49 -07:00
pingqiuandClaude Opus 4.6 a27569358b feat: Phase 20 T4 — local activation gate on promoted primary
After assignment executes through V2 core, evaluateActivationGate()
checks the resulting projection locally. If mode is degraded,
needs_rebuild, bootstrap_pending, or allocated_only, the volume is
gated from serving. Gate is enforced immediately after assignment,
before the next heartbeat round-trip.

Gate cleared only when projection reaches publish_healthy or
replica_ready. IsActivationGated() provides the query surface for
iSCSI/NVMe adapter enforcement. Heartbeat carries ActivationGated
and ActivationGateReason fields so master can observe the gated state
(report path, not enforcement path).

activationGated map on BlockService tracks per-volume gate state.
Initialized in constructor. Test helper updated to include it.

6 tests: degraded gates, needs_rebuild gates, healthy clears gate,
gate enforced before heartbeat, recovery re-enables, assignment with
degraded projection triggers gate.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 18:13:20 -07:00
pingqiuandClaude Opus 4.6 85dad8e0c9 feat: Phase 20 T1 — EngineProjectionMode in heartbeat
Add engine_projection_mode as a distinct proto/wire/registry field
that carries pure V2 engine-derived local projection mode from VS
to master. Reads ONLY from CoreProjection — no ad-hoc fallback.

Separate from existing VolumeMode: EngineProjectionMode is VS-local
V2 engine truth, VolumeMode is the existing field that conflates V2
and V1 paths. Both exist during transition; only EngineProjectionMode
is V2-authoritative.

Clears stale value on primary turnover: when a newly promoted primary
heartbeats without the field, the old primary's projection is not
preserved (prevents synthetic master-side truth).

5 focused tests: propagation, distinctness (hard assertion), backward
compat preservation, turnover-clears, turnover-with-field.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 15:45:26 -07:00
pingqiuandClaude Opus 4.6 044a6d770b feat: Phase 19 — bounded working RF2 block path
Live HTTP evidence transport, continuous Loop2 service, bounded auto
failover trigger, runtime-managed frontend export, bounded replica
repair, end-to-end RF2 handoff with continued I/O on new primary,
bounded operator HTTP surface, and CSI V2 runtime backend adapter.

11 new proof tests covering the full M6-M10 chain plus CSI create/
lookup/publish through the V2 runtime path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 15:12:00 -07:00
pingqiuandClaude Opus 4.6 cf16e53b04 feat: Phase 16M/17 + promote fixes + testrunner updates
Phase 16M: explicit replica readiness on heartbeat seam
- master.proto: optional bool replica_ready = 19 (proto regenerated on M01)
- block_heartbeat_proto.go: write/read ReplicaReady with presence semantics
- master_block_registry.go: replicaReadyObservedFromHeartbeat prefers
  explicit proto field, falls back to address heuristic when absent
- volume_server_block.go: heartbeat emits ReplicaReady from core projection

Phase 17: host effects extraction + stop line
- phase-17-log.md: Batch 10/11 delivery notes

Promote fixes:
- master_block_failover.go: deterministic replica addrs from path hash
- qa_promote_replication_test.go: address-upgrade trigger test
- qa_promote_rejoin_live_test.go: new live rejoin test

Testrunner:
- devops.go: action improvements
- recovery-baseline-failover.yaml, suite-ha-failover.yaml: scenario updates
- cp11b3-manual-promote.yaml: promote scenario alignment
- fresh_volume_write_test.go: new component test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 11:38:05 -07:00
pingqiu 10833c8b68 refactor: preserve bounded volume mode reason heartbeat truth
Carry explicit volume_mode_reason across the heartbeat/master/API seam so outward surfaces retain the bounded core-owned explanation behind mode transitions.

Made-with: Cursor
2026-04-04 14:21:31 -07:00
pingqiu 6cad5bb8e1 refactor: rebind bounded volume mode heartbeat truth
Make the heartbeat/master boundary preserve explicit volume_mode truth so master consume no longer reconstructs outward mode only from secondary heartbeat signals. Keep backward compatibility by falling back to the previous reconstruction when older heartbeats do not send the field.

Made-with: Cursor
2026-04-04 13:56:41 -07:00
pingqiu 6794f79df9 refactor: preserve bounded publish healthy heartbeat truth
Make the heartbeat/master boundary preserve explicit publish_healthy truth so master consume no longer reconstructs healthy publication only from secondary readiness and degraded heuristics. Keep backward compatibility by falling back to the previous reconstruction when older heartbeats do not send the field.

Made-with: Cursor
2026-04-04 13:43:19 -07:00
pingqiu eb610deb92 refactor: preserve bounded needs_rebuild heartbeat truth
Make the heartbeat/master boundary preserve explicit needs_rebuild truth so primary heartbeat consume no longer collapses that stronger mode into a generic degraded signal. Keep backward compatibility by falling back to the previous heuristic when older heartbeats do not send the field.

Made-with: Cursor
2026-04-04 13:11:42 -07:00
pingqiu 69b41a7f16 refactor: rebind bounded replica-ready heartbeat truth
Make the heartbeat/master boundary carry explicit replica readiness truth so the registry no longer depends only on replica transport-address presence as a readiness proxy. Keep backward compatibility by falling back to the old address heuristic when older heartbeats do not send the field.

Made-with: Cursor
2026-04-04 12:06:53 -07:00
pingqiuandClaude Opus 4.6 38b5042997 refactor: extract command bindings and service ops from volume server
Move BlockVol-backed command bindings into v2bridge and move non-BlockVol
command operations into weed/server/blockcmd. This keeps dispatch and host
effects in weed/server, keeps backend binding in v2bridge, and further
shrinks volume_server_block.go toward a host shell while preserving
current command-driven proofs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 08:11:39 -07:00
pingqiuandClaude Opus 4.6 263611004e refactor: Batch 5 — recovery binding factory moved to v2bridge
New v2bridge.BuildRecoveryBundle(vol, rebuildAddr) assembles all
recovery bindings (Reader + Pinner + StorageAdapter + Executor) from
a real BlockVol instance in one call.

block_recovery.go changes:
- Removed local recoveryBundle type
- buildRecoveryBundle now delegates to v2bridge.BuildRecoveryBundle
  inside WithVolume, returns (driver, executor, err)
- Removed direct v2bridge.NewReader/NewPinner/NewExecutor construction
- Removed bridge import (no longer needed)
- runCatchUp/runRebuild use (driver, executor, err) directly

block_recovery.go no longer knows how to construct Reader, Pinner,
StorageAdapter, or Executor. It only knows: resolve volPath, ask the
factory for bindings, plan, branch to legacy or core-present path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 01:39:40 -07:00
pingqiuandClaude Opus 4.6 680b530314 refactor: Task E — reader returns bridge.BlockVolState directly
Reader backend-binding extraction:
- v2bridge/reader.go: Reader.ReadState() now returns bridge.BlockVolState
  directly instead of a local v2bridge.BlockVolState mirror type.
  Removed the local BlockVolState type entirely.
- block_recovery.go: removed readerShimForRecovery (12 lines of 1:1
  field copying). Reader is now passed directly as bridge.BlockVolReader.

Before: v2bridge.Reader → v2bridge.BlockVolState → readerShim → bridge.BlockVolState
After:  v2bridge.Reader → bridge.BlockVolState (direct)

v2bridge now imports sw-block/bridge/blockvol for the contract type
(control.go already did this, reader.go now follows the same pattern).

Validation:
- go test ./sw-block/bridge/blockvol/... → PASS
- go test ./weed/storage/blockvol/v2bridge/ -run "TestReader_" → PASS
- go test ./weed/server/ -run "TestP4_|TestP16B_" → PASS (8 tests)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 00:43:30 -07:00
pingqiuandClaude Opus 4.6 a38e04c03b refactor: Task A — canonical identity/recovery rules via bridge helpers
Remove direct fmt.Sprintf identity construction from v2bridge/control.go.
Both convertReplicaAssignment and convertRebuildAssignment now use:
- bridge.ReplicaAssignmentForServer (canonical ReplicaID derivation)
- bridge.RecoveryTargetForRole (canonical role → SessionKind mapping)

Before: 3 call sites with inline fmt.Sprintf("%s/%s", vol, server)
After: 0 — all identity construction goes through sw-block canonical helpers

volume_server_block.go already used bridge helpers (no change needed).

Validation:
- go test ./sw-block/bridge/blockvol/... → PASS (10 tests)
- go test ./weed/storage/blockvol/v2bridge/ -run "TestControl_|TestBridge_" → PASS (7 tests)
- go test ./weed/server/ -run "TestBlockService_ApplyAssignments_RebuildingRole_" → PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 00:10:48 -07:00
pingqiuandClaude Opus 4.6 6e1b8efd68 feat: CP13-9 — mode normalization for constrained V1 runtime
Add computed VolumeMode to BlockVolumeEntry with 5 normalized modes:
- allocated_only: RF=1, no replicas (standalone)
- bootstrap_pending: RF>1 but replicas not yet ready (first-write pending)
- publish_healthy: all replicas ready, no transport degradation
- degraded: replication impaired but recoverable
- needs_rebuild: unrecoverable gap, rebuild required

Code changes:
- master_block_registry.go: computeVolumeMode() called from
  recomputeReplicaState(), VolumeMode field on BlockVolumeEntry
- master_server_handlers_block.go: VolumeMode exposed in REST API
- blockapi/types.go: VolumeMode field in VolumeInfo
- testrunner types: VolumeMode for scenario assertions

7 tests prove mode normalization:
- AllocatedOnly, BootstrapPending (2 cases), PublishHealthy,
  Degraded, NeedsRebuild, SurfaceConsistency (transition proof)

Interpretation rule: current integrated tests validate V1 runtime
under V2 constraints, not a completed V2 runtime (Phase 14 scope).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 15:02:50 -07:00
pingqiuandClaude Opus 4.6 4c7fbefe25 feat: CP13-8 PASSES — real-workload validation on RF=2 sync_all
CP13-8 scenario results on m01/M02 (25Gbps RoCE):
  fsck_ext4:       CLEAN
  file count:      200 (assert_equal PASS)
  checksum match:  MATCH (assert_contains PASS)
  pgbench TPS:     565.69 (assert_greater PASS)
  auto-failover:   10.0.0.1:18480 → 10.0.0.3:18480

Code changes (tester + scenario):
- volume_server_block.go: readiness state, assignment lifecycle cleanup
- block_heartbeat_loop.go: readiness-aware heartbeat reporting
- store_blockvol.go: readiness tracking
- master_server_handlers_block.go: block API handler updates
- cp13-8-real-workload-validation.yaml: redesigned scenario
  (removed block_promote, use natural auto-failover flow,
  bootstrap write before wait_volume_healthy)
- testrunner/actions/devops.go: scenario action improvements
- replica_read_test.go: component-level replica read test

Phase docs: CP13-7 accepted, CP13-8/8A technical packs updated,
design docs updated for protocol closure evidence.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 14:24:13 -07:00
pingqiuandClaude Opus 4.6 7012383c3f fix: StartReplicaReceiver idempotency guard — skip if already running
P0 bug on real hardware: assignments are re-delivered every heartbeat
cycle (5s). First setupReplicaReceiver succeeds (receiver starts on
deterministic port). Second call fails with "bind: address already in
use" because the listener is already bound. The volume stays permanently
degraded, blocking all RF=2 sync_all replication.

Fix: skip StartReplicaReceiver if v.replRecv is already set. The
receiver only needs to start once per volume lifetime.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 13:18:30 -07:00
pingqiuandClaude Opus 4.6 2c305f9e7f fix: CP13-8 — use correct assert params + add pgbench TPS gate
1. assert_contains: change actual/expected to value/contains (matches
   the action implementation in system.go)
2. Add assert_greater for pgbench TPS > 0 after pgbench_run (closes
   the pgbench durability pass criterion in the doc)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 09:17:11 -07:00
pingqiuandClaude Opus 4.6 d7cd415714 feat: CP13-8 — bounded real-workload validation scenario + envelope
One named workload validation package for RF=2 sync_all:
- Scenario: cp13-8-real-workload-validation.yaml (6 phases)
- ext4 proof: write 200 files → failover → fsck + file count + md5sum diff
- pgbench proof: TPC-B on promoted replica (database durability)
- Disturbance: one bounded failover (kill primary, promote replica)

Workload envelope doc: phase-13-cp8-workload-validation.md
- Named topology, transport, workloads, disturbance, exclusions
- Pass criteria: fsck passes, 200 files, checksums match, pgbench TPS > 0
- Maps each pass criterion to accepted CP13-1..7 semantics
- Explicit non-claims: not rollout approval, not NVMe, not soak, not CP13-9

Reuses existing infrastructure:
- cp85-db-ext4-fsck.yaml pattern (extended with checksums + pgbench)
- benchmark-pgbench.yaml actions (pgbench_init/pgbench_run)

Must run on real hardware (m01/M02). Cannot run in unit test harness.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 08:59:46 -07:00
pingqiuandClaude Opus 4.6 4f7283b6be fix: registry role-aware failover + devops action + failover scenario update
- master_block_registry.go: minor role-handling fixes
- qa_failover_role_test.go: new failover role test
- testrunner/actions/devops.go: new devops action helpers
- recovery-baseline-failover.yaml: scenario alignment

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 08:48:13 -07:00
pingqiuandClaude Opus 4.6 1d3fb1f119 fix: CP13-7 rev3 — require NeedsRebuild, not Degraded, after handshake gap
Tighten TestReconnect_GapBeyondRetainedWal_NeedsRebuild assertion from
"NeedsRebuild or Degraded" to strictly "NeedsRebuild". The handshake
R < S path returns NeedsRebuild directly — tolerating Degraded weakened
the proof.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 08:36:00 -07:00
pingqiuandClaude Opus 4.6 ec63c18438 fix: CP13-7 rev2 — real handshake gap detection, reclassify rebuild test
Two fixes:
1. TestReconnect_GapBeyondRetainedWal_NeedsRebuild: rewritten to test the
   real reconnect handshake gap detection path (R < S in
   reconnectWithHandshake). Sequence: establish sync → disconnect →
   release retention hold via timeout → write + flush to advance WAL past
   replica position → reconnect → handshake detects R=0 < S=9 → NeedsRebuild.
   Log proves: "reconnect: gap too large R=0 H=8 S=9"

2. TestReplicaState_RebuildComplete_ReentersInSync: reclassified from
   primary proof to support evidence (does not start from live NeedsRebuild
   shipper state, but proves rebuild mechanics work end-to-end).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 08:24:56 -07:00
pingqiuandClaude Opus 4.6 88c336b1c1 feat: CP13-7 — NeedsRebuild fail-closed fallback + rebuild handoff proof
Last baseline FAIL closed:
- TestAdversarial_NeedsRebuildBlocksAllPaths: rewritten to use
  EvaluateRetentionBudgets for NeedsRebuild trigger, then asserts
  5 properties: state=NeedsRebuild, Ship drops, Barrier rejects,
  state sticky after failed barrier, second SyncCache still fails

Last baseline PASS* closed:
- TestReconnect_GapBeyondRetainedWal_NeedsRebuild: rewritten with
  hard NeedsRebuild state assertion + SyncCache failure assertion

6 tests promoted to CP13-7 primary proof:
- NeedsRebuildBlocksAllPaths (fail-closed lifecycle)
- GapBeyondRetainedWal (transition)
- HeartbeatReportsNeedsRebuild (visibility)
- RebuildComplete_ReentersInSync (handoff)
- Rebuild_AbortOnEpochChange (epoch safety)
- PostRebuild_FlushedLSN_IsCheckpoint (progress initialization)

Baseline: 43 PASS / 0 FAIL / 1 PASS* (address witness only)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 00:13:33 -07:00
pingqiuandClaude Opus 4.6 0ce5aa32e9 fix: CP13-6 rev3 — hard hold-release assertion + stale comment cleanup
1. TestWalRetention_TimeoutTriggersNeedsRebuild: add hard assertion that
   checkpoint advances past replicaFlushedLSN after NeedsRebuild (proves
   hold is actually released, not just state transition)
2. TestWalRetention_RequiredReplicaBlocksReclaim: remove stale "EXPECTED
   TO FAIL" / duplicate comment block

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 23:59:44 -07:00
pingqiuandClaude Opus 4.6 4e55b53bef fix: CP13-6 rev2 — upgrade all 3 retention tests to hard assertions, block-size-aware budget
Three fixes:
1. TestWalRetention_RequiredReplicaBlocksReclaim: rewritten from log-only
   placeholder to hard assertion (checkpointLSN <= replicaFlushedLSN)
2. TestWalRetention_TimeoutTriggersNeedsRebuild: rewritten from log-only
   to hard assertion (State() == NeedsRebuild after 1ns timeout)
3. EvaluateRetentionBudgets: uses RetentionBudgetParams struct with
   actual BlockSize from volume config instead of hardcoded 4096

All 3 retention tests now have real state/progress assertions.
No placeholder or log-only evidence remains in CP13-6 proof package.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 23:45:29 -07:00
pingqiuandClaude Opus 4.6 0ca57dc2eb feat: CP13-6 — replica-aware WAL retention with max-bytes budget
Add max-bytes retention budget alongside existing timeout budget:
- shipper_group.go: EvaluateRetentionBudgets now checks both timeout
  (last contact time) and max-bytes (entry lag * 4KB > maxBytes).
  Either exceeding budget → NeedsRebuild state transition.
- blockvol.go: add walRetentionMaxBytes (64MB default), pass to
  EvaluateRetentionBudgets with primaryHeadLSN.

TestWalRetention_MaxBytesTriggersNeedsRebuild upgraded from PASS*
(log-only placeholder) to real PASS: asserts State()==NeedsRebuild
after lag exceeds configured max-bytes budget.

Retention contract: hold-back blocks reclaim for recoverable replicas,
timeout and max-bytes budgets escalate to NeedsRebuild and release hold.
Full rebuild lifecycle remains CP13-7 scope.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 23:10:06 -07:00
pingqiuandClaude Opus 4.6 4681df6b56 fix: CP13-5 — tighten reconnect proof with observable handshake evidence
Findings fixed:
1. TestAdversarial_ReconnectUsesHandshakeNotBootstrap now has 3 observable
   proof points instead of just "SyncCache succeeded":
   - new shipper HasFlushedProgress=true (seeded from old group)
   - replica receivedLSN advances during SyncCache (catch-up delivered entries)
   - shipper replicaFlushedLSN > 0 after barrier (durable progress established)
   Bootstrap alone would not advance receivedLSN — it only sends the barrier.

2. TestBug2 stale comment removed: "must NOT call SetReplicaAddr" replaced
   with accurate CP13-5 explanation that SetReplicaAddrs now preserves
   hasFlushedProgress across shipper replacement.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 22:56:20 -07:00
pingqiuandClaude Opus 4.6 80be2ec05a feat: CP13-5 — reconnect handshake + WAL catch-up on SetReplicaAddrs
Bug: SetReplicaAddrs created fresh shippers (hasFlushedProgress=false),
so after disconnect, the new shipper used bootstrap instead of reconnect
handshake. Bootstrap doesn't replay missed WAL entries — barrier hung.

Fix:
- blockvol.go: SetReplicaAddrs checks if old shipper group had durable
  progress (AnyHasFlushedProgress). If so, seeds new shippers with
  hasFlushedProgress=true → they use reconnect handshake + catch-up.
- shipper_group.go: add AnyHasFlushedProgress() helper.

3 baseline FAILs now PASS:
- ReconnectUsesHandshakeNotBootstrap: reconnect path used, not bootstrap
- CatchupMultipleDisconnects: repeated disconnect/reconnect recovers
- CatchupDoesNotOverwriteNewerData: catch-up completes, safety exercised

7 tests promoted to CP13-5 primary proof.
TestAdversarial_NeedsRebuildBlocksAllPaths still FAIL (CP13-7 scope).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 22:39:08 -07:00
pingqiuandClaude Opus 4.6 1c294af169 feat: CP13-4 — replica state machine / barrier eligibility contract + proof
Contract review: 6-state set (Disconnected, Connecting, CatchingUp,
InSync, Degraded, NeedsRebuild). Only InSync proceeds to barrier
request path. All other states either fail immediately or attempt
reconnect (must succeed before reaching barrier).

New test: TestBarrier_NonEligibleStates_FailClosed — systematically
verifies each non-eligible state (Connecting, CatchingUp, NeedsRebuild,
Disconnected) is rejected by Barrier(), and InSync is the only state
that enters the barrier request path.

5 baseline tests promoted to CP13-4 primary proof.
No production code changed — contract review + new focused test only.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 22:01:05 -07:00
pingqiuandClaude Opus 4.6 d4ff6b482b fix: CP13-3 test — exercise real shipper.Barrier() against legacy server
The previous test only checked wire decode + fresh shipper state, never
calling shipper.Barrier() against a legacy response source.

New test runs a fake TCP control server that responds with a 1-byte
BarrierOK (no FlushedLSN). Shipper.Barrier() is called against it and
must return an error containing "no FlushedLSN". Verifies the real
rejection path at wal_shipper.go:229-231.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 21:47:58 -07:00
pingqiuandClaude Opus 4.6 08dc592d29 fix: CP13-3 — reject legacy BarrierOK with FlushedLSN=0 in sync_all
Bug: BarrierOK with FlushedLSN == 0 (legacy 1-byte response) was counted
as successful sync_all durability even though no authoritative durable
progress was established. This allowed a legacy replica to silently pass
through the sync_all barrier without proving any LSN was fsynced.

Fix (wal_shipper.go): BarrierOK with FlushedLSN == 0 now returns an
error instead of nil. Barrier success requires the replica to report a
non-zero FlushedLSN proving which LSN was durably persisted. This makes
the code match the CP13-3 contract: replicaFlushedLSN is the sole
authority for sync_all durability.

New test: TestBarrier_LegacyResponseRejectedBySyncAll — proves legacy
1-byte responses don't establish durable authority.

Contract review doc updated to reflect the code fix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 21:42:51 -07:00
pingqiuandClaude Opus 4.6 4bdf6c604e fix: CP13-2 — use advertisedIP (routable), not localServerID (opaque)
Bug: setupReplicaReceiver derived the advertised host from localServerID,
which can be an opaque string (from -id flag, e.g., "my-custom-server-id").
This would publish unusable endpoints like "my-custom-server-id:14260".

Fix:
- volume_server_block.go: add advertisedIP field (always a real IP from
  -ip flag), use it instead of localServerID for replica canonicalization
- volume.go: wire *v.ip → blockService.SetAdvertisedIP() at startup
- blockvol.go: StartReplicaReceiver variadic advertisedHost unchanged

Proof (sync_all_bug_test.go TestBug3, 4 sub-cases):
- fallback: wildcard bind without advertisedHost → outbound-IP
- advertisedHost: explicit IP appears in exported addresses
- StartReplicaReceiver_API: public API forwards host correctly
- opaque_identity_not_routable: proves opaque string produces
  non-routable address, confirming production must use advertisedIP

Identity vs transport separation preserved:
- localServerID: stable identity for V2 control (may be opaque)
- advertisedIP: routable IP for transport endpoints (always real IP)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 20:51:47 -07:00
pingqiuandClaude Opus 4.6 2d47383df7 feat: CP13-2 — canonical replica addressing on production truth surface
Problem: StartReplicaReceiver didn't forward advertisedHost to
NewReplicaReceiver, so wildcard-bind listeners relied on outbound-IP
fallback for canonicalization. On multi-NIC hosts this could select
the wrong interface, leaking non-routable addresses into replication
truth.

Fix:
- blockvol.go: StartReplicaReceiver now accepts optional advertisedHost
  variadic param and forwards it to NewReplicaReceiver
- volume_server_block.go: setupReplicaReceiver extracts host from
  localServerID (the canonical VS identity) and passes it as
  advertisedHost — wildcard-bind addresses now resolve to the
  authoritative server IP, not outbound-IP fallback

Proof (sync_all_bug_test.go TestBug3, upgraded from PASS* to PASS):
- fallback: wildcard bind without advertisedHost still produces ip:port
- advertisedHost: explicit host appears in exported DataAddr/CtrlAddr
- StartReplicaReceiver_API: public API forwards advertisedHost correctly

What CP13-2 does NOT change:
- No reconnect handshake changes (CP13-5)
- No retention policy changes (CP13-6)
- No rebuild behavior changes (CP13-7)
- No barrier protocol changes (CP13-3)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 17:45:42 -07:00
pingqiuandClaude Opus 4.6 bdf20fde71 feat: Phase 12 — production hardening (disturbance, soak, testrunner scenarios)
P1 Disturbance: restart/reconnect correctness tests — assignment delivery
  through real proto → ProcessAssignments, epoch validation on promoted
  volume, mandatory reconnect assertions

P2 Soak: repeated create/failover/recover cycles with end-of-cycle truth
  checks, runtime hygiene (no stale tasks/entries), steady-state idempotence

Testrunner recovery actions + scenarios:
- recovery.go: wait_recovery_complete, assert_recovery_state, trigger_rebuild
- 8 new YAML scenarios: baseline (failover/crash/partition), stability
  (replication-tax, netem-sweep, packet-loss, degraded), robust shipper

HA edge case and EC6 fix tests for regression coverage.

(P3 diagnosability + P4 perf floor committed separately in 643a5a107)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 16:26:17 -07:00
pingqiuandClaude Opus 4.6 bdf83e350e feat: Phase 11 — product-surface rebinding (snapshot, CSI, publication, restore)
P1 Snapshots: CoW snapshot lifecycle through V2 engine path, create/list/delete
  via master RPC, BaseLSN tracking in manifest, ImportSnapshotForRebuild

P2 CSI Lifecycle: masterServerBackend calling real MasterServer in-process,
  CreateVolume/DeleteVolume/ExpandVolume through CSI → master → VS flow,
  ExportedControllerServer/ExportedNodeServer for cross-package testing

P3 Publication: LookupBlockVolume coherence across failover, iSCSI + NVMe
  address switching on promotion, repeated lookup self-consistency

P4 Restore: RestoreBlockSnapshot RPC through master and volume server,
  snapshot restore with runtime convergence, epoch/role validation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 16:25:58 -07:00
pingqiuandClaude Opus 4.6 3ec8fab2f1 feat: Phase 10 — control-plane closure (identity, convergence, idempotence)
Stable identity on wire:
- ServerID fields in proto (replica_server_id, server_id on ReplicaAddrMessage)
- volumeServerId wired through volume.go → BlockService.SetServerID
- Identity derived from canonical server ID, not transport addresses

Assignment convergence:
- V2 idempotence via lastAppliedAssignment.equals (full replica set comparison)
- setupPrimaryReplication/Multi idempotence guards
- ProcessAssignments with V2 + V1 dual-path assignment handling

Master-driven control loop:
- RecoveryManager: serialized cancel-and-drain via done channels
- Per-replica heartbeat state reporting (ReplicaShipperStatus)
- masterServerBackend: VolumeBackend calling real MasterServer in-process
- RestoreBlockSnapshot RPC (master + volume server proto)

QA tests (P10 P1-P4):
- Identity: ServerID on wire, fail-closed on missing
- Convergence: assignment delivery, epoch monotonicity, registry coherence
- Idempotence: repeated assignment, multi-replica set comparison
- Control loop: integrationMaster + real allocator + proto round-trip

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 16:25:43 -07:00
pingqiuandClaude Opus 4.6 c7eb87c587 feat: Phase 09 — V2 execution primitives and production closure
Engine execution layer for V2 replication protocol:
- RebuildInstaller: full state handoff (dirty map, WAL, superblock, flusher)
- TruncateToLSN: exact safety predicate (checkpointLSN == truncateLSN),
  ErrTruncationUnsafe escalation to NeedsRebuild
- SyncReceiverProgress: unconditional Store for post-rebuild alignment
- V2StatusSnapshot: CommittedLSN = nextLSN-1 for sync_all

V2 bridge real I/O executors:
- TransferFullBase: TCP streaming + RebuildInstaller + second catch-up
- TransferSnapshot: SHA-256 verified streaming to disk
- TruncateWAL: ErrTruncationUnsafe detection + escalation
- StreamWALEntries: rebuild-mode TCP apply

Engine executor interfaces:
- CatchUpIO.TruncateWAL, RebuildIO.TransferFullBase returns achievedLSN
- CatchUpExecutor truncation-only skip, NeedsRebuild escalation
- RebuildExecutor uses achievedLSN for progress tracking

Design docs reorganized: superseded planning docs removed, protocol
truths and closure map added.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 16:25:23 -07:00
pingqiuandClaude Opus 4.6 ebe95b6e2e fix: flusher OOM on multi-block writes + testrunner enhancements
Bug: flusher.go:336 allocated make([]byte, entryLen) per dirty block
instead of per unique WAL entry. A 4MB WriteLBA creates 1024 dirty map
entries (one per 4KB block), all sharing the same WAL offset. The flusher
read the full 4MB WAL entry 1024 times into separate buffers:
1024 × 4MB = 4GB per 4MB write → OOM on mkfs.ext4.

Root cause: flusher assumed 1:1 dirty-block-to-WAL-entry mapping.
WriteLBA supports multi-block writes but the flusher never deduplicated
shared WAL offsets.

Fix: deduplicate WAL reads by WalOffset in flushOnceLocked(). Multiple
dirty blocks from the same WAL entry share one read buffer and one
DecodeWALEntry call. Memory: O(WAL_entries × size) not O(blocks × size).
For a 4MB write: 4GB → 4MB.

Verified on hardware (m01/M02 25Gbps RoCE):
- Before: mkfs.ext4 → VS RSS 100MB→25GB → OOM killed
- After: mkfs.ext4 → VS RSS 129MB stable, mkfs succeeds
- pgbench TPC-B c=4: 1,248 TPS (RF=1, previously blocked by OOM)

Tests added:
- flusher_test.go: flush_multiblock_shared_wal_read (16 blocks share
  one WAL offset, flush dedup verified)
- flusher_test.go: flush_multiblock_data_correct (3 mixed multi-block
  writes, all data correct after flush)
- test/component/large_write_test.go: 7 component tests (single 4MB,
  sequential mkfs sim, concurrent, mixed sizes, production volume,
  flusher throughput 30s sustained)
- iscsi/large_write_mem_test.go: 2 iSCSI session memory tests (4MB
  R2T flow, slow device)

Testrunner enhancements (same commit — all tested on hardware):
- discover_primary action: maps primary IP → topology node name,
  supports alt_ips for multi-NIC (RoCE + management)
- NodeSpec.AltIPs field for multi-NIC node identification
- 5 new YAML scenarios: ec3, ec5, degraded sync_all/best_effort, pgbench
- All 13 hardware-verified scenarios PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 14:24:10 -07:00