seaweedfs/sw-block/design/v2-two-loop-protocol.md

# V2 Two-Loop Protocol

Date: 2026-04-05
Status: active

## Purpose

This note fixes the protocol boundary for the next V2 step.

The goal is not to finalize every wire field before implementation.
The goal is to make the ownership boundary stable enough that automata,
constraints, and runtime packages can be reorganized without mixing identity
control and replication consensus again.

## Core Rule

The protocol is split into two loops:

1. `Loop 1`: identity control
2. `Loop 2`: data control

These loops must not be collapsed into one heartbeat or one state owner.

## Authority Principle

Each volume should be treated as a small distributed cluster:

1. `masterv2` is the identity authority outside the cluster
2. the selected primary is the data-control authority inside the cluster
3. replicas report bounded facts to the primary, not full truth to `masterv2`

This means:

1. `masterv2` decides who is allowed to own the role
2. the new primary decides how takeover, catch-up, and rebuild proceed
3. `masterv2` may query bounded facts for arbitration, but it does not choreograph
   data recovery step by step

## Loop 1: Identity Control

Owner:

- `masterv2 <-> volumev2`

Frequency:

- low
- heartbeat scale
- assignment scale
- promotion-query scale

## Three Control Channels

Within `Loop 1`, the control plane should be split into three different
channels. They must not be collapsed into one message type.

### 1. Heartbeat

Direction:

- `volumev2 -> masterv2`

Frequency:

- periodic
- lightweight

Purpose:

1. liveness detection
2. compressed outward mode
3. confirmation that assignment was applied

Heartbeat should carry only:

1. `NodeID`
2. per-volume `Mode`
3. applied `Epoch`
4. applied `Role`
5. `RoleApplied`
6. `ReplicaReady`
7. optional passive `CommittedLSN` cache

Heartbeat should not carry:

1. per-replica progress
2. catch-up targets
3. rebuild detail
4. full failover evidence

If `CommittedLSN` is carried in heartbeat, it is only a passive cache. The
authoritative failover-time value still comes from promotion query.

### 2. Promotion Query

Direction:

- `masterv2 -> candidate volumev2`
- candidate `volumev2 -> masterv2`

Frequency:

- on demand
- only during failover or promotion arbitration

Purpose:

1. obtain fresh failover evidence
2. avoid treating stale heartbeat cache as authority

Candidate response should include:

1. `CommittedLSN`
2. `WALHeadLSN` as a weaker tiebreaker
3. `Epoch`
4. `Role`
5. `ReceiverReady`
6. bounded eligibility reason if not promotable

The promotion query is where fresh identity-loop evidence is collected.
It is not a replacement for the data-control loop, and it is not a continuous
replication-progress feed.

If heartbeat also carries `CommittedLSN`, promotion query still wins whenever
fresh arbitration is required.

### 3. Assignment

Direction:

- `masterv2 -> volumev2`

Frequency:

- on demand
- role change or membership change

Purpose:

1. authorize role ownership
2. fence stale owners
3. deliver replica-set identity

### Master To Volume

`masterv2 -> volumev2` carries only:

1. `Epoch`
2. `Role`
3. `LeaseTTL`
4. `ReplicaSet` identities and addresses

It does not carry:

1. per-replica progress
2. catch-up target history
3. detailed rebuild plan

### Volume To Master

`volumev2 -> masterv2` carries only bounded identity evidence:

1. applied `Epoch`
2. applied `Role`
3. outward `Mode`
4. `RoleApplied`
5. `ReplicaReady`

This channel must remain lightweight. Fresh failover evidence belongs to the
promotion-query channel, not the periodic heartbeat.

## Loop 2: Data Control

Owner:

- `primary engine <-> replica engine`

Frequency:

- high
- write scale
- barrier scale
- reconnect scale

This is where replication consensus lives.

### Primary To Replica

Primary-side data-control messages should cover:

1. WAL entry stream
2. barrier request with `Epoch` and target LSN
3. reconnect or resume handshake
4. rebuild/catch-up execution requests when needed

### Replica To Primary

Replica-side data-control messages should cover:

1. `FlushedLSN`
2. bounded status such as `ok`, `epoch_mismatch`, `timeout`, `fsync_failed`
3. reconnect gap evidence
4. coarse local recovery state

### Primary-Owned Per-Replica State

The primary brain should own:

1. `ReplicaFlushedLSN`
2. `ShippedLSN` as diagnostic only
3. replica `State`
4. `CatchUpTarget`
5. `RetentionFloor`
6. `LastContactTime`

## Role Of Masterv2

`masterv2` authorizes:

1. who is primary
2. who is replica
3. which epoch is active
4. when stale owners must be fenced

`masterv2` must not decide:

1. replay from LSN `X` to `Y`
2. whether the next action is `keepup` or `catchup`
3. how rebuild is executed
4. continuous commit progress

`masterv2` may query candidates for fresh promotion evidence, but it still does
not become the owner of replication history.

## Reconstruction And Takeover

Promotion and reconstruction are related, but they do not have the same owner.

### What Masterv2 Leads

`masterv2` leads:

1. failover detection
2. epoch fencing
3. candidate query for fresh promotion evidence
4. primary selection
5. assignment of the new primary role

### What The New Primary Leads

The selected replacement primary leads:

1. local assignment realization
2. collection of self and peer replica summaries
3. bounded truth reconstruction
4. fail-closed takeover gating
5. follow-on `keepup`, `catchup`, or `rebuild` orchestration

### Rule

`masterv2` may say:

1. "you are now the authorized primary candidate for epoch `E`"
2. "these are the members of the replica set"

But `masterv2` must not say:

1. "replay from LSN `X` to `Y`"
2. "use replica `R` as the rebuild source"
3. "enter `catchup` before `rebuild`"
4. "the cluster is safe because my last cached view looked healthy"

## Role Of Primary Brain

The primary brain discovers:

1. current replica state
2. gap or retention situation
3. barrier success or failure
4. whether the volume is `keepup`, `catchup`, `degraded`, or `needs_rebuild`

The primary brain decides:

1. keep shipping
2. start catch-up
3. escalate to `needs_rebuild`
4. start rebuild after role/assignment allows it

## Distributed State-Machine Rules

### 1. Different Nodes Have Different Views

Each node must distinguish:

1. local execution truth
2. last observed peer truth
3. cluster identity truth from `masterv2`

Do not collapse these into one blob.

### 2. All Peer Observations Are Epoch-Scoped

Any peer observation that affects recovery must be tied to:

1. `Epoch`
2. session or generation token

Old-epoch observations must be ignored or fail closed.

### 3. New Primary Reconstructs Truth

After failover, the new primary must rebuild its own data-control truth from:

1. local state
2. peer summaries
3. reconnect handshakes

It must not trust `masterv2` as a cache of full recovery history.
It may use `masterv2` only as the source of authorization and replica identity.

### 4. Outward Mode Is Compressed Evidence

`allocated_only`, `bootstrap_pending`, `publish_healthy`, `degraded`, and
`needs_rebuild` are public meanings, not the full internal recovery automaton.

### 5. Ambiguity Fails Closed

When barrier lineage, progress lineage, or epoch lineage is unclear, the system
must prefer:

1. `degraded`
2. `needs_rebuild`
3. no promotion without enough eligibility evidence

## Constraint Migration

Most of the last week's V2 work remains valid. The important change is where
each constraint belongs.

### Keep As-Is

These constraints still stand:

1. epoch fencing
2. one active session per replica per epoch
3. `catchup` and `rebuild` are different paths
4. fail closed on ambiguous recovery truth
5. semantics first, adapters later

### Move To Loop 1

These belong to identity control:

1. assignment application
2. role ownership
3. lease ownership
4. stable replica identity and addressing
5. compressed heartbeat evidence
6. on-demand promotion query for fresh evidence

### Move To Loop 2

These belong to data control:

1. committed/durable/checkpoint boundaries
2. barrier result meaning
3. replica progress
4. keepup/catchup/rebuild progression
5. retention-floor and catch-up targeting

## Existing V2 Seeds To Reuse

The current V2 code already has the right seeds for the primary-led loop:

1. `sw-block/engine/replication/state.go`
2. `sw-block/engine/replication/event.go`
3. `sw-block/engine/replication/command.go`
4. `sw-block/engine/replication/sender.go`
5. `sw-block/engine/replication/session.go`
6. `sw-block/engine/replication/registry.go`

The current MVP already has the right seeds for the identity loop:

1. `sw-block/runtime/masterv2/master.go`
2. `sw-block/runtime/volumev2/control_session.go`
3. `sw-block/runtime/volumev2/orchestrator.go`

## Immediate Next Step

Before deeper implementation, the codebase should next define:

1. the minimal `Loop 1` contract types in code, split into heartbeat, promotion
   query, and assignment
2. the minimal `Loop 2` progress and reconnect contract draft
3. the automata ownership map showing which engine events and commands belong
   to identity control versus data control

## Promotion Logic

Promotion should use fresh on-demand evidence, not stale heartbeat cache.

Recommended judgment order:

1. fence the old primary by epoch
2. query all surviving candidates
3. reject any candidate with wrong epoch, wrong role lineage, or not-ready
   receiver state
4. choose the candidate with highest `CommittedLSN`
5. use `WALHeadLSN` only as a tiebreaker for equally committed candidates
6. assign new primary role at a new epoch

This keeps the durability boundary centered on `CommittedLSN`, which is the
last LSN that satisfied the configured durability mode such as `sync_all` or
`sync_quorum`.