seaweedfs/sw-block/design/protocol-anti-patterns.md

Protocol Anti-Patterns in Distributed Storage Systems

Date: 2026-04-09 Status: reference document

Purpose

This document catalogs recurring protocol anti-patterns found in distributed storage systems. Each anti-pattern is defined, then illustrated with concrete cases from real codebases: SeaweedFS (upstream object store), SeaweedFS sw-block (block storage extension), Ceph RADOS, and DRBD.

The goal is to provide a reusable reference for protocol review and design decisions. When a new protocol change is proposed, check it against this list.

---

A1: Heartbeat Timing Defines Recovery Semantics

Definition: A timer or heartbeat interval directly determines a semantic outcome (node death, volume reassignment, recovery type) without collecting independent facts.

Why it is wrong: Heartbeat is transport-layer liveness observation. It measures "when did I last hear from you," not "what is your data state." Using it as the sole input for semantic decisions conflates observation latency with data truth.

Correct principle: Timers trigger observation. Facts determine semantics.

Case: SeaweedFS upstream — dead node detection

topology/volume_location_list.go:69-86

When LastSeen < freshThreshHold (hardcoded 60 seconds), the node is removed from volume location lists. When the heartbeat gRPC stream drops (master_grpc_server.go:67-101), the defer block immediately unregisters the node and marks all volumes unavailable.

A 60-second network hiccup causes immediate volume unavailability. No probe, no corroboration, no fact collection. One missed heartbeat cycle = topology change.

Case: SeaweedFS sw-block — fast replica rejoin missed

When a replica restarts faster than the master's heartbeat detection (~5 seconds), the master never detects the disconnect. No failoverBlockVolumes fires, no recoverBlockVolumes fires, no primary refresh is sent. The primary's shipper holds a dead TCP connection and the volume stays degraded indefinitely.

The system's recovery depends on the timing of the restart relative to the heartbeat cycle, not on the fact of the restart.

Case: Ceph — OSD flapping storms

Early Ceph used OSD heartbeat timeout (20s default) as the trigger for PG peering. Network congestion caused heartbeat delays without actual OSD failure, leading to "flapping" — OSDs bouncing between up and down, triggering unnecessary recovery storms that made congestion worse.

Fix: Ceph separated heartbeat (liveness signal) from PG peering (data-plane fact collection). The monitor marks a node down, but PG peering independently collects pg_info from all OSDs before deciding recovery. The osd_heartbeat_grace and mon_osd_min_down_reporters tuning knobs are artifacts of this evolution.

Case: DRBD — ping timeout triggers resync

DRBD's ping-timeout and connect-int timers originally drove state transitions directly. A slow network caused DRBD to declare the peer dead and start resync — but the peer was alive, just slow.

Fix: DRBD 8.4+ added fencing — you need an external fencing agent to confirm death before split-brain resolution proceeds. The timer triggers the check; the fencing agent provides the fact.

Fix pattern

Timer fires → trigger probe/observation
Probe collects bounded facts (R, S, H, epoch, endpoint)
Facts determine semantic action (no-op, catchup, rebuild, promote)

---

A2: Transport Failure Directly Defines Recovery Type

Definition: A single transport-layer error (TCP reset, write timeout, dial failure) directly triggers a semantic state change (rebuild required, volume degraded, write failed) without distinguishing transient from permanent failure.

Why it is wrong: Transport failure and data-plane recovery are different truth domains. A TCP reset means "I cannot reach you right now," not "your data is divergent and you need a full rebuild."

Correct principle: Transport failure changes reachability truth. Recovery type requires bounded data-plane facts.

Case: SeaweedFS upstream — one replica failure fails entire write

topology/store_replicate.go:73-137

DistributedOperation fans out writes to all replicas in parallel. If ANY replica returns an error, the entire write fails and returns error to the client. No distinction between:

• Transient network hiccup (retry would succeed)
• Replica disk full (permanent, needs different handling)
• Replica truly unreachable (degrade and continue)

A single TCP error from one replica out of three fails the write for the client.

Case: SeaweedFS sw-block — Ship() 3s deadline causes NeedsRebuild

The WAL shipper's Ship() sets a 3-second write deadline on the TCP connection. Under sustained qd=16 writes, the replica couldn't drain the TCP buffer fast enough. The deadline fired, markDegraded() was called, and subsequent WAL recycling made the gap unrecoverable — escalating to NeedsRebuild.

A 3-second TCP write timeout caused a full 1GB rebuild. The transport timeout was over-interpreted as a semantic failure.

Case: DRBD — TCP drop triggers full resync

In DRBD 8.x, a single TCP connection failure could trigger a full resync. The activity log bitmap was supposed to track only changed blocks, but connection drops during heavy I/O could corrupt the bitmap tracking, causing a full resync of the entire device.

Fix: DRBD 9 introduced quorum — transport failure alone doesn't trigger resync. You need quorum loss (confirmed by multiple paths) before the system decides the peer needs rebuilding.

Case: Ceph — transport loss vs acting set change

Ceph's PG peering distinguishes transport loss from acting set changes. A transport failure moves the OSD out of the acting set, but the PG doesn't start recovery until peering collects actual data-plane facts (pg_info exchange). Transport loss changes reachability; recovery requires facts.

Fix pattern

Transport error → mark reachability lost
Keep probing last known endpoint (harmless retries)
When reachable again → collect facts (R, S, H)
Facts determine: no recovery, catchup, or rebuild

---

A3: Ack Arrival Defines Terminal Success

Definition: An acknowledgment message (ack, barrier response, completion signal) is treated as terminal success before the full protocol round-trip closes. The system announces success, then discovers the operation is not actually complete.

Why it is wrong: Observed completion is not the same as closed recovery authority. An ack means "the remote side received and processed your request." It does not mean "all side effects are complete, all cleanup is done, and the system is in a consistent terminal state."

Correct principle: Terminal success is emitted only after executor/session closure completes, not on ack arrival.

Case: SeaweedFS sw-block — double event on rebuild completion

The remote rebuild ack callback emitted SessionCompleted immediately on receiving SessionAckCompleted from the replica. The V2 core transitioned to publish_healthy. Then the executor's cleanup path encountered a "sender stopped" error and emitted SessionFailed, knocking the mode back to degraded. Two contradictory terminal events from the same operation.

Root cause: ack arrival was treated as terminal truth, but the executor was still unwinding.

Fix: Ack updates observed facts only. Terminal success is emitted after executor/session closure succeeds.

Case: SeaweedFS upstream — local write before replica ack

topology/store_replicate.go:49-64

The primary writes locally first, then replicates to remote locations. If local write succeeds but remote replication fails, the primary has data the replicas don't. There is no reconciliation mechanism. The write is partially committed.

If the client connection drops after local write but before replication error is returned, the client may see success (from a previous response) while the replicas don't have the data.

Case: Ceph — committed before fsync

Ceph had a bug where the primary OSD declared a write "committed" after receiving replica acks, but the replica crashed before fsyncing. The fix was min_size enforcement — the write isn't committed until the required number of replicas confirm durable persistence via journal/WAL fsync.

Fix pattern

Ack arrives → update observed facts (progress, achieved LSN)
Do NOT emit terminal success yet
Executor completes cleanup (sender close, session close)
Then emit terminal success

---

A4: Event Ordering Determines Semantics

Definition: The same set of facts produces different semantic outcomes depending on which event arrives first. Two orderings of the same events lead to different recovery paths or different final states.

Why it is wrong: If the system has the same facts, it should make the same decision. Event order should affect convergence latency, not semantic result.

Correct principle: Same facts, same decision, regardless of arrival order.

Case: SeaweedFS sw-block — NeedsRebuildObserved removes sender before rebuild starts

handleReplicaProbeResult calls applyCoreEvent(NeedsRebuildObserved) which emits InvalidateSessionCommand. The command removes the sender's session from the orchestrator registry. Then installSession tries to re-create it, and RebuildStarted fires. But by the time the first ack arrives, a registry reconciliation (triggered by syncProtocolExecutionState inside applyCoreEvent) has removed the sender again.

The sender survives or dies depending on the exact ordering of:

1. InvalidateSessionCommand dispatch
2. installSession ProcessAssignment
3. syncProtocolExecutionState reconciliation
4. First ack arrival

Different timings of these four events produce different outcomes (rebuild succeeds or fails with "sender not found").

Case: SeaweedFS upstream — volume growth race

topology/volume_growth.go:288-349

Concurrent Assign requests race on volume slot reservation. Thread A reserves node1+node2+node3. Thread B sees node3 full (A got it), fails, and releases its reservations. Under high concurrency, different request ordering produces different capacity decisions.

Case: Ceph — PG peering collects all facts first

Ceph's solution to this anti-pattern: the PG peering state machine has GetInfo and GetLog states that wait for responses from ALL peers before advancing to Active. Event order within the collection phase doesn't matter because no decision is made until all facts are present.

Case: Raft — log position, not arrival order

The Raft consensus algorithm is the canonical solution. All decisions are based on log position (term + index), not arrival order. If two messages carry the same log position information, the same decision is made regardless of which arrives first.

Fix pattern

Collect all required facts before making decisions
Use monotonic identifiers (epoch, LSN, sessionID) to order facts
Decision function: f(facts) → action (deterministic, no ordering dependency)
R >= H → no recovery
R >= S → catchup
R < S → rebuild

---

A5: Projection Drives Control Truth

Definition: A derived status field (mode, health, state label) is read back as input to control decisions. The projection is supposed to be a report of truth, but it becomes a hidden control input that creates feedback loops.

Why it is wrong: Projection must be derived only. When a derived field is used as input, the system creates a circular dependency: state determines projection, projection determines state.

Correct principle: Projection reports truth. It must never create truth.

Case: SeaweedFS upstream — ReadOnly status drives assignment

topology/volume_layout.go:267-271

Volume ReadOnly status (self-reported by the volume server) directly controls whether the master adds the volume to the writable list and whether it is eligible for new write assignments. The master has no independent verification.

If a volume server incorrectly marks itself readonly (bug, misconfiguration), the master stops all writes to that volume. The volume server's self-reported status becomes authoritative for the master's assignment decisions.

Case: SeaweedFS upstream — volume compacting status

topology/topology_vacuum.go:347-349

The master skips vacuum for volumes marked readonly — but readonly is itself derived from the volume server's report. A volume server can prevent vacuuming by claiming readonly, even if the volume should be vacuumed.

Case: Kubernetes — Pod phase feedback loop

Pod status.phase (Running, Pending, Failed) is supposed to be a derived report. But many controllers use status.phase as input for reconciliation loops, creating feedback cycles. The Kubernetes community has been migrating toward conditions (explicit fact-based fields) instead of phase (derived summary).

Case: Ceph — PG state as recovery scheduler input

Ceph's PG state (active+clean, active+degraded) was used internally to prioritize recovery scheduling. But degraded was itself derived from the recovery state — creating a loop. Fix: the recovery scheduler works from pg_missing (actual missing objects), not the PG state label.

Fix pattern

Projection = f(facts) — pure derivation, no side effects
Control decisions read facts directly, not projections
If you find a control path reading a projection, refactor to read the
underlying fact instead

---

A6: Timing Workaround States Accumulate in the Engine

Definition: New states, flags, or enum values are added to the semantic engine to survive a specific race condition or timing issue. Over time, the engine becomes a patch history rather than a semantic model.

Why it is wrong: Each workaround state increases the state machine's surface area without adding semantic meaning. The state exists for timing reasons, not because the problem space has a new distinction. Future developers cannot tell which states are essential and which are workarounds.

Correct principle: New state enters the engine only if it expresses a new semantic distinction that cannot be represented by existing dimensions.

Case: SeaweedFS upstream — phantom volume server Counter

data_node.go:24

Counter int // in race condition, the previous dataNode was not dead

When a VS reconnects before the old heartbeat stream's defer runs, two streams exist for the same node. The Counter field tracks active streams and only unregisters when it reaches 0. This is a timing workaround for the race between stream closure and new stream registration. The comment explicitly says "in race condition."

Case: SeaweedFS upstream — reservation timeout

topology/node.go:294

const reservationTimeout = 5 * time.Minute // TODO: make this configurable

Reservations are cleaned up after an arbitrary 5-minute timeout. This is a workaround for failed volume creation that doesn't release its reservation. The timeout exists because there is no proper completion/failure signal for the reservation lifecycle.

Case: Ceph — PG peering state explosion

Ceph's PG peering state machine grew from ~8 states in early versions to ~20+ states. Many intermediate states (Stray, GetMissing, WaitUpThru, Incomplete) were introduced for specific race conditions between OSD restart timing and monitor OSDMap propagation.

WaitUpThru exists solely because the monitor's OSDMap epoch update can arrive before or after PG peering completes — it is a timing workaround state that has persisted for over a decade.

Case: DRBD — connection state proliferation

DRBD's original 4 connection states (StandAlone, Connecting, Connected, Disconnecting) grew to include NetworkFailure, Unconnected, Timeout, BrokenPipe, TearDown — many of which handle specific TCP failure modes rather than semantic differences.

Fix pattern

Before adding a new state, ask:
1. Does this express a new semantic distinction? (keep)
2. Or does it exist to survive a specific timing race? (reject)

If the answer is (2), fix the timing issue in the runtime layer
(retry, epoch comparison, idempotent processing) rather than adding
a new state to the semantic engine.

---

A7: Transport Mechanics Leak Into the Semantic Engine

Definition: TCP connection management, retry logic, goroutine scheduling, or wire protocol details appear inside the semantic state machine instead of being handled by runtime adapters.

Why it is wrong: The semantic engine should process facts and produce decisions. It should not know or care about TCP, gRPC, goroutines, or wire formats. When transport mechanics leak in, the engine becomes coupled to a specific runtime implementation and cannot be tested or reasoned about independently.

Correct principle: The engine processes facts. Runtime adapters handle transport.

Case: SeaweedFS sw-block — sender registry as transport gate

The V2 engine's sender registry (Orchestrator.Registry) serves dual purpose: semantic session tracking AND transport connection management. When InvalidateSessionCommand fires, it nulls the sender's session (semantic) and the same sender is used by the ack observation path to validate incoming TCP frames (transport). The semantic invalidation kills the transport path.

This coupling caused the "sender not found" rebuild failure: the engine's semantic state change (invalidate session) removed the transport path needed by the rebuild coordinator.

Case: Ceph — messenger abstraction

Ceph explicitly separates the semantic layer (PG peering state machine) from the transport layer (msgr2 messenger). The PG state machine never directly touches TCP connections. It produces messages that the messenger delivers. Connection failures are reported to the state machine as facts, not as transport errors.

Fix pattern

Engine interface:
  Input: facts (epoch, R, S, H, ack kind, session ID)
  Output: decisions (no-op, catchup, rebuild, promote)

Runtime adapter:
  Translates TCP events → facts
  Translates decisions → TCP actions
  Handles retry, timeout, connection lifecycle

The engine never imports net, io, or sync packages.

---

Summary Table

Anti-Pattern	SeaweedFS upstream	sw-block V2	Ceph	DRBD
A1: Timer defines semantics	Dead node detection (60s)	Fast rejoin missed	OSD flapping storms	Ping timeout resync
A2: Transport = recovery	One replica fails all	Ship() 3s deadline	—	TCP drop = full resync
A3: Ack = terminal success	Local write before ack	Double SessionCompleted/Failed	Committed before fsync	—
A4: Ordering determines result	Volume growth race	Sender removed before rebuild	Solved: collect all facts first	—
A5: Projection as control	ReadOnly drives assignment	—	PG state as scheduler input	—
A6: Workaround states	Counter field, reservation timeout	—	WaitUpThru, 20+ PG states	9+ connection states
A7: Transport in engine	—	Sender registry dual-purpose	Solved: messenger abstraction	—

---

Using This Document

When reviewing a protocol change:

1. Check if the change matches any anti-pattern above
2. If it does, ask: can the same goal be achieved without the anti-pattern?
3. Apply the "fix pattern" suggested for that anti-pattern
4. If no clean alternative exists, document why the exception is necessary

When designing a new protocol:

1. Separate identity truth (who), reachability truth (can I reach), recovery truth (what data action), and execution (transport mechanics)
2. Ensure each truth domain has a single authority
3. Ensure timers trigger observations, never semantic decisions
4. Ensure same facts produce same decisions regardless of event order
5. Keep transport mechanics in runtime adapters, not the semantic engine

---

This document is a living reference. Add new cases as they are discovered.