diff --git a/sw-block/design/external-failure-taxonomy.md b/sw-block/design/external-failure-taxonomy.md new file mode 100644 index 000000000..dee09df78 --- /dev/null +++ b/sw-block/design/external-failure-taxonomy.md @@ -0,0 +1,420 @@ +# External Failure Taxonomy + +Date: 2026-04-11 +Status: reference +Purpose: Catalog real production bugs from Ceph/DRBD/Mayastor/Longhorn, +extract semantic lessons, and map to V2 exposure and V3 prevention rules. + +## How to read this document + +Each entry answers: +1. What happened (the failure) +2. Why it happened (the root semantic mistake) +3. Did V2 hit this? (or could it) +4. What V3 rule prevents it + +The value is not the bugfix — it is the semantic lesson. + +--- + +## Category 1: Membership / Liveness Misjudgment + +### 1.1 Ceph: Asymmetric heartbeat flapping +**Source**: [Ceph #14181](https://tracker.ceph.com/issues/14181) + +**What**: OSDs flapped up/down continuously. Monitors saw them as dead +(cluster network heartbeat failed), but OSDs saw themselves as alive +(public network to monitor worked). Flap loop: mark down → mark up → repeat. + +**Root semantic mistake**: Heartbeat and control-plane used different +network paths. A partial network failure created contradictory liveness +views. Timer-based death declaration with no corroboration. + +**V2 exposed?**: Yes. Our fast-rejoin test showed: master didn't detect +3-second replica disconnect because heartbeat cycle was 5 seconds. +Recovery depended on timing, not facts. + +**V3 rule**: P1 (timers trigger, facts decide). Liveness observation +on one path must not contradict identity truth on another. Primary +probes independently of master heartbeat. + +--- + +### 1.2 Ceph: Stale epoch checked before sync +**Source**: [Ceph #22673](https://tracker.ceph.com/issues/22673) + +**What**: After OSD destroy + recreate, OSD refused to start — checked +DESTROYED flag against locally-cached stale OSDMap epoch instead of +syncing to current epoch first. + +**Root semantic mistake**: Membership decision made against stale local +state. The check ran before the sync. + +**V2 exposed?**: Similar pattern. Our stale-cleanup race: the master's +heartbeat handler checked the block registry (stale) before the VS had +time to report its volumes. Entry deleted prematurely. + +**V3 rule**: P5 (monotonic facts win). Never validate identity against +stale local epoch. Sync to authoritative source first. + +--- + +### 1.3 DRBD: Split brain from asymmetric UUID rules +**Source**: [DRBD 9.0.20-1 release notes](https://github.com/LINBIT/drbd/releases/tag/drbd-9.0.20-1) + +**What**: Split brain detected despite quorum, after resize operation. +UUID propagation differed between diskless and disk-backed nodes. + +**Root semantic mistake**: Generation identifier (UUID) rules were +asymmetric across node types. Same logical operation produced different +identity artifacts depending on node role. + +**V2 exposed?**: Yes. Our replicaID format mismatch: shipper used +`serverID`, engine used `path/serverID`, orchestrator used yet another +format. Same identity, three representations. + +**V3 rule**: Single canonical identity format. `MakeReplicaID(path, serverID)` +is the only constructor. No format conversion in hot paths. + +--- + +### 1.4 Longhorn: Instance manager restart blindspot +**Source**: [Longhorn #5809](https://github.com/longhorn/longhorn/issues/5809) + +**What**: After instance manager pod restart, Longhorn was unaware +replica processes were dead for 60 seconds. Replicas appeared alive +but were actually killed. + +**Root semantic mistake**: The liveness monitor died with the component +it was monitoring. No watchdog survived the restart. + +**V2 exposed?**: Yes. When the primary's shipper held a dead TCP +connection after replica restart, it didn't detect the failure until +the next barrier attempt (~30s later). + +**V3 rule**: Liveness detection must be independent of the transport +being monitored. The watchdog recheck fires on a timer regardless of +whether Ship/Barrier calls are happening. + +--- + +## Category 2: Recovery Decision Error + +### 2.1 DRBD: Full resync when partial would suffice +**Source**: [DRBD 9.1.19 ChangeLog](https://github.com/LINBIT/drbd/blob/drbd-9.1/ChangeLog) + +**What**: After brief disconnect + reconnect, DRBD performed full +resync of all data instead of bitmap-tracked partial resync. Multiple +root causes across versions: bitmap loss, UUID mismatch during resync, +UUID history slot corruption. + +**Root semantic mistake**: The UUID comparison that decides partial vs +full resync was fragile against transient disconnects. A UUID change +during active resync was misinterpreted as data divergence. + +**V2 exposed?**: Yes. Our "no WAL retention" caused every disconnect +(even 3 seconds) to force full 1GB rebuild instead of catchup. The +decision boundary (R < S → rebuild) was too coarse. + +**V3 rule**: P4 (deterministic recovery choice). Recovery decision +from bounded facts only: `R >= S → catchup, R < S → rebuild`. +SmartWAL + LBA dirty map eliminates the S boundary entirely — recovery +is always "send dirty blocks from extent." + +--- + +### 2.2 Longhorn: Rebuild sent to wrong replica (address reuse) +**Source**: [Longhorn #5709](https://github.com/longhorn/longhorn/issues/5709) + +**What**: After instance-manager restart, rebuild command was sent to +wrong replica. A new volume's replica received the same IP:port as the +old failed replica. Rebuild targeted wrong volume. + +**Root semantic mistake**: Recovery command routed by network address, +not by stable replica identity. Address reuse after restart caused +identity collision. + +**V2 exposed?**: Yes. Our nil-interface panic in `senderByID` was +caused by a sender lookup returning nil `*Sender` wrapped in a non-nil +interface. The registry keyed by `replicaID` strings, and format +mismatch caused lookup failures. + +**V3 rule**: P2 (single authority per truth domain). Replica identity +must use stable, unique IDs (epoch + path + serverID). Never derive +identity from reusable network addresses. + +--- + +### 2.3 Ceph: Backfill cancellation crashes target OSD +**Source**: [Ceph #21613](https://tracker.ceph.com/issues/21613) + +**What**: Primary sent REJECT message to backfill target to cancel +backfill. Target crashed — REJECT is a message the target sends to the +primary, not the other way around. + +**Root semantic mistake**: Protocol message repurposed in wrong +direction. The state machine on the receiving end never expected this +message from that sender. + +**V2 exposed?**: Similar pattern. Our `InvalidateSessionCommand` +dispatch path invalidated the sender's session (semantic cleanup) which +killed the transport path needed by the rebuild coordinator. The +command was semantically valid but operationally destructive in context. + +**V3 rule**: Protocol messages have fixed sender/receiver roles. +Recovery commands flow from primary to replica. Acknowledgments flow +from replica to primary. Never reverse the direction. + +--- + +## Category 3: Completion / Durability Illusion + +### 3.1 Longhorn: Corrupted replica kept, healthy one discarded +**Source**: [Longhorn #7425](https://github.com/longhorn/longhorn/issues/7425) + +**What**: After rebuild timeout + node failure, corrupted replica was +kept and healthy replica was discarded. Autosalvage selected replicas +by `healthyAt` timestamp — corrupted replica had a valid timestamp from +before the rebuild started. + +**Root semantic mistake**: Health metadata was not invalidated when +rebuild started. The metadata said "healthy" but the data was corrupt +from a partial rebuild. + +**V2 exposed?**: Yes. Our double-event bug: `SessionCompleted` emitted +on ack arrival, then `SessionFailed("sender stopped")` emitted during +executor cleanup. The system briefly declared success then reverted to +failure. + +**V3 rule**: A3 (ack arrival ≠ terminal success). Health metadata must +be invalidated at rebuild START, not just at rebuild FAILURE. Terminal +success is emitted only after executor/session closure completes. + +--- + +### 3.2 Longhorn: Healthy replica deleted due to address collision +**Source**: [Longhorn #9216](https://github.com/longhorn/longhorn/issues/9216) + +**What**: In a 3-replica degraded volume, the last healthy replica was +deleted. A stopped replica was assigned the same port as a previously +deleted one. The engine saw two replicas at the same address and +deleted the "redundant" one — which was the only healthy copy. + +**Root semantic mistake**: Replica identity derived from network address. +Port reuse after restart created identity collision. Two physically +different replicas appeared as the same entity. + +**V2 exposed?**: Yes. The `Counter` field in SeaweedFS upstream's +`data_node.go` is a workaround for the same problem — two heartbeat +streams for the same IP:port after fast restart. + +**V3 rule**: Replica identity = `(epoch, volumePath, serverID)`. +Immutable, never derived from ports or transport addresses. + +--- + +### 3.3 Ceph: Rolling restarts orphan objects +**Source**: [Ceph #52385](https://tracker.ceph.com/issues/52385) + +**What**: After restarting all nodes sequentially without waiting for +`HEALTH_OK`, PGs entered `recovery_unfound` with permanently lost +objects. Objects existed on stray OSDs excluded from the acting set. + +**Root semantic mistake**: Peering completion was not waited for before +proceeding with the next restart. Partial membership view during +peering excluded OSDs that had the data. + +**V2 exposed?**: Similar risk. If the master's primary refresh isn't +delivered before the next heartbeat cycle, the primary operates with a +partial roster. The rebuild may target the wrong set of replicas. + +**V3 rule**: Recovery operations must wait for complete membership view. +Never start recovery with a partial roster. + +--- + +## Category 4: Ordering / Race Conditions + +### 4.1 DRBD: Write races with very short resync +**Source**: [DRBD 9.2.14 ChangeLog](https://github.com/LINBIT/drbd/blob/drbd-9.1/ChangeLog) + +**What**: A write during a very short resync was not resynced, leaving +replicas divergent. The resync marked the region "clean" while the write +was in-flight. + +**Root semantic mistake**: Region bitmap clear was not ordered after +concurrent write drain. The bitmap said "synced" but an in-flight write +had not been applied to the peer. + +**V2 exposed?**: Our rebuild bitmap has the same structure. The rule +"WAL-applied wins over base-applied" handles this: the bitmap is SET +by the live lane, and base lane checks before installing. But a race +between bitmap check and base install could produce the same bug if +the lock granularity is wrong. + +**V3 rule**: Recovery completion must not mark a region clean while +writes are in-flight to that region. Drain before clear. + +--- + +### 4.2 DRBD: Parallel connection handshake interference +**Source**: [DRBD 9.1.23 ChangeLog](https://github.com/LINBIT/drbd/blob/drbd-9.1/ChangeLog) + +**What**: In 3+ node setups, parallel connection establishment left one +connection in `WFBitMapT/Established` — an inconsistent half-syncing +state. + +**Root semantic mistake**: Per-connection state transitions were not +isolated. One connection's handshake interfered with another's. + +**V2 exposed?**: Our `ShipperGroup` manages multiple shippers. If +`ProbeReconnectAll` probes all replicas in parallel, one probe's state +change could interfere with another's if they share state. + +**V3 rule**: Per-replica state must be fully isolated. No shared +mutable state between recovery sessions for different replicas. + +--- + +### 4.3 Longhorn: Zombie engine from rapid attach/detach +**Source**: [Longhorn #11605](https://github.com/longhorn/longhorn/issues/11605) + +**What**: Attach + detach within 1 second left an orphaned engine +process. Detach happened before the instance appeared in the CR, +so nothing knew to clean it up. + +**Root semantic mistake**: Deletion checked once and gave up. +The target wasn't visible during that single check window because +of CR sync delay. + +**V2 exposed?**: Our `cancelAndDrain` with 5-second timeout is the +same pattern. If the catch-up goroutine doesn't exit within 5s, we +abandon it. The zombie goroutine may hold resources indefinitely. + +**V3 rule**: Cleanup must retry until confirmed absent. A single +check-miss is not proof of absence. + +--- + +### 4.4 Mayastor: Internal/external state divergence on fault +**Source**: [Mayastor #549](https://github.com/openebs/mayastor/issues/549) + +**What**: Faulting a nexus child removed it from internal structure but +NOT from external child list. Destroy caused state machine assertion +failure because internal and external representations disagreed. + +**Root semantic mistake**: Partial cleanup. Internal state changed but +external-facing state was not updated atomically. + +**V2 exposed?**: Our sender registry vs shipper state divergence. +`InvalidateSessionCommand` nulled the sender's session but the shipper's +transport state was unchanged. The registry said "no session" but the +shipper still had a live TCP connection. + +**V3 rule**: State changes must be atomic across all representations. +If you invalidate a session, both the engine state AND the transport +state must reflect it. + +--- + +## Category 5: Background Work Corrupts Semantics + +### 5.1 Ceph: Recovery priority starvation +**Source**: [Ceph #62811](https://tracker.ceph.com/issues/62811) + +**What**: PGs stuck in `backfilling` indefinitely. Recovery work was +continuously preempted by peering events and sub-op reads from other +PGs. Recovery could never make forward progress. + +**Root semantic mistake**: No guaranteed forward-progress for recovery. +Each individual preemption was "correct" but infinite preemption = no +recovery = semantic failure. + +**V2 exposed?**: Our recovery goroutine competes with heartbeat +processing on the same thread. If heartbeat processing takes too long +(e.g., many volumes), recovery stalls. + +**V3 rule**: Recovery must have guaranteed forward-progress slots that +cannot be infinitely preempted. Rate-limit interruptions. + +--- + +### 5.2 Mayastor: Rebuild thundering herd after node loss +**Source**: [Mayastor #1714](https://github.com/openebs/mayastor/issues/1714) + +**What**: When 1 node went down in a 3-node cluster, all volumes +attempted rebuild simultaneously. Connection failures cascaded because +rebuild descriptors couldn't acquire IO channels. Each failed rebuild +faulted its child, and retry created a thundering herd. + +**Root semantic mistake**: No admission control for concurrent rebuilds. +Automated remediation without backoff worsened the overload. + +**V2 exposed?**: Currently only 1 volume in tests. With 100+ volumes, +simultaneous rebuild after a node failure would create the same +thundering herd on our rebuild server. + +**V3 rule**: Rebuild scheduling after node failure must be staggered. +Maximum concurrent rebuilds per node. Exponential backoff on failure. + +--- + +### 5.3 Mayastor: HA feedback loop under shared pool load +**Source**: [Mayastor #1331](https://github.com/openebs/mayastor/issues/1331) + +**What**: Shared DiskPool under concurrent IO from 6 pods triggered +timeouts. HA interpreted timeouts as node failure, initiated target +replacement, generating more IO, causing more timeouts. Positive +feedback loop. + +**Root semantic mistake**: HA reaction to resource exhaustion did not +include backoff. The "fix" (replace target) consumed more resources +than the failure it was trying to fix. + +**V2 exposed?**: If the rebuild server saturates the disk with base +block reads while the primary is also serving live writes, the same +resource exhaustion → timeout → retry loop could occur. + +**V3 rule**: HA/rebuild reactions must detect resource exhaustion and +back off. Automated remediation without admission control cascades. + +--- + +## Mapping to V3 Anti-Patterns + +| External Lesson | V3 Anti-Pattern | V3 Principle | +|---|---|---| +| Timer-based death without corroboration | A1 | P1: timers trigger, facts decide | +| Stale epoch used for membership decision | A1 | P5: monotonic facts win | +| Transport error → full rebuild | A2 | P6: transport ≠ semantic overreach | +| Ack → success before close | A3 | P2: terminal authority = executor close | +| Address reuse → identity collision | A4 | Stable IDs, never address-derived | +| Health metadata survives rebuild | A5 | P3: projection is derived only | +| Partial cleanup leaves split state | A4 | Atomic state changes across all representations | +| Priority starvation blocks recovery | A6 | Guaranteed forward progress | +| Thundering herd of rebuilds | A6 | Admission control + stagger | + +--- + +## "Would V2 Have This Bug?" Checklist + +Use this for self-audit. For each external bug, ask: could our system +hit the same failure class? + +| # | Failure Class | V2 Exposed? | Status | +|---|---|---|---| +| 1 | Heartbeat timing = semantic death | Yes (fast rejoin) | Fixed: primary probes independently | +| 2 | Stale local state for membership check | Yes (stale-cleanup race) | Fixed: RegisteredAt grace period | +| 3 | Asymmetric identity rules | Yes (replicaID format mismatch) | Fixed: resolveEngineReplicaID | +| 4 | Liveness monitor dies with target | Yes (shipper holds dead conn) | Partially fixed: watchdog recheck | +| 5 | Transport error → full rebuild | Yes (no WAL retention) | Designed: SmartWAL + LBA dirty map | +| 6 | Address reuse → wrong target | Not yet (single volume tests) | Risk at scale: need stable IDs | +| 7 | Ack = terminal success | Yes (double event) | Fixed: suppression + manual OnRebuildCompleted | +| 8 | Health metadata survives rebuild | Not yet | Risk: need to track | +| 9 | Partial cleanup (sender vs session) | Yes (sender registry) | Fixed: emitTerminal=false | +| 10 | Recovery starvation | Not yet | Risk at scale with many volumes | +| 11 | Rebuild thundering herd | Not yet | Risk at scale: need admission control | + +--- + +*This document is updated as new external cases are discovered.*