mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-21 22:56:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

v3-batch-process §12-§13 + v3-architecture.md (architect first-order)

Browse Source 
Architect additions to v3-batch-process.md:
  §12 Architect review checklist — system-level drift check (4 questions:
      scope truth, V2/new-build decision, engine/adapter impact,
      product usability level). Close-sign template enforces honest
      "Done / Not done / Product level reached / Next gate" statement.
  §13 Failure modes process must prevent — 5 observed waste sources:
      V2 porting drift, engine/adapter change without reason, function
      close = happy path only, product unusable despite green tests,
      missing architecture component (catches like binary wiring +
      G9A placement gap).
  §14 (renumbered from §12) — ownership table unchanged.

v3-architecture.md (NEW first-order doc, architect-authored):
  369 lines, 15 sections covering component map, truth domains,
  control-plane + data-plane flows, recovery architecture, failure
  model, operator interface, P15 gate alignment, product completion
  ladder, open architecture decisions, change discipline.

  Bridges the gap surfaced in conversation: WHAT (gates) + behavior
  contracts + anti-patterns existed; HOW the system fits together
  was missing. v3-architecture.md is now peer of mvp-scope-gates.md
  + block-behavior-contract-index.md as first-order references.

  v3-batch-process §8 control-doc table updated to include
  v3-architecture.md as architect-owned, "component/responsibility/
  flow changes" trigger.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
pingqiu
2026-04-26 20:55:07 -07:00
co-authored by Claude Opus 4.7
parent a8b0999c45
commit 0965a36b16
2 changed files with 409 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sw-block/design/v3-architecture.md
+369
View File
@@ -0,0 +1,369 @@
# V3 Architecture Blueprint
**Date**: 2026-04-26
**Status**: Draft first-order architecture reference for P15+
**Owner**: V3 Architect
**Purpose**: describe how V3 fits together as a product system, so P15 gates and compact batch plans do not rediscover major boundaries late.
---
## 1. Why This Document Exists
P15 has several first-order references:
- `v3-phase-15-mvp-scope-gates.md` defines what must pass.
- `v3-block-behavior-contract-index.md` defines behavioral contracts.
- `protocol-anti-patterns.md` defines what must not re-enter.
- `v3-product-placement-authority-rationale.md` explains placement and authority philosophy.
- `v3-invariant-ledger.md` records tested claims.
This document fills the missing middle:
```text
product goal
-> system architecture
-> gates
-> batch plans
-> code and evidence
```
It is not a replacement for the gates. It is the blueprint that shows how each gate belongs to one coherent product architecture.
---
## 2. Architecture Thesis
V3 is a replicated block storage product with a strict separation between intent, authority, and execution.
```text
operator / CSI / API intent
-> placement controller computes desired topology
-> authority publisher mints epoch / endpoint version / assignment
-> volume daemons execute frontend IO, durability, replication, recovery
-> evidence surfaces report health, lag, rebuild, and gate status
```
The master is not just a rigid assignment server. It is the production control-plane interface through which operators and orchestrators express safe intent. Internally, it must keep policy and authority separate:
- policy/controller decides what topology is desired
- publisher mints what authority is current
- runtime executes without minting authority
This preserves V3's protocol discipline while still allowing V2-like product ergonomics.
---
## 3. Component Map
### 3.1 Product Control Plane
| Component | Responsibility | Must not do |
|---|---|---|
| Operator / CLI / API / CSI | Express user intent: create, attach, drain, replace, resize, snapshot, delete | Mint epoch, stuff assignment, choose local role |
| Product controller | Convert intent and policy into desired topology generation | Treat heartbeat as authority, mutate runtime state directly |
| Authority publisher / master | Validate observations against desired topology; mint epoch, endpoint version, assignment | Hide placement policy in heartbeat timing; silently reverse authority |
| Durable authority store | Persist current authority and topology generation | Depend on process memory as sole truth |
| Evidence/status surface | Explain current line, health, lag, rebuild, failures, unsupported decisions | Return green while data path is not usable |
### 3.2 Product Data Plane
| Component | Responsibility | Must not do |
|---|---|---|
| Volume daemon | Host one or more volume runtimes and report observations | Invent placement or authority |
| Frontend target | Serve iSCSI/NVMe/API IO only when projection authorizes | Re-derive authority or bypass stale-primary fence |
| Durable provider/backend | Persist local block data and WAL/checkpoint state | Let session lifecycle close shared persistent volume state incorrectly |
| Replication volume | Fan out primary writes to eligible peers under authority lineage | Ship writes without lineage or peer-set context |
| Replica listener | Accept incoming live/recovery traffic and apply through gates | Treat recovery traffic as live traffic or skip stale live writes silently |
| Engine/adapter | Own recovery state, session fate, and projection | Parse diagnostic strings as protocol decisions |
---
## 4. Truth Domains
Each fact has one owning domain.
| Truth | Owner | Consumers |
|---|---|---|
| Desired topology generation | Product controller | Publisher, diagnostics, placement plan API |
| Current epoch / endpoint version / assignment | Authority publisher | Volume daemon, frontend, replication, status |
| Local replica observation | Volume daemon | Master observation host |
| Frontend write/read readiness | Adapter projection | iSCSI/NVMe/API providers |
| Durability of local write | Durable backend | Replication coordinator, frontend ack path |
| Remote durability ack | Replication transport + peer | Durability coordinator |
| Recovery decision and retry budget | Engine | Adapter/executor |
| Per-LBA applied frontier | Replica apply gate / substrate contract | Recovery apply path |
Review rule: if a change creates a second owner for any row, it is architectural drift.
---
## 5. Control-Plane Flow
### 5.1 Create / Placement
```text
CreateVolume(size, rf, class)
-> controller filters eligible nodes
-> controller writes desired topology generation
-> publisher validates observations against generation
-> publisher mints assignment line
-> volume daemons subscribe and self-determine role
```
P15 minimum: flat-topology placement is enough if G9A is accepted as MVP. Rack/AZ awareness and load-based rebalance can defer, but declared topology/manual placement must not be silently presented as dynamic assignment.
### 5.2 Assignment Subscription
Assignment facts are volume-scoped and master-minted. Every subscriber may see the same fact:
- `fact.ReplicaID == self.ReplicaID`: this daemon is primary for the line
- `fact.ReplicaID != self.ReplicaID`: this daemon is supporting replica or superseded
The volume daemon records and reacts to the fact; it does not reinterpret local role as authority.
### 5.3 Node Lifecycle
```text
node join
-> observation appears
-> controller may include node in future topology generation
node drain
-> controller excludes node from new placements
-> controller emits replacement topology changes
-> runtime catches up / rebuilds replacements
-> old replica is fenced or retired
node decommission
-> no new assignment
-> no remaining durable responsibility
```
This is why placement is part of product architecture, not a UI convenience.
---
## 6. Data-Plane Flow
### 6.1 Write Path
```text
frontend write
-> projection/stale-primary check
-> local durable append / fsync boundary
-> replication fan-out to current peers
-> durability mode evaluation
-> host-visible ack or error
```
Durability modes must remain explicit:
- `best_effort`: local durable boundary plus best-effort shipping
- `sync_all`: all eligible peers satisfy barrier
- `sync_quorum`: quorum satisfies authority-aligned durability
No remote ack can count without authority lineage.
### 6.2 Read Path
```text
frontend read
-> projection/stale-primary check
-> local durable read
-> protocol response
```
Reads are authority-gated because stale reads can poison filesystems and databases just as badly as stale writes.
### 6.3 Replication Path
```text
primary local write observed
-> ReplicationVolume ships entry to ReplicaPeer
-> BlockExecutor dials peer DataAddr
-> ReplicaListener receives frame
-> apply gate dispatches live or recovery lane
-> substrate applies or rejects
```
`--data-addr` is the replication listener bind for the current G5 binary path because peer data traffic dials `AssignmentFact.Peers[*].DataAddr`.
---
## 7. Recovery Architecture
Recovery is engine-owned, not transport-owned.
```text
probe
-> engine classifies catch-up vs rebuild
-> adapter dispatches session via executor
-> transport streams catch-up or rebuild
-> replica apply gate prevents per-LBA regression
-> session closes explicitly
-> engine updates truth
```
Required rules:
- recovery starts from engine state, not raw probe payload
- catch-up scans from `replica flushed + 1`
- WAL recycled is a typed failure and escalates to rebuild
- catch-up exhaustion escalates to rebuild
- rebuild escalation is sticky until terminal
- session success is terminal close, not an intermediate ack
---
## 8. Failure Model
| Failure | Architectural behavior |
|---|---|
| Master process restart | Reload durable authority; volumes reconnect and replay observations; no new authority from heartbeat alone |
| Primary crash | Publisher mints new authority for eligible replica; old primary is stale-fenced |
| Supporting replica short disconnect | Engine-driven catch-up within retention |
| Supporting replica beyond retention | Rebuild escalation |
| Replica disk loss/corruption | Mark unsuitable; rebuild/recreate or explicit unsupported gate |
| Network partition | Old authority cannot count toward current writes or acks |
| Node drain | Controller removes node through desired topology generation, not manual assignment edits |
| Volume daemon restart | Reopens durable state, resubscribes, reconstructs runtime from assignment and probe facts |
P15 beta may choose single-master availability, but it must not blur "single master process" with "single primary per volume."
---
## 9. Operator Interface
The product should move toward this user model:
```text
seaweed block volume create --name db --size 100GiB --replicas 3 --class fast
seaweed block volume status db
seaweed block node drain nodeB
seaweed block replica replace db --from nodeB --to nodeD
seaweed block volume attach db --frontend nvme
```
The operator should see:
- desired topology generation
- current assignment line
- primary and supporting replicas
- peer set generation
- durability mode
- WAL pressure and checkpoint status
- replica lag
- recovery session state
- rebuild progress
- degraded reason
- unsupported/deferred feature explanations
If P15 cannot implement a surface, it must document the limitation rather than imply production support.
---
## 10. P15 Gate Alignment
| Gate family | Architecture responsibility |
|---|---|
| G0/G1 | Product processes host control plane and RPC without authority stuffing |
| G2/G3 | Frontends consume projection and enforce stale-path rejection |
| G4 | Durable local backend provides real persistence |
| G5 | Replicated write path and durability semantics |
| G6/G7/G12 | Catch-up, rebuild, and disk-failure handling |
| G8 | Failover with acknowledged-data continuity |
| G9/G9A | Lifecycle and placement intent become desired topology |
| G10/G11 | Snapshot and resize decisions align with product truth |
| G13 | Node lifecycle flows through placement/controller, not manual authority edits |
| G14/G15 | API/CSI expose safe intent, not authority internals |
| G16-G21 | Security, diagnostics, deployment, migration, deferred product policies, SLOs |
| G22 | End-to-end cluster evidence that the architecture holds |
`G9A` is the explicit placement-controller MVP decision point. It exists to prevent P15 lifecycle and CSI gates from overclaiming V2-like dynamic placement while topology is still manual.
---
## 11. P15 Non-Goals
Unless explicitly promoted by the gates, P15 does not claim:
- multi-master HA or distributed authority-store safety
- rack/AZ-aware placement
- automatic hot-volume rebalance
- advanced scheduler scoring
- ALUA/multipath parity beyond the accepted frontend gate
- snapshot-based long-lag catch-up
- wire protocol negotiation
- auth/encryption/mTLS beyond the accepted security gate
- multi-volume-per-binary if the product host remains single-volume in P15
Non-goals are acceptable only when documented with customer-visible implication.
---
## 12. How Mini-Plans Use This Document
Every P15 mini-plan should include an architecture reference in its scope section:
```text
Architecture touchpoints:
- v3-architecture.md §5.1 Create / Placement
- v3-architecture.md §6.1 Write Path
- v3-architecture.md §7 Recovery Architecture
```
The reviewer should ask:
1. Which architecture flow does this batch change?
2. Which truth domain owns the new fact?
3. Does this introduce a second authority path?
4. Does it preserve the product/control/data-plane split?
5. Which gate and invariant rows will prove the claim?
If the answer is unclear, update this document or the gate before coding.
---
## 13. Product Completion Ladder
Each close should state which product level it reaches. The level is not a precise percentage, but it prevents component completion from being reported as product completion.
| Level | Meaning | Typical proof |
|---|---|---|
| L0 Internal component | Logic works inside a package or fixture | Unit/component tests |
| L1 Product binary composition | Real product binary composes the needed components | Subprocess/L2 binary test |
| L2 Single-node usable IO | Real frontend IO works against durable local storage | OS or protocol harness attach/write/read |
| L3 Replicated IO | Real frontend writes move through replication to another node | Cross-node byte-equal test |
| L4 Failure/recovery under IO | Disconnect, crash, restart, catch-up/rebuild work under real IO | m01/L3 fault scenario |
| L5 Lifecycle/operator path | User/API/CSI can create, attach, drain, replace, delete without manual authority edits | API/CSI/admin scenario |
| L6 Beta-scope product | Accepted P15 gates pass or have explicit product-owner deferrals | G22 evidence bundle |
Example: G5-4 reached L1 for replicated write path. It did not reach L3, because byte movement under real frontend IO moved to G5-5.
---
## 14. Known Architecture Decisions Still Open
| Decision | Default for now | Bind point |
|---|---|---|
| `G5-DECISION-001`: primary restart recovery state | Path B runtime, Path A serializable seam open | G5-6 |
| `G5-DECISION-002`: walstore checkpoint cadence | Verify and document tuning policy | G5-2 / G5 close |
| Placement controller MVP | Add `G9A`; flat topology MVP recommended in P15 | G9/G13/G15 planning |
| Master HA | Single-node beta unless A1 changes | Post-P15 or explicit scope change |
| Rack-aware placement | Deferred; flat topology only | G20 or P16 |
| Real handler-context lane signal | Deferred hardening | Post-G5 |
---
## 15. Change Discipline
Update this document when:
- a gate adds/removes a component responsibility
- a batch changes a control-plane, data-plane, or recovery flow
- a new product operation is accepted
- a non-goal is promoted into P15 scope
- an invariant reveals that the architecture map is incomplete
Do not update this document for local implementation details that do not change system shape.
sw-block/design/v3-batch-process.md
+40 -1
View File
@@ -115,6 +115,7 @@ These are first-order references; keep updated as batches close:
| Doc | Owner | Update trigger |
|---|---|---|
| [`v3-dev-roadmap.md`](./v3-dev-roadmap.md) | QA | every gate-close |
| [`v3-architecture.md`](./v3-architecture.md) | architect | component/responsibility/flow changes |
| [`v3-phase-15-mvp-scope-gates.md`](./v3-phase-15-mvp-scope-gates.md) | architect | scope changes (e.g. G9A added 2026-04-26) |
| [`v3-invariant-ledger.md`](./v3-invariant-ledger.md) | sw + QA | every batch-close (PR-atomic) |
| [`v3-block-behavior-contract-index.md`](./v3-block-behavior-contract-index.md) | architect | new behavior contract ratifications |
@@ -159,7 +160,45 @@ Drop ceremony. Keep discipline.
---
## §12 Who owns what step
## §12 Architect review checklist
Architect review is not plan authorship. It is the system-level check that prevents drift.
Every mini-plan ratification and §close sign must answer:
| Check | Question |
|---|---|
| Scope truth | What is done, what is explicitly not done, and what product risk remains? |
| V2 / new-build decision | Is this a full V2 port, partial V2 port, reference-only read, explicit reject, or V3-native design? Does it need a new gate? |
| Engine / adapter impact | Does this change authority, session fate, recovery, durability, projection, or truth ownership? If yes, where is the invariant and adversarial test? |
| Product usability level | What can an operator or user actually do end-to-end after this batch? Which test proves it? Which next gate makes it more usable? |
Close signs should include a short statement of:
```text
Done:
Not done:
Product level reached:
Next gate that makes it usable:
```
---
## §13 Failure modes this process must prevent
These are the observed sources of waste. Every batch should actively check them.
| Failure mode | Required countermeasure |
|---|---|
| V2 porting drift | Top-down mapping: V2 behavior → V3 product contract → truth owner → accepted/rejected semantics → invariant/test |
| Engine/adapter change without reason | Require reason, truth-domain impact, bad state prevented, and proof test |
| Function close means happy path only | Acceptance must include stale input, reordering/overlap where relevant, partial progress, retry/restart, and explicit non-claims |
| Product unusable despite green tests | Mini-plan must name the end-to-end operator/user path and whether it is real, simulated, manual, or deferred |
| Missing architecture component | If a batch discovers a product component late (for example placement or binary wiring), update `v3-architecture.md` or the gate before continuing |
---
## §14 Who owns what step
Based on what actually worked across T4 + G5: