seaweedfs/sw-block/design/v3-dev-process.md

V3 Development Process

Date: 2026-04-13 Status: design draft Purpose: define branch strategy, code review, CI/CD, release, and engineering management for sw-block development

1. Current SeaweedFS Upstream Process

1.1 What upstream does

Aspect	Implementation
Branch model	Master-only, feature branches merged via PR
Merge strategy	Squash-merge with PR number in commit message
CI	46 GitHub Actions workflows (build, test, Docker, CodeQL)
Test suite	663 _test.go files + 25 integration test workflows
Release	Tag-based: multi-platform binaries + Docker images
Code review	PR-based, CodeQL security scan, no mandatory reviewers
Linting	go vet only (no golangci-lint)
Documentation	Wiki-based, minimal CONTRIBUTING.md
Issue tracking	GitHub Issues + Discussions
Engineering mgmt	Informal, sponsor-driven

1.2 What upstream does well

• One-command build: make builds everything
• 46 CI workflows: comprehensive S3, FUSE, EC, backend integration tests
• Multi-platform releases: Linux/Windows/Darwin × amd64/arm64
• Docker multi-arch: amd64, arm64, arm, 386 with GHCR + Docker Hub
• Dev builds on every push: :dev tagged images auto-published
• Concurrency cancellation: cancel-in-progress: true saves CI minutes

1.3 What upstream is missing

• No golangci-lint or external linters
• No pre-commit hooks
• No formal CONTRIBUTING.md
• No branch protection rules
• No mandatory code review
• No test coverage tracking
• No public roadmap or project boards
• No release notes automation

2. sw-block Branch Strategy

2.1 Branch model

master (upstream SeaweedFS)
  │
  └── feature/sw-block (long-lived, periodic rebase on master)
        │
        ├── sw-block/v1-pre-phase13    (historical checkpoint)
        ├── sw-block/v1.5-ec6-fix      (historical checkpoint)
        ├── sw-block/v1.5-post-phase13 (V1.5 perf baseline)
        │
        └── (working tree — direct commits)

2.2 Rules

feature/sw-block is the main development branch:

• All sw-block work commits here
• Periodic rebase on master to stay current with upstream
• Never force-push without team notification
• Historical checkpoints preserved as named branches for perf comparison

master is upstream SeaweedFS:

• sw-block does not commit directly to master
• Upstream PRs go through the normal SeaweedFS PR process
• sw-block-specific code stays on feature/sw-block

When to create a checkpoint branch:

• Before a major architectural change (V1→V2, V2→V3)
• Before removing/replacing a subsystem
• When a perf baseline needs to be preserved for comparison

2.3 Commit conventions

# Feature
feat: add RemoteRebuildIO — primary coordinates, replica installs

# Bug fix
fix: suppress SessionFailed after successful rebuild completion

# Documentation
doc: external failure taxonomy — 20 real bugs from Ceph/DRBD/Longhorn

# Refactor
refactor: unified primary onboarding — assignment is the trigger

# Diagnostic (temporary, remove before release)
diag: add sender registry verification after installSession

# Test
test: 16/16 SmartWAL crash tests PASS

# Performance
perf: RF=1 baseline — V1.5=47233 vs V2=46666 IOPS (within noise)

Format: type: short description (imperative mood)

For multi-line commits:

feat: RemoteRebuildIO — primary coordinates rebuild, replica installs

- New file: block_rebuild_remote.go
- SessionControlMsg extended with RebuildAddr trailer (v2 wire format)
- replica_barrier.go: runBaseLaneClient auto-connects to rebuild server
- Shipper state: NeedsRebuild → Rebuilding → InSync on ack sequence

3. Code Review Process

3.1 Review requirements

Change type	Required review	Who reviews
Engine semantic change	Architect	Must review before merge
Runtime/adapter change	sw peer or architect	At least one review
Test-only change	Self-merge OK	Optional review
Design doc	Architect	Review before merge
Scenario YAML	Self-merge OK	Run on hardware first
Diagnostic/temp code	Self-merge OK	Must be removed before release

3.2 Review checklist

For every code change, the reviewer checks:

□ Does this change match a V3 anti-pattern? (check protocol-anti-patterns.md)
□ Does it introduce a new state to the engine? (must justify per v3-clean-recovery-draft.md)
□ Does it add timer-dependent semantics? (reject — P1: timers trigger, facts decide)
□ Does it change authority for a truth domain? (must be explicit and documented)
□ Are error messages actionable? (say what happened + why + what to do)
□ Is there a test? (unit test for logic, hardware scenario for integration)
□ Does it pass go vet? (no new warnings)
□ Does it compile for linux/amd64? (GOOS=linux GOARCH=amd64 go build)

3.3 Self-review for solo work

When working alone (common with AI agents), apply the same checklist before committing. The commit message should document the review:

fix: WAL retention exhaustion — remove keepup retention floor

Review: checked against A2 (transport failure → recovery type).
The retention removal means every disconnect triggers rebuild,
not catchup. This is correct for current architecture (no LBA
dirty map). Future SmartWAL + dirty map will eliminate this.

Tested: I-V3 43/43, I-R8 58/58, fast-rejoin 43/43 on hardware.

4. Testing Standards

4.1 Test levels

Level	What	Where	When
Unit	Pure logic, no I/O	*_test.go next to source	Every commit
Unit-TCP	Localhost TCP (shipper, iSCSI, NVMe)	*_test.go with network	Every commit
Component	Real weed processes on localhost	test/component/	Before merge
Integration	Real hardware, SSH, kernel iSCSI	testrunner/scenarios/	Before release
Crash	Kill -9 + recovery verification	SmartWAL crash tests	Before release

4.2 Test counts (current)

Domain	Count
Engine (BlockVol core)	18
WAL	81
Replication	73
Fencing	7
Operations (snapshot, expand, scrub)	143
iSCSI	19
NVMe	252
Control plane (registry, failover, heartbeat)	683
Test runner	179
SmartWAL prototype	16
Total	~1,600

Hardware scenarios: 147 YAML files, 6 verified on m01/m02 (196 actions).

4.3 Required tests for each change type

Change type	Required test
Engine state machine	Unit test with assertion on every state transition
WAL/extent I/O	Crash test (write → kill → recover → verify)
Replication protocol	Two-node unit test (primary + replica)
Server layer (weed integration)	Component test or hardware scenario
Performance-sensitive	Perf comparison (before/after IOPS)
Bug fix	Regression test that reproduces the bug

4.4 Hardware acceptance gate

Before any release, these 4 scenarios must PASS:

1. I-V3 auto-failover            43/43 actions
2. I-R8 rebuild-rejoin           58/58 actions
3. Fast rejoin                   43/43 actions
4. RF=1 perf baseline            22/22 actions (within 5% of V1.5)

Total: 166 actions, 0 failures allowed.

5. CI/CD Pipeline

5.1 Current state

sw-block runs on the upstream SeaweedFS CI. The go.yml workflow builds and tests on every push. Block-specific tests run as part of the full go test ./... suite.

5.2 Desired pipeline

On every push to feature/sw-block:
  1. go vet ./...
  2. go build ./weed                    (linux/amd64)
  3. go test ./weed/storage/blockvol/... (unit + unit-TCP)
  4. go test ./weed/server/ -run "Block" (server-layer block tests)
  5. go test ./sw-block/...             (protocol engine tests)

On PR to feature/sw-block:
  6. All of above +
  7. go build linux/arm64               (cross-compile check)
  8. golangci-lint run                  (if added)

On tag (release):
  9. All of above +
  10. Build linux/amd64 binary
  11. Build sw-test-runner binary
  12. Deploy to hardware, run 4 acceptance scenarios
  13. Build Docker image
  14. Publish release artifacts

5.3 CI configuration

# .github/workflows/sw-block.yml
name: sw-block
on:
  push:
    branches: [feature/sw-block]
  pull_request:
    branches: [feature/sw-block]

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version: '1.23'
      - run: go vet ./...
      - run: GOOS=linux GOARCH=amd64 go build ./weed
      - run: go test ./weed/storage/blockvol/... -count=1 -timeout 5m
      - run: go test ./weed/server/ -run "Block|TestP16" -count=1 -timeout 5m
      - run: go test ./sw-block/... -count=1 -timeout 5m

5.4 Hardware CI (future)

When hardware is available for CI:

  hardware-acceptance:
    runs-on: self-hosted  # m01/m02 runner
    needs: build-and-test
    steps:
      - run: sw-test-runner run recovery-baseline-failover.yaml
      - run: sw-test-runner run v2-rebuild-rejoin.yaml
      - run: sw-test-runner run v2-fast-rejoin-catchup.yaml
      - run: sw-test-runner run rf1-perf-compare.yaml

6. Release Process

6.1 Versioning

sw-block follows SeaweedFS upstream versioning with a block suffix:

SeaweedFS 4.19 + sw-block Phase 20 = sw-block 4.19-block.20

For standalone releases (before upstream merge):

sw-block-v2.0  (V2 stable baseline, 2026-04-11)
sw-block-v3.0  (V3 with SmartWAL, future)

6.2 Release checklist

□ All 4 hardware acceptance scenarios PASS (166 actions)
□ RF=1 perf within 5% of baseline
□ No known data-loss bugs
□ All diagnostic/temp code removed
□ Commit history clean (no WIP commits)
□ Design docs up to date
□ Release notes written
□ Binary built for linux/amd64
□ sw-test-runner built for linux/amd64
□ Docker image built and tagged

6.3 Release artifacts

sw-block-v2.0/
  weed-linux-amd64              ← main binary
  sw-test-runner-linux-amd64    ← test runner
  scenarios/                    ← YAML test scenarios
  dashboards/
    block-overview.json         ← Grafana dashboard
  alerts/
    sw-block-alerts.yml         ← Prometheus alert rules
  RELEASE-NOTES.md

7. Issue / PR Templates

7.1 Bug report template

## Bug Report

**Component**: [engine / replication / iSCSI / NVMe / master / CSI]
**Version**: [git SHA or release tag]
**Hardware**: [m01/m02, or describe]

### What happened
[Clear description]

### Expected behavior
[What should have happened]

### Steps to reproduce
1. ...
2. ...

### Logs
[Relevant log lines with timestamps]

### Scenario (if applicable)
[YAML scenario file or inline]

### Anti-pattern check
Does this match a known anti-pattern from protocol-anti-patterns.md?
- [ ] A1: Timer defines semantics
- [ ] A2: Transport failure → recovery type
- [ ] A3: Ack = terminal success
- [ ] A4: Ordering determines result
- [ ] A5: Projection as control
- [ ] A6: Workaround state
- [ ] A7: Transport in engine
- [ ] None / New class

7.2 PR template

## What this PR does

[Short description]

## Why

[Problem being solved]

## How it was tested

- [ ] Unit tests pass: `go test ./weed/storage/blockvol/...`
- [ ] Server tests pass: `go test ./weed/server/ -run "Block"`
- [ ] Hardware scenario: [which scenario, result]
- [ ] Performance check: [IOPS before/after, or N/A]

## Review checklist

- [ ] No new anti-patterns (checked against protocol-anti-patterns.md)
- [ ] No new engine state without justification
- [ ] No timer-dependent semantics
- [ ] Error messages are actionable
- [ ] go vet clean
- [ ] Compiles for linux/amd64

## Files changed

| File | Change |
|------|--------|
| ... | ... |

7.3 Design doc template

# [Title]

Date: YYYY-MM-DD
Status: [draft / active / superseded]
Supersedes: [previous doc, if any]

## Problem
[What problem does this solve]

## Design
[The solution]

## Anti-pattern check
[Which anti-patterns does this avoid, and how]

## Test plan
[How to verify this works]

## Alternatives considered
[What else was considered and why it was rejected]

8. Engineering Management

8.1 Work tracking

What	Where
Active bugs	GitHub Issues on feature/sw-block
Design decisions	sw-block/design/*.md (append-only decision log)
Phase progress	sw-block/design/v2-acceptance-evidence.md
Test results	sw-test-runner results/ (runs.db index)
Performance history	sw-test-runner trend command

8.2 Decision log

Every significant decision is recorded:

## Decision: Remove WAL retention for keepup
Date: 2026-04-08
Context: WAL fills during sustained writes, causing 30s dd_write timeout
Decision: Set RetentionFloorFn to nil. Flusher recycles freely.
Consequence: Every replica disconnect triggers rebuild, not catchup.
Future: SmartWAL + LBA dirty map will eliminate this tension.
Tested: I-V3 43/43, dd_write 132ms (was 30s)

8.3 Phase development model

Each development phase follows:

1. Define scope and acceptance criteria
2. Write/update design doc
3. Implement
4. Test on hardware (4 acceptance scenarios)
5. Record results in evidence doc
6. Checkpoint branch if needed
7. Move to next phase

8.4 Agent collaboration model

For AI agent (sw/tester) driven development:

Role	Responsibility
Architect (human)	Direction, design review, acceptance
sw agent	Implementation, unit tests, component tests
tester agent	Hardware scenarios, evidence collection
manager (Claude)	Progress tracking, research, documentation

Rules:

• Agents must not silently redefine semantic authority
• Every agent commit includes what was tested
• Agent code is reviewed against the anti-pattern checklist
• Hardware test results are the acceptance gate, not unit tests

9. Code Quality Tools

9.1 Current

Tool	Status
go vet	Active (CI)
go build cross-compile	Active (CI)
-race flag	Not on Windows (CGO required)
golangci-lint	Not configured
Pre-commit hooks	None

9.2 Recommended additions

Tool	Purpose	Priority
golangci-lint	Catch common bugs, style	P1
go test -race on Linux CI	Race condition detection	P1
Pre-commit: go vet + go build	Fast local check	P2
Test coverage tracking	Identify untested code	P3

9.3 golangci-lint config (recommended)

# .golangci.yml
linters:
  enable:
    - govet
    - errcheck
    - staticcheck
    - unused
    - gosimple
    - ineffassign
  disable:
    - exhaustive     # too noisy for proto enums
    - wrapcheck      # too strict for internal code

issues:
  exclude-rules:
    - path: _test\.go
      linters: [errcheck]  # test files don't need error checking
    - path: \.pb\.go
      linters: [all]       # generated protobuf code

10. Upstream Contribution Path

When sw-block features are ready for upstream SeaweedFS:

10.1 What goes upstream

Component	Upstream candidate?	When
BlockVol engine	Yes	After V3 stable
iSCSI target	Yes	After V3 stable
NVMe-oF target	Yes	After V3 stable
CSI driver	Yes	After K8s validation
SmartWAL	Maybe (if generalized)	After prototype proven
Protocol engine (sw-block/)	No (sw-block specific)	Never
Test runner	Yes (general purpose)	After P1 features
Design docs	No (internal)	Never

10.2 Upstream PR process

1. Cherry-pick from feature/sw-block to a new branch
2. Ensure no sw-block-specific dependencies
3. Add tests that work without hardware
4. Follow upstream commit conventions
5. Submit PR to master
6. Address upstream review feedback

10.3 Upstream improvements to contribute back

From this session's audit (protocol-anti-patterns.md):

Fix	File	Impact
Per-replica failure in replication	store_replicate.go	P0: one failure shouldn't fail all
Epoch-based connection identity	data_node.go	P1: replace Counter workaround
Fix Stats threshold inversion	volume_location_list.go	P2: reports stale data