seaweedfs/sw-block/design/testrunner-roadmap.md

sw-test-runner Roadmap

Date: 2026-04-11 Status: active

1. Current State

The sw-test-runner is a YAML-driven test platform for sw-block hardware validation. It deploys binaries, orchestrates multi-node scenarios, runs benchmarks, injects faults, and reports results.

Metric	Value
Registered actions	37
YAML scenarios (internal)	72
YAML scenarios (external)	75
Scenario categories	smoke, HA, rebuild, chaos, perf, soak, CSI
Hardware tested	m01/m02, 25Gbps RoCE

What it does well

• Deploy weed binaries to remote nodes via SSH
• Start/stop/kill weed processes (master, volume server)
• Create/delete block volumes via master gRPC API
• iSCSI and NVMe-oF connect/disconnect
• fio benchmark with JSON output + metric parsing
• dd write/read/verify with md5 checksum
• Fault injection: netem delay, iptables partition, disk fill, WAL corrupt
• Phase-based scenario sequencing with variable substitution
• Assertions: equal, greater, contains, block field checks
• Artifact collection on failure
• JUnit XML output for CI
• Parallel phase support
• Prometheus metrics scraping

What it cannot do yet

• Compare multiple binaries (V1/V2/V3) in one run
• Compare against other systems (Ceph RBD, DRBD)
• Track results over time (no persistent history)
• Detect performance regressions automatically
• Debug failures (auto-grep logs, stop-on-fail)
• Deploy clusters from scratch (relies on pre-installed weed)
• Reuse cluster setup across scenarios (boilerplate every time)

2. Proven Scenarios (V2 Baseline)

These 4 scenarios define the V2 acceptance bar. All PASS as of 2026-04-11.

Scenario	File	Actions	What it proves
I-V3 Auto-Failover	recovery-baseline-failover.yaml	43/43	Create→write→kill→promote→IO verified
I-R8 Rebuild-Rejoin	v2-rebuild-rejoin.yaml	58/58	Failover→write→restart→1GB rebuild→data verified
Fast Rejoin	v2-fast-rejoin-catchup.yaml	43/43	Kill replica→3s restart→recovery→data verified
RF=1 Perf Baseline	rf1-perf-compare.yaml	22/22	V1.5 vs V2 IOPS comparison (within 1.2%)

Additional proven scenarios:

Scenario	File	Actions	What it proves
Rebuild Retry	v2-rebuild-failure-retry.yaml	52/52	Kill during rebuild→restart→data verified
dm-stripe 2-server	dm-stripe-two-server.yaml	42/42	Linux striping across 2 BlockVols: 1.87x write IOPS

3. Roadmap

P0: Multi-Version Comparison Matrix

Goal: Run the same benchmark scenario against V1.5, V2, V3, and Ceph in one invocation. Produce a side-by-side comparison table.

Why P0: CEO needs head-to-head performance numbers for investor/customer presentations. Current workflow requires manual binary swaps and separate runs.

Design:

# New top-level field in scenario YAML:
matrix:
  binary:
    - name: v1.5
      weed: /opt/work/weed-v1.5
    - name: v2
      weed: /opt/work/weed
    - name: ceph
      type: ceph
      pool: bench-pool

# Runner executes the scenario once per matrix entry,
# collects metrics, produces comparison table.

Output:

=== perf-compare: 4K randwrite qd=32, RF=1, 15s ===

| Binary | Write IOPS | Write P99 | Read IOPS | Read P99 |
|--------|-----------|-----------|-----------|----------|
| v1.5   | 47,233    | 1,234 us  | 62,100    | 890 us   |
| v2     | 46,666    | 1,250 us  | 61,800    | 910 us   |
| ceph   | 4,533     | 14,483 us | 65,761    | 1,319 us |

New code:

Component	Description	Est. lines
Matrix parser	Parse matrix: from YAML, generate run combinations	80
Matrix executor	Run scenario N times, collect results per variant	120
Comparison reporter	Markdown/terminal table from collected metrics	80
Total		~280

Acceptance: sw-test-runner compare rf1-perf-compare.yaml produces the table above with real numbers from hardware.

---

P0: Ceph RBD Adapter

Goal: Run fio benchmarks against Ceph RBD on the same hardware, using the same scenario YAML. Enable head-to-head comparison.

Why P0: Validates the 13.5x write IOPS advantage claim. Without this, performance claims are from separate, non-reproducible runs.

Prerequisites: Ceph cluster deployed on m01/m02 (one monitor, two OSDs). This is a one-time manual setup.

New actions:

Action	Parameters	What it does
ceph_create_image	pool, name, size	rbd create --size {size} {pool}/{name}
ceph_map	pool, name, node, save_as	rbd map {pool}/{name} → returns /dev/rbdN
ceph_unmap	device, node	rbd unmap {device}
ceph_delete_image	pool, name	rbd rm {pool}/{name}

Est. lines: ~100 (4 actions, each ~25 lines wrapping CLI commands)

Acceptance: ceph_create_image + ceph_map + fio_json + ceph_unmap works in a scenario YAML.

---

P1: Result Index + History (text-db)

Goal: Index all run bundles in a lightweight text database. Support listing, searching, comparing, and cleaning up old runs.

Why P1: Runs accumulate as directories on the test node. Without an index, you can't find old results, compare trends, or clean up disk space.

What already exists (RunBundle system):

results/
  20260409-175214-d099/           ← one dir per run (timestamp-based)
    manifest.json                 ← run identity: scenario, git SHA, binary hash
    scenario.yaml                 ← frozen copy of input YAML
    result.json                   ← full structured result (phases, actions, vars)
    result.xml                    ← JUnit XML
    result.html                   ← HTML report
    artifacts/                    ← collected logs on failure
  20260410-012330-5d99/
    ...

Each run is self-contained. manifest.json has: run_id, scenario_name, scenario_sha256, git_sha, host, status, command_line, started_at, finished_at.

What's missing: an index across runs.

Design: Add a runs.db text file (one JSON line per run) that acts as a lightweight index. No external database dependency.

results/
  runs.db                         ← newline-delimited JSON (one line per run)
  20260409-175214-d099/
    manifest.json
    result.json
    ...

runs.db format (one JSON line per entry):

{"run_id":"20260409-175214-d099","scenario":"v2-rebuild-rejoin","status":"pass","actions":58,"passed":58,"failed":0,"duration_ms":181795,"iops_write":46666,"iops_read":0,"git_sha":"8ecc50645","binary_md5":"4bcf08","started_at":"2026-04-09T17:52:14Z","dir":"20260409-175214-d099","disk_mb":12}
{"run_id":"20260410-012330-5d99","scenario":"recovery-baseline-failover","status":"pass","actions":43,"passed":43,"failed":0,"duration_ms":111583,"iops_write":28733,"git_sha":"8ecc50645","binary_md5":"4bcf08","started_at":"2026-04-10T01:23:30Z","dir":"20260410-012330-5d99","disk_mb":8}

New CLI commands:

# List all runs, most recent first
sw-test-runner list
  RUN_ID                  SCENARIO                    STATUS  ACTIONS  IOPS_W   DURATION
  20260411-140236-8012    dm-stripe-two-server        pass    42/42    79001    1m10s
  20260410-012330-5d99    recovery-baseline-failover   pass    43/43    28733    1m52s
  20260409-175214-d099    v2-rebuild-rejoin           pass    58/58    —        3m02s

# Filter by scenario
sw-test-runner list --scenario rebuild
sw-test-runner list --status fail
sw-test-runner list --since 2026-04-10

# Show trend for a scenario
sw-test-runner trend rf1-perf-compare
  DATE        GIT_SHA     IOPS_WRITE  IOPS_READ   STATUS
  2026-04-10  8ecc506     47233       —           pass (v1.5)
  2026-04-10  8ecc506     46666       —           pass (v2)

# Compare two runs
sw-test-runner diff 20260410-run1 20260410-run2

# Clean up old runs (keep last N per scenario)
sw-test-runner gc --keep 10
  Deleted 23 runs, freed 1.2 GB

# Rebuild index from existing run directories
sw-test-runner reindex
  Scanned 45 run directories, indexed 45 entries

New code:

Component	Description	Est. lines
Index writer	Append one JSON line to runs.db after each run	50
Index reader	Parse runs.db, filter/sort/search	80
list command	Terminal table from index	60
trend command	Filter by scenario, show metrics over time	60
gc command	Delete old run dirs, update index	50
reindex command	Scan run dirs, rebuild runs.db from manifest+result	60
Disk size calculator	du -s each run dir, store in index	20
Total		~380

Regression detection:

sw-test-runner trend rf1-perf-compare --check
  Latest: 46,666 IOPS
  Baseline (last 10): 47,100 ± 800 IOPS
  Status: OK (within 1 stddev)

sw-test-runner trend rf1-perf-compare --check
  Latest: 38,000 IOPS
  Baseline (last 10): 47,100 ± 800 IOPS
  Status: REGRESSION (-19.3%, > 2 stddev)

Acceptance: After 5 runs, sw-test-runner list shows all 5 with metrics. sw-test-runner gc --keep 3 deletes the 2 oldest.

---

P1: Cluster Templates

Goal: Remove the 20-line cluster boilerplate from every scenario. Define reusable topology+cluster configs.

Why P1: Every new scenario copies the same cluster-start phase. Changes to cluster config (ports, dirs) require editing every scenario.

Design:

# templates/two-node-roce.yaml
cluster:
  master:
    node: m02
    port: 9433
    dir: /tmp/sw-master
    extra_args: "-ip=10.0.0.3"
  volumes:
    - node: m02
      port: 18480
      dir: /tmp/sw-vs1
      block_dir: /tmp/sw-vs1/blocks
      block_listen: ":3295"
      extra_args: "-ip=10.0.0.3"
    - node: m01
      port: 18480
      dir: /tmp/sw-vs2
      block_dir: /tmp/sw-vs2/blocks
      block_listen: ":3295"
      extra_args: "-ip=10.0.0.1"

# scenario references template:
name: rebuild-rejoin
cluster: two-node-roce       # ← replaces 20 lines of cluster-start
phases:
  - name: create-and-write
    ...

New code:

Component	Description	Est. lines
Template loader	Parse cluster template YAML	60
Auto cluster-start phase	Generate start actions from template	80
Auto cleanup phase	Generate stop actions from template	40
Variable injection	Expose {{ master_pid }}, {{ vs1_pid }}, etc.	30
Total		~210

Acceptance: v2-rebuild-rejoin.yaml works with cluster: two-node-roce instead of inline cluster-start phase.

---

P2: Debug Mode

Goal: When a scenario fails, automatically collect diagnostic information and optionally pause for manual investigation.

Why P2: This session's debugging workflow was: scenario fails → SSH to node → grep logs → find root cause → fix → redeploy → rerun. The runner should automate the grep+collect step.

New CLI flags:

sw-test-runner run scenario.yaml --debug          # auto-grep + stop on fail
sw-test-runner run scenario.yaml --stop-on-fail   # pause, don't cleanup
sw-test-runner run scenario.yaml --grep "pattern"  # grep all node logs

Auto-diagnostic on failure:

=== FAILURE DIAGNOSTIC ===
Phase: restart-old-primary
Action: wait_volume_healthy (timeout 60s)

--- m01 volume.log (last 20 lines matching "error|fail|warn|panic") ---
W0409 15:41:18 recovery: rebuild execution failed: sender not found
...

--- m02 volume.log (last 20 lines matching "error|fail|warn|panic") ---
(clean)

--- m02 master.log (last 10 lines matching "failover|promote|assign") ---
I0409 15:41:12 failover: promoted replica for "v2-rebuild"
...

New code:

Component	Description	Est. lines
--stop-on-fail handler	Skip cleanup, print SSH commands	30
Auto-grep on failure	SSH to all nodes, grep key patterns	80
Log merge by timestamp	Merge multi-node logs, sort by time	60
Diagnostic formatter	Terminal-friendly failure report	40
Total		~210

Acceptance: When v2-rebuild-rejoin.yaml fails, the runner automatically shows the "sender not found" error from the logs without manual SSH.

---

P2: RF=2 Performance Benchmark Suite

Goal: Standardized benchmark scenario that matches the V1 bench parameters exactly. Produces results comparable to the CEO's numbers.

Why P2: Our current V2 perf numbers are RF=1 only. The CEO's benchmark was RF=2 sync_all over NVMe/TCP. We need the same config.

Scenario:

name: rf2-perf-benchmark
cluster: two-node-roce

phases:
  - name: create-volume
    actions:
      - action: create_block_volume
        name: perf-vol
        size_bytes: "1073741824"
        replica_factor: "2"
        durability_mode: "sync_all"

  - name: benchmark
    actions:
      # Warmup
      - action: fio_json
        rw: randwrite
        bs: 4k
        iodepth: "32"
        runtime: "10"

      # 4K random write
      - action: fio_json
        rw: randwrite
        bs: 4k
        iodepth: "32"
        runtime: "15"
        save_as: write_result

      # 4K random read
      - action: fio_json
        rw: randread
        bs: 4k
        iodepth: "32"
        runtime: "15"
        save_as: read_result

      # 128K sequential write
      - action: fio_json
        rw: write
        bs: 128k
        iodepth: "16"
        runtime: "15"
        save_as: seq_write_result

      # Report
      - action: print
        msg: |
          RF=2 sync_all Performance:
          4K randwrite: {{ write_iops }} IOPS, P99={{ write_p99 }}us
          4K randread:  {{ read_iops }} IOPS, P99={{ read_p99 }}us
          128K seqwrite: {{ seq_bw }} MB/s

---

P3: Full Cluster Deployment

Goal: Deploy a complete sw-block cluster (master + N volume servers + CSI driver) from scratch on bare metal or K8s.

Why P3: Currently assumes pre-installed binaries and manual OS setup. For CI/CD and customer POC, need push-button deployment.

Scope:

Component	What it does
OS prereqs	Install iscsiadm, nvme-cli, fio, dmsetup
Binary deployment	Build + SCP weed binary to all nodes
Cluster bootstrap	Start master, wait ready, start volume servers
Volume provisioning	Create volumes, configure replication
Client setup	iSCSI/NVMe-oF discovery + login
Health check	Verify all components healthy

Effort: ~500 lines. Depends on target environment (bare metal vs K8s).

---

P3: pgbench / Application Benchmark

Goal: Run real database benchmarks (pgbench TPC-B, sysbench OLTP) on sw-block volumes to produce application-level metrics.

Why P3: fio shows raw block performance. Customers care about "how fast is PostgreSQL on your storage."

New actions:

Action	What it does
pgbench_init	pgbench -i -s {scale} -h {host} {db}
pgbench_run	pgbench -c {clients} -T {time} -h {host} {db} → parse TPS
start_postgres	Start PostgreSQL on a block PV
stop_postgres	Stop PostgreSQL cleanly

---

4. Implementation Priority

Priority	Feature	Lines	Impact
P0	Multi-version comparison matrix	280	CEO benchmark table
P0	Ceph RBD adapter	100	Head-to-head validation
P1	Structured results + trends	300	Regression detection
P1	Cluster templates	210	Scenario authoring speed
P2	Debug mode	210	Debugging speed
P2	RF=2 perf benchmark suite	1 scenario	V1 number comparison
P3	Full cluster deployment	500	CI/CD, customer POC
P3	pgbench / app benchmarks	200	Customer-facing metrics
Total		~1,800

5. Versioning Strategy

The testrunner should work across sw-block versions:

Version	Binary	Scenarios	Expected behavior
V1.5	weed-v1.5	rf1-perf-compare only	RF=1 perf baseline
V2	weed (current)	All 147 scenarios	Full coverage
V3	weed-v3 (future)	Same 4 acceptance scenarios	Must match V2 results
Ceph	rbd CLI	Perf scenarios via adapter	Comparison baseline

The matrix runner handles this — same scenario, different binary, automatic comparison.

6. Non-Goals

The testrunner should NOT become:

• A full CI/CD pipeline (use GitHub Actions / Jenkins for that)
• A monitoring system (use Prometheus/Grafana for ongoing monitoring)
• A configuration management tool (use Ansible/Terraform for cluster setup)
• A log aggregation system (use ELK/Loki for production log analysis)

It IS: a focused hardware validation and benchmark tool for sw-block development and customer demos.