From 676539d3b968846aeb4a8e4087ef275ef45af046 Mon Sep 17 00:00:00 2001 From: pingqiu Date: Sun, 12 Apr 2026 21:57:03 -0700 Subject: [PATCH] doc: testrunner roadmap + dm-stripe scenario (42/42 PASS, 1.87x write IOPS) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit testrunner-roadmap.md: P0-P3 feature plan for multi-version comparison, Ceph adapter, result tracking, cluster templates, debug mode. dm-stripe-two-server.yaml: proven Linux dm-stripe across 2 sw-block volumes on 2 servers. Results: single=42K IOPS → striped=79K IOPS (1.87x). Data integrity verified via md5. Zero sw-block code changes needed. Co-Authored-By: Claude Opus 4.6 (1M context) --- sw-block/design/testrunner-roadmap.md | 448 ++++++++++++++++++ .../internal/dm-stripe-two-server.yaml | 285 +++++++++++ 2 files changed, 733 insertions(+) create mode 100644 sw-block/design/testrunner-roadmap.md create mode 100644 weed/storage/blockvol/testrunner/scenarios/internal/dm-stripe-two-server.yaml diff --git a/sw-block/design/testrunner-roadmap.md b/sw-block/design/testrunner-roadmap.md new file mode 100644 index 000000000..f21055613 --- /dev/null +++ b/sw-block/design/testrunner-roadmap.md @@ -0,0 +1,448 @@ +# sw-test-runner Roadmap + +Date: 2026-04-11 +Status: active + +## 1. Current State + +The sw-test-runner is a YAML-driven test platform for sw-block hardware +validation. It deploys binaries, orchestrates multi-node scenarios, runs +benchmarks, injects faults, and reports results. + +| Metric | Value | +|--------|-------| +| Registered actions | 37 | +| YAML scenarios (internal) | 72 | +| YAML scenarios (external) | 75 | +| Scenario categories | smoke, HA, rebuild, chaos, perf, soak, CSI | +| Hardware tested | m01/m02, 25Gbps RoCE | + +### What it does well + +- Deploy weed binaries to remote nodes via SSH +- Start/stop/kill weed processes (master, volume server) +- Create/delete block volumes via master gRPC API +- iSCSI and NVMe-oF connect/disconnect +- fio benchmark with JSON output + metric parsing +- dd write/read/verify with md5 checksum +- Fault injection: netem delay, iptables partition, disk fill, WAL corrupt +- Phase-based scenario sequencing with variable substitution +- Assertions: equal, greater, contains, block field checks +- Artifact collection on failure +- JUnit XML output for CI +- Parallel phase support +- Prometheus metrics scraping + +### What it cannot do yet + +- Compare multiple binaries (V1/V2/V3) in one run +- Compare against other systems (Ceph RBD, DRBD) +- Track results over time (no persistent history) +- Detect performance regressions automatically +- Debug failures (auto-grep logs, stop-on-fail) +- Deploy clusters from scratch (relies on pre-installed weed) +- Reuse cluster setup across scenarios (boilerplate every time) + +## 2. Proven Scenarios (V2 Baseline) + +These 4 scenarios define the V2 acceptance bar. All PASS as of 2026-04-11. + +| Scenario | File | Actions | What it proves | +|----------|------|---------|----------------| +| I-V3 Auto-Failover | `recovery-baseline-failover.yaml` | 43/43 | Create→write→kill→promote→IO verified | +| I-R8 Rebuild-Rejoin | `v2-rebuild-rejoin.yaml` | 58/58 | Failover→write→restart→1GB rebuild→data verified | +| Fast Rejoin | `v2-fast-rejoin-catchup.yaml` | 43/43 | Kill replica→3s restart→recovery→data verified | +| RF=1 Perf Baseline | `rf1-perf-compare.yaml` | 22/22 | V1.5 vs V2 IOPS comparison (within 1.2%) | + +Additional proven scenarios: + +| Scenario | File | Actions | What it proves | +|----------|------|---------|----------------| +| Rebuild Retry | `v2-rebuild-failure-retry.yaml` | 52/52 | Kill during rebuild→restart→data verified | +| dm-stripe 2-server | `dm-stripe-two-server.yaml` | 42/42 | Linux striping across 2 BlockVols: 1.87x write IOPS | + +## 3. Roadmap + +### P0: Multi-Version Comparison Matrix + +**Goal**: Run the same benchmark scenario against V1.5, V2, V3, and Ceph +in one invocation. Produce a side-by-side comparison table. + +**Why P0**: CEO needs head-to-head performance numbers for investor/customer +presentations. Current workflow requires manual binary swaps and separate runs. + +**Design**: + +```yaml +# New top-level field in scenario YAML: +matrix: + binary: + - name: v1.5 + weed: /opt/work/weed-v1.5 + - name: v2 + weed: /opt/work/weed + - name: ceph + type: ceph + pool: bench-pool + +# Runner executes the scenario once per matrix entry, +# collects metrics, produces comparison table. +``` + +**Output**: + +``` +=== perf-compare: 4K randwrite qd=32, RF=1, 15s === + +| Binary | Write IOPS | Write P99 | Read IOPS | Read P99 | +|--------|-----------|-----------|-----------|----------| +| v1.5 | 47,233 | 1,234 us | 62,100 | 890 us | +| v2 | 46,666 | 1,250 us | 61,800 | 910 us | +| ceph | 4,533 | 14,483 us | 65,761 | 1,319 us | +``` + +**New code**: + +| Component | Description | Est. lines | +|-----------|-------------|-----------| +| Matrix parser | Parse `matrix:` from YAML, generate run combinations | 80 | +| Matrix executor | Run scenario N times, collect results per variant | 120 | +| Comparison reporter | Markdown/terminal table from collected metrics | 80 | +| Total | | ~280 | + +**Acceptance**: `sw-test-runner compare rf1-perf-compare.yaml` produces +the table above with real numbers from hardware. + +--- + +### P0: Ceph RBD Adapter + +**Goal**: Run fio benchmarks against Ceph RBD on the same hardware, using +the same scenario YAML. Enable head-to-head comparison. + +**Why P0**: Validates the 13.5x write IOPS advantage claim. Without this, +performance claims are from separate, non-reproducible runs. + +**Prerequisites**: Ceph cluster deployed on m01/m02 (one monitor, two OSDs). +This is a one-time manual setup. + +**New actions**: + +| Action | Parameters | What it does | +|--------|-----------|--------------| +| `ceph_create_image` | pool, name, size | `rbd create --size {size} {pool}/{name}` | +| `ceph_map` | pool, name, node, save_as | `rbd map {pool}/{name}` → returns /dev/rbdN | +| `ceph_unmap` | device, node | `rbd unmap {device}` | +| `ceph_delete_image` | pool, name | `rbd rm {pool}/{name}` | + +**Est. lines**: ~100 (4 actions, each ~25 lines wrapping CLI commands) + +**Acceptance**: `ceph_create_image` + `ceph_map` + `fio_json` + `ceph_unmap` +works in a scenario YAML. + +--- + +### P1: Structured Results + Trend Tracking + +**Goal**: Persist results in a queryable format. Detect performance +regressions automatically. + +**Why P1**: Without history, we can't tell if a code change degraded +performance. Each run is ephemeral. + +**Design**: + +``` +results/ + index.json ← append-only run index + runs/ + 20260411-140236/ + meta.json ← scenario, binary, topology, timestamp + metrics.json ← {iops_write: 46666, iops_read: 61800, ...} + phases.json ← per-phase timing and pass/fail + artifacts/ ← logs, core dumps on failure + baselines/ + rf1-perf.json ← rolling baseline: mean, stddev, threshold +``` + +**New capabilities**: + +| Feature | Description | Est. lines | +|---------|-------------|-----------| +| `metrics.json` writer | Extract metrics from fio/dd results, write structured JSON | 80 | +| `index.json` updater | Append each run to index with scenario/binary/timestamp | 40 | +| Baseline comparator | Compare latest metrics against baseline, flag regressions | 100 | +| `sw-test-runner trend` | CLI command to show metric trend over last N runs | 80 | +| Total | | ~300 | + +**Regression detection rule**: + +``` +if latest.iops < baseline.mean - 2 * baseline.stddev: + WARN: performance regression detected + latest: 38,000 IOPS + baseline: 46,666 ± 1,200 IOPS (last 10 runs) +``` + +**Acceptance**: After 5 runs of `rf1-perf-compare.yaml`, `sw-test-runner trend` +shows IOPS over time and would flag a >5% regression. + +--- + +### P1: Cluster Templates + +**Goal**: Remove the 20-line cluster boilerplate from every scenario. +Define reusable topology+cluster configs. + +**Why P1**: Every new scenario copies the same `cluster-start` phase. +Changes to cluster config (ports, dirs) require editing every scenario. + +**Design**: + +```yaml +# templates/two-node-roce.yaml +cluster: + master: + node: m02 + port: 9433 + dir: /tmp/sw-master + extra_args: "-ip=10.0.0.3" + volumes: + - node: m02 + port: 18480 + dir: /tmp/sw-vs1 + block_dir: /tmp/sw-vs1/blocks + block_listen: ":3295" + extra_args: "-ip=10.0.0.3" + - node: m01 + port: 18480 + dir: /tmp/sw-vs2 + block_dir: /tmp/sw-vs2/blocks + block_listen: ":3295" + extra_args: "-ip=10.0.0.1" +``` + +```yaml +# scenario references template: +name: rebuild-rejoin +cluster: two-node-roce # ← replaces 20 lines of cluster-start +phases: + - name: create-and-write + ... +``` + +**New code**: + +| Component | Description | Est. lines | +|-----------|-------------|-----------| +| Template loader | Parse cluster template YAML | 60 | +| Auto cluster-start phase | Generate start actions from template | 80 | +| Auto cleanup phase | Generate stop actions from template | 40 | +| Variable injection | Expose `{{ master_pid }}`, `{{ vs1_pid }}`, etc. | 30 | +| Total | | ~210 | + +**Acceptance**: `v2-rebuild-rejoin.yaml` works with `cluster: two-node-roce` +instead of inline cluster-start phase. + +--- + +### P2: Debug Mode + +**Goal**: When a scenario fails, automatically collect diagnostic +information and optionally pause for manual investigation. + +**Why P2**: This session's debugging workflow was: scenario fails → SSH +to node → grep logs → find root cause → fix → redeploy → rerun. The +runner should automate the grep+collect step. + +**New CLI flags**: + +```bash +sw-test-runner run scenario.yaml --debug # auto-grep + stop on fail +sw-test-runner run scenario.yaml --stop-on-fail # pause, don't cleanup +sw-test-runner run scenario.yaml --grep "pattern" # grep all node logs +``` + +**Auto-diagnostic on failure**: + +``` +=== FAILURE DIAGNOSTIC === +Phase: restart-old-primary +Action: wait_volume_healthy (timeout 60s) + +--- m01 volume.log (last 20 lines matching "error|fail|warn|panic") --- +W0409 15:41:18 recovery: rebuild execution failed: sender not found +... + +--- m02 volume.log (last 20 lines matching "error|fail|warn|panic") --- +(clean) + +--- m02 master.log (last 10 lines matching "failover|promote|assign") --- +I0409 15:41:12 failover: promoted replica for "v2-rebuild" +... +``` + +**New code**: + +| Component | Description | Est. lines | +|-----------|-------------|-----------| +| `--stop-on-fail` handler | Skip cleanup, print SSH commands | 30 | +| Auto-grep on failure | SSH to all nodes, grep key patterns | 80 | +| Log merge by timestamp | Merge multi-node logs, sort by time | 60 | +| Diagnostic formatter | Terminal-friendly failure report | 40 | +| Total | | ~210 | + +**Acceptance**: When `v2-rebuild-rejoin.yaml` fails, the runner +automatically shows the "sender not found" error from the logs +without manual SSH. + +--- + +### P2: RF=2 Performance Benchmark Suite + +**Goal**: Standardized benchmark scenario that matches the V1 bench +parameters exactly. Produces results comparable to the CEO's numbers. + +**Why P2**: Our current V2 perf numbers are RF=1 only. The CEO's +benchmark was RF=2 sync_all over NVMe/TCP. We need the same config. + +**Scenario**: + +```yaml +name: rf2-perf-benchmark +cluster: two-node-roce + +phases: + - name: create-volume + actions: + - action: create_block_volume + name: perf-vol + size_bytes: "1073741824" + replica_factor: "2" + durability_mode: "sync_all" + + - name: benchmark + actions: + # Warmup + - action: fio_json + rw: randwrite + bs: 4k + iodepth: "32" + runtime: "10" + + # 4K random write + - action: fio_json + rw: randwrite + bs: 4k + iodepth: "32" + runtime: "15" + save_as: write_result + + # 4K random read + - action: fio_json + rw: randread + bs: 4k + iodepth: "32" + runtime: "15" + save_as: read_result + + # 128K sequential write + - action: fio_json + rw: write + bs: 128k + iodepth: "16" + runtime: "15" + save_as: seq_write_result + + # Report + - action: print + msg: | + RF=2 sync_all Performance: + 4K randwrite: {{ write_iops }} IOPS, P99={{ write_p99 }}us + 4K randread: {{ read_iops }} IOPS, P99={{ read_p99 }}us + 128K seqwrite: {{ seq_bw }} MB/s +``` + +--- + +### P3: Full Cluster Deployment + +**Goal**: Deploy a complete sw-block cluster (master + N volume servers + +CSI driver) from scratch on bare metal or K8s. + +**Why P3**: Currently assumes pre-installed binaries and manual OS setup. +For CI/CD and customer POC, need push-button deployment. + +**Scope**: + +| Component | What it does | +|-----------|-------------| +| OS prereqs | Install iscsiadm, nvme-cli, fio, dmsetup | +| Binary deployment | Build + SCP weed binary to all nodes | +| Cluster bootstrap | Start master, wait ready, start volume servers | +| Volume provisioning | Create volumes, configure replication | +| Client setup | iSCSI/NVMe-oF discovery + login | +| Health check | Verify all components healthy | + +**Effort**: ~500 lines. Depends on target environment (bare metal vs K8s). + +--- + +### P3: pgbench / Application Benchmark + +**Goal**: Run real database benchmarks (pgbench TPC-B, sysbench OLTP) +on sw-block volumes to produce application-level metrics. + +**Why P3**: fio shows raw block performance. Customers care about +"how fast is PostgreSQL on your storage." + +**New actions**: + +| Action | What it does | +|--------|-------------| +| `pgbench_init` | `pgbench -i -s {scale} -h {host} {db}` | +| `pgbench_run` | `pgbench -c {clients} -T {time} -h {host} {db}` → parse TPS | +| `start_postgres` | Start PostgreSQL on a block PV | +| `stop_postgres` | Stop PostgreSQL cleanly | + +--- + +## 4. Implementation Priority + +| Priority | Feature | Lines | Impact | +|----------|---------|-------|--------| +| **P0** | Multi-version comparison matrix | 280 | CEO benchmark table | +| **P0** | Ceph RBD adapter | 100 | Head-to-head validation | +| **P1** | Structured results + trends | 300 | Regression detection | +| **P1** | Cluster templates | 210 | Scenario authoring speed | +| **P2** | Debug mode | 210 | Debugging speed | +| **P2** | RF=2 perf benchmark suite | 1 scenario | V1 number comparison | +| **P3** | Full cluster deployment | 500 | CI/CD, customer POC | +| **P3** | pgbench / app benchmarks | 200 | Customer-facing metrics | +| **Total** | | ~1,800 | | + +## 5. Versioning Strategy + +The testrunner should work across sw-block versions: + +| Version | Binary | Scenarios | Expected behavior | +|---------|--------|-----------|-------------------| +| V1.5 | `weed-v1.5` | `rf1-perf-compare` only | RF=1 perf baseline | +| V2 | `weed` (current) | All 147 scenarios | Full coverage | +| V3 | `weed-v3` (future) | Same 4 acceptance scenarios | Must match V2 results | +| Ceph | `rbd` CLI | Perf scenarios via adapter | Comparison baseline | + +The matrix runner handles this — same scenario, different binary, automatic +comparison. + +## 6. Non-Goals + +The testrunner should NOT become: + +- A full CI/CD pipeline (use GitHub Actions / Jenkins for that) +- A monitoring system (use Prometheus/Grafana for ongoing monitoring) +- A configuration management tool (use Ansible/Terraform for cluster setup) +- A log aggregation system (use ELK/Loki for production log analysis) + +It IS: a focused hardware validation and benchmark tool for sw-block +development and customer demos. diff --git a/weed/storage/blockvol/testrunner/scenarios/internal/dm-stripe-two-server.yaml b/weed/storage/blockvol/testrunner/scenarios/internal/dm-stripe-two-server.yaml new file mode 100644 index 000000000..3be7a73f6 --- /dev/null +++ b/weed/storage/blockvol/testrunner/scenarios/internal/dm-stripe-two-server.yaml @@ -0,0 +1,285 @@ +name: dm-stripe-two-server +timeout: 5m + +# Test: dm-stripe across 2 BlockVols on 2 servers. +# Proves Linux-layer striping works with sw-block for high throughput. +# +# Topology: +# m01: client + VS2 (BlockVol B) +# m02: master + VS1 (BlockVol A) +# Client connects iSCSI to both, dm-stripe combines them. + +env: + master_url: "http://10.0.0.3:9433" + vol_size: "1073741824" + +topology: + nodes: + m01: + host: 192.168.1.181 + alt_ips: ["10.0.0.1"] + user: testdev + key: "/opt/work/testdev_key" + m02: + host: 192.168.1.184 + alt_ips: ["10.0.0.3"] + user: testdev + key: "/opt/work/testdev_key" + +phases: + - name: cluster-start + actions: + - action: exec + node: m02 + cmd: "fuser -k 9433/tcp 18480/tcp 2>/dev/null; sleep 1; rm -rf /tmp/sw-stripe-master /tmp/sw-stripe-vs1 && mkdir -p /tmp/sw-stripe-master /tmp/sw-stripe-vs1/blocks" + root: "true" + ignore_error: true + - action: exec + node: m01 + cmd: "fuser -k 18480/tcp 2>/dev/null; sleep 1; rm -rf /tmp/sw-stripe-vs2 && mkdir -p /tmp/sw-stripe-vs2/blocks" + root: "true" + ignore_error: true + + - action: start_weed_master + node: m02 + port: "9433" + dir: /tmp/sw-stripe-master + extra_args: "-ip=10.0.0.3" + save_as: master_pid + + - action: sleep + duration: 3s + + - action: start_weed_volume + node: m02 + port: "18480" + master: "10.0.0.3:9433" + dir: /tmp/sw-stripe-vs1 + extra_args: "-block.dir=/tmp/sw-stripe-vs1/blocks -block.listen=:3295 -ip=10.0.0.3" + save_as: vs1_pid + + - action: start_weed_volume + node: m01 + port: "18480" + master: "10.0.0.3:9433" + dir: /tmp/sw-stripe-vs2 + extra_args: "-block.dir=/tmp/sw-stripe-vs2/blocks -block.listen=:3295 -ip=10.0.0.1" + save_as: vs2_pid + + - action: sleep + duration: 3s + + - action: wait_cluster_ready + node: m02 + master_url: "{{ master_url }}" + + - action: wait_block_servers + count: "2" + + - name: create-volumes + actions: + # Create 2 RF=1 volumes — one will land on each server + - action: create_block_volume + name: stripe-a + size_bytes: "{{ vol_size }}" + replica_factor: "1" + + - action: create_block_volume + name: stripe-b + size_bytes: "{{ vol_size }}" + replica_factor: "1" + + - action: sleep + duration: 5s + + - name: discover-placement + actions: + # Find where each volume was placed + - action: discover_primary + name: stripe-a + save_as: pri_a + + - action: discover_primary + name: stripe-b + save_as: pri_b + + - action: print + msg: "stripe-a on {{ pri_a }} ({{ pri_a_server }}), stripe-b on {{ pri_b }} ({{ pri_b_server }})" + + - name: connect-iscsi + actions: + # Connect to both volumes from m01 using discovered addresses + - action: lookup_block_volume + name: stripe-a + save_as: vol_a + + - action: lookup_block_volume + name: stripe-b + save_as: vol_b + + - action: iscsi_login_direct + node: m01 + host: "{{ vol_a_iscsi_host }}" + port: "{{ vol_a_iscsi_port }}" + iqn: "{{ vol_a_iqn }}" + save_as: dev_a + + - action: iscsi_login_direct + node: m01 + host: "{{ vol_b_iscsi_host }}" + port: "{{ vol_b_iscsi_port }}" + iqn: "{{ vol_b_iqn }}" + save_as: dev_b + + - action: print + msg: "Connected: dev_a={{ dev_a }} (m02), dev_b={{ dev_b }} (m01)" + + - name: setup-stripe + actions: + # Create dm-stripe: 64KB stripe unit across both devices + # stripe_sectors = 64KB / 512B = 128 sectors + # total_sectors = size of one device (they must be equal) + - action: exec + node: m01 + cmd: | + SECTORS_A=$(blockdev --getsz {{ dev_a }}) + SECTORS_B=$(blockdev --getsz {{ dev_b }}) + TOTAL=$((SECTORS_A + SECTORS_B)) + echo "Device A: {{ dev_a }} = ${SECTORS_A} sectors" + echo "Device B: {{ dev_b }} = ${SECTORS_B} sectors" + echo "Total: ${TOTAL} sectors" + echo "0 ${TOTAL} striped 2 128 {{ dev_a }} 0 {{ dev_b }} 0" | dmsetup create sw-stripe0 + ls -la /dev/mapper/sw-stripe0 + root: "true" + save_as: stripe_setup + + - action: print + msg: "dm-stripe created: /dev/mapper/sw-stripe0 ({{ stripe_setup }})" + + - name: benchmark-single + actions: + # Baseline: single volume performance (dev_a only) + - action: fio_json + node: m01 + device: "{{ dev_a }}" + rw: randwrite + bs: 4k + iodepth: "32" + runtime: "10" + time_based: "true" + name: single_vol_write + save_as: single_vol_write + + - action: fio_parse + json_var: single_vol_write + metric: iops + save_as: single_write_iops + + - action: fio_json + node: m01 + device: "{{ dev_a }}" + rw: randread + bs: 4k + iodepth: "32" + runtime: "10" + time_based: "true" + name: single_vol_read + save_as: single_vol_read + + - action: fio_parse + json_var: single_vol_read + metric: iops + save_as: single_read_iops + + - action: print + msg: "SINGLE VOLUME: write={{ single_write_iops }} IOPS, read={{ single_read_iops }} IOPS" + + - name: benchmark-stripe + actions: + # Striped: both volumes combined + - action: fio_json + node: m01 + device: /dev/mapper/sw-stripe0 + rw: randwrite + bs: 4k + iodepth: "32" + runtime: "10" + time_based: "true" + name: stripe_write + save_as: stripe_write + + - action: fio_parse + json_var: stripe_write + metric: iops + save_as: stripe_write_iops + + - action: fio_json + node: m01 + device: /dev/mapper/sw-stripe0 + rw: randread + bs: 4k + iodepth: "32" + runtime: "10" + time_based: "true" + name: stripe_read + save_as: stripe_read + + - action: fio_parse + json_var: stripe_read + metric: iops + save_as: stripe_read_iops + + - action: print + msg: "STRIPED (2x): write={{ stripe_write_iops }} IOPS, read={{ stripe_read_iops }} IOPS" + + - action: print + msg: "COMPARISON: single_write={{ single_write_iops }} → stripe_write={{ stripe_write_iops }} | single_read={{ single_read_iops }} → stripe_read={{ stripe_read_iops }}" + + - name: verify-data + actions: + # Write known data to stripe, read back, verify + - action: dd_write + node: m01 + device: /dev/mapper/sw-stripe0 + bs: 1M + count: "10" + sync_mode: fsync + save_as: stripe_md5 + + - action: dd_read_md5 + node: m01 + device: /dev/mapper/sw-stripe0 + bs: 1M + count: "10" + save_as: stripe_verify + + - action: assert_equal + actual: "{{ stripe_verify }}" + expected: "{{ stripe_md5 }}" + + - action: print + msg: "STRIPE DATA INTEGRITY VERIFIED: md5={{ stripe_md5 }}" + + - name: cleanup + always: true + actions: + - action: exec + node: m01 + cmd: "dmsetup remove sw-stripe0 2>/dev/null; true" + root: "true" + ignore_error: true + - action: iscsi_cleanup + node: m01 + ignore_error: true + - action: stop_weed + node: m01 + pid: "{{ vs2_pid }}" + ignore_error: true + - action: stop_weed + node: m02 + pid: "{{ vs1_pid }}" + ignore_error: true + - action: stop_weed + node: m02 + pid: "{{ master_pid }}" + ignore_error: true