mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-21 22:56:55 +00:00
doc: update testrunner roadmap — add runs.db text index for result tracking
P1 feature updated: replace generic "structured results" with concrete runs.db design (newline-delimited JSON, one line per run). Leverages existing RunBundle system (manifest.json, result.json already exist). New CLI commands: list, trend, gc, reindex, diff. Regression detection via stddev comparison against rolling baseline. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
pingqiu
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
676539d3b9
commit
2bc8dfcdde
@@ -142,50 +142,115 @@ works in a scenario YAML.
|
||||
|
||||
---
|
||||
|
||||
### P1: Structured Results + Trend Tracking
|
||||
### P1: Result Index + History (text-db)
|
||||
|
||||
**Goal**: Persist results in a queryable format. Detect performance
|
||||
regressions automatically.
|
||||
**Goal**: Index all run bundles in a lightweight text database. Support
|
||||
listing, searching, comparing, and cleaning up old runs.
|
||||
|
||||
**Why P1**: Without history, we can't tell if a code change degraded
|
||||
performance. Each run is ephemeral.
|
||||
**Why P1**: Runs accumulate as directories on the test node. Without an
|
||||
index, you can't find old results, compare trends, or clean up disk space.
|
||||
|
||||
**Design**:
|
||||
**What already exists** (RunBundle system):
|
||||
|
||||
```
|
||||
results/
|
||||
index.json ← append-only run index
|
||||
runs/
|
||||
20260411-140236/
|
||||
meta.json ← scenario, binary, topology, timestamp
|
||||
metrics.json ← {iops_write: 46666, iops_read: 61800, ...}
|
||||
phases.json ← per-phase timing and pass/fail
|
||||
artifacts/ ← logs, core dumps on failure
|
||||
baselines/
|
||||
rf1-perf.json ← rolling baseline: mean, stddev, threshold
|
||||
20260409-175214-d099/ ← one dir per run (timestamp-based)
|
||||
manifest.json ← run identity: scenario, git SHA, binary hash
|
||||
scenario.yaml ← frozen copy of input YAML
|
||||
result.json ← full structured result (phases, actions, vars)
|
||||
result.xml ← JUnit XML
|
||||
result.html ← HTML report
|
||||
artifacts/ ← collected logs on failure
|
||||
20260410-012330-5d99/
|
||||
...
|
||||
```
|
||||
|
||||
**New capabilities**:
|
||||
Each run is self-contained. manifest.json has: run_id, scenario_name,
|
||||
scenario_sha256, git_sha, host, status, command_line, started_at, finished_at.
|
||||
|
||||
| Feature | Description | Est. lines |
|
||||
|---------|-------------|-----------|
|
||||
| `metrics.json` writer | Extract metrics from fio/dd results, write structured JSON | 80 |
|
||||
| `index.json` updater | Append each run to index with scenario/binary/timestamp | 40 |
|
||||
| Baseline comparator | Compare latest metrics against baseline, flag regressions | 100 |
|
||||
| `sw-test-runner trend` | CLI command to show metric trend over last N runs | 80 |
|
||||
| Total | | ~300 |
|
||||
**What's missing**: an index across runs.
|
||||
|
||||
**Regression detection rule**:
|
||||
**Design**: Add a `runs.db` text file (one JSON line per run) that acts
|
||||
as a lightweight index. No external database dependency.
|
||||
|
||||
```
|
||||
if latest.iops < baseline.mean - 2 * baseline.stddev:
|
||||
WARN: performance regression detected
|
||||
latest: 38,000 IOPS
|
||||
baseline: 46,666 ± 1,200 IOPS (last 10 runs)
|
||||
results/
|
||||
runs.db ← newline-delimited JSON (one line per run)
|
||||
20260409-175214-d099/
|
||||
manifest.json
|
||||
result.json
|
||||
...
|
||||
```
|
||||
|
||||
**Acceptance**: After 5 runs of `rf1-perf-compare.yaml`, `sw-test-runner trend`
|
||||
shows IOPS over time and would flag a >5% regression.
|
||||
**runs.db format** (one JSON line per entry):
|
||||
|
||||
```json
|
||||
{"run_id":"20260409-175214-d099","scenario":"v2-rebuild-rejoin","status":"pass","actions":58,"passed":58,"failed":0,"duration_ms":181795,"iops_write":46666,"iops_read":0,"git_sha":"8ecc50645","binary_md5":"4bcf08","started_at":"2026-04-09T17:52:14Z","dir":"20260409-175214-d099","disk_mb":12}
|
||||
{"run_id":"20260410-012330-5d99","scenario":"recovery-baseline-failover","status":"pass","actions":43,"passed":43,"failed":0,"duration_ms":111583,"iops_write":28733,"git_sha":"8ecc50645","binary_md5":"4bcf08","started_at":"2026-04-10T01:23:30Z","dir":"20260410-012330-5d99","disk_mb":8}
|
||||
```
|
||||
|
||||
**New CLI commands**:
|
||||
|
||||
```bash
|
||||
# List all runs, most recent first
|
||||
sw-test-runner list
|
||||
RUN_ID SCENARIO STATUS ACTIONS IOPS_W DURATION
|
||||
20260411-140236-8012 dm-stripe-two-server pass 42/42 79001 1m10s
|
||||
20260410-012330-5d99 recovery-baseline-failover pass 43/43 28733 1m52s
|
||||
20260409-175214-d099 v2-rebuild-rejoin pass 58/58 — 3m02s
|
||||
|
||||
# Filter by scenario
|
||||
sw-test-runner list --scenario rebuild
|
||||
sw-test-runner list --status fail
|
||||
sw-test-runner list --since 2026-04-10
|
||||
|
||||
# Show trend for a scenario
|
||||
sw-test-runner trend rf1-perf-compare
|
||||
DATE GIT_SHA IOPS_WRITE IOPS_READ STATUS
|
||||
2026-04-10 8ecc506 47233 — pass (v1.5)
|
||||
2026-04-10 8ecc506 46666 — pass (v2)
|
||||
|
||||
# Compare two runs
|
||||
sw-test-runner diff 20260410-run1 20260410-run2
|
||||
|
||||
# Clean up old runs (keep last N per scenario)
|
||||
sw-test-runner gc --keep 10
|
||||
Deleted 23 runs, freed 1.2 GB
|
||||
|
||||
# Rebuild index from existing run directories
|
||||
sw-test-runner reindex
|
||||
Scanned 45 run directories, indexed 45 entries
|
||||
```
|
||||
|
||||
**New code**:
|
||||
|
||||
| Component | Description | Est. lines |
|
||||
|-----------|-------------|-----------|
|
||||
| Index writer | Append one JSON line to runs.db after each run | 50 |
|
||||
| Index reader | Parse runs.db, filter/sort/search | 80 |
|
||||
| `list` command | Terminal table from index | 60 |
|
||||
| `trend` command | Filter by scenario, show metrics over time | 60 |
|
||||
| `gc` command | Delete old run dirs, update index | 50 |
|
||||
| `reindex` command | Scan run dirs, rebuild runs.db from manifest+result | 60 |
|
||||
| Disk size calculator | `du -s` each run dir, store in index | 20 |
|
||||
| Total | | ~380 |
|
||||
|
||||
**Regression detection**:
|
||||
|
||||
```
|
||||
sw-test-runner trend rf1-perf-compare --check
|
||||
Latest: 46,666 IOPS
|
||||
Baseline (last 10): 47,100 ± 800 IOPS
|
||||
Status: OK (within 1 stddev)
|
||||
|
||||
sw-test-runner trend rf1-perf-compare --check
|
||||
Latest: 38,000 IOPS
|
||||
Baseline (last 10): 47,100 ± 800 IOPS
|
||||
Status: REGRESSION (-19.3%, > 2 stddev)
|
||||
```
|
||||
|
||||
**Acceptance**: After 5 runs, `sw-test-runner list` shows all 5 with
|
||||
metrics. `sw-test-runner gc --keep 3` deletes the 2 oldest.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user