From d7cd41571417e1e474d0f4c14477c05cfcccc8a3 Mon Sep 17 00:00:00 2001 From: pingqiu Date: Fri, 3 Apr 2026 08:59:46 -0700 Subject: [PATCH] =?UTF-8?q?feat:=20CP13-8=20=E2=80=94=20bounded=20real-wor?= =?UTF-8?q?kload=20validation=20scenario=20+=20envelope?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One named workload validation package for RF=2 sync_all: - Scenario: cp13-8-real-workload-validation.yaml (6 phases) - ext4 proof: write 200 files → failover → fsck + file count + md5sum diff - pgbench proof: TPC-B on promoted replica (database durability) - Disturbance: one bounded failover (kill primary, promote replica) Workload envelope doc: phase-13-cp8-workload-validation.md - Named topology, transport, workloads, disturbance, exclusions - Pass criteria: fsck passes, 200 files, checksums match, pgbench TPS > 0 - Maps each pass criterion to accepted CP13-1..7 semantics - Explicit non-claims: not rollout approval, not NVMe, not soak, not CP13-9 Reuses existing infrastructure: - cp85-db-ext4-fsck.yaml pattern (extended with checksums + pgbench) - benchmark-pgbench.yaml actions (pgbench_init/pgbench_run) Must run on real hardware (m01/M02). Cannot run in unit test harness. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../phase/phase-13-cp8-workload-validation.md | 75 ++++++ .../cp13-8-real-workload-validation.yaml | 237 ++++++++++++++++++ 2 files changed, 312 insertions(+) create mode 100644 sw-block/.private/phase/phase-13-cp8-workload-validation.md create mode 100644 weed/storage/blockvol/testrunner/scenarios/internal/cp13-8-real-workload-validation.yaml diff --git a/sw-block/.private/phase/phase-13-cp8-workload-validation.md b/sw-block/.private/phase/phase-13-cp8-workload-validation.md new file mode 100644 index 000000000..a9dcc339f --- /dev/null +++ b/sw-block/.private/phase/phase-13-cp8-workload-validation.md @@ -0,0 +1,75 @@ +# CP13-8 Real-Workload Validation — Envelope + Contract + +Date: 2026-04-03 + +## Workload Envelope + +| Parameter | Value | +|-----------|-------| +| Topology | RF=2, sync_all, cross-machine (m01 ↔ M02) | +| Transport | iSCSI (primary frontend) | +| Filesystem workload | ext4: 200 files, write + sync + failover + fsck + checksum verify | +| Application workload | PostgreSQL pgbench TPC-B (scale=1, c=1, 10s) on promoted replica | +| Disturbance | One bounded failover: kill primary, promote replica (epoch 1→2) | +| **NOT included** | NVMe-TCP, RF>2, hours/days soak, degraded-mode perf, mode normalization | + +## Scenario + +`weed/storage/blockvol/testrunner/scenarios/internal/cp13-8-real-workload-validation.yaml` + +### Phase flow + +1. **Setup**: RF=2 sync_all pair (primary on M02, replica on m01), standalone `iscsi-target` binary +2. **ext4 write**: iSCSI login → mkfs ext4 → write 200 files → md5sum → sync → umount → wait replication +3. **Failover**: kill primary → promote replica to primary (epoch 2) +4. **ext4 verify**: iSCSI login to promoted replica → fsck (filesystem integrity) → mount → file count == 200 → md5sum diff == MATCH +5. **pgbench**: iSCSI login → pgbench_init (ext4, scale=1) → TPC-B run (c=1, 10s) → TPS reported +6. **Cleanup**: always-run phase + +### Pass criteria + +| Proof | Assertion | What it validates | +|-------|-----------|-------------------| +| ext4 integrity | `fsck_ext4` passes | Replicated writes are filesystem-consistent after failover | +| ext4 completeness | `file_count == 200` | No files lost during replication + failover | +| ext4 correctness | `md5sum diff == MATCH` | File content identical to pre-failover (no corruption) | +| pgbench durability | `pgbench_run` completes with TPS > 0 | Database transactions are durable on sync_all promoted replica | + +## Relation to CP13-1..7 + +| Accepted checkpoint | What this workload validates | +|---------------------|------------------------------| +| CP13-2 (address truth) | Cross-machine iSCSI replication uses canonical addresses | +| CP13-3 (durable progress) | ext4 data survives failover because barrier guarantees flushed durability | +| CP13-4 (state eligibility) | Only InSync replica was eligible for barrier during replication | +| CP13-5 (reconnect/catch-up) | Replication completed before failover (all writes reached replica) | +| CP13-6 (retention) | WAL retained long enough for replication to complete | +| CP13-7 (rebuild fallback) | Not directly exercised — failover is clean (no WAL gap). Support-only. | + +## Existing infrastructure reused + +| Existing scenario | Relation | +|-------------------|----------| +| `cp85-db-ext4-fsck.yaml` | CP13-8 extends this pattern with checksums, pgbench, and explicit envelope | +| `benchmark-pgbench.yaml` | CP13-8 pgbench phase uses same `pgbench_init` + `pgbench_run` actions | + +## Run instructions + +```bash +# From m01 (client node): +sw-test-runner run cp13-8-real-workload-validation.yaml + +# Or from Windows dev machine (testrunner SSH): +cd C:/work/seaweedfs +go run ./weed/storage/blockvol/testrunner/cmd/sw-test-runner run \ + weed/storage/blockvol/testrunner/scenarios/internal/cp13-8-real-workload-validation.yaml +``` + +## What CP13-8 Does NOT Close + +- Mode normalization (CP13-9) +- Broad launch approval +- Performance floor (see Phase 12 P4) +- Degraded-mode validation +- NVMe-TCP transport validation +- Hours/days soak under sustained load diff --git a/weed/storage/blockvol/testrunner/scenarios/internal/cp13-8-real-workload-validation.yaml b/weed/storage/blockvol/testrunner/scenarios/internal/cp13-8-real-workload-validation.yaml new file mode 100644 index 000000000..c0b5c84f5 --- /dev/null +++ b/weed/storage/blockvol/testrunner/scenarios/internal/cp13-8-real-workload-validation.yaml @@ -0,0 +1,237 @@ +name: cp13-8-real-workload-validation +timeout: 20m + +# CP13-8: Bounded real-workload validation for RF=2 sync_all. +# +# Workload envelope: +# Topology: RF=2 sync_all, cross-machine replication (m01 ↔ M02) +# Transport: iSCSI (primary frontend) +# Workloads: ext4 (filesystem) + PostgreSQL pgbench (application) +# Disturbance: one bounded failover (kill primary, promote replica) +# Exclusions: NVMe-TCP, RF>2, hours/days soak, degraded-mode perf +# +# What this validates: +# The accepted CP13-1..7 replication contract survives contact with +# real filesystem and database consumers. Specifically: +# - Replicated writes are durable on both nodes (ext4 file integrity) +# - Post-failover data is consistent (fsck + file count) +# - Database transactions are durable under sync_all (pgbench TPC-B) +# +# What this does NOT validate: +# - Production rollout readiness +# - Performance floor (see Phase 12 P4) +# - Degraded mode behavior +# - NVMe-TCP transport path +# - Mode normalization (CP13-9) + +env: + repo_dir: "C:/work/seaweedfs" + +topology: + nodes: + target_node: + host: "192.168.1.184" + user: testdev + key: "C:/work/dev_server/testdev_key" + client_node: + host: "192.168.1.181" + user: testdev + key: "C:/work/dev_server/testdev_key" + +targets: + primary: + node: target_node + vol_size: 100M + iscsi_port: 3280 + admin_port: 8095 + replica_data_port: 9040 + replica_ctrl_port: 9041 + rebuild_port: 9042 + iqn_suffix: cp13-8-primary + replica: + node: client_node + vol_size: 100M + iscsi_port: 3281 + admin_port: 8096 + replica_data_port: 9043 + replica_ctrl_port: 9044 + rebuild_port: 9045 + iqn_suffix: cp13-8-replica + +phases: + # --- Phase 1: Setup RF=2 sync_all pair --- + - name: setup + actions: + - action: kill_stale + node: target_node + ignore_error: true + - action: kill_stale + node: client_node + ignore_error: true + - action: iscsi_cleanup + node: client_node + ignore_error: true + - action: build_deploy + - action: start_target + target: primary + create: "true" + durability_mode: sync_all + - action: start_target + target: replica + create: "true" + durability_mode: sync_all + - action: assign + target: replica + epoch: "1" + role: replica + lease_ttl: 60s + - action: assign + target: primary + epoch: "1" + role: primary + lease_ttl: 60s + - action: set_replica + target: primary + replica: replica + - action: sleep + duration: 2s + + # --- Phase 2: ext4 filesystem workload --- + - name: ext4-write + actions: + - action: iscsi_login + target: primary + node: client_node + save_as: device + - action: mkfs + node: client_node + device: "{{ device }}" + fstype: ext4 + - action: mount + node: client_node + device: "{{ device }}" + mountpoint: /mnt/cp13-8 + # Write 200 files with known content. + - action: exec + node: client_node + root: "true" + cmd: "bash -c 'for i in $(seq 1 200); do dd if=/dev/urandom of=/mnt/cp13-8/file_$i bs=4k count=1 2>/dev/null; done && sync'" + # Compute checksums for later verification. + - action: exec + node: client_node + root: "true" + cmd: "md5sum /mnt/cp13-8/file_* | sort > /tmp/cp13-8-checksums.txt && cat /tmp/cp13-8-checksums.txt | wc -l" + save_as: checksum_count + - action: assert_equal + actual: "{{ checksum_count }}" + expected: "200" + - action: umount + node: client_node + mountpoint: /mnt/cp13-8 + - action: iscsi_cleanup + node: client_node + ignore_error: true + # Wait for replication to catch up. + - action: wait_lsn + target: replica + min_lsn: "1" + timeout: 30s + - action: sleep + duration: 3s + + # --- Phase 3: Failover (kill primary, promote replica) --- + - name: failover + actions: + - action: kill_target + target: primary + - action: assign + target: replica + epoch: "2" + role: primary + lease_ttl: 60s + - action: wait_role + target: replica + role: primary + timeout: 10s + + # --- Phase 4: ext4 verification on promoted replica --- + - name: ext4-verify + actions: + - action: iscsi_login + target: replica + node: client_node + save_as: device2 + # fsck: filesystem integrity. + - action: fsck_ext4 + node: client_node + device: "{{ device2 }}" + save_as: fsck_result + # Mount and verify file count. + - action: mount + node: client_node + device: "{{ device2 }}" + mountpoint: /mnt/cp13-8 + - action: exec + node: client_node + root: "true" + cmd: "ls /mnt/cp13-8/file_* | wc -l" + save_as: post_failover_count + - action: assert_equal + actual: "{{ post_failover_count }}" + expected: "200" + # Verify checksums match pre-failover. + - action: exec + node: client_node + root: "true" + cmd: "md5sum /mnt/cp13-8/file_* | sort > /tmp/cp13-8-checksums-post.txt && diff /tmp/cp13-8-checksums.txt /tmp/cp13-8-checksums-post.txt && echo MATCH" + save_as: checksum_match + - action: assert_contains + actual: "{{ checksum_match }}" + expected: "MATCH" + - action: umount + node: client_node + mountpoint: /mnt/cp13-8 + - action: iscsi_cleanup + node: client_node + ignore_error: true + + # --- Phase 5: pgbench on promoted replica (application workload) --- + - name: pgbench-on-replica + actions: + - action: iscsi_login + target: replica + node: client_node + save_as: device3 + - action: pgbench_init + node: client_node + device: "{{ device3 }}" + mount: "/mnt/cp13-8-pg" + port: "5440" + scale: "1" + fstype: ext4 + - action: pgbench_run + node: client_node + clients: "1" + duration: "10" + save_as: tps_post_failover + - action: print + msg: "CP13-8: pgbench TPC-B post-failover: {{ tps_post_failover }} TPS" + # pgbench succeeded = database transactions are durable on the promoted replica. + - action: pgbench_cleanup + node: client_node + mount: "/mnt/cp13-8-pg" + port: "5440" + ignore_error: true + - action: iscsi_cleanup + node: client_node + ignore_error: true + + # --- Phase 6: Cleanup --- + - name: cleanup + always: true + actions: + - action: iscsi_cleanup + node: client_node + ignore_error: true + - action: stop_all_targets + ignore_error: true