no need to set default scripts now

4.15
admin: auto migrating master maintenance scripts to admin_script plugin config (#8509 )
2026-10-06 22:55:51 +00:00 · 2026-03-04 22:27:02 -08:00 · 2026-03-04 22:13:57 -08:00 · 2026-03-04 22:11:07 -08:00 · 2026-03-04 19:22:39 -08:00 · 2026-03-04 19:20:28 -08:00
1129 changed files with 11664 additions and 315069 deletions
@@ -135,7 +135,7 @@ jobs:

    - name: Archive logs
      if: always()
-      uses: actions/upload-artifact@v6
+      uses: actions/upload-artifact@v7
      with:
        name: output-logs
        path: docker/output.log
@@ -52,7 +52,7 @@ jobs:

    - name: Archive logs
      if: failure()
-      uses: actions/upload-artifact@v6
+      uses: actions/upload-artifact@v7
      with:
        name: ec-integration-test-logs
        path: |
@@ -42,7 +42,7 @@ jobs:
        
    - name: Upload test logs on failure
      if: failure()
-      uses: actions/upload-artifact@v6
+      uses: actions/upload-artifact@v7
      with:
        name: ec-test-logs
        path: test/erasure_coding/admin_dockertest/tmp/logs/
@@ -183,7 +183,7 @@ jobs:
        
    - name: Upload Test Artifacts
      if: always()
-      uses: actions/upload-artifact@v6
+      uses: actions/upload-artifact@v7
      with:
        name: fuse-integration-test-results
        path: |
@@ -49,7 +49,7 @@ jobs:
      
      - name: Upload Test Reports
        if: always()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: test-reports-java-${{ matrix.java }}
          path: |
@@ -70,7 +70,7 @@ jobs:
        
    - name: Archive logs on failure
      if: failure()
-      uses: actions/upload-artifact@v6
+      uses: actions/upload-artifact@v7
      with:
        name: metadata-subscribe-test-logs
        path: |
@@ -62,7 +62,7 @@ jobs:

    - name: Archive logs
      if: always()
-      uses: actions/upload-artifact@v6
+      uses: actions/upload-artifact@v7
      with:
        name: postgres-logs
        path: test/postgres/postgres-output.log
@@ -57,7 +57,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: integration-test-logs
          path: test/s3/normal/*.log
@@ -77,7 +77,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-filer-group-test-logs
          path: test/s3/filer_group/weed-test*.log
@@ -76,7 +76,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-versioning-test-logs-${{ matrix.test-type }}
          path: test/s3/versioning/weed-test*.log
@@ -124,7 +124,7 @@ jobs:

      - name: Upload server logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-versioning-compatibility-logs
          path: test/s3/versioning/weed-test*.log
@@ -172,7 +172,7 @@ jobs:

      - name: Upload server logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-cors-compatibility-logs
          path: test/s3/cors/weed-test*.log
@@ -239,7 +239,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-retention-test-logs-${{ matrix.test-type }}
          path: test/s3/retention/weed-test*.log
@@ -306,7 +306,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-cors-test-logs-${{ matrix.test-type }}
          path: test/s3/cors/weed-test*.log
@@ -355,7 +355,7 @@ jobs:

      - name: Upload server logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-retention-worm-logs
          path: test/s3/retention/weed-test*.log
@@ -422,7 +422,7 @@ jobs:

      - name: Upload stress test logs
        if: always()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-versioning-stress-logs
          path: test/s3/versioning/weed-test*.log
@@ -478,7 +478,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-tagging-test-logs
          path: test/s3/tagging/weed-test*.log
@@ -531,7 +531,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-remote-cache-test-logs
          path: |
@@ -65,7 +65,7 @@ jobs:

      - name: Upload test results on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: iam-unit-test-results
          path: |
@@ -162,7 +162,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-iam-integration-logs-${{ matrix.test-type }}
          path: test/s3/iam/weed-*.log
@@ -222,7 +222,7 @@ jobs:

      - name: Upload distributed test logs
        if: always()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-iam-distributed-logs
          path: test/s3/iam/weed-*.log
@@ -274,7 +274,7 @@ jobs:

      - name: Upload performance test results
        if: always()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-iam-performance-results
          path: |
@@ -152,7 +152,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-keycloak-test-logs
          path: |
@@ -121,7 +121,7 @@ jobs:
      
      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: test-logs-python-${{ matrix.python-version }}
          path: |
@@ -70,7 +70,7 @@ jobs:

      - name: Upload test results on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: policy-unit-test-results
          path: |
@@ -178,7 +178,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-policy-variables-test-logs
          path: /tmp/weed_policy_test_server.log
@@ -299,7 +299,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-policy-enforcement-logs-${{ matrix.test-case }}
          path: /tmp/weed_policy_enforcement_${{ matrix.test-case }}.log
@@ -386,7 +386,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: trusted-proxy-test-logs
          path: /tmp/weed_proxy_test.log
@@ -73,7 +73,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-spark-test-logs
          path: test/s3/spark/test-output.log
@@ -95,7 +95,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-sse-test-logs-${{ matrix.test-type }}
          path: test/s3/sse/weed-test*.log
@@ -143,7 +143,7 @@ jobs:

      - name: Upload server logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-sse-compatibility-logs
          path: test/s3/sse/weed-test*.log
@@ -192,7 +192,7 @@ jobs:

      - name: Upload server logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-sse-metadata-persistence-logs
          path: test/s3/sse/weed-test*.log
@@ -241,7 +241,7 @@ jobs:

      - name: Upload server logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-sse-copy-operations-logs
          path: test/s3/sse/weed-test*.log
@@ -290,7 +290,7 @@ jobs:

      - name: Upload server logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-sse-multipart-logs
          path: test/s3/sse/weed-test*.log
@@ -340,7 +340,7 @@ jobs:

      - name: Upload performance test logs
        if: always()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-sse-performance-logs
          path: test/s3/sse/weed-test*.log
@@ -389,7 +389,7 @@ jobs:

      - name: Upload server logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-volume-encryption-logs
          path: /tmp/seaweedfs-sse-*.log
@@ -66,7 +66,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: s3-tables-test-logs
          path: test/s3tables/table-buckets/test-output.log
@@ -122,7 +122,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: iceberg-catalog-test-logs
          path: test/s3tables/catalog/test-output.log
@@ -188,12 +188,73 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: trino-iceberg-catalog-test-logs
          path: test/s3tables/catalog_trino/test-output.log
          retention-days: 3

+  polaris-integration-tests:
+    name: Polaris Integration Tests
+    runs-on: ubuntu-22.04
+    timeout-minutes: 30
+
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v6
+
+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: 'go.mod'
+        id: go
+
+      - name: Run go mod tidy
+        run: go mod tidy
+
+      - name: Install SeaweedFS
+        run: |
+          go install -buildvcs=false ./weed
+
+      - name: Pre-pull Polaris image
+        run: docker pull apache/polaris:latest
+
+      - name: Run Polaris Integration Tests
+        timeout-minutes: 25
+        run: |
+          set -x
+          set -o pipefail
+          echo "=== System Information ==="
+          uname -a
+          free -h
+          df -h
+          echo "=== Starting Polaris Tests ==="
+
+          go test -v -timeout 20m ./test/s3tables/polaris 2>&1 | tee test/s3tables/polaris/test-output.log || {
+            echo "Polaris integration tests failed"
+            exit 1
+          }
+
+      - name: Show test output on failure
+        if: failure()
+        working-directory: test/s3tables/polaris
+        run: |
+          echo "=== Test Output ==="
+          if [ -f test-output.log ]; then
+            tail -200 test-output.log
+          fi
+
+          echo "=== Process information ==="
+          ps aux | grep -E "(weed|test|docker)" || true
+
+      - name: Upload test logs on failure
+        if: failure()
+        uses: actions/upload-artifact@v7
+        with:
+          name: polaris-test-logs
+          path: test/s3tables/polaris/test-output.log
+          retention-days: 3
+
  spark-iceberg-catalog-tests:
    name: Spark Iceberg Catalog Integration Tests
    runs-on: ubuntu-22.04
@@ -254,7 +315,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: spark-iceberg-catalog-test-logs
          path: test/s3tables/catalog_spark/test-output.log
@@ -322,7 +383,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: risingwave-catalog-test-logs
          path: test/s3tables/catalog_risingwave/test-output.log
@@ -388,7 +449,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: sts-integration-test-logs
          path: test/s3tables/sts_integration/test-output.log
@@ -457,7 +518,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: lakekeeper-integration-test-logs
          path: test/s3tables/lakekeeper/test-output.log
@@ -125,7 +125,7 @@ jobs:

    - name: Upload test results
      if: always()
-      uses: actions/upload-artifact@v6
+      uses: actions/upload-artifact@v7
      with:
        name: spark-test-results
        path: test/java/spark/target/surefire-reports/
@@ -103,7 +103,7 @@ jobs:

      - name: Upload server logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: seaweedfs-logs
          # Note: actions don't use defaults.run.working-directory, so path is relative to workspace root
@@ -106,7 +106,7 @@ jobs:

      - name: Upload test logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: tus-test-logs
          path: |
@@ -90,7 +90,7 @@ jobs:

      - name: Archive logs on failure
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
        with:
          name: volume-server-integration-test-logs
          path: /tmp/volume-server-it-logs/
@@ -1,247 +0,0 @@
-# SeaweedFS Block Storage -- Getting Started
-
-Block storage exposes SeaweedFS volumes as `/dev/sdX` block devices via iSCSI.
-You can format them with ext4/xfs, mount them, and use them like any disk.
-
-## Prerequisites
-
- Linux host with `open-iscsi` installed
- Docker with compose plugin (`docker compose`)
-
-```bash
-# Install iSCSI initiator (Ubuntu/Debian)
-sudo apt-get install -y open-iscsi
-
-# Verify
-sudo systemctl start iscsid
-```
-
-## Quick Start (5 minutes)
-
-### 1. Build the image
-
-```bash
-# From the seaweedfs repo root
-GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o docker/compose/weed ./weed
-cd docker
-docker build -f Dockerfile.local -t seaweedfs-block:local .
-```
-
-### 2. Start the cluster
-
-```bash
-cd docker/compose
-
-# Set HOST_IP to your machine's IP (for remote iSCSI clients)
-# Use 127.0.0.1 for local-only testing
-HOST_IP=127.0.0.1 docker compose -f local-block-compose.yml up -d
-```
-
-Wait ~5 seconds for the volume server to register with the master.
-
-### 3. Create a block volume
-
-```bash
-curl -s -X POST http://localhost:9333/block/volume \
-  -H "Content-Type: application/json" \
-  -d '{"name":"myvolume","size_bytes":1073741824}'
-```
-
-This creates a 1GB block volume, auto-assigns it as primary, and starts the
-iSCSI target. The response includes the IQN and iSCSI address.
-
-### 4. Connect via iSCSI
-
-```bash
-# Discover targets
-sudo iscsiadm -m discovery -t sendtargets -p 127.0.0.1:3260
-
-# Login
-sudo iscsiadm -m node -T iqn.2024-01.com.seaweedfs:vol.myvolume \
-  -p 127.0.0.1:3260 --login
-
-# Find the new device
-lsblk | grep sd
-```
-
-### 5. Format and mount
-
-```bash
-# Format with ext4
-sudo mkfs.ext4 /dev/sdX
-
-# Mount
-sudo mkdir -p /mnt/myvolume
-sudo mount /dev/sdX /mnt/myvolume
-
-# Use it like any filesystem
-echo "hello" | sudo tee /mnt/myvolume/test.txt
-```
-
-### 6. Cleanup
-
-```bash
-sudo umount /mnt/myvolume
-sudo iscsiadm -m node -T iqn.2024-01.com.seaweedfs:vol.myvolume \
-  -p 127.0.0.1:3260 --logout
-docker compose -f local-block-compose.yml down -v
-```
-
-## API Reference
-
-All endpoints are on the master server (default: port 9333).
-
-### Create volume
-
-```
-POST /block/volume
-Content-Type: application/json
-
-{
-  "name": "myvolume",
-  "size_bytes": 1073741824,
-  "disk_type": "ssd",
-  "replica_placement": "001",
-  "durability_mode": "best_effort"
-}
-```
-
-| Field | Required | Default | Description |
-|-------|----------|---------|-------------|
-| `name` | yes | -- | Volume name (alphanumeric + hyphens) |
-| `size_bytes` | yes | -- | Volume size in bytes |
-| `disk_type` | no | `""` | Disk type hint: `ssd`, `hdd` |
-| `replica_placement` | no | `000` | SeaweedFS placement: `000` (no replica), `001` (1 replica same rack) |
-| `durability_mode` | no | `best_effort` | `best_effort`, `sync_all`, `sync_quorum` |
-| `replica_factor` | no | `2` | Number of copies: 1, 2, or 3 |
-
-### List volumes
-
-```
-GET /block/volumes
-```
-
-Returns JSON array of all block volumes with status, role, epoch, IQN, etc.
-
-### Lookup volume
-
-```
-GET /block/volume/{name}
-```
-
-### Delete volume
-
-```
-DELETE /block/volume/{name}
-```
-
-### Assign role
-
-```
-POST /block/assign
-Content-Type: application/json
-
-{
-  "name": "myvolume",
-  "epoch": 2,
-  "role": "primary",
-  "lease_ttl_ms": 30000
-}
-```
-
-Roles: `primary`, `replica`, `stale`, `rebuilding`.
-
-### Cluster status
-
-```
-GET /block/status
-```
-
-Returns volume count, server count, failover stats, queue depth.
-
-## Remote Client Setup
-
-To connect from a remote machine (not the Docker host):
-
-1. Set `HOST_IP` to the Docker host's network-reachable IP:
-   ```bash
-   HOST_IP=192.168.1.100 docker compose -f local-block-compose.yml up -d
-   ```
-
-2. On the client machine:
-   ```bash
-   sudo iscsiadm -m discovery -t sendtargets -p 192.168.1.100:3260
-   sudo iscsiadm -m node -T iqn.2024-01.com.seaweedfs:vol.myvolume \
-     -p 192.168.1.100:3260 --login
-   ```
-
-## Volume Lifecycle
-
-```
-create  -->  primary (serving I/O via iSCSI)
-                |
-           unmount/remount OK (lease auto-renewed by master)
-                |
-         assign replica  -->  WAL shipping active
-                |
-          kill primary   -->  promote replica  -->  new primary
-                |
-          old primary    -->  rebuild from new primary
-```
-
-Key points:
- **Lease renewal is automatic.** The master continuously renews the primary's
-  write lease via the heartbeat stream. Unmount/remount works without manual
-  intervention.
- **Epoch fencing.** Each role change bumps the epoch. Old primaries cannot
-  write after being demoted -- even if they still have the lease.
- **Volumes survive container restart.** Data is stored in the Docker volume
-  at `/data/blocks/`. The volume server re-registers with the master on restart.
-
-## Troubleshooting
-
-**iSCSI login fails with "No records found"**
- Run discovery first: `sudo iscsiadm -m discovery -t sendtargets -p HOST:3260`
-
-**Device not appearing after login**
- Check `dmesg | tail` for SCSI errors
- Verify the volume is assigned as primary: `curl http://HOST:9333/block/volumes`
-
-**I/O errors on write**
- Check volume role is `primary` (not `none` or `stale`)
- Check master is running (lease renewal requires master heartbeat)
-
-**Stuck iSCSI session after container restart**
- Force logout: `sudo iscsiadm -m node -T IQN -p HOST:PORT --logout`
- If stuck: `sudo ss -K dst HOST dport = 3260` to kill the TCP connection
- Then re-discover and login
-
-## Docker Compose Reference
-
-```yaml
-# local-block-compose.yml
-services:
-  master:
-    image: seaweedfs-block:local
-    ports:
-      - "9333:9333"      # HTTP API
-      - "19333:19333"     # gRPC
-    command: ["master", "-ip=master", "-ip.bind=0.0.0.0", "-mdir=/data"]
-
-  volume:
-    image: seaweedfs-block:local
-    ports:
-      - "8280:8080"       # Volume HTTP
-      - "18280:18080"     # Volume gRPC
-      - "3260:3260"       # iSCSI target
-    command: >
-      volume -ip=volume -master=master:9333 -dir=/data
-      -block.dir=/data/blocks
-      -block.listen=0.0.0.0:3260
-      -block.portal=${HOST_IP:-127.0.0.1}:3260,1
-```
-
-Key flags:
- `-block.dir`: Directory for `.blk` volume files
- `-block.listen`: iSCSI target listen address (inside container)
- `-block.portal`: iSCSI portal address reported to clients (must be reachable)
@@ -1,38 +0,0 @@
-## SeaweedFS Block Storage — Docker Compose
-##
-## Usage:
-##   HOST_IP=192.168.1.100 docker compose -f local-block-compose.yml up -d
-##
-## The HOST_IP is used for iSCSI discovery so external clients can connect.
-## If running on the same host, you can use: HOST_IP=127.0.0.1
-
-services:
-  master:
-    image: seaweedfs-block:local
-    entrypoint: ["/usr/bin/weed"]
-    ports:
-      - "9333:9333"
-      - "19333:19333"
-    command: ["master", "-ip=master", "-ip.bind=0.0.0.0", "-mdir=/data"]
-
-  volume:
-    image: seaweedfs-block:local
-    ports:
-      - "8280:8080"
-      - "18280:18080"
-      - "3260:3260"
-    entrypoint: ["/bin/sh", "-c"]
-    command:
-      - >
-        mkdir -p /data/blocks &&
-        exec /usr/bin/weed volume
-        -ip=volume
-        -master=master:9333
-        -ip.bind=0.0.0.0
-        -port=8080
-        -dir=/data
-        -block.dir=/data/blocks
-        -block.listen=0.0.0.0:3260
-        -block.portal=${HOST_IP:-127.0.0.1}:3260,1
-    depends_on:
-      - master
@@ -7,6 +7,7 @@

 [master.maintenance]
 # periodically run these scripts are the same as running them from 'weed shell'
+# Scripts are skipped while an admin server is connected.
 scripts = """
  lock
  ec.encode -fullPercent=95 -quietFor=1h
@@ -1,2 +1,105 @@
 #!/bin/sh
-exec /usr/bin/weed "$@"
+
+# Enable FIPS 140-3 mode by default (Go 1.24+)
+# To disable: docker run -e GODEBUG=fips140=off ...
+export GODEBUG="${GODEBUG:+$GODEBUG,}fips140=on"
+
+# Fix permissions for mounted volumes
+# If /data is mounted from host, it might have different ownership
+# Fix this by ensuring seaweed user owns the directory
+if [ "$(id -u)" = "0" ]; then
+  # Running as root, check and fix permissions if needed
+  SEAWEED_UID=$(id -u seaweed)
+  SEAWEED_GID=$(id -g seaweed)
+  
+  # Verify seaweed user and group exist
+  if [ -z "$SEAWEED_UID" ] || [ -z "$SEAWEED_GID" ]; then
+    echo "Error: 'seaweed' user or group not found. Cannot fix permissions." >&2
+    exit 1
+  fi
+  
+  DATA_UID=$(stat -c '%u' /data 2>/dev/null)
+  DATA_GID=$(stat -c '%g' /data 2>/dev/null)
+
+  # Only run chown -R if ownership doesn't already match (avoids expensive
+  # recursive chown on subsequent starts, and is a no-op on OpenShift when
+  # fsGroup has already set correct ownership on the PVC).
+  if [ "$DATA_UID" != "$SEAWEED_UID" ] || [ "$DATA_GID" != "$SEAWEED_GID" ]; then
+    echo "Fixing /data ownership for seaweed user (uid=$SEAWEED_UID, gid=$SEAWEED_GID)"
+    if ! chown -R seaweed:seaweed /data; then
+      echo "Warning: Failed to change ownership of /data. This may cause permission errors." >&2
+      echo "If /data is read-only or has mount issues, the application may fail to start." >&2
+    fi
+  fi
+  
+  # Use su-exec to drop privileges and run as seaweed user
+  exec su-exec seaweed "$0" "$@"
+fi
+
+isArgPassed() {
+  arg="$1"
+  argWithEqualSign="$1="
+  shift
+  while [ $# -gt 0 ]; do
+    passedArg="$1"
+    shift
+    case $passedArg in
+    "$arg")
+      return 0
+      ;;
+    "$argWithEqualSign"*)
+      return 0
+      ;;
+    esac
+  done
+  return 1
+}
+
+case "$1" in
+
+  'master')
+  	ARGS="-mdir=/data -volumeSizeLimitMB=1024"
+  	shift
+  	exec /usr/bin/weed -logtostderr=true master $ARGS $@
+	;;
+
+  'volume')
+  	ARGS="-dir=/data -max=0"
+  	if isArgPassed "-max" "$@"; then
+  	  ARGS="-dir=/data"
+  	fi
+  	shift
+  	exec /usr/bin/weed -logtostderr=true volume $ARGS $@
+	;;
+
+  'server')
+  	ARGS="-dir=/data -volume.max=0 -master.volumeSizeLimitMB=1024"
+  	if isArgPassed "-volume.max" "$@"; then
+  	  ARGS="-dir=/data -master.volumeSizeLimitMB=1024"
+  	fi
+ 	shift
+  	exec /usr/bin/weed -logtostderr=true server $ARGS $@
+  	;;
+
+  'filer')
+  	ARGS=""
+  	shift
+  	exec /usr/bin/weed -logtostderr=true filer $ARGS $@
+	;;
+
+  's3')
+  	ARGS="-domainName=$S3_DOMAIN_NAME -key.file=$S3_KEY_FILE -cert.file=$S3_CERT_FILE"
+  	shift
+  	exec /usr/bin/weed -logtostderr=true s3 $ARGS $@
+	;;
+
+  'shell')
+  	ARGS="-cluster=$SHELL_CLUSTER -filer=$SHELL_FILER -filerGroup=$SHELL_FILER_GROUP -master=$SHELL_MASTER -options=$SHELL_OPTIONS"
+  	shift
+  	exec echo "$@" | /usr/bin/weed -logtostderr=true shell $ARGS
+  ;;
+
+  *)
+  	exec /usr/bin/weed $@
+	;;
+esac
@@ -26,7 +26,7 @@ require (
 	github.com/facebookgo/stats v0.0.0-20151006221625-1b76add642e4
 	github.com/facebookgo/subset v0.0.0-20200203212716-c811ad88dec4 // indirect
 	github.com/fsnotify/fsnotify v1.9.0 // indirect
-	github.com/go-redsync/redsync/v4 v4.15.0
+	github.com/go-redsync/redsync/v4 v4.16.0
 	github.com/go-sql-driver/mysql v1.9.3
 	github.com/go-zookeeper/zk v1.0.3 // indirect
 	github.com/golang/protobuf v1.5.4
@@ -89,20 +89,20 @@ require (
 	go.etcd.io/etcd/client/v3 v3.6.7
 	go.mongodb.org/mongo-driver v1.17.6
 	go.opencensus.io v0.24.0 // indirect
-	gocloud.dev v0.44.0
-	gocloud.dev/pubsub/natspubsub v0.44.0
+	gocloud.dev v0.45.0
+	gocloud.dev/pubsub/natspubsub v0.45.0
 	gocloud.dev/pubsub/rabbitpubsub v0.44.0
 	golang.org/x/crypto v0.48.0
 	golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546
 	golang.org/x/image v0.36.0
 	golang.org/x/net v0.49.0
-	golang.org/x/oauth2 v0.34.0
+	golang.org/x/oauth2 v0.35.0
 	golang.org/x/sys v0.41.0
 	golang.org/x/text v0.34.0 // indirect
 	golang.org/x/tools v0.41.0 // indirect
 	golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
 	google.golang.org/api v0.258.0
-	google.golang.org/genproto v0.0.0-20250922171735-9219d122eba9 // indirect
+	google.golang.org/genproto v0.0.0-20251124214823-79d6a2a48846 // indirect
 	google.golang.org/grpc v1.78.0
 	google.golang.org/protobuf v1.36.11
 	gopkg.in/inf.v0 v0.9.1 // indirect
@@ -129,7 +129,6 @@ require (
 	github.com/aws/aws-sdk-go-v2/credentials v1.19.7
 	github.com/aws/aws-sdk-go-v2/service/s3 v1.95.0
 	github.com/cognusion/imaging v1.0.2
-	github.com/container-storage-interface/spec v1.10.0
 	github.com/fluent/fluent-logger-golang v1.10.1
 	github.com/getsentry/sentry-go v0.42.0
 	github.com/go-ldap/ldap/v3 v3.4.12
@@ -139,7 +138,7 @@ require (
 	github.com/hashicorp/raft-boltdb/v2 v2.3.1
 	github.com/hashicorp/vault/api v1.22.0
 	github.com/jhump/protoreflect v1.18.0
-	github.com/linkedin/goavro/v2 v2.14.1
+	github.com/linkedin/goavro/v2 v2.15.0
 	github.com/mattn/go-sqlite3 v1.14.34
 	github.com/minio/crc64nvme v1.1.1
 	github.com/orcaman/concurrent-map/v2 v2.0.1
@@ -151,7 +150,7 @@ require (
 	github.com/redis/go-redis/v9 v9.18.0
 	github.com/schollz/progressbar/v3 v3.19.0
 	github.com/seaweedfs/go-fuse/v2 v2.9.1
-	github.com/shirou/gopsutil/v4 v4.26.1
+	github.com/shirou/gopsutil/v4 v4.26.2
 	github.com/tarantool/go-tarantool/v2 v2.4.1
 	github.com/testcontainers/testcontainers-go v0.39.0
 	github.com/tikv/client-go/v2 v2.0.7
@@ -172,7 +171,7 @@ require (
 	atomicgo.dev/keyboard v0.2.9 // indirect
 	atomicgo.dev/schedule v0.1.0 // indirect
 	cloud.google.com/go/longrunning v0.7.0 // indirect
-	cloud.google.com/go/pubsub/v2 v2.2.1 // indirect
+	cloud.google.com/go/pubsub/v2 v2.3.0 // indirect
 	dario.cat/mergo v1.0.2 // indirect
 	github.com/Azure/azure-sdk-for-go/sdk/keyvault/internal v0.7.1 // indirect
 	github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
@@ -227,7 +226,6 @@ require (
 	github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect
 	github.com/hashicorp/go-sockaddr v1.0.7 // indirect
 	github.com/hashicorp/hcl v1.0.1-vault-7 // indirect
-	github.com/iceber/iouring-go v0.0.0-20230403020409-002cfd2e2a90 // indirect
 	github.com/internxt/rclone-adapter v0.0.0-20260213125353-6f59c89fcb7c // indirect
 	github.com/jackc/pgpassfile v1.0.0 // indirect
 	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
@@ -257,7 +255,6 @@ require (
 	github.com/openzipkin/zipkin-go v0.4.3 // indirect
 	github.com/parquet-go/bitpack v1.0.0 // indirect
 	github.com/parquet-go/jsonlite v1.0.0 // indirect
-	github.com/pawelgaczynski/giouring v0.0.0-20230826085535-69588b89acb9 // indirect
 	github.com/petermattis/goid v0.0.0-20260113132338-7c7de50cc741 // indirect
 	github.com/pierrre/geohash v1.0.0 // indirect
 	github.com/pquerna/otp v1.5.0 // indirect
@@ -282,10 +279,10 @@ require (
 	github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
 	github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
 	github.com/zeebo/xxh3 v1.0.2 // indirect
-	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.37.0 // indirect
-	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.37.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.38.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.38.0 // indirect
 	go.opentelemetry.io/otel/exporters/zipkin v1.36.0 // indirect
-	go.opentelemetry.io/proto/otlp v1.7.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.9.0 // indirect
 	go.uber.org/mock v0.5.2 // indirect
 	go.yaml.in/yaml/v2 v2.4.3 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
@@ -295,12 +292,12 @@ require (
 )

 require (
-	cel.dev/expr v0.24.0 // indirect
+	cel.dev/expr v0.25.1 // indirect
 	cloud.google.com/go/auth v0.17.0 // indirect
 	cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
 	cloud.google.com/go/compute/metadata v0.9.0 // indirect
 	cloud.google.com/go/iam v1.5.3 // indirect
-	cloud.google.com/go/monitoring v1.24.2 // indirect
+	cloud.google.com/go/monitoring v1.24.3 // indirect
 	filippo.io/edwards25519 v1.1.1 // indirect
 	github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0
 	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1
@@ -330,7 +327,7 @@ require (
 	github.com/arangodb/go-velocypack v0.0.0-20200318135517-5af53c29c67e // indirect
 	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4 // indirect
 	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.17 // indirect
-	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.4 // indirect
+	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.12 // indirect
 	github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.17 // indirect
 	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.17 // indirect
 	github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
@@ -339,11 +336,11 @@ require (
 	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.7 // indirect
 	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.17 // indirect
 	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.16 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sns v1.34.7 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sqs v1.38.8 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sns v1.39.7 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sqs v1.42.17 // indirect
 	github.com/aws/aws-sdk-go-v2/service/sso v1.30.9 // indirect
 	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.13 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sts v1.41.6 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sts v1.41.6
 	github.com/aws/smithy-go v1.24.0
 	github.com/boltdb/bolt v1.3.1 // indirect
 	github.com/bradenaw/juniper v0.15.3 // indirect
@@ -355,7 +352,7 @@ require (
 	github.com/cloudinary/cloudinary-go/v2 v2.13.0 // indirect
 	github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc // indirect
 	github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc // indirect
-	github.com/cncf/xds/go v0.0.0-20251022180443-0feb69152e9f // indirect
+	github.com/cncf/xds/go v0.0.0-20251110193048-8bfbf64dc13e // indirect
 	github.com/colinmarc/hdfs/v2 v2.4.0 // indirect
 	github.com/creasty/defaults v1.8.0 // indirect
 	github.com/cronokirby/saferith v0.33.0 // indirect
@@ -363,11 +360,11 @@ require (
 	github.com/d4l3k/messagediff v1.2.1 // indirect
 	github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 // indirect
 	github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5 // indirect
-	github.com/ebitengine/purego v0.9.1 // indirect
+	github.com/ebitengine/purego v0.10.0 // indirect
 	github.com/elastic/gosigar v0.14.3 // indirect
 	github.com/emersion/go-message v0.18.2 // indirect
 	github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff // indirect
-	github.com/envoyproxy/go-control-plane/envoy v1.35.0 // indirect
+	github.com/envoyproxy/go-control-plane/envoy v1.36.0 // indirect
 	github.com/envoyproxy/protoc-gen-validate v1.2.1 // indirect
 	github.com/fatih/color v1.18.0 // indirect
 	github.com/felixge/httpsnoop v1.0.4 // indirect
@@ -431,8 +428,8 @@ require (
 	github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4
 	github.com/montanaflynn/stats v0.7.1 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
-	github.com/nats-io/nats.go v1.43.0 // indirect
-	github.com/nats-io/nkeys v0.4.11 // indirect
+	github.com/nats-io/nats.go v1.48.0 // indirect
+	github.com/nats-io/nkeys v0.4.12 // indirect
 	github.com/nats-io/nuid v1.0.1 // indirect
 	github.com/ncruces/go-strftime v1.0.0 // indirect
 	github.com/ncw/swift/v2 v2.0.5 // indirect
@@ -496,11 +493,11 @@ require (
 	go.opentelemetry.io/contrib/detectors/gcp v1.38.0 // indirect
 	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.63.0 // indirect
 	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0 // indirect
-	go.opentelemetry.io/otel v1.38.0 // indirect
-	go.opentelemetry.io/otel/metric v1.38.0 // indirect
-	go.opentelemetry.io/otel/sdk v1.38.0 // indirect
-	go.opentelemetry.io/otel/sdk/metric v1.38.0 // indirect
-	go.opentelemetry.io/otel/trace v1.38.0 // indirect
+	go.opentelemetry.io/otel v1.40.0 // indirect
+	go.opentelemetry.io/otel/metric v1.40.0 // indirect
+	go.opentelemetry.io/otel/sdk v1.40.0 // indirect
+	go.opentelemetry.io/otel/sdk/metric v1.40.0 // indirect
+	go.opentelemetry.io/otel/trace v1.40.0 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
 	go.uber.org/zap v1.27.1 // indirect
 	golang.org/x/term v0.40.0 // indirect
@@ -523,14 +520,3 @@ require (
 )

 // replace github.com/seaweedfs/raft => /Users/chrislu/go/src/github.com/seaweedfs/raft
-
-// V2 engine bridge modules (Phase 07)
-require (
-	github.com/seaweedfs/seaweedfs/sw-block/engine/replication v0.0.0
-	github.com/seaweedfs/seaweedfs/sw-block/bridge/blockvol v0.0.0
-)
-
-replace (
-	github.com/seaweedfs/seaweedfs/sw-block/engine/replication => ./sw-block/engine/replication
-	github.com/seaweedfs/seaweedfs/sw-block/bridge/blockvol => ./sw-block/bridge/blockvol
-)
@@ -1,6 +1,6 @@
 apiVersion: v1
 description: SeaweedFS
 name: seaweedfs
-appVersion: "4.13"
+appVersion: "4.15"
 # Dev note: Trigger a helm chart release by `git tag -a helm-<version>`
-version: 4.0.413
+version: 4.15.0
@@ -100,12 +100,19 @@ filer:

 # S3 gateway (if enabled)
 s3:
+  enabled: true
+  replicas: 1
+  port: 8333
+  enableAuth: true
  podSecurityContext:
    enabled: true
    # On OpenShift, we omit runAsUser/runAsGroup/fsGroup to let the admission
    # controller assign them automatically based on the namespace's SCC.
    runAsNonRoot: true

+  logs:
+    type: "emptyDir"
+
  containerSecurityContext:
    enabled: true
    allowPrivilegeEscalation: false
@@ -118,8 +118,10 @@ spec:
                  fieldPath: metadata.namespace
            - name: SEAWEEDFS_FULLNAME
              value: "{{ include "seaweedfs.fullname" . }}"
-            {{- if .Values.admin.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.admin.extraEnvironmentVars }}
+            {{- $mergedExtraEnvironmentVars := dict }}
+            {{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global "component" .Values.admin "target" $mergedExtraEnvironmentVars) }}
+            {{- range $key := keys $mergedExtraEnvironmentVars | sortAlpha }}
+            {{- $value := index $mergedExtraEnvironmentVars $key }}
            - name: {{ $key }}
            {{- if kindIs "string" $value }}
              value: {{ tpl $value $ | quote }}
@@ -128,18 +130,6 @@ spec:
                {{ toYaml $value | nindent 16 | trim }}
            {{- end -}}
            {{- end }}
-            {{- end }}
-            {{- if .Values.global.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.global.extraEnvironmentVars }}
-            - name: {{ $key }}
-            {{- if kindIs "string" $value }}
-              value: {{ tpl $value $ | quote }}
-            {{- else }}
-              valueFrom:
-                {{ toYaml $value | nindent 16 | trim }}
-            {{- end -}}
-            {{- end }}
-            {{- end }}
          command:
            - "/bin/sh"
            - "-ec"
@@ -96,8 +96,10 @@ spec:
            - name: WEED_GRPC_CA
              value: /usr/local/share/ca-certificates/client/ca.crt
            {{- end }}
-            {{- if .Values.cosi.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.cosi.extraEnvironmentVars }}
+            {{- $mergedExtraEnvironmentVars := dict }}
+            {{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global "component" .Values.cosi "target" $mergedExtraEnvironmentVars) }}
+            {{- range $key := keys $mergedExtraEnvironmentVars | sortAlpha }}
+            {{- $value := index $mergedExtraEnvironmentVars $key }}
            - name: {{ $key }}
            {{- if kindIs "string" $value }}
              value: {{ tpl $value $ | quote }}
@@ -106,18 +108,6 @@ spec:
                {{ toYaml $value | nindent 16 | trim }}
            {{- end -}}
            {{- end }}
-            {{- end }}
-            {{- if .Values.global.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.global.extraEnvironmentVars }}
-            - name: {{ $key }}
-            {{- if kindIs "string" $value }}
-              value: {{ tpl $value $ | quote }}
-            {{- else }}
-              valueFrom:
-                {{ toYaml $value | nindent 16 | trim }}
-            {{- end -}}
-            {{- end }}
-            {{- end }}
          volumeMounts:
            - mountPath: /var/lib/cosi
              name: socket
@@ -114,8 +114,10 @@ spec:
                  optional: true
            - name: SEAWEEDFS_FULLNAME
              value: "{{ include "seaweedfs.fullname" . }}"
-            {{- if .Values.filer.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.filer.extraEnvironmentVars }}
+            {{- $mergedExtraEnvironmentVars := dict }}
+            {{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global "component" .Values.filer "target" $mergedExtraEnvironmentVars) }}
+            {{- range $key := keys $mergedExtraEnvironmentVars | sortAlpha }}
+            {{- $value := index $mergedExtraEnvironmentVars $key }}
            - name: {{ $key }}
            {{- if kindIs "string" $value }}
              value: {{ tpl $value $ | quote }}
@@ -124,18 +126,6 @@ spec:
                {{ toYaml $value | nindent 16 | trim }}
            {{- end -}}
            {{- end }}
-            {{- end }}
-            {{- if .Values.global.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.global.extraEnvironmentVars }}
-            - name: {{ $key }}
-            {{- if kindIs "string" $value }}
-              value: {{ tpl $value $ | quote }}
-            {{- else }}
-              valueFrom:
-                {{ toYaml $value | nindent 16 | trim }}
-            {{- end -}}
-            {{- end }}
-            {{- end }}
            {{- if .Values.filer.secretExtraEnvironmentVars }}
            {{- range $key, $value := .Values.filer.secretExtraEnvironmentVars }}
            - name: {{ $key }}
@@ -69,9 +69,7 @@ spec:
      priorityClassName: {{ .Values.master.priorityClassName | quote }}
      {{- end }}
      enableServiceLinks: false
-      {{- if .Values.global.createClusterRole }}
      serviceAccountName: {{ .Values.master.serviceAccountName | default (include "seaweedfs.serviceAccountName" .) | quote }} # for deleting statefulset pods after migration
-      {{- end }}
      {{- if .Values.master.initContainers }}
      initContainers:
        {{ tpl .Values.master.initContainers . | nindent 8 | trim }}
@@ -98,8 +96,10 @@ spec:
                  fieldPath: metadata.namespace
            - name: SEAWEEDFS_FULLNAME
              value: "{{ include "seaweedfs.fullname" . }}"
-            {{- if .Values.master.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.master.extraEnvironmentVars }}
+            {{- $mergedExtraEnvironmentVars := dict }}
+            {{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global "component" .Values.master "target" $mergedExtraEnvironmentVars) }}
+            {{- range $key := keys $mergedExtraEnvironmentVars | sortAlpha }}
+            {{- $value := index $mergedExtraEnvironmentVars $key }}
            - name: {{ $key }}
            {{- if kindIs "string" $value }}
              value: {{ tpl $value $ | quote }}
@@ -108,18 +108,6 @@ spec:
                {{ toYaml $value | nindent 16 | trim }}
            {{- end -}}
            {{- end }}
-            {{- end }}
-            {{- if .Values.global.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.global.extraEnvironmentVars }}
-            - name: {{ $key }}
-            {{- if kindIs "string" $value }}
-              value: {{ tpl $value $ | quote }}
-            {{- else }}
-              valueFrom:
-                {{ toYaml $value | nindent 16 | trim }}
-            {{- end -}}
-            {{- end }}
-            {{- end }}
          command:
            - "/bin/sh"
            - "-ec"
@@ -90,8 +90,10 @@ spec:
                  fieldPath: metadata.namespace
            - name: SEAWEEDFS_FULLNAME
              value: "{{ include "seaweedfs.fullname" . }}"
-            {{- if .Values.s3.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.s3.extraEnvironmentVars }}
+            {{- $mergedExtraEnvironmentVars := dict }}
+            {{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global "component" .Values.s3 "target" $mergedExtraEnvironmentVars) }}
+            {{- range $key := keys $mergedExtraEnvironmentVars | sortAlpha }}
+            {{- $value := index $mergedExtraEnvironmentVars $key }}
            - name: {{ $key }}
            {{- if kindIs "string" $value }}
              value: {{ tpl $value $ | quote }}
@@ -100,18 +102,6 @@ spec:
                {{ toYaml $value | nindent 16 | trim }}
            {{- end -}}
            {{- end }}
-            {{- end }}
-            {{- if .Values.global.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.global.extraEnvironmentVars }}
-            - name: {{ $key }}
-            {{- if kindIs "string" $value }}
-              value: {{ tpl $value $ | quote }}
-            {{- else }}
-              valueFrom:
-                {{ toYaml $value | nindent 16 | trim }}
-            {{- end -}}
-            {{- end }}
-            {{- end }}
          command:
            - "/bin/sh"
            - "-ec"
@@ -90,8 +90,10 @@ spec:
                  fieldPath: metadata.namespace
            - name: SEAWEEDFS_FULLNAME
              value: "{{ include "seaweedfs.fullname" . }}"
-            {{- if .Values.sftp.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.sftp.extraEnvironmentVars }}
+            {{- $mergedExtraEnvironmentVars := dict }}
+            {{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global "component" .Values.sftp "target" $mergedExtraEnvironmentVars) }}
+            {{- range $key := keys $mergedExtraEnvironmentVars | sortAlpha }}
+            {{- $value := index $mergedExtraEnvironmentVars $key }}
            - name: {{ $key }}
            {{- if kindIs "string" $value }}
              value: {{ tpl $value $ | quote }}
@@ -100,18 +102,6 @@ spec:
                {{ toYaml $value | nindent 16 | trim }}
            {{- end -}}
            {{- end }}
-            {{- end }}
-            {{- if .Values.global.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.global.extraEnvironmentVars }}
-            - name: {{ $key }}
-            {{- if kindIs "string" $value }}
-              value: {{ tpl $value $ | quote }}
-            {{- else }}
-              valueFrom:
-                {{ toYaml $value | nindent 16 | trim }}
-            {{- end -}}
-            {{- end }}
-            {{- end }}
          command:
            - "/bin/sh"
            - "-ec"
@@ -59,6 +59,18 @@ Inject extra environment vars in the format key:value, if populated
 {{- end -}}
 {{- end -}}

+{{- define "seaweedfs.mergeExtraEnvironmentVars" -}}
+{{- $global := ((.global | default dict).extraEnvironmentVars | default dict) -}}
+{{- $component := ((.component | default dict).extraEnvironmentVars | default dict) -}}
+{{- $target := .target -}}
+{{- range $key, $value := $global }}
+{{- $_ := set $target $key $value }}
+{{- end }}
+{{- range $key, $value := $component }}
+{{- $_ := set $target $key $value }}
+{{- end }}
+{{- end -}}
+
 {{/* Return the proper filer image */}}
 {{- define "filer.image" -}}
 {{- if .Values.filer.imageOverride -}}
@@ -69,9 +69,7 @@ spec:
      priorityClassName: {{ $volume.priorityClassName | quote }}
      {{- end }}
      enableServiceLinks: false
-      {{- if $.Values.global.createClusterRole }}
      serviceAccountName: {{ $volume.serviceAccountName | default (include "seaweedfs.serviceAccountName" $) | quote }} # for deleting statefulset pods after migration
-      {{- end }}
      {{- $initContainers_exists := include "volume.initContainers_exists" $ -}}
      {{- if $initContainers_exists }}
      initContainers:
@@ -118,8 +116,10 @@ spec:
                  fieldPath: status.hostIP
            - name: SEAWEEDFS_FULLNAME
              value: "{{ include "seaweedfs.fullname" $ }}"
-            {{- if $volume.extraEnvironmentVars }}
-            {{- range $key, $value := $volume.extraEnvironmentVars }}
+            {{- $mergedExtraEnvironmentVars := dict }}
+            {{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" $.Values.global "component" $volume "target" $mergedExtraEnvironmentVars) }}
+            {{- range $key := keys $mergedExtraEnvironmentVars | sortAlpha }}
+            {{- $value := index $mergedExtraEnvironmentVars $key }}
            - name: {{ $key }}
            {{- if kindIs "string" $value }}
              value: {{ tpl $value $ | quote }}
@@ -128,18 +128,6 @@ spec:
                {{ toYaml $value | nindent 16 | trim }}
            {{- end -}}
            {{- end }}
-            {{- end }}
-            {{- if $.Values.global.extraEnvironmentVars }}
-            {{- range $key, $value := $.Values.global.extraEnvironmentVars }}
-            - name: {{ $key }}
-            {{- if kindIs "string" $value }}
-              value: {{ tpl $value $ | quote }}
-            {{- else }}
-              valueFrom:
-                {{ toYaml $value | nindent 16 | trim }}
-            {{- end -}}
-            {{- end }}
-            {{- end }}
          command:
            - "/bin/sh"
            - "-ec"
@@ -93,8 +93,10 @@ spec:
                  fieldPath: metadata.namespace
            - name: SEAWEEDFS_FULLNAME
              value: "{{ include "seaweedfs.fullname" . }}"
-            {{- if .Values.worker.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.worker.extraEnvironmentVars }}
+            {{- $mergedExtraEnvironmentVars := dict }}
+            {{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global "component" .Values.worker "target" $mergedExtraEnvironmentVars) }}
+            {{- range $key := keys $mergedExtraEnvironmentVars | sortAlpha }}
+            {{- $value := index $mergedExtraEnvironmentVars $key }}
            - name: {{ $key }}
            {{- if kindIs "string" $value }}
              value: {{ tpl $value $ | quote }}
@@ -103,18 +105,6 @@ spec:
                {{ toYaml $value | nindent 16 | trim }}
            {{- end -}}
            {{- end }}
-            {{- end }}
-            {{- if .Values.global.extraEnvironmentVars }}
-            {{- range $key, $value := .Values.global.extraEnvironmentVars }}
-            - name: {{ $key }}
-            {{- if kindIs "string" $value }}
-              value: {{ tpl $value $ | quote }}
-            {{- else }}
-              valueFrom:
-                {{ toYaml $value | nindent 16 | trim }}
-            {{- end -}}
-            {{- end }}
-            {{- end }}
          command:
            - "/bin/sh"
            - "-ec"
@@ -1,104 +0,0 @@
-# Phase 5 Dev Log
-
-Append-only communication between agents. Newest entries at bottom.
-Each entry: `[date] [role] message`
-
-Roles: `DEV`, `REVIEWER`, `TESTER`, `ARCHITECT`
-
---
-
-[2026-03-03] [DEV] CP5-1 ALUA + multipath complete. Added ALUA provider + REPORT TPG (implicit ALUA), VPD 0x83
-NAA+TPG+RTP descriptors, TPGS=01 in INQUIRY, standby write fencing, and -tpg-id flag. Added UUID to VolumeInfo for
-shared NAA. Added multipath config and setup script. 4 multipath integration tests added. 10 ALUA unit tests added
-(SCSI tests total 53). Reviewer fixes applied: RoleNone maps to Active/Optimized to avoid single-node regression;
-REPORT TPG advertises T_SUP when state is Transitioning; TPG ID validation; non-ASCII log fix. Added 2 tests:
-alua_role_none_allows_writes and alua_report_tpg_transitioning. All unit tests pass, Linux cross-compile verified.
-
-[2026-03-03] [TESTER] CP5-1 adversarial suite: 16 tests added/validated (state boundaries, VPD 0x83, REPORT TPG,
-concurrency, INQUIRY invariants). All 16 PASS. No regressions in engine + iSCSI tests.
-
-[2026-03-03] [DEV] CP5-2 CoW snapshots completed. Fixes applied from review: DeleteSnapshot pauses flusher before
-closing delta; RestoreSnapshot checks PauseAndFlush error + defers Resume; CreateSnapshot holds snapMu across check/insert;
-Delete/Restore use beginOp/endOp; lock order documented (flushMu -> snapMu); non-ASCII punctuation removed; persistSuperblock
-now returns error and callers propagate. All tests passing (known pre-existing flaky
-rebuild_full_extent_midcopy_writes under full-suite load).
-
-[2026-03-03] [TESTER] CP5-2 QA adversarial suite: 22 tests in 5 groups (races, role rejection, edge cases, lifecycle,
-restore correctness) all PASS. Confirms fixes for delete_during_flush_cow, concurrent_create_same_id, and restore path
-nextLSN reset.
-
-[2026-03-03] [DEV] CP5-3 implementation complete. CHAP: ValidateCHAPConfig with ErrCHAPSecretEmpty and CLI guard
-requires -chap-secret when -chap-user is set. Login SecurityNeg echoes AuthMethod=CHAP on second PDU after verify; test
-assertion added. Metrics adapter docs clarify counters count attempts; /metrics inherits admin auth noted in header
-comment. All CP5-3 tests pass; only pre-existing flaky rebuild_catchup_concurrent_writes observed under full suite.
-
-[2026-03-03] [TESTER] CP5-3 QA adversarial: 28 tests added (16 CHAP + 12 resize) all PASS. No new bugs. Full
-regression clean except pre-existing flaky rebuild_catchup_concurrent_writes.
-
-[2026-03-03] [TESTER] Failover latency probe (10 iterations, m01->M02) shows bimodal iSCSI login time dominates pause.
-Promote avg 16ms (8-20ms), FirstIO avg 12ms (6-19ms), login avg 552ms with bimodal split (~130-180ms vs ~1170ms).
-Total avg 588ms, min 99ms, max/P99 1217ms. Conclusion: storage path is fast; pause is iSCSI client reconnect.
-Multipath should keep failover near ~100-200ms; otherwise tune open-iscsi/login timeout and avoid stale portals.
-
-[2026-03-03] [DEV] CP5-4 failure injection + distributed consistency tests implemented. 5 new files:
- `test/fault_test.go` — 7 failure injection tests (F1-F7)
- `test/fault_helpers.go` — netem, iptables, diskfill, WAL corrupt helpers
- `test/consistency_test.go` — 17 distributed consistency tests (C1-C17)
- `test/pgcrash_test.go` — Postgres crash loop (50 iterations, replicated failover)
- `test/pg_helper.go` — Postgres lifecycle helper (initdb, start, stop, pgbench, mount)
-
-Port assignments: iSCSI 3280-3281, admin 8100-8101, replData 9031, replCtrl 9032 (fault/consistency);
-iSCSI 3290-3291, admin 8110-8111, replData 9041, replCtrl 9042 (pgcrash).
-
-[2026-03-03] [TESTER] CP5-4 QA on m01/M02 remote environment. Multiple issues found and fixed:
-
-**BUG-CP54-1: Lease expiry during PgCrashLoop bootstrap** — 30s lease too short for initdb+pgbench
-(which generate hundreds of fsyncs through distributed group commit). Postgres PANIC after exactly 30s.
-Fix: increased bootstrap lease to 600000ms (10min), iteration leases to 120000ms (2min).
-
-**BUG-CP54-2: SCP volume copy auth failure** — pgcrash_test.go hardcoded `id_rsa` SSH key path.
-Fix: use `clientNode.KeyFile` and `*flagSSHUser` for cross-node scp.
-
-**BUG-CP54-3: Replica volume file permission denied** — scp as root created root-owned file,
-but iscsi-target runs as testdev. Fix: added `chown` after scp.
-
-**BUG-CP54-4: C2 EpochMonotonicThreePromotions data mismatch** — dd with `oflag=direct` doesn't
-issue SYNCHRONIZE CACHE, so WAL buffer not fsync'd before kill-9. Data lost on restart.
-Fix: added `conv=fdatasync` to dd writes in C2 test.
-
-**BUG-CP54-5: PG start failure on promoted replica** — WAL shipper degrades under pgbench fdatasync
-pressure (5s barrier timeout too short for burst writes). Promoted replica has incomplete PG data.
-Fix: added `e2fsck -y` before mount in pg_helper.go; made pg start failures non-fatal with
-mkfs+initdb reinit fallback.
-
-**BUG-CP54-6: pgbench_branches relation missing after failover** — Data divergence from degraded
-replication left pgbench database with missing tables. Fix: added dropdb+recreate fallback when
-pgbench init fails.
-
-Final combined run: **25/25 ALL PASS** (994.8s total on m01/M02):
- TestConsistency: 17/17 PASS (194.6s)
- TestFault: 7/7 PASS (75.5s)
- TestPgCrashLoop: PASS — 48/49 recovered, 1 reinit (723.9s)
-
-Known limitation: WAL shipper barrier timeout (5s) causes degradation under heavy fdatasync
-workloads (pgbench). Data divergence occurs on ~50% of failovers without full rebuild between
-role swaps. This is expected behavior — production deployments would use a master-driven rebuild
-after each failover.
-
-[2026-03-03] [TESTER] CP5-4 QA review identified gap: no clean failover test proving PG data
-survives with volume-copy replication. Added `CleanFailoverNoDataLoss` test to pgcrash_test.go:
- Bootstrap 500 rows on primary (no replication — avoids WAL shipper degradation from PG background writes)
- Copy volume to replica, set up replication, verify with lightweight dd write
- Kill primary, promote replica, start PG on promoted replica
- Verify: 500 rows intact, content correct (first="row-1", last="row-500"), post-failover INSERT works
- Proves full stack: PG → ext4 → iSCSI → BlockVol → volume copy → failover → WAL recovery → ext4 → PG recovery
-
-Design note: PG cannot run under active replication without degrading the WAL shipper (background
-checkpointer/WAL writer generate continuous iSCSI writes that hit 5s barrier timeout). The test
-separates data creation (bootstrap without replication) from replication verification (dd only).
-
-Final combined run with CleanFailoverNoDataLoss: **26/26 ALL PASS** (1067.7s total on m01/M02):
- TestConsistency: 17/17 PASS (194.7s)
- TestFault: 7/7 PASS (75.6s)
- TestPgCrashLoop/CleanFailoverNoDataLoss: PASS (90.3s)
- TestPgCrashLoop/ReplicatedFailover50: PASS — 48/49 recovered, 1 reinit (706.3s)
@@ -1,80 +0,0 @@
-# Phase 5 Progress
-
-## Status
- CP5-1 through CP5-4 complete. Phase 5 DONE.
-
-## Completed
- CP5-1: ALUA implicit support, REPORT TARGET PORT GROUPS, VPD 0x83 descriptors, write fencing on standby.
- CP5-1: Multipath config + setup script, 4 multipath integration tests.
- CP5-1: Reviewer fixes (RoleNone write regression, T_SUP flag, TPG ID validation, ASCII log).
- CP5-1: 10 ALUA unit tests + 16 adversarial tests (all PASS).
- CP5-2: CoW snapshots implemented with flusher-based CoW, delta files, and recovery.
- CP5-2: Review fixes applied (PauseAndFlush safety, snapMu race fix, beginOp/endOp, lock order doc, error propagation).
- CP5-2: 10 unit tests + 22 adversarial tests (all PASS).
- CP5-3: CHAP auth, online resize, Prometheus metrics, admin endpoints.
- CP5-3: Review fixes applied (empty secret validation, AuthMethod echo, docs).
- CP5-3: 12 dev tests + 28 QA adversarial tests (all PASS).
- CP5-4: Failure injection (7 tests) + distributed consistency (17 tests) + Postgres crash loop (50 iters).
- CP5-4: 6 bugs found and fixed (lease expiry, scp auth, permissions, fdatasync, pg reinit, pgbench tables).
- CP5-4: 26/26 tests ALL PASS on m01/M02 remote environment (1067.7s combined).
- CP5-4: Added CleanFailoverNoDataLoss (500 PG rows survive failover via volume copy).
-
-## In Progress
- None.
-
-## Blockers
- None.
-
-## Next Steps
- Phase 5 complete. Ready for Phase 6 (NVMe-oF) or other priorities.
-
-## Notes
- SCSI test count: 53 (12 ALUA). Integration multipath tests require multipath-tools + sg3_utils.
- Known flaky: rebuild_full_extent_midcopy_writes under full-suite CPU contention (pre-existing).
- Known flaky: rebuild_catchup_concurrent_writes (WAL_RECYCLED timing, pre-existing).
- Known limitation: WAL shipper barrier timeout (5s) causes degradation under heavy fdatasync
-  workloads. PgCrashLoop shows ~50% data divergence per failover without full rebuild. Expected
-  behavior — production would use master-driven rebuild after each failover.
- Failover latency probe (10 iters): promote+first I/O ~30ms; total pause dominated by iSCSI
-  login (avg 552ms, bimodal 130-180ms vs ~1170ms). Multipath should keep pause near 100-200ms;
-  otherwise tune open-iscsi login timeout and avoid stale portals.
-
-## CP5-4 Test Catalog
-
-### Failure Injection (`test/fault_test.go`)
-| ID | Test | What it proves |
-|----|------|----------------|
-| F1 | PowerLossDuringFio | fdatasync'd data survives kill-9 + failover |
-| F2 | DiskFullENOSPC | reads survive ENOSPC, writes recover after space freed |
-| F3 | WALCorruption | WAL recovery discards corrupted tail, early data intact |
-| F4 | ReplicaDownDuringWrites | primary keeps serving after replica crash mid-write |
-| F5 | SlowNetworkBarrierTimeout | writes continue under 200ms netem delay (remote only) |
-| F6 | NetworkPartitionSelfFence | primary self-fences on iptables partition (remote only) |
-| F7 | SnapshotDuringFailover | snapshot + replication interaction, both patterns survive |
-
-### Distributed Consistency (`test/consistency_test.go`)
-| ID | Test | What it proves |
-|----|------|----------------|
-| C1 | EpochPersistedOnPromotion | epoch survives kill-9 + restart (superblock persistence) |
-| C2 | EpochMonotonicThreePromotions | 3 failovers, epoch 1→2→3, data from all phases intact |
-| C3 | StaleEpochWALRejected | replica at epoch=2 rejects WAL entries from epoch=1 |
-| C4 | LeaseExpiredWriteRejected | writes fail after lease expiry |
-| C5 | LeaseRenewalUnderJitter | lease survives 100ms netem jitter with 30s TTL (remote) |
-| C6 | PromotionDataIntegrityChecksum | 10MB byte-for-byte match after failover |
-| C7 | PromotionPostgresRecovery | postgres recovers from crash (single-node, no repl) |
-| C8 | DeadZoneNoWrites | fencing gap verified between old/new primary |
-| C9 | RebuildWALCatchup | WAL catch-up rebuild after brief replica outage |
-| C10 | RebuildFullExtent | full extent rebuild after heavy writes |
-| C11 | RebuildDuringActiveWrites | fio uninterrupted during rebuild |
-| C12 | GracefulDemoteNoDataLoss | data intact after demote + re-promote |
-| C13 | RapidRoleFlip10x | 10 rapid epoch bumps, no crash or panic |
-| C14 | LeaseTimerRealExpiry | lease transitions true→false at ~5s mark |
-| C15 | DistGroupCommitEndToEnd | replica WAL advances during fdatasync fio |
-| C16 | DistGroupCommitReplicaCrash | primary continues in degraded mode |
-| C17 | DistGroupCommitBarrierVerify | replica LSN >= primary after fdatasync |
-
-### Postgres Crash Loop (`test/pgcrash_test.go`)
-| ID | Test | What it proves |
-|----|------|----------------|
-| PG1 | CleanFailoverNoDataLoss | 500 PG rows survive volume-copy failover, content verified |
-| PG2 | ReplicatedFailover50 | 49 kill→promote→recover→pgbench cycles, PG recovers |
@@ -1,202 +0,0 @@
-# Phase 6 Dev Log
-
-Append-only communication between agents. Newest entries at bottom.
-Each entry: `[date] [role] message`
-
-Roles: `DEV`, `REVIEWER`, `TESTER`, `ARCHITECT`
-
---
-
-[2026-03-03] [DEV] CP6-1 implementation complete. 7 tasks (Task 0-6) delivered:
- Task 0: Extracted BlockVolAdapter to shared `blockvol/adapter.go` (exported fields, ALUA methods,
-  RoleToALUA/UUIDToNAA helpers). Added DisconnectVolume(iqn) to TargetServer (kills sessions + removes
-  volume). Added Session.TargetIQN() accessor. Updated standalone binary to use shared adapter. 3 adapter tests.
- Task 1: VolumeManager -- multi-volume BlockVol + shared TargetServer lifecycle. sync.Once start guard,
-  atomic ready flag, IQN sanitization with hash suffix for collision avoidance. 10 tests.
- Task 2: CSI Identity service (GetPluginInfo, GetPluginCapabilities, Probe). 3 tests.
- Task 3: CSI Controller service (CreateVolume with idempotency + size mismatch, DeleteVolume). 4 tests.
- Task 4: CSI Node service (Stage/Unstage/Publish/Unpublish) with ISCSIUtil/MountUtil interfaces. 7 tests.
- Task 5: gRPC server + binary entry point (unix/tcp socket, signal handler, graceful shutdown).
- Task 6: K8s manifests (DaemonSet, StorageClass, RBAC, example PVC) + smoke-test.sh.
-Total: 12 new Go files, 2 modified, 4 YAML, 1 shell script, 25+3=28 tests. CSI spec v1.10.0 added.
-
-[2026-03-03] [REVIEWER] CP6-1 review returned 5 findings:
-1. (High) CreateVolume not idempotent after restart -- only checks in-memory map, misses existing .blk files.
-2. (Medium) NodePublishVolume doesn't validate empty StagingTargetPath.
-3. (Medium) NodeStageVolume resource leak -- OpenVolume not cleaned up on discovery/login/mount failure.
-4. (Medium) Target start race -- ListenAndServe in goroutine, ready=true set before bind confirmed.
-5. (Low) IQN collision -- truncation without hash suffix causes identical IQNs for long names.
-Open Q1: How should CreateVolume handle pre-existing .blk files on disk?
-Open Q2: What happens in NodeUnstageVolume if unmount succeeds but logout fails?
-
-[2026-03-03] [DEV] All 5 review findings + 2 open questions resolved:
- Finding 1: CreateVolume now checks os.Stat for existing .blk files, adopts via OpenBlockVol.
-  Added ErrVolumeSizeMismatch. Controller maps it to codes.AlreadyExists.
- Finding 2: Added stagingPath=="" check in NodePublishVolume returning InvalidArgument.
- Finding 3: Added success flag + deferred CloseVolume after OpenVolume in NodeStageVolume.
- Finding 4: Listener created synchronously via net.Listen before ready=true. Serve in goroutine.
- Finding 5: SanitizeIQN appends SHA256 hash suffix (8 hex chars) when truncating to 64.
- Open Q1: Pre-existing files adopted as idempotent success if size >= requested.
- Open Q2: NodeUnstageVolume uses best-effort cleanup (firstErr pattern), always attempts CloseVolume.
-3 new tests: CreateIdempotentAfterRestart, IQNCollision, StageLoginFailureCleanup, PublishMissingStagingPath.
-All 25 CSI tests + full regression PASS.
-
-[2026-03-03] [TESTER] CP6-1 QA adversarial suite: 30 tests in qa_csi_test.go. 26 PASS, 4 FAIL confirming 5 bugs.
-Groups: QA-VM (8), QA-CTRL (5), QA-NODE (7), QA-SRV (3), QA-ID (1), QA-IQN (5), QA-X (1).
-Bugs: BUG-QA-1 snapshot leak, BUG-QA-2/3 sync.Once restart, BUG-QA-4 LimitBytes ignored, BUG-QA-5 case divergence.
-
-[2026-03-03] [DEV] All 5 QA bugs fixed:
- BUG-QA-1: DeleteVolume now globs+removes volPath+".snap.*" (both tracked and untracked paths).
- BUG-QA-2+3: Replaced sync.Once+atomic.Bool with managerState enum (stopped/starting/ready/failed).
-  Start() retryable after failure or Stop(). Stop() sets state=stopped, nils target.
-  Goroutine captures target locally before launch (prevents nil deref after Stop).
- BUG-QA-4: Controller CreateVolume validates LimitBytes. When RequiredBytes=0 and LimitBytes set,
-  uses LimitBytes as target size. Rejects RequiredBytes > LimitBytes and post-rounding overflow.
- BUG-QA-5: sanitizeFilename now lowercases (matching SanitizeIQN). "VolA" and "vola" produce
-  same file and same IQN — treated as same volume via file adoption path.
- QA-CTRL-4 test updated from bug-detection to behavior-documentation (NotFound is by design;
-  volumes re-tracked via CreateVolume after restart).
-All 54 CSI tests + full regression PASS (blockvol 63s, iscsi 2.3s, csi 0.4s).
-
-[2026-03-03] [DEV] CP6-2 complete. See separate CP6-2 entries in progress.md.
-
-[2026-03-04] [TESTER] CSI Testing Ladder Levels 2-4 complete on M02 (192.168.1.184):
-
-**Level 2: csi-sanity gRPC Conformance**
- cross-compiled block-csi (linux/amd64), installed csi-sanity on M02
- Result: 33 Passed, 0 Failed, 58 Skipped (optional RPCs), 1 Pending
- 6 bugs found and fixed: empty VolumeCapabilities validation (3 RPCs), bind mount for NodePublish,
-  target path removal in NodeUnpublish, IsMounted check before unmount
- All 226 unit tests updated with VolumeCapabilities/VolumeCapability in requests
-
-**Level 3: Integration Smoke**
- Verified via csi-sanity's "should work" tests exercising real iSCSI on M02
- 489 real SCSI commands processed (READ_10, WRITE_10, SYNC_CACHE, INQUIRY, etc.)
- Full lifecycle: Create → Stage (discovery+login+mkfs+mount) → Publish → Unpublish → Unstage (unmount+logout) → Delete
- Clean state: no leftover sessions, mounts, or volume files
-
-**Level 4: k3s PVC→Pod**
- Installed k3s v1.34.4 on M02, deployed CSI DaemonSet (block-csi + csi-provisioner + registrar)
- DaemonSet uses nsenter wrappers for host iscsiadm/mount/umount/blkid/mountpoint/mkfs.ext4
- Test: PVC (100Mi) → Pod writes "hello sw-block" → md5 7be761488cf480c966077c7aca4ea3ed
-  → Pod deleted → PVC retained → New pod reads same data → PASS
- 1 additional bug: IsLoggedIn didn't handle iscsiadm exit code 21 (nsenter suppresses output)
-  → Fixed by checking ExitError.ExitCode() == 21 directly
-
-Code changes from Levels 2-4:
- controller.go: +VolumeCapabilities validation in CreateVolume, ValidateVolumeCapabilities
- node.go: +VolumeCapability nil check, BindMount for publish, IsMounted+RemoveAll in unpublish
- iscsi_util.go: +BindMount interface+impl (real+mock), IsLoggedIn exit code 21 handling
- controller_test.go, node_test.go, qa_csi_test.go, qa_cp62_test.go: testVolCaps()/testVolCap() helpers
-
-[2026-03-04] [DEV] CP6-3 Review 1+2 findings fixed (12 total, 5 High, 5 Medium, 2 Low):
- R1-1 (High): AllocateBlockVolume now returns ReplicaDataAddr/CtrlAddr/RebuildListenAddr from ReplicationPorts().
- R1-2 (High): setupPrimaryReplication now calls vol.StartRebuildServer(rebuildAddr) with deterministic port.
- R1-3 (High): VS sends periodic full block heartbeat (5×sleepInterval) enabling assignment confirmation.
- R2-F1 (High): LastLeaseGrant moved to entry initializer before Register (was after → stale-lease race).
- R1-4 (Medium): BlockService.CollectBlockVolumeHeartbeat fills ReplicaDataAddr/CtrlAddr from replStates.
- R1-5 (Medium): UpdateFullHeartbeat refreshes LastLeaseGrant on every heartbeat.
- R2-F2 (Medium): Deferred promotion timers stored and cancelled on VS reconnect (prevents split-brain).
- R2-F3 (Medium): SwapPrimaryReplica uses blockvol.RoleToWire(blockvol.RolePrimary) instead of uint32(1).
- R2-F4 (Medium): DeleteBlockVolume now deletes replica (best-effort, non-fatal).
- R2-F5 (Medium): SwapPrimaryReplica computes epoch+1 atomically inside lock, returns newEpoch.
- R2-F6 (Low): Removed redundant string(server) casts.
- R2-F7 (Low): Documented rebuild feedback as future work.
-All 293 tests PASS: blockvol (24s), csi (1.6s), iscsi (2.6s), server (3.3s).
-
-[2026-03-04] [DEV] CP6-3 implementation complete. 8 tasks (Task 0-7) delivered:
- Task 0: Proto extension — replica/rebuild address fields in master.proto, volume_server.proto,
-  generated pb.go files, wire types, converters. AssignmentsToProto batch helper. 8 tests.
- Task 1: Assignment queue — BlockAssignmentQueue with retain-until-confirmed (F1).
-  Enqueue/Peek/Confirm/ConfirmFromHeartbeat. Stale epoch pruning. Wired into HeartbeatResponse. 11 tests.
- Task 2: VS assignment receiver — extracts block_volume_assignments from HeartbeatResponse,
-  calls BlockService.ProcessAssignments.
- Task 3: BlockService replication — ProcessAssignments dispatches HandleAssignment +
-  setupPrimaryReplication/setupReplicaReceiver/startRebuild. Deterministic ports via FNV hash (F3).
-  Heartbeat reports replica addresses (F5). 9 tests.
- Task 4: Registry replica + CreateVolume — SetReplica/ClearReplica/SwapPrimaryReplica.
-  CreateBlockVolume creates primary + replica, enqueues assignments. Single-copy mode (F4). 10 tests.
- Task 5: Failover — failoverBlockVolumes on VS disconnect. Lease-aware promotion (F2):
-  promote only after lease expires, deferred via time.AfterFunc. SwapPrimaryReplica + epoch bump.
-  11 failover tests.
- Task 6: ControllerPublish — ControllerPublishVolume returns fresh primary address via LookupVolume.
-  ControllerUnpublishVolume no-op. PUBLISH_UNPUBLISH_VOLUME capability. NodeStageVolume prefers
-  publish_context over volume_context. 8 tests.
- Task 7: Rebuild on recovery — recoverBlockVolumes on VS reconnect drains pendingRebuilds,
-  enqueues Rebuilding assignments. 10 tests (shared file with Task 5).
-Total: 4 new files, ~15 modified, 67 new tests. All 5 review findings (F1-F5) addressed.
-All tests PASS: blockvol (43s), csi (1.4s), iscsi (2.5s), server (3.2s).
-Cumulative Phase 6: 293 tests.
-
-[2026-03-04] [TESTER] CP6-3 QA adversarial suite: 48 tests in qa_block_cp63_test.go. 47 PASS, 1 FAIL confirming 1 bug.
-Groups: QA-Queue (8), QA-Reg (7), QA-Failover (7), QA-Create (5), QA-Rebuild (3), QA-Integration (2), QA-Edge (5), QA-Master (5), QA-VS (6).
-
-**BUG-QA-CP63-1 (Medium): `SetReplica` leaks old replica server in `byServer` index.**
- When calling `SetReplica("vol1", "vs3", ...)` on a volume whose replica was previously `vs2`,
-  `vs2` remains in the `byServer` index. `ListByServer("vs2")` still returns `vol1`.
- Impact: `PickServer` over-counts old replica server's volume count (wrong placement).
-  Failover could trigger on stale index entries.
- Fix: Added `removeFromServer(oldReplicaServer, name)` before setting new replica in `SetReplica()`.
- File: `master_block_registry.go:285` (3 lines added).
- Test: `TestQA_Reg_SetReplicaTwice_ReplacesOld`.
-
-All 48 QA tests + full regression PASS: blockvol (23s), csi (1.1s), iscsi (2.5s), server (4.8s).
-Cumulative Phase 6: 293 + 48 = 341 tests.
-
-[2026-03-04] [TESTER] CP6-3 integration tests: 8 tests in integration_block_test.go. All 8 PASS.
-
-**Required Tests:**
-1. `TestIntegration_FailoverCSIPublish` — Create replicated vol → kill primary → verify
-   LookupBlockVolume (CSI ControllerPublishVolume path) returns promoted replica's iSCSI addr.
-2. `TestIntegration_RebuildOnRecovery` — Failover → reconnect old primary → verify Rebuilding
-   assignment enqueued with correct epoch → confirm via heartbeat.
-3. `TestIntegration_AssignmentDeliveryConfirmation` — Create replicated vol → verify pending
-   assignments → wrong epoch doesn't confirm → correct heartbeat confirms → queue cleared.
-
-**Nice-to-have Tests:**
-4. `TestIntegration_LeaseAwarePromotion` — Lease not expired → promotion deferred → after TTL → promoted.
-5. `TestIntegration_ReplicaFailureSingleCopy` — Replica alloc fails → single-copy mode → no replica
-   assignments → failover is no-op (no replica to promote).
-6. `TestIntegration_TransientDisconnectNoSplitBrain` — VS disconnects with active lease → deferred
-   timer → VS reconnects → timer cancelled → no promotion (split-brain prevented).
-
-**Extra coverage:**
-7. `TestIntegration_FullLifecycle` — Create → publish → confirm assignments → failover → re-publish
-   → confirm → recover → rebuild → confirm → delete. Full 11-phase lifecycle.
-8. `TestIntegration_DoubleFailover` — Primary dies → promoted → promoted replica also dies → original
-   server re-promoted (epoch=3).
-9. `TestIntegration_MultiVolumeFailoverRebuild` — 3 volumes across 2 servers → kill one server → all
-   primaries promoted → reconnect → rebuild assignments for each.
-
-All 349 server+QA+integration tests PASS (6.8s).
-Cumulative Phase 6: 293 + 48 + 8 = 349 tests.
-
-[2026-03-05] [TESTER] CP6-3 real integration tests on M02 (192.168.1.184): 3 tests, all PASS.
-
-**Bug found during testing: RoleNone → RoleRebuilding transition not allowed.**
- After VS restart, volume is RoleNone. Master sends Rebuilding assignment, but both
-  `validTransitions` (role.go) and `HandleAssignment` (promotion.go) rejected this path.
- Fix: Added `RoleRebuilding: true` to `validTransitions[RoleNone]` in role.go.
-  Added `RoleNone → RoleRebuilding` case in HandleAssignment (promotion.go) with
-  SetEpoch + SetMasterEpoch + SetRole.
- Infrastructure: Added `action:"connect"` to admin.go `/rebuild` endpoint to start
-  rebuild client (calls `blockvol.StartRebuild` in background goroutine).
-  Added `StartRebuildClient` method to ha_target.go.
-
-**Tests (cp63_test.go, `//go:build integration`):**
-1. `FailoverCSIAddressSwitch` (3.2s) — Write data A → kill primary → promote replica
-   → client re-discovers at new iSCSI address → verify data A → write data B →
-   verify A+B. Simulates CSI ControllerPublishVolume address-switch flow.
-2. `RebuildDataConsistency` (5.3s) — Write A (replicated) → kill replica → write B
-   (missed) → restart replica as Rebuilding → start rebuild server on primary →
-   connect rebuild client → wait for role→replica → kill primary → promote rebuilt
-   replica → verify A+B intact. Full end-to-end rebuild with data verification.
-3. `FullLifecycleFailoverRebuild` (6.4s) — Write A → kill primary → promote replica
-   → write B → start rebuild server → restart old primary as Rebuilding → rebuild
-   → write C → kill new primary → promote rebuilt old-primary → verify A+B intact.
-   11-phase lifecycle simulating master's failover→recoverBlockVolumes→rebuild flow.
-
-Existing 7 HA tests: all PASS (no regression). Total real integration: 10 tests on M02.
-Code changes: role.go (+1 line), promotion.go (+7 lines), admin.go (+15 lines),
-ha_target.go (+20 lines), cp63_test.go (new, ~350 lines).
-
@@ -1,526 +0,0 @@
-# Phase 6 Progress
-
-## Status
- CP6-1 complete. 54 CSI tests (25 dev + 30 QA - 1 removed).
- CP6-2 complete. 172 CP6-2 tests (118 dev/review + 54 QA). 1 QA bug found and fixed.
- **Phase 6 cumulative: 226 tests, all PASS.**
-
-## Completed
- CP6-1 Task 0: Extracted BlockVolAdapter to shared `blockvol/adapter.go`, added DisconnectVolume to TargetServer, added Session.TargetIQN().
- CP6-1 Task 1: VolumeManager (multi-volume BlockVol + shared TargetServer lifecycle). 10 tests.
- CP6-1 Task 2: CSI Identity service (GetPluginInfo, GetPluginCapabilities, Probe). 3 tests.
- CP6-1 Task 3: CSI Controller service (CreateVolume, DeleteVolume, ValidateVolumeCapabilities). 4 tests.
- CP6-1 Task 4: CSI Node service (NodeStageVolume, NodeUnstageVolume, NodePublishVolume, NodeUnpublishVolume). 7 tests.
- CP6-1 Task 5: gRPC server + binary entry point (`csi/cmd/block-csi/main.go`).
- CP6-1 Task 6: K8s manifests (DaemonSet, StorageClass, RBAC, example PVC) + smoke-test.sh.
- CP6-1 Review fixes: 5 findings + 2 open questions resolved, 3 new tests added.
-  - Finding 1: CreateVolume idempotency after restart (adopts existing .blk files on disk).
-  - Finding 2: NodePublishVolume validates empty StagingTargetPath.
-  - Finding 3: Resource leak cleanup on error paths (success flag + deferred CloseVolume).
-  - Finding 4: Synchronous listener creation (bind errors surface immediately).
-  - Finding 5: IQN collision avoidance (SHA256 hash suffix on truncation).
-
- CP6-1 QA adversarial: 30 tests in qa_csi_test.go. 5 bugs found and fixed:
-  - BUG-QA-1 (Medium): DeleteVolume leaked .snap.* delta files. Fixed: glob+remove snapshot files.
-  - BUG-QA-2 (High): Start not retryable after failure (sync.Once). Fixed: state machine.
-  - BUG-QA-3 (High): Stop then Start broken (sync.Once already fired). Fixed: same state machine.
-  - BUG-QA-4 (Low): CreateVolume ignored LimitBytes. Fixed: validate and cap size.
-  - BUG-QA-5 (Medium): sanitizeFilename case divergence with SanitizeIQN. Fixed: lowercase both.
-  - Additional: goroutine captured m.target by reference (nil after Stop). Fixed: local capture.
-
- CP6-2 complete. All 7 tasks done. 63 CSI tests + 48 server block tests = 111 CP6-2 tests, all PASS.
-
-## CP6-2: Control-Plane Integration
-
-### Completed Tasks
-
- **Task 0: Proto Extension + Code Generation** — block volume messages in master.proto/volume_server.proto, Go stubs regenerated, conversion helpers + 5 tests.
- **Task 1: Master Block Volume Registry** — in-memory registry with Pending→Active status tracking, full/delta heartbeat reconciliation, per-name inflight lock (TOCTOU prevention), placement (fewest volumes), block-capable server tracking. 11 tests.
- **Task 2: Volume Server Block Volume gRPC** — AllocateBlockVolume/DeleteBlockVolume gRPC handlers on VolumeServer, CreateBlockVol/DeleteBlockVol on BlockService, shared naming (blockvol/naming.go). 5 tests.
- **Task 3: Master Block Volume RPC Handlers** — CreateBlockVolume (idempotent, inflight lock, retry up to 3 servers), DeleteBlockVolume (idempotent), LookupBlockVolume. Mock VS call injection for testability. 9 tests.
- **Task 4: Heartbeat Wiring** — block volume fields in heartbeat stream, volume server sends initial full heartbeat + deltas, master processes via UpdateFullHeartbeat/UpdateDeltaHeartbeat.
- **Task 5: CSI Controller Refactor** — VolumeBackend interface (LocalVolumeBackend + MasterVolumeClient), controller uses backend instead of VolumeManager, returns volume_context with iscsiAddr+iqn, mode flag (controller/node/all). 5 backend tests.
- **Task 6: CSI Node Refactor + K8s Manifests** — Node reads volume_context for remote targets, staged volume tracking with IQN derivation fallback on restart, split K8s manifests (csi-driver.yaml, csi-controller.yaml Deployment, csi-node.yaml DaemonSet). 4 new node tests (11 total).
-
-### New Files (CP6-2)
-| File | Description |
-|------|-------------|
-| `blockvol/naming.go` | Shared SanitizeIQN + SanitizeFilename |
-| `blockvol/naming_test.go` | 4 naming tests |
-| `blockvol/block_heartbeat_proto.go` | Go wire type ↔ proto conversion |
-| `blockvol/block_heartbeat_proto_test.go` | 5 conversion tests |
-| `server/master_block_registry.go` | Block volume registry + placement |
-| `server/master_block_registry_test.go` | 11 registry tests |
-| `server/volume_grpc_block.go` | VS block volume gRPC handlers |
-| `server/volume_grpc_block_test.go` | 5 VS tests |
-| `server/master_grpc_server_block.go` | Master block volume RPC handlers |
-| `server/master_grpc_server_block_test.go` | 9 master handler tests |
-| `csi/volume_backend.go` | VolumeBackend interface + clients |
-| `csi/volume_backend_test.go` | 5 backend tests |
-| `csi/deploy/csi-controller.yaml` | Controller Deployment manifest |
-| `csi/deploy/csi-node.yaml` | Node DaemonSet manifest |
-
-### Modified Files (CP6-2)
-| File | Changes |
-|------|---------|
-| `pb/master.proto` | Block volume messages, Heartbeat fields 24-27, RPCs |
-| `pb/volume_server.proto` | AllocateBlockVolume, VolumeServerDeleteBlockVolume |
-| `server/master_server.go` | BlockVolumeRegistry + VS call fields |
-| `server/master_grpc_server.go` | Block volume heartbeat processing |
-| `server/volume_grpc_client_to_master.go` | Block volume in heartbeat stream |
-| `server/volume_server_block.go` | CreateBlockVol/DeleteBlockVol on BlockService |
-| `csi/controller.go` | VolumeBackend instead of VolumeManager |
-| `csi/controller_test.go` | Updated for VolumeBackend |
-| `csi/node.go` | Remote target support + staged volume tracking |
-| `csi/node_test.go` | 4 new remote target tests |
-| `csi/server.go` | Mode flag, MasterAddr, VolumeBackend config |
-| `csi/cmd/block-csi/main.go` | --master, --mode flags |
-| `csi/deploy/csi-driver.yaml` | CSIDriver object only (split out workloads) |
-| `csi/qa_csi_test.go` | Updated for VolumeBackend |
-
-### CP6-2 Review Fixes
-All findings from both reviewers addressed. 4 new tests added (118 total CP6-2 tests).
-
-| # | Finding | Severity | Fix |
-|---|---------|----------|-----|
-| R1-F1 | DeleteBlockVol doesn't terminate active sessions | High | Use DisconnectVolume instead of RemoveVolume |
-| R1-F2 | Block registry server list never pruned | Medium | UnmarkBlockCapable on VS disconnect in SendHeartbeat defer |
-| R1-F3 | Block volume status never updates after create | Medium | Mark StatusActive immediately after successful VS allocate |
-| R1-F4 | IQN generation on startup scan doesn't sanitize | Low | Apply blockvol.SanitizeIQN(name) in scan path |
-| R1-F5/R2-F3 | CreateBlockVol idempotent path skips TargetServer | Medium | Re-add adapter to TargetServer on idempotent path |
-| R2-F1 | UpdateFullHeartbeat doesn't update SizeBytes | Low | Copy info.VolumeSize to existing.SizeBytes |
-| R2-F2 | inflightEntry.done channel is dead code | Low | Removed done channel, simplified to empty struct |
-| R2-F4 | CreateBlockVolume idempotent check doesn't validate size | Medium | Return error if existing size < requested size |
-| R2-F5 | Full + delta heartbeat can fire on same message | Low | Changed second `if` to `else if` + comment |
-| R2-F6 | NodeUnstageVolume deletes staged entry before cleanup | Medium | Delete from staged map only after successful cleanup |
-
-New tests: TestMaster_CreateIdempotentSizeMismatch, TestRegistry_UnmarkDeadServer, TestRegistry_FullHeartbeatUpdatesSizeBytes, TestNode_UnstageRetryKeepsStagedEntry.
-
-### CP6-2 QA Adversarial Tests
-54 tests across 2 files. 1 bug found and fixed.
-
-| File | Tests | Areas |
-|------|-------|-------|
-| `server/qa_block_cp62_test.go` | 22 | Registry (8), Master RPCs (8), VS BlockService (6) |
-| `csi/qa_cp62_test.go` | 32 | Node remote (6), Controller backend (5), Backend (2), Naming (2), Lifecycle (4), Server/Driver (2), VolumeManager (4), Edge cases (7) |
-
-**BUG-QA-CP62-1 (Medium): `NewCSIDriver` accepts invalid mode strings.**
- `NewCSIDriver(DriverConfig{Mode: "invalid"})` returns nil error. Driver runs with only identity server — no controller, no node. K8s reports capabilities but all operations fail `Unimplemented`.
- Fix: Added `switch` validation after mode defaulting. Returns `"csi: invalid mode %q, must be controller/node/all"`.
- Test: `TestQA_ModeInvalid`.
-
-**Final CP6-2 test count: 118 dev/review + 54 QA = 172 CP6-2 tests, all PASS.**
-
-**Cumulative Phase 6 test count: 54 CP6-1 + 172 CP6-2 = 226 tests.**
-
-## CSI Testing Ladder
-
-| Level | What | Tools | Status |
-|-------|------|-------|--------|
-| 1. Unit tests | Mock iscsiadm/mount. Confirm idempotency, error handling, edge cases. | `go test` | DONE (226 tests) |
-| 2. gRPC conformance | `csi-sanity` tool validates all CSI RPCs against spec. No K8s needed. | [csi-sanity](https://github.com/kubernetes-csi/csi-test) | DONE (33 pass, 58 skip) |
-| 3. Integration smoke | Full iSCSI lifecycle with real filesystem (via csi-sanity "should work" tests). | csi-sanity + iscsiadm | DONE (489 SCSI cmds) |
-| 4. Single-node K8s (k3s) | Deploy CSI DaemonSet on k3s. PVC → Pod → write data → delete/recreate → verify persistence. | k3s v1.34.4 | DONE |
-| 5. Failure/chaos | Kill CSI controller pod; ensure no IO outage for existing volumes. Node restart with staged volumes. | chaos-mesh or manual | TODO |
-| 6. K8s E2E suite | SIG-Storage tests validate provisioning, attach/detach, resize, snapshots. | `e2e.test` binary | TODO |
-
-### Level 2: csi-sanity Conformance (M02)
-
-**Result: 33 Passed, 0 Failed, 58 Skipped, 1 Pending.**
-
-Run on M02 (192.168.1.184) with block-csi in local mode. Used helper scripts for staging/target path management.
-
-Bugs found and fixed during csi-sanity:
-| # | Bug | Severity | Fix |
-|---|-----|----------|-----|
-| BUG-SANITY-1 | CreateVolume accepted empty VolumeCapabilities | Medium | Added `len(req.VolumeCapabilities) == 0` check |
-| BUG-SANITY-2 | ValidateVolumeCapabilities accepted empty VolumeCapabilities | Medium | Same check added |
-| BUG-SANITY-3 | NodeStageVolume accepted nil VolumeCapability | Medium | Added nil check |
-| BUG-SANITY-4 | NodePublishVolume used `mount -t ext4` instead of bind mount | High | Added BindMount method to MountUtil interface |
-| BUG-SANITY-5 | NodeUnpublishVolume didn't remove target path | Medium | Added os.RemoveAll per CSI spec |
-| BUG-SANITY-6 | NodeUnpublishVolume failed on unmounted path | Medium | Added IsMounted check before unmount |
-
-All existing unit tests updated with VolumeCapabilities/VolumeCapability in test requests.
-
-### Level 3: Integration Smoke (M02)
-
-Verified through csi-sanity's full lifecycle tests which exercised real iSCSI:
- 489 real SCSI commands processed (READ_10, WRITE_10, SYNC_CACHE, INQUIRY, etc.)
- Full cycle: CreateVolume → NodeStageVolume (iSCSI login + mkfs.ext4 + mount) → NodePublishVolume → NodeUnpublishVolume → NodeUnstageVolume (unmount + iSCSI logout) → DeleteVolume
- Clean state verified: no leftover iSCSI sessions, mounts, or volume files
-
-### Level 4: k3s PVC→Pod (M02)
-
-**Result: PASS — data persists across pod deletion/recreation.**
-
-k3s v1.34.4 single-node on M02. CSI deployed as DaemonSet with 3 containers:
-1. block-csi (privileged, nsenter wrappers for host iscsiadm/mount/umount/mkfs/blkid/mountpoint)
-2. csi-provisioner (v5.1.0, --node-deployment for single-node)
-3. csi-node-driver-registrar (v2.12.0)
-
-Test sequence:
-1. Created PVC (100Mi, sw-block StorageClass) → Bound
-2. Created pod → wrote "hello sw-block" to /data/test.txt → md5: `7be761488cf480c966077c7aca4ea3ed`
-3. Deleted pod (PVC retained) → iSCSI session cleanly closed
-4. Recreated pod with same PVC → read "hello sw-block" → same md5 verified
-5. Appended "persistence works!" → confirmed read-write
-
-Additional bug fixed during k3s testing:
-| # | Bug | Severity | Fix |
-|---|-----|----------|-----|
-| BUG-K3S-1 | IsLoggedIn didn't handle iscsiadm exit code 21 (nsenter suppresses output) | Medium | Added `exitErr.ExitCode() == 21` check |
-
-DaemonSet manifest: `learn/projects/sw-block/test/csi-k3s-node.yaml`
-
- CP6-3 complete. 67 CP6-3 tests. All PASS.
-
-## CP6-3: Failover + Rebuild in Kubernetes
-
-### Completed Tasks
-
- **Task 0: Proto Extension + Wire Type Updates** — Added replica_data_addr, replica_ctrl_addr to BlockVolumeInfoMessage/BlockVolumeAssignment; rebuild_addr to BlockVolumeAssignment; replica_server to Create/LookupBlockVolumeResponse; replica fields to AllocateBlockVolumeResponse. Updated wire types and converters. 8 tests.
- **Task 1: Master Assignment Queue + Delivery** — BlockAssignmentQueue with Enqueue/Peek/Confirm/ConfirmFromHeartbeat. Retain-until-confirmed pattern (F1): assignments resent on every heartbeat until VS confirms via matching (path, epoch, role). Stale epoch pruning during Peek. Wired into HeartbeatResponse delivery. 11 tests.
- **Task 2: VS Assignment Receiver Wiring** — VS extracts block_volume_assignments from HeartbeatResponse and calls BlockService.ProcessAssignments.
- **Task 3: BlockService Replication Support** — ProcessAssignments dispatches to HandleAssignment + setupPrimaryReplication/setupReplicaReceiver/startRebuild per role. ReplicationPorts deterministic hash (F3). Heartbeat reports replica addresses (F5). 9 tests.
- **Task 4: Registry Replica Tracking + CreateVolume** — Added SetReplica/ClearReplica/SwapPrimaryReplica to registry. CreateBlockVolume creates on 2 servers (primary + replica), enqueues assignments. Single-copy mode if only 1 server or replica fails (F4). LookupBlockVolume returns ReplicaServer. 10 tests.
- **Task 5: Master Failover Detection** — failoverBlockVolumes on VS disconnect. Lease-aware promotion (F2): promote only after LastLeaseGrant + LeaseTTL expires. Deferred promotion via time.AfterFunc for unexpired leases. promoteReplica swaps primary/replica, bumps epoch, enqueues new primary assignment. 11 tests.
- **Task 6: ControllerPublishVolume/UnpublishVolume** — ControllerPublishVolume calls backend.LookupVolume, returns publish_context{iscsiAddr, iqn}. ControllerUnpublishVolume is no-op. Added PUBLISH_UNPUBLISH_VOLUME capability. NodeStageVolume prefers publish_context over volume_context (reflects current primary after failover). 8 tests.
- **Task 7: Rebuild on Recovery** — recoverBlockVolumes on VS reconnect drains pendingRebuilds, sets reconnected server as replica, enqueues Rebuilding assignments. 10 tests (shared with Task 5 test file).
-
-### Design Review Findings Addressed
-
-| # | Finding | Severity | Resolution |
-|---|---------|----------|------------|
-| F1 | Assignment delivery can be dropped | Critical | Retain-until-confirmed: Peek+Confirm pattern, assignments resent every heartbeat |
-| F2 | Failover without lease check → split-brain | Critical | Gate promotion on `now > lastLeaseGrant + leaseTTL`; deferred promotion for unexpired leases |
-| F3 | Replication ports change on VS restart | Critical | Deterministic port = FNV hash of path, offset from base iSCSI port |
-| F4 | Partial create (replica fails) | Medium | Single-copy mode with ReplicaServer="", skip replica assignments |
-| F5 | UpdateFullHeartbeat ignores replica addresses | Medium | VS includes replica_data/ctrl in InfoMessage; registry updates on heartbeat |
-
-### Code Review 1 Findings Addressed
-
-| # | Finding | Severity | Resolution |
-|---|---------|----------|------------|
-| R1-1 | AllocateBlockVolume missing repl addrs | High | AllocateBlockVolume now returns ReplicaDataAddr/CtrlAddr/RebuildListenAddr from ReplicationPorts() |
-| R1-2 | Primary never starts rebuild server | High | setupPrimaryReplication now calls vol.StartRebuildServer(rebuildAddr) |
-| R1-3 | Assignment queue never confirms after startup | High | VS sends periodic full block heartbeat (5×sleepInterval tick) enabling master confirmation |
-| R1-4 | Replica addresses not reported in heartbeat | Medium | BlockService.CollectBlockVolumeHeartbeat wraps store's collector, fills ReplicaDataAddr/CtrlAddr from replStates |
-| R1-5 | Lease never refreshed after create | Medium | UpdateFullHeartbeat refreshes LastLeaseGrant on every heartbeat; periodic block heartbeats keep it current |
-
-### Code Review 2 Findings Addressed
-
-| # | Finding | Severity | Resolution |
-|---|---------|----------|------------|
-| R2-F1 | LastLeaseGrant set AFTER Register → stale-lease race | High | Moved to entry initializer BEFORE Register |
-| R2-F2 | Deferred promotion timer has no cancellation | Medium | Timers stored in blockFailoverState.deferredTimers; cancelled in recoverBlockVolumes on reconnect |
-| R2-F3 | SwapPrimaryReplica hardcodes uint32(1) | Medium | Changed to blockvol.RoleToWire(blockvol.RolePrimary) |
-| R2-F4 | DeleteBlockVolume doesn't delete replica | Medium | Added best-effort replica delete (non-fatal if replica VS is down) |
-| R2-F5 | promoteReplica reads epoch without lock | Medium | SwapPrimaryReplica now computes epoch+1 atomically inside lock, returns newEpoch |
-| R2-F6 | Redundant string(server) casts | Low | Removed — servers already typed as string |
-| R2-F7 | startRebuild goroutine has no feedback path | Low | Documented as future work (VS could report via heartbeat) |
-
-### New Files (CP6-3)
-
-| File | Description |
-|------|-------------|
-| `server/master_block_assignment_queue.go` | Assignment queue with retain-until-confirmed |
-| `server/master_block_assignment_queue_test.go` | 11 queue tests |
-| `server/master_block_failover.go` | Failover detection + rebuild on recovery |
-| `server/master_block_failover_test.go` | 21 failover + rebuild tests |
-
-### Modified Files (CP6-3)
-
-| File | Changes |
-|------|---------|
-| `pb/master.proto` | Replica/rebuild fields on assignment/info/response messages |
-| `pb/volume_server.proto` | Replica/rebuild fields on AllocateBlockVolumeResponse |
-| `pb/master_pb/master.pb.go` | New fields + getters |
-| `pb/volume_server_pb/volume_server.pb.go` | New fields + getters |
-| `storage/blockvol/block_heartbeat.go` | ReplicaDataAddr/CtrlAddr on InfoMessage, RebuildAddr on Assignment |
-| `storage/blockvol/block_heartbeat_proto.go` | Updated converters + AssignmentsToProto |
-| `server/master_server.go` | blockAssignmentQueue, blockFailover, blockAllocResult struct |
-| `server/master_grpc_server.go` | Assignment delivery in heartbeat, failover on disconnect, recovery on reconnect |
-| `server/master_grpc_server_block.go` | Replica creation, assignment enqueueing, tryCreateReplica; R2-F1 LastLeaseGrant fix; R2-F4 replica delete; R2-F6 cast cleanup |
-| `server/master_block_registry.go` | Replica fields, lease fields, SetReplica/ClearReplica/SwapPrimaryReplica; R2-F3 RoleToWire; R2-F5 atomic epoch; R1-5 lease refresh |
-| `server/volume_grpc_client_to_master.go` | Assignment processing from HeartbeatResponse; R1-3 periodic block heartbeat tick |
-| `server/volume_grpc_block.go` | R1-1 replication ports in AllocateBlockVolumeResponse |
-| `server/volume_server_block.go` | ProcessAssignments, replication setup, ReplicationPorts; R1-2 StartRebuildServer; R1-4 CollectBlockVolumeHeartbeat with repl addrs |
-| `server/master_block_failover.go` | R2-F2 deferred timer cancellation; R2-F5 new SwapPrimaryReplica API; R2-F7 rebuild feedback comment |
-| `storage/store_blockvol.go` | WithVolume (exported) |
-| `csi/controller.go` | ControllerPublishVolume/UnpublishVolume, PUBLISH_UNPUBLISH capability |
-| `csi/node.go` | Prefer publish_context over volume_context |
-
-### CP6-3 Test Count
-
-| File | New Tests |
-|------|-----------|
-| `blockvol/block_heartbeat_proto_test.go` | 7 |
-| `server/master_block_assignment_queue_test.go` | 11 |
-| `server/volume_server_block_test.go` | 9 |
-| `server/master_block_registry_test.go` | 5 |
-| `server/master_grpc_server_block_test.go` | 6 |
-| `server/master_block_failover_test.go` | 21 |
-| `csi/controller_test.go` | 6 |
-| `csi/node_test.go` | 2 |
-| **Total CP6-3** | **67** |
-
-**Cumulative Phase 6 test count: 54 CP6-1 + 172 CP6-2 + 67 CP6-3 = 293 tests.**
-
-### CP6-3 QA Adversarial Tests
-48 tests in `server/qa_block_cp63_test.go`. 1 bug found and fixed.
-
-| Group | Tests | Areas |
-|-------|-------|-------|
-| Assignment Queue | 8 | Wrong epoch confirm, partial heartbeat confirm, same-path different roles, concurrent ops |
-| Registry | 7 | Double swap, swap no-replica, concurrent swap+lookup, SetReplica replace, heartbeat clobber |
-| Failover | 7 | Deferred cancel on reconnect, double disconnect, mixed lease states, volume deleted during timer |
-| Create+Delete | 5 | Lease non-zero after create, replica delete on vol delete, replica delete failure |
-| Rebuild | 3 | Double reconnect, nil failover state, full cycle |
-| Integration | 2 | Failover enqueues assignment, heartbeat confirms failover assignment |
-| Edge Cases | 5 | Epoch monotonic, cancel timers no rebuilds, replica server dies, empty batch |
-| Master-level | 5 | Delete VS unreachable, sanitized name, concurrent create/delete, all VS fail, slow allocate |
-| VS-level | 6 | Concurrent create, concurrent create/delete, delete cleans snapshots, sanitization collision, idempotent re-add, nil block service |
-
-**BUG-QA-CP63-1 (Medium): `SetReplica` leaks old replica server in `byServer` index.**
- `SetReplica` didn't remove old replica server from `byServer` when replacing with a new one.
- Fix: Added `removeFromServer(oldReplicaServer, name)` before setting new replica (3 lines).
- Test: `TestQA_Reg_SetReplicaTwice_ReplacesOld`.
-
-**Final CP6-3 test count: 67 dev/review + 48 QA = 115 CP6-3 tests, all PASS.**
-
-### CP6-3 Integration Tests
-8 tests in `server/integration_block_test.go`. Full cross-component flows.
-
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | FailoverCSIPublish | LookupBlockVolume returns new iSCSI addr after failover |
-| 2 | RebuildOnRecovery | Rebuilding assignment enqueued + heartbeat confirms it |
-| 3 | AssignmentDeliveryConfirmation | Queue retains until heartbeat confirms matching (path, epoch) |
-| 4 | LeaseAwarePromotion | Promotion deferred until lease TTL expires |
-| 5 | ReplicaFailureSingleCopy | Single-copy mode: no replica assignments, failover is no-op |
-| 6 | TransientDisconnectNoSplitBrain | Deferred timer cancelled on reconnect, no split-brain |
-| 7 | FullLifecycle | 11-phase lifecycle: create→publish→confirm→failover→re-publish→recover→rebuild→delete |
-| 8 | DoubleFailover | Two successive failovers: epoch 1→2→3 |
-| 9 | MultiVolumeFailoverRebuild | 3 volumes, kill 1 server, rebuild all affected |
-
-**Final CP6-3 test count: 67 dev/review + 48 QA + 8 mock integration + 3 real integration = 126 CP6-3 tests, all PASS.**
-
-**Cumulative Phase 6 with QA: 54 CP6-1 + 172 CP6-2 + 126 CP6-3 = 352 tests.**
-
-### CP6-3 Real Integration Tests (M02)
-3 tests in `blockvol/test/cp63_test.go`, run on M02 (192.168.1.184) with real iSCSI.
-
-**Bug found: RoleNone → RoleRebuilding transition not allowed.**
-After VS restart, volume is RoleNone. Master sends Rebuilding assignment, but both
-`validTransitions` (role.go) and `HandleAssignment` (promotion.go) rejected this path.
- Fix: Added `RoleRebuilding: true` to `validTransitions[RoleNone]` in role.go.
-  Added `RoleNone → RoleRebuilding` case in HandleAssignment with SetEpoch + SetRole.
- Admin API: Added `action:"connect"` to `/rebuild` endpoint (starts rebuild client).
-
-| # | Test | Time | What it proves |
-|---|------|------|----------------|
-| 1 | FailoverCSIAddressSwitch | 3.2s | Write A → kill primary → promote replica → re-discover at new iSCSI address → verify A → write B → verify A+B. Simulates CSI ControllerPublishVolume address-switch. |
-| 2 | RebuildDataConsistency | 5.3s | Write A (replicated) → kill replica → write B (missed) → restart replica as Rebuilding → rebuild server + client → wait role→Replica → kill primary → promote rebuilt → verify A+B. Full end-to-end rebuild with data verification. |
-| 3 | FullLifecycleFailoverRebuild | 6.4s | Write A → kill primary → promote → write B → rebuild old primary → write C → kill new primary → promote old → verify A+B. 11-phase lifecycle: failover→recoverBlockVolumes→rebuild. |
-
-All 7 existing HA tests: PASS (no regression). Total real integration: 10 tests on M02.
-
-## In Progress
- None.
-
-## Blockers
- None.
-
-## Next Steps
- CP6-4: Soak testing, lease renewal timers, monitoring dashboards.
-
-## Notes
- CSI spec dependency: `github.com/container-storage-interface/spec v1.10.0`.
- Architecture: CSI binary embeds TargetServer + BlockVol in-process (loopback iSCSI).
- Interface-based ISCSIUtil/MountUtil for unit testing without real iscsiadm/mount.
- k3s deployment requires: hostNetwork, hostPID, privileged, /dev mount, nsenter wrappers for host commands.
- Known pre-existing flaky: `TestQAPhase4ACP1/role_concurrent_transitions` (unrelated to CSI).
-
-## CP6-1 Test Catalog
-
-### VolumeManager (`csi/volume_manager_test.go`) — 10 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | CreateOpenClose | Create, verify IQN, close, reopen lifecycle |
-| 2 | DeleteRemovesFile | .blk file removed on delete |
-| 3 | DuplicateCreate | Same size idempotent; different size returns ErrVolumeSizeMismatch |
-| 4 | ListenAddr | Non-empty listen address after start |
-| 5 | OpenNonExistent | Error on opening non-existent volume |
-| 6 | CloseAlreadyClosed | Idempotent close of non-tracked volume |
-| 7 | ConcurrentCreateDelete | 10 parallel create+delete, no races |
-| 8 | SanitizeIQN | Special char replacement, truncation to 64 chars |
-| 9 | CreateIdempotentAfterRestart | Existing .blk file adopted on restart |
-| 10 | IQNCollision | Long names with same prefix get distinct IQNs via hash suffix |
-
-### Identity (`csi/identity_test.go`) — 3 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | GetPluginInfo | Returns correct driver name + version |
-| 2 | GetPluginCapabilities | Returns CONTROLLER_SERVICE capability |
-| 3 | Probe | Returns ready=true |
-
-### Controller (`csi/controller_test.go`) — 4 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | CreateVolume | Volume created and tracked |
-| 2 | CreateIdempotent | Same name+size succeeds, different size returns AlreadyExists |
-| 3 | DeleteVolume | Volume removed after delete |
-| 4 | DeleteNotFound | Delete non-existent returns success (CSI spec) |
-
-### Node (`csi/node_test.go`) — 7 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | StageUnstage | Full stage flow (discovery+login+mount) and unstage (unmount+logout+close) |
-| 2 | PublishUnpublish | Bind mount from staging to target path |
-| 3 | StageIdempotent | Already-mounted staging path returns OK without side effects |
-| 4 | StageLoginFailure | iSCSI login error propagated as Internal |
-| 5 | StageMkfsFailure | mkfs error propagated as Internal |
-| 6 | StageLoginFailureCleanup | Volume closed after login failure (no resource leak) |
-| 7 | PublishMissingStagingPath | Empty StagingTargetPath returns InvalidArgument |
-
-### Adapter (`blockvol/adapter_test.go`) — 3 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | AdapterALUAProvider | ALUAState/TPGroupID/DeviceNAA correct values |
-| 2 | RoleToALUA | All role→ALUA state mappings |
-| 3 | UUIDToNAA | NAA-6 byte layout from UUID |
-
-## CP6-2 Test Catalog
-
-### Registry (`server/master_block_registry_test.go`) — 11 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | RegisterLookup | Register + Lookup returns entry |
-| 2 | DuplicateRegister | Second register same name errors |
-| 3 | Unregister | Unregister removes entry |
-| 4 | ListByServer | Returns only entries for given server |
-| 5 | FullHeartbeat | Marks active, removes stale, adds new |
-| 6 | DeltaHeartbeat | Add/remove deltas applied correctly |
-| 7 | PickServer | Fewest-volumes placement |
-| 8 | Inflight | AcquireInflight blocks duplicate, ReleaseInflight unblocks |
-| 9 | BlockCapable | MarkBlockCapable / UnmarkBlockCapable tracking |
-| 10 | UnmarkDeadServer | R1-F2 regression test |
-| 11 | FullHeartbeatUpdatesSizeBytes | R2-F1 regression test |
-
-### Master RPCs (`server/master_grpc_server_block_test.go`) — 9 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | CreateHappyPath | Create → register → lookup works |
-| 2 | CreateIdempotent | Same name+size returns same entry |
-| 3 | CreateIdempotentSizeMismatch | Same name, smaller size → error |
-| 4 | CreateInflightBlock | Concurrent create same name → one fails |
-| 5 | Delete | Delete → VS called → unregistered |
-| 6 | DeleteNotFound | Delete non-existent → success |
-| 7 | Lookup | Lookup returns entry |
-| 8 | LookupNotFound | Lookup non-existent → NotFound |
-| 9 | CreateRetryNextServer | First VS fails → retries on next |
-
-### VS Block gRPC (`server/volume_grpc_block_test.go`) — 5 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | Allocate | Create via gRPC returns path+iqn+addr |
-| 2 | AllocateEmptyName | Empty name → error |
-| 3 | AllocateZeroSize | Zero size → error |
-| 4 | Delete | Delete via gRPC succeeds |
-| 5 | DeleteNilService | Nil blockService → error |
-
-### Naming (`blockvol/naming_test.go`) — 4 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | SanitizeFilename | Lowercases, replaces invalid chars |
-| 2 | SanitizeIQN | Lowercases, replaces, truncates with hash |
-| 3 | IQNMaxLength | 64-char names pass through unchanged |
-| 4 | IQNHashDeterministic | Same input → same hash suffix |
-
-### Proto conversion (`blockvol/block_heartbeat_proto_test.go`) — 5 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | RoundTrip | Go→proto→Go preserves all fields |
-| 2 | NilSafe | Nil input → nil output |
-| 3 | ShortRoundTrip | Short info round-trip |
-| 4 | AssignmentRoundTrip | Assignment round-trip |
-| 5 | SliceHelpers | Slice conversion helpers |
-
-### Backend (`csi/volume_backend_test.go`) — 5 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | LocalCreate | LocalVolumeBackend.CreateVolume creates + returns info |
-| 2 | LocalDelete | LocalVolumeBackend.DeleteVolume removes volume |
-| 3 | LocalLookup | LocalVolumeBackend.LookupVolume returns info |
-| 4 | LocalLookupNotFound | Lookup non-existent returns not-found |
-| 5 | LocalDeleteNotFound | Delete non-existent returns success |
-
-### Node remote (`csi/node_test.go` additions) — 4 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | StageRemoteTarget | volume_context drives iSCSI instead of local mgr |
-| 2 | UnstageRemoteTarget | Staged map IQN used for logout |
-| 3 | UnstageAfterRestart | IQN derived from iqnPrefix when staged map empty |
-| 4 | UnstageRetryKeepsStagedEntry | R2-F6 regression: staged entry preserved on failure |
-
-### QA Server (`server/qa_block_cp62_test.go`) — 22 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | Reg_FullHeartbeatCrossTalk | Heartbeat from s2 doesn't remove s1 volumes |
-| 2 | Reg_FullHeartbeatEmptyServer | Empty heartbeat marks server block-capable |
-| 3 | Reg_ConcurrentHeartbeatAndRegister | 10 goroutines heartbeat+register, no races |
-| 4 | Reg_DeltaHeartbeatUnknownPath | Delta for unknown path is no-op |
-| 5 | Reg_PickServerTiebreaker | PickServer returns first server on tie |
-| 6 | Reg_ReregisterDifferentServer | Re-register same name on different server fails |
-| 7 | Reg_InflightIndependence | Inflight lock for vol-a doesn't block vol-b |
-| 8 | Reg_BlockCapableServersAfterUnmark | Unmark removes from block-capable list |
-| 9 | Master_DeleteVSUnreachable | Delete fails if VS delete fails (no orphan) |
-| 10 | Master_CreateSanitizedName | Names with special chars go through |
-| 11 | Master_ConcurrentCreateDelete | Concurrent create+delete on same name, no panic |
-| 12 | Master_AllVSFailNoOrphan | All 3 servers fail → error, no registry entry |
-| 13 | Master_SlowAllocateBlocksSecond | Inflight lock blocks concurrent same-name create |
-| 14 | Master_CreateZeroSize | Zero size → InvalidArgument |
-| 15 | Master_CreateEmptyName | Empty name → InvalidArgument |
-| 16 | Master_EmptyNameValidation | Whitespace-only name → InvalidArgument |
-| 17 | VS_ConcurrentCreate | 20 goroutines create same vol, no crash |
-| 18 | VS_ConcurrentCreateDelete | 20 goroutines create+delete interleaved |
-| 19 | VS_DeleteCleansSnapshots | Delete removes .snap.* files |
-| 20 | VS_SanitizationCollision | Idempotent create after sanitization matches |
-| 21 | VS_CreateIdempotentReaddTarget | Idempotent create re-adds adapter to TargetServer |
-| 22 | VS_GrpcNilBlockService | Nil blockService returns error (not panic) |
-
-### QA CSI (`csi/qa_cp62_test.go`) — 32 tests
-| # | Test | What it proves |
-|---|------|----------------|
-| 1 | Node_RemoteUnstageNoCloseVolume | Remote unstage doesn't call CloseVolume |
-| 2 | Node_RemoteUnstageFailPreservesStaged | Failed unstage preserves staged entry |
-| 3 | Node_ConcurrentStageUnstage | 20 concurrent stage+unstage, no races |
-| 4 | Node_RemotePortalUsedCorrectly | Remote portal used for discovery (not local) |
-| 5 | Node_PartialVolumeContext | Missing iqn falls back to local mgr |
-| 6 | Node_UnstageNoMgrNoPrefix | No mgr + no prefix → empty IQN (graceful) |
-| 7 | Ctrl_VolumeContextPresent | CreateVolume returns iscsiAddr+iqn in context |
-| 8 | Ctrl_ValidateUsesBackend | ValidateVolumeCapabilities uses backend lookup |
-| 9 | Ctrl_CreateLargerSizeRejected | Existing vol + larger size → AlreadyExists |
-| 10 | Ctrl_ExactBlockSizeBoundary | Exact 4MB boundary succeeds |
-| 11 | Ctrl_ConcurrentCreate | 10 concurrent creates, one succeeds |
-| 12 | Backend_LookupAfterRestart | Volume found after VolumeManager restart |
-| 13 | Backend_DeleteThenLookup | Lookup after delete → not found |
-| 14 | Naming_CrossLayerConsistency | CSI and blockvol SanitizeIQN produce same result |
-| 15 | Naming_LongNameHashCollision | Two 70-char names → distinct IQNs |
-| 16 | RemoteLifecycleFull | Full remote stage→publish→unpublish→unstage→delete |
-| 17 | ModeControllerNoMgr | Controller mode with masterAddr, no local mgr |
-| 18 | ModeNodeOnly | Node mode creates mgr but no controller |
-| 19 | ModeInvalid | Invalid mode → error (BUG-QA-CP62-1) |
-| 20 | Srv_AllModeLocalBackend | All mode without master uses local backend |
-| 21 | Srv_DoubleStop | Double Stop doesn't panic |
-| 22 | VM_CreateAfterStop | Create after stop returns error |
-| 23 | VM_OpenNonExistent | Open non-existent returns error |
-| 24 | VM_ListenAddrAfterStop | ListenAddr after stop returns empty |
-| 25 | VM_VolumeIQNSanitized | VolumeIQN applies sanitization |
-| 26 | Edge_MinSize | Minimum 4MB volume succeeds |
-| 27 | Edge_BelowMinSize | Below minimum → error |
-| 28 | Edge_RequiredEqualsLimit | Required == limit succeeds |
-| 29 | Edge_RoundingExceedsLimit | Rounding up exceeds limit → error |
-| 30 | Edge_EmptyVolumeIDNode | Empty volumeID → InvalidArgument |
-| 31 | Node_PublishWithoutStaging | Publish unstaged vol → still works (mock) |
-| 32 | Node_DoubleUnstage | Double unstage → idempotent success |
@@ -1,4 +0,0 @@
-This directory holds cached build artifacts from the Go build system.
-Run "go clean -cache" if the directory is getting too large.
-Run "go clean -fuzzcache" to delete the fuzz cache.
-See go.dev to learn more about Go.
@@ -1 +0,0 @@
-1774577367
@@ -1,27 +0,0 @@
-# .private
-
-Private working area for `sw-block`.
-
-Use this for:
- phase development notes
- roadmap/progress tracking
- draft handoff notes
- temporary design comparisons
- prototype scratch work not ready for `design/` or `prototype/`
-
-Recommended layout:
- `.private/phase/`: phase-by-phase development notes
- `.private/roadmap/`: short-term and medium-term execution notes
- `.private/handoff/`: notes for `sw`, `qa`, or future sessions
-
-Phase protocol:
- each phase should normally have:
-  - `phase-xx.md`
-  - `phase-xx-log.md`
-  - `phase-xx-decisions.md`
- details are defined in `.private/phase/README.md`
-
-Promotion rules:
- stable vision/design docs go to `../design/`
- real prototype code stays in `../prototype/`
- `.private/` is for working material, not source of truth
@@ -1,36 +0,0 @@
-# Phase Dev
-
-Use this directory for private phase development notes.
-
-## Phase Protocol
-
-Each phase should use this file set:
-
- `phase-01.md`
-  - plan
-  - scope
-  - progress
-  - active tasks
-  - exit criteria
- `phase-01-log.md`
-  - dated development log
-  - experiments
-  - test runs
-  - failures and findings
- `phase-01-decisions.md`
-  - key algorithm decisions
-  - tradeoffs
-  - rejected alternatives
-
-Suggested naming pattern:
- `phase-01.md`
- `phase-01-log.md`
- `phase-01-decisions.md`
- `phase-02.md`
- `phase-02-log.md`
- `phase-02-decisions.md`
-
-Rule of use:
-1. if it is what we are doing -> `phase-xx.md`
-2. if it is what happened -> `phase-xx-log.md`
-3. if it is why we chose something -> `phase-xx-decisions.md`
@@ -1,97 +0,0 @@
-# Phase 01 Decisions
-
-Date: 2026-03-26
-Status: active
-
-## Purpose
-
-Capture the key design decisions made during Phase 01 simulator work.
-
-## Initial Decisions
-
-### 1. `design/` vs `.private/phase/`
-
-Decision:
- `sw-block/design/` holds shared design truth
- `sw-block/.private/phase/` holds execution planning and progress
-
-Reason:
- design backlog and execution checklist should not be mixed
-
-### 2. Scenario source of truth
-
-Decision:
- `sw-block/design/v2_scenarios.md` is the scenario backlog and coverage matrix
-
-Reason:
- all contributors need one visible scenario list
-
-### 3. Phase 01 priority
-
-Decision:
- first close:
-  - `S19`
-  - `S20`
-
-Reason:
- they are the biggest remaining distributed lineage/partition scenarios
-
-### 4. Current simulator scope
-
-Decision:
- use the simulator as a V2 design-validation tool, not a product/perf harness
-
-Reason:
- current goal is correctness and protocol coverage, not productization
-
-### 5. Phase execution format
-
-Decision:
- keep phase execution in three files:
-  - `phase-xx.md`
-  - `phase-xx-log.md`
-  - `phase-xx-decisions.md`
-
-Reason:
- separates plan, evidence, and reasoning
- reduces drift between roadmap and findings
-
-### 6. Design backlog vs execution plan
-
-Decision:
- `sw-block/design/v2_scenarios.md` remains the source of truth for scenario backlog and coverage
- `.private/phase/phase-01.md` is the execution layer for `sw`
-
-Reason:
- design truth should be stable and shareable
- execution tasks should be easier to edit without polluting design docs
-
-### 7. Immediate Phase 01 priorities
-
-Decision:
- prioritize:
-  - `S19` chain of custody across multiple promotions
-  - `S20` live partition with competing writes
-
-Reason:
- these are the biggest remaining distributed-lineage gaps after current simulator milestone
-
-### 8. Coverage status should be conservative
-
-Decision:
- mark scenarios as `partial` unless the test actually exercises the core protocol obligation, not just a simplified happy path
-
-Reason:
- avoids overstating simulator coverage
- keeps the backlog honest for follow-up strengthening
-
-### 9. Protocol-version comparison belongs in the simulator
-
-Decision:
- compare `V1`, `V1.5`, and `V2` using the same scenario set where possible
-
-Reason:
- this is the clearest way to show:
-  - where V1 breaks
-  - where V1.5 improves but still strains
-  - why V2 is architecturally cleaner
@@ -1,67 +0,0 @@
-# Phase 01 Log
-
-Date: 2026-03-26
-Status: active
-
-## Log Protocol
-
-Use dated entries like:
-
-## 2026-03-26
- work completed
- tests run
- failures found
- seeds/traces worth keeping
- follow-up items
-
-## Initial State
-
- Phase 01 created from the earlier `phase-01-v2-scenarios.md` working note
- scenario source of truth remains:
-  - `sw-block/design/v2_scenarios.md`
- current active asks for `sw`:
-  - `S19`
-  - `S20`
-
-## 2026-03-26
-
- created Phase 01 file set:
-  - `phase-01.md`
-  - `phase-01-log.md`
-  - `phase-01-decisions.md`
- promoted scenario execution checklist into `phase-01.md`
- kept `sw-block/design/v2_scenarios.md` as the shared backlog and coverage matrix
- current simulator milestone:
-  - `fsmv2` passing
-  - `volumefsm` passing
-  - `distsim` passing
-  - randomized `distsim` seeds passing
-  - event/interleaving simulator work present in `sw-block/prototype/distsim/simulator.go`
- current immediate development priority for `sw`:
-  - implement `S19`
-  - implement `S20`
- `sw` added Phase 01 P0/P1 scenario tests in `distsim`:
-  - `S19`
-  - `S20`
-  - `S5`
-  - `S6`
-  - `S18`
-  - stronger `S12`
- review result:
-  - `S19` looks solid
-  - stronger `S12` now looks solid
-  - `S20`, `S5`, `S6`, `S18` are better classified as `partial` than fully closed
- updated `v2_scenarios.md` coverage matrix to reflect actual status
- next development focus:
-  - P2 scenarios
-  - stronger versions of current partial scenarios
- added protocol-version comparison design:
-  - `sw-block/design/protocol-version-simulation.md`
- added minimal protocol policy prototype in `distsim`:
-  - `ProtocolV1`
-  - `ProtocolV15`
-  - `ProtocolV2`
-  - focused on:
-    - catch-up policy
-    - tail-chasing outcome policy
-    - restart/rejoin policy
@@ -1,11 +0,0 @@
-# Deprecated
-
-This file is deprecated.
-
-Use instead:
- `phase-01.md`
- `phase-01-log.md`
- `phase-01-decisions.md`
-
-The scenario source of truth remains:
- `sw-block/design/v2_scenarios.md`
@@ -1,164 +0,0 @@
-# Phase 01
-
-Date: 2026-03-26
-Status: completed
-Purpose: drive V2 simulator development by closing the scenario backlog in `sw-block/design/v2_scenarios.md`
-
-## Goal
-
-Make the V2 simulator cover the important protocol scenarios as explicitly as possible.
-
-This phase is about:
- simulator fidelity
- scenario coverage
- invariant quality
-
-This phase is not about:
- product integration
- SPDK
- raw allocator
- production transport
-
-## Source Of Truth
-
-Design/source-of-truth:
- `sw-block/design/v2_scenarios.md`
-
-Prototype code:
- `sw-block/prototype/fsmv2/`
- `sw-block/prototype/volumefsm/`
- `sw-block/prototype/distsim/`
-
-## Assigned Tasks For `sw`
-
-### P0
-
-1. `S19` chain of custody across multiple promotions
- add fixed test(s)
- verify committed data from `A -> B -> C`
- update coverage matrix
-
-2. `S20` live partition with competing writes
- add fixed test(s)
- stale side must not advance committed lineage
- update coverage matrix
-
-### P1
-
-3. `S5` flapping replica stays recoverable
- repeated disconnect/reconnect
- no unnecessary rebuild while recovery remains possible
-
-4. `S6` tail-chasing under load
- primary keeps writing while replica catches up
- explicit outcome:
-  - converge and promote
-  - or abort to rebuild
-
-5. `S18` primary restart without failover
- same-lineage restart behavior
- no stale session assumptions
-
-6. stronger `S12`
- more than one promotion candidate
- choose valid lineage, not merely highest apparent LSN
-
-### P2
-
-7. protocol-version comparison support
- model:
-  - `V1`
-  - `V1.5`
-  - `V2`
- use the same scenario set to show:
-  - V1 breaks
-  - V1.5 improves but still strains
-  - V2 handles recovery more explicitly
-
-8. richer Smart WAL scenarios
- time-varying `ExtentReferenced` availability
- recoverable then unrecoverable transitions
-
-9. delayed/drop network scenarios beyond simple disconnect
-
-10. multi-node reservation expiry / rebuild timeout cases
-
-## Invariants To Preserve
-
-After every scenario or random run, preserve:
-
-1. committed data is durable per policy
-2. uncommitted data is not revived as committed
-3. stale epoch traffic does not mutate current lineage
-4. recovered/promoted node matches reference state at target `LSN`
-5. committed prefix remains contiguous
-
-## Required Updates Per Task
-
-For each completed scenario:
-
-1. add or update test(s)
-2. update `sw-block/design/v2_scenarios.md`
-   - package
-   - test name
-   - status
-3. note any missing simulator capability
-
-## Current Progress
-
-Already in place before this phase:
- `fsmv2` local FSM prototype
- `volumefsm` orchestrator prototype
- `distsim` distributed simulator
- randomized `distsim` runs
- first event/interleaving simulator work in `distsim/simulator.go`
-
-Open focus:
- `S19` covered in `distsim`
- `S20` partially covered in `distsim`
- `S5` partially covered in `distsim`
- `S6` partially covered in `distsim`
- `S18` partially covered in `distsim`
- stronger `S12` covered in `distsim`
- protocol-version comparison design added in:
-  - `sw-block/design/protocol-version-simulation.md`
- remaining focus is now P2 plus stronger versions of partial scenarios
-
-## Phase Status
-
-### P0
-
- `S19` chain of custody across multiple promotions: done
- `S20` live partition with competing writes: partial
-
-### P1
-
- `S5` flapping replica stays recoverable: partial
- `S6` tail-chasing under load: partial
- `S18` primary restart without failover: partial
- stronger `S12`: done
-
-### P2
-
- active next step:
-  - protocol-version comparison support
-  - stronger versions of current partial scenarios
-
-## Exit Criteria
-
-Phase 01 is done when:
-
-1. `S19` and `S20` are covered
-2. `S5`, `S6`, `S18`, and stronger `S12` are at least partially covered
-3. coverage matrix in `v2_scenarios.md` is current
-4. random simulation still passes after added scenarios
-
-## Completion Note
-
-Phase 01 completed with:
- `S19` covered
- stronger `S12` covered
- `S20`, `S5`, `S6`, `S18` strengthened but correctly left as `partial`
-
-Next execution phase:
- `sw-block/.private/phase/phase-02.md`
@@ -1,51 +0,0 @@
-# Phase 02 Decisions
-
-Date: 2026-03-26
-Status: active
-
-## Decision 1: Extend `distsim` Instead Of Forking A New Protocol Simulator
-
-Reason:
- current `distsim` already has:
-  - node/storage model
-  - coordinator/epoch model
-  - reference oracle
-  - randomized runs
- the missing layer is protocol-state fidelity, not a new simulation foundation
-
-Implication:
- add lightweight per-node replication state and protocol decisions to `distsim`
- do not build a separate fourth simulator yet
-
-## Decision 2: Keep Coverage Status Conservative
-
-Reason:
- `S20`, `S6`, and `S18` currently prove important safety properties
- but they do not yet fully assert message-level or explicit state-transition behavior
-
-Implication:
- leave them `partial` until the model can assert protocol behavior directly
-
-## Decision 3: Use Versioned Scenario Comparison To Justify V2
-
-Reason:
- the simulator should not only say "V2 works"
- it should show:
-  - where `V1` fails
-  - where `V1.5` improves but still strains
-  - why `V2` is worth the complexity
-
-Implication:
- Phase 02 includes explicit `V1` / `V1.5` / `V2` scenario comparison work
-
-## Decision 4: V2 Must Not Be Described As "Always Catch-Up"
-
-Reason:
- that wording is too optimistic and hides the real V2 design rule
- V2 is better because it makes recoverability explicit, not because it retries forever
-
-Implication:
- describe V2 as:
-  - catch-up if explicitly recoverable
-  - otherwise explicit rebuild
- keep this wording consistent in tests and docs
@@ -1,93 +0,0 @@
-# Phase 02 Log
-
-Date: 2026-03-26
-Status: active
-
-## 2026-03-26
-
- Phase 02 created to move `distsim` from final-state safety validation toward explicit protocol-state simulation.
- Initial focus:
-  - close `S20`, `S6`, and `S18` at protocol level
-  - compare `V1`, `V1.5`, and `V2` on the same scenarios
- Known model gap at phase start:
-  - current `distsim` is strong at final-state safety invariants
-  - current `distsim` is weaker at mid-flow protocol assertions and message-level rejection reasons
- Phase 02 progress now in place:
-  - delivery accept/reject tracking
-  - protocol-level stale-epoch rejection assertions
-  - explicit non-convergent catch-up state transition assertions
-  - initial version-comparison tests for disconnect, tail-chasing, and restart/rejoin policy
- Next simulator target:
-  - reproduce real `V1.5` address-instability and control-plane-recovery failures as named scenarios
- Immediate coding asks for `sw`:
-  - changed-address restart failure in `V1.5`
-  - same-address transient outage comparison across `V1` / `V1.5` / `V2`
-  - slow control-plane reassignment scenario derived from `CP13-8 T4b`
- Local housekeeping done:
-  - corrected V2 wording from "always catch-up" to "catch-up if explicitly recoverable; otherwise rebuild"
-  - added explicit brief-disconnect and changed-address restart policy helpers
-  - verified `distsim` test suite still passes with the Windows-safe runner
- Scenario status update:
-  - `S20` now covered via protocol-level stale-traffic rejection + committed-prefix stability
-  - `S6` now covered via explicit `CatchingUp -> NeedsRebuild` assertions
-  - `S18` now covered via explicit stale `MsgBarrierAck` rejection + prefix stability
- Next asks for `sw` after this closure:
-  - changed-address restart scenario tied directly to `CP13-8 T4b`
-  - same-address transient outage comparison across `V1` / `V1.5` / `V2`
-  - slow control-plane reassignment scenario
-  - Smart WAL recoverable -> unrecoverable transition scenarios
- Additional closure completed:
-  - `S5` now covered with both:
-    - repeated recoverable flapping
-    - budget-exceeded escalation to `NeedsRebuild`
-  - Smart WAL transitions now exercised with:
-    - recoverable -> unrecoverable during active recovery
-    - mixed `WALInline` + `ExtentReferenced` success
-    - time-varying payload availability
- Updated next asks for `sw`:
-  - changed-address restart scenario tied directly to `CP13-8 T4b`
-  - same-address transient outage comparison across `V1` / `V1.5` / `V2`
-  - slow control-plane reassignment scenario
-  - delayed/drop network beyond simple disconnect
-  - multi-node reservation expiry / rebuild timeout cases
- Additional Phase 02 coverage delivered:
-  - delayed stale messages after promote/failover
-  - delayed stale barrier ack rejection
-  - selective write-drop with barrier delivery under `sync_all`
-  - multi-node mixed reservation expiry outcome
-  - multi-node `NeedsRebuild` / snapshot rebuild recovery
-  - partial rebuild timeout / retry completion
- Remaining asks are now narrower:
-  - changed-address restart scenario tied directly to `CP13-8 T4b`
-  - same-address transient outage comparison across `V1` / `V1.5` / `V2`
-  - slow control-plane reassignment scenario
-  - stronger coordinator candidate-selection scenarios
- Additional closure after review:
-  - safe default promotion selector now refuses `NeedsRebuild` candidates
-  - explicit desperate-promotion API separated from safe selection
-  - changed-address and slow-control-plane comparison tests now prove actual data divergence / healing, not only policy shape
- New next-step assignment:
-  - strengthen model depth around endpoint identity and control-plane reassignment
-  - replace abstract repair helpers with more explicit event flow where practical
-  - reduce direct recovery state injection in comparison tests
-  - extend candidate selection from ranking into validity rules
-
-## 2026-03-27
-
- Phase 02 core simulator hardening is effectively complete.
- Delivered since the previous checkpoint:
-  - endpoint identity / endpoint-version modeling
-  - stale-endpoint rejection in delivery path
-  - heartbeat -> coordinator detect -> assignment-update control-plane flow
-  - recovery-session trigger API for `V1.5` and `V2`
-  - explicit candidate eligibility checks:
-    - running
-    - epoch alignment
-    - state eligibility
-    - committed-prefix sufficiency
-  - safe default promotion now rejects candidates without the committed prefix
- Current `distsim` status at latest review:
-  - 73 tests passing
- Manager bookkeeping decision:
-  - keep Phase 02 active only for doc maintenance / wrap-up
-  - treat further simulator depth as likely Phase 03 work, not unbounded Phase 02 scope creep
@@ -1,191 +0,0 @@
-# Phase 02
-
-Date: 2026-03-27
-Status: active
-Purpose: extend the V2 simulator from final-state safety checking into protocol-state simulation that can reproduce `V1`, `V1.5`, and `V2` behavior on the same scenarios
-
-## Goal
-
-Make the simulator model enough node-local replication state and message-level behavior to:
-
-1. reproduce `V1` / `V1.5` failure modes
-2. show why those failures are structural
-3. close the current `partial` V2 scenarios with stronger protocol assertions
-
-This phase is about:
- protocol-version comparison
- per-node replication state
- message-level fencing / accept / reject behavior
- explicit catch-up abort / rebuild transitions
-
-This phase is not about:
- product integration
- production transport
- SPDK
- raw allocator
-
-## Source Of Truth
-
-Design/source-of-truth:
- `sw-block/design/v2_scenarios.md`
- `sw-block/design/protocol-version-simulation.md`
- `sw-block/design/v1-v15-v2-simulator-goals.md`
-
-Prototype code:
- `sw-block/prototype/distsim/`
-
-## Assigned Tasks For `sw`
-
-### P0
-
-1. Add per-node replication state to `distsim`
- minimum states:
-  - `InSync`
-  - `Lagging`
-  - `CatchingUp`
-  - `NeedsRebuild`
-  - `Rebuilding`
- keep state lightweight; do not clone full `fsmv2` into `distsim`
-
-2. Add message-level protocol decisions
- stale-epoch write / ship / barrier traffic must be explicitly rejected
- record whether a message was:
-  - accepted
-  - rejected by epoch
-  - rejected by state
-
-3. Add explicit catch-up abort / rebuild entry
- non-convergent catch-up must move to explicit modeled failure:
-  - `NeedsRebuild`
-  - or equivalent abort outcome
-
-### P1
-
-4. Re-close `S20` at protocol level
- stale-side writes must go through protocol delivery path
- prove stale-side traffic cannot advance committed lineage
-
-5. Re-close `S6` at protocol level
- assert explicit abort/escalation on non-convergence
- not only final-state safety
-
-6. Re-close `S18` at protocol level
- assert committed-prefix behavior around delayed old ack / restart races
- not only final-state oracle checks
-
-### P2
-
-7. Expand protocol-version comparison
- run selected scenarios under:
-  - `V1`
-  - `V1.5`
-  - `V2`
- at minimum:
-  - brief disconnect
-  - restart with changed address
-  - tail-chasing
-
-8. Add V1.5-derived failure scenarios
- replica restart with changed receiver address
- same-address transient outage
- slow control-plane recovery vs fast local reconnect
-
-9. Prepare richer recovery modeling
- time-varying recoverability
- reservation loss during active catch-up
- rebuild timeout / retry in mixed-state cluster
-
-## Invariants To Preserve
-
-After every scenario or random run, preserve:
-
-1. committed data is durable per policy
-2. uncommitted data is not revived as committed
-3. stale epoch traffic does not mutate current lineage
-4. recovered/promoted node matches reference state at target `LSN`
-5. committed prefix remains contiguous
-6. protocol-state transitions are explicit, not inferred from final data only
-
-## Required Updates Per Task
-
-For each completed task:
-
-1. add or update test(s)
-2. update `sw-block/design/v2_scenarios.md`
-   - package
-   - test name
-   - status
-   - source if new scenario was derived from V1/V1.5 behavior
-3. add a short note to:
-   - `sw-block/.private/phase/phase-02-log.md`
-4. if a design choice changed, record it in:
-   - `sw-block/.private/phase/phase-02-decisions.md`
-
-## Current Progress
-
-Already in place before this phase:
- `distsim` final-state safety invariants
- randomized simulation
- event/interleaving simulator work
- initial `ProtocolVersion` / policy scaffold
- `S19` covered
- stronger `S12` covered
-
-Known partials to close in this phase:
- none in the current named backlog slice
-
-Delivered in this phase so far:
- delivery accept/reject tracking added
- protocol-level rejection assertions added
- explicit `CatchingUp -> NeedsRebuild` state transition tested
- selected protocol-version comparison tests added
- `S20`, `S6`, and `S18` moved from `partial` to `covered`
- Smart WAL transition scenarios added
- `S5` moved from `partial` to `covered`
- endpoint identity / endpoint-version modeling added
- explicit heartbeat -> detect -> assignment-update control-plane flow added for changed-address restart
- explicit recovery-session triggers added for `V1.5` and `V2`
- promotion selection now uses explicit eligibility, including committed-prefix gating
- safe and desperate promotion paths are separated
- full `distsim` suite at latest review: 73 tests passing
-
-Remaining focus for `sw`:
- Phase 02 core scope is now largely delivered
- remaining work should be treated as future-strengthening, not baseline closure
- if more simulator depth is needed next, it should likely start as Phase 03:
-  - timeout semantics
-  - timer races
-  - richer event/interleaving behavior
-  - stronger endpoint/control-plane realism beyond the current abstract model
-
-## Immediate Next Tasks For `sw`
-
-1. Add a documented compare artifact for new scenarios
- for each new `V1` / `V1.5` / `V2` comparison:
-  - record scenario name
-  - what fails in `V1`
-  - what improves in `V1.5`
-  - what is explicit in `V2`
- keep `sw-block/design/v1-v15-v2-comparison.md` updated
-
-2. Keep the coverage matrix honest
- do not mark a scenario `covered` unless the test asserts protocol behavior directly
- final-state oracle checks alone are not enough
-
-3. Prepare Phase 03 proposal instead of broadening ad hoc
- if more depth is needed, define it cleanly first:
-  - timers / timeout events
-  - event ordering races
-  - richer endpoint lifecycle
-  - recovery-session uniqueness across competing triggers
-
-## Exit Criteria
-
-Phase 02 is done when:
-
-1. `S5`, `S6`, `S18`, and `S20` are covered at protocol level
-2. `distsim` can reproduce at least one `V1` failure, one `V1.5` failure, and the corresponding `V2` behavior on the same named scenario
-3. protocol-level rejection/accept behavior is asserted in tests, not only inferred from final-state oracle checks
-4. coverage matrix in `v2_scenarios.md` is current
-5. changed-address and reconnect scenarios are modeled through explicit endpoint / control-plane behavior rather than helper-only abstraction
-6. promotion selection uses explicit eligibility, including committed-prefix safety
@@ -1,97 +0,0 @@
-# Phase 03 Decisions
-
-Date: 2026-03-27
-Status: initial
-
-## Why Phase 03 Exists
-
-Phase 02 already covered the main protocol-state story:
-
- V1 / V1.5 / V2 comparison
- stale traffic rejection
- catch-up vs rebuild
- changed-address restart control-plane flow
- committed-prefix-safe promotion eligibility
-
-The next simulator problems are different:
-
- timer semantics
- timeout races
- event ordering under contention
-
-That deserves a separate phase so the model boundary stays clear.
-
-## Initial Boundary
-
-### `distsim`
-
-Keep for:
-
- protocol correctness
- reference-state validation
- recoverability logic
- promotion / lineage rules
-
-### `eventsim`
-
-Grow for:
-
- explicit event queue behavior
- timeout events
- equal-time scheduling choices
- race exploration
-
-## Working Rule
-
-Do not move all scenarios into `eventsim`.
-
-Only move or duplicate scenarios when:
-
- timer or event ordering is the real bug surface
- `distsim` abstraction hides the important behavior
-
-## Accepted Phase 03 Decisions
-
-### Same-tick rule
-
-Within one tick:
-
- data/message delivery is evaluated before timeout firing
-
-Meaning:
-
- if an ack arrives in the same tick as a timeout deadline, the ack wins and may cancel the timeout
-
-This is now an explicit simulator rule, not accidental behavior.
-
-### Timeout authority
-
-Not every timeout that reaches its deadline still has authority to mutate state.
-
-So we now distinguish:
-
- `FiredTimeouts`
-  - timeout had authority and changed the model
- `IgnoredTimeouts`
-  - timeout reached deadline but was stale and ignored
-
-This keeps replay/debug output honest.
-
-### Late barrier ack rule
-
-Once a barrier instance times out:
-
- it is marked expired
- late ack for that barrier instance is rejected
-
-That prevents a stale ack from reviving old durability state.
-
-### Review gate rule for timer work
-
-Timer/race work is easy to get subtly wrong while still having green tests.
-
-So timer-related work is not accepted until:
-
- code path is reviewed
- tests assert the real protocol obligation
- stale and authoritative timer behavior are clearly distinguished
@@ -1,36 +0,0 @@
-# Phase 03 Log
-
-Date: 2026-03-27
-Status: active
-
-## 2026-03-27
-
- Phase 03 created after Phase 02 core scope was effectively delivered.
- Reason for new phase:
-  - remaining simulator work is about timer semantics and race behavior, not basic protocol-state coverage
- Initial target:
-  - define `distsim` vs `eventsim` split more clearly
-  - add explicit timeout semantics
-  - add timer-race scenarios without bloating `distsim` ad hoc
- P0 delivered:
-  - timeout model added for barrier / catch-up / reservation
-  - timeout-backed scenarios added
-  - same-tick ordering rule defined as data-before-timers
- First review result:
-  - timeout semantics accepted only after making cancellation model-driven
-  - late barrier ack after timeout required explicit rejection
- P0 hardening delivered:
-  - recovery timeout cancellation moved into model logic
-  - stale late barrier ack rejected via expired-barrier tracking
-  - stale vs authoritative timeout distinction added:
-    - `FiredTimeouts`
-    - `IgnoredTimeouts`
- P1 delivered and reviewed:
-  - promotion vs stale timeout race
-  - rebuild completion vs epoch bump race
-  - trace builder moved into reusable code
- Current suite state at latest accepted review:
-  - 86 `distsim` tests passing
- Manager decision:
-  - Phase 03 P0/P1 are accepted
-  - next work should move to deliberate P2 selection rather than broadening the phase ad hoc
@@ -1,193 +0,0 @@
-# Phase 03
-
-Date: 2026-03-27
-Status: active
-Purpose: define the next simulator tier after Phase 02, focused on timeout semantics, timer races, and a cleaner split between protocol simulation and event/interleaving simulation
-
-## Goal
-
-Phase 03 exists to cover behavior that current `distsim` still abstracts away:
-
-1. timeout semantics
-2. timer races
-3. event ordering under competing triggers
-4. clearer separation between:
-   - protocol / lineage simulation
-   - event / race simulation
-
-This phase should not reopen already-closed Phase 02 protocol scope unless a clear bug is found.
-
-## Why A New Phase
-
-Phase 02 already delivered:
-
- protocol-state assertions
- V1 / V1.5 / V2 comparison scenarios
- endpoint identity modeling
- control-plane assignment-update flow
- committed-prefix-aware promotion eligibility
-
-What remains is different in character:
-
- timers
- delayed events racing with each other
- timeout-triggered state changes
- more explicit event scheduling
-
-That deserves a new phase boundary.
-
-## Source Of Truth
-
-Design/source-of-truth:
- `sw-block/design/v2_scenarios.md`
- `sw-block/design/v2-dist-fsm.md`
- `sw-block/design/v2-scenario-sources-from-v1.md`
- `sw-block/design/v1-v15-v2-comparison.md`
-
-Current prototype base:
- `sw-block/prototype/distsim/`
- `sw-block/prototype/distsim/simulator.go`
-
-## Scope
-
-### In scope
-
-1. timeout semantics
- barrier timeout
- catch-up timeout
- reservation expiry timeout
- rebuild timeout
-
-2. timer races
- delayed ack vs timeout
- timeout vs promotion
- reconnect vs timeout
- catch-up completion vs expiry
- rebuild completion vs epoch bump
-
-3. simulator split clarification
- `distsim` keeps:
-  - protocol correctness
-  - lineage
-  - recoverability
-  - reference-state checking
- `eventsim` grows into:
-  - event scheduling
-  - timer firing
-  - same-time interleavings
-  - race exploration
-
-### Out of scope
-
- production integration
- real transport
- real disk timings
- SPDK
- raw allocator
-
-## Assigned Tasks For `sw`
-
-### P0
-
-1. Write a concrete `eventsim` scope note in code/docs
- define what stays in `distsim`
- define what moves to `eventsim`
- avoid overlap and duplicated semantics
-
-2. Add minimal timeout event model
- first-class timeout event type(s)
- at minimum:
-  - barrier timeout
-  - catch-up timeout
-  - reservation expiry
-
-3. Add timeout-backed scenarios
- stale delayed ack vs timeout
- catch-up timeout before convergence
- reservation expiry during active recovery
-
-### P1
-
-4. Add race-focused tests
- promotion vs delayed stale ack
- rebuild completion vs epoch bump
- reconnect success vs timeout firing
-
-5. Keep traces debuggable
- failing runs must dump:
-  - seed
-  - event order
-  - timer events
-  - node states
-  - committed prefix
-
-### P2
-
-6. Decide whether selected `distsim` scenarios should also exist in `eventsim`
- only when timer/event ordering is the real point
- do not duplicate every scenario blindly
-
-## Current Progress
-
-Delivered in this phase so far:
-
- `eventsim` scope note added in code
- explicit timeout model added:
-  - barrier timeout
-  - catch-up timeout
-  - reservation timeout
- timeout-backed scenarios added and reviewed
- same-tick rule made explicit:
-  - data before timers
- recovery timeout cancellation is now model-driven, not test-driven
- stale barrier ack after timeout is explicitly rejected
- stale timeouts are separated from authoritative timeouts:
-  - `FiredTimeouts`
-  - `IgnoredTimeouts`
- race-focused scenarios added and reviewed:
-  - promotion vs stale catch-up timeout
-  - promotion vs stale barrier timeout
-  - rebuild completion vs epoch bump
-  - epoch bump vs stale catch-up timeout
- reusable trace builder added for replay/debug support
- current `distsim` suite at latest review:
-  - 86 tests passing
-
-Remaining focus for `sw`:
-
- Phase 03 P0 and P1 are effectively complete
- Phase 03 P2 is also effectively complete after review
- any further simulator work should now be narrow and evidence-driven
- recommended next simulator additions only:
-  - control-plane latency parameter
-  - sustained-write convergence / tail-chasing load test
-  - one multi-promotion lineage extension
-
-## Invariants To Preserve
-
-1. committed data remains durable per policy
-2. uncommitted data is never revived as committed
-3. stale epoch traffic never mutates current lineage
-4. committed prefix remains contiguous
-5. timeout-triggered transitions are explicit and explainable
-6. races do not silently bypass fencing or rebuild boundaries
-
-## Required Updates Per Task
-
-For each completed task:
-
-1. add or update tests
-2. update `sw-block/design/v2_scenarios.md` if scenario coverage changed
-3. add a short note to:
-   - `sw-block/.private/phase/phase-03-log.md`
-4. if the simulator boundary changed, record it in:
-   - `sw-block/.private/phase/phase-03-decisions.md`
-
-## Exit Criteria
-
-Phase 03 is done when:
-
-1. timeout semantics exist as explicit simulator behavior
-2. at least three important timer-race scenarios are modeled and tested
-3. `distsim` vs `eventsim` responsibilities are clearly separated
-4. failure traces from race/timeout scenarios are replayable enough to debug
@@ -1,200 +0,0 @@
-# Phase 04 Decisions
-
-Date: 2026-03-27
-Status: complete
-
-## First Slice Decision
-
-The first standalone V2 implementation slice is:
-
- per-replica sender ownership
- one active recovery session per replica per epoch
-
-## Why Not Start In V1
-
-V1/V1.5 remains:
-
- production line
- maintenance/fix line
-
-It should not be the place where V2 architecture is first implemented.
-
-## Why This Slice
-
-This slice:
-
- directly addresses the clearest V1.5 structural pain
- maps cleanly to the V2-boundary tests
- is narrow enough to implement without dragging in the entire future architecture
-
-## Accepted P0 Refinements
-
-### Sender epoch coherence
-
-Sender-owned epoch is real state, not decoration.
-
-So:
-
- reconcile/update paths must refresh sender epoch
- stale active session must be invalidated on epoch advance
-
-### Session lifecycle
-
-The first slice should not use a totally loose lifecycle shell.
-
-So:
-
- session phase changes now follow an explicit transition map
- invalid jumps are rejected
-
-### Session attach rule
-
-Attaching a session at the wrong epoch is invalid.
-
-So:
-
- `AttachSession(epoch, kind)` must reject epoch mismatch with the owning sender
-
-## Accepted P1 Refinements
-
-### Session identity fencing
-
-The standalone V2 slice must reject stale completion by explicit session identity.
-
-So:
-
- `RecoverySession` has stable unique identity
- sender completion must be by session ID, not by "current pointer"
- stale session results are rejected at the sender authority boundary
-
-### Ownership vs execution
-
-Ownership creation is not the same as execution start.
-
-So:
-
- `AttachSession()` and `SupersedeSession()` establish ownership only
- `BeginConnect()` is the first execution-state mutation
-
-### Completion authority
-
-An ID match alone is not enough to complete recovery.
-
-So:
-
- completion must require a valid completion-ready phase
- normal completion requires converged catch-up
- zero-gap fast completion is allowed explicitly from handshake
-
-## P2 Direction
-
-The next prototype step is not broader simulation.
-
-It is:
-
- recovery outcome branching
- assignment-intent orchestration
- prototype-level end-to-end recovery flow
-
-## Accepted P2 Refinements
-
-### Recovery boundary
-
-Recovery classification must use a lineage-safe boundary, not a raw primary WAL head.
-
-So:
-
- handshake outcome classification uses committed/safe recovery boundary
- stale or divergent extra tail must not be treated as zero-gap by default
-
-### Stale assignment fencing
-
-Assignment intent must not create current live sessions from stale epoch input.
-
-So:
-
- stale assignment epoch is rejected
- assignment result distinguishes:
-  - created
-  - superseded
-  - failed
-
-### Phase discipline on outcome classification
-
-The outcome API must respect execution entry rules.
-
-So:
-
- handshake-with-outcome requires valid connecting phase before acting
-
-## P3 Direction
-
-The next prototype step is:
-
- minimal historical-data model
- recoverability proof
- explicit safe-boundary / divergent-tail handling
-
-## Accepted P3 Refinements
-
-### Recoverability proof
-
-The historical-data prototype must prove why catch-up is allowed.
-
-So:
-
- recoverability now checks retained start, end within head, and contiguous coverage
- rebuild fallback is backed by executable unrecoverability
-
-### Historical state after recycling
-
-Retained-prefix modeling needs a base state, not only remaining WAL entries.
-
-So:
-
- tail advance captures a base snapshot
- historical state reconstruction uses snapshot + retained WAL
-
-### Divergent tail handling
-
-Replica-ahead state must not collapse directly to `InSync`.
-
-So:
-
- divergent tail requires explicit truncation
- completion is gated on recorded truncation when required
-
-## P4 Direction
-
-The next prototype step is:
-
- prototype scenario closure
- acceptance-criteria to prototype traceability
- explicit expression of the 4 V2-boundary cases against `enginev2`
-
-## Accepted P4 Refinements
-
-### Prototype scenario closure
-
-The prototype must stop being only a set of local mechanisms.
-
-So:
-
- acceptance criteria are mapped to prototype evidence
- key V2-boundary scenarios are expressed directly against `enginev2`
- prototype behavior is reviewable scenario-by-scenario
-
-### Phase 04 completion decision
-
-Phase 04 has now met its intended prototype scope:
-
- ownership
- execution gating
- outcome branching
- minimal historical-data model
- prototype scenario closure
-
-So:
-
- no broad new Phase 04 work should be added
- next work should move to `Phase 4.5` gate-hardening
@@ -1,76 +0,0 @@
-# Phase 04 Log
-
-Date: 2026-03-27
-Status: complete
-
-## 2026-03-27
-
- Phase 04 created to start the first standalone V2 implementation slice.
- Decision:
-  - do not begin in `weed/storage/blockvol/`
-  - begin under `sw-block/`
- first slice chosen:
-  - per-replica sender ownership
-  - explicit recovery-session ownership
- Initial slice delivered under `sw-block/prototype/enginev2/`:
-  - sender
-  - recovery session
-  - sender group
- First review found:
-  - sender/session epoch coherence gap
-  - session lifecycle was shell-only, not enforcing real transitions
-  - attach-session epoch mismatch was not rejected
- Follow-up delivered and accepted:
-  - reconcile updates preserved sender epoch
-  - epoch bump invalidates stale session
-  - session transition map enforced
-  - attach-session rejects epoch mismatch
-  - enginev2 tests increased to 26 passing
- Phase 04a created to close the ownership-validation gap:
-  - explicit session identity in `distsim`
-  - bridge tests into `enginev2`
- Phase 04a ownership problem closed well enough:
-  - stale completion rejected by session ID
-  - endpoint invalidation includes `CtrlAddr`
-  - boundary doc aligned with real simulator/prototype evidence
- Phase 04 P1 delivered and accepted:
-  - sender-owned execution APIs added
-  - all execution APIs fence on `sessionID`
-  - completion now requires valid completion point
-  - attach/supersede now establish ownership only
-  - handshake range validation added
-  - enginev2 tests increased to 46 passing
- Phase 04 P2 delivered and accepted:
-  - outcome branching added:
-    - `OutcomeZeroGap`
-    - `OutcomeCatchUp`
-    - `OutcomeNeedsRebuild`
-  - assignment-intent orchestration added
-  - stale assignment epoch now rejected
-  - assignment result now distinguishes created / superseded / failed
-  - end-to-end prototype recovery tests added
-  - zero-gap classification tightened:
-    - exact equality to committed boundary only
-    - replica-ahead is not zero-gap
-  - enginev2 tests increased to 63 passing
- Phase 04 P3 delivered and accepted:
-  - `WALHistory` added as minimal historical-data model
-  - recoverability proof strengthened:
-    - retained start
-    - end within head
-    - contiguous coverage
-  - base snapshot added for correct `StateAt()` after tail advance
-  - divergent-tail truncation made explicit in sender/session execution
-  - WAL-backed prototype recovery tests added
-  - enginev2 tests increased to 83 passing
- Phase 04 P4 delivered and accepted:
-  - acceptance criteria mapped to prototype evidence
-  - V2-boundary scenarios expressed against `enginev2`
-  - prototype scenario closure achieved
-  - enginev2 tests increased to 95 passing
- Phase 04 is now complete for its intended prototype scope.
- Next recommended phase:
-  - `Phase 4.5`
-  - tighten bounded `CatchUp`
-  - formalize `Rebuild`
-  - strengthen crash-consistency / recoverability / liveness proof
@@ -1,216 +0,0 @@
-# Phase 04
-
-Date: 2026-03-27
-Status: complete
-Purpose: start the first standalone V2 implementation slice under `sw-block/`, centered on per-replica sender ownership and explicit recovery-session ownership
-
-## Goal
-
-Build the first real V2 implementation slice without destabilizing V1.
-
-This slice should prove:
-
-1. per-replica sender identity
-2. explicit one-session-per-replica recovery ownership
-3. endpoint/assignment-driven recovery updates
-4. clean handoff between normal sender and recovery session
-
-## Why This Phase Exists
-
-The simulator and design work are now strong enough to support a narrow implementation slice.
-
-We should not start with:
-
- Smart WAL
- new storage engine
- frontend integration
-
-We should start with the ownership problem that most clearly separates V2 from V1.5.
-
-## Source Of Truth
-
-Design:
- `sw-block/docs/archive/design/v2-first-slice-session-ownership.md`
- `sw-block/design/v2-acceptance-criteria.md`
- `sw-block/design/v2-open-questions.md`
-
-Simulator reference:
- `sw-block/prototype/distsim/`
-
-## Scope
-
-### In scope
-
-1. per-replica sender owner object
-2. explicit recovery session object
-3. session lifecycle rules
-4. endpoint update handling
-5. basic tests for sender/session ownership
-
-### Out of scope
-
- Smart WAL in production code
- real block backend redesign
- V1 integration
- frontend publication
-
-## Assigned Tasks For `sw`
-
-### P0
-
-1. create standalone V2 implementation area under `sw-block/`
- recommended:
-  - `sw-block/prototype/enginev2/`
-
-2. define sender/session types
- sender owner per replica
- recovery session per replica per epoch
-
-3. implement basic lifecycle
- create sender
- attach session
- supersede stale session
- close session on success / invalidation
-
-## Current Progress
-
-Delivered in this phase so far:
-
- standalone V2 area created under:
-  - `sw-block/prototype/enginev2/`
- core types added:
-  - `Sender`
-  - `RecoverySession`
-  - `SenderGroup`
- sender/session lifecycle shell implemented
- per-replica ownership implemented
- endpoint-change invalidation implemented
- sender epoch coherence implemented
- session epoch attach validation implemented
- session phase transitions now enforce a real transition map
- session identity fencing implemented
- stale completion rejected by session ID
- execution APIs implemented:
-  - `BeginConnect`
-  - `RecordHandshake`
-  - `RecordHandshakeWithOutcome`
-  - `BeginCatchUp`
-  - `RecordCatchUpProgress`
-  - `CompleteSessionByID`
- completion authority tightened:
-  - catch-up must converge
-  - zero-gap handshake fast path allowed
- attach/supersede now establish ownership only
- sender-group orchestration tests added
- recovery outcome branching implemented:
-  - `OutcomeZeroGap`
-  - `OutcomeCatchUp`
-  - `OutcomeNeedsRebuild`
- assignment-intent orchestration implemented:
-  - reconcile + recovery target session creation
-  - stale assignment epoch rejected
-  - created/superseded/failed outcomes distinguished
- P2 data-boundary correction accepted:
-  - zero-gap now requires exact equality to committed boundary
-  - replica-ahead is not zero-gap
- minimal historical-data prototype implemented:
-  - `WALHistory`
-  - retained-prefix / recycled-range semantics
-  - executable recoverability proof
-  - base snapshot for historical state after tail advance
- explicit safe-boundary handling implemented:
-  - divergent tail requires truncation before `InSync`
-  - truncation recorded via sender-owned execution API
- WAL-backed prototype tests added:
-  - catch-up recovery with data verification
-  - rebuild fallback with proof of unrecoverability
-  - truncate-then-`InSync` with committed-boundary verification
- current `enginev2` test state at latest review:
-  - 95 tests passing
- prototype scenario closure completed:
-  - acceptance criteria mapped to prototype evidence
-  - V2-boundary scenarios expressed against `enginev2`
-  - small end-to-end prototype harness added
-
-Next phase:
-
- `Phase 4.5`
-  - bounded `CatchUp`
-  - first-class `Rebuild`
-  - crash-consistency / recoverability / liveness proof hardening
- do not integrate into V1 production tree yet
-
-### P1
-
-4. implement endpoint update handling
- changed-address update must refresh the right sender owner
-
-5. implement epoch invalidation
- stale session must stop after epoch bump
-
-6. add tests matching the slice acceptance
-
-### P2
-
-7. add recovery outcome branching
- distinguish:
-  - zero-gap fast completion
-  - positive-gap catch-up completion
-  - unrecoverable gap / `NeedsRebuild`
-
-8. add assignment-intent driven orchestration
- move beyond raw reconcile-only tests
- make sender-group react to explicit recovery intent
-
-9. add prototype-level end-to-end flow tests
- assignment/update
- session creation
- execution
- completion / invalidation
- rebuild escalation
-
-### P3
-
-10. add minimal historical-data prototype
- retained prefix/window
- minimal recoverability state
- explicit "why catch-up is allowed" proof
-
-11. make safe-boundary data handling explicit
- divergent tail cleanup / truncate rule
- or equivalent explicit boundary handling before `InSync`
-
-12. strengthen recoverability/rebuild tests
- executable proof of:
- recoverable gap
- unrecoverable gap
- rebuild fallback boundary
-
-### P4
-
-13. close prototype scenario coverage
- map key acceptance criteria onto `enginev2` scenarios/tests
- make prototype evidence reviewable scenario-by-scenario
-
-14. express the 4 V2-boundary cases against the prototype
- changed-address identity-preserving recovery
- `NeedsRebuild` persistence
- catch-up without overwriting safe data
- repeated disconnect/reconnect cycles
-
-15. add one small prototype harness if needed
- enough to show assignment -> recovery -> outcome flow end-to-end
- no product/backend integration yet
-
-## Exit Criteria
-
-Phase 04 is done when:
-
-1. standalone V2 sender/session slice exists under `sw-block/`
-2. sender ownership is per replica, not set-global
-3. one active recovery session per replica per epoch is enforced
-4. endpoint update and epoch invalidation are tested
-5. sender-owned execution flow is validated
-6. recovery outcome branching exists at prototype level
-7. minimal historical-data / recoverability model exists at prototype level
-8. prototype scenario closure is achieved for key V2 acceptance cases
@@ -1,49 +0,0 @@
-# Phase 04a Decisions
-
-Date: 2026-03-27
-Status: initial
-
-## Core Decision
-
-The next must-fix validation problem is:
-
- sender/session ownership semantics
-
-This outranks:
-
- more timing realism
- more WAL detail
- broader scenario growth
-
-## Why
-
-V2's core claim over V1.5 is not only:
-
- better recovery policy
-
-It is also:
-
- stable per-replica sender identity
- one active recovery owner
- stale work cannot mutate current state
-
-If those ownership rules are not validated, the simulator can overstate confidence.
-
-## Validation Rule
-
-For this phase, a scenario is only complete when it is expressed at two levels:
-
-1. simulator ownership model (`distsim`)
-2. standalone implementation slice (`enginev2`)
-
-Real `weed/` adversarial tests remain the system-level gate.
-
-## Scope Discipline
-
-Do not expand this phase into:
-
- generic simulator feature growth
- Smart WAL design growth
- V1 integration work
-
-Keep it focused on the ownership model.
@@ -1,22 +0,0 @@
-# Phase 04a Log
-
-Date: 2026-03-27
-Status: active
-
-## 2026-03-27
-
- Phase 04a created as a narrow validation phase.
- Reason:
-  - the biggest remaining V2 validation gap is ownership semantics
-  - not general scenario count
-  - not more timer realism
-  - not more WAL detail
- Scope chosen:
-  - sender identity
-  - recovery session identity
-  - supersede / invalidate rules
-  - stale completion rejection
-  - `distsim` to `enginev2` bridge tests
- This phase is intentionally separate from broad Phase 04 implementation growth.
- Goal:
-  - gain confidence that V2 is validated as owned session/sender protocol state, not only as policy
@@ -1,113 +0,0 @@
-# Phase 04a
-
-Date: 2026-03-27
-Status: active
-Purpose: close the critical V2 ownership-validation gap by making sender/session ownership explicit in both simulation and the standalone `enginev2` slice
-
-## Goal
-
-Validate the core V2 claim more deeply:
-
-1. one stable sender identity per replica
-2. one active recovery session per replica
-3. endpoint change, epoch bump, and supersede rules invalidate stale work
-4. stale late results from old sessions cannot mutate current state
-
-This phase is not about adding broad new simulator surface.
-It is about proving the ownership model that is supposed to make V2 better than V1.5.
-
-## Why This Phase Exists
-
-Current simulation is already strong on:
-
- quorum / commit rules
- stale epoch rejection
- catch-up vs rebuild
- timeout / race ordering
- changed-address recovery at the policy level
-
-The remaining critical risk is narrower:
-
- the simulator still validates V2 strongly as policy
- but not yet strongly enough as owned sender/session protocol state
-
-That is the highest-value validation gap to close before trusting V2 too much.
-
-## Source Of Truth
-
-Design:
- `sw-block/docs/archive/design/v2-first-slice-session-ownership.md`
- `sw-block/design/v2-acceptance-criteria.md`
- `sw-block/design/v2-open-questions.md`
- `sw-block/design/protocol-development-process.md`
-
-Simulator / prototype:
- `sw-block/prototype/distsim/`
- `sw-block/prototype/enginev2/`
-
-Historical / review context:
- `learn/projects/sw-block/phases/phase-13-v2-boundary-tests.md`
- `sw-block/design/v2-scenario-sources-from-v1.md`
-
-## Scope
-
-### In scope
-
-1. explicit sender/session identity validation in `distsim`
-2. explicit stale-session invalidation rules
-3. bridge tests from `distsim` scenarios to `enginev2` sender/session invariants
-4. doc cleanup so V2-boundary tests point to real simulator and `enginev2` coverage
-
-### Out of scope
-
- Smart WAL expansion
- broad new timing realism
- TCP / disk realism
- V1 production integration
- new backend/storage engine work
-
-## Critical Questions To Close
-
-1. can an old session completion mutate state after a new session supersedes it?
-2. does endpoint change invalidate or supersede the active session cleanly?
-3. does epoch bump remove all authority from prior sessions?
-4. can duplicate recovery triggers create overlapping active sessions?
-
-## Assigned Tasks For `sw`
-
-### P0
-
-1. add explicit session identity to `distsim`
- model session ID or equivalent ownership token
- make stale session results rejectable by identity, not just by coarse state
-
-2. add ownership scenarios to `distsim`
- endpoint change during active catch-up
- epoch bump during active catch-up
- stale late completion from old session
- duplicate recovery trigger while a session is already active
-
-3. add bridge tests in `enginev2`
- same-address reconnect preserves sender identity
- endpoint bump supersedes or invalidates active session
- epoch bump rejects stale completion
- only one active session per sender
-
-### P1
-
-4. tighten `learn/projects/sw-block/phases/phase-13-v2-boundary-tests.md`
- point to actual `distsim` scenarios
- point to actual `enginev2` bridge tests
- state what remains real-engine-only
-
-5. only add simulator mechanics if a bridge test exposes a real ownership gap
-
-## Exit Criteria
-
-Phase 04a is done when:
-
-1. `distsim` explicitly validates sender/session ownership invariants
-2. `enginev2` has bridge tests for the same invariants
-3. stale session work is shown unable to mutate current sender state
-4. V2-boundary doc no longer has stale simulator references
-5. we can say with confidence that V2 ownership semantics, not just V2 policy, are validated at prototype level
@@ -1,94 +0,0 @@
-# Phase 05 Decisions
-
-## Decision 1: Real V2 engine work lives under `sw-block/engine/replication/`
-
-The first real engine slice is established under:
-
- `sw-block/engine/replication/`
-
-This keeps V2 separate from:
-
- `sw-block/prototype/`
- `weed/storage/blockvol/`
-
-## Decision 2: Slice 1 is accepted
-
-Accepted scope:
-
-1. stable per-replica sender identity
-2. stable recovery-session identity
-3. stale authority fencing
-4. endpoint / epoch invalidation
-5. ownership registry
-
-## Decision 3: Stable identity must not be address-shaped
-
-The engine registry is now keyed by stable `ReplicaID`, not mutable endpoint address.
-
-This is a required structural break from the V1/V1.5 identity-loss pattern.
-
-## Decision 4: Slice 2 is accepted
-
-Accepted scope:
-
-1. connect / handshake / catch-up flow
-2. zero-gap / catch-up / needs-rebuild branching
-3. stale execution rejection during active recovery
-4. bounded catch-up semantics in engine path
-5. rebuild execution shell
-
-## Decision 5: Slice 3 owns real recoverability inputs
-
-Slice 3 should be the point where:
-
-1. recoverable vs unrecoverable gap uses real engine inputs
-2. trusted-base / rebuild-source decision uses real engine data inputs
-3. truncation / safe-boundary handling is tied to real engine state
-4. historical correctness at recovery target is validated from engine inputs
-
-## Decision 6: Slice 3 is accepted
-
-Accepted scope:
-
-1. real engine recoverability input path
-2. trusted-base / rebuild-source decision from engine data inputs
-3. truncation / safe-boundary handling tied to engine state
-4. recoverability gating without overclaiming full historical reconstruction in engine
-
-## Decision 7: Slice 3 should replace carried-forward heuristics where appropriate
-
-In particular:
-
-1. simple rebuild-source heuristics carried from prototype should not become permanent engine policy
-2. Slice 3 should tighten these decisions against real engine recoverability inputs
-
-## Decision 8: Slice 4 is the engine integration closure slice
-
-Next focus:
-
-1. real assignment/control intent entry path
-2. engine observability / debug surface
-3. focused integration tests for V2-boundary cases
-4. validation against selected real failure classes from `learn/projects/sw-block/` and `weed/storage/block*`
-
-## Decision 9: Slice 4 is accepted
-
-Accepted scope:
-
-1. real orchestrator entry path
-2. assignment/update-driven recovery through that path
-3. engine observability / causal recovery logging
-4. diagnosable V2-boundary integration tests
-
-## Decision 10: Phase 05 is complete
-
-Reason:
-
-1. ownership core is accepted
-2. recovery execution core is accepted
-3. data / recoverability core is accepted
-4. integration closure is accepted
-
-Next:
-
- `Phase 06` broader engine implementation stage
@@ -1,78 +0,0 @@
-# Phase 05 Log
-
-## 2026-03-29
-
-### Opened
-
-`Phase 05` opened as:
-
- V2 engine planning + Slice 1 ownership core
-
-### Accepted
-
-1. engine module location
-   - `sw-block/engine/replication/`
-
-2. Slice 1 ownership core
-   - stable per-replica sender identity
-   - stable recovery-session identity
-   - sender/session fencing
-   - endpoint / epoch invalidation
-   - ownership registry
-
-3. Slice 1 identity correction
-   - registry now keyed by stable `ReplicaID`
-   - mutable `Endpoint` separated from identity
-   - real changed-`DataAddr` preservation covered by test
-
-4. Slice 1 encapsulation
-   - mutable sender/session authority state no longer exposed directly
-   - snapshot/read-only inspection path in place
-
-5. Slice 2 recovery execution core
-   - connect / handshake / catch-up flow
-   - explicit zero-gap / catch-up / needs-rebuild branching
-   - stale execution rejection during active recovery
-   - bounded catch-up semantics
-   - rebuild execution shell
-
-6. Slice 2 validation
-   - corrected tester summary accepted
-   - `12` ownership tests + `18` recovery tests = `30` total
-   - Slice 2 accepted for progression to Slice 3 planning
-
-7. Slice 3 data / recoverability core
-   - `RetainedHistory` introduced as engine-level recoverability input
-   - history-driven sender APIs added for handshake and rebuild-source selection
-   - trusted-base decision now requires both checkpoint trust and replayable tail
-   - truncation remains a completion gate / protocol boundary
-
-8. Slice 3 validation
-   - corrected tester summary accepted
-   - `12` ownership tests + `18` recovery tests + `18` recoverability tests = `48` total
-   - accepted boundary:
-     - engine proves historical-correctness prerequisites
-     - simulator retains stronger historical reconstruction proof
-   - Slice 3 accepted for progression to Slice 4 planning
-
-9. Slice 4 integration closure
-   - `RecoveryOrchestrator` added as integrated engine entry path
-   - assignment/update-driven recovery is exercised through orchestrator
-   - observability surface added:
-     - `RegistryStatus`
-     - `SenderStatus`
-     - `SessionSnapshot`
-     - `RecoveryLog`
-   - causal recovery logging now covers invalidation, escalation, truncation, completion, rebuild transitions
-
-10. Slice 4 validation
-   - corrected tester summary accepted
-   - `12` ownership tests + `18` recovery tests + `18` recoverability tests + `11` integration tests = `59` total
-   - Slice 4 accepted
-   - `Phase 05` accepted as complete
-
-### Next
-
-1. `Phase 06` planning
-2. broader engine implementation stage
-3. real-engine integration against selected `weed/storage/block*` constraints and failure classes
@@ -1,356 +0,0 @@
-# Phase 05
-
-Date: 2026-03-29
-Status: complete
-Purpose: begin the real V2 engine track under `sw-block/` by moving from prototype proof to the first engine slice
-
-## Why This Phase Exists
-
-The project has now completed:
-
-1. V2 design/FSM closure
-2. V2 protocol/simulator validation
-3. Phase 04 prototype closure
-4. Phase 4.5 evidence hardening
-
-So the next step is no longer:
-
- extend prototype breadth
-
-The next step is:
-
- start disciplined real V2 engine work
-
-## Phase Goal
-
-Start the real V2 engine line under `sw-block/` with:
-
-1. explicit engine module location
-2. Slice 1 ownership-core boundaries
-3. first engine ownership-core implementation
-4. engine-side validation tied back to accepted prototype invariants
-
-## Relationship To Previous Phases
-
-`Phase 05` is built on:
-
- `sw-block/docs/archive/design/v2-engine-readiness-review.md`
- `sw-block/docs/archive/design/v2-engine-slicing-plan.md`
- `sw-block/.private/phase/phase-04.md`
- `sw-block/.private/phase/phase-4.5.md`
-
-This is a new implementation phase.
-
-It is not:
-
-1. more prototype expansion
-2. V1 integration
-3. backend redesign
-
-## Scope
-
-### In scope
-
-1. choose real V2 engine module location under `sw-block/`
-2. define Slice 1 file/module boundaries
-3. write short engine ownership-core spec
-4. start Slice 1 implementation:
-   - stable per-replica sender object
-   - stable recovery-session object
-   - session identity fencing
-   - endpoint / epoch invalidation
-   - ownership registry / sender-group equivalent
-5. add focused engine-side ownership/fencing tests
-
-### Out of scope
-
-1. Smart WAL expansion
-2. full storage/backend redesign
-3. full rebuild-source decision logic
-4. V1 production integration
-5. performance work
-6. full product integration
-
-## Planned Slices
-
-### P0: Engine Planning Setup
-
-1. choose real V2 engine module location under `sw-block/`
-2. define Slice 1 file/module boundaries
-3. write ownership-core spec
-4. map 3-5 acceptance scenarios to Slice 1 expectations
-
-Status:
-
- accepted
- engine module location chosen: `sw-block/engine/replication/`
- Slice 1 boundaries are explicit enough to start implementation
-
-### P1: Slice 1 Ownership Core
-
-1. implement stable per-replica sender object
-2. implement stable recovery-session object
-3. implement sender/session identity fencing
-4. implement endpoint / epoch invalidation
-5. implement ownership registry
-
-Status:
-
- accepted
- stable `ReplicaID` is now explicit and separate from mutable `Endpoint`
- engine registry is keyed by stable identity, not address-shaped strings
- real changed-`DataAddr` preservation is covered by test
-
-### P2: Slice 1 Validation
-
-1. engine-side tests for ownership/fencing
-2. changed-address case
-3. stale-session rejection case
-4. epoch-bump invalidation case
-5. traceability back to accepted prototype behavior
-
-Status:
-
- accepted
- Slice 1 ownership/fencing tests are in place and passing
- acceptance/gate mapping is strong enough to move to Slice 2
-
-### P3: Slice 2 Planning Setup
-
-1. define Slice 2 boundaries explicitly
-2. distinguish Slice 2 core from carried-forward prototype support
-3. map Slice 2 engine expectations from accepted prototype evidence
-4. prepare Slice 2 validation targets
-
-Status:
-
- accepted
- Slice 2 recovery execution core is implemented and validated
- corrected tester summary accepted:
-  - `12` ownership tests
-  - `18` recovery tests
-  - `30` total
-
-### P4: Slice 3 Planning Setup
-
-1. define Slice 3 boundaries explicitly
-2. connect recovery decisions to real engine recoverability inputs
-3. make trusted-base / rebuild-source decision use real engine data inputs
-4. prepare Slice 3 validation targets
-
-Status:
-
- accepted
- Slice 3 data / recoverability core is implemented and validated
- corrected tester summary accepted:
-  - `12` ownership tests
-  - `18` recovery tests
-  - `18` recoverability tests
-  - `48` total
- important boundary preserved:
-  - engine proves historical-correctness prerequisites
-  - full historical reconstruction proof remains simulator-side
-
-## Slice 3 Guardrails
-
-Slice 3 is the point where V2 must move from:
-
- recovery automaton is coherent
-
-to:
-
- recovery basis is provable
-
-So Slice 3 must stay tight.
-
-### Guardrail 1: No optimistic watermark in place of recoverability proof
-
-Do not accept:
-
- loose head/tail watermarks
- "looks retained enough"
- heuristic recoverability
-
-Slice 3 should prove:
-
-1. why a gap is recoverable
-2. why a gap is unrecoverable
-
-### Guardrail 2: No current extent state pretending to be historical correctness
-
-Do not accept:
-
- current extent image as substitute for target-LSN truth
- checkpoint/base state that leaks newer state into older historical queries
-
-Slice 3 should prove historical correctness at the actual recovery target.
-
-### Guardrail 3: No `snapshot + tail` without trusted-base proof
-
-Do not accept:
-
- "snapshot exists" as sufficient
-
-Require:
-
-1. trusted base exists
-2. trusted base covers the required base state
-3. retained tail can be replayed continuously from that base to the target
-
-If not, recovery must use:
-
- `FullBase`
-
-### Guardrail 4: Truncation is protocol boundary, not cleanup policy
-
-Do not treat truncation as:
-
- optional cleanup
- post-recovery tidying
-
-Treat truncation as:
-
-1. divergent tail removal
-2. explicit safe-boundary restoration
-3. prerequisite for safe `InSync` / recovery completion where applicable
-
-### P5: Slice 4 Planning Setup
-
-1. define Slice 4 boundaries explicitly
-2. connect engine control/recovery core to real assignment/control intent entry path
-3. add engine observability / debug surface for ownership and recovery failures
-4. prepare integration validation against V2-boundary failure classes
-
-Status:
-
- accepted
- Slice 4 integration closure is implemented and validated
- corrected tester summary accepted:
-  - `12` ownership tests
-  - `18` recovery tests
-  - `18` recoverability tests
-  - `11` integration tests
-  - `59` total
-
-## Slice 4 Guardrails
-
-Slice 4 should close integration, not just add an entry point and some logs.
-
-### Guardrail 1: Entry path must actually drive recovery
-
-Do not accept:
-
- tests that manually push sender/session state while only pretending to use integration entry points
-
-Require:
-
-1. real assignment/control intent entry path
-2. session creation / invalidation / restart triggered through that path
-3. recovery flow driven from that path, not only from unit-level helper calls
-
-### Guardrail 2: Changed-address must survive the real entry path
-
-Do not accept:
-
- changed-address correctness proven only at local object level
-
-Require:
-
-1. stable `ReplicaID` survives real assignment/update entry path
-2. endpoint update invalidates old session correctly
-3. new recovery session is created correctly on updated endpoint
-
-### Guardrail 3: Observability must show protocol causality
-
-Do not accept:
-
- only state snapshots
- only phase dumps
-
-Require observability that can explain:
-
-1. why recovery entered `NeedsRebuild`
-2. why a session was superseded
-3. why a completion or progress update was rejected
-4. why endpoint / epoch change caused invalidation
-
-### Guardrail 4: Failure replay must be explainable
-
-Do not accept:
-
- a replay that reproduces failure but cannot explain the cause from engine observability
-
-Require:
-
-1. selected failure-class replays through the real entry path
-2. observability sufficient to explain the control/recovery decision
-3. reviewability against key V2-boundary failures
-
-## Exit Criteria
-
-Phase 05 Slice 1 is done when:
-
-1. the real V2 engine module location is chosen
-2. Slice 1 boundaries are explicit
-3. engine ownership core exists under `sw-block/`
-4. engine-side ownership/fencing tests pass
-5. Slice 1 evidence is reviewable against prototype expectations
-
-This bar is now met.
-
-Phase 05 Slice 2 is done when:
-
-1. engine-side recovery execution flow exists
-2. zero-gap / catch-up / needs-rebuild branching is explicit
-3. stale execution is rejected during active recovery
-4. bounded catch-up semantics are enforced in engine path
-5. rebuild execution shell is validated
-
-This bar is now met.
-
-Phase 05 Slice 3 is done when:
-
-1. recoverable vs unrecoverable gap uses real engine recoverability inputs
-2. trusted-base / rebuild-source decision uses real engine data inputs
-3. truncation / safe-boundary handling is tied to real engine state
-4. history-driven engine APIs exist for recovery decisions
-5. Slice 3 validation is reviewable without overclaiming full historical reconstruction
-
-This bar is now met.
-
-Phase 05 Slice 4 is done when:
-
-1. real assignment/control intent entry path exists
-2. changed-address recovery works through the real entry path
-3. observability explains protocol causality, not only state snapshots
-4. selected V2-boundary failures are replayable and diagnosable through engine integration tests
-
-This bar is now met.
-
-## Assignment For `sw`
-
-Phase 05 is now complete.
-
-Next phase:
-
- `Phase 06` broader engine implementation stage
-
-## Assignment For `tester`
-
-Phase 05 validation is complete.
-
-Next phase:
-
- `Phase 06` engine implementation validation against real-engine constraints and failure classes
-
-## Management Rule
-
-`Phase 05` should stay narrow.
-
-It should start the engine line with:
-
-1. ownership
-2. fencing
-3. validation
-
-It should not try to absorb later slices early.
@@ -1,68 +0,0 @@
-# Phase 06 Decisions
-
-## Decision 1: Phase 06 is broader engine implementation, not new design
-
-The protocol shape and engine core contracts were already accepted.
-
-Phase 06 implemented around them.
-
-## Decision 2: Phase 06 must connect to real constraints
-
-This phase explicitly used:
-
-1. `learn/projects/sw-block/` for failure gates and test lineage
-2. `weed/storage/block*` for real implementation constraints
-
-without importing V1 structure as the V2 design template.
-
-## Decision 3: Phase 06 should replace key synchronous conveniences
-
-The accepted Slice 4 convenience flows were sufficient for closure work, but broader engine work required real step boundaries.
-
-This is now satisfied via planner/executor separation.
-
-## Decision 4: Phase 06 ends with a runnable engine stage decision
-
-Result:
-
- yes, the project now has a broader runnable engine stage that is ready to proceed to real-system integration / product-path work
-
-## Decision 5: Phase 06 P0 is accepted
-
-Accepted scope:
-
-1. adapter/module boundaries
-2. convenience-flow classification
-3. initial real-engine stage framing
-
-## Decision 6: Phase 06 P1 is accepted
-
-Accepted scope:
-
-1. storage/control adapter interfaces
-2. `RecoveryDriver` planner/resource-acquisition layer
-3. full-base and WAL retention resource contracts
-4. fail-closed preconditions on planning paths
-
-## Decision 7: Phase 06 P2 is accepted
-
-Accepted scope:
-
-1. explicit planner/executor split on top of `RecoveryPlan`
-2. executor-owned cleanup symmetry on success/failure/cancellation
-3. plan-bound rebuild execution with no policy re-derivation at execute time
-4. synchronous orchestrator completion helpers remain test-only convenience
-
-## Decision 8: Phase 06 P3 is accepted
-
-Accepted scope:
-
-1. selected real failure classes validated through the engine path
-2. cross-layer engine/storage proof validation
-3. diagnosable failure when proof or resource acquisition cannot be established
-
-## Decision 9: Phase 06 is complete
-
-Next step:
-
- `Phase 07` real-system integration / product-path decision
@@ -1,51 +0,0 @@
-# Phase 06 Log
-
-## 2026-03-30
-
-### Opened
-
-`Phase 06` opened as:
-
- broader engine implementation stage
-
-### Starting basis
-
-1. `Phase 05`: complete
-2. engine core and integration closure accepted
-3. next work moves from slice proof to broader runnable engine stage
-
-### Accepted
-
-1. Phase 06 P0
-   - adapter/module boundaries defined
-   - convenience flows explicitly classified
-
-2. Phase 06 P1
-   - storage/control adapter surfaces defined
-   - `RecoveryDriver` added as planner/resource-acquisition layer
-   - full-base rebuild now has explicit resource contract
-   - WAL pin contract tied to actual recovery need
-   - driver preconditions fail closed
-
-3. Phase 06 P2
-   - explicit planner/executor split accepted
-   - executor owns release symmetry on success, failure, and cancellation
-   - rebuild execution now consumes plan-bound source/target values
-   - tester final validation accepted with reduced-but-sufficient rebuild failure-path coverage
-
-4. Phase 06 P3
-   - selected real failure classes validated through the engine path
-   - changed-address restart now uses plan cancellation and re-plan flow
-   - stale execution is caught through the executor-managed loop
-   - cross-layer trusted-base / replayable-tail proof path validated end-to-end
-   - rebuild planning failures now clean up sessions and remain diagnosable
-
-### Closed
-
-`Phase 06` closed as complete.
-
-### Next
-
-1. Phase 07 real-system integration / product-path decision
-2. service-slice integration against real control/storage surroundings
-3. first product-path gating decision
@@ -1,193 +0,0 @@
-# Phase 06
-
-Date: 2026-03-30
-Status: complete
-Purpose: move from validated engine slices to the first broader runnable V2 engine stage
-
-## Why This Phase Exists
-
-`Phase 05` established and validated:
-
-1. ownership core
-2. recovery execution core
-3. recoverability/data gating core
-4. integration closure
-
-What still does not exist is a broader engine stage that can run with:
-
-1. real control-plane inputs
-2. real persistence/backing inputs
-3. non-trivial execution loops instead of only synchronous convenience paths
-
-So `Phase 06` exists to turn the accepted engine shape into the first broader runnable engine stage.
-
-Phase 06 must connect the accepted engine core to real control and real storage truth, not just wrap current abstractions with adapters.
-
-## Phase Goal
-
-Build the first broader V2 engine stage without reopening protocol shape.
-
-This phase should focus on:
-
-1. real engine adapters around the accepted core
-2. asynchronous or stepwise execution paths where Slice 4 used synchronous helpers
-3. real retained-history / checkpoint input plumbing
-4. validation against selected real failure classes and real implementation constraints
-
-## Overall Roadmap
-
-Completed:
-
-1. Phase 01-03: design + simulator
-2. Phase 04: prototype closure
-3. Phase 4.5: evidence hardening
-4. Phase 05: engine slice closure
-5. Phase 06: broader engine implementation stage
-
-Next:
-
-1. Phase 07: real-system integration / product-path decision
-
-This roadmap should stay strict:
-
- no return to broad prototype expansion
- no uncontrolled engine sprawl
-
-## Scope
-
-### In scope
-
-1. control-plane adapter into `sw-block/engine/replication/`
-2. retained-history / checkpoint adapter into engine recoverability APIs
-3. replacement of synchronous convenience flows with explicit engine steps where needed
-4. engine error taxonomy and observability tightening
-5. validation against selected real failure classes from:
-   - `learn/projects/sw-block/`
-   - `weed/storage/block*`
-
-### Out of scope
-
-1. Smart WAL expansion
-2. full backend redesign
-3. performance optimization as primary goal
-4. V1 replacement rollout
-5. full product integration
-
-## Phase 06 Items
-
-### P0: Engine Stage Plan
-
-Status:
-
- accepted
- module boundaries now explicit:
-  - `adapter.go`
-  - `driver.go`
-  - `orchestrator.go` classification
- convenience flows are now classified as:
-  - test-only convenience wrapper
-  - stepwise engine task
-  - planner/executor split
-
-### P1: Control / History Adapters
-
-Status:
-
- accepted
- `StorageAdapter` boundary exists and is exercised by tests
- full-base rebuild now has a real pin/release contract
- WAL pinning is tied to actual recovery contract, not loose watermark use
- planner fails closed on missing sender / missing session / wrong session kind
-
-### P2: Execution Driver
-
-Status:
-
- accepted
- executor now owns resource lifecycle on success / failure / cancellation
- catch-up execution is stepwise and budget-checked per progress step
- rebuild execution consumes plan-bound source/target values and does not re-derive policy at execute time
- `CompleteCatchUp` / `CompleteRebuild` remain test-only convenience wrappers
- tester validation accepted with reduced-but-sufficient rebuild failure-path coverage
-
-### P3: Validation Against Real Failure Classes
-
-Status:
-
- accepted
- changed-address restart now validated through planner/executor path with plan cancellation
- stale epoch/session during active execution now validated through the executor-managed loop
- cross-layer trusted-base / replayable-tail proof path validated end-to-end
- rebuild fallback and pin-failure cleanup now fail closed and are diagnosable
-
-## Guardrails
-
-### Guardrail 1: Do not reopen protocol shape
-
-Phase 06 implemented around accepted engine slices and did not reopen:
-
-1. sender/session authority model
-2. bounded catch-up contract
-3. recoverability/truncation boundary
-
-### Guardrail 2: Do not let adapters smuggle V1 structure back in
-
-V1 code and docs remain:
-
-1. constraints
-2. failure gates
-3. integration references
-
-not the V2 architecture template.
-
-### Guardrail 3: Prefer explicit engine steps over synchronous convenience
-
-Key convenience helpers remain test-only. Real engine work now has explicit planner/executor boundaries.
-
-### Guardrail 4: Keep evidence quality high
-
-Phase 06 improved:
-
-1. cross-layer traceability
-2. diagnosability
-3. real-failure validation
-
-without growing protocol surface.
-
-### Guardrail 5: Do not fake storage truth with metadata-only adapters
-
-Phase 06 now requires:
-
-1. trusted base to come from storage-side truth
-2. replayable tail to be grounded in retention state
-3. observable rejection when those proofs cannot be established
-
-## Exit Criteria
-
-Phase 06 is done when:
-
-1. engine has real control/history adapters into the accepted core
-2. engine has real storage/base adapters into the accepted core
-3. key synchronous convenience paths are explicitly classified or replaced by real engine steps where necessary
-4. selected real failure classes are validated against the engine stage
-5. at least one cross-layer storage/engine proof path is validated end-to-end
-6. engine observability remains good enough to explain recovery causality
-
-Status:
-
- met
-
-## Closeout
-
-`Phase 06` is complete.
-
-It established:
-
-1. a broader runnable engine stage around the accepted Phase 05 core
-2. real planner/executor/resource contracts
-3. validated failure-class behavior through the engine path
-4. diagnosable proof rejection and cleanup behavior
-
-Next step:
-
- `Phase 07` real-system integration / product-path decision
@@ -1,119 +0,0 @@
-# Phase 07 Decisions
-
-## Decision 1: Phase 07 is real-system integration, not protocol redesign
-
-The V2 protocol shape, engine core, and broader runnable engine stage are already accepted.
-
-Phase 07 should integrate them into a real-system service slice.
-
-## Decision 2: Phase 07 should make the first product-path decision
-
-This phase should not only integrate a service slice.
-
-It should also decide:
-
-1. what the first product path is
-2. what remains before pre-production hardening
-
-## Decision 3: Phase 07 must preserve accepted V2 boundaries
-
-Phase 07 should preserve:
-
-1. narrow catch-up semantics
-2. rebuild as the formal recovery path
-3. trusted-base / replayable-tail proof boundaries
-4. stable identity / fenced execution / diagnosable failure handling
-
-## Decision 4: Phase 07 P0 service-slice direction is set
-
-Current direction:
-
-1. first service slice = `RF=2` block volume primary + one replica
-2. engine remains in `sw-block/engine/replication/`
-3. current bridge work starts in `sw-block/bridge/blockvol/`
-4. deferred real blockvol-side bridge target = `weed/storage/blockvol/v2bridge/`
-5. stable identity mapping is explicit:
-   - `ReplicaID = <volume-name>/<server-id>`
-6. `blockvol` executes I/O but does not own recovery policy
-
-## Decision 5: Phase 07 P1 is accepted with explicit scope limits
-
-Accepted `P1` coverage is:
-
-1. real reader mapping from `BlockVol` state
-2. real retention hold / release wiring into the flusher retention floor
-3. one real WAL catch-up scan path through `v2bridge`
-4. direct real-adapter tests under `weed/storage/blockvol/v2bridge/`
-
-This acceptance means:
-
-1. the real bridge path is now integrated and evidenced
-2. `P1` is not yet acceptance proof of general post-checkpoint catch-up viability
-
-Not accepted as part of `P1`:
-
-1. snapshot transfer execution
-2. full-base transfer execution
-3. WAL truncation execution
-4. master-side confirmed failover / control-intent integration
-
-## Decision 6: Interim committed-truth limitation remains active
-
-`Phase 07 P1` is accepted with an explicit carry-forward limitation:
-
-1. interim `CommittedLSN = CheckpointLSN` is a service-slice mapping, not final V2 protocol truth
-2. post-checkpoint catch-up semantics are therefore narrower than final V2 intent
-3. later `Phase 07` work must not overclaim this limitation as solved until commit truth is separated from checkpoint truth
-
-## Decision 7: Phase 07 P2 is accepted with scoped replay claims
-
-Accepted `P2` coverage is:
-
-1. real service-path replay for changed-address restart
-2. stale epoch / stale session invalidation through the integrated path
-3. unrecoverable-gap / needs-rebuild replay with diagnosable proof
-4. explicit replay of the post-checkpoint boundary under the interim model
-
-Not accepted as part of `P2`:
-
-1. general integrated engine-driven post-checkpoint catch-up semantics
-2. real control-plane delivery from master heartbeat into the bridge
-3. rebuild execution beyond the already-deferred executor stubs
-
-## Decision 8: Phase 07 now moves to product-path choice, not more bridge-shape proof
-
-With `P0`, `P1`, and `P2` accepted, the next step is:
-
-1. choose the first product path from accepted service-slice evidence
-2. define what remains before pre-production hardening
-3. keep unresolved limits explicit rather than hiding them behind broader claims
-
-## Decision 7: Phase 07 P2 must replay the interim limitation explicitly
-
-`Phase 07 P2` should not only replay happy-path or ordinary failure-path integration.
-
-It should also include one explicit replay where:
-
-1. the live bridge path is exercised after checkpoint truth has advanced
-2. the observed catch-up limitation is diagnosed as a consequence of the interim mapping
-3. the result is not overclaimed as proof of final V2 post-checkpoint catch-up semantics
-
-## Decision 10: Phase 07 P3 is accepted and Phase 07 is complete
-
-The first V2 product path is now explicitly chosen as:
-
-1. `RF=2`
-2. `sync_all`
-3. existing master / volume-server heartbeat path
-4. V2 engine owns recovery policy
-5. `v2bridge` provides real storage truth
-
-This decision is accepted with explicit non-claims:
-
-1. not production-ready
-2. no real master-side control delivery proof yet
-3. no full rebuild execution proof yet
-4. no general post-checkpoint catch-up proof yet
-5. no full integrated engine -> executor -> `v2bridge` catch-up proof yet
-
-Phase 07 is therefore complete, and the next phase is pre-production hardening.
@@ -1,63 +0,0 @@
-# Phase 07 Log
-
-## 2026-03-30
-
-### Opened
-
-`Phase 07` opened as:
-
- real-system integration / product-path decision
-
-### Starting basis
-
-1. `Phase 06`: complete
-2. broader runnable engine stage accepted
-3. next work moves from engine-stage validation to real-system service-slice integration
-
-### Delivered
-
-1. Phase 07 P0
-   - service-slice plan defined
-   - implementation slice proposal delivered
-   - bridge layer introduced as:
-     - `sw-block/bridge/blockvol/` for current bridge work
-     - `weed/storage/blockvol/v2bridge/` as the deferred real integration target
-   - stable identity mapping made explicit:
-     - `ReplicaID = <volume-name>/<server-id>`
-   - engine / blockvol policy boundary made explicit
-   - initial bridge tests delivered (`8`)
-2. Phase 07 P1
-   - real blockvol reader integrated via `weed/storage/blockvol/v2bridge/reader.go`
-   - real pinner integrated via `weed/storage/blockvol/v2bridge/pinner.go`
-   - one real catch-up executor path integrated via `weed/storage/blockvol/v2bridge/executor.go`
-   - direct real-adapter tests delivered in:
-     - `weed/storage/blockvol/v2bridge/bridge_test.go`
-   - accepted with explicit carry-forward:
-     - interim `CommittedLSN = CheckpointLSN` limits post-checkpoint catch-up semantics and is not final V2 commit truth
-     - acceptance is for the real integrated bridge path, not for general post-checkpoint catch-up viability
-3. Phase 07 P2
-   - real service-path failure replay accepted
-   - accepted replay set includes:
-     - changed-address restart
-     - stale epoch / stale session invalidation
-     - unrecoverable-gap / needs-rebuild replay
-     - explicit post-checkpoint boundary replay
-   - evidence kept explicitly scoped:
-     - real `v2bridge` WAL-scan execution proven
-     - general integrated post-checkpoint catch-up semantics not overclaimed under the interim model
-4. Phase 07 P3
-   - product-path decision accepted
-   - first product path chosen as:
-     - `RF=2`
-     - `sync_all`
-     - existing master / volume-server heartbeat path
-     - V2 engine recovery ownership with `v2bridge` real storage truth
-   - pre-hardening prerequisites made explicit
-   - intentional deferrals and non-claims recorded
-   - `Phase 07` completed
-
-### Next
-
-1. Phase 08 pre-production hardening
-2. real master/control delivery integration
-3. integrated catch-up / rebuild execution closure
@@ -1,220 +0,0 @@
-# Phase 07
-
-Date: 2026-03-30
-Status: complete
-Purpose: connect the broader runnable V2 engine stage to a real-system service slice and decide the first product path
-
-## Why This Phase Exists
-
-`Phase 06` completed the broader runnable engine stage:
-
-1. planner/executor/resource contracts are real
-2. selected real failure classes are validated through the engine path
-3. cross-layer trusted-base / replayable-tail proof path is validated
-
-What still does not exist is a real-system slice where the engine runs inside actual service boundaries with real control/storage surroundings.
-
-So `Phase 07` exists to answer:
-
-1. how the engine runs as a real subsystem
-2. what the first product path should be
-3. what integration risks remain before pre-production hardening
-
-## Phase Goal
-
-Establish a real-system integration slice for the V2 engine and make the first product-path decision without reopening protocol shape.
-
-## Scope
-
-### In scope
-
-1. service-slice integration around `sw-block/engine/replication/`
-2. real control-plane / lifecycle entry path into the engine
-3. real storage-side adapter hookup into existing system boundaries
-4. selected real-system failure replay and diagnosis
-5. explicit product-path decision framing
-
-### Out of scope
-
-1. broad performance optimization
-2. Smart WAL expansion
-3. full V1 replacement rollout
-4. broad backend redesign
-5. production rollout itself
-
-## Phase 07 Items
-
-### P0: Service-Slice Plan
-
-1. define the first real-system service slice that will host the engine
-2. define adapter/module boundaries at the service boundary
-3. choose the concrete integration path to exercise first
-4. identify which current adapters are still mock/test-only and must be replaced first
-5. make the first-slice identity/epoch mapping explicit
-6. treat `blockvol` as execution backend only, not recovery-policy owner
-
-Status:
-
- delivered
- planning artifact:
-  - `sw-block/docs/archive/design/phase-07-service-slice-plan.md`
- implementation slice proposal:
-  - engine core: `sw-block/engine/replication/`
-  - bridge adapters: `sw-block/bridge/blockvol/`
-  - real blockvol integration target: `weed/storage/blockvol/v2bridge/` (`P1`)
- adapter replacement order:
-  - `control_adapter.go` (`P0`) done
-  - `storage_adapter.go` (`P0`) done
-  - `executor_bridge.go` (`P1`) deferred
-  - `observe_adapter.go` (`P1`) deferred
- first-slice identity mapping is explicit:
-  - `ReplicaID = <volume-name>/<server-id>`
-  - not derived from any address field
- engine / blockvol boundary is explicit:
-  - bridge maps intent and state
-  - `blockvol` executes I/O
-  - `blockvol` does not own recovery policy
- service-slice validation gaps called out for `P1`:
-  - real blockvol field mapping
-  - real pin/release lifecycle against reclaim/GC
-  - assignment timing vs engine session lifecycle
-  - executor bridge into real WAL/snapshot work
-
-### P1: Real Entry-Path Integration
-
-1. connect real control/lifecycle events into the engine entry path
-2. connect real storage/base/recoverability signals into the engine adapters
-3. preserve accepted engine authority/execution/recoverability contracts
-
-Status:
-
- accepted
- real integration now established for:
-  - reader via `weed/storage/blockvol/v2bridge/reader.go`
-  - pinner via `weed/storage/blockvol/v2bridge/pinner.go`
-  - catch-up executor path via `weed/storage/blockvol/v2bridge/executor.go`
- direct real-adapter tests now exist in:
-  - `weed/storage/blockvol/v2bridge/bridge_test.go`
- accepted scope is explicit:
-  - real reader
-  - real retention hold / release
-  - real WAL catch-up scan path
-  - direct real bridge evidence for the integrated path
- still deferred:
-  - `TransferSnapshot`
-  - `TransferFullBase`
-  - `TruncateWAL`
-  - control intent from confirmed failover / master-side integration
- carry-forward limitation:
-  - under interim `CommittedLSN = CheckpointLSN`, this slice proves a real bridge path, not general post-checkpoint catch-up viability
-  - post-checkpoint catch-up semantics therefore remain narrower than final V2 intent and do not represent final V2 commit semantics
-
-### P2: Real-System Failure Replay
-
-1. replay selected real failure classes against the integrated service slice
-2. confirm diagnosability from logs/status
-3. identify any remaining mismatch between engine-stage assumptions and real system behavior
-
-Status:
-
- accepted
- real service-path replay now accepted for:
-  - changed-address restart
-  - stale epoch / stale session invalidation
-  - unrecoverable-gap / needs-rebuild replay
-  - explicit post-checkpoint boundary replay under the interim model
- accepted with scoped limitation:
-  - real `v2bridge` WAL-scan execution is proven
-  - full integrated engine-driven catch-up semantics are not overclaimed under interim `CommittedLSN = CheckpointLSN`
- control-plane delivery remains simulated via direct `AssignmentIntent` construction
- carry-forward remains explicit:
-  - post-checkpoint catch-up semantics are still narrower than final V2 intent
-
-### P3: Product-Path Decision
-
-1. choose the first product path for V2
-2. define what remains before pre-production hardening
-3. record what is still intentionally deferred
-
-Status:
-
- accepted
- first product path chosen:
-  - `RF=2`
-  - `sync_all`
-  - existing master / volume-server heartbeat path
-  - V2 engine owns recovery policy
-  - `v2bridge` provides real storage truth
- proposal is evidence-grounded and explicitly bounded by accepted `P0/P1/P2` evidence
- pre-hardening prerequisites are explicit:
-  - real master control delivery
-  - full integrated engine -> executor -> `v2bridge` catch-up chain
-  - separation of committed truth from checkpoint truth
-  - rebuild execution (`snapshot` / `full-base` / `truncation`)
-  - pinner / flusher behavior under concurrent load
- intentionally deferred:
-  - `RF>2`
-  - Smart WAL optimizations
-  - `best_effort` background recovery
-  - performance tuning
-  - full V1 replacement
- non-claims remain explicit:
-  - not production-ready
-  - no end-to-end rebuild proof yet
-  - no general post-checkpoint catch-up proof
-  - no real master heartbeat/control delivery proof yet
-  - no full integrated engine -> executor -> `v2bridge` catch-up proof yet
-
-## Guardrails
-
-### Guardrail 1: Do not re-import V1 structure as the design owner
-
-Use `weed/storage/block*` and `learn/projects/sw-block/` as constraints and validation sources, not as the architecture template.
-
-### Guardrail 2: Keep catch-up narrow and rebuild explicit
-
-Do not use integration work as an excuse to widen catch-up semantics or blur rebuild as the formal recovery path.
-
-### Guardrail 3: Prefer real entry paths over test-only wrappers
-
-The integrated slice should exercise real service boundaries, not only internal engine helpers.
-
-### Guardrail 4: Observability must explain causality
-
-Integrated logs/status must explain:
-
-1. why rebuild was required
-2. why proof was rejected
-3. why execution was cancelled or invalidated
-4. why a product-path integration failed
-
-### Guardrail 5: Stable identity must not collapse back to address shape
-
-For the first slice, `ReplicaID` must be derived from master/block-registry identity, not current endpoint addresses.
-
-### Guardrail 6: `blockvol` executes I/O but does not own recovery policy
-
-The service bridge may translate engine decisions into concrete blockvol actions, but it must not re-decide:
-
-1. zero-gap / catch-up / rebuild
-2. trusted-base validity
-3. replayable-tail sufficiency
-4. rebuild fallback requirement
-
-## Exit Criteria
-
-Phase 07 is done when:
-
-1. one real-system service slice is integrated with the engine
-2. selected real-system failure classes are replayed through that slice
-3. diagnosability is sufficient for service-slice debugging
-4. the first product path is explicitly chosen
-5. the remaining work to pre-production hardening is clear
-
-## Assignment For `sw`
-
-Next tasks move to `Phase 08`.
-
-## Assignment For `tester`
-
-Next tasks move to `Phase 08`.
@@ -1,187 +0,0 @@
-# Phase 08 Decisions
-
-## Decision 1: Phase 08 is pre-production hardening, not protocol rediscovery
-
-The accepted V2 product path from `Phase 07` is the basis.
-
-`Phase 08` should harden that path rather than reopen accepted protocol shape.
-
-## Decision 2: The first hardening priorities are control delivery and execution closure
-
-The most important remaining gaps are:
-
-1. real master/control delivery into the bridge/engine path
-2. integrated engine -> executor -> `v2bridge` catch-up execution closure
-3. first rebuild execution path for the chosen product path
-
-## Decision 3: Carry-forward limitations remain explicit until closed
-
-Phase 08 must keep explicit:
-
-1. committed truth is still not separated from checkpoint truth
-2. rebuild execution is still incomplete
-3. current control delivery is still simulated
-
-## Decision 4: Phase 08 P0 is accepted
-
-The hardening plan is sufficiently specified to begin implementation work.
-
-In particular, `P0` now fixes:
-
-1. the committed-truth gate decision requirement
-2. the unified replay requirement after control and execution closure
-3. the need for at least one real failover / reassignment validation target
-## Decision 5: The committed-truth limitation must become a hardening gate
-
-Phase 08 must explicitly decide one of:
-
-1. `CommittedLSN != CheckpointLSN` separation is mandatory before a production-candidate phase
-2. the first candidate path is intentionally bounded to the currently proven pre-checkpoint replay behavior
-
-It must not remain only a documented carry-forward.
-
-## Decision 6: Unified-path replay is required after control and execution closure
-
-Once real control delivery and integrated execution closure land, `Phase 08` must replay the accepted failure-class set again on the unified live path.
-
-This prevents independent closure of:
-
-1. control delivery
-2. execution closure
-
-without proving that they behave correctly together.
-
-## Decision 7: Real failover / reassignment validation is mandatory for the chosen path
-
-Because the chosen product path depends on the existing master / volume-server heartbeat path, at least one real failover / promotion / reassignment cycle must be a named hardening target in `Phase 08`.
-
-## Decision 8: Phase 08 should reuse the existing Seaweed control/runtime path, not invent a new one
-
-For the first hardening path, implementation should preferentially reuse:
-
-1. existing master / heartbeat / assignment delivery
-2. existing volume-server assignment receive/apply path
-3. existing `blockvol` runtime and `v2bridge` storage/runtime hooks
-
-This reuse is about:
-
-1. control-plane reality
-2. storage/runtime reality
-3. execution-path reality
-
-It is not permission to inherit old policy semantics as V2 truth.
-
-The hard rule remains:
-
-1. engine owns recovery policy
-2. bridge translates confirmed control/storage truth
-3. `blockvol` executes I/O
-
-## Decision 9: Phase 08 P1 is accepted with explicit scope limits
-
-Accepted `P1` coverage is:
-
-1. real `ProcessAssignments()` path drives V2 engine sender/session state change
-2. stable remote `ReplicaID` is derived from `ServerID`, not address
-3. address change preserves sender identity through the live control path
-4. stale epoch/session invalidation occurs through the live control path
-5. missing `ServerID` fails closed
-
-Not accepted as part of `P1`:
-
-1. full end-to-end gRPC heartbeat delivery proof
-2. integrated catch-up execution through the live path
-3. rebuild execution through the live path
-4. final local stable identity beyond transport-shaped `listenAddr`
-
-## Decision 10: Phase 08 P2 is accepted as real execution closure
-
-Accepted `P2` coverage is:
-
-1. `CommittedLSN` is separated from `CheckpointLSN` on the chosen `sync_all` path
-2. catch-up is proven as one live chain:
-   - engine plan
-   - engine executor
-   - `v2bridge`
-   - real `blockvol` I/O
-   - completion
-   - cleanup
-3. rebuild is proven as one live chain for the delivered path
-4. cleanup/pin release is asserted after execution
-
-Residual non-blocking scope notes:
-
-1. `CatchUpStartLSN` is not directly asserted in tests
-2. rebuild source variants are not all forced and individually asserted
-
-## Decision 11: Phase 08 now moves to unified hardening validation
-
-With `P1` and `P2` accepted, the next required step is:
-
-1. replay the accepted failure-class set again on the unified live path
-2. validate at least one real failover / reassignment cycle
-3. validate concurrent retention/pinner behavior
-4. make the committed-truth gate decision explicit for the chosen candidate path
-
-## Decision 12: Phase 08 P3 is accepted as unified hardening validation
-
-Accepted `P3` coverage is:
-
-1. replay of the accepted failure-class set on the unified `P1` + `P2` live path
-2. at least one real failover / reassignment cycle through the live control path
-3. one true simultaneous-overlap retention/pinner safety proof
-4. stronger causality assertions for invalidation, escalation, catch-up, and completion
-
-## Decision 13: The committed-truth gate is decided for the chosen candidate path
-
-For the chosen `RF=2 sync_all` candidate path:
-
-1. `CommittedLSN = WALHeadLSN`
-2. `CheckpointLSN` remains the durable base-image boundary
-3. this separation is accepted as sufficient for the candidate-path hardening boundary
-
-This decision is intentionally scoped:
-
-1. it is accepted for the chosen candidate path
-2. it is not yet a blanket truth for every future path or durability mode
-
-## Decision 14: Phase 08 P4 is candidate-path judgment, not broad new engineering expansion
-
-`P4` should close `Phase 08` by producing one explicit candidate-path judgment.
-
-Its main output is not more isolated engineering progress, but:
-
-1. a bounded candidate-path statement
-2. an evidence-to-claim mapping from accepted `P1` / `P2` / `P3` results
-3. an explicit list of accepted bounds, remaining deferrals, and production blockers
-
-`P4` may include small closure work if needed to make the candidate statement coherent, but it should not reopen protocol design or grow into another broad hardening slice.
-
-## Decision 15: Phase 08 P4 is accepted as candidate package closure
-
-Accepted `P4` coverage is:
-
-1. one explicit candidate package for the chosen `RF=2 sync_all` path
-2. candidate-safe claims mapped to accepted `P1` / `P2` / `P3` evidence
-3. explicit bounds, deferred items, and production blockers
-4. committed-truth decision scoped to the chosen candidate path
-5. module/package boundary summary for the next heavy engineering phase
-
-Accepted judgment:
-
-1. candidate-safe-with-bounds
-2. not production-ready
-
-## Decision 16: Phase 08 is closed and the next heavy phase is production execution closure
-
-With `P0` through `P4` accepted, `Phase 08` is closed.
-
-The next phase should not be a light packaging-only round.
-It should begin with:
-
-1. `Phase 09: Production Execution Closure`
-2. `P0` planning for:
-   - real `TransferFullBase`
-   - real `TransferSnapshot`
-   - real `TruncateWAL`
-   - stronger live runtime execution ownership
@@ -1,414 +0,0 @@
-# Phase 08 Log
-
-## 2026-03-31
-
-### Opened
-
-`Phase 08` opened as:
-
- pre-production hardening
-
-### Starting basis
-
-1. `Phase 07`: complete
-2. first V2 product path chosen
-3. remaining gaps are integration and hardening gaps, not protocol-discovery gaps
-
-### Next
-
-1. Phase 08 P0 accepted
-2. Phase 08 P1 accepted
-3. Phase 08 P2 accepted
-4. Phase 08 P3 hardening validation on the unified live path
-5. Phase 08 P4 candidate package closure accepted
-6. Phase 08 closeout bookkeeping complete
-7. next: open Phase 09 P0 for production execution closure planning
-
-### P3 Technical Pack
-
-Purpose:
-
- provide the minimum design/algo/test detail needed to execute `P3`
- reuse accepted `P1` / `P2` live-path closure
- avoid broad scenario growth or repeated proof of already accepted mechanics
-
-#### Design / algo focus
-
-`P3` is not another execution-closure slice.
-It assumes these are already accepted on the chosen path:
-
- real control delivery
- real catch-up one-chain closure
- real rebuild one-chain closure
-
-What `P3` adds is hardening evidence on top of that live path:
-
-1. replay accepted failure classes again on the unified path
-2. prove one real failover / reassignment cycle
-3. prove one overlapping retention/pinner safety case
-4. produce one explicit committed-truth gate decision
-
-Key algorithm rules for `P3`:
-
- control truth remains primary:
-  - failover / reassignment is driven by new assignment / epoch truth
-  - storage/runtime must not invent role changes
- recovery choice remains engine-owned:
-  - engine chooses `zero_gap` / `catchup` / `needs_rebuild`
-  - bridge and `blockvol` execute what the engine already decided
- overlapping recovery must remain fail-closed:
-  - retained floor = minimum active retention requirement
-  - stale or cancelled plan must release its hold
-  - a new authoritative plan must not inherit leaked resources from an old one
- committed-truth gate must be output, not discussed informally:
-  - either the chosen candidate path is accepted with current committed/checkpoint semantics
-  - or the next phase is blocked on further separation/bounding work
-
-#### Validation matrix
-
-Use one compact replay matrix rather than many near-duplicate tests.
-
-1. Changed-address restart
-   - trigger: address refresh / reassignment while prior identity is preserved
-   - expected: old session invalidated, same logical `ReplicaID`, new recovery starts cleanly
-   - assert:
-     - no stale session mutation
-     - no leaked pins
-     - logs show why identity stayed and session changed
-
-2. Stale epoch / stale session
-   - trigger: epoch bump during or before recovery continuation
-   - expected: stale execution loses authority immediately
-   - assert:
-     - old session cannot mutate
-     - replacement assignment/session becomes the only live authority
-     - logs show invalidation reason
-
-3. Unrecoverable gap / needs-rebuild
-   - trigger: replica falls behind retained WAL
-   - expected: engine chooses `needs_rebuild`, rebuild path executes or is prepared according to accepted boundary
-   - assert:
-     - no catch-up overclaim
-     - correct rebuild source/result logged
-     - no leaked pins after completion/failure
-
-4. Post-checkpoint boundary behavior
-   - trigger: replica state around checkpoint / committed boundary
-   - expected: classification and execution match the chosen candidate-path semantics
-   - assert:
-     - chosen path does not overclaim beyond the accepted boundary
-     - committed/checkpoint truth used here matches the explicit gate decision
-
-#### Required extra cases
-
-Besides the replay matrix, `P3` should add only two new validation cases:
-
-1. One real failover / promotion / reassignment cycle
-   - primary change or reassignment through the live control path
-   - verify old authority dies, new authority starts, recovery resumes/starts correctly
-
-2. One true simultaneous-overlap retention/pinner case
-   - two live recovery holds coexist before the earlier one is released
-   - verify:
-     - minimum retention floor is respected while both are live
-     - releasing one hold leaves the other hold still contributing the correct floor
-     - released/cancelled plan stops contributing to retention floor
-     - final hold count returns to zero
-
-#### Expected evidence
-
-For each accepted `P3` case, prefer explicit evidence blocks:
-
- entry truth:
-  - assignment / epoch / role that started the case
- engine result:
-  - selected outcome or invalidation result
- execution result:
-  - completion / cancel / failure
- cleanup result:
-  - `ActiveHoldCount() == 0`
-  - no surviving active session when case should be closed
- observability result:
-  - logs explain:
-    - why control truth changed
-    - why session changed
-    - why catch-up vs rebuild happened
-    - why execution completed / failed / cancelled
-
-#### Efficient test plan
-
-Keep `P3` small and high-signal:
-
- one unified replay test package or compact matrix
- one real failover-cycle test
- one overlapping-retention test
- one explicit gate-decision record in delivery / phase status
-
-Avoid:
-
- re-proving isolated `P2` one-chain mechanics
- broad combinatorial growth across many replicas / roles / timing permutations
- turning `P3` into another protocol-design slice
-
-### P4 Technical Pack
-
-Purpose:
-
- provide the minimum design/algo/test detail needed to close `Phase 08`
- convert accepted `P1` / `P2` / `P3` evidence into one candidate-path judgment
- keep `P4` as a closure slice, not another broad engineering slice
-
-#### Delivery sequence
-
-Use this order:
-
-1. `sw` develops the candidate package
-2. `architect` reviews code/claim shape before tester time is spent
-3. `tester` validates the evidence-to-claim mapping
-4. `manager` records the final phase/accounting decision
-
-Do not collapse these roles:
-
- `sw` builds the candidate statement and supporting artifacts
- `architect` checks whether the resulting package has obvious semantic, scope, or evidence-shape problems before tester validation
- `tester` checks whether every claim is actually supported
- `manager` decides acceptance/bookkeeping after architect + tester feedback
-
-Recommended handoff gate before tester:
-
- if architect finds obvious overclaim, missing evidence mapping, or broken candidate shape, return to `sw` first
- do not spend tester time on a package that is clearly not ready
-
-#### Design / algo focus
-
-`P4` should not introduce new protocol shape.
-It consumes already accepted results:
-
- `P1`: real control delivery
- `P2`: real execution closure
- `P3`: unified hardening validation
-
-The main design task is to classify the chosen path into three buckets:
-
-1. candidate-safe
-   - supported by accepted evidence
-   - allowed to appear in the candidate statement
-2. intentionally bounded
-   - accepted only within narrow limits
-   - must appear as explicit candidate bounds
-3. deferred or blocking
-   - not yet supported enough
-   - must not be implied as candidate-ready
-
-Algorithmically, `P4` is a classification/output slice:
-
- no new recovery FSM
- no new identity model
- no new rebuild policy
- no new durability model
-
-It should only:
-
- map accepted evidence to accepted candidate claims
- map residual limitations to explicit bounds or blockers
- separate candidate readiness from production readiness
-
-#### Required output artifacts
-
-`sw` should produce exactly these artifacts:
-
-1. Candidate statement
-   - what the chosen `RF=2 sync_all` path is allowed to claim
-
-2. Evidence-to-claim map
-   - each candidate claim points to accepted evidence from `P1` / `P2` / `P3`
-
-3. Bound list
-   - explicit candidate-safe bounds, for example:
-     - chosen path only
-     - chosen durability mode only
-     - accepted rebuild coverage only
-
-4. Deferred / blocking list
-   - what remains outside the candidate path
-   - what still blocks production readiness
-
-#### Candidate statement shape
-
-Keep the candidate statement short and structured.
-It should answer only:
-
-1. What path is the candidate?
-2. What is proven for that path?
-3. What is intentionally bounded for that path?
-4. What is still deferred or blocking?
-
-Good pattern:
-
- candidate path:
-  - `RF=2 sync_all` on the accepted master/heartbeat control path
- proven:
-  - real control delivery
-  - real catch-up closure
-  - real rebuild closure for accepted coverage
-  - unified replay and failover validation
- bounded:
-  - only the chosen path / mode
-  - only accepted rebuild/source coverage
- not yet claimed:
-  - general future path/mode truth
-  - production readiness
-
-#### Candidate statement template
-
-Use this exact structure for the `P4` delivery statement:
-
-1. Candidate path
-   - The first candidate path is:
-     - `<path / topology / durability mode>`
-
-2. Candidate-safe claims
-   - The candidate path is supported for:
-     - `<claim 1>` — evidence: `<P1/P2/P3 reference>`
-     - `<claim 2>` — evidence: `<P1/P2/P3 reference>`
-     - `<claim 3>` — evidence: `<P1/P2/P3 reference>`
-
-3. Explicit bounds
-   - This candidate statement is intentionally bounded to:
-     - `<bound 1>`
-     - `<bound 2>`
-     - `<bound 3>`
-
-4. Deferred or blocking items
-   - Not yet claimed as candidate-safe:
-     - `<deferred item 1>`
-     - `<deferred item 2>`
-   - Still blocking production readiness:
-     - `<blocker 1>`
-     - `<blocker 2>`
-
-5. Committed-truth decision
-   - For this candidate path:
-     - `<committed-truth decision>`
-   - Scope:
-     - `<why this does not automatically generalize>`
-
-6. Overall judgment
-   - Judgment:
-     - `<candidate-safe / candidate-safe-with-bounds / not-yet-candidate>`
-   - Reason:
-     - `<one short paragraph tying evidence to judgment>`
-
-When `sw` fills this template:
-
- every positive claim must carry an evidence reference
- every important missing area must appear either under:
-  - explicit bounds
-  - deferred
-  - blockers
- avoid prose that mixes candidate judgment with production-readiness language
-
-#### Assignment template
-
-Use this template when assigning `P4` work to `sw`:
-
-1. Goal
-   - Build the `P4` candidate package for the chosen path.
-
-2. Required outputs
-   - candidate statement
-   - evidence-to-claim mapping
-   - explicit bounds list
-   - deferred / blocking list
-   - committed-truth decision statement
-
-3. Hard rules
-   - no new protocol redesign
-   - no broad scope growth without candidate impact
-   - every positive claim must map to accepted `P1` / `P2` / `P3` evidence
-   - do not mix candidate readiness with production readiness
-
-4. Delivery order
-   - first hand to architect review
-   - only after architect review passes, hand to tester validation
-   - manager records final acceptance/bookkeeping last
-
-5. Reject before handoff if
-   - evidence-to-claim mapping is incomplete
-   - important limitations are not classified as bounded / deferred / blocking
-   - claims exceed accepted evidence
-
-Use this template when assigning `P4` validation to `tester`:
-
-1. Goal
-   - Validate that the candidate package is fully supported by accepted evidence.
-
-2. Validate
-   - each claim has accepted evidence
-   - each bound/deferred/blocker is explicit
-   - committed-truth decision stays scoped correctly
-   - no candidate-to-production overclaim exists
-
-3. Output
-   - pass/fail on each candidate claim group
-   - findings on unsupported claims, missing bounds, or hidden blockers
-
-#### Tester validation checklist
-
-`tester` should validate:
-
-1. every positive candidate claim has accepted evidence
-2. every important limitation appears in either:
-   - bounded
-   - deferred
-   - blocking
-3. no accepted evidence is stretched into a broader product claim
-4. committed-truth decision stays scoped to the chosen candidate path
-5. candidate readiness is not confused with production readiness
-
-#### Architect review focus
-
-`architect` should review only:
-
-1. semantic correctness of the candidate statement
-2. whether the evidence-to-claim mapping is honest
-3. whether bounds are explicit enough to prevent future drift
-4. whether any hidden overclaim remains
-
-This review should not reopen already accepted `P1` / `P2` / `P3` mechanics unless the candidate statement contradicts them.
-
-#### Efficient test / evidence plan
-
-`P4` should mostly reuse accepted evidence rather than add new broad tests.
-
-Preferred work:
-
- collect accepted evidence references
- compress them into candidate-safe claims
- write one explicit residual-gap list
-
-Only add new code/tests if a small missing blocker prevents a coherent candidate statement.
-
-Avoid:
-
- large new replay matrices
- new protocol experiments
- broad implementation growth without candidate impact
-
-### Closeout bookkeeping
-
-Manager follow-up after `P4` acceptance found only a minor bookkeeping concern:
-
- ensure `phase-08.md` is explicitly closed before treating `Phase 09` as opened
-
-Closeout check:
-
-1. `phase-08.md` is `Status: complete`
-2. `P4` is recorded as accepted
-3. `Phase-close note` points to `Phase 09: Production Execution Closure`
-4. `phase-08-decisions.md` records `Decision 16`
-
-Final bookkeeping judgment:
-
- `Phase 08` is closed
- `Phase 09 P0` is the active next planning/engineering package
@@ -1,535 +0,0 @@
-# Phase 08
-
-Date: 2026-03-31
-Status: complete
-Purpose: convert the accepted Phase 07 product path into a pre-production-hardening program without reopening accepted V2 protocol shape
-
-## Why This Phase Exists
-
-`Phase 07` completed:
-
-1. a real service-slice integration around the V2 engine
-2. real storage-truth bridge evidence through `v2bridge`
-3. selected real-system failure replay
-4. the first explicit product-path decision
-
-What still does not exist is a pre-production-ready system path. The remaining work is no longer protocol discovery. It is closing the operational and integration gaps between the accepted product path and a hardened deployment candidate.
-
-## Phase Goal
-
-Harden the first accepted V2 product path until the remaining gap to a production candidate is explicit, bounded, and implementation-driven.
-
-This phase doc is the canonical hardening contract for `sw` and `tester`.
-Use `phase-08-log.md` for deeper engineering process, alternatives, and implementation detail.
-
-Algorithm note:
-
- the accepted V2 algorithm / protocol shape is treated as fixed for this phase
- remaining work is engineering closure over real Seaweed/V1 runtime paths under V2 boundaries
- do not reopen protocol design unless a live contradiction is found
-
-## Scope
-
-### In scope
-
-1. real master/control delivery into the engine service path
-2. integrated engine -> executor -> `v2bridge` execution closure
-3. rebuild execution closure for the accepted product path
-4. operational/debuggability hardening
-5. concurrency/load validation around retention and recovery
-
-### Out of scope
-
-1. new protocol redesign
-2. `RF>2` coordination
-3. Smart WAL optimization work
-4. broad performance tuning beyond validation needed for hardening
-5. full V1 replacement rollout
-
-## Phase 08 Items
-
-### P0: Hardening Plan
-
-1. convert the accepted `Phase 07` product path into a hardening plan
-2. define the minimum pre-production gates
-3. order the remaining integration closures by risk
-4. make an explicit gate decision on committed truth vs checkpoint truth:
-   - either separate `CommittedLSN` from `CheckpointLSN` before a production-candidate phase
-   - or explicitly bound the first candidate path to the currently proven pre-checkpoint replay behavior
-
-Status:
-
- planning package accepted in this phase doc
- first hardening priorities are fixed as:
-  - real master/control delivery
-  - integrated engine -> executor -> `v2bridge` catch-up execution chain
-  - first rebuild execution path
- the committed-truth carry-forward is now a required hardening gate, not just a note:
-  - either separate `CommittedLSN` from `CheckpointLSN` before a production-candidate phase
-  - or explicitly bound the first candidate path to the currently proven pre-checkpoint replay behavior
- at least one real failover / promotion / reassignment cycle is a required hardening target
- once `P1` and `P2` land, the accepted failure-class set must be replayed again on the newly unified live path
- the validation oracle for `Phase 08` is expected to reject overclaiming around:
-  - catch-up semantics
-  - rebuild execution
-  - master/control delivery
-  - candidate-path readiness vs production readiness
- accepted
-
-Reference:
-
- `sw-block/docs/archive/design/phase-08-engine-skeleton-map.md` is the implementation-side skeleton map for this phase
- it is subordinate to `sw-block/design/v2-protocol-truths.md` and this `phase-08.md`; use it for module layout, execution order, interim fields, hard gates, and reuse guidance
-
-### P1: Real Control Delivery
-
-1. connect real master/heartbeat assignment delivery into the bridge
-2. replace direct `AssignmentIntent` construction for the first live path
-3. preserve stable identity and fenced authority through the real control path
-4. include at least one real failover / promotion / reassignment validation target on the chosen `sync_all` path
-
-Technical focus:
-
- keep the control-path split explicit:
-  - master confirms assignment / epoch / role
-  - bridge translates confirmed control truth into engine intent
-  - engine owns sender/session/recovery policy
-  - `blockvol` does not re-decide recovery policy
- preserve the identity rule through the live path:
-  - `ReplicaID = <volume>/<server>`
-  - endpoint change updates location but must not recreate logical identity
- preserve the fencing rule through the live path:
-  - stale epoch must invalidate old authority
-  - stale session must not mutate current lineage
-  - address change must invalidate the old live session before the new path proceeds
- treat failover / promotion / reassignment as control-truth events first, not storage-side heuristics
-
-Implementation route (`reuse map`):
-
- reuse directly as the first hardening carrier:
-  - `weed/server/master_grpc_server.go`
-  - `weed/server/volume_grpc_client_to_master.go`
-  - `weed/server/volume_server_block.go`
-  - `weed/server/master_block_registry.go`
-  - `weed/server/master_block_failover.go`
- reuse as storage/runtime execution reality:
-  - `weed/storage/blockvol/blockvol.go`
-  - `weed/storage/blockvol/replica_apply.go`
-  - `weed/storage/blockvol/replica_barrier.go`
-  - `weed/storage/blockvol/v2bridge/`
- preserve the V2 boundary while reusing these files:
-  - reuse transport/control/runtime reality
-  - do not inherit old policy semantics as V2 truth
-  - keep engine as the recovery-policy owner
-  - keep `blockvol` as the I/O executor
-
-Validation focus:
-
- prove live assignment delivery into the bridge/engine path
- prove stable `ReplicaID` across address refresh on the live path
- prove stale epoch / stale session invalidation through the live path
- prove at least one real failover / promotion / reassignment cycle on the chosen `sync_all` path
- prove the resulting logs explain:
-  - why reassignment happened
-  - why a session was invalidated
-  - which epoch / identity / endpoint drove the transition
-
-Reject if:
-
- address-shaped identity reappears anywhere in the control path
- bridge starts re-deriving catch-up vs rebuild policy from convenience inputs
- old epoch or old session can still mutate after the new control truth arrives
- failover / reassignment is claimed without a real replay target
- delivery claims general production readiness rather than control-path closure
-
-Status:
-
- accepted
- real assignment delivery into the V2 path is now proven through `ProcessAssignments()`
- accepted evidence includes:
-  - live assignment -> engine sender/session creation
-  - stable remote `ReplicaID = <volume>/<ServerID>`
-  - address-change identity preservation through the live path
-  - stale epoch/session invalidation through the live path
-  - fail-closed skip on missing `ServerID`
- accepted with explicit carry-forwards:
-  - `localServerID = listenAddr` remains transport-shaped for local identity
-  - heartbeat -> `ProcessAssignments()` is proven, but not full end-to-end gRPC delivery
-  - integrated catch-up execution is not yet proven through the live path
-  - rebuild execution remains deferred
-  - `CommittedLSN = CheckpointLSN` remains unresolved
-
-### P2: Execution Closure
-
-1. close the live engine -> executor -> `v2bridge` execution chain
-2. make catch-up execution evidence integrated rather than split across layers
-3. close the first rebuild execution path required by the product path
-
-Technical focus:
-
- keep execution ownership explicit:
-  - engine plans and owns recovery state transitions
-  - engine executor drives stepwise execution
-  - `v2bridge` translates execution requests into real blockvol work
-  - `blockvol` performs I/O only
- prove catch-up as one real path:
-  - accepted control delivery
-  - real retained-history input
-  - real WAL retention pin
-  - real WAL scan / progress return
-  - real session completion
- choose the narrowest rebuild closure required by the current product path:
-  - first real `full-base` rebuild path is preferred
-  - `snapshot + tail` can remain later unless needed by the chosen path
- keep resource ownership fail-closed:
-  - pin acquisition before execution
-  - release on success
-  - release on cancel / invalidation
-  - release on partial failure
- keep observability causal:
-  - execution start
-  - execution progress
-  - execution cancel / invalidation
-  - execution failure
-  - completion
-
-Implementation route:
-
- reuse engine-side execution core:
-  - `sw-block/engine/replication/driver.go`
-  - `sw-block/engine/replication/executor.go`
-  - `sw-block/engine/replication/orchestrator.go`
- reuse storage/runtime execution bridge:
-  - `weed/storage/blockvol/v2bridge/executor.go`
-  - `weed/storage/blockvol/v2bridge/pinner.go`
-  - `weed/storage/blockvol/v2bridge/reader.go`
- reuse block runtime execution reality:
-  - `weed/storage/blockvol/blockvol.go`
-  - `weed/storage/blockvol/replica_apply.go`
-  - `weed/storage/blockvol/replica_barrier.go`
-  - rebuild-side files under `weed/storage/blockvol/`
- preserve the boundary:
-  - do not move zero-gap / catch-up / rebuild classification into `blockvol`
-  - do not let executor convenience paths redefine protocol semantics
-
-Validation focus:
-
- prove one live integrated catch-up chain:
-  - assignment/control arrives through accepted `P1` path
-  - engine plans
-  - executor drives `v2bridge`
-  - `blockvol` executes
-  - progress returns
-  - session completes
- prove one real rebuild execution path for the chosen product path
- prove retention pin / release symmetry on the live path
- prove rebuild resource pin / release symmetry on the live path
- prove invalidation / cancel cleanup on the live path
- prove execution logs explain:
-  - why catch-up started
-  - why rebuild started
-  - why execution failed
-  - why execution was cancelled
-  - why completion succeeded
-
-Reject if:
-
- catch-up is still only proven by split evidence
- rebuild remains only a detection outcome
- `blockvol` starts deciding recovery mode or rebuild fallback
- resources leak on cancel / invalidation / partial failure
- execution logs are too weak to replay causality offline
- the slice quietly broadens protocol semantics beyond the current accepted boundary
-
-Recommended first cut:
-
-1. close the live catch-up chain first
-2. close the first real `full-base` rebuild path second
-3. leave unified replay to `P3`
-
-Minimum closure threshold:
-
- do not accept `P2` on glue code + partial chain tests alone
- at least one accepted catch-up proof must drive the real engine executor path:
-  - `PlanRecovery(...)`
-  - `NewCatchUpExecutor(...)`
-  - executor-managed progress / completion
-  - real `v2bridge` / `blockvol` execution underneath
- at least one accepted rebuild proof must drive the real engine executor path:
-  - rebuild assignment
-  - `PlanRebuild(...)`
-  - `NewRebuildExecutor(...)`
-  - executor-managed completion
-  - real `TransferFullBase(...)` underneath
- resource-cleanup proof must include live-path assertions, not only logs:
-  - active holds released
-  - retention floor no longer pinned after release
-  - no surviving session/plan ownership after cancel / invalidation / failure
- observability proof should include executor-generated events, not only planner-side events
- if these thresholds are not met, record `P2` as partial execution progress, not execution closure
-
-Carry-forward note:
-
- on the chosen `RF=2 sync_all` path, `CommittedLSN` separation is resolved in this slice:
-  - `CommittedLSN = WALHeadLSN`
-  - `CheckpointLSN` remains the durable base-image boundary
- this is not yet a blanket truth for every future path or durability mode
- post-checkpoint catch-up remains bounded unless explicitly closed
- rebuild coverage is limited to the first chosen executable path if that is all that lands
-
-Status:
-
- accepted
- real one-chain execution is now proven for:
-  - catch-up
-  - rebuild
- accepted evidence includes:
-  - `CommittedLSN` separated from `CheckpointLSN` on the chosen `sync_all` path
-  - live engine plan -> executor -> `v2bridge` -> `blockvol` catch-up chain
-  - live engine plan -> executor -> `v2bridge` -> `blockvol` rebuild chain
-  - explicit pin cleanup assertions after execution
- accepted with explicit residual scope:
-  - `CatchUpStartLSN` is not directly asserted in tests
-  - rebuild source is not yet forced/verified per source variant
-  - broader rebuild-source coverage can remain follow-up work
-
-Review checklist:
-
- is there one accepted catch-up proof from real `P1` control path to real session completion, using `CatchUpExecutor`
- is there one accepted first rebuild proof on the chosen path, using `RebuildExecutor`
- do live-path assertions prove pin/hold release on success, cancel, invalidation, and failure
- do logs/status explain start, cancel, failure, and completion without hidden transitions
- does the delivery avoid overclaiming general post-checkpoint catch-up, broad rebuild coverage, or production readiness
-
-### P3: Hardening Validation
-
-1. replay the accepted failure-class set again on the unified live path after `P1` + `P2`
-2. validate at least one real failover / promotion / reassignment cycle through the live control path
-3. validate concurrent retention/pinner behavior under overlapping recovery activity
-4. make the committed-truth gate decision explicit for the chosen candidate path
-
-Slice adjustment note:
-
- if `P2` lands only partially, `P3` should first close the missing execution outcome:
-  - real catch-up closure if still missing
-  - real first rebuild closure if still missing
- only after both are real should `P3` spend most of its weight on unified replay, failover / reassignment validation, and concurrent retention / cleanup hardening
-
-Efficiency note:
-
- `P3` is a hardening-validation slice, not another execution-closure slice
- reuse the accepted `P1` / `P2` live path as the base; do not re-prove already accepted chain mechanics in isolation
- prefer one compact replay matrix over many near-duplicate tests
- prefer one real failover cycle and one true simultaneous-overlap retention case over broad scenario expansion
- the required new outputs are:
-  - unified replay evidence
-  - one real failover / reassignment replay
-  - one concurrent retention/pinner safety result
-  - one explicit committed-truth gate decision
-
-Validation focus:
-
- unified replay for:
-  - changed-address restart
-  - stale epoch / stale session
-  - unrecoverable gap / needs-rebuild
-  - post-checkpoint boundary behavior
- at least one real failover / promotion / reassignment cycle
- concurrent retention/pinner safety under at least one true simultaneous-overlap hold case
- logs explain:
-  - why control truth changed
-  - why a session was invalidated
-  - why catch-up vs rebuild was chosen
-  - why execution completed, failed, or was cancelled
-
-Reject if:
-
- accepted failure classes are still only partially replayed on the unified path
- failover / reassignment is claimed without a real live-path replay
- concurrent retention/pinner behavior leaks pins or violates recovery safety
- logs are too weak to replay causality offline
- the committed-truth gate is still just a note instead of an explicit decision
-
-Status:
-
- accepted
- unified hardening replay is now proven on the accepted live path
- accepted evidence includes:
-  - replay of the accepted failure-class set on the unified `P1` + `P2` path
-  - at least one real failover / reassignment cycle through the live control path
-  - one true simultaneous-overlap retention/pinner safety proof
-  - stronger causality assertions for invalidation, escalation, catch-up, and completion
- committed-truth gate decision for the chosen candidate path:
-  - for the chosen `RF=2 sync_all` candidate path, `CommittedLSN = WALHeadLSN` with `CheckpointLSN` kept separate is accepted as sufficient for the candidate-path hardening boundary
-  - this is not yet a blanket truth for every future path or durability mode
-
-### P4: Candidate Package Closure
-
-1. classify what is truly ready for a first candidate path
-2. package the accepted `P1` / `P2` / `P3` evidence into one bounded candidate package
-3. turn carry-forwards into explicit candidate bounds or hard gates
-4. state clearly what still remains before production readiness
-
-Goal:
-
- finish `Phase 08` with one explicit candidate package, not just a collection of accepted slices
-
-Verification mechanism:
-
- evidence map:
-  - every candidate claim must point to accepted evidence from `P1` / `P2` / `P3`
- tester validation:
-  - verify each candidate claim is supported by accepted evidence
-  - reject any claim that exceeds the proven boundary
- manager validation:
-  - verify the candidate statement is explicit, bounded, and not confused with production readiness
-
-Output artifacts:
-
-1. candidate-path statement in `phase-08.md`
-2. candidate/gate decision record in `phase-08-decisions.md`
-3. concise candidate package summary:
-   - candidate-safe capabilities
-   - explicit bounds
-   - deferred / blocking items
-4. concise residual-gap summary:
-   - candidate-safe
-   - intentionally bounded
-   - still deferred / still blocking
-5. short module/package boundary summary for later phases:
-   - what is already strong enough
-   - what moves to the next heavy engineering phase
-
-Efficiency note:
-
- `P4` should mostly consume already accepted evidence, not create broad new engineering work
- only add implementation work if a small remaining blocker must be closed to make the candidate statement coherent
- if a gap is real but not worth closing in `Phase 08`, classify it explicitly rather than expanding scope implicitly
- `P4` exists inside `Phase 08` so the next phase can begin with substantial engineering work, not a light packaging-only round
-
-Validation focus:
-
- make the candidate-path boundary explicit:
-  - what is proven
-  - what is intentionally bounded
-  - what is still deferred
- make the candidate package explicit:
-  - candidate-safe capability list
-  - evidence-to-claim mapping
-  - short module/package boundary summary
- make the committed-truth decision explicit:
-  - accepted for the chosen `RF=2 sync_all` candidate path
-  - still unclassified for future paths / durability modes unless separately proven
- prove the accepted product path can be described as an engineering candidate, not only as a set of slice-local proofs
- provide one explicit residual-gap list that separates:
-  - candidate-safe bounds
-  - future hardening work
-  - production blockers
-
-Reject if:
-
- `P4` reopens protocol design instead of closing engineering gaps
- candidate claims are broader than the proven path
- carry-forwards remain informal notes rather than bounds or gates
- production readiness is implied from candidate readiness
- `P4` produces only prose summary without an evidence-to-claim mapping
- `P4` is too thin to leave the next phase with substantial engineering closure work
-
-Status:
-
- accepted
- the first candidate package is now explicit for the chosen path
- accepted evidence includes:
-  - candidate-safe claims mapped to accepted `P1` / `P2` / `P3` evidence
-  - explicit bounds for `RF=2 sync_all`
-  - explicit deferred / blocking items before production use
-  - committed-truth decision scoped to the chosen candidate path
-  - short module/package boundary summary for the next heavy engineering phase
- accepted judgment:
-  - candidate-safe-with-bounds
-  - not production-ready
-
-## Guardrails
-
-### Guardrail 1: Do not reopen accepted V2 protocol truths casually
-
-`Phase 08` is a hardening phase. New work should preserve the accepted protocol truth set unless a real contradiction is demonstrated.
-
-### Guardrail 2: Keep product-path claims evidence-bound
-
-Do not claim more than the hardened path actually proves. Distinguish:
-
-1. live integrated path
-2. hardened product path
-3. production candidate
-
-### Guardrail 3: Identity and policy boundaries remain hard rules
-
-1. `ReplicaID` must remain stable and never collapse to address shape
-2. engine decides recovery policy
-3. bridge translates intent/state
-4. `blockvol` executes I/O only
-
-### Guardrail 4: Carry-forward limitations must remain explicit until closed
-
-Especially:
-
-1. committed truth vs checkpoint truth
-2. rebuild execution coverage
-3. real master/control delivery coverage
-
-### Guardrail 5: The committed-truth carry-forward must become a gate, not a note
-
-For the chosen `RF=2 sync_all` candidate path, this gate is now decided:
-
-1. `CommittedLSN = WALHeadLSN`
-2. `CheckpointLSN` remains the durable base-image boundary
-3. this separation is accepted as sufficient for the candidate-path hardening boundary
-
-For future paths or durability modes, the gate must still be classified explicitly rather than carried forward informally.
-
-## Exit Criteria
-
-Phase 08 is done when:
-
-1. the first product path runs through a real control delivery path
-2. the critical execution chain is integrated and validated
-3. rebuild execution for the chosen path is no longer just detected but executed
-4. at least one real failover / reassignment cycle is replayed through the live control path
-5. the accepted failure-class set is replayed again on the unified live path
-6. operational/debug evidence is sufficient for pre-production use
-7. the remaining gap to a production candidate is small and explicit
-
-Phase-close note:
-
- `Phase 08` is now closed
- next phase:
-  - `Phase 09: Production Execution Closure`
-  - start with `P0` planning for real execution completeness:
-    - real `TransferFullBase`
-    - real `TransferSnapshot`
-    - real `TruncateWAL`
-    - stronger live runtime execution ownership
-
-## Assignment For `sw`
-
-Current next tasks:
-
-1. close out `Phase 08` bookkeeping only if any wording drift remains
-2. move to `Phase 09 P0` planning for production execution closure
-3. focus the next heavy engineering package on:
-   - real `TransferFullBase`
-   - real `TransferSnapshot`
-   - real `TruncateWAL`
-   - stronger live runtime execution ownership
-
-## Assignment For `tester`
-
-Current next tasks:
-
-1. treat `Phase 08` as closed after any final wording/bookkeeping sync
-2. prepare the `Phase 09 P0` validation oracle for production execution closure
-3. keep no-overclaim active around:
-   - validation-grade transfer vs production-grade transfer
-   - truncation execution
-   - stronger runtime ownership vs current bounded path
@@ -1,177 +0,0 @@
-# Phase 09 Decisions
-
-## Decision 1: Phase 09 is production execution closure, not packaging
-
-The candidate-path packaging/judgment work remains inside `Phase 08 P4`.
-
-`Phase 09` starts directly with substantial backend engineering closure.
-
-## Decision 2: The first Phase 09 targets are real transfer, truncation, and stronger runtime ownership
-
-The initial heavy execution blockers are:
-
-1. real `TransferFullBase`
-2. real `TransferSnapshot`
-3. real `TruncateWAL`
-4. stronger live runtime execution ownership
-
-## Decision 3: Phase 09 remains bounded to the chosen candidate path unless evidence forces expansion
-
-Default scope remains:
-
-1. `RF=2`
-2. `sync_all`
-3. existing master / volume-server heartbeat path
-
-Future paths or durability modes should not be absorbed casually into this phase.
-
-## Decision 4: Full-base rebuild completion is defined by an achieved boundary, not exact target equality
-
-For the chosen `RF=2 sync_all` backend path, `full_base` rebuild does not require:
-
-1. extent image exactly equal to the engine's frozen `targetLSN`
-
-It does require:
-
-1. the engine plans a frozen minimum target `targetLSN`
-2. the backend produces an actual rebuilt boundary `achievedLSN`
-3. correctness requires `achievedLSN >= targetLSN`
-4. after install, local runtime state and engine-visible completion must align to the same `achievedLSN`
-5. the system must not keep engine truth at `targetLSN` while local runtime truth has advanced to `achievedLSN`
-
-Reason:
-
-1. the current full-base path copies a mutable extent image from the live backend
-2. this backend does not provide an immutable extent export at an exact requested LSN
-3. forcing exact-target extent equality would require a different protocol, not just a tighter implementation
-4. rollback to an older target after a newer stable base is installed is much harder than accepting the newer stable boundary
-
-Algorithm guarantees required by this decision:
-
-1. minimum-target guarantee:
-   - rebuild completion must never leave the replica behind the engine's frozen minimum target
-2. single-truth guarantee:
-   - `checkpoint`
-   - `nextLSN`
-   - receiver progress
-   - flusher checkpoint
-   - engine-visible rebuild progress/completion
-   must all converge to the same `achievedLSN`
-3. no split-truth guarantee:
-   - do not allow local runtime state to reflect a newer boundary while engine/accounting still records the older one
-4. backend-realism guarantee:
-   - it is acceptable for the achieved boundary to be newer than the frozen minimum target
-   - it is not acceptable for the achieved boundary to remain implicit
-
-## Decision 5: P1 full-base execution closure accepted
-
-P1 delivers real full-base execution closure under the Decision 4 contract.
-
-Accepted properties:
-
-1. `TransferFullBase(committedLSN) → (achievedLSN, error)` — achieved boundary surfaced explicitly
-2. rebuild server pre-flushes before extent copy — no unflushed-entry hole
-3. full state handoff on install — dirty map, WAL, superblock, flusher, receiver progress all aligned
-4. second catch-up bounded to target — no unbounded replay
-5. engine uses `achievedLSN` for progress recording — no split truth
-6. rebuild server fail-closes on pre-copy flush failure
-7. stale-higher local/runtime state is reset to the rebuilt achieved boundary, not preserved by monotonic advance
-
-Evidence closure:
-
-1. live-receiver convergence is now covered directly in `P1`
-2. `P1` accepted state is final for full-base closure on the chosen path
-
-## Decision 6: P2 snapshot execution closure accepted
-
-`P2` delivers real `snapshot_tail` execution closure on the chosen path.
-
-Accepted properties:
-
-1. `TransferSnapshot(snapshotLSN)` now performs real TCP snapshot transfer
-2. snapshot base boundary is exact, not conservative:
-   - requested `snapshotLSN` must match the transferred base
-   - newer checkpoints are rejected instead of silently accepted
-3. snapshot transfer carries explicit boundary metadata through `SnapshotArtifactManifest.BaseLSN`
-4. snapshot install converges local runtime to the exact snapshot boundary before tail replay begins
-5. the `snapshot_tail` path now closes through one executor:
-   - `TransferSnapshot(snapshotLSN)`
-   - `StreamWALEntries(snapshotLSN, targetLSN)`
-6. tail replay remains bounded to `targetLSN`
-7. temporary snapshot ownership is cleaned up on both success and failure paths
-
-Evidence closure:
-
-1. component proof now covers real snapshot transfer and exact-boundary install
-2. one-chain proof now covers `engine -> RebuildExecutor -> v2bridge -> blockvol -> tail replay -> InSync`
-3. boundary-drift rejection is covered directly in `P2`
-
-## Decision 7: P3 truncation execution closure accepted under the narrowed Option A contract
-
-`P3` does not mean "all replica-ahead cases can be corrected by local truncate."
-
-Accepted contract:
-
-1. local truncation is allowed only when the local base boundary exactly matches the kept boundary:
-   - `checkpointLSN == truncateLSN`
-2. if `checkpointLSN > truncateLSN`:
-   - ahead entries already contaminated extent
-   - truncation is unsafe
-   - the path must escalate to rebuild
-3. if `checkpointLSN < truncateLSN`:
-   - part of the kept range may still exist only in WAL
-   - truncation would discard committed kept data
-   - the path must escalate to rebuild
-4. no path may record truncation completion while extent/base truth is known to be unsafe for local truncate
-5. execution-time escalation to `NeedsRebuild` is acceptable for `P3`
-
-Accepted properties:
-
-1. `TruncateWAL(truncateLSN)` now performs real local correction for the truncation-safe case
-2. `TruncateToLSN()` pauses the flusher and drains I/O before mutating local runtime truth
-3. `blockvol.ErrTruncationUnsafe` is bridged to `engine.ErrTruncationUnsafe`
-4. `CatchUpExecutor` escalates unsafe truncation cases to `StateNeedsRebuild`
-5. the mixed case `checkpointLSN < truncateLSN < headLSN` is now covered directly in tests
-
-Evidence closure:
-
-1. component proof covers exact local truncation only for the safe case
-2. one-chain proof covers both:
-   - safe truncation to `InSync`
-   - unsafe truncation escalation to `NeedsRebuild`
-3. `P3` accepted state is final for truncation execution closure on the chosen path
-
-## Decision 8: P4 stronger live runtime ownership accepted
-
-`P4` closes the bounded runtime-ownership gap for the chosen `RF=2 sync_all` live volume-server path.
-
-Accepted properties:
-
-1. `ProcessAssignments()` now drives live recovery ownership through:
-   - assignment conversion
-   - orchestrator session creation/supersede
-   - `RecoveryManager` start/cancel/replace/cleanup
-2. runtime inputs are sourced from the live path rather than test-only injection:
-   - live volume path
-   - live storage adapter / pinner / reader
-   - rebuild address scoped by volume path
-3. replacement is serialized:
-   - stale owner is cancelled and drained before replacement starts
-   - no concurrent live owners remain for the same `replicaID`
-4. shutdown drains live recovery owners before the block service closes volumes
-5. engine policy remains in engine; `P4` does not move policy into the volume-server runtime
-
-Evidence closure:
-
-1. live-path proof now covers:
-   - `ProcessAssignments -> plan_catchup -> exec_catchup_started -> exec_completed -> in_sync`
-2. serialized replacement proof now directly demonstrates:
-   - old owner alive
-   - old owner `done` still open before supersede
-   - `ProcessAssignments(epoch+1)` returns only after old owner `done` closes
-3. shutdown proof now covers a live blocked task, not only an already-finished task
-
-Residual note:
-
-1. repeated primary assignment on the same volume still logs a low-severity rebuild-server double-start warning
-2. broader control-plane closure remains outside `Phase 09`
@@ -1,250 +0,0 @@
-# Phase 09
-
-Date: 2026-03-31
-Status: complete
-Purpose: turn the accepted candidate-safe backend path into a production-grade execution path without reopening accepted V2 recovery semantics
-
-## Why This Phase Exists
-
-`Phase 08` closed:
-
-1. real control delivery on the chosen path
-2. real one-chain catch-up and rebuild closure on the chosen path
-3. unified hardening replay on the accepted live path
-4. one bounded candidate package for `RF=2 sync_all`
-
-What still does not exist is production-grade execution completeness.
-
-The main remaining gap is no longer:
-
-1. whether the path is candidate-safe
-
-It is now:
-
-1. whether the backend execution path is production-grade rather than validation-grade
-
-## Phase Goal
-
-Close the main backend execution gaps so the chosen path is no longer blocked by validation-grade transfer/truncation behavior.
-
-## Scope
-
-### In scope
-
-1. real `TransferFullBase`
-2. real `TransferSnapshot`
-3. real `TruncateWAL`
-4. stronger live runtime execution ownership on the volume-server path
-
-### Out of scope
-
-1. broad control-plane redesign
-2. `RF>2`
-3. `best_effort` / `sync_quorum` recovery semantics
-4. product-surface rebinding (`CSI` / `NVMe` / `iSCSI`)
-5. broad performance optimization
-
-## Phase 09 Items
-
-### P0: Production Execution Closure Plan
-
-1. convert the accepted candidate package into a production-execution closure plan
-2. define the minimum execution blockers that must be closed in this phase
-3. order the execution work by dependency and risk
-4. keep the chosen-path bound explicit while making the backend path production-grade
-
-Goal:
-
- start `Phase 09` with one substantial execution-closure plan, not another light packaging round
-
-Must prove:
-
-1. the phase is centered on real backend execution work
-2. the required closures are explicit:
-   - `TransferFullBase`
-   - `TransferSnapshot`
-   - `TruncateWAL`
-   - stronger runtime ownership
-3. the phase remains bounded to the chosen candidate path unless new evidence expands it
-
-Verification mechanism:
-
-1. architect review:
-   - phase shape is substantial and outcome-based
-   - work is ordered by real engineering dependency
-2. tester review:
-   - validation expectations are explicit for each execution closure target
-3. manager review:
-   - the phase is large enough to justify a full engineering round
-
-Output artifacts:
-
-1. explicit execution-closure target list
-2. explicit execution blocker list
-3. initial slice/package order inside `Phase 09`
-
-Execution note:
-
- use `phase-09-log.md` as the technical pack for:
-  - the definition of "real" for each execution target
-  - recommended slice order
-  - validation expectations
-  - assignment templates for `sw` and `tester`
-
-Reject if:
-
-1. `Phase 09` is framed as another packaging/documentation phase
-2. execution blockers remain implicit
-3. the phase quietly expands into product surfaces or unrelated control-plane work
-4. the phase has no clear verification mechanism
-
-Status:
-
- accepted
-
-### P1: Full-Base Execution Closure
-
-Goal:
-
- make `TransferFullBase` a real production-grade execution path for the chosen `RF=2 sync_all` candidate path
-
-Accepted scope:
-
-1. real TCP full-base transfer
-2. explicit local install ownership in `blockvol`
-3. second catch-up after extent copy
-4. achieved-boundary reporting back to engine
-5. local runtime convergence to the achieved boundary
-6. fail-closed behavior for transfer/runtime errors
-
-Accepted evidence shape:
-
-1. component proof:
-   - TCP transfer
-   - local install
-2. one-chain proof:
-   - `engine plan -> RebuildExecutor -> v2bridge -> blockvol -> InSync`
-3. convergence proof:
-   - `achievedLSN >= targetLSN`
-   - no split truth between engine and local runtime
-4. fail-closed proof:
-   - connection refused
-   - epoch mismatch
-   - no address
-   - partial transfer
-5. runtime proof:
-   - stale non-empty replica state cleared
-   - active receiver progress converges
-
-Status:
-
- accepted
-
-Carry-forward from `P1`:
-
-1. `TransferSnapshot` still not real
-2. `TruncateWAL` still not real
-3. stronger live runtime ownership still not closed
-
-### P2: Snapshot Execution Closure
-
-Goal:
-
- make `TransferSnapshot` a real production-grade execution path for the chosen `RF=2 sync_all` candidate path
-
-Accepted scope:
-
-1. real TCP snapshot/base transfer
-2. exact snapshot-boundary verification
-3. explicit manifest boundary metadata
-4. local runtime convergence to the exact snapshot boundary before tail replay
-5. single-executor snapshot + tail replay execution chain
-6. bounded tail replay to the planned target
-
-Accepted evidence shape:
-
-1. component proof:
-   - real snapshot image transfer
-   - exact base-boundary install
-2. one-chain proof:
-   - `engine plan -> RebuildExecutor -> v2bridge -> blockvol -> tail replay -> InSync`
-3. exact-boundary proof:
-   - requested `snapshotLSN` is transferred exactly
-   - newer checkpoint is rejected rather than silently accepted
-4. convergence proof:
-   - post-install local runtime converges to `snapshotLSN`
-   - post-replay engine/runtime converge to `targetLSN`
-5. cleanup proof:
-   - temporary snapshot ownership released on success/failure
-
-Status:
-
- accepted
-
-Carry-forward from `P2`:
-
-1. `TruncateWAL` still not real
-2. stronger live runtime ownership still not closed
-
-### P3: Truncation Execution Closure
-
-Goal:
-
- make `TruncateWAL` a real production-grade execution path for the chosen `RF=2 sync_all` candidate path
-
-Required scope:
-
-1. real truncation execution closure for the truncation-safe replica-ahead case
-2. explicit rebuild escalation for replica-ahead cases that are not truncation-safe
-3. one-chain proof through the catch-up executor path
-4. fail-closed / no-overclaim behavior when local truncation is unsafe
-5. no overclaim of broader runtime-ownership closure
-
-Status:
-
- accepted
-
-Carry-forward from `P3`:
-
-1. truncation-safe vs rebuild-required replica-ahead split still happens at execution time, not planning time
-2. stronger live runtime ownership still not closed
-
-### P4: Stronger Live Runtime Ownership
-
-Goal:
-
- move the accepted execution logic from bounded test/adapter ownership into a stronger live runtime path on the chosen `RF=2 sync_all` volume-server path
-
-Required scope:
-
-1. stronger volume-server/runtime ownership of recovery execution
-2. explicit live start / cancel / replace / cleanup semantics
-3. real runtime wiring for current execution inputs and addresses
-4. one-chain proof on the live runtime path, not only bounded executor tests
-5. no overclaim of broader control-plane closure
-
-Status:
-
- accepted
-
-Carry-forward from `P4`:
-
-1. repeated primary assignment still logs a low-severity rebuild-server double-start warning on the same volume
-2. broader control-plane closure remains out of scope for `Phase 09`
-
-## Assignment For `sw`
-
-Current next tasks:
-
-1. `Phase 09` is complete
-2. no further `P4` implementation work is open in this phase
-3. any next work should open under the next phase, not extend `Phase 09` implicitly
-
-## Assignment For `tester`
-
-Current next tasks:
-
-1. `Phase 09` validation/bookkeeping is complete
-2. keep any residual notes bounded:
-   - low-severity rebuild-server double-start warning on repeated primary assignment
-   - broader control-plane closure still belongs to a later phase
@@ -1,109 +0,0 @@
-# Phase 10 Decisions
-
-## Decision 1: Phase 10 is control-plane closure, not backend execution rework
-
-`Phase 09` already closed the main backend execution gaps on the chosen path.
-
-`Phase 10` should therefore focus on:
-
-1. real control delivery
-2. reassignment / result convergence
-3. identity cleanup
-
-It should not reopen accepted backend execution semantics unless a true control-plane bug forces a narrow correction.
-
-## Decision 2: Phase 10 remains bounded to the chosen path
-
-Default scope remains:
-
-1. `RF=2`
-2. `sync_all`
-3. existing master / volume-server heartbeat path
-
-Future durability modes or wider topology support should not be absorbed casually into this phase.
-
-## Decision 3: Identity cleanup belongs to control-plane closure
-
-The current local server identity remains transport-shaped (`listenAddr`).
-
-`Phase 10` is the right place to strengthen this because identity coherence affects:
-
-1. assignment truth
-2. sender/replica identity continuity
-3. end-to-end control-path correctness
-
-## Decision 4: Rebuild-server idempotence cleanup is bounded residual work, not the phase itself
-
-The repeated-primary-assignment warning around rebuild-server start is a valid residual note.
-
-It may be addressed in `Phase 10` only if:
-
-1. it is directly relevant to real control/runtime ownership or assignment idempotence
-2. it stays bounded
-
-It must not turn `Phase 10` into a broad runtime polish phase.
-
-## Decision 5: P1 identity and control-truth closure accepted
-
-`P1` closes the stable-identity/control-truth gap on the chosen block assignment wire.
-
-Accepted properties:
-
-1. stable server identity is now carried additively on the block assignment proto wire:
-   - scalar `replica_server_id`
-   - per-replica `server_id`
-2. generated protobuf output, not hand-maintained output, is now the accepted basis for the wire shape
-3. master create-path and chosen failover/primary-refresh assignment generation now preserve stable identity on the chosen path
-4. volume-server block/control path now uses the same canonical `volumeServerId` as the main volume server
-5. `ControlBridge` continues to fail closed when stable identity is missing
-
-Evidence closure:
-
-1. proto/decode proof now covers stable identity round-trip
-2. real ingress proof now covers:
-   - proto assignment
-   - decode
-   - `ProcessAssignments()`
-   - `ControlBridge`
-   - engine sender `ReplicaID`
-3. canonical local identity proof now covers non-default local ID
-4. missing-ID fail-closed proof is covered directly
-
-## Decision 6: P2 reassignment/result convergence accepted under the chosen-path volume-server ingress bound
-
-`P2` closes the main reassignment/result-convergence gap on the chosen path without reopening accepted backend execution semantics.
-
-Accepted properties:
-
-1. reassignment through the accepted chosen-path ingress now proves old sender truth is removed and new sender truth is created
-2. stale runtime ownership is now proved as a live drain case, not only a bookkeeping absence case
-3. reported truth is now checked through `CollectBlockVolumeHeartbeat()`, the same reporting surface used by the live heartbeat loop
-4. the accepted no-split-truth claim is bounded to:
-   - engine sender truth
-   - stale-runtime residue removed
-   - heartbeat output truth
-5. `P2` does not claim full master-driven failover/gRPC-infrastructure closure beyond the accepted volume-server-side ingress boundary
-
-Evidence closure:
-
-1. real reassignment proof covers `vs2 -> vs3` sender replacement on the chosen path
-2. stale-owner proof now blocks a live old goroutine and verifies drain during reassignment
-3. heartbeat proof now checks actual heartbeat output rather than local helper state
-4. delivery wording is bounded so it does not overclaim a proved live replacement owner in the no-split-truth test
-
-## Decision 7: P3 bounded repeated-assignment/idempotence cleanup accepted on the chosen path
-
-`P3` closes the bounded repeated-assignment residual left after accepted `P2`.
-
-Accepted properties:
-
-1. repeated unchanged chosen-path assignment is now skipped before duplicate V2 orchestrator/recovery work is started
-2. the corresponding V1 primary-replication setup path is also absorbed idempotently for unchanged truth
-3. changed chosen-path assignment still takes the accepted replacement/update path rather than being suppressed incorrectly
-4. `P3` remains bounded cleanup and does not claim general multi-replica idempotence or broad production hardening
-
-Evidence closure:
-
-1. repeated-assignment proof now checks stable V2 event count rather than only stable helper/reporting state
-2. changed-assignment guard proof keeps accepted replacement behavior intact
-3. externally visible heartbeat state remains coherent after repeated unchanged assignment
@@ -1,228 +0,0 @@
-# Phase 10
-
-Date: 2026-04-02
-Status: complete
-Purpose: close the main end-to-end control-plane gaps on the chosen `RF=2 sync_all` path without reopening accepted backend execution semantics
-
-## Why This Phase Exists
-
-`Phase 09` closed the main backend execution gaps on the chosen path:
-
-1. real `TransferFullBase`
-2. real `TransferSnapshot`
-3. real `TruncateWAL` under the accepted narrowed contract
-4. stronger live runtime ownership on the volume-server path
-
-What still does not exist is stronger end-to-end control-plane closure.
-
-The main remaining gap is no longer:
-
-1. whether the backend execution path is real
-
-It is now:
-
-1. whether the real control path drives and reflects the chosen path coherently enough for product use
-
-## Phase Goal
-
-Strengthen from accepted assignment-entry closure to stronger end-to-end control-plane closure on the chosen path.
-
-## Scope
-
-### In scope
-
-1. heartbeat / gRPC-level control delivery proof on the chosen path
-2. reassignment / failover result convergence through the real control path
-3. cleaner local identity than transport-shaped `listenAddr`
-4. bounded idempotence / repeated-assignment cleanup when it directly affects live control/runtime ownership
-
-### Out of scope
-
-1. reopening accepted `P1` / `P2` / `P3` / `P4` backend execution semantics
-2. `RF>2`
-3. `best_effort` / `sync_quorum`
-4. product-surface rebinding (`CSI` / `NVMe` / `iSCSI`)
-5. broad performance optimization
-
-## Phase 10 Items
-
-### P0: Control-Plane Closure Plan
-
-Goal:
-
- start `Phase 10` with one substantial control-plane closure package, not a loose collection of follow-up fixes
-
-Must prove:
-
-1. the phase is centered on real control-path closure rather than backend execution rework
-2. the required closure targets are explicit:
-   - heartbeat / gRPC delivery
-   - reassignment / result convergence
-   - identity cleanup
-   - bounded repeated-assignment/idempotence cleanup
-3. the chosen-path bound remains explicit
-
-Verification mechanism:
-
-1. architect review:
-   - control-plane scope is explicit and bounded
-   - proposed slices do not reopen accepted backend execution semantics
-2. tester review:
-   - required end-to-end proofs are explicit
-3. manager review:
-   - the package is concrete enough to assign the first implementation slice
-
-Output artifacts:
-
-1. explicit control-plane closure targets
-2. explicit reject shapes
-3. initial slice order inside `Phase 10`
-
-Execution note:
-
- use `phase-10-log.md` as the technical pack for:
-  - semantic scope
-  - execution scope
-  - proof shapes
-  - assignment templates for `sw` and `tester`
-
-Reject if:
-
-1. `Phase 10` is framed as a vague "polish/control" phase without concrete closure targets
-2. accepted `Phase 09` execution semantics are quietly reopened
-3. product surfaces or unrelated hardening work are absorbed into this phase
-4. no explicit end-to-end proof shape is defined
-
-Status:
-
- accepted
-
-### P1: Identity And Control-Truth Closure
-
-Goal:
-
- close stable identity on the real chosen-path control wire so assignment truth, local ingest truth, and `ReplicaID` construction no longer depend on transport-shaped fallback
-
-Accepted scope:
-
-1. stable server identity preserved on the block assignment proto wire
-2. master assignment generation preserves stable identity on the chosen path
-3. volume-server local identity uses the same canonical server identity as the main volume server
-4. real ingress proof:
-   - proto/decode
-   - `ProcessAssignments()`
-   - `ControlBridge`
-   - engine sender identity
-5. fail-closed behavior for missing stable identity
-
-Status:
-
- accepted
-
-Carry-forward from `P1`:
-
-1. fuller reassignment / failover result convergence is still open
-2. broader control-plane reporting closure is still open
-
-### P2: Reassignment / Result Convergence
-
-Goal:
-
- prove that reassignment and failover converge through the real control path without stale local ownership or stale reported truth lingering after control truth changes
-
-Accepted scope:
-
-1. real failover / reassignment convergence through the chosen control path
-2. no stale local runtime owner after control truth changes
-3. no stale control/reporting truth after reassignment
-4. one-chain proof through the real control path, not only local helper logic
-5. no overclaim of broader hardening or product-surface closure
-
-Status:
-
- accepted
-
-Carry-forward from `P2`:
-
-1. `P2` proves stale owner removal and no stale residue after control truth changes
-2. bounded repeated-assignment/idempotence cleanup is still open where repeated primary assignment can still emit rebuild-server relisten warnings
-3. `P2` does not claim broad master-driven failover infrastructure closure beyond the accepted volume-server-side chosen-path ingress
-
-### P3: Bounded Repeated-Assignment / Idempotence Cleanup
-
-Goal:
-
- close the remaining low-severity repeated-assignment/runtime-idempotence gap on the chosen path so duplicate or replacement primary assignments do not leave avoidable relisten/restart noise or ambiguous live-control ownership
-
-Accepted scope:
-
-1. repeated primary assignment on the same chosen-path volume should converge idempotently
-2. rebuild-server/runtime side effects should not relaunch noisily when the authoritative control truth is unchanged or already active
-3. bounded proof that repeated-assignment cleanup does not reopen accepted `P2` convergence or accepted `Phase 09` execution semantics
-4. no expansion into broad runtime polish, product surfaces, or unrelated restart hardening
-
-Status:
-
- accepted
-
-Carry-forward from `P3`:
-
-1. chosen-path repeated unchanged assignment is now absorbed idempotently across the accepted V2 + V1 live path
-2. `P3` remains bounded cleanup; it does not itself close the remaining master-driven heartbeat/gRPC control-loop gap
-3. fuller master-originated control delivery proof is still open
-
-### P4: Master-Driven Control-Loop Closure
-
-Goal:
-
- close the remaining chosen-path control-plane gap by proving that master-originated assignment truth delivered through the real heartbeat / gRPC control loop reaches the live volume-server path and converges without split truth
-
-Required scope:
-
-1. one bounded end-to-end proof from real master-produced chosen-path assignment truth into the live volume-server control path
-2. proof that the real heartbeat / gRPC delivery path preserves the already accepted identity and convergence properties
-3. proof that externally visible post-delivery state reflects the same new truth after the real master-driven path runs
-4. no reopening of accepted `P1` / `P2` / `P3` semantics except for narrow bugs directly exposed by the fuller control-loop proof
-5. no expansion into product surfaces, `RF>2`, or broad cluster-hardening work
-
-Status:
-
- accepted
-
-Carry-forward from `P4`:
-
-1. bounded chosen-path master-driven heartbeat / gRPC control-loop closure is now accepted
-2. `P4` does not claim full live transport-stream deployment proof or broad product hardening
-3. the next phase should move to `Phase 11` product-surface rebinding
-
-### Planned slice direction after `P0`
-
-1. `P1`:
-   - identity and control-truth closure on the live control path
-2. `P2`:
-   - reassignment / failover result convergence through the real control path
-3. `P3`:
-   - bounded idempotence / repeated-assignment cleanup after accepted `P1` / `P2`
-4. `P4`:
-   - master-driven heartbeat / gRPC control-loop closure on the chosen path
-
-## Assignment For `sw`
-
-Current next tasks:
-
-1. treat `Phase 10` as closed and keep accepted `P1` / `P2` / `P3` / `P4` semantics stable
-2. start `Phase 11` product-surface rebinding from `v2-phase-development-plan.md`
-3. keep the first `Phase 11` slice bounded to selected product surfaces rather than broad hardening
-4. do not reopen accepted backend execution or control-plane closure except for narrow bug fixes
-
-## Assignment For `tester`
-
-Current next tasks:
-
-1. treat `P4` as accepted bounded control-loop closure on the chosen path
-2. validate the first `Phase 11` slice as bounded product-surface rebinding rather than renewed control-plane work
-3. keep no-overclaim active around:
-   - accepted `Phase 09` execution closure
-   - accepted `Phase 10` control-plane closure
-   - selected `Phase 11` surface scope vs broader product readiness
-   - chosen path vs future paths/modes
@@ -1,488 +0,0 @@
-# Phase 11
-
-Date: 2026-04-02
-Status: complete
-Purpose: bind selected product-facing surfaces onto the accepted V2-backed chosen path without reopening accepted backend execution or control-plane closure
-
-## Why This Phase Exists
-
-`Phase 09` accepted production-grade execution closure on the chosen path.
-`Phase 10` accepted bounded master-driven control-plane closure on that same path.
-
-What remains is no longer:
-
-1. whether the chosen backend path executes correctly
-2. whether accepted control truth can reach the live volume-server path coherently
-
-It is now:
-
-1. whether selected product-facing surfaces can be rebound onto that accepted path without semantic drift
-2. whether reuse of older V1-facing adapters reintroduces V1 recovery truth implicitly
-3. whether the first product-facing surface can be proven in a bounded way before broader surface expansion
-
-## Phase Goal
-
-Move from accepted backend/control closure on one bounded chosen path to the first bounded product-surface rebinding proof.
-
-Execution note:
-
-1. treat `P0` as real planning work, not placeholder prose
-2. use `phase-11-log.md` as the technical pack for:
-   - step breakdown
-   - hard indicators
-   - reject shapes
-   - assignment text for `sw` and `tester`
-
-## Scope
-
-### In scope
-
-1. one bounded first product-surface slice
-2. explicit no-overclaim around what that first surface proves and does not prove
-3. reuse of existing implementation only where V2 truth still owns placement, recovery, and correctness claims
-4. focused integration tests and contract checks for the chosen first surface
-
-### Out of scope
-
-1. reopening accepted `Phase 09` execution semantics
-2. reopening accepted `Phase 10` control-plane closure
-3. broad multi-surface product completion in one slice
-4. `RF>2`, new durability modes, or broad cluster hardening
-5. full production readiness / soak / rollout gates
-
-## Phase 11 Items
-
-### P0: First Surface Selection
-
-Goal:
-
- choose the first product-facing surface that gives real product completion movement without turning the phase into a multi-system rewrite
-
-Accepted decision:
-
-1. the first bounded slice is `snapshot product path`
-2. `CSI` is deferred to a later `Phase 11` slice because it pulls controller/node lifecycle, staging/publish, and broader cluster contract surface
-3. `NVMe` / `iSCSI` rebinding are also deferred because they are transport/front-end adapters whose useful proof should come after one simpler product surface is already closed
-
-Why this first:
-
-1. snapshot is closest to already accepted backend truth
-2. it exercises a real product-facing contract without immediately absorbing node/attach orchestration
-3. it keeps the first `Phase 11` slice bounded to metadata/visibility/restore-contract correctness rather than transport and lifecycle breadth
-
-Status:
-
- accepted
-
-### P1: Snapshot Product-Path Rebinding
-
-Goal:
-
- prove that the snapshot product path can be rebound onto the accepted V2-backed chosen path without semantic drift between snapshot-visible behavior and the accepted backend snapshot truth
-
-Execution steps:
-
-1. Step 1: contract freeze
-   - define exactly what the first slice claims:
-     - snapshot create
-     - snapshot list
-     - snapshot delete
-   - explicitly exclude clone/restore unless a later slice accepts them
-2. Step 2: implementation binding
-   - bind product-visible snapshot operations onto the accepted backend snapshot path
-   - keep master/volume-server state and visible metadata coherent
-3. Step 3: proof package
-   - prove create/list/delete on the chosen path
-   - prove fail-closed behavior for unsupported/invalid inputs
-   - prove no-overclaim around broader snapshot workflows
-
-Required scope:
-
-1. snapshot create/list/delete product-visible behavior on the chosen path
-2. proof that snapshot metadata and visible snapshot set reflect the same accepted backend truth
-3. proof that snapshot claims do not exceed the accepted V2 snapshot contract
-4. explicit boundedness around restore/clone if they are not part of the first slice
-
-Must prove:
-
-1. snapshot creation on the product path maps to the accepted backend snapshot boundary rather than an implicit V1 truth
-2. listing and deletion reflect the real volume-server/master state coherently
-3. fail-closed behavior is preserved when snapshot prerequisites are missing or the volume is not eligible
-4. the slice does not silently imply clone/restore/product workflow support that is not yet proven
-
-Reuse discipline:
-
-1. V1/master-facing snapshot RPC surface may be reused only as a product wrapper:
-   - `CreateBlockSnapshot`
-   - `DeleteBlockSnapshot`
-   - `ListBlockSnapshots`
-2. V1/volume-server-facing snapshot surface may be reused only as the bounded execution adapter:
-   - `SnapshotBlockVol`
-   - `DeleteBlockSnapshot`
-   - `ListBlockSnapshots`
-3. underlying `blockvol` snapshot implementation may be reused as execution reality, not as product truth ownership
-4. every reused V1 surface must be called out explicitly in `phase-11-log.md` with one of:
-   - `update in place`
-   - `reference only`
-   - `reuse as bounded adapter`
-5. no reused V1 surface may silently redefine snapshot semantics, placement truth, or product support claims
-
-Verification mechanism:
-
-1. focused integration tests for create/list/delete on the chosen path
-2. contract checks that visible snapshot metadata matches the accepted backend snapshot truth
-3. no-overclaim review on what user-visible snapshot behavior is actually supported after the slice
-
-Hard indicators:
-
-1. one accepted create proof:
-   - product-visible create succeeds on the chosen path
-   - created snapshot is observable through list/readback metadata
-2. one accepted delete proof:
-   - deleted snapshot disappears from the visible snapshot set
-   - repeated delete is either idempotent-success or explicitly fail-closed as designed
-3. one accepted list coherence proof:
-   - listed snapshot IDs/metadata match the real backend snapshot state
-4. one accepted fail-closed proof:
-   - invalid volume / missing snapshot / unsupported preconditions do not imply false success
-5. one accepted boundedness proof:
-   - docs/tests do not imply clone/restore/full snapshot workflow readiness unless separately proven
-6. one accepted reuse-boundary proof:
-   - all V1 reuse surfaces touched by the slice are explicitly listed and their role is bounded
-
-Reject if:
-
-1. the slice proves only local helper behavior rather than product-visible snapshot behavior
-2. visible snapshot metadata can drift from backend truth
-3. the first slice quietly absorbs clone/restore or broader workflow work
-4. the slice claims product readiness beyond create/list/delete on the chosen path
-5. reuse of V1 surfaces is implicit or lets V1 semantics become the source of truth
-
-Status:
-
- accepted
-
-Carry-forward from `P1`:
-
-1. bounded snapshot create/list/delete product rebinding is now accepted on the chosen path
-2. `P1` does not claim restore/clone/full snapshot workflow readiness
-3. `CSI` rebinding is now the next active `Phase 11` slice
-
-### Later candidate slices inside `Phase 11`
-
-1. `P2`: `CSI` rebinding after snapshot product-path closure
-2. `P3`: `NVMe` / `iSCSI` front-end rebinding after one simpler product-visible surface is already accepted
-3. `P4`: broader snapshot workflow closure (`restore` / `clone`) or other residual product workflow work only after earlier slices are bounded and proven
-
-### P2: CSI Rebinding
-
-Goal:
-
- bind the accepted V2-backed chosen path to the `CSI` controller/node product surface without reintroducing V1 recovery truth
-
-Execution steps:
-
-1. Step 1: contract freeze
-   - define the first bounded `CSI` surface claims:
-     - `CreateVolume`
-     - `DeleteVolume`
-     - `ControllerPublishVolume`
-     - `NodeStageVolume`
-     - `NodePublishVolume`
-     - `NodeUnpublishVolume`
-     - `NodeUnstageVolume`
-   - explicitly exclude CSI snapshot, expand, and NVMe-specific transport work unless a later slice accepts them
-2. Step 2: backend rebinding
-   - bind CSI controller operations to the accepted master-backed chosen-path volume surface
-   - bind CSI node operations to the accepted chosen-path access contract for remote attach/stage/publish
-3. Step 3: proof package
-   - prove bounded create/publish/stage/use/delete lifecycle on the chosen path
-   - prove fail-closed behavior for unsupported or invalid cases
-   - prove no-overclaim around broader CSI/product workflow breadth
-
-Required scope:
-
-1. bounded CSI controller/node lifecycle on the chosen path
-2. explicit separation between accepted backend/control truth and CSI orchestration wrappers
-3. remote target publication/staging behavior for the chosen path
-4. no-overclaim around snapshots via CSI, expand, NVMe transport preference, multi-node topology breadth, or broad K8s readiness
-
-Must prove:
-
-1. CSI controller create/delete/publish map to the accepted master-backed chosen-path truth rather than a local V1 shortcut
-2. CSI node stage/publish/unstage/unpublish consume the same chosen-path access truth without redefining recovery semantics
-3. product-visible CSI lifecycle behavior is coherent across controller and node surfaces
-4. fail-closed behavior is preserved when required publish/volume context or target information is missing
-
-Reuse discipline:
-
-1. V1/CSI-facing controller and node RPC surfaces may be reused only as bounded product adapters:
-   - `controller.go`
-   - `node.go`
-   - `server.go`
-2. `volume_backend.go` may be reused only as the bounded bridge between CSI and accepted master/local surfaces
-3. `volume_manager.go` may be reused only as bounded local execution reality where the slice explicitly proves that local manager behavior does not become semantic owner
-4. accepted master block RPC surfaces may be reused only as bounded control/product adapters underneath the CSI backend bridge:
-   - `CreateBlockVolume`
-   - `DeleteBlockVolume`
-   - `LookupBlockVolume`
-5. every reused V1 surface must be called out explicitly in `phase-11-log.md` with one of:
-   - `update in place`
-   - `reference only`
-   - `reuse as bounded adapter`
-   - `reuse as bounded bridge`
-   - `reuse as execution reality only`
-6. no reused V1 surface may silently redefine lifecycle semantics, placement truth, or product support claims
-
-Verification mechanism:
-
-1. focused CSI controller/node integration tests on the chosen path
-2. contract checks that controller-visible and node-visible truth match accepted backend/control truth
-3. no-overclaim review on what CSI behavior is actually supported after the slice
-
-Hard indicators:
-
-1. one accepted controller create/publish proof:
-   - CSI create returns coherent volume/publish context on the chosen path
-2. one accepted node stage/publish proof:
-   - node consumes the published target info and stages/publishes coherently on the chosen path
-3. one accepted unpublish/unstage/delete proof:
-   - teardown/deletion complete without leaving false-visible ownership
-4. one accepted fail-closed proof:
-   - missing or partial transport/context information does not imply false success
-5. one accepted reuse-boundary proof:
-   - all CSI/V1 reuse surfaces touched by the slice are explicitly listed and bounded
-6. one accepted boundedness proof:
-   - docs/tests do not imply CSI snapshot, expand, NVMe transport preference, or broad K8s/product readiness unless separately proven
-
-Reject if:
-
-1. the slice proves only CSI wrapper-local behavior without chosen-path backend/control coherence
-2. controller truth and node truth can drift from accepted master-backed volume truth
-3. the first CSI slice quietly absorbs snapshot, expand, NVMe, or broad multi-node/K8s readiness work
-4. reuse of V1 surfaces is implicit or lets V1 semantics become the source of truth
-
-Status:
-
- accepted
-
-Carry-forward from `P2`:
-
-1. bounded CSI controller/node lifecycle rebinding is now accepted on the chosen path
-2. accepted proof uses the real master-backed create/lookup/delete path plus `mgr=nil` node consumption of published target truth
-3. `P2` does not claim CSI snapshot, CSI expand, NVMe preference/failover closure, or broad Kubernetes readiness
-
-### P3: NVMe / iSCSI Front-End Rebinding
-
-Goal:
-
- bind transport/front-end publication surfaces onto the accepted V2-backed chosen path so the product-visible access path matches accepted backend/control truth
-
-Execution steps:
-
-1. Step 1: contract freeze
-   - define the first bounded front-end publication claims:
-     - create returns coherent front-end publication data
-     - lookup returns coherent front-end publication data
-     - heartbeat refresh preserves and updates publication truth
-     - failover switches publication truth to the new primary coherently
-   - explicitly exclude broad transport-performance claims, real initiator benchmarking, and broad cluster rollout readiness
-2. Step 2: publication rebinding
-   - bind `iSCSI` and `NVMe` publication fields onto the accepted master-backed chosen-path truth
-   - keep registry-visible, lookup-visible, and CSI-visible publication truth coherent
-3. Step 3: proof package
-   - prove bounded create/lookup/failover/restart publication truth on the chosen path
-   - prove fallback behavior is explicit where `NVMe` is absent
-   - prove no-overclaim around full transport runtime/performance closure
-
-Required scope:
-
-1. publication/address/naming truth for front-end adapters on the chosen path
-2. bounded integration proof that master-visible and product-visible access metadata stay coherent
-3. `NVMe` primary publication and `iSCSI` fallback publication where supported by the chosen path
-4. explicit boundedness around real initiator behavior, transport performance, and broad cluster hardening
-
-Must prove:
-
-1. create/lookup publication fields map to accepted chosen-path truth rather than ad hoc wrapper-local construction
-2. heartbeat refresh and failover preserve or update front-end publication truth coherently
-3. `NVMe` and `iSCSI` publication fields do not drift between registry, lookup, and product-facing responses
-4. mixed-capability or fallback behavior is explicit rather than silently overclaimed
-
-Reuse discipline:
-
-1. master-facing product/control publication surfaces may be reused only as bounded adapters:
-   - `CreateBlockVolume`
-   - `LookupBlockVolume`
-2. registry publication fields may be reused only as bounded truth carriers, not independent semantic owners:
-   - `ISCSIAddr`
-   - `IQN`
-   - `NvmeAddr`
-   - `NQN`
-3. volume-server allocation/publication surfaces may be reused only as bounded front-end publication sources:
-   - `AllocateBlockVolume`
-   - block heartbeat publication of `NvmeAddr` / `NQN`
-4. existing `CSI` controller consumption of publication fields may be reused only as a bounded downstream consumer, not as the source of truth for `P3`
-5. every reused V1 surface must be called out explicitly in `phase-11-log.md` with one of:
-   - `update in place`
-   - `reference only`
-   - `reuse as bounded adapter`
-   - `reuse as bounded truth carrier`
-   - `reuse as publication source only`
-6. no reused V1 surface may silently redefine publication truth, failover truth, or supported transport claims
-
-Verification mechanism:
-
-1. focused integration tests for create/lookup publication truth on the chosen path
-2. contract checks that registry-visible, lookup-visible, and consumer-visible publication fields match
-3. failover/restart checks that front-end publication truth is reconstructed or updated coherently
-4. no-overclaim review on what transport/front-end behavior is actually supported after the slice
-
-Hard indicators:
-
-1. one accepted create/lookup publication proof:
-   - create returns coherent front-end publication fields
-   - lookup returns the same chosen-path publication truth
-2. one accepted failover publication proof:
-   - front-end publication fields move to the new primary coherently after failover
-3. one accepted restart/heartbeat reconstruction proof:
-   - publication fields can be reconstructed or refreshed from accepted heartbeat truth
-4. one accepted fallback proof:
-   - `iSCSI` fallback or mixed-capability behavior is explicit and coherent when `NVMe` is absent
-5. one accepted reuse-boundary proof:
-   - all front-end publication surfaces touched by the slice are explicitly listed and bounded
-6. one accepted boundedness proof:
-   - docs/tests do not imply real transport runtime, performance leadership, or broad production readiness unless separately proven
-
-Reject if:
-
-1. the slice proves only field plumbing without chosen-path publication coherence
-2. publication truth can drift across create, lookup, heartbeat, or failover
-3. the slice quietly absorbs full transport runtime or performance claims
-4. reuse of V1/publication surfaces is implicit or lets wrappers become the truth owner
-
-Status:
-
- accepted
-
-Carry-forward from `P3`:
-
-1. bounded front-end publication/address truth rebinding is now accepted on the chosen path
-2. accepted proof closes create/lookup coherence, failover publication switch, heartbeat reconstruction, and no-`NVMe` fallback
-3. `P3` does not claim full initiator/runtime transport proof, performance claims, or broad production readiness
-
-### P4: Broader Product Workflow Closure
-
-Goal:
-
- close the remaining bounded snapshot product workflow gaps downstream of accepted `P1` / `P2` / `P3` without reopening earlier accepted truth
-
-Execution steps:
-
-1. Step 1: contract freeze
-   - define the first bounded `P4` workflow claim as snapshot `restore`
-   - explicitly defer `clone` unless and until a real product-facing clone surface exists and is accepted into scope
-2. Step 2: workflow rebinding
-   - bind product-visible restore behavior onto the accepted snapshot and chosen-path execution truth
-   - keep restore-visible state coherent across master-visible and volume-server/backend-visible truth
-3. Step 3: proof package
-   - prove bounded restore success, destructive semantics, and post-restore visible truth
-   - prove fail-closed behavior for missing snapshot or unsupported conditions
-   - prove no-overclaim around clone or broader workflow productization
-
-Required scope:
-
-1. bounded snapshot restore product workflow on the chosen path
-2. explicit proof that restore uses accepted snapshot/backend truth rather than reopening new execution ownership
-3. explicit post-restore visible truth checks
-4. explicit boundedness around `clone` and any broader workflow work
-
-Must prove:
-
-1. product-visible restore maps to accepted backend restore execution truth on the chosen path
-2. restore-visible outcome matches the selected snapshot truth after the operation completes
-3. destructive restore semantics are explicit rather than hidden
-4. fail-closed behavior is preserved for missing snapshot, missing volume, or unsupported preconditions
-
-Reuse discipline:
-
-1. accepted master-facing snapshot RPC surfaces may be reused only as bounded product adapters for restore if a restore entry surface exists
-2. accepted volume-server-facing snapshot/restore surfaces may be reused only as bounded execution adapters
-3. underlying `blockvol.RestoreSnapshot` may be reused only as execution reality, not as product-truth ownership
-4. `clone` must stay explicitly deferred unless a real product-facing surface is brought into scope and written into `phase-11-log.md`
-5. every reused V1 surface must be called out explicitly in `phase-11-log.md` with one of:
-   - `update in place`
-   - `reference only`
-   - `reuse as bounded adapter`
-   - `reuse as execution reality only`
-6. no reused V1 surface may silently redefine restore semantics, workflow readiness, or clone claims
-
-Verification mechanism:
-
-1. focused restore integration tests on the chosen path
-2. contract checks that post-restore visible truth matches selected snapshot truth
-3. fail-closed checks for invalid or unsupported restore conditions
-4. no-overclaim review on what restore/clone workflow behavior is actually supported after the slice
-
-Hard indicators:
-
-1. one accepted restore success proof:
-   - product-visible restore succeeds on the chosen path
-   - visible post-restore state matches the selected snapshot truth
-2. one accepted destructive-semantics proof:
-   - writes after the snapshot are lost as designed and this is explicitly verified
-3. one accepted fail-closed proof:
-   - missing snapshot / missing volume / unsupported conditions do not imply false success
-4. one accepted post-restore coherence proof:
-   - list/readback/visible workflow state are coherent after restore
-5. one accepted reuse-boundary proof:
-   - all restore-facing V1 surfaces touched by the slice are explicitly listed and bounded
-6. one accepted boundedness proof:
-   - docs/tests do not imply clone or broad snapshot workflow readiness unless separately proven
-
-Reject if:
-
-1. the slice proves only backend-local restore mechanics without product-visible restore behavior
-2. post-restore visible truth is not asserted
-3. destructive semantics are left implicit
-4. the slice quietly absorbs `clone` or broader workflow readiness work
-5. reuse of V1 surfaces is implicit or lets V1 semantics become the source of truth
-
-Status:
-
- accepted
-
-Carry-forward from `P4`:
-
-1. bounded snapshot restore workflow closure is now accepted on the chosen path
-2. accepted proof closes restore success, destructive semantics, post-restore visible truth, and fail-closed behavior
-3. `clone` remains explicitly deferred because no real product-facing clone surface is yet accepted into scope
-
-## Phase 11 Completion Judgment
-
-`Phase 11` is complete because:
-
-1. `P1` accepted bounded snapshot create/list/delete product rebinding
-2. `P2` accepted bounded `CSI` controller/node lifecycle rebinding
-3. `P3` accepted bounded `NVMe` / `iSCSI` publication/address truth rebinding
-4. `P4` accepted bounded snapshot restore workflow closure
-5. the chosen-path product surface rebinding goal is now closed without reopening accepted `Phase 09` / `Phase 10` semantics
-6. remaining work is no longer product-surface rebinding inside `Phase 11`, but production hardening in `Phase 12`
-
-## Assignment For `sw`
-
-Current next tasks:
-
-1. `Phase 11` is closed
-2. move next to `Phase 12 P0` production-hardening planning
-3. do not reopen accepted `P1` / `P2` / `P3` / `P4` semantics casually during hardening planning
-4. keep `clone` deferred unless separately re-scoped in a future phase
-
-## Assignment For `tester`
-
-Current next tasks:
-
-1. `Phase 11` is closed
-2. validate `Phase 12 P0` as real planning work rather than placeholder prose
-3. keep no-overclaim active around accepted `P1` / `P2` / `P3` / `P4` closure
-4. treat `clone` or any other future workflow work as separate re-scoping work, not implicit `Phase 11` residue
@@ -1,29 +0,0 @@
-# Phase 12 P3 — Blocker Ledger
-
-Date: 2026-04-02
-Scope: bounded diagnosability / blocker accounting for the accepted RF=2 sync_all chosen path
-
-## Diagnosed and Bounded
-
-| ID | Symptom | Evidence Surface | Owning Truth | Status |
-|----|---------|-----------------|--------------|--------|
-| B1 | Failover does not converge | failover logs + registry Lookup epoch/primary | registry authority | Diagnosed: convergence depends on lease expiry + heartbeat cycle; bounded by lease TTL |
-| B2 | Lookup publication stale after failover | LookupBlockVolume response vs registry entry | registry ISCSIAddr/VolumeServer | Diagnosed: publication updates on failover assignment delivery; bounded by assignment queue delivery |
-| B3 | Recovery tasks remain after volume delete | RecoveryManager.DiagnosticSnapshot | RecoveryManager task map | Diagnosed: tasks drain on shutdown/cancel; bounded by RecoveryManager lifecycle |
-
-## Unresolved but Explicit
-
-| ID | Symptom | Current Evidence | Why Unresolved | Blocks P4/Rollout? |
-|----|---------|-----------------|----------------|-------------------|
-| U1 | V2 engine accepts stale-epoch assignments at orchestrator level | V2 idempotence check skips only same-epoch; lower epoch creates new sender | Engine ApplyAssignment does not check epoch monotonicity on Reconcile | No — V1 HandleAssignment rejects epoch regression; V2 is secondary |
-| U2 | Single-process test cannot exercise Primary→Rebuilding role transition | HandleAssignment rejects transition in shared store | Test harness limitation, not production bug | No — production VS has separate stores |
-| U3 | gRPC stream transport not exercised in control-loop tests | All logic above/below stream is real; stream itself bypassed | Would require live master+VS gRPC servers in test | Blocks full integration test, not correctness |
-
-## Out of Scope for P3
-
- Performance floor characterization
- Rollout-gate criteria
- Hours/days soak
- RF>2 topology
- NVMe runtime transport proof
- CSI snapshot/expand
@@ -1,70 +0,0 @@
-# Phase 12 P3 — Bounded Runbook
-
-Scope: diagnosis of three symptom classes on the accepted RF=2 sync_all chosen path.
-
-All diagnosis steps reference ONLY explicit bounded read-only surfaces:
- `LookupBlockVolume` — gRPC RPC returning current primary VS + iSCSI address
- `FailoverDiagnostic` — volume-oriented failover state snapshot
- `PublicationDiagnostic` — lookup vs authority coherence snapshot
- `RecoveryDiagnostic` — active recovery task set snapshot
- Blocker ledger — finite file at `phase-12-p3-blockers.md`
-
-## S1: Failover/Recovery Convergence Stall
-
-**Visible symptom:** Volume remains unavailable after a VS death; lookup still returns the old primary.
-
-**Diagnosis surfaces:**
- `LookupBlockVolume(volumeName)` — check if `VolumeServer` is still the dead server
- `FailoverDiagnostic` — check `Volumes[]` for the affected volume
-
-**Diagnosis steps:**
-1. Call `LookupBlockVolume(volumeName)`. If `VolumeServer` changed from the dead server, failover succeeded.
-2. If unchanged: read `FailoverDiagnostic`. Find the volume by name in `Volumes[]`.
-3. If found with `DeferredPromotion=true`: lease-wait — failover is deferred until lease expires.
-4. If found with `PendingRebuild=true`: failover completed, rebuild is pending for the dead server.
-5. If `DeferredPromotionCount[deadServer] > 0` in the aggregate: deferred promotions are queued.
-6. If the volume does not appear in either lookup change or `FailoverDiagnostic`: escalate.
-
-**Conclusion classes (from surfaces only):**
- **Lease-wait:** `FailoverDiagnostic.DeferredPromotionCount[deadServer] > 0` — normal, bounded by lease TTL.
- **Rebuild-pending:** `FailoverDiagnostic.Volumes[].PendingRebuild=true` — failover done, rebuild queued.
- **Converged:** `LookupBlockVolume` shows new primary, no failover entries — resolved.
- **Unresolved:** None of the above — escalate.
-
-## S2: Publication/Lookup Mismatch
-
-**Visible symptom:** `LookupBlockVolume` returns an iSCSI address or volume server that doesn't match expected state.
-
-**Diagnosis surfaces:**
- `LookupBlockVolume(volumeName)` — operator-visible publication
- `PublicationDiagnostic` — explicit coherence check (lookup vs authority)
-
-**Diagnosis steps:**
-1. Call `PublicationDiagnosticFor(volumeName)`. Check `Coherent` field.
-2. If `Coherent=true`: lookup matches registry authority — no mismatch.
-3. If `Coherent=false`: read `Reason` for explanation. Compare `LookupVolumeServer` vs `AuthorityVolumeServer` and `LookupIscsiAddr` vs `AuthorityIscsiAddr`.
-4. Cross-check with `LookupBlockVolume` directly: repeated lookups should be self-consistent.
-
-**Conclusion classes (from surfaces only):**
- **Coherent:** `PublicationDiagnostic.Coherent=true` — no mismatch.
- **Stale client:** Coherent but client sees old value — bounded by client re-query.
- **Unresolved:** `PublicationDiagnostic.Coherent=false` with no transient cause — escalate.
-
-## S3: Leftover Runtime Work After Convergence
-
-**Visible symptom:** After volume deletion or steady-state convergence, recovery tasks should have drained.
-
-**Diagnosis surfaces:**
- `RecoveryDiagnostic` — `ActiveTasks` list (replicaIDs with active recovery work)
-
-**Diagnosis steps:**
-1. Call `RecoveryManager.DiagnosticSnapshot()`. Read `ActiveTasks`.
-2. If `ActiveTasks` is empty: clean — no leftover work.
-3. If non-empty: check whether any task replicaID contains the deleted volume's path.
-4. If a deleted volume's replicaID is present in `ActiveTasks`: residue — escalate.
-5. If all tasks are for live volumes: non-empty but expected — normal in-flight work.
-
-**Conclusion classes (from surfaces only):**
- **Clean:** `RecoveryDiagnostic.ActiveTasks` is empty — runtime converged.
- **Non-empty, no residue:** Tasks present but none for the deleted/converged volume — normal.
- **Residue:** Deleted volume's replicaID still in `ActiveTasks` — escalate.
@@ -1,101 +0,0 @@
-# Phase 12 P4 — Performance Floor Summary
-
-Date: 2026-04-02
-Scope: bounded performance floor for the accepted RF=2, sync_all chosen path.
-
-## Workload Envelope
-
-| Parameter | Value |
-|-----------|-------|
-| Topology | RF=2, sync_all |
-| Operations | 4K random write, 4K random read, sequential write, sequential read |
-| Runtime | Steady-state, no failover, no disturbance |
-| Path | Accepted chosen path (same as P1/P2/P3) |
-
-## Environment
-
-### Unit Test Harness (engine-local)
-
-| Parameter | Value |
-|-----------|-------|
-| Name | `TestP12P4_PerformanceFloor_Bounded` |
-| Location | `weed/server/qa_block_perf_test.go` |
-| Platform | Single-process, local disk |
-| Volume | 64MB, 4K blocks, 16MB WAL |
-| Writer | Single-threaded (worst-case for group commit) |
-| Replication | Not exercised (engine-local only) |
-| Measurement | Worst of 3 iterations (floor, not peak) |
-
-### Production Baseline (cross-machine)
-
-| Parameter | Value |
-|-----------|-------|
-| Name | `baseline-roce-20260401` |
-| Location | `learn/projects/sw-block/test/results/baseline-roce-20260401.md` |
-| Hardware | m01 (10.0.0.1) - M02 (10.0.0.3), 25Gbps RoCE |
-| Protocol | NVMe-TCP |
-| Volume | 2GB, RF=2, sync_all, cross-machine replication |
-| Writer | fio, QD1-128, j=4 |
-
-## Floor Table: Production (RF=2, sync_all, NVMe-TCP, 25Gbps RoCE)
-
-These are measured floor values from the production baseline, not the unit test.
-
-| Workload | Floor IOPS | Notes |
-|----------|-----------|-------|
-| 4K random write QD1 | 28,347 | Barrier round-trip limited (flat across QD) |
-| 4K random write QD32 | 28,453 | Same barrier ceiling |
-| 4K random read QD32 | 136,648 | No replication overhead |
-| Mixed 70/30 QD32 | 28,423 | Write-side limited |
-
-Latency: Write latency is bounded by sync_all barrier round-trip (~35us at QD1).
-Read latency: sub-microsecond for cached, single-digit microseconds for extent.
-
-## Floor Table: Engine-Local (unit test harness)
-
-These values are measured by `TestP12P4_PerformanceFloor_Bounded` on the dev machine.
-They characterize the engine I/O floor WITHOUT transport or replication.
-Actual values vary by hardware; the test produces them on each run.
-
-| Workload | Metric | Method | Gate |
-|----------|--------|--------|------|
-| 4K random write | Floor IOPS, Avg/P50/P99/Max latency | Worst of 3 iterations | >= 1,000 IOPS, P99 <= 100ms |
-| 4K random read | Floor IOPS, Avg/P50/P99/Max latency | Worst of 3 iterations | >= 5,000 IOPS |
-| 4K sequential write | Floor IOPS, Avg/P50/P99/Max latency | Worst of 3 iterations | >= 2,000 IOPS, P99 <= 100ms |
-| 4K sequential read | Floor IOPS, Avg/P50/P99/Max latency | Worst of 3 iterations | >= 10,000 IOPS |
-
-Gate thresholds are regression gates enforced in code (`perfFloorGates` in `qa_block_perf_test.go`).
-Set at ~10% of measured values to tolerate slow CI/VM hardware while catching catastrophic regressions.
-
-## Cost Summary
-
-| Cost | Value | Source |
-|------|-------|--------|
-| WAL write amplification | 2x minimum | Engine design: each write → WAL + eventual extent flush |
-| Replication tax (RF=2 sync_all vs RF=1) | -56% | baseline-roce-20260401.md (NVMe-TCP, 25Gbps RoCE) |
-| Replication tax (RF=2 sync_all vs RF=1, iSCSI 1Gbps) | -56% | baseline-roce-20260401.md |
-| Degraded mode penalty (sync_all RF=2, one replica dead) | -66% | baseline-roce-20260401.md (barrier timeout) |
-| Group commit | 1 fdatasync per batch | Amortizes sync cost across concurrent writers |
-
-## Acceptance Evidence
-
-| Item | Evidence | Type |
-|------|----------|------|
-| Floor gates pass | `perfFloorGates` thresholds enforced per workload | Acceptance |
-| Workload runs repeatably | `TestP12P4_PerformanceFloor_Bounded` passes | Acceptance |
-| Cost statement is bounded | `TestP12P4_CostCharacterization_Bounded` passes | Acceptance |
-| Production baseline exists | `baseline-roce-20260401.md` with measured values | Acceptance |
-| Floor is worst-of-N, not peak | Test takes minimum IOPS across 3 iterations | Method |
-| Regression-safe | Test fails if floor drops below gate (blocks rollout) | Acceptance |
-| Replication tax documented | -56% from measured production baseline | Support telemetry |
-
-## What P4 does NOT claim
-
- This is not a claim that the measured floor is "good enough" for any specific application.
- This does not claim readiness for failover-under-load scenarios.
- This does not claim readiness for hours/days soak under load.
- This does not claim readiness for RF>2 topologies.
- This does not claim readiness for all transport combinations (iSCSI + NVMe + kernel versions).
- This does not claim readiness for production rollout beyond the explicitly named launch envelope.
- Engine-local floor numbers are not production floor numbers.
- The replication tax is measured on one specific hardware configuration and may differ on other hardware.
@@ -1,64 +0,0 @@
-# Phase 12 P4 — Rollout Gates
-
-Date: 2026-04-02
-Scope: bounded first-launch envelope for the accepted RF=2, sync_all chosen path.
-
-This is a bounded first-launch envelope, not general readiness.
-
-## Supported Launch Envelope
-
-Only the transport/network combinations with measured baselines are included.
-
-| Parameter | Value |
-|-----------|-------|
-| Topology | RF=2, sync_all |
-| Transport + Network | NVMe-TCP @ 25Gbps RoCE (measured), iSCSI @ 25Gbps RoCE (measured), iSCSI @ 1Gbps (measured) |
-| NOT included | NVMe-TCP @ 1Gbps (not measured) |
-| Volume size | Up to 2GB (tested baseline) |
-| Failover | Lease-based, bounded by TTL (30s default) |
-| Recovery | Catch-up-first, rebuild fallback |
-| Degraded mode | Documented -66% write penalty (sync_all RF=2, one replica dead) |
-
-## Cleared Gates
-
-| Gate | Evidence | Status | Notes |
-|------|----------|--------|-------|
-| G1 | P1 disturbance tests pass | Cleared | Restart/reconnect correctness under disturbance |
-| G2 | P2 soak tests pass | Cleared | Repeated create/failover/recover cycles, no drift |
-| G3 | P3 diagnosability tests pass | Cleared | Explicit bounded diagnosis surfaces for all symptom classes |
-| G4 | P4 floor gates pass | Cleared | Explicit IOPS thresholds + P99 ceilings enforced per workload in code |
-| G5 | P4 cost characterization bounded | Cleared | WAL 2x write amp, -56% replication tax documented |
-| G6 | Production baseline exists | Cleared | baseline-roce-20260401.md: 28.4K write IOPS, 136.6K read IOPS |
-| G8 | Floor gates are regression-safe | Cleared | Test fails if any workload drops below defined minimum IOPS or exceeds P99 ceiling |
-| G7 | Blocker ledger finite | Cleared | 3 diagnosed (B1-B3) + 3 unresolved (U1-U3), all explicit |
-
-## Remaining Blockers / Exclusions
-
-| Exclusion | Why | Impact |
-|-----------|-----|--------|
-| E1 | Failover-under-load perf not measured | Cannot claim bounded perf during failover |
-| E2 | Hours/days soak not run | Cannot claim long-run stability under sustained load |
-| E3 | RF>2 not measured | Cannot claim perf floor for RF=3+ |
-| E4 | Broad transport matrix not tested | Cannot claim parity across all kernel/NVMe/iSCSI versions |
-| E5 | Degraded mode is severe (-66%) | sync_all RF=2 has sharp write cliff on replica death |
-| E6 | V2 stale-epoch at orchestrator level (U1 from P3) | V1 guards suffice; V2 is secondary path |
-| E7 | gRPC stream transport not exercised in unit tests (U3 from P3) | Blocks full integration test, not correctness |
-
-## Reject Conditions
-
-This launch envelope should be REJECTED if:
-
-1. Any P1/P2/P3 test regresses (correctness/stability/diagnosability gate violated)
-2. Production baseline numbers are not reproducible on the target hardware
-3. Degraded mode behavior (-66% cliff) is not acceptable for the deployment scenario
-4. The deployment requires RF>2, failover-under-load guarantees, or long soak proof
-5. The deployment requires transport combinations not covered by the baseline
-
-## What P4 does NOT claim
-
- This does not claim general production readiness.
- This does not claim readiness for any deployment outside the named launch envelope.
- This does not claim that the performance floor is optimal or final.
- This does not claim that the degraded-mode penalty is acceptable (deployment-specific decision).
- This does not claim hours/days stability under sustained load.
- This is a bounded first-launch gate, not a broad rollout approval.
@@ -1,443 +0,0 @@
-# Phase 12
-
-Date: 2026-04-02
-Status: accepted
-Purpose: move the accepted chosen-path implementation from candidate-safe product closure toward production-safe behavior under restart, disturbance, and operational reality
-
-## Why This Phase Exists
-
-`Phase 09` accepted production-grade execution closure on the chosen path.
-`Phase 10` accepted bounded control-plane closure on that same path.
-`Phase 11` accepted bounded product-surface rebinding on that same path.
-
-What remains is no longer:
-
-1. whether the chosen backend path works
-2. whether selected product surfaces can be rebound onto it
-
-It is now:
-
-1. whether the chosen path stays correct under restart, failover, rejoin, and repeated disturbance
-2. whether long-run behavior is stable enough for serious production use
-3. whether operators can diagnose, bound, and reason about failures in practice
-4. whether remaining production blockers are explicit and finite
-
-## Phase Goal
-
-Move from candidate-safe chosen-path closure to explicit production-hardening closure planning and execution.
-
-Execution note:
-
-1. treat `P0` as real planning work, not placeholder prose
-2. use `phase-12-log.md` as the technical pack for:
-   - step breakdown
-   - hard indicators
-   - reject shapes
-   - assignment text for `sw` and `tester`
-
-## Scope
-
-### In scope
-
-1. restart/recovery stability under repeated disturbance
-2. long-run / soak viability planning and evidence design
-3. operational diagnosability and blocker accounting
-4. bounded hardening slices that do not reopen accepted earlier semantics casually
-
-### Out of scope
-
-1. re-discovering core protocol semantics already accepted in `Phase 09` / `Phase 10`
-2. re-scoping `Phase 11` product rebinding work unless a hardening proof exposes a real bug
-3. broad new feature expansion unrelated to hardening
-4. unbounded product-surface additions
-
-## Phase 12 Items
-
-### P0: Hardening Plan Freeze
-
-Goal:
-
- convert `Phase 12` from a broad “hardening” label into a bounded execution plan with explicit first slices, hard indicators, and reject shapes
-
-Accepted decision target:
-
-1. define the first hardening slices and their order
-2. define what counts as production-hardening evidence versus support evidence
-3. define which accepted surfaces become the first disturbance targets
-
-Planned first hardening areas:
-
-1. restart / rejoin / repeated failover disturbance
-2. long-run / soak stability
-3. operational diagnosis quality and blocker accounting
-4. performance floor and cost characterization only after correctness-hardening slices are bounded
-
-Status:
-
- accepted
-
-### Later candidate slices inside `Phase 12`
-
-1. `P1`: restart / recovery disturbance hardening
-2. `P2`: soak / long-run stability hardening
-3. `P3`: diagnosability / blocker accounting / runbook hardening
-4. `P4`: performance floor and rollout-gate hardening
-
-### P1: Restart / Recovery Disturbance Hardening
-
-Goal:
-
- prove the accepted chosen path remains correct under restart, rejoin, repeated failover, and disturbance ordering
-
-Acceptance object:
-
-1. `P1` accepts correctness under restart/disturbance on the chosen path
-2. it does not accept merely that recovery-related code paths exist
-3. it does not accept merely that the system eventually seems to recover in a loose or approximate sense
-
-Execution steps:
-
-1. Step 1: disturbance contract freeze
-   - define the bounded disturbance classes for the first hardening slice:
-     - restart with same lineage
-     - restart with changed address / refreshed publication
-     - repeated failover / rejoin cycles
-     - delayed or stale signal arrival after restart/failover
-2. Step 2: implementation hardening
-   - harden ownership/control reconstruction on the already accepted chosen path
-   - keep identity, epoch, session, and publication truth coherent across disturbance
-3. Step 3: proof package
-   - prove repeated disturbance correctness on the chosen path
-   - prove stale or delayed signals fail closed rather than silently corrupting ownership truth
-   - prove no-overclaim around soak, perf, or broader production readiness
-
-Required scope:
-
-1. restart/rejoin correctness for the accepted chosen path
-2. publication/address refresh correctness without identity drift
-3. repeated ownership/control transitions under failover and rejoin
-4. bounded reject behavior for stale heartbeat/control signals after disturbance
-
-Must prove:
-
-1. post-restart chosen-path ownership is reconstructed from accepted truth rather than accidental local leftovers
-2. stale or delayed signals after restart/failover are rejected or explicitly bounded
-3. repeated failover/rejoin cycles preserve identity, epoch/session monotonicity, and convergence on the chosen path
-4. acceptance wording stays bounded to disturbance correctness rather than broad production-readiness claims
-
-Reuse discipline:
-
-1. `weed/server/block_recovery.go` and related tests may be updated in place as the primary restart/recovery ownership surface
-2. `weed/server/master_block_failover.go`, `weed/server/master_block_registry.go`, and `weed/server/volume_server_block.go` may be updated in place as the accepted control/runtime disturbance surfaces
-3. `weed/server/block_recovery_test.go`, `weed/server/block_recovery_adversarial_test.go`, and focused `qa_block_*` tests should carry the main proof burden
-4. `weed/storage/blockvol/*` and `weed/storage/blockvol/v2bridge/*` are reference only unless disturbance hardening exposes a real bug in accepted earlier closure
-5. no reused V1 surface may silently redefine chosen-path ownership truth, recovery choice, or disturbance acceptance wording
-
-Verification mechanism:
-
-1. focused restart/rejoin/failover integration tests on the chosen path
-2. adversarial checks for stale or delayed control/heartbeat arrival after disturbance
-3. explicit no-overclaim review so `P1` does not absorb soak/perf/product-expansion work
-
-Hard indicators:
-
-1. one accepted restart correctness proof:
-   - restart on the chosen path reconstructs valid ownership/control state
-   - post-restart behavior does not depend on accidental pre-restart leftovers
-2. one accepted rejoin/publication-refresh proof:
-   - changed address or publication refresh does not break identity truth or visibility
-3. one accepted repeated-disturbance proof:
-   - repeated failover/rejoin cycles converge without epoch/session regression
-4. one accepted stale-signal proof:
-   - delayed heartbeat/control signals after disturbance do not re-authorize stale ownership
-5. one accepted boundedness proof:
-   - `P1` claims correctness under disturbance, not soak, perf, or rollout readiness
-
-Reject if:
-
-1. evidence only shows that recovery code paths execute, rather than that correctness is preserved under disturbance
-2. tests prove only one happy restart path and skip stale/delayed signal shapes
-3. identity, epoch/session, or publication truth can drift across restart/rejoin
-4. `P1` quietly absorbs soak, diagnosability, perf, or new product-surface work
-
-Status:
-
- accepted
-
-Carry-forward from `P0`:
-
-1. the hardening object is the accepted chosen path from `Phase 09` + `Phase 10` + `Phase 11`
-2. `P1` is the first correctness-hardening slice because disturbance threatens correctness before soak or perf
-3. later `P2` / `P3` / `P4` remain distinct acceptance objects and should not be absorbed into `P1`
-
-### P2: Soak / Long-Run Stability Hardening
-
-Goal:
-
- prove the accepted chosen path remains viable over longer duration and repeated operation without hidden state drift
-
-Acceptance object:
-
-1. `P2` accepts bounded long-run stability on the chosen path under repeated operation or soak-like repetition
-2. it does not accept merely that one disturbance test can be repeated many times manually
-3. it does not accept diagnosability, performance floor, or rollout readiness by implication
-
-Execution steps:
-
-1. Step 1: soak contract freeze
-   - define one bounded repeated-operation envelope for the chosen path:
-     - repeated create / failover / recover / steady-state cycles
-     - repeated heartbeat / control / recovery interaction
-     - repeated publication / ownership convergence checks
-   - define what counts as state drift versus expected bounded churn
-2. Step 2: harness and evidence path
-   - build or adapt one repeatable soak/repeated-cycle harness on the accepted chosen path
-   - collect stable end-of-cycle truth rather than only transient pass/fail output
-3. Step 3: proof package
-   - prove no hidden state drift across repeated cycles
-   - prove no unbounded growth/leak in the bounded chosen-path runtime state
-   - prove no-overclaim around diagnosability, perf, or production rollout
-
-Required scope:
-
-1. repeated-cycle correctness on the accepted chosen path
-2. stable end-of-cycle ownership/control/publication truth after many cycles
-3. bounded runtime-state hygiene across repeated operation
-4. explicit distinction between acceptance evidence and support telemetry
-
-Must prove:
-
-1. repeated chosen-path cycles converge to the same bounded truth rather than accumulating semantic drift
-2. registry / VS-visible / product-visible state remain mutually coherent after repeated cycles
-3. repeated operation does not leave unbounded leftover tasks, sessions, or stale runtime ownership artifacts within the tested envelope
-4. acceptance wording stays bounded to long-run stability rather than diagnosability/perf/launch claims
-
-Reuse discipline:
-
-1. `weed/server/qa_block_*test.go`, `block_recovery_test.go`, and related hardening tests may be updated in place as the primary repeated-cycle proof surface
-2. testrunner / infra / metrics helpers may be reused as support instrumentation, but support telemetry must not replace acceptance assertions
-3. `weed/server/master_block_failover.go`, `master_block_registry.go`, `volume_server_block.go`, and `block_recovery.go` may be updated in place only if repeated-cycle hardening exposes a real bug
-4. `weed/storage/blockvol/*` and `weed/storage/blockvol/v2bridge/*` remain reference only unless soak evidence exposes a real accepted-path mismatch
-5. no reused V1 surface may silently redefine the chosen-path steady-state truth, drift criteria, or soak acceptance wording
-
-Verification mechanism:
-
-1. one bounded repeated-cycle or soak harness on the chosen path
-2. explicit end-of-cycle assertions for ownership/control/publication truth
-3. explicit checks for bounded runtime-state hygiene after repeated cycles
-4. no-overclaim review so `P2` does not absorb `P3` diagnosability or `P4` perf/rollout work
-
-Hard indicators:
-
-1. one accepted repeated-cycle proof:
-   - the chosen path completes many bounded cycles without semantic drift
-   - end-of-cycle truth remains coherent after each cycle
-2. one accepted state-hygiene proof:
-   - no unbounded leftover runtime artifacts accumulate within the tested envelope
-3. one accepted long-run stability proof:
-   - stability claims are based on repeated evidence, not one-shot reruns
-4. one accepted boundedness proof:
-   - `P2` claims soak/long-run stability only, not diagnosability, perf, or rollout readiness
-
-Reject if:
-
-1. evidence is only a renamed rerun of `P1` disturbance tests
-2. the slice counts iterations but never checks end-of-cycle truth for drift
-3. support telemetry is presented without a hard acceptance assertion
-4. `P2` quietly absorbs diagnosability, perf, or launch-readiness claims
-
-Status:
-
- accepted
-
-Carry-forward from `P1`:
-
-1. bounded restart/disturbance correctness is now accepted on the chosen path
-2. `P2` now asks whether that accepted path stays stable across repeated operation without hidden drift
-3. later `P3` / `P4` remain distinct acceptance objects and should not be absorbed into `P2`
-
-### P3: Diagnosability / Blocker Accounting / Runbook Hardening
-
-Goal:
-
- make failures, residual blockers, and operator-visible diagnosis quality explicit and reviewable on the accepted chosen path
-
-Acceptance object:
-
-1. `P3` accepts bounded diagnosability / blocker accounting on the chosen path
-2. it does not accept merely that some logs or debug strings exist
-3. it does not accept performance floor or rollout readiness by implication
-
-Execution steps:
-
-1. Step 1: diagnosability contract freeze
-   - define one bounded diagnosis envelope for the accepted chosen path:
-     - failover / recovery does not converge in time
-     - publication / lookup truth does not match authority truth
-     - residual runtime work or stale ownership artifacts remain after an operation
-     - known production blockers remain open and must be made explicit
-   - define what counts as operator-visible diagnosis versus engineer-only source spelunking
-2. Step 2: evidence-surface and blocker-ledger hardening
-   - identify or harden the minimum operator-visible surfaces needed to classify the bounded failure classes
-   - make residual blockers explicit, finite, and reviewable rather than implicit tribal knowledge
-3. Step 3: proof package
-   - prove at least one bounded diagnosis loop closes from symptom to owning truth/blocker
-   - prove blocker accounting is explicit and does not hide unknown gaps behind “hardening later” language
-   - prove no-overclaim around perf, launch readiness, or broad topology support
-
-Required scope:
-
-1. operator-visible symptoms/logs/status for bounded chosen-path failure classes
-2. one explicit mapping from symptom to ownership/control/runtime/publication truth
-3. one explicit blocker ledger for unresolved production-hardening gaps
-4. bounded runbook guidance for diagnosis of the accepted chosen path
-
-Must prove:
-
-1. bounded chosen-path failures can be distinguished with explicit operator-visible evidence rather than debugger-only knowledge
-2. at least one diagnosis loop closes from visible symptom to the relevant authority/runtime truth without semantic ambiguity
-3. residual blockers are explicit, finite, and named with a clear boundary rather than scattered across chats or memory
-4. acceptance wording stays bounded to diagnosability / blocker accounting rather than perf or rollout claims
-
-Reuse discipline:
-
-1. `weed/server/qa_block_*test.go`, `block_recovery*_test.go`, and focused hardening tests may be updated in place where they can prove a bounded diagnosis loop on the accepted path
-2. `weed/server/master_block_registry.go`, `master_block_failover.go`, `volume_server_block.go`, and `block_recovery.go` may be updated in place only if diagnosability work exposes a real visibility gap in accepted-path behavior
-3. lightweight status/logging surfaces and bounded runbook docs may be updated in place as support artifacts, but support artifacts must not replace acceptance assertions
-4. `weed/storage/blockvol/*` and `weed/storage/blockvol/v2bridge/*` remain reference only unless diagnosability work exposes a real accepted-path mismatch
-5. no reused V1 surface may silently redefine chosen-path truth, blocker boundaries, or diagnosis acceptance wording
-
-Verification mechanism:
-
-1. one bounded diagnosis-loop proof on the accepted chosen path
-2. one explicit blocker ledger or equivalent review artifact with finite named items
-3. one explicit check that operator-visible evidence matches the underlying accepted truth being diagnosed
-4. no-overclaim review so `P3` does not absorb `P4` perf/rollout work
-
-Hard indicators:
-
-1. one accepted symptom-classification proof:
-   - bounded failure classes can be told apart by explicit operator-visible evidence
-2. one accepted diagnosis-loop proof:
-   - a visible symptom can be traced to the relevant ownership/control/runtime/publication truth
-3. one accepted blocker-accounting proof:
-   - unresolved blockers are explicit, finite, and reviewable
-4. one accepted boundedness proof:
-   - `P3` claims diagnosability / blockers only, not perf floor or rollout readiness
-
-Reject if:
-
-1. the slice merely adds logs or debug strings without proving diagnostic usefulness
-2. blockers remain implicit, scattered, or dependent on private memory of prior chats
-3. diagnosis requires debugger/source-level spelunking instead of bounded operator-visible evidence
-4. `P3` quietly absorbs perf, rollout, or broad product/topology expansion claims
-
-Status:
-
- accepted
-
-Carry-forward from `P2`:
-
-1. bounded restart/disturbance correctness and bounded long-run stability are now accepted on the chosen path
-2. `P3` now asks whether bounded failures and residual gaps are explicit and diagnosable in operator-facing terms
-3. later `P4` remains a distinct acceptance object and should not be absorbed into `P3`
-
-### P4: Performance Floor / Rollout Gates
-
-Goal:
-
- define explicit performance floor, cost characterization, and rollout-gate criteria without letting perf claims replace correctness hardening
-
-Acceptance object:
-
-1. `P4` accepts a bounded performance floor and a bounded rollout-gate package for the accepted chosen path
-2. it does not accept generic “performance is good” prose or one-off fast runs
-3. it does not accept broad production rollout readiness outside the explicitly named launch envelope
-
-Execution steps:
-
-1. Step 1: performance-floor contract freeze
-   - define one bounded workload envelope for the accepted chosen path
-   - define which metrics count as acceptance evidence:
-     - throughput / latency floor
-     - resource-cost envelope
-     - disturbance-free steady-state behavior
-   - define which metrics are support-only telemetry
-2. Step 2: benchmark and cost characterization
-   - run one repeatable benchmark package against the accepted chosen path
-   - record measured floor values and cost trade-offs rather than “fast enough” wording
-3. Step 3: rollout-gate package
-   - translate accepted correctness, soak, diagnosability, and perf evidence into one bounded launch envelope
-   - make explicit which blockers are cleared, which remain, and what the first supported rollout shape is
-
-Required scope:
-
-1. one bounded benchmark matrix on the accepted chosen path
-2. one explicit performance floor statement backed by measured evidence
-3. one explicit resource-cost characterization
-4. one rollout-gate / launch-envelope artifact with finite named requirements and exclusions
-
-Must prove:
-
-1. performance claims are tied to a named workload envelope rather than generic optimism
-2. the chosen path has a measurable minimum acceptable floor within that envelope
-3. rollout discussion is bounded by explicit gates and supported scope, not implied from prior slice acceptance
-4. acceptance wording stays bounded to performance floor / rollout gates rather than broad production success claims
-
-Reuse discipline:
-
-1. `weed/server/qa_block_*test.go`, testrunner scenarios, and focused perf/support harnesses may be updated in place as the primary measurement surface
-2. `weed/server/*`, `weed/storage/blockvol/*`, and `weed/storage/blockvol/v2bridge/*` may be updated in place only if performance-floor work exposes a real bug or a measurement-surface gap
-3. `sw-block/.private/phase/` docs may be updated in place for the rollout-gate artifact and measured envelope
-4. support telemetry may help characterize cost, but support telemetry must not replace the explicit floor/gate assertions
-5. no reused V1 surface may silently redefine chosen-path truth, launch envelope, or rollout-gate wording
-
-Verification mechanism:
-
-1. one repeatable bounded benchmark package on the accepted chosen path
-2. one explicit measured floor summary with named workload and cost envelope
-3. one explicit rollout-gate artifact naming:
-   - supported launch envelope
-   - cleared blockers
-   - remaining blockers
-   - reject conditions for rollout
-4. no-overclaim review so `P4` does not turn into generic launch optimism
-
-Hard indicators:
-
-1. one accepted performance-floor proof:
-   - measured floor values exist for the named workload envelope
-2. one accepted cost-characterization proof:
-   - resource/replication tax or similar bounded cost is explicit
-3. one accepted rollout-gate proof:
-   - the first supported launch envelope is explicit and finite
-4. one accepted boundedness proof:
-   - `P4` claims only the bounded floor/gates it actually measures
-
-Reject if:
-
-1. the slice presents isolated benchmark numbers without a named workload contract
-2. rollout gates are replaced by vague “looks ready” wording
-3. support telemetry is presented without an explicit acceptance threshold or gate
-4. `P4` quietly absorbs broad new topology, product-surface, or generic ops-tooling expansion
-
-Status:
-
- accepted
-
-Carry-forward from `P3`:
-
-1. bounded disturbance correctness, bounded soak stability, and bounded diagnosability / blocker accounting are now accepted on the chosen path
-2. `P4` now asks whether that accepted path has an explicit measured floor and an explicit first-launch envelope
-3. later work after `Phase 12` should be a productionization program, not another hidden hardening slice
-
-## Phase Close-Out Note
-
-`Phase 12` is now accepted as bounded production hardening on the chosen path:
-
-1. `P1` accepted disturbance correctness
-2. `P2` accepted bounded soak / long-run stability
-3. `P3` accepted diagnosability / blocker accounting / runbook hardening
-4. `P4` accepted bounded performance floor / rollout-gate hardening
-
-Next work should open a new phase or program rather than silently continuing inside `Phase 12`.
@@ -1,124 +0,0 @@
-# CP13-1 Baseline Report
-
-Date: 2026-04-02
-Commit: c0a805184 (feature/sw-block HEAD)
-Runner: `go test ./weed/storage/blockvol/ -v -count=1 -timeout 120s`
-Protocol changes in this checkpoint: NONE — test-first baseline only
-
-## Category 1: Address Truth
-
-| Result | Test | Reason |
-|--------|------|--------|
-| PASS | `TestCanonicalizeAddr_WildcardIPv4_UsesAdvertised` | canonicalization infra works |
-| PASS | `TestCanonicalizeAddr_WildcardIPv6_UsesAdvertised` | canonicalization infra works |
-| PASS | `TestCanonicalizeAddr_NilIP_UsesAdvertised` | canonicalization infra works |
-| PASS | `TestCanonicalizeAddr_AlreadyCanonical_Unchanged` | no-op on canonical input |
-| PASS | `TestCanonicalizeAddr_Loopback_Unchanged` | loopback preserved intentionally |
-| PASS | `TestCanonicalizeAddr_NoAdvertised_FallsBackToOutbound` | fallback path works |
-| PASS* | `TestBug3_ReplicaAddr_MustBeIPPort_WildcardBind` | documents gap: ReplicaReceiver may return `:port` not `ip:port` on wildcard bind; test passes as documentation, not as proof of fix → CP13-2 |
-
-## Category 2: Durable Progress Truth
-
-| Result | Test | Reason |
-|--------|------|--------|
-| PASS | `TestReplicaProgress_BarrierUsesFlushedLSN` | current code passes this test; suggests CP13-3 behavior may already exist |
-| PASS | `TestReplicaProgress_FlushedLSNMonotonicWithinEpoch` | current code passes this test; suggests CP13-3 behavior may already exist |
-| PASS | `TestBarrier_RejectsReplicaNotInSync` | barrier rejects non-InSync replica |
-| PASS | `TestBarrier_EpochMismatchRejected` | barrier rejects epoch mismatch |
-| PASS | `TestBarrier_DuringCatchup_Rejected` | current code passes this test; suggests CP13-4 behavior may already exist |
-| PASS | `TestBarrier_ReplicaSlowFsync_Timeout` | barrier timeout on slow replica |
-| PASS | `TestBarrierResp_FlushedLSN_Roundtrip` | barrier response wire format carries flushedLSN |
-| PASS | `TestBarrierResp_BackwardCompat_1Byte` | backward compat with old 1-byte response |
-| PASS | `TestReplica_FlushedLSN_OnlyAfterSync` | flushedLSN only updated after fdatasync |
-| PASS | `TestReplica_FlushedLSN_NotOnReceive` | flushedLSN not updated on entry receive |
-| PASS | `TestShipper_ReplicaFlushedLSN_UpdatedOnBarrier` | shipper tracks replica flushedLSN from barrier |
-| PASS | `TestShipper_ReplicaFlushedLSN_Monotonic` | tracked flushedLSN is monotonic |
-| PASS | `TestShipperGroup_MinReplicaFlushedLSN` | group computes min flushedLSN across replicas |
-| PASS | `TestDistSync_SyncAll_NilGroup_Succeeds` | sync_all with no replicas succeeds locally |
-| PASS | `TestDistSync_SyncAll_AllDegraded_Fails` | sync_all fails when all replicas degraded |
-| PASS | `TestBug2_SyncAll_SyncCache_AfterDegradedShipperRecovers` | current code passes this test; suggests CP13-5 behavior may already exist |
-| PASS | `TestBug1_SyncAll_WriteDuringDegraded_SyncCacheMustFail` | SyncCache correctly fails during degraded |
-
-## Category 3: Reconnect / Catch-up
-
-| Result | Test | Reason |
-|--------|------|--------|
-| PASS | `TestReconnect_CatchupFromRetainedWal` | current code passes this test; suggests CP13-5 catch-up behavior may already exist |
-| PASS* | `TestReconnect_GapBeyondRetainedWal_NeedsRebuild` | correctly fails SyncCache after large gap, but does NOT assert NeedsRebuild state transition — asserts barrier failure only → CP13-5+CP13-7 |
-| PASS | `TestReconnect_EpochChangeDuringCatchup_Aborts` | catch-up aborts on epoch change |
-| PASS | `TestReconnect_CatchupTimeout_TransitionsDegraded` | catch-up timeout → degraded |
-| PASS | `TestAdversarial_FreshShipperUsesBootstrapNotReconnect` | fresh shipper uses bootstrap path |
-| FAIL | `TestAdversarial_ReconnectUsesHandshakeNotBootstrap` | **gap: degraded shipper with prior flushed progress reconnects but barrier fails** — shipper does not catch up before attempting barrier → CP13-5 |
-| PASS | `TestAdversarial_ReplicaRejectsDuplicateLSN` | replica rejects duplicate LSN |
-| PASS | `TestAdversarial_ReplicaRejectsGapLSN` | replica rejects LSN gap |
-| FAIL | `TestAdversarial_CatchupMultipleDisconnects` | **gap: catch-up across multiple disconnect/reconnect cycles fails** — first reconnect barrier fails, subsequent cycles never recover → CP13-5 |
-| PASS | `TestAdversarial_ConcurrentBarrierDoesNotCorruptCatchupFailures` | concurrent barriers don't corrupt counter |
-
-## Category 4: Retention / Rebuild Boundary
-
-| Result | Test | Reason |
-|--------|------|--------|
-| PASS | `TestWalRetention_RequiredReplicaBlocksReclaim` | current code passes this test; suggests CP13-6 retention behavior may already exist |
-| PASS | `TestWalRetention_TimeoutTriggersNeedsRebuild` | current code passes this test; suggests CP13-6 timeout behavior may already exist |
-| PASS* | `TestWalRetention_MaxBytesTriggersNeedsRebuild` | passes but logs "max-bytes retention trigger not implemented yet" — shipper stays Degraded, does not transition to NeedsRebuild → CP13-6 |
-| FAIL | `TestAdversarial_NeedsRebuildBlocksAllPaths` | **gap: after large WAL gap, shipper stays Degraded instead of NeedsRebuild; Ship/Barrier not blocked** → CP13-5+CP13-7 |
-| FAIL | `TestAdversarial_CatchupDoesNotOverwriteNewerData` | **gap: catch-up after disconnect fails at barrier level** — catch-up doesn't complete, so newer-data safety not actually exercised → CP13-5 |
-| PASS | `TestHeartbeat_ReportsPerReplicaState` | heartbeat reports per-replica shipper state |
-| PASS | `TestHeartbeat_ReportsNeedsRebuild` | heartbeat reports NeedsRebuild per-replica |
-| PASS | `TestReplicaState_RebuildComplete_ReentersInSync` | full rebuild cycle: NeedsRebuild → rebuild → InSync |
-| PASS | `TestRebuild_AbortOnEpochChange` | rebuild aborts on epoch change |
-| PASS | `TestRebuild_PostRebuild_FlushedLSN_IsCheckpoint` | post-rebuild flushedLSN = checkpoint |
-
-## Summary
-
-| Category | PASS | FAIL | PASS* | Total |
-|----------|------|------|-------|-------|
-| 1. Address Truth | 6 | 0 | 1 | 7 |
-| 2. Durable Progress Truth | 17 | 0 | 0 | 17 |
-| 3. Reconnect / Catch-up | 7 | 2 | 1 | 10 |
-| 4. Retention / Rebuild | 7 | 2 | 1 | 10 |
-| **Total** | **37** | **4** | **3** | **44** |
-
-## Failure → Checkpoint Mapping
-
-| FAIL Test | Root Cause | Expected to close in |
-|-----------|-----------|----------------------|
-| `TestAdversarial_ReconnectUsesHandshakeNotBootstrap` | degraded shipper reconnects but doesn't catch up before barrier | CP13-5 (reconnect handshake) |
-| `TestAdversarial_CatchupMultipleDisconnects` | repeated disconnect/reconnect cycles don't recover | CP13-5 (reconnect handshake) |
-| `TestAdversarial_NeedsRebuildBlocksAllPaths` | shipper stays Degraded after large gap, should be NeedsRebuild | CP13-5 (gap detection) + CP13-7 (rebuild fallback) |
-| `TestAdversarial_CatchupDoesNotOverwriteNewerData` | catch-up fails at barrier, newer-data safety not exercised | CP13-5 (catch-up protocol) |
-
-Main remaining failures cluster around CP13-5 (reconnect/catch-up), but CP13-7 (rebuild fallback) and part of CP13-6 (max-bytes retention) also remain open.
-
-## PASS* → Checkpoint Mapping
-
-| PASS* Test | Why Not Full Proof | Expected to close in |
-|------------|-------------------|----------------------|
-| `TestBug3_ReplicaAddr_MustBeIPPort_WildcardBind` | documents gap, doesn't prove fix | CP13-2 (canonical addressing) |
-| `TestReconnect_GapBeyondRetainedWal_NeedsRebuild` | asserts barrier failure, not NeedsRebuild state transition | CP13-5 (gap detection) + CP13-7 (rebuild fallback) |
-| `TestWalRetention_MaxBytesTriggersNeedsRebuild` | logs "not implemented", shipper stays Degraded | CP13-6 (max-bytes retention) |
-
-## Remaining Open Checkpoints
-
-This baseline does NOT close any checkpoint. Checkpoint closure requires dedicated review per checkpoint. The baseline only records which tests pass or fail on current code.
-
-Tests passing on current code **suggests** the behavior may already exist, but does not constitute checkpoint acceptance. The following checkpoints still require dedicated review:
-
- **CP13-2** (canonical addressing): 1 PASS* test documents the gap
- **CP13-5** (reconnect/catch-up): 2 FAILs + 1 PASS* directly expose missing protocol
- **CP13-6** (WAL retention): 1 PASS* exposes missing max-bytes trigger
- **CP13-7** (rebuild fallback): 1 FAIL + 1 PASS* expose missing NeedsRebuild transition
-
-## What Was NOT Changed
-
-This baseline was captured on current code without any protocol modifications:
-
- No reconnect handshake changes
- No WAL catch-up logic changes
- No retention policy changes
- No rebuild behavior changes
- No barrier protocol changes
- No state machine changes
- No new protocol code of any kind
-
-All 4 FAILs and 3 PASS* entries expose real gaps that exist in the current codebase.
@@ -1,119 +0,0 @@
-# CP13-3 Durable Progress Truth — Contract Review + Proof Package
-
-Date: 2026-04-03
-Commit: ac962fc83 → updated with legacy-response rejection fix
-
-## Durable Progress Contract
-
-### Definition
-
-`replicaFlushedLSN` is the **sole authority** for replica durability in the sync_all path.
-
-It means: the replica has called `fd.Sync()` (WAL fdatasync) for all entries through this LSN, and the barrier response carrying this value has reached the primary.
-
-### What is NOT durable authority
-
-| Variable | Location | Role | Why NOT authority |
-|----------|----------|------|-------------------|
-| `shippedLSN` | `wal_shipper.go:269` | Diagnostic | Tracks last LSN sent over TCP; receipt not confirmed |
-| `receivedLSN` | `replica_apply.go:362` | Intermediate | Entry applied to WAL buffer; not yet fsynced |
-| `sentLSN` / transport progress | shipper send loop | Diagnostic | TCP write completed; no durability guarantee |
-
-### Where durable authority lives
-
-| Component | File | How it works |
-|-----------|------|-------------|
-| Replica: barrier handler | `replica_barrier.go:53-110` | Waits for `receivedLSN >= req.LSN`, calls `fd.Sync()`, advances `flushedLSN` only after sync succeeds, returns `BarrierResponse{FlushedLSN: flushed}` |
-| Shipper: barrier consumer | `wal_shipper.go:220-238` | Reads `resp.FlushedLSN`, updates `replicaFlushedLSN` via monotonic CAS (never decreases) |
-| Shipper: explicit API | `wal_shipper.go:273-278` | `ReplicaFlushedLSN()` is the authoritative API; `ShippedLSN()` has explicit "NOT authoritative" comment at line 268 |
-| Group commit: sync_all | `dist_group_commit.go:15-83` | `BarrierAll(lsnMax)` called in parallel with local WAL sync; sync_all fails if any barrier fails |
-| SyncCache entry | `blockvol.go:774-782` | `groupCommit.Submit()` → distributed sync → barrier → durability |
-
-### Durability proof chain
-
-```
-WriteLBA → appendWithRetry → WAL.Append + Ship (fire-and-forget)
-    ↓
-SyncCache → groupCommit.Submit → distributedSync:
-    ├─ local: walSync (fd.Sync on primary WAL)
-    └─ remote: group.BarrierAll(lsnMax)
-              → shipper.Barrier(lsnMax)
-                → WriteFrame(MsgBarrierReq) to replica
-                → replica handleBarrier:
-                    1. wait receivedLSN >= LSN
-                    2. fd.Sync() — THIS IS THE DURABILITY EVENT
-                    3. advance flushedLSN
-                    4. return BarrierResponse{FlushedLSN}
-                ← ReadFrame(MsgBarrierResp)
-                ← update replicaFlushedLSN (monotonic CAS)
-              → if BarrierOK: markInSync
-    ↓
-sync_all: ALL barriers must succeed → SyncCache returns nil
-sync_all: ANY barrier fails → ErrDurabilityBarrierFailed
-```
-
-## Baseline Test Promotion
-
-The following CP13-1 baseline PASS tests are promoted to CP13-3 proof:
-
-### Primary proofs (directly verify the durable-progress contract)
-
-| Test | What it proves for CP13-3 |
-|------|--------------------------|
-| `TestReplicaProgress_BarrierUsesFlushedLSN` | Barrier success is gated on `replicaFlushedLSN`, not `shippedLSN` |
-| `TestReplicaProgress_FlushedLSNMonotonicWithinEpoch` | `replicaFlushedLSN` never decreases within an epoch |
-| `TestReplica_FlushedLSN_OnlyAfterSync` | `flushedLSN` only advanced after `fd.Sync()` — not on entry receive |
-| `TestReplica_FlushedLSN_NotOnReceive` | Receiving an entry does NOT advance `flushedLSN` — confirms receive != durable |
-| `TestShipper_ReplicaFlushedLSN_UpdatedOnBarrier` | Shipper's tracked `replicaFlushedLSN` comes from barrier response, not from send |
-| `TestShipper_ReplicaFlushedLSN_Monotonic` | Shipper's tracked progress is monotonic (CAS-only, never decreases) |
-| `TestBarrierResp_FlushedLSN_Roundtrip` | Barrier response wire format correctly carries `flushedLSN` |
-| `TestBarrierResp_BackwardCompat_1Byte` | Old 1-byte responses decode to `FlushedLSN=0` (wire compat) |
-| `TestBarrier_LegacyResponseRejectedBySyncAll` | Legacy `BarrierOK` with `FlushedLSN=0` is rejected — no false durability authority |
-
-### Support evidence (adjacent to the contract, not primary proof)
-
-| Test | What it supports |
-|------|-----------------|
-| `TestBarrier_RejectsReplicaNotInSync` | Barrier rejects non-InSync replica — guards barrier correctness |
-| `TestBarrier_EpochMismatchRejected` | Barrier rejects epoch mismatch — guards against stale durability claims |
-| `TestBarrier_ReplicaSlowFsync_Timeout` | Barrier times out on slow fsync — bounded, not unbounded wait |
-| `TestShipperGroup_MinReplicaFlushedLSN` | Group computes min flushedLSN across replicas — multi-replica support |
-| `TestDistSync_SyncAll_NilGroup_Succeeds` | sync_all with no replicas = local-only (correct degenerate case) |
-| `TestDistSync_SyncAll_AllDegraded_Fails` | sync_all fails when all replicas degraded — fail-closed |
-
-### Out of scope for CP13-3
-
-| Test | Why out of scope |
-|------|-----------------|
-| `TestBarrier_DuringCatchup_Rejected` | Barrier during CatchingUp — this is CP13-4 (state machine) |
-| `TestReconnect_*` | Reconnect/catch-up — this is CP13-5 |
-| `TestWalRetention_*` | WAL retention — this is CP13-6 |
-| `TestAdversarial_NeedsRebuild*` | NeedsRebuild state — this is CP13-7 |
-
-## Sender-Side Progress: Explicitly Diagnostic
-
-Code evidence that `shippedLSN` / `sentLSN` are non-authoritative:
-
-```go
-// wal_shipper.go:266-271
-// ShippedLSN returns the highest LSN sent to the replica.
-// This is NOT authoritative for sync durability — use ReplicaFlushedLSN() instead.
-func (s *WALShipper) ShippedLSN() uint64 {
-    return s.shippedLSN.Load()
-}
-```
-
-The comment at line 268 is explicit: sender-side progress is diagnostic only.
-
-## Code Change
-
-One targeted fix in `wal_shipper.go`: `BarrierOK` with `FlushedLSN == 0` now returns
-an error instead of counting as successful sync_all durability. This closes the gap
-where a legacy 1-byte barrier response could pass through as durable authority.
-
-## What CP13-3 Does NOT Close
-
- Reconnect/catch-up protocol (CP13-5)
- WAL retention policy (CP13-6)
- Rebuild fallback (CP13-7)
- Replica state machine transitions beyond barrier eligibility (CP13-4)
@@ -1,91 +0,0 @@
-# CP13-4 Replica State Machine / Barrier Eligibility — Contract Review + Proof Package
-
-Date: 2026-04-03
-Code change: one new test (`TestBarrier_NonEligibleStates_FailClosed`)
-
-## Replica State Set
-
-The replication path uses a bounded 6-state set (`wal_shipper.go:25-30`):
-
-| State | Value | Meaning | Barrier behavior |
-|-------|-------|---------|-----------------|
-| `Disconnected` | 0 | No session (initial state) | Attempts bootstrap/reconnect inside Barrier(); fails if no progress or reconnect fails |
-| `Connecting` | 1 | Socket open, handshake pending | Immediate `ErrReplicaDegraded` |
-| `CatchingUp` | 2 | Connected, replaying missed WAL | Immediate `ErrReplicaDegraded` |
-| `InSync` | 3 | Eligible for sync_all barriers | **Proceeds to barrier request** — only state that can complete barrier successfully |
-| `Degraded` | 4 | Transient failure, retry allowed | Attempts reconnect inside Barrier(); fails if reconnect fails |
-| `NeedsRebuild` | 5 | WAL gap too large, rebuild required | Immediate `ErrReplicaDegraded` |
-
-## Barrier State Gate
-
-`WALShipper.Barrier()` at `wal_shipper.go:160-182`:
-
-```go
-st := s.State()
-switch st {
-case ReplicaInSync:
-    // proceed normally to barrier
-case ReplicaDisconnected, ReplicaDegraded:
-    // attempt reconnect; error if fails
-default:
-    // Connecting, CatchingUp, NeedsRebuild — reject immediately
-    return ErrReplicaDegraded
-}
-```
-
-**Contract (precise):**
-
- **Only `InSync` can complete barrier successfully.** It is the only state that proceeds
-  directly to the barrier request (ensureCtrlConn → MsgBarrierReq → wait for BarrierOK).
- **`Disconnected` and `Degraded` use Barrier() as a recovery entry point.** They attempt
-  bootstrap/reconnect inside the Barrier() call. If recovery succeeds and transitions to
-  InSync, the barrier request proceeds. If recovery fails, the barrier fails.
- **`Connecting`, `CatchingUp`, `NeedsRebuild` are rejected immediately** with `ErrReplicaDegraded`.
-
-The key distinction: Barrier() can be *invoked* from Disconnected/Degraded (as a recovery
-trigger), but only InSync can *satisfy* barrier success. The Disconnected/Degraded paths
-are recovery attempts, not barrier eligibility.
-
-## sync_all Gate
-
-`dist_group_commit.go:59-66`: sync_all counts barrier failures. Any shipper that returns an error from `Barrier()` increments `failCount`. If `failCount > 0`, sync_all returns `ErrDurabilityBarrierFailed`.
-
-Combined with the CP13-3 fix (FlushedLSN=0 rejected), the full chain is:
-1. Only `InSync` shippers proceed to the barrier request
-2. Disconnected/Degraded may recover inside Barrier(), transitioning to InSync before requesting
-3. Only `BarrierOK` with `FlushedLSN > 0` counts as success
-4. sync_all fails if any barrier fails
-
-## Proof Promotion
-
-### Primary proofs (directly verify state/eligibility contract)
-
-| Test | What it proves for CP13-4 |
-|------|--------------------------|
-| `TestBarrier_NonEligibleStates_FailClosed` | 5 sub-cases: Connecting/CatchingUp/NeedsRebuild rejected immediately; Disconnected fails (no recovery on dead addr); InSync enters barrier path (verified by MsgBarrierReq receipt on fake server) |
-| `TestBarrier_RejectsReplicaNotInSync` | SyncCache fails when replica is not InSync (end-to-end) |
-| `TestBarrier_DuringCatchup_Rejected` | Barrier rejected while replica is CatchingUp |
-| `TestDistSync_SyncAll_AllDegraded_Fails` | sync_all fails when all replicas degraded |
-| `TestAdversarial_FreshShipperUsesBootstrapNotReconnect` | Fresh (Disconnected, no prior progress) shipper uses bootstrap path |
-
-### Support evidence
-
-| Test | What it supports |
-|------|-----------------|
-| `TestBarrier_EpochMismatchRejected` | Barrier rejects epoch mismatch — adjacent to eligibility |
-| `TestBarrier_ReplicaSlowFsync_Timeout` | Barrier timeout — bounded failure, not silent success |
-
-### Out of scope for CP13-4
-
-| Test | Why |
-|------|-----|
-| `TestReconnect_*` | Reconnect protocol — CP13-5 |
-| `TestWalRetention_*` | Retention — CP13-6 |
-| `TestAdversarial_NeedsRebuildBlocksAllPaths` | Full NeedsRebuild lifecycle — CP13-5+CP13-7 |
-
-## What CP13-4 Does NOT Close
-
- Reconnect/catch-up protocol (CP13-5)
- WAL retention policy (CP13-6)
- Rebuild fallback (CP13-7)
- The Disconnected/Degraded reconnect paths are tested for failure on dead addresses, but the actual reconnect protocol is CP13-5 scope
@@ -1,89 +0,0 @@
-# CP13-5 Reconnect Handshake + WAL Catch-up — Contract Review + Proof Package
-
-Date: 2026-04-03
-Code change: `blockvol.go` SetReplicaAddrs + `shipper_group.go` AnyHasFlushedProgress
-
-## Reconnect Decision Matrix
-
-| Prior durable progress? | WAL covers gap? | Outcome |
-|------------------------|----------------|---------|
-| No (`hasFlushedProgress=false`) | N/A | Bootstrap: bare Ship + Barrier |
-| Yes | Yes (gap within retained WAL) | Reconnect: ResumeShipReq handshake → catch-up replay → InSync |
-| Yes | No (gap exceeds retained WAL) | Fail closed: NeedsRebuild (CP13-7 scope for full lifecycle) |
-
-## What Changed
-
-**Bug:** `SetReplicaAddrs` created fresh shippers with `hasFlushedProgress=false`, so after
-disconnect + reconnect, the shipper used the bootstrap path instead of the reconnect handshake.
-Bootstrap doesn't replay missed WAL entries, so the barrier waited forever for entries the
-replica never received.
-
-**Fix (`blockvol.go`):** `SetReplicaAddrs` now checks if the old shipper group had any
-shipper with durable progress (`AnyHasFlushedProgress`). If so, new shippers are seeded
-with `hasFlushedProgress=true`, routing them through the reconnect handshake + catch-up path.
-
-**New helper (`shipper_group.go`):** `AnyHasFlushedProgress()` — returns true if any shipper
-in the group has ever received a valid `FlushedLSN > 0` from a barrier response.
-
-## Reconnect Path (production flow)
-
-```
-SetReplicaAddrs(new addresses after reconnect)
-  ├─ old group had flushedProgress? → seed new shippers with hasFlushedProgress=true
-  └─ new shipper created with WAL access
-
-SyncCache → groupCommit.Submit → Barrier(lsnMax)
-  ├─ state=Disconnected + hasFlushedProgress=true + wal != nil
-  │   → doReconnectAndCatchUp()
-  │     → reconnectWithHandshake()
-  │       → TCP connect to new replica address
-  │       → ResumeShipReq{Epoch, PrimaryHeadLSN, RetainStart}
-  │       → replica responds with {Status, ReplicaFlushedLSN}
-  │       → gap analysis: R (replica flushed) vs H (primary head) vs S (retain start)
-  │         ├─ R >= H: already caught up → InSync
-  │         ├─ R >= S: recoverable gap → CatchingUp → runCatchUp(R)
-  │         └─ R < S: gap exceeds retention → NeedsRebuild
-  │     → runCatchUp: stream WAL entries from R to H → replica applies
-  │     → catch-up complete → InSync
-  └─ barrier request proceeds (InSync)
-```
-
-## Baseline FAILs Now Closed
-
-| Test | Was | Now | Why |
-|------|-----|-----|-----|
-| `TestAdversarial_ReconnectUsesHandshakeNotBootstrap` | FAIL | PASS | 3 observable signals: seeded hasFlushedProgress, receivedLSN advance, non-zero replicaFlushedLSN |
-| `TestAdversarial_CatchupMultipleDisconnects` | FAIL | PASS | Repeated SetReplicaAddrs preserves progress seed |
-| `TestAdversarial_CatchupDoesNotOverwriteNewerData` | FAIL | PASS | Catch-up now completes, safety invariant exercised |
-
-## Baseline Tests Promoted to CP13-5 Proof
-
-### Primary proofs
-
-| Test | What it proves |
-|------|---------------|
-| `TestAdversarial_ReconnectUsesHandshakeNotBootstrap` | 3 observable proofs: (1) new shipper seeded with `hasFlushedProgress=true`, (2) replica `receivedLSN` advances during SyncCache (catch-up delivered entries), (3) shipper `replicaFlushedLSN > 0` after barrier |
-| `TestAdversarial_CatchupMultipleDisconnects` | Repeated disconnect/reconnect cycles recover cleanly |
-| `TestAdversarial_CatchupDoesNotOverwriteNewerData` | Catch-up replays missing entries without overwriting newer replica data |
-| `TestReconnect_CatchupFromRetainedWal` | Retained-WAL gap replays and returns to InSync |
-| `TestReconnect_EpochChangeDuringCatchup_Aborts` | Epoch change during catch-up aborts cleanly |
-| `TestReconnect_CatchupTimeout_TransitionsDegraded` | Catch-up timeout → Degraded (bounded failure) |
-| `TestAdversarial_FreshShipperUsesBootstrapNotReconnect` | Fresh shipper (no prior progress) uses bootstrap, not reconnect |
-
-### Support evidence
-
-| Test | What it supports |
-|------|-----------------|
-| `TestReconnect_GapBeyondRetainedWal_NeedsRebuild` | PASS* — asserts barrier failure on large gap, but full NeedsRebuild lifecycle is CP13-7 |
-
-### Still FAIL (CP13-7 scope)
-
-| Test | Why still fails |
-|------|----------------|
-| `TestAdversarial_NeedsRebuildBlocksAllPaths` | Full NeedsRebuild lifecycle — lease expiry + WAL overflow timing; CP13-7 scope |
-
-## What CP13-5 Does NOT Close
-
- Replica-aware WAL retention policy (CP13-6)
- Full NeedsRebuild lifecycle / rebuild execution (CP13-7)
- The `TestAdversarial_NeedsRebuildBlocksAllPaths` failure is a CP13-7 gap, not CP13-5
@@ -1,76 +0,0 @@
-# CP13-6 Replica-Aware WAL Retention — Contract Review + Proof Package
-
-Date: 2026-04-03
-Code change: `shipper_group.go` EvaluateRetentionBudgets (params struct + block-size-aware) + `blockvol.go` caller updates + 3 tests rewritten with hard assertions
-
-## Retention Contract
-
-### Inputs
-
-| Input | Source | How it's used |
-|-------|--------|---------------|
-| `replicaFlushedLSN` | Barrier response (CP13-3 authority) | Retention floor: WAL must keep entries from this LSN forward |
-| `primaryHeadLSN` | `nextLSN.Load() - 1` | Lag calculation: head - replicaFlushed = entries the replica still needs |
-| `lastContactTime` | Barrier/handshake success time | Timeout budget: how long since the replica was heard from |
-
-### Decision matrix
-
-| Condition | Action |
-|-----------|--------|
-| Recoverable replica needs WAL entries | Hold: flusher does not advance tail past `minRecoverableFlushedLSN` |
-| Replica last contact exceeds `walRetentionTimeout` (5min) | Escalate to `NeedsRebuild`, release hold |
-| Replica lag exceeds `walRetentionMaxBytes` (64MB default) | Escalate to `NeedsRebuild`, release hold |
-| Replica in `NeedsRebuild` | Excluded from retention floor (`MinRecoverableFlushedLSN` skips it) |
-| No recoverable replicas | No retention hold (flusher advances freely) |
-
-### Code path
-
-```
-Flusher.FlushOnce()
-  ├─ EvaluateRetentionBudgetsFn()  → shipper_group.EvaluateRetentionBudgets(timeout, maxBytes, primaryHead)
-  │   ├─ for each recoverable shipper:
-  │   │   ├─ timeout exceeded? → state.Store(NeedsRebuild)
-  │   │   └─ lag * 4KB > maxBytes? → state.Store(NeedsRebuild)
-  │   └─ NeedsRebuild shippers excluded from future floor computation
-  ├─ RetentionFloorFn()  → shipper_group.MinRecoverableFlushedLSN()
-  │   └─ returns min flushedLSN of non-NeedsRebuild shippers with prior progress
-  └─ if maxLSN > floorLSN: hold WAL (don't advance tail)
-     else: advance tail normally
-```
-
-## What Changed
-
-**`shipper_group.go`:** `EvaluateRetentionBudgets` now takes `RetentionBudgetParams` struct
-with `Timeout`, `MaxBytes`, `PrimaryHeadLSN`, and `BlockSize` (from volume config).
-Max-bytes lag computed as `entryLag * BlockSize`, not hardcoded 4096.
-Both timeout and max-bytes checks transition to `NeedsRebuild` with real state effects.
-
-**`blockvol.go`:** Added `walRetentionMaxBytes` (64MB default). Callers pass `RetentionBudgetParams`
-with actual `v.super.BlockSize`.
-
-**`sync_all_protocol_test.go`:** All 3 retention tests rewritten with hard assertions (no log-only placeholders).
-
-## Tests Upgraded
-
-All 3 retention tests rewritten from placeholder/PASS* to hard-assertion proofs:
-
-| Test | Was | Now | Hard assertion |
-|------|-----|-----|----------------|
-| `TestWalRetention_RequiredReplicaBlocksReclaim` | PASS (log-only, no assertion) | PASS (hard assert) | `checkpointLSN <= replicaFlushedLSN` — flusher did not advance past retention floor |
-| `TestWalRetention_TimeoutTriggersNeedsRebuild` | PASS (log-only, no assertion) | PASS (hard assert) | `s.State() == NeedsRebuild` + `checkpointAfter > replicaFlushedLSN` (hold released) |
-| `TestWalRetention_MaxBytesTriggersNeedsRebuild` | PASS* (logged "not implemented") | PASS (hard assert) | `s.State() == NeedsRebuild` after lag exceeds 8KB budget |
-
-## Proof Promotion
-
-### Primary proofs
-
-| Test | What it proves |
-|------|---------------|
-| `TestWalRetention_RequiredReplicaBlocksReclaim` | Flusher checkpoint does not advance past `replicaFlushedLSN` while recoverable replica is behind |
-| `TestWalRetention_TimeoutTriggersNeedsRebuild` | Timeout budget → `NeedsRebuild` (State assertion) + checkpoint advances past replicaFlushedLSN after flush (hold-release assertion) |
-| `TestWalRetention_MaxBytesTriggersNeedsRebuild` | Max-bytes budget evaluation transitions shipper to `NeedsRebuild` (verified via `State()` assertion, uses actual `BlockSize` from volume config) |
-
-## What CP13-6 Does NOT Close
-
- Full NeedsRebuild lifecycle / rebuild execution (CP13-7)
- `TestAdversarial_NeedsRebuildBlocksAllPaths` still FAIL (CP13-7)
@@ -1,89 +0,0 @@
-# CP13-7 Rebuild Fallback — Contract Review + Proof Package
-
-Date: 2026-04-03
-Code change: `sync_all_adversarial_test.go` + `sync_all_protocol_test.go` test rewrites
-
-## NeedsRebuild Contract
-
-### Entry
-
-| Trigger | Source | Result |
-|---------|--------|--------|
-| Timeout budget exceeded | `EvaluateRetentionBudgets` (CP13-6) | `state.Store(NeedsRebuild)` |
-| Max-bytes budget exceeded | `EvaluateRetentionBudgets` (CP13-6) | `state.Store(NeedsRebuild)` |
-| Reconnect detects impossible progress | `reconnectWithHandshake` (CP13-5) | Returns `NeedsRebuild` |
-| Reconnect detects gap beyond retained WAL | `reconnectWithHandshake` (CP13-5) | Returns `NeedsRebuild` |
-| Catch-up failures exceed max retries | `doReconnectAndCatchUp` | `state.Store(NeedsRebuild)` |
-
-### Blocking (fail-closed)
-
-| Path | Behavior when NeedsRebuild |
-|------|---------------------------|
-| `Ship()` | Silently drops (state != InSync and != Disconnected) |
-| `Barrier()` | Immediate `ErrReplicaDegraded` (default case in state switch) |
-| `MinRecoverableFlushedLSN` | Excluded (NeedsRebuild shippers skipped) |
-| `EvaluateRetentionBudgets` | Skipped (already escalated) |
-
-### Visibility
-
-| Surface | What it reports |
-|---------|----------------|
-| Heartbeat `ReplicaShipperStates` | `state: "needs_rebuild"` per-replica |
-| `WALShipper.State()` | `ReplicaNeedsRebuild` (5) |
-
-### Rebuild handoff
-
-| Step | What happens |
-|------|-------------|
-| Master detects `NeedsRebuild` in heartbeat | Sends Rebuilding assignment to replica VS |
-| Replica `HandleAssignment(RoleRebuilding)` | Starts rebuild from primary |
-| `StartRebuild` completes | 3-phase copy (full extent + WAL catch-up) |
-| Post-rebuild: `flushedLSN = checkpointLSN` | Not stale/zero — initialized from durable baseline |
-| Master sends fresh Primary assignment | `SetReplicaAddrs` → fresh shipper → bootstrap → InSync |
-
-### Abort
-
-| Condition | Result |
-|-----------|--------|
-| Epoch changes during rebuild | `RebuildServer` rejects with `EPOCH_MISMATCH` |
-| Rebuild copy fails | Error returned, role stays `RoleRebuilding` |
-
-## Baseline Closures
-
-| Test | Was | Now | What it proves |
-|------|-----|-----|----------------|
-| `TestAdversarial_NeedsRebuildBlocksAllPaths` | FAIL | PASS | NeedsRebuild blocks Ship (drops) + Barrier (rejects) + is sticky across retries |
-| `TestReconnect_GapBeyondRetainedWal_NeedsRebuild` | PASS* | PASS | Real reconnect handshake gap detection (R < S path), not budget trigger |
-
-## Proof Promotion
-
-### Primary proofs
-
-| Test | What it proves for CP13-7 |
-|------|--------------------------|
-| `TestAdversarial_NeedsRebuildBlocksAllPaths` | 5 assertions: NeedsRebuild state, Ship drops, Barrier rejects, state sticky after barrier, second SyncCache still fails |
-| `TestReconnect_GapBeyondRetainedWal_NeedsRebuild` | Real reconnect handshake detects R < S (gap beyond retained WAL) → SyncCache fails |
-| `TestHeartbeat_ReportsNeedsRebuild` | Heartbeat carries per-replica `needs_rebuild` state |
-| `TestRebuild_AbortOnEpochChange` | Epoch mismatch during rebuild → abort |
-| `TestRebuild_PostRebuild_FlushedLSN_IsCheckpoint` | Post-rebuild `flushedLSN = checkpointLSN` (not stale/zero) |
-
-### Support evidence
-
-| Test | What it supports |
-|------|-----------------|
-| `TestReplicaState_RebuildComplete_ReentersInSync` | Rebuild completion flow (reopen volume → RoleRebuilding → StartRebuild → fresh shipper → InSync). Support evidence: does not start from live NeedsRebuild shipper state, but proves the rebuild mechanics work end-to-end. |
-
-## Updated Baseline Summary
-
-| | PASS | FAIL | PASS* |
-|---|---|---|---|
-| CP13-1 (original) | 37 | 4 | 3 |
-| After CP13-2..CP13-7 | **43** | **0** | **1** |
-
-Remaining PASS*: `TestBug3_ReplicaAddr_MustBeIPPort_WildcardBind` (CP13-2 address witness — already upgraded to real proof in test but baseline doc still lists it as PASS*).
-
-## What CP13-7 Does NOT Close
-
- Real-workload validation (CP13-8)
- Broad rollout or performance claims
- Mode normalization (CP13-9)
@@ -1,75 +0,0 @@
-# CP13-8 Real-Workload Validation — Envelope + Contract
-
-Date: 2026-04-03
-
-## Workload Envelope
-
-| Parameter | Value |
-|-----------|-------|
-| Topology | RF=2, sync_all, cross-machine (m01 ↔ M02) |
-| Transport | iSCSI (primary frontend) |
-| Filesystem workload | ext4: 200 files, write + sync + failover + fsck + checksum verify |
-| Application workload | PostgreSQL pgbench TPC-B (scale=1, c=1, 10s) on promoted replica |
-| Disturbance | One bounded failover: kill primary, promote replica (epoch 1→2) |
-| **NOT included** | NVMe-TCP, RF>2, hours/days soak, degraded-mode perf, mode normalization |
-
-## Scenario
-
-`weed/storage/blockvol/testrunner/scenarios/internal/cp13-8-real-workload-validation.yaml`
-
-### Phase flow
-
-1. **Setup**: RF=2 sync_all pair (primary on M02, replica on m01), standalone `iscsi-target` binary
-2. **ext4 write**: iSCSI login → mkfs ext4 → write 200 files → md5sum → sync → umount → wait replication
-3. **Failover**: kill primary → promote replica to primary (epoch 2)
-4. **ext4 verify**: iSCSI login to promoted replica → fsck (filesystem integrity) → mount → file count == 200 → md5sum diff == MATCH
-5. **pgbench**: iSCSI login → pgbench_init (ext4, scale=1) → TPC-B run (c=1, 10s) → TPS reported
-6. **Cleanup**: always-run phase
-
-### Pass criteria
-
-| Proof | Assertion | What it validates |
-|-------|-----------|-------------------|
-| ext4 integrity | `fsck_ext4` passes | Replicated writes are filesystem-consistent after failover |
-| ext4 completeness | `file_count == 200` | No files lost during replication + failover |
-| ext4 correctness | `md5sum diff == MATCH` | File content identical to pre-failover (no corruption) |
-| pgbench durability | `pgbench_run` completes with TPS > 0 | Database transactions are durable on sync_all promoted replica |
-
-## Relation to CP13-1..7
-
-| Accepted checkpoint | What this workload validates |
-|---------------------|------------------------------|
-| CP13-2 (address truth) | Cross-machine iSCSI replication uses canonical addresses |
-| CP13-3 (durable progress) | ext4 data survives failover because barrier guarantees flushed durability |
-| CP13-4 (state eligibility) | Only InSync replica was eligible for barrier during replication |
-| CP13-5 (reconnect/catch-up) | Replication completed before failover (all writes reached replica) |
-| CP13-6 (retention) | WAL retained long enough for replication to complete |
-| CP13-7 (rebuild fallback) | Not directly exercised — failover is clean (no WAL gap). Support-only. |
-
-## Existing infrastructure reused
-
-| Existing scenario | Relation |
-|-------------------|----------|
-| `cp85-db-ext4-fsck.yaml` | CP13-8 extends this pattern with checksums, pgbench, and explicit envelope |
-| `benchmark-pgbench.yaml` | CP13-8 pgbench phase uses same `pgbench_init` + `pgbench_run` actions |
-
-## Run instructions
-
-```bash
-# From m01 (client node):
-sw-test-runner run cp13-8-real-workload-validation.yaml
-
-# Or from Windows dev machine (testrunner SSH):
-cd C:/work/seaweedfs
-go run ./weed/storage/blockvol/testrunner/cmd/sw-test-runner run \
-  weed/storage/blockvol/testrunner/scenarios/internal/cp13-8-real-workload-validation.yaml
-```
-
-## What CP13-8 Does NOT Close
-
- Mode normalization (CP13-9)
- Broad launch approval
- Performance floor (see Phase 12 P4)
- Degraded-mode validation
- NVMe-TCP transport validation
- Hours/days soak under sustained load
@@ -1,128 +0,0 @@
-# CP13-9 Mode Normalization Under V2 Constraints
-
-Date: 2026-04-03
-
-Status: accepted
-
-## Current Interpretation Rule
-
-Before an explicit `V2 core` exists as a real code structure and live
-event/command owner, current integrated tests are interpreted as:
-
-1. validation of current `V1` runtime behavior under `V2` constraints
-2. not proof that a completed `V2 runtime` already exists
-
-`CP13-9` keeps that rule explicit.
-It does not try to rewrite current constrained-runtime evidence into a claim that
-the pure `V2 core` has already landed.
-
-## Bounded Contract
-
-`CP13-9` accepts one bounded thing:
-
-1. explicit mode/publication normalization for the accepted chosen path
-
-Scope remains bounded to:
-
-1. `RF=2`
-2. `sync_all`
-3. current master / volume-server heartbeat path
-4. `blockvol` as execution backend
-
-It does not accept:
-
-1. `Phase 14` pure `V2 core` extraction
-2. broad launch approval
-3. broad transport/product expansion
-
-## Why This Checkpoint Exists
-
-`CP13-8` and `CP13-8A` now prove:
-
-1. one bounded real-workload package passes on the chosen path
-2. assignment/readiness/publication closure is explicit enough for that path
-
-What still needs freezing is the external mode meaning of the current path.
-
-In particular:
-
-1. a fresh volume before the first real replicated durability proof is not yet the
-   same as replicated-healthy
-2. `degraded` and `NeedsRebuild` are not interchangeable
-3. lookup / heartbeat / tester / debug surfaces should not silently use different
-   meanings of "healthy"
-
-## Recommended Mode Contract
-
-The semantic split below is the first-cut target.
-Exact mode names may change, but the distinctions should remain explicit.
-
-| Mode | Meaning | What it is allowed to claim |
-|------|---------|-----------------------------|
-| `allocated_only` | volume exists locally but runtime closure has not begun | existence only; not ready, not healthy |
-| `bootstrap_pending` | assignment exists and the pair may need the first real replicated write/connect proof | not replicated-healthy; may be publishable only under bounded non-healthy wording |
-| `replica_ready` | receiver / readiness closure exists on replica side | replica wiring is ready; not by itself proof of end-to-end healthy publication |
-| `publish_healthy` | chosen-path publication conditions are closed | allowed to surface healthy publication on bounded chosen path |
-| `degraded` | the bounded healthy path is not currently satisfied, but rebuild is not yet required | fail-closed for healthy replication claims |
-| `needs_rebuild` | unrecoverable gap or equivalent fail-closed state | explicitly not healthy; normal replication path blocked |
-
-## First-Write Bootstrap Rule
-
-`CP13-9` should freeze this rule explicitly:
-
-1. a freshly created `RF=2 sync_all` volume before the first real replicated write
-   or equivalent bounded durability proof must not be overclaimed as
-   replicated-healthy
-2. if the current runtime needs the first replicated write to establish the first
-   real sync/connect proof, that is a mode-policy fact that must be surfaced
-   explicitly rather than hidden inside ambiguous degraded/healthy output
-
-## Proof Shape
-
-`CP13-9` should close with a bounded proof package:
-
-| Proof | What it must show |
-|-------|-------------------|
-| Interpretation proof | current integrated evidence is described as constrained `V1` under `V2` constraints |
-| Bootstrap proof | fresh volume before first replicated write is surfaced as bootstrap-pending or equivalent bounded non-healthy mode |
-| Surface-consistency proof | lookup / heartbeat / tester / debug surfaces use one bounded mode meaning |
-| Fail-closed proof | `publish_healthy`, `degraded`, and `needs_rebuild` remain distinct and do not overclaim health |
-
-## Accepted Validation Summary
-
-Tester verdict: `ACCEPT`
-
-| Proof | Claim | Evidence |
-|------|-------|----------|
-| `AllocatedOnly` | `RF=1` maps to `allocated_only` | focused mode test |
-| `BootstrapPending` (`Replicas` empty) | `RF=2` before replica set closure maps to `bootstrap_pending` | focused mode test |
-| `BootstrapPending` (replica not ready) | `RF=2` with replica not ready maps to `bootstrap_pending` | focused mode test |
-| `PublishHealthy` | ready + not transport degraded maps to `publish_healthy` | focused mode test |
-| `Degraded` | transport degraded maps to `degraded` | focused mode test |
-| `NeedsRebuild` | rebuilding role maps to `needs_rebuild` | focused mode test |
-| `SurfaceConsistency` | mode / ready / degraded meaning stays aligned across transitions | focused transition checks |
-| `InterpretationRule` | current integrated tests are constrained `V1` under `V2` constraints | explicit wording in contract + design docs |
-| `NoOverclaim` | checkpoint does not claim pure `V2 core`, launch, or broad transport expansion | explicit boundedness wording |
-
-Minor note kept bounded:
-
-1. `assert_block_field` in the testrunner does not yet expose `volume_mode` as a first-class assert case
-2. this does not block checkpoint acceptance because the bounded unit and API-surface proofs are already direct
-
-## Relation to Earlier Checkpoints
-
-| Prior checkpoint | What CP13-9 reuses |
-|------------------|--------------------|
-| `CP13-1..7` | accepted replication contract and fail-closed semantics |
-| `CP13-8` | bounded real-workload pass on the chosen path |
-| `CP13-8A` | assignment/readiness/publication closure |
-
-`CP13-9` is therefore about policy/meaning on top of the corrected constrained
-runtime, not about redoing replication correctness or workload validation.
-
-## What CP13-9 Does NOT Close
-
- Pure `V2 core` extraction (`Phase 14`)
- Broad product launch approval
- Broad transport matrix claims
- Broad product-surface expansion beyond the chosen path
@@ -1,886 +0,0 @@
-# Phase 13
-
-Date: 2026-04-02
-Status: accepted
-Purpose: carry one explicit engineering gap beyond accepted `Phase 12` hardening into a bounded implementation phase so `RF=2 sync_all` becomes a correct, test-backed replicated durability mode under real reconnect, catch-up, retention, and rebuild conditions
-
-## Why This Phase Exists
-
-`Phase 09` accepted chosen-path execution closure.
-`Phase 10` accepted bounded control-plane closure.
-`Phase 11` accepted bounded product-surface rebinding.
-`Phase 12` accepted bounded hardening, diagnosability, and first-launch envelope evidence.
-
-What still remains is not broad protocol discovery.
-It is one concrete engineering problem:
-
-1. `sync_all` still needs a cleaner replicated-durability contract under cross-machine reconnect and replica recovery reality
-2. that contract must be expressed in code and tests so later feature work can reuse it rather than reopen replication semantics repeatedly
-
-## Phase Goal
-
-Turn `RF=2 sync_all` from a bounded chosen-path mode with accepted launch-hardening evidence into a correct, reusable replicated-durability model for reconnect, catch-up, retention, and rebuild on real workloads.
-
-Execution note:
-
-1. use `phase-13-log.md` as the technical pack for:
-   - checkpoint breakdown
-   - acceptance objects
-   - reject shapes
-   - assignment text for `sw` and `tester`
-2. prefer test-first baseline plus checkpointed implementation
-3. keep the goal narrow: replication correctness first, not broad optimization or new transport work
-
-## Scope
-
-### In scope
-
-1. canonical replica address truth
-2. authoritative per-replica durable-progress tracking
-3. reconnect handshake and WAL catch-up
-4. replica-aware WAL retention / truncation
-5. rebuild fallback when catch-up is impossible
-6. real ext4 / PostgreSQL validation on real block devices for cross-machine `sync_all`
-7. mode normalization work that depends directly on the corrected replication model
-
-### Out of scope
-
-1. broad new protocol discovery outside the replication path
-2. new transport projects such as `SPDK`, `io_uring`, or striped-layout redesign
-3. generic benchmark positioning beyond correctness-backed validation
-4. unrelated control-plane or product-surface expansion
-5. reopening accepted `Phase 09` / `Phase 10` / `Phase 11` / `Phase 12` semantics unless this phase exposes a real bug
-
-## Phase 13 Items
-
-### `CP13-1`: Test-First Baseline
-
-Goal:
-
- freeze a failing/passing baseline that exposes the current replication gaps before protocol work begins
-
-Acceptance object:
-
-1. the focused sync-replication gap tests exist
-2. they are run on current code before major implementation work
-3. the fail/pass split is captured explicitly so later checkpoint claims are grounded
-
-Status:
-
- accepted
-
-Carry-forward:
-
-1. the baseline report is frozen in `phase-13-cp1-baseline.md`
-2. no protocol code was changed in `CP13-1`
-3. `CP13-2` and later checkpoints must treat the baseline as the starting truth, not redefine it after implementation
-
-### `CP13-2`: Canonical Replica Addressing
-
-Goal:
-
- make replica endpoint truth canonical and routable so cross-machine replication never depends on wildcard listener strings, incomplete `:port` forms, or other non-authoritative address leakage
-
-Acceptance object:
-
-1. `CP13-2` accepts canonical replica address truth for the replication path
-2. it does not accept durable-progress truth, reconnect protocol, WAL retention, or rebuild fallback by implication
-3. it does not accept broad networking redesign beyond endpoint canonicalization
-
-Execution steps:
-
-1. Step 1: address truth contract freeze
-   - define the canonical replica endpoint form for replication surfaces as routable `host:port`
-   - define which forms are invalid for exported/registered truth:
-     - bare `:port`
-     - wildcard listener strings such as `[::]:port`
-     - accidental loopback when cross-machine routing is intended
-2. Step 2: implementation hardening
-   - canonicalize replica listener addresses at the source where receiver/registration surfaces expose them
-   - keep authoritative endpoint truth aligned across local listener state, registration/heartbeat publication, and any registry copies
-3. Step 3: proof package
-   - prove canonical `host:port` truth is emitted under wildcard-bind cases
-   - prove no wildcard or incomplete address string leaks into exported replication truth
-   - prove no-overclaim around reconnect, retention, or rebuild semantics
-
-Required scope:
-
-1. replica receiver endpoint truth
-2. registration / heartbeat / registry path carrying replica endpoints
-3. one focused wildcard-bind proof plus bounded cross-machine truth checks
-4. explicit distinction between address canonicalization and later reconnect protocol work
-
-Must prove:
-
-1. cross-machine replica addresses exported for replication are canonical routable `host:port`
-2. wildcard bind strings do not escape into replication truth
-3. local canonicalization does not silently rewrite intentionally loopback-only cases into incorrect external truth
-4. acceptance wording stays bounded to endpoint truth rather than later replication recovery semantics
-
-Reuse discipline:
-
-1. `weed/storage/blockvol/replica_receiver.go`, `replica_meta.go`, and nearby address helpers may be updated in place as the primary endpoint-truth surface
-2. `weed/server/master_block_registry.go` and heartbeat/registration paths may be updated in place only if needed to keep authoritative endpoint truth aligned
-3. focused unit/protocol tests should carry the main proof burden; component tests are support-only unless they prove an otherwise unreachable leak
-4. no checkpoint work may silently introduce reconnect protocol, retention policy, or rebuild logic
-
-Verification mechanism:
-
-1. one focused wildcard-bind canonicalization proof
-2. explicit checks that exported/registered replica endpoints are routable `host:port`
-3. no-overclaim review so `CP13-2` does not absorb `CP13-3+`
-
-Hard indicators:
-
-1. one accepted canonical-endpoint proof:
-   - wildcard-bind listener state resolves to canonical exported `host:port`
-2. one accepted no-leak proof:
-   - bare `:port` / wildcard listener strings no longer escape into replication truth
-3. one accepted boundedness proof:
-   - `CP13-2` claims endpoint truth only, not reconnect or durability semantics
-
-Reject if:
-
-1. the checkpoint fixes only one test string shape but leaves other exported endpoint paths unchanged
-2. canonicalization happens only in tests rather than at the production truth surface
-3. the checkpoint quietly broadens into reconnect, retention, or rebuild protocol work
-
-Status:
-
- accepted
-
-Carry-forward:
-
-1. `localServerID` remains stable control identity and may be opaque
-2. `advertisedHost` is now the transport-facing canonicalization input for wildcard-bind replica endpoints
-3. `CP13-3` and later checkpoints must not reopen identity-vs-transport separation unless a new concrete bug is exposed
-
-### `CP13-3`: Durable Progress Truth
-
-Goal:
-
- make durable replication progress explicit and authoritative so sync correctness is grounded in replica flushed durability rather than sender-side send progress or loosely inferred health
-
-Acceptance object:
-
-1. `CP13-3` accepts durable progress truth for the replication path
-2. it does not accept reconnect/catch-up protocol, retention policy, rebuild fallback, or broader state-machine closure by implication
-3. it does not accept generic “tests pass” reasoning without an explicit durable-progress contract review
-
-Execution steps:
-
-1. Step 1: durable-progress contract freeze
-   - define `replicaFlushedLSN` as replica-side WAL durability confirmed at barrier time
-   - define sender-side shipped/sent progress as diagnostic only, not authority for sync correctness
-   - define what barrier responses must expose as explicit durable progress truth
-2. Step 2: implementation hardening or proof confirmation
-   - update the durable-progress path only where current code fails to meet the contract
-   - if current code already satisfies the contract, keep changes minimal and make the proof package explicit instead of broadening scope
-3. Step 3: proof package
-   - prove barrier success is grounded in replica flushed durability
-   - prove flushed progress is monotonic within epoch and not updated on mere receive
-   - prove no-overclaim around `CP13-4+`
-
-Required scope:
-
-1. replica receiver durable-progress state
-2. barrier request/response path
-3. sender/group tracking of replica durable progress
-4. explicit separation between durable-progress truth and later reconnect / retention semantics
-
-Must prove:
-
-1. `replicaFlushedLSN` means replica durability, not sender transmission progress
-2. barrier responses expose durable progress explicitly enough for sync correctness decisions
-3. sender-side progress such as shipped/sent LSN is diagnostic only and cannot authorize sync success
-4. acceptance wording stays bounded to durable-progress truth rather than broader recovery/state-machine closure
-
-Reuse discipline:
-
-1. `weed/storage/blockvol/replica_apply.go`, `wal_shipper.go`, `dist_group_commit.go`, and related protocol message code may be updated in place as the primary durable-progress surfaces
-2. focused unit/protocol tests should carry the main proof burden
-3. `weed/server/*` should remain reference only unless durable-progress truth requires an exposed wiring change
-4. no checkpoint work may silently introduce reconnect protocol, retention policy, rebuild policy, or broader transport redesign
-
-Verification mechanism:
-
-1. one focused proof set around barrier/flushed progress truth
-2. explicit checks that receive progress alone does not advance durable authority
-3. no-overclaim review so `CP13-3` does not absorb `CP13-4+`
-
-Hard indicators:
-
-1. one accepted barrier-truth proof:
-   - barrier success is tied to replica flushed durability
-2. one accepted monotonicity proof:
-   - `replicaFlushedLSN` is monotonic within epoch
-3. one accepted no-false-authority proof:
-   - sender-side shipped/sent progress is diagnostic only
-4. one accepted boundedness proof:
-   - `CP13-3` claims durable-progress truth only
-
-Reject if:
-
-1. the checkpoint treats passing baseline tests as automatic closure without reviewing the durable-progress contract
-2. durable-progress truth is still mixed with sender-side transmission progress
-3. the checkpoint quietly broadens into reconnect, retention, rebuild, or general replication redesign
-
-Status:
-
- accepted
-
-Carry-forward:
-
-1. `replicaFlushedLSN` is now the authoritative durable-progress variable for `sync_all`
-2. legacy `BarrierOK` responses without `FlushedLSN` are rejected and cannot count as durable authority
-3. `CP13-4` and later checkpoints must treat sender-side send progress as diagnostic only, not as sync-correctness authority
-
-### `CP13-4`: Replica State Machine / Barrier Eligibility
-
-Goal:
-
- make replica state and barrier eligibility explicit so only `InSync` replicas can satisfy sync durability while non-eligible states fail closed instead of drifting into accidental success
-
-Acceptance object:
-
-1. `CP13-4` accepts the replica state machine and barrier-eligibility contract
-2. it does not accept reconnect/catch-up protocol, retention policy, rebuild fallback, or broader rollout claims by implication
-3. it does not accept vague “state seems fine” reasoning without an explicit eligibility contract
-
-Execution steps:
-
-1. Step 1: state contract freeze
-   - define the bounded state set used by the replication path:
-     - `Disconnected`
-     - `Connecting`
-     - `CatchingUp`
-     - `InSync`
-     - `Degraded`
-     - `NeedsRebuild`
-   - define barrier eligibility:
-     - only `InSync` replicas count toward sync durability
-     - non-eligible states must pre-reject or fail closed
-2. Step 2: implementation hardening or proof confirmation
-   - update the state/eligibility path only where current code fails the contract
-   - if current code already satisfies much of the contract, keep code changes minimal and make the proof package explicit
-3. Step 3: proof package
-   - prove barrier rejects replicas not eligible for sync durability
-   - prove degraded or catching-up replicas do not silently count toward `sync_all`
-   - prove no-overclaim around `CP13-5+`
-
-Required scope:
-
-1. replica shipper state transitions and eligibility checks
-2. barrier admission path
-3. `sync_all` failure semantics when replicas are non-eligible
-4. explicit separation between state eligibility and later reconnect/rebuild protocol work
-
-Must prove:
-
-1. only `InSync` replicas count toward sync durability
-2. `Disconnected`, `Connecting`, `CatchingUp`, `Degraded`, and `NeedsRebuild` do not silently satisfy barrier eligibility
-3. degraded/non-eligible replicas fail closed for `sync_all` rather than producing false durability success
-4. acceptance wording stays bounded to state/eligibility truth rather than reconnect, retention, or rebuild closure
-
-Reuse discipline:
-
-1. `weed/storage/blockvol/wal_shipper.go`, `dist_group_commit.go`, `shipper_group.go`, and nearby replication coordination code may be updated in place as the primary state/eligibility surfaces
-2. focused unit/protocol/adversarial tests should carry the main proof burden
-3. `weed/server/*` should remain reference only unless state eligibility requires a surfaced wiring correction
-4. no checkpoint work may silently introduce reconnect handshake, retention policy, rebuild flow, or broader transport redesign
-
-Verification mechanism:
-
-1. one focused proof set around replica state and barrier eligibility
-2. explicit checks that non-`InSync` states cannot satisfy `sync_all`
-3. no-overclaim review so `CP13-4` does not absorb `CP13-5+`
-
-Hard indicators:
-
-1. one accepted eligibility proof:
-   - only `InSync` replicas count toward sync durability
-2. one accepted fail-closed proof:
-   - non-eligible replicas cause bounded failure rather than false success
-3. one accepted state-boundary proof:
-   - barrier rejects or excludes disallowed states explicitly
-4. one accepted boundedness proof:
-   - `CP13-4` claims state/eligibility truth only
-
-Reject if:
-
-1. the checkpoint treats passing baseline tests as automatic closure without restating the state/eligibility contract
-2. non-eligible replica states can still satisfy sync durability
-3. the checkpoint quietly broadens into reconnect, retention, rebuild, or general replication redesign
-
-Status:
-
- accepted
-
-Carry-forward:
-
-1. the replica state set and barrier-eligibility contract are now explicit
-2. only `InSync` may satisfy sync durability; `Disconnected`/`Degraded` may invoke `Barrier()` only as bounded recovery entry paths
-3. `CP13-5` and later checkpoints must preserve this eligibility boundary rather than reopening it implicitly
-
-### `CP13-5`: Reconnect Handshake + WAL Catch-up
-
-Goal:
-
- make reconnect after replica disturbance explicit and correct so a replica with known durable progress can resume from retained WAL, catch up, and re-enter `InSync` without false bootstrap success or barrier hangs
-
-Acceptance object:
-
-1. `CP13-5` accepts the reconnect handshake and WAL catch-up contract for recoverable gaps on the replication path
-2. it does not accept replica-aware WAL retention policy, full rebuild fallback lifecycle, or broader rollout claims by implication
-3. it does not accept vague “reconnect seems to work” reasoning without an explicit resume/catch-up contract
-
-Execution steps:
-
-1. Step 1: reconnect contract freeze
-   - define when a replica must use bootstrap versus reconnect:
-     - fresh replica with no prior durable progress may bootstrap
-     - replica with prior flushed progress must reconnect via explicit resume truth
-   - define reconnect decision outcomes:
-     - already caught up
-     - recoverable gap within retained WAL
-     - unrecoverable gap that must fail closed and defer full rebuild handling to `CP13-7`
-2. Step 2: implementation hardening
-   - update the reconnect path only where current code still fails the resume/catch-up contract
-   - ensure catch-up replays retained WAL before barrier success is allowed
-   - ensure repeated disconnect/reconnect cycles remain bounded and do not silently fall back to unsafe bootstrap
-3. Step 3: proof package
-   - prove degraded replicas with prior durable progress use handshake/reconnect rather than bootstrap
-   - prove retained-WAL catch-up completes and re-enters `InSync` on recoverable gaps
-   - prove reconnect fails closed on unrecoverable or incomplete recovery cases
-   - prove no-overclaim around `CP13-6+`
-
-Required scope:
-
-1. `wal_shipper` reconnect discriminator and resume handshake
-2. retained-WAL catch-up replay path
-3. repeated disconnect/reconnect recovery behavior
-4. bounded failure semantics for gaps that cannot be recovered within this checkpoint
-5. explicit separation between reconnect/catch-up closure and later retention/rebuild policy work
-
-Must prove:
-
-1. fresh shippers bootstrap, but previously-synced shippers reconnect using resume truth
-2. barrier success after disturbance is allowed only after reconnect/catch-up has re-established `InSync`
-3. repeated disconnect/reconnect cycles do not strand the replica in false degraded recovery
-4. recoverable gaps replay retained WAL correctly without overwriting newer replica data
-5. acceptance wording stays bounded to reconnect/catch-up truth rather than retention or rebuild closure
-
-Reuse discipline:
-
-1. `weed/storage/blockvol/wal_shipper.go`, reconnect/catch-up helpers, and nearby replication protocol code may be updated in place as the primary reconnect surface
-2. focused protocol/adversarial tests should carry the main proof burden; component tests are support-only unless a protocol gap is otherwise unreachable
-3. `weed/server/*` should remain reference only unless reconnect correctness requires surfaced wiring changes
-4. no checkpoint work may silently broaden into retention policy, rebuild orchestration, or performance tuning
-
-Verification mechanism:
-
-1. one focused proof set around reconnect discriminator, catch-up replay, and post-reconnect barrier behavior
-2. explicit checks for repeated disconnect/reconnect recovery
-3. explicit checks that recoverable gaps replay retained WAL before sync success
-4. no-overclaim review so `CP13-5` does not absorb `CP13-6+`
-
-Hard indicators:
-
-1. one accepted reconnect-discriminator proof:
-   - prior durable progress uses handshake/reconnect rather than bootstrap
-2. one accepted catch-up proof:
-   - recoverable retained-WAL gap replays and returns to `InSync`
-3. one accepted repeated-recovery proof:
-   - multiple disconnect/reconnect cycles recover without hanging or drifting
-4. one accepted fail-closed proof:
-   - reconnect does not falsely succeed when recovery is incomplete or impossible within retained WAL
-5. one accepted boundedness proof:
-   - `CP13-5` claims reconnect/catch-up truth only
-
-Reject if:
-
-1. a previously-synced replica can still skip resume truth and succeed via unsafe bootstrap
-2. barrier success can occur before reconnect/catch-up has restored `InSync`
-3. repeated reconnect cycles still hang, strand, or silently degrade correctness
-4. the checkpoint quietly broadens into retention, explicit `NeedsRebuild` lifecycle closure, rebuild execution, or general replication redesign
-
-Status:
-
- accepted
-
-Carry-forward:
-
-1. replacement shippers now preserve prior durable-progress intent across `SetReplicaAddrs`
-2. previously-synced replicas must reconnect through resume truth and retained-WAL catch-up rather than unsafe bootstrap
-3. `CP13-6` and later checkpoints must preserve the reconnect/catch-up contract rather than weakening it through reclaim or rebuild shortcuts
-
-### `CP13-6`: Replica-Aware WAL Retention
-
-Goal:
-
- make WAL retention explicit and replica-aware so reclaim is gated by recoverable replica progress and bounded retention budgets rather than silently discarding catch-up-critical WAL
-
-Acceptance object:
-
-1. `CP13-6` accepts replica-aware WAL retention and retention-budget truth on the replication path
-2. it does not accept full rebuild fallback lifecycle, rebuild execution, or broader rollout claims by implication
-3. it does not accept vague “reclaim seems safe” reasoning without an explicit retention contract
-
-Execution steps:
-
-1. Step 1: retention contract freeze
-   - define which replica progress is authoritative for WAL retention:
-     - only replicas with prior durable progress and still recoverable state may hold WAL
-   - define bounded retention outcomes:
-     - reclaim blocked while a recoverable replica still needs retained WAL
-     - timeout / max-bytes budgets may escalate boundedly and release the WAL hold
-     - full rebuild handling after escalation remains `CP13-7`
-2. Step 2: implementation hardening
-   - update the retention path only where current code still fails the bounded retention contract
-   - ensure retention decisions use replica-aware progress rather than primary-local heuristics alone
-   - ensure budget-triggered escalation is explicit and fail-closed rather than silent reclaim
-3. Step 3: proof package
-   - prove recoverable replicas block reclaim of needed WAL
-   - prove timeout / max-bytes budgets trigger bounded escalation instead of indefinite WAL growth
-   - prove retention remains aligned with `CP13-5` reconnect/catch-up truth
-   - prove no-overclaim around `CP13-7+`
-
-Required scope:
-
-1. WAL retention/reclaim gates
-2. shipper-group retention inputs derived from recoverable replica progress
-3. bounded timeout / max-bytes escalation behavior
-4. explicit separation between retention truth and full rebuild lifecycle closure
-
-Must prove:
-
-1. reclaim does not drop WAL still required by a recoverable replica
-2. retention inputs come from replica-aware durable progress, not sender-side guesses
-3. timeout / max-bytes budgets trigger bounded escalation when WAL cannot be held indefinitely
-4. acceptance wording stays bounded to retention truth rather than full rebuild closure
-
-Reuse discipline:
-
-1. `weed/storage/blockvol` WAL-retention, flusher, shipper-group, and adjacent replication coordination code may be updated in place as the primary retention surface
-2. focused unit/protocol tests should carry the main proof burden; component tests are support-only unless a retention gap is otherwise unreachable
-3. `weed/server/*` should remain reference only unless retention truth requires surfaced reporting changes
-4. no checkpoint work may silently broaden into rebuild execution, broad control-plane redesign, or performance tuning
-
-Verification mechanism:
-
-1. one focused proof set around retention hold, reclaim gating, and budget-triggered escalation
-2. explicit checks that max-bytes and timeout paths are real production behaviors, not just comments/logs
-3. explicit checks that retention stays compatible with `CP13-5` recoverable catch-up
-4. no-overclaim review so `CP13-6` does not absorb `CP13-7+`
-
-Hard indicators:
-
-1. one accepted hold-back proof:
-   - recoverable replicas block reclaim of required WAL
-2. one accepted timeout-budget proof:
-   - timeout can escalate a stalled recoverable replica into bounded fail-closed behavior
-3. one accepted max-bytes-budget proof:
-   - max-bytes pressure triggers explicit bounded escalation rather than silent reclaim or TODO-only behavior
-4. one accepted boundedness proof:
-   - `CP13-6` claims retention truth only
-
-Reject if:
-
-1. reclaim can still silently discard WAL needed for a recoverable replica
-2. max-bytes behavior is still only log text / placeholder behavior without real state effect
-3. the checkpoint quietly broadens into full `NeedsRebuild` lifecycle closure, rebuild execution, or general replication redesign
-
-Status:
-
- accepted
-
-Carry-forward:
-
-1. retention inputs and bounded retention budgets are now replica-aware
-2. timeout and max-bytes escalation can move a stalled recoverable replica into `NeedsRebuild`
-3. `CP13-7` must turn that escalation into a real fail-closed rebuild lifecycle rather than leaving `NeedsRebuild` as a partially-signaled state
-
-### `CP13-7`: Rebuild Fallback
-
-Goal:
-
- make `NeedsRebuild` a real fail-closed recovery state so unrecoverable replicas stop participating in normal replication paths, surface rebuild intent clearly, and re-enter the replication contract only through bounded rebuild handoff
-
-Acceptance object:
-
-1. `CP13-7` accepts the `NeedsRebuild` fallback and bounded rebuild handoff lifecycle on the replication path
-2. it does not accept broad rollout claims or real-workload validation by implication
-3. it does not accept vague “rebuild eventually works” reasoning without an explicit fail-closed lifecycle contract
-
-Execution steps:
-
-1. Step 1: rebuild-fallback contract freeze
-   - define what `NeedsRebuild` means:
-     - unrecoverable via retained WAL catch-up
-     - excluded from normal ship/barrier success
-     - visible to rebuild orchestration and observability surfaces
-   - define lifecycle boundaries:
-     - detection/escalation into `NeedsRebuild`
-     - fail-closed behavior while in `NeedsRebuild`
-     - bounded rebuild handoff and post-rebuild re-entry
-2. Step 2: implementation hardening
-   - update the rebuild-fallback path only where current code still leaves `NeedsRebuild` partial, leaky, or inconsistent
-   - ensure ship/barrier paths block correctly while `NeedsRebuild`
-   - ensure successful rebuild resets progress/state in a way compatible with later re-entry
-3. Step 3: proof package
-   - prove unrecoverable gaps transition to `NeedsRebuild`
-   - prove `NeedsRebuild` blocks normal replication participation
-   - prove rebuild handoff can re-establish a bounded healthy starting point
-   - prove no-overclaim around `CP13-8+`
-
-Required scope:
-
-1. `NeedsRebuild` detection and state ownership on the primary shipper side
-2. fail-closed behavior for ship/barrier and related replication paths while `NeedsRebuild`
-3. rebuild start/abort/complete handoff boundaries
-4. post-rebuild progress/state initialization needed for safe re-entry
-5. explicit separation between rebuild fallback closure and later real-workload validation
-
-Must prove:
-
-1. unrecoverable gaps do not remain merely degraded; they transition to `NeedsRebuild`
-2. a shipper in `NeedsRebuild` cannot silently participate in ship/barrier success
-3. rebuild completion restores a bounded re-entry point without faking immediate `InSync`
-4. acceptance wording stays bounded to rebuild fallback truth rather than `CP13-8` rollout/workload claims
-
-Reuse discipline:
-
-1. `weed/storage/blockvol` rebuild, shipper-group, wal-shipper, and adjacent replication coordination code may be updated in place as the primary rebuild-fallback surface
-2. focused unit/protocol/adversarial tests should carry the main proof burden; component tests are support-only unless a rebuild gap is otherwise unreachable
-3. `weed/server/*` should remain reference only unless rebuild fallback requires surfaced status/reporting changes
-4. no checkpoint work may silently broaden into real-workload benchmarking, performance tuning, or new protocol discovery
-
-Verification mechanism:
-
-1. one focused proof set around `NeedsRebuild` transition, blocking semantics, and rebuild re-entry
-2. explicit checks that `NeedsRebuild` blocks normal replication paths rather than merely logging/marking degraded
-3. explicit checks that post-rebuild progress initializes from bounded truth such as checkpoint state
-4. no-overclaim review so `CP13-7` does not absorb `CP13-8+`
-
-Hard indicators:
-
-1. one accepted transition proof:
-   - unrecoverable retained-WAL gap transitions to `NeedsRebuild`
-2. one accepted fail-closed proof:
-   - `NeedsRebuild` blocks ship/barrier participation
-3. one accepted rebuild-handoff proof:
-   - rebuild start/complete path restores a bounded re-entry state
-4. one accepted post-rebuild-progress proof:
-   - replica progress after rebuild is initialized from checkpoint truth, not stale/zeroed state
-5. one accepted boundedness proof:
-   - `CP13-7` claims rebuild fallback only
-
-Reject if:
-
-1. an unrecoverable gap can still linger in `Degraded` without escalating to `NeedsRebuild`
-2. a `NeedsRebuild` shipper can still satisfy normal ship/barrier paths
-3. rebuild completion jumps directly to misleading healthy semantics without bounded re-entry proof
-4. the checkpoint quietly broadens into `CP13-8` real-workload validation or general replication redesign
-
-Status:
-
- accepted
-
-Carry-forward:
-
-1. `NeedsRebuild` is now a real fail-closed fallback state
-2. rebuild handoff and post-rebuild progress are bounded by checkpoint truth rather than implicit recovery assumptions
-3. `CP13-8` must validate the accepted replication contract on named real workloads without reopening protocol semantics or quietly broadening into mode policy work
-
-### `CP13-8`: Real-Workload Validation
-
-Goal:
-
- validate the accepted `RF=2 sync_all` replication contract on one bounded set of real workloads so the engineering proof is no longer only protocol/unit-level but also demonstrated on named real block-device consumers
-
-Acceptance object:
-
-1. `CP13-8` accepts one bounded real-workload validation package for the accepted `RF=2 sync_all` path
-2. it does not accept broad rollout claims, broad benchmark positioning, or mode normalization by implication
-3. it does not accept vague “worked in a manual run” reasoning without named workloads, bounded envelope, and replayable evidence
-
-Execution steps:
-
-1. Step 1: workload envelope freeze
-   - name one bounded validation matrix:
-     - workload(s)
-     - topology
-     - transport/frontend
-     - filesystem/application surface
-     - disturbance shapes included and excluded
-   - recommended first-cut surfaces:
-     - real filesystem behavior such as `ext4`
-     - one database/application surface such as `PostgreSQL`
-2. Step 2: harness and evidence hardening
-   - wire the workload run through real block-device consumers on the accepted path
-   - keep the environment reproducible and bounded enough that failures are attributable
-   - collect evidence at the same semantic layer as accepted prior checkpoints
-3. Step 3: proof package
-   - prove the named real workloads complete correctly on the accepted path
-   - prove disturbance/failover behavior is bounded inside the named envelope if included
-   - prove no-overclaim around `CP13-9+`
-
-Required scope:
-
-1. one bounded workload matrix on the accepted `RF=2 sync_all` path
-2. real block-device consumer validation (not only protocol/unit tests)
-3. bounded disturbance cases only if explicitly named in the envelope
-4. explicit separation between real-workload proof and later mode normalization / rollout claims
-
-Must prove:
-
-1. the accepted replication contract survives contact with named real workloads
-2. evidence is tied to a bounded environment and workload envelope, not generic “production ready” rhetoric
-3. failures, if any, are attributable to explicit workload-envelope gaps rather than ambiguous harness drift
-4. acceptance wording stays bounded to real-workload validation rather than `CP13-9` policy/mode closure
-
-Reuse discipline:
-
-1. prefer existing `testrunner`, bounded component scenarios, and real-device harnesses where possible
-2. update `weed/storage/blockvol/*` only when the real workload exposes a concrete bug in accepted semantics
-3. `weed/server/*` should remain reference only unless workload validation exposes a surfaced control/runtime issue
-4. no checkpoint work may silently broaden into generic benchmark marketing, launch approval, or mode policy redesign
-
-Verification mechanism:
-
-1. one named workload matrix with explicit environment description
-2. replayable runs or artifacts for the chosen workload package
-3. explicit pass/fail conditions tied back to accepted `CP13-1..7` semantics
-4. no-overclaim review so `CP13-8` does not absorb `CP13-9+`
-
-Hard indicators:
-
-1. one accepted filesystem proof:
-   - a named real filesystem workload completes correctly on the accepted path
-2. one accepted application proof:
-   - a named real application/database workload completes correctly on the accepted path
-3. one accepted envelope proof:
-   - the validation matrix is explicit about topology, frontend, workload, and exclusions
-4. one accepted boundedness proof:
-   - `CP13-8` claims real-workload validation only
-
-Reject if:
-
-1. the checkpoint relies on ad hoc manual runs with no bounded envelope
-2. a claimed real-workload proof is actually only a synthetic benchmark or unit test
-3. delivery wording quietly broadens into mode normalization, launch approval, or general production-readiness claims
-
-Status:
-
- accepted
-
-Carry-forward:
-
-1. one bounded real-workload package now passes on the chosen path:
-   - `RF=2`
-   - `sync_all`
-   - iSCSI
-   - `ext4 + pgbench`
-   - one failover
-2. this checkpoint validates current runtime behavior under accepted `V2` constraints
-3. it does not by itself mean a pure `V2 runtime` already exists
-4. `CP13-8A` and `CP13-9` must keep that interpretation explicit
-
-### `CP13-8A`: Assignment-to-Publication Closure
-
-Goal:
-
- close the control/runtime/publication contradiction exposed by `CP13-8` so the system no longer treats allocation or assignment presence as equivalent to replica publication readiness
-
-Acceptance object:
-
-1. `CP13-8A` accepts one bounded closure slice for assignment-to-publication truth on the accepted `RF=2 sync_all` path
-2. it does not accept broad mode normalization, launch approval, or backend replacement by implication
-3. it does not accept sleep-based or timing-based fixes that leave readiness semantics implicit
-
-Execution steps:
-
-1. Step 1: unify assignment lifecycle
-   - ensure assignment delivery flows through one authoritative path from role apply to receiver/shipper wiring to readiness bookkeeping
-   - remove semantic split between store-only role application and service-level replication/publication setup
-2. Step 2: name readiness and publication truth
-   - define explicit readiness states for the chosen path
-   - ensure heartbeat / lookup / tester surfaces distinguish:
-     - allocated
-     - role applied
-     - receiver ready
-     - publish healthy
-3. Step 3: bounded rerun
-   - rerun the bounded `CP13-8` workload package after closure lands
-   - determine whether the remaining contradiction is backend data visibility, adapter timing/publication, or a true core-rule gap
-
-Required scope:
-
-1. assignment-to-publication closure only
-2. chosen path only: `RF=2 sync_all`
-3. existing master / volume-server heartbeat path only
-4. `blockvol` remains the execution backend
-
-Must prove:
-
-1. assignment delivered does not by itself imply receiver ready or publish healthy
-2. replica publication requires explicit readiness closure rather than allocation completion or precomputed port presence
-3. master lookup / REST / tester health checks consume the same bounded readiness truth
-4. `CP13-8A` remains about closure, not mode normalization or backend redesign
-
-Reuse discipline:
-
-1. prefer `weed/server/*` and bridge-layer updates first because this is a surfaced control/runtime issue
-2. update `weed/storage/blockvol/*` only if closure work exposes a concrete backend bug rather than a publication-path contradiction
-3. keep `CP13-1..7` semantics fixed unless the closure work exposes a live contradiction
-4. no checkpoint work may silently broaden into `CP13-9` mode policy or broad rollout claims
-
-Verification mechanism:
-
-1. one focused proof set around assignment lifecycle closure and readiness/publication gating
-2. explicit tests that heartbeat / lookup / tester surfaces do not publish a replica before readiness closes
-3. bounded `CP13-8` rerun or equivalent evidence showing the contradiction moves from mixed-state ambiguity to an attributable remaining cause
-4. no-overclaim review so `CP13-8A` does not absorb `CP13-9`
-
-Hard indicators:
-
-1. one accepted lifecycle proof:
-   - assignment processing uses one authoritative path from role apply through runtime wiring
-2. one accepted readiness proof:
-   - replica-ready is explicit and not inferred from mere existence/allocation
-3. one accepted publication proof:
-   - lookup / heartbeat / tester gates do not publish a replica before readiness closure
-4. one accepted boundedness proof:
-   - `CP13-8A` claims closure only and leaves broader mode policy untouched
-
-Reject if:
-
-1. assignment still reaches different semantic outcomes depending on whether it flows through heartbeat/store-only or service-level processing
-2. a replica can still be surfaced as healthy/ready before receiver/session readiness closes
-3. the slice relies on delays or ad hoc retries rather than explicit readiness semantics
-4. delivery wording broadens into `CP13-9` mode normalization, launch approval, or generic backend replacement
-
-Status:
-
- accepted
-
-Carry-forward:
-
-1. assignment/readiness/publication closure is now explicit enough for the bounded chosen path
-2. the corrected path no longer treats replica allocation or assignment presence as equivalent to replica publication readiness
-3. the remaining next step is mode-policy normalization on top of this closed assignment/publication path
-
-### `CP13-9`: Mode Normalization Under `V2` Constraints
-
-Goal:
-
- freeze one bounded mode-policy contract for the current chosen path so external health/publication meaning no longer drifts between implicit `V1` runtime behavior and `V2` constraint language
-
-Acceptance object:
-
-1. `CP13-9` accepts one bounded mode-normalization package for the accepted `RF=2 sync_all` path
-2. it accepts mode/publication semantics for the current runtime only under explicit `V2` constraints
-3. it does not accept pure `V2 core` extraction, launch approval, or broad transport/product expansion by implication
-
-Execution steps:
-
-1. Step 1: interpretation rule freeze
-   - make explicit that current integrated tests are evaluating `V1` runtime behavior under `V2` constraints
-   - define `CP13-9` as policy/meaning closure for the constrained current path, not proof that a completed `V2 runtime` already exists
-2. Step 2: mode contract freeze
-   - define one bounded external mode set for the chosen path
-   - at minimum distinguish:
-     - allocated / assigned
-     - bootstrap-pending
-     - replica-ready
-     - publish-healthy
-     - degraded
-     - `NeedsRebuild`
-   - define what each surface is allowed to claim for each mode:
-     - heartbeat
-     - lookup / REST / tester surfaces
-     - operator/debug surfaces
-3. Step 3: bootstrap-policy closure
-   - make the first-write / first-connect bootstrap behavior explicit
-   - ensure a freshly created `RF=2 sync_all` volume is not overclaimed as replicated-healthy before the first real replicated durability proof exists
-4. Step 4: proof package
-   - prove all relevant surfaces agree on the bounded mode meanings
-   - prove no-overclaim around future pure-core extraction or broad launch claims
-
-Required scope:
-
-1. chosen path only: `RF=2 sync_all`
-2. current master / volume-server heartbeat path only
-3. `blockvol` remains the execution backend
-4. current integrated runtime is interpreted as constrained `V1`, not yet as a completed `V2 runtime`
-
-Must prove:
-
-1. health/publication meaning is explicit and consistent across product/tester/operator surfaces
-2. `bootstrap-pending` or equivalent first-write state is explicit rather than hidden inside ambiguous degraded/healthy output
-3. publish/ready semantics remain fail-closed under the accepted replication contract
-4. acceptance wording stays bounded to mode normalization for the constrained current path rather than `V2 core` extraction
-
-Reuse discipline:
-
-1. prefer surfaced policy/diagnostic/projection work first because this checkpoint is about external mode meaning
-2. update `weed/storage/blockvol/*` only if mode normalization exposes a concrete backend leak rather than a surface-meaning gap
-3. keep `CP13-1..8A` semantics fixed unless a live contradiction is exposed
-4. no checkpoint work may silently broaden into `Phase 14` pure-core extraction or broad rollout claims
-
-Verification mechanism:
-
-1. one focused proof set around mode/publication semantics across heartbeat / lookup / tester / debug surfaces
-2. explicit tests or bounded evidence that a fresh volume before first replicated write is not overpublished as replicated-healthy
-3. explicit checks that degraded / rebuild-required surfaces remain distinguishable and bounded
-4. no-overclaim review so `CP13-9` does not absorb `Phase 14`
-
-Hard indicators:
-
-1. one accepted interpretation proof:
-   - current integrated evidence is explicitly described as constrained `V1` under `V2` constraints
-2. one accepted bootstrap proof:
-   - a fresh `RF=2 sync_all` volume before first replicated write is surfaced as bootstrap-pending or equivalent bounded non-healthy mode
-3. one accepted surface-consistency proof:
-   - heartbeat / lookup / tester / debug surfaces agree on the same bounded mode meanings
-4. one accepted boundedness proof:
-   - `CP13-9` claims mode normalization only and leaves pure-core extraction to later phases
-
-Reject if:
-
-1. the slice still uses one meaning of “healthy” for lookup and a different one for tester/debug/operator surfaces
-2. a fresh volume can still appear fully replicated-healthy before first real replicated durability proof exists
-3. the checkpoint quietly claims a completed `V2 runtime` already exists
-4. delivery wording broadens into launch approval, broad productization, or `Phase 14` pure-core extraction
-
-Status:
-
- accepted
-
-Carry-forward:
-
-1. one bounded mode set is now explicit for the current constrained chosen path:
-   - `allocated_only`
-   - `bootstrap_pending`
-   - `publish_healthy`
-   - `degraded`
-   - `needs_rebuild`
-2. current integrated tests remain explicitly interpreted as constrained `V1` under `V2` constraints
-3. `CP13-9` does not claim pure `V2 core` extraction, launch approval, or broad transport expansion
-
-## Reuse Discipline
-
-1. `weed/storage/blockvol/*` is the primary implementation surface and may be updated in place
-2. focused unit/component/adversarial tests should carry the main proof burden
-3. real-node / real-device validation belongs in testrunner or bounded component scenarios, not chat prose
-4. `weed/server/*` may be updated only when replication correctness requires registry / assignment / heartbeat truth to change
-5. no checkpoint may silently broaden into performance-optimization or broad rollout work
-
-## Expected Outcome
-
-`Phase 13` now succeeds with the following closure:
-
-1. reconnect / catch-up / rebuild semantics become explicit and test-backed
-2. `sync_all` correctness no longer depends on partial or implicit sender-state assumptions
-3. later feature work can reuse a clearer replication contract instead of re-deriving durability semantics each time
-4. one bounded real-workload package and one bounded mode-normalization package are both accepted on the current constrained path
@@ -1,709 +0,0 @@
-Purpose: append-only technical pack and delivery log for `Phase 14` V2 core
-extraction.
-
---
-
-### `14A` Technical Pack
-
-Date: 2026-04-03
-Goal: freeze the first explicit `V2 core` shell inside
-`sw-block/engine/replication` so current accepted semantic constraints become
-executable state/event/command/projection ownership, not only design wording
-
-#### Layer 1: Semantic Core
-
-##### Problem statement
-
-`Phase 13` accepted:
-
-1. bounded replication correctness
-2. bounded assignment/publication closure
-3. bounded mode normalization
-
-But those results are still interpreted mainly as:
-
-1. constrained-`V1` runtime behavior under `V2` rules
-
-`14A` accepts one narrower thing:
-
-1. the first real `V2 core` semantic shell exists as code in
-   `sw-block/engine/replication`
-
-It does not accept:
-
-1. live runtime cutover
-2. adapter rebinding
-3. product-surface migration
-4. launch or performance claims
-
-##### State / contract
-
-`14A` must make these truths explicit in code:
-
-1. one bounded `VolumeState` owns normalized mode, readiness, boundary, and
-   desired replica truth
-2. one bounded event set expresses assignment, readiness observation, durable
-   boundary change, and rebuild escalation
-3. one bounded command set expresses semantic decisions without runtime side
-   effects
-4. one bounded projection expresses outward publication meaning from the same
-   state owner
-5. the current interpretation remains:
-   - explicit `V2 core` shell exists
-   - integrated runtime authority is still `constrained_v1` until later phases
-
-##### Must preserve
-
-1. stable `ReplicaID` ownership
-2. durable boundary truth is not inferred from diagnostic shipped progress
-3. `publish_healthy` requires named readiness plus durable boundary closure
-4. `degraded` and `needs_rebuild` remain distinct fail-closed modes
-5. the code does not overclaim live `V2` runtime ownership
-
-##### Reject shapes
-
-Reject `14A` if:
-
-1. the new core shell is only a naming wrapper with no deterministic state
-   update path
-2. `publish_healthy` can be reached from assignment or transport convenience
-   without durable boundary truth
-3. diagnostic sender progress is allowed to establish durable authority
-4. `degraded` and `needs_rebuild` collapse into one ambiguous unhealthy bucket
-5. the delivery wording implies live path cutover
-
-#### Layer 2: Execution Core
-
-##### Files in scope
-
-Primary files:
-
-1. `sw-block/engine/replication/state.go`
-2. `sw-block/engine/replication/event.go`
-3. `sw-block/engine/replication/command.go`
-4. `sw-block/engine/replication/projection.go`
-5. `sw-block/engine/replication/engine.go`
-6. `sw-block/engine/replication/phase14_core_test.go`
-7. `sw-block/engine/replication/doc.go`
-
-Existing substrate kept in place:
-
-1. `sw-block/engine/replication/registry.go`
-2. `sw-block/engine/replication/sender.go`
-3. `sw-block/engine/replication/session.go`
-4. `sw-block/engine/replication/orchestrator.go`
-5. nearby ownership/recovery tests
-
-##### Execution order
-
-`14A` follows the `Phase 14+` framework strictly:
-
-1. explicit state
-2. explicit events
-3. explicit commands
-4. explicit projection
-5. deterministic engine loop
-6. bounded structural tests
-
-##### Acceptance basis
-
-Keep the proof set small and structural:
-
-1. identity / ownership
-   - stable `ReplicaID`
-   - endpoint change invalidates active ownership session
-2. state eligibility
-   - only eligible primary path can reach `publish_healthy`
-3. durable boundary
-   - barrier durability updates authority
-   - diagnostic shipped progress stays diagnostic
-4. fail-closed modes
-   - `degraded` and `needs_rebuild` stay distinct and non-healthy
-5. interpretation rule
-   - the shell begins `V2 core`
-   - it does not yet claim live runtime authority
-
-##### Delivery posture
-
-This phase uses the larger-slice execution model:
-
-1. main developer owns semantic design and implementation
-2. `sw` is used only for bounded support tasks if needed
-3. `tester` validates the structural acceptance basis
-4. `manager` challenges semantic adequacy and overclaim control
-
-##### Review gate
-
-Every `14A` code change or acceptance note should answer:
-
-1. semantic constraint satisfied
-2. overclaim avoided
-3. accepted proof preserved
-
-#### Starting point inventory
-
-Current explicit shell already present in repo:
-
-1. `state.go`
-   - `RuntimeAuthority`
-   - `VolumeRole`
-   - `ModeName`
-   - `ReadinessView`
-   - `BoundaryView`
-   - `ModeView`
-   - `VolumeState`
-2. `event.go`
-   - assignment
-   - readiness observation
-   - barrier accepted / rejected
-   - checkpoint advance
-   - rebuild observation / commit
-3. `command.go`
-   - `ApplyRoleCommand`
-   - `StartReceiverCommand`
-   - `ConfigureShipperCommand`
-   - `InvalidateSessionCommand`
-   - `PublishProjectionCommand`
-4. `projection.go`
-   - `PublicationProjection`
-5. `engine.go`
-   - deterministic `ApplyEvent()`
-   - recompute mode/readiness/publication
-   - emit bounded commands and projection
-6. `phase14_core_test.go`
-   - structural acceptance basis for the shell
-
-#### Immediate development target
-
-The next development target under `14A` is not to broaden the shell.
-
-It is to make the shell the clear semantic owner for the first complete chain:
-
-1. `mode`
-2. `readiness`
-3. `publication`
-
-and verify the package stays internally coherent before `14B` begins.
-
-#### Verification status
-
-Current package verification on 2026-04-03:
-
-1. `go test ./...` in `sw-block/engine/replication`
-2. result: `PASS`
-3. interpretation:
-   - the current explicit shell is a valid starting point for `Phase 14`
-   - this verifies bounded internal coherence only
-   - this does not claim live runtime cutover
-
---
-
-### `14A` Delivery Note Rev 1
-
-Date: 2026-04-03
-Scope: strengthen the first `mode -> readiness -> publication` chain inside the
-explicit `V2 core` shell without adding any live adapter hook
-
-What changed:
-
-1. publication is now explicit core-owned state, not only an implicit boolean
-   threaded through readiness/projection
-2. the engine now emits normalized publication-gate reasons for bootstrap and
-   non-primary states
-3. `RF=1 / no replicas -> allocated_only` is now frozen directly in the core
-   shell, aligning the code with accepted `CP13-9` semantics
-
-Files changed:
-
-1. `sw-block/engine/replication/state.go`
-   - added `PublicationView`
-   - `VolumeState` now owns publication truth explicitly
-2. `sw-block/engine/replication/projection.go`
-   - `PublicationProjection` now carries explicit publication state
-3. `sw-block/engine/replication/engine.go`
-   - split publication recompute away from raw readiness bits
-   - added explicit gate reasons:
-     - `awaiting_role_apply`
-     - `awaiting_shipper_configured`
-     - `awaiting_shipper_connected`
-     - `awaiting_barrier_durability`
-     - `replica_not_primary`
-     - `allocated_only`
-   - enforced `no replicas => allocated_only`
-4. `sw-block/engine/replication/phase14_core_test.go`
-   - strengthened the primary publication chain proof with gate-reason checks
-   - strengthened replica-ready proof with non-primary publication reason
-   - added direct `allocated_only` proof for no-replica path
-
-Proofs added or strengthened:
-
-1. primary publication closure proof
-   - assignment -> role applied -> shipper configured -> shipper connected ->
-     barrier durability now produces the expected gate reason at each stage
-2. replica-ready is not publication proof
-   - `replica_ready` stays non-healthy with explicit reason
-     `replica_not_primary`
-3. `CP13-9` allocated-only proof
-   - a primary assignment with no replicas remains `allocated_only`, not
-     `bootstrap_pending`
-
-Validation:
-
-1. `gofmt -w state.go projection.go engine.go phase14_core_test.go`
-2. `go test ./...`
-3. result: `PASS`
-
-Constraint / overclaim / proof review:
-
-1. semantic constraint satisfied
-   - `CP13-8A`: assignment/readiness/publication closure must be explicit
-   - `CP13-9`: `allocated_only`, `bootstrap_pending`, `replica_ready`,
-     `publish_healthy`, `degraded`, and `needs_rebuild` must stay bounded and
-     non-overlapping
-2. overclaim avoided
-   - publication health can no longer be inferred from assignment presence,
-     shipper connection alone, or replica readiness
-   - RF=1/no-replica path no longer overclaims `bootstrap_pending`
-3. proof preserved
-   - barrier durability remains the authority for `publish_healthy`
-   - diagnostic shipped progress remains non-authoritative
-   - constrained-`V1` runtime interpretation remains explicit
-
---
-
-### `14B` Delivery Note Rev 1
-
-Date: 2026-04-03
-Scope: freeze first bounded command-emission rules so the explicit `V2 core`
-decides commands from semantic gaps, not from repeated event convenience
-
-What changed:
-
-1. repeated assignments no longer blindly reset semantic state and re-emit the
-   same commands
-2. command emission is now gap-driven:
-   - apply role only when epoch/role command state is stale
-   - start receiver only when replica path still needs receiver start for the
-     current epoch
-   - configure shipper only when primary path still needs current replica
-     configuration
-   - invalidate session only on a new failure transition, not every repeated
-     degraded event
-3. assignment changes still re-emit the needed command when semantic intent
-   really changes
-
-Files changed:
-
-1. `sw-block/engine/replication/state.go`
-   - added private command-state tracking to `VolumeState`
-2. `sw-block/engine/replication/engine.go`
-   - extracted assignment handling into gap-driven command logic
-   - preserved readiness when the assignment is repeated without semantic change
-   - reset only the relevant readiness edges when role/epoch/replica-set changes
-   - deduplicated repeated invalidation commands for the same failure reason
-3. `sw-block/engine/replication/phase14_command_test.go`
-   - added exact command-sequence proofs
-
-Proofs added:
-
-1. primary repeated-assignment boundedness
-   - first assignment emits:
-     - `apply_role`
-     - `configure_shipper`
-     - `publish_projection`
-   - repeated identical assignment emits only:
-     - `publish_projection`
-2. replica repeated-assignment boundedness
-   - first replica assignment emits:
-     - `apply_role`
-     - `start_receiver`
-     - `publish_projection`
-   - repeated identical assignment emits only:
-     - `publish_projection`
-3. assignment-change selective reissue
-   - changed replica endpoint on primary path reissues only
-     `configure_shipper`, not the whole initial command bundle
-4. repeated-failure boundedness
-   - first `BarrierRejected(timeout)` emits `invalidate_session`
-   - repeated `BarrierRejected(timeout)` does not emit duplicate invalidation
-
-Validation:
-
-1. `gofmt -w state.go engine.go phase14_command_test.go`
-2. `go test ./...`
-3. result: `PASS`
-
-Constraint / overclaim / proof review:
-
-1. semantic constraint satisfied
-   - `Phase 14B`: command emission must come from semantic state, not runtime
-     convenience
-   - `CP13-8A`: assignment/readiness/publication closure must stay explicit
-   - `CP13-9`: bounded mode meaning must not be destabilized by repeated command
-     churn
-2. overclaim avoided
-   - repeated assignment no longer acts like proof that role apply / receiver
-     start / shipper configure still need to happen
-   - repeated failure does not create unbounded invalidation spam that looks like
-     fresh semantic transitions
-3. proof preserved
-   - `14A` publication-gate proofs still hold
-   - barrier durability is still the only path to `publish_healthy`
-   - constrained-`V1` interpretation is still explicit, not broadened
-
---
-
-### `14B` Delivery Note Rev 2
-
-Date: 2026-04-03
-Scope: tighten `publish_projection` so it is also emitted from semantic change,
-not from raw event frequency
-
-What changed:
-
-1. `PublishProjectionCommand` is now emitted only when the outward projection
-   actually changes
-2. repeated identical events on an already-converged state now become true
-   no-op command sequences
-
-Files changed:
-
-1. `sw-block/engine/replication/engine.go`
-   - compare previous and new projection
-   - emit `publish_projection` only on real outward change
-2. `sw-block/engine/replication/phase14_command_test.go`
-   - repeated identical primary assignment now expects no commands
-   - repeated identical replica assignment now expects no commands
-   - repeated identical failure now expects no commands after the first
-     invalidation
-   - added direct proof that repeated unchanged projection events emit no
-     `publish_projection`
-
-Proofs strengthened:
-
-1. repeated identical assignment is now a true no-op command sequence
-2. repeated identical failure is now a true no-op command sequence
-3. publish emission is now tied to projection change, not event arrival
-
-Validation:
-
-1. `gofmt -w engine.go phase14_command_test.go`
-2. `go test ./...`
-3. result: `PASS`
-
-Constraint / overclaim / proof review:
-
-1. semantic constraint satisfied
-   - `14B`: command emission is further frozen to semantic deltas only
-2. overclaim avoided
-   - repeated identical events no longer look like fresh publication work
-   - projection emission no longer overstates outward change when nothing changed
-3. proof preserved
-   - all `14A` and `14B` proofs still pass
-   - publication remains bounded by the same explicit state owner
-
---
-
-### `14C` Delivery Note Rev 1
-
-Date: 2026-04-03
-Scope: make the first bounded boundary/recovery truths explicit in the core
-shell so recovery-in-progress and rebuild closure affect mode/publication
-semantics directly
-
-What changed:
-
-1. `BoundaryView` now carries more explicit boundary truth:
-   - `CommittedLSN`
-   - `TargetLSN`
-   - `AchievedLSN`
-   - plus the previously separated durable/checkpoint/diagnostic fields
-2. `RecoveryView` is now an explicit core-owned state with bounded phases:
-   - `idle`
-   - `catching_up`
-   - `needs_rebuild`
-   - `rebuilding`
-3. the event vocabulary now includes:
-   - `CommittedLSNAdvanced`
-   - `CatchUpPlanned`
-   - `RecoveryProgressObserved`
-   - `RebuildStarted`
-   - extended `RebuildCommitted` with explicit achieved boundary support
-4. recovery-in-progress now blocks `replica_ready` / publication overclaim
-   through mode recompute:
-   - active catch-up or rebuild forces `bootstrap_pending`
-     with reason `recovery_in_progress`
-   - rebuild-required stays `needs_rebuild`
-
-Files changed:
-
-1. `sw-block/engine/replication/state.go`
-   - added explicit `RecoveryView`
-   - expanded `BoundaryView`
-2. `sw-block/engine/replication/event.go`
-   - added boundary/recovery events
-3. `sw-block/engine/replication/engine.go`
-   - boundary truth is now updated explicitly and monotonically
-   - recovery state now participates directly in mode/publication recompute
-   - assignment changes clear stale recovery target/achieved truth
-4. `sw-block/engine/replication/phase14_boundary_test.go`
-   - added structural boundary/recovery proofs
-
-Proofs added:
-
-1. boundary-truth separation
-   - `CommittedLSN`, `CheckpointLSN`, `DurableLSN`, and diagnostic shipped
-     progress remain distinct truths
-2. catch-up blocks ready overclaim
-   - a replica with role applied + receiver ready still falls back to
-     `bootstrap_pending` with reason `recovery_in_progress` while catch-up is
-     active
-3. rebuild boundary closure
-   - `needs_rebuild` -> `rebuilding` -> `idle` is explicit in recovery truth
-   - rebuild commit aligns achieved/durable/checkpoint boundaries
-   - rebuild completion on replica returns to `replica_ready`, not
-     `publish_healthy`
-
-Validation:
-
-1. `gofmt -w state.go event.go engine.go phase14_boundary_test.go`
-2. `go test ./...`
-3. result: `PASS`
-
-Constraint / overclaim / proof review:
-
-1. semantic constraint satisfied
-   - `CP13-3`: durable truth remains distinct from diagnostic sender progress
-   - `CP13-7`: rebuild is explicit fail-closed truth, not an ambiguous degraded
-     tail
-   - `T14`: engine owns recovery policy and meaning, not backend convenience
-2. overclaim avoided
-   - receiver-ready during catch-up no longer looks like final ready state
-   - rebuild-in-progress no longer risks being interpreted as ordinary
-     bootstrap/readiness closure
-   - rebuild completion on replica does not overclaim publication health
-3. proof preserved
-   - all `14A` and `14B` proofs still pass
-   - publication remains derived from explicit core-owned truth
-
---
-
-### `14C` Delivery Note Rev 2
-
-Date: 2026-04-03
-Scope: close the first bounded recovery-closure gap by making catch-up
-completion explicit and projecting recovery truth outward
-
-What changed:
-
-1. `PublicationProjection` now carries `RecoveryView`, so recovery truth is part
-   of outward normalized meaning rather than hidden only in internal state
-2. catch-up now has an explicit closure event:
-   - `CatchUpCompleted`
-3. catch-up completion now:
-   - advances achieved boundary
-   - advances durable boundary on the bounded replica path
-   - returns recovery phase to `idle`
-   - allows mode to return from `bootstrap_pending` to `replica_ready`
-
-Files changed:
-
-1. `sw-block/engine/replication/projection.go`
-   - projection now exposes `RecoveryView`
-2. `sw-block/engine/replication/event.go`
-   - added `CatchUpCompleted`
-3. `sw-block/engine/replication/engine.go`
-   - catch-up planning resets achieved progress for the new plan
-   - catch-up completion explicitly closes recovery phase and updates boundaries
-4. `sw-block/engine/replication/phase14_boundary_test.go`
-   - strengthened catch-up proof with completion semantics
-   - strengthened rebuild proof with outward recovery projection checks
-
-Proofs strengthened:
-
-1. recovery truth is projection-visible
-   - `catching_up`, `rebuilding`, and `idle` are now asserted through outward
-     projection, not only internal state snapshots
-2. catch-up completion closure
-   - replica catch-up returns to `replica_ready`
-   - achieved and durable boundaries converge to the explicit completed target
-   - no publication-health overclaim appears
-
-Validation:
-
-1. `gofmt -w projection.go event.go engine.go phase14_boundary_test.go`
-2. `go test ./...`
-3. result: `PASS`
-
-Constraint / overclaim / proof review:
-
-1. semantic constraint satisfied
-   - recovery closure is now expressed as explicit core-owned truth, not timing
-     intuition
-2. overclaim avoided
-   - catch-up no longer stays indefinitely in an ambiguous in-progress state
-   - recovery truth no longer disappears from outward projection
-3. proof preserved
-   - `14A`, `14B`, and `14C rev 1` proofs still pass
-
---
-
-### `14C` Delivery Note Rev 3
-
-Date: 2026-04-03
-Scope: turn recovery start into explicit bounded command semantics so `catch-up`
-and `rebuild` are not only state/projection truth but also first-class core
-decisions
-
-What changed:
-
-1. added explicit recovery-start commands:
-   - `StartCatchUpCommand`
-   - `StartRebuildCommand`
-2. recovery plan/start events now emit bounded commands:
-   - `CatchUpPlanned(target)` -> `start_catchup` when the target is newly needed
-   - `RebuildStarted(target)` -> `start_rebuild` when the target is newly needed
-3. repeated identical recovery-start events are now true no-op command
-   sequences
-
-Files changed:
-
-1. `sw-block/engine/replication/command.go`
-   - added explicit recovery-start commands
-2. `sw-block/engine/replication/state.go`
-   - extended private command-state tracking for catch-up/rebuild targets
-3. `sw-block/engine/replication/engine.go`
-   - emits bounded recovery-start commands from recovery events
-   - deduplicates repeated identical recovery-start requests
-4. `sw-block/engine/replication/phase14_command_test.go`
-   - added bounded catch-up start proof
-   - added bounded rebuild start proof
-
-Proofs strengthened:
-
-1. catch-up start boundedness
-   - first `CatchUpPlanned(55)` emits:
-     - `start_catchup`
-     - `publish_projection`
-   - repeated identical `CatchUpPlanned(55)` emits no commands
-2. rebuild start boundedness
-   - first `RebuildStarted(80)` emits:
-     - `start_rebuild`
-     - `publish_projection`
-   - repeated identical `RebuildStarted(80)` emits no commands
-
-Validation:
-
-1. `gofmt -w command.go state.go engine.go phase14_command_test.go`
-2. `go test ./...`
-3. result: `PASS`
-
-Constraint / overclaim / proof review:
-
-1. semantic constraint satisfied
-   - recovery policy is now explicit as both state truth and command decision
-2. overclaim avoided
-   - recovery-start intent no longer hides only in state mutation
-   - repeated planning/start events no longer look like fresh work every time
-3. proof preserved
-   - all `14A`, `14B`, and `14C` proofs still pass
-
---
-
-### `14C` Delivery Note Rev 4
-
-Date: 2026-04-03
-Scope: close the stale-recovery leakage gap so old recovery truth and old
-recovery-start intent cannot survive into a new assignment/epoch cycle
-
-What changed:
-
-1. added proof that assignment change clears stale recovery truth:
-   - recovery phase returns to `idle`
-   - target and achieved boundaries are cleared
-   - mode/publication fall back to the new assignment bootstrap state
-2. added proof that a fresh assignment cycle may legitimately re-emit the same
-   recovery-start command for the same target
-
-Files changed:
-
-1. `sw-block/engine/replication/phase14_boundary_test.go`
-   - added stale-recovery-reset proof across assignment/epoch change
-2. `sw-block/engine/replication/phase14_command_test.go`
-   - added fresh-cycle recovery-start reissue proof
-
-Proofs strengthened:
-
-1. stale recovery does not leak across assignment cycles
-2. recovery-start command dedupe is cycle-bounded rather than globally sticky
-
-Validation:
-
-1. `gofmt -w phase14_boundary_test.go phase14_command_test.go`
-2. `go test ./...`
-3. result: `PASS`
-
-Constraint / overclaim / proof review:
-
-1. semantic constraint satisfied
-   - recovery truth and recovery command intent are now scoped to the active
-     assignment/epoch cycle
-2. overclaim avoided
-   - old target/achieved/recovery phase cannot make a new assignment look
-     partially recovered
-   - dedupe state cannot suppress valid fresh-cycle recovery work
-3. proof preserved
-   - all previous `14A/14B/14C` proofs still pass
-
-#### `Phase 14` first-round closure
-
-At this point the first bounded `Phase 14` core shell is in place:
-
-1. `14A` delivered
-   - explicit mode / readiness / publication ownership
-2. `14B` delivered
-   - bounded command-emission rules
-3. `14C` delivered
-   - explicit boundary / recovery truth, projection visibility, recovery-start
-     commands, and assignment-cycle reset rules
-
-Interpretation:
-
-1. this is a real explicit `V2 core` shell in `sw-block/engine/replication`
-2. it is still not a live runtime cutover
-3. the best next step is `Phase 15A` adapter ingress/egress rebinding on one
-   narrow path
-
---
-
-### Post-Closure Tightening
-
-Date: 2026-04-03
-Reason: manager review correctly identified two remaining risks:
-
-1. `14A/14B/14C` slice-boundary blur in top-level phase wording
-2. duplicated `publish_healthy` authority in core state/projection
-
-Actions taken:
-
-1. `sw-block/.private/phase/phase-14.md`
-   - tightened `14A` so it owns only mode/readiness/publication shell closure
-   - made `14B` the explicit owner of command-sequence closure
-   - made `14C` the explicit owner of durable-boundary and recovery closure
-2. `sw-block/engine/replication/state.go`
-   - removed `ReadinessView.PublishHealthy`
-   - documented `PublicationView` as the semantic owner for publication truth
-3. `sw-block/engine/replication/projection.go`
-   - removed duplicate top-level `PublishHealthy` convenience field
-4. `sw-block/engine/replication/engine.go`
-   - publication truth is now carried only through `PublicationView`
-5. `phase14_*_test.go`
-   - switched assertions to `Projection.Publication.Healthy`
-
-Result:
-
-1. `PublicationView` is now the single semantic owner for publication health
-2. `ReadinessView` and `PublicationProjection` no longer carry parallel
-   publication-health truth
-3. top-level `Phase 14` wording now matches the actual `14A/14B/14C` ownership
-   split more closely
@@ -1,206 +0,0 @@
-# Phase 14
-
-Date: 2026-04-03
-Status: delivered
-Purpose: make the `V2 core` explicit inside `sw-block/engine/replication` so
-accepted semantic constraints become executable ownership, rather than staying
-only as design and constrained-`V1` interpretation
-
-## Why This Phase Exists
-
-`Phase 13` accepted a bounded replication-correctness package on the current
-chosen path, including:
-
-1. corrected `sync_all` replication semantics
-2. bounded real-workload validation
-3. assignment/publication closure
-4. bounded mode normalization
-
-That package matters, but it still mostly evaluates `V1` runtime behavior under
-`V2` constraints.
-
-`Phase 14` exists to change that.
-
-The new problem is no longer:
-
-1. keep deepening constrained-`V1` validation as the primary path
-
-It is:
-
-1. make `V2 core` an explicit owner inside the repo
-2. turn accepted claims into core-owned state, events, commands, and projections
-3. create a bounded executable basis for later adapter rebinding
-
-## Phase Goal
-
-Build the first real `V2 core` inside `sw-block/engine/replication` as a
-deterministic, side-effect-free semantic owner for:
-
-1. state and transitions
-2. command decisions
-3. outward projection meaning
-
-This phase does not yet claim live runtime cutover.
-
-## Execution Rule
-
-For all `Phase 14` work, implementation order must be:
-
-1. define core-owned state and transitions
-2. define command-emission rules
-3. define projection contracts
-4. only then connect adapters in later phases
-
-Do not invert this order.
-
-If runtime wiring comes first, `V1` mixed runtime state will silently retake
-semantic authority.
-
-## Execution Model
-
-This phase uses the new working model:
-
-1. primary developer
-   - owns `V2 core` semantic design and implementation
-   - decides state/transition/command/projection shape
-2. `sw`
-   - supports bounded implementation work after semantic ownership is already
-     defined
-   - should receive only narrow, easy-to-accept tasks
-3. `tester`
-   - validates bounded acceptance basis and checks for overclaim
-4. `manager`
-   - performs phase challenge/review gates against semantic discipline
-
-## Scope
-
-### In scope
-
-1. explicit core-owned state in `sw-block/engine/replication`
-2. explicit bounded event vocabulary
-3. explicit bounded command vocabulary
-4. explicit normalized projection vocabulary
-5. structural acceptance tests proving accepted constraints can be represented by
-   the new core
-
-### Out of scope
-
-1. no live `weed/` adapter hook yet
-2. no product-surface rebinding yet
-3. no broad runtime migration
-4. no launch or performance claims
-5. no reopening accepted `Phase 13` claim boundaries
-
-## Phase 14 Slices
-
-### `14A`: Mode / Readiness / Publication Core Closure
-
-Goal:
-
-1. make mode, readiness, and publication first-class core-owned meanings
-
-Acceptance object:
-
-1. `VolumeState`, normalized mode/readiness/publication state, and bounded
-   outward projection exist in `sw-block/engine/replication`
-2. `publish_healthy` is derived from named semantic state rather than runtime
-   convenience
-3. fail-closed mode distinctions stay explicit:
-   - `allocated_only`
-   - `bootstrap_pending`
-   - `replica_ready`
-   - `publish_healthy`
-   - `degraded`
-   - `needs_rebuild`
-4. the structural acceptance tests prove:
-   - `replica_ready` and `publish_healthy` stay distinct
-   - no-replica path stays `allocated_only`
-   - `degraded` and `needs_rebuild` remain distinct fail-closed meanings
-   - the current integrated interpretation remains `constrained_v1`, not live
-     `v2_core` cutover
-
-Ownership boundary:
-
-1. `14A` owns semantic shell closure for:
-   - mode
-   - readiness
-   - publication
-2. `14A` does not own:
-   - command-sequence closure
-   - durable-boundary closure
-   - recovery closure
-
-Status:
-
-1. delivered
-
-### `14B`: Assignment / Command Semantics Closure
-
-Goal:
-
-1. make assignment transitions and command emission rules explicit from semantic
-   state rather than runtime convenience
-
-Acceptance object:
-
-1. assignment intent, role application, receiver start, shipper configuration,
-   and invalidation commands are emitted as bounded semantic decisions
-2. one bounded event sequence produces one bounded command sequence
-3. command emission does not depend on `weed/` internals
-
-Ownership boundary:
-
-1. `14B` owns command-sequence closure
-2. `14B` does not redefine mode/publication ownership from `14A`
-3. `14B` does not absorb durable-boundary or recovery closure from `14C`
-
-Status:
-
-1. delivered
-
-### `14C`: Boundary / Recovery Semantic Closure
-
-Goal:
-
-1. make durable boundary and recovery semantics explicit in the same core owner
-
-Acceptance object:
-
-1. boundary truth distinguishes durable progress, checkpoint truth, and
-   diagnostic sender progress
-2. recovery semantics preserve the accepted constraints around eligibility,
-   fail-closed degradation, and rebuild escalation
-3. structural tests stay bounded and do not claim live path migration yet
-
-Ownership boundary:
-
-1. `14C` owns durable-boundary and recovery closure
-2. `14C` may affect mode/publication only through explicit boundary/recovery
-   truth
-3. `14C` does not reopen `14A` shell ownership or `14B` command-sequence
-   closure
-
-Status:
-
-1. delivered
-
-## Manager Review Gate
-
-Every `Phase 14` slice must survive one challenge review that asks:
-
-1. which semantic constraint does this slice satisfy?
-2. which overclaim does this slice prevent?
-3. which accepted checkpoint proof does this slice preserve?
-
-Reject the slice if any of those questions can only be answered by vague runtime
-intuition.
-
-## Immediate Next Step
-
-Phase 14's first bounded core shell is now in place.
-
-The best next step is `Phase 15A`:
-
-1. connect one narrow adapter ingress into the explicit core
-2. connect one bounded command path back out
-3. prove the live path does not split semantic truth from the new core owner
--- a/Show More
+++ b/Show More