metrics: graph the plugin workers

Nothing displayed the worker metrics, and the panels that look like they
did are about something else: Workers Connected, Worker Slots and Worker
Events in the Admin / Maintenance row read SeaweedFS_admin_*, which the
older maintenance queue feeds. A cluster running plugin workers - Go or
Rust - reads zero there while they are connected and busy.

A Plugin Workers row graphs what the workers themselves publish: how many
are connected, jobs and their failures, detection and proposal rates, job
duration, slot usage, stream events, and what the Lance jobs reclaimed.
The panel worth having is Objects Seen vs Skipped, since a sweep with
nothing to do and a sweep that could read nothing report the same number
of proposals.

Also a commented scrape target in the sample Prometheus config. It is 9328
rather than 9327: the sample compose already gives 9327 to the S3 gateway,
so the port the worker's own usage text suggests collides with it on a
single host.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

.github/workflows/codeql.yml

@@ -27,7 +27,7 @@ jobs:
 
       # Initializes the CodeQL tools for scanning.
       - name: Initialize CodeQL
-        uses: github/codeql-action/init@v4.37.9
+        uses: github/codeql-action/init@v4.37.6
         # Override language selection by uncommenting this and choosing your languages
         with:
           languages: go
@@ -35,7 +35,7 @@ jobs:
       # Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
       # If this step fails, then you should remove it and run the build manually (see below).
       - name: Autobuild
-        uses: github/codeql-action/autobuild@v4.37.9
+        uses: github/codeql-action/autobuild@v4.37.6
 
       # ℹ️ Command-line programs to run using the OS shell.
       # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -49,4 +49,4 @@ jobs:
       #     make release
 
       - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@v4.37.9
+        uses: github/codeql-action/analyze@v4.37.6

.github/workflows/container_dev.yml

@@ -6,7 +6,6 @@ on:
     paths:
       - 'weed/**'
       - 'seaweed-volume/**'
-      - 'seaweed-worker/**'
       - 'docker/**'
       - 'go.mod'
       - 'go.sum'
@@ -17,7 +16,7 @@ permissions:
 
 jobs:
 
-  # ── Pre-build the Rust binaries natively ────────────────────────────
+  # ── Pre-build Rust volume server binaries natively ──────────────────
   build-rust-binaries:
     runs-on: ubuntu-22.04
     strategy:
@@ -56,26 +55,10 @@ jobs:
             ~/.cargo/registry
             ~/.cargo/git
             seaweed-volume/target
-            seaweed-worker/target/${{ matrix.target }}/release
-          key: rust-docker-dev-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock') }}
+          key: rust-docker-dev-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock') }}
           restore-keys: |
             rust-docker-dev-${{ matrix.target }}-
 
-      # lance's build scripts compile their own protos and look for a protoc.
-      # Point them at the one protoc-bin-vendored ships, which seaweed-worker's
-      # own build already uses, so no job depends on a system package and every
-      # build sees the same version.
-      - name: Use the vendored protoc
-        run: |
-          cd seaweed-worker
-          cargo fetch
-          # The version from the lock, not whatever else a restored cache holds.
-          version=$(awk '/^name = "protoc-bin-vendored-linux-x86_64"$/{found=1; next} found && /^version = /{gsub(/"/,"",$3); print $3; exit}' Cargo.lock)
-          test -n "$version" || { echo "protoc-bin-vendored-linux-x86_64 is not in Cargo.lock" >&2; exit 1; }
-          protoc=$(find ~/.cargo/registry/src -path "*protoc-bin-vendored-linux-x86_64-$version/bin/protoc" | head -1)
-          test -x "$protoc" || { echo "no vendored protoc $version in the registry" >&2; exit 1; }
-          echo "PROTOC=$protoc" >> "$GITHUB_ENV"
-
       - name: Build normal variant
         env:
           SEAWEEDFS_COMMIT: ${{ github.sha }}
@@ -84,19 +67,11 @@ jobs:
           cargo build --release --target ${{ matrix.target }} --no-default-features
           cp target/${{ matrix.target }}/release/weed-volume ../weed-volume-normal-${{ matrix.arch }}
 
-      - name: Build the Rust maintenance worker
-        run: |
-          cd seaweed-worker
-          cargo build --release -p weed-lance-worker --target ${{ matrix.target }}
-          cp target/${{ matrix.target }}/release/weed-worker ../weed-worker-${{ matrix.arch }}
-
       - name: Upload artifacts
         uses: actions/upload-artifact@v7
         with:
-          name: rust-bins-${{ matrix.arch }}
-          path: |
-            weed-volume-normal-${{ matrix.arch }}
-            weed-worker-${{ matrix.arch }}
+          name: rust-volume-${{ matrix.arch }}
+          path: weed-volume-normal-${{ matrix.arch }}
 
   build-dev-containers:
     needs: [build-rust-binaries]
@@ -109,7 +84,7 @@ jobs:
       - name: Download pre-built Rust binaries
         uses: actions/download-artifact@v8
         with:
-          pattern: rust-bins-*
+          pattern: rust-volume-*
           merge-multiple: true
           path: ./rust-bins
 
@@ -123,16 +98,7 @@ jobs:
               echo "Placed pre-built Rust binary for ${arch}"
             fi
           done
-          mkdir -p docker/weed-worker-prebuilt
-          for arch in amd64 arm64; do
-            src="./rust-bins/weed-worker-${arch}"
-            if [ -f "$src" ]; then
-              cp "$src" "docker/weed-worker-prebuilt/weed-worker-${arch}"
-              echo "Placed pre-built Rust worker for ${arch}"
-            fi
-          done
           ls -la docker/weed-volume-prebuilt/
-          ls -la docker/weed-worker-prebuilt/
 
       - name: Docker meta
         id: docker_meta

.github/workflows/container_latest.yml

@@ -59,7 +59,7 @@ jobs:
             echo "publish=true" >> "$GITHUB_OUTPUT"
           fi
 
-  # ── Pre-build the Rust binaries natively ────────────────────────────
+  # ── Pre-build Rust volume server binaries natively ──────────────────
   build-rust-binaries:
     runs-on: ubuntu-22.04
     strategy:
@@ -100,26 +100,10 @@ jobs:
             ~/.cargo/registry
             ~/.cargo/git
             seaweed-volume/target
-            seaweed-worker/target/${{ matrix.target }}/release
-          key: rust-docker-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock') }}
+          key: rust-docker-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock') }}
           restore-keys: |
             rust-docker-${{ matrix.target }}-
 
-      # lance's build scripts compile their own protos and look for a protoc.
-      # Point them at the one protoc-bin-vendored ships, which seaweed-worker's
-      # own build already uses, so no job depends on a system package and every
-      # build sees the same version.
-      - name: Use the vendored protoc
-        run: |
-          cd seaweed-worker
-          cargo fetch
-          # The version from the lock, not whatever else a restored cache holds.
-          version=$(awk '/^name = "protoc-bin-vendored-linux-x86_64"$/{found=1; next} found && /^version = /{gsub(/"/,"",$3); print $3; exit}' Cargo.lock)
-          test -n "$version" || { echo "protoc-bin-vendored-linux-x86_64 is not in Cargo.lock" >&2; exit 1; }
-          protoc=$(find ~/.cargo/registry/src -path "*protoc-bin-vendored-linux-x86_64-$version/bin/protoc" | head -1)
-          test -x "$protoc" || { echo "no vendored protoc $version in the registry" >&2; exit 1; }
-          echo "PROTOC=$protoc" >> "$GITHUB_ENV"
-
       - name: Build large-disk variant
         env:
           SEAWEEDFS_COMMIT: ${{ github.sha }}
@@ -136,20 +120,13 @@ jobs:
           cargo build --release --target ${{ matrix.target }} --no-default-features
           cp target/${{ matrix.target }}/release/weed-volume ../weed-volume-normal-${{ matrix.arch }}
 
-      - name: Build the Rust maintenance worker
-        run: |
-          cd seaweed-worker
-          cargo build --release -p weed-lance-worker --target ${{ matrix.target }}
-          cp target/${{ matrix.target }}/release/weed-worker ../weed-worker-${{ matrix.arch }}
-
       - name: Upload artifacts
         uses: actions/upload-artifact@v7
         with:
-          name: rust-bins-${{ matrix.arch }}
+          name: rust-volume-${{ matrix.arch }}
           path: |
             weed-volume-large-disk-${{ matrix.arch }}
             weed-volume-normal-${{ matrix.arch }}
-            weed-worker-${{ matrix.arch }}
 
   build:
     needs: [setup, build-rust-binaries]
@@ -197,7 +174,7 @@ jobs:
       - name: Download pre-built Rust binaries
         uses: actions/download-artifact@v8
         with:
-          pattern: rust-bins-*
+          pattern: rust-volume-*
           merge-multiple: true
           path: ./rust-bins
 
@@ -211,16 +188,7 @@ jobs:
               echo "Placed pre-built Rust binary for ${arch}"
             fi
           done
-          mkdir -p docker/weed-worker-prebuilt
-          for arch in amd64 arm64; do
-            src="./rust-bins/weed-worker-${arch}"
-            if [ -f "$src" ]; then
-              cp "$src" "docker/weed-worker-prebuilt/weed-worker-${arch}"
-              echo "Placed pre-built Rust worker for ${arch}"
-            fi
-          done
           ls -la docker/weed-volume-prebuilt/
-          ls -la docker/weed-worker-prebuilt/
 
       - name: Docker meta
         id: docker_meta
@@ -318,7 +286,7 @@ jobs:
         if: needs.setup.outputs.publish != 'true'
         uses: actions/download-artifact@v8
         with:
-          pattern: rust-bins-*
+          pattern: rust-volume-*
           merge-multiple: true
           path: ./rust-bins
       - name: Place Rust binaries in Docker context for local scan
@@ -336,16 +304,7 @@ jobs:
               echo "Placed pre-built Rust binary for ${arch}"
             fi
           done
-          mkdir -p docker/weed-worker-prebuilt
-          for arch in amd64 arm64; do
-            src="./rust-bins/weed-worker-${arch}"
-            if [ -f "$src" ]; then
-              cp "$src" "docker/weed-worker-prebuilt/weed-worker-${arch}"
-              echo "Placed pre-built Rust worker for ${arch}"
-            fi
-          done
           ls -la docker/weed-volume-prebuilt/
-          ls -la docker/weed-worker-prebuilt/
       - name: Create BuildKit config for local scan build
         if: needs.setup.outputs.publish != 'true'
         run: |
@@ -405,7 +364,7 @@ jobs:
           output: trivy-results.sarif
           exit-code: '0'
       - name: Upload Trivy scan results to GitHub Security
-        uses: github/codeql-action/upload-sarif@v4.37.9
+        uses: github/codeql-action/upload-sarif@v4.37.6
         if: always()
         with:
           sarif_file: trivy-results.sarif

.github/workflows/container_release_unified.yml

@@ -42,10 +42,9 @@ concurrency:
 
 jobs:
 
-  # ── Pre-build the Rust binaries natively ────────────────────────────
-  # The volume server and the Rust maintenance worker, cross-compiled for
-  # amd64 and arm64 without QEMU, turning a 5-hour emulated cargo build into
-  # ~15 minutes of native compilation.
+  # ── Pre-build Rust volume server binaries natively ──────────────────
+  # Cross-compiles for amd64 and arm64 without QEMU, turning a 5-hour
+  # emulated cargo build into ~15 minutes of native compilation.
   build-rust-binaries:
     runs-on: ubuntu-22.04
     strategy:
@@ -84,26 +83,10 @@ jobs:
             ~/.cargo/registry
             ~/.cargo/git
             seaweed-volume/target
-            seaweed-worker/target/${{ matrix.target }}/release
-          key: rust-docker-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock') }}
+          key: rust-docker-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock') }}
           restore-keys: |
             rust-docker-${{ matrix.target }}-
 
-      # lance's build scripts compile their own protos and look for a protoc.
-      # Point them at the one protoc-bin-vendored ships, which seaweed-worker's
-      # own build already uses, so no job depends on a system package and every
-      # build sees the same version.
-      - name: Use the vendored protoc
-        run: |
-          cd seaweed-worker
-          cargo fetch
-          # The version from the lock, not whatever else a restored cache holds.
-          version=$(awk '/^name = "protoc-bin-vendored-linux-x86_64"$/{found=1; next} found && /^version = /{gsub(/"/,"",$3); print $3; exit}' Cargo.lock)
-          test -n "$version" || { echo "protoc-bin-vendored-linux-x86_64 is not in Cargo.lock" >&2; exit 1; }
-          protoc=$(find ~/.cargo/registry/src -path "*protoc-bin-vendored-linux-x86_64-$version/bin/protoc" | head -1)
-          test -x "$protoc" || { echo "no vendored protoc $version in the registry" >&2; exit 1; }
-          echo "PROTOC=$protoc" >> "$GITHUB_ENV"
-
       - name: Build large-disk variant
         env:
           SEAWEEDFS_COMMIT: ${{ github.sha }}
@@ -120,20 +103,13 @@ jobs:
           cargo build --release --target ${{ matrix.target }} --no-default-features
           cp target/${{ matrix.target }}/release/weed-volume ../weed-volume-normal-${{ matrix.arch }}
 
-      - name: Build the Rust maintenance worker
-        run: |
-          cd seaweed-worker
-          cargo build --release -p weed-lance-worker --target ${{ matrix.target }}
-          cp target/${{ matrix.target }}/release/weed-worker ../weed-worker-${{ matrix.arch }}
-
       - name: Upload artifacts
         uses: actions/upload-artifact@v7
         with:
-          name: rust-bins-${{ matrix.arch }}
+          name: rust-volume-${{ matrix.arch }}
           path: |
             weed-volume-large-disk-${{ matrix.arch }}
             weed-volume-normal-${{ matrix.arch }}
-            weed-worker-${{ matrix.arch }}
 
   # One job per (variant, platform) on a native runner, pushed by digest;
   # the merge job stitches the digests into one multi-arch tag.
@@ -179,7 +155,7 @@ jobs:
         if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
         uses: actions/download-artifact@v8
         with:
-          pattern: rust-bins-*
+          pattern: rust-volume-*
           merge-multiple: true
           path: ./rust-bins
 
@@ -194,16 +170,7 @@ jobs:
               echo "Placed pre-built Rust binary for ${arch}"
             fi
           done
-          mkdir -p docker/weed-worker-prebuilt
-          for arch in amd64 arm64; do
-            src="./rust-bins/weed-worker-${arch}"
-            if [ -f "$src" ]; then
-              cp "$src" "docker/weed-worker-prebuilt/weed-worker-${arch}"
-              echo "Placed pre-built Rust worker for ${arch}"
-            fi
-          done
           ls -la docker/weed-volume-prebuilt/
-          ls -la docker/weed-worker-prebuilt/
 
       - name: Free Disk Space
         if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
@@ -430,7 +397,7 @@ jobs:
 
       - name: Upload Trivy scan results to GitHub Security
         if: always()
-        uses: github/codeql-action/upload-sarif@v4.37.9
+        uses: github/codeql-action/upload-sarif@v4.37.6
         with:
           sarif_file: trivy-results.sarif
           category: trivy-${{ matrix.variant }}

.github/workflows/helm_ci.yml

@@ -3,10 +3,10 @@ name: "helm: lint and test charts"
 on:
   push:
     branches: [ master ]
-    paths: ['k8s/**', '.github/workflows/helm_ci.yml']
+    paths: ['k8s/**']
   pull_request:
     branches: [ master ]
-    paths: ['k8s/**', '.github/workflows/helm_ci.yml']
+    paths: ['k8s/**']
 
 permissions:
   contents: read
@@ -58,52 +58,10 @@ jobs:
           grep -q "kind: Deployment" /tmp/s3.yaml && grep -q "seaweedfs-s3" /tmp/s3.yaml
           echo "S3 deployment renders correctly"
           
-          echo "=== Testing S3 credentials from an existing secret ==="
-          credential_args=(
-            --set s3.credentials.admin.existingSecret=minio-root
-            --set s3.credentials.admin.accessKeyKey=root-user
-            --set s3.credentials.admin.secretKeyKey=root-password
-            --set s3.credentials.read.existingSecret=minio-root
-          )
-          for workload in \
-            "s3.enabled=true,s3.enableAuth=true" \
-            "filer.s3.enabled=true,filer.s3.enableAuth=true" \
-            "allInOne.enabled=true,allInOne.s3.enabled=true,allInOne.s3.enableAuth=true"
-          do
-            helm template test $CHART_DIR --set "$workload" "${credential_args[@]}" > /tmp/s3-existing-credentials.yaml
-            # The identities file names the variables, and each is bound to the key it was pointed at.
-            grep -q 'accessKey":"${SEAWEEDFS_S3_ADMIN_ACCESS_KEY_ID}' /tmp/s3-existing-credentials.yaml
-            grep -q 'secretKey":"${SEAWEEDFS_S3_ADMIN_SECRET_ACCESS_KEY}' /tmp/s3-existing-credentials.yaml
-            grep -q 'accessKey":"${SEAWEEDFS_S3_READ_ACCESS_KEY_ID}' /tmp/s3-existing-credentials.yaml
-            for pair in \
-              "SEAWEEDFS_S3_ADMIN_ACCESS_KEY_ID root-user" \
-              "SEAWEEDFS_S3_ADMIN_SECRET_ACCESS_KEY root-password" \
-              "SEAWEEDFS_S3_READ_ACCESS_KEY_ID read_access_key_id" \
-              "SEAWEEDFS_S3_READ_SECRET_ACCESS_KEY read_secret_access_key"
-            do
-              set -- $pair
-              grep -A 4 -- "- name: $1\$" /tmp/s3-existing-credentials.yaml | grep -q "name: \"minio-root\""
-              grep -A 4 -- "- name: $1\$" /tmp/s3-existing-credentials.yaml | grep -q "key: \"$2\""
-            done
-            # The keys stay in the user's secret rather than being copied into the chart's.
-            ! grep -qE "^  (admin|read)_(access_key_id|secret_access_key):" /tmp/s3-existing-credentials.yaml
-            echo "S3 credentials reference the existing secret for $workload"
-          done
-
           echo "=== Testing with all-in-one mode ==="
           helm template test $CHART_DIR --set allInOne.enabled=true > /tmp/allinone.yaml
           grep -q "seaweedfs-all-in-one" /tmp/allinone.yaml
           echo "All-in-one deployment renders correctly"
-
-          echo "=== Testing the all-in-one s3 secret is created by every flag that mounts it ==="
-          for auth in allInOne.s3.enableAuth s3.enableAuth filer.s3.enableAuth; do
-            helm template test $CHART_DIR \
-              --set allInOne.enabled=true --set allInOne.s3.enabled=true --set "$auth=true" \
-              > /tmp/allinone-s3-auth.yaml
-            grep -q "secretName: test-seaweedfs-s3-secret" /tmp/allinone-s3-auth.yaml
-            grep -q "name: test-seaweedfs-s3-secret" /tmp/allinone-s3-auth.yaml
-            echo "All-in-one s3 secret renders for $auth"
-          done
           
           echo "=== Testing with security enabled ==="
           helm template test $CHART_DIR --set global.seaweedfs.enableSecurity=true > /tmp/security.yaml
@@ -482,41 +440,6 @@ jobs:
             --set filer.s3.enableAuth=true > /tmp/filer-s3.yaml
           echo "Filer S3 gateway renders correctly"
           
-          echo "=== Testing the mysql filer store gates its secret and env ==="
-          helm template test $CHART_DIR \
-            --set-string filer.extraEnvironmentVars.WEED_MONGODB_ENABLED=true \
-            --set-string filer.extraEnvironmentVars.WEED_LEVELDB2_ENABLED=false > /tmp/filer-mongodb.yaml
-          ! grep -q "db-secret" /tmp/filer-mongodb.yaml
-          ! grep -q "WEED_MYSQL" /tmp/filer-mongodb.yaml
-          grep -q "name: WEED_MONGODB_ENABLED" /tmp/filer-mongodb.yaml
-          helm template test $CHART_DIR \
-            --set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true > /tmp/filer-mysql.yaml
-          grep -q "name: test-seaweedfs-db-secret" /tmp/filer-mysql.yaml
-          grep -q "name: WEED_MYSQL_USERNAME" /tmp/filer-mysql.yaml
-          grep -q "name: WEED_MYSQL_PASSWORD" /tmp/filer-mysql.yaml
-          grep -q "name: WEED_MYSQL_HOSTNAME" /tmp/filer-mysql.yaml
-          # Secret-backed keys follow the same rule as the plain ones.
-          helm template test $CHART_DIR \
-            --set filer.secretExtraEnvironmentVars.WEED_MYSQL_PASSWORD.secretKeyRef.name=db \
-            --set filer.secretExtraEnvironmentVars.WEED_MYSQL_PASSWORD.secretKeyRef.key=password > /tmp/filer-mysql-off-secret.yaml
-          ! grep -q "WEED_MYSQL" /tmp/filer-mysql-off-secret.yaml
-          helm template test $CHART_DIR \
-            --set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true \
-            --set filer.secretExtraEnvironmentVars.WEED_MYSQL_PASSWORD.secretKeyRef.name=db \
-            --set filer.secretExtraEnvironmentVars.WEED_MYSQL_PASSWORD.secretKeyRef.key=password > /tmp/filer-mysql-on-secret.yaml
-          grep -A 4 -- "- name: WEED_MYSQL_PASSWORD$" /tmp/filer-mysql-on-secret.yaml | grep -q "name: db"
-          # A flag the chart cannot read counts as selected, not as off.
-          helm template test $CHART_DIR \
-            --set filer.secretExtraEnvironmentVars.WEED_MYSQL_ENABLED.secretKeyRef.name=store \
-            --set filer.secretExtraEnvironmentVars.WEED_MYSQL_ENABLED.secretKeyRef.key=enabled > /tmp/filer-mysql-secret.yaml
-          grep -q "name: test-seaweedfs-db-secret" /tmp/filer-mysql-secret.yaml
-          grep -q "name: WEED_MYSQL_HOSTNAME" /tmp/filer-mysql-secret.yaml
-          helm template test $CHART_DIR \
-            --set-string filer.extraEnvironmentVars.WEED_MYSQL2_HOSTNAME=other > /tmp/filer-mysql2.yaml
-          grep -q "name: WEED_MYSQL2_HOSTNAME" /tmp/filer-mysql2.yaml
-          ! grep -q "WEED_MYSQL_" /tmp/filer-mysql2.yaml
-          echo "The mysql secret and env follow the selected filer store"
-
           echo "=== Testing SFTP enabled ==="
           helm template test $CHART_DIR --set sftp.enabled=true > /tmp/sftp.yaml
           grep -q "seaweedfs-sftp" /tmp/sftp.yaml
@@ -1549,37 +1472,11 @@ jobs:
 
           echo "All template rendering tests passed!"
 
-      - name: Resolve an image tag that is published
-        run: |
-          set -e
-          # A release bumps appVersion on master ~40 minutes before the container
-          # build publishes that tag, and this workflow runs on the bump commit.
-          # Install the last released image for the length of that window instead
-          # of failing on ImagePullBackOff.
-          IMAGE=$(helm template test k8s/charts/seaweedfs \
-            -s templates/master/master-statefulset.yaml | awk '$1 == "image:" {print $2; exit}')
-          REPO=${IMAGE%:*}
-          TAG=${IMAGE##*:}
-          # Anything but a published tag installs latest, which between releases
-          # is the same digest as the chart's own appVersion, so a registry blip
-          # costs nothing while failing the job on one would cost a red build.
-          STATUS=$(curl -sSL --connect-timeout 5 --max-time 15 -o /dev/null \
-            -w '%{http_code}' "https://hub.docker.com/v2/repositories/$REPO/tags/$TAG" || true)
-          if [ "$STATUS" = 200 ]; then
-            echo "installing $IMAGE"
-          else
-            echo "$IMAGE is unavailable (HTTP ${STATUS:-none}), installing $REPO:latest"
-            TAG=latest
-          fi
-          echo "IMAGE_TAG=$TAG" >> $GITHUB_ENV
-
       - name: Create kind cluster
         uses: helm/kind-action@v1.14.0
 
       - name: Run chart-testing (install)
-        run: |
-          ct install --target-branch ${{ github.event.repository.default_branch }} --all --chart-dirs k8s/charts \
-            --helm-extra-set-args "--set=image.tag=$IMAGE_TAG"
+        run: ct install --target-branch ${{ github.event.repository.default_branch }} --all --chart-dirs k8s/charts
 
       - name: Verify SFTP host key secret lifecycle
         run: |
@@ -1587,7 +1484,7 @@ jobs:
           CHART_DIR="k8s/charts/seaweedfs"
           NS="sftp-hostkey"
           SECRET="hk-seaweedfs-sftp-ssh-secret"
-          SFTP_ARGS="--set image.tag=$IMAGE_TAG --set sftp.enabled=true --set master.enabled=false --set volume.enabled=false --set filer.enabled=false"
+          SFTP_ARGS="--set sftp.enabled=true --set master.enabled=false --set volume.enabled=false --set filer.enabled=false"
           kubectl create namespace "$NS"
 
           echo "=== install generates a host key, upgrade keeps it ==="
@@ -1664,7 +1561,6 @@ jobs:
           # release if the hook Job does not finish, so a clean install is the
           # assertion.
           helm install np $CHART_DIR -n "$NS" --wait --timeout 8m \
-            --set image.tag=$IMAGE_TAG \
             --set s3.enabled=true \
             --set s3.createBuckets[0].name=testbucket \
             --set networkPolicy.enabled=true \

.github/workflows/java_integration_tests.yml

@@ -34,7 +34,7 @@ jobs:
         id: go
       
       - name: Set up Java
-        uses: actions/setup-java@v6
+        uses: actions/setup-java@v5
         with:
           java-version: ${{ matrix.java }}
           distribution: 'temurin'

.github/workflows/java_release.yml

@@ -73,7 +73,7 @@ jobs:
           echo "version=${VERSION}" >> "$GITHUB_OUTPUT"
 
       - name: Set up JDK 17
-        uses: actions/setup-java@v6
+        uses: actions/setup-java@v5
         with:
           java-version: '17'
           distribution: 'temurin'

.github/workflows/java_unit_tests.yml

@@ -26,7 +26,7 @@ jobs:
         uses: actions/checkout@v7
       
       - name: Set up Java
-        uses: actions/setup-java@v6
+        uses: actions/setup-java@v5
         with:
           java-version: ${{ matrix.java }}
           distribution: 'temurin'

.github/workflows/mount-benchmark.yml

@@ -131,7 +131,7 @@ jobs:
 
     - name: Results
       if: always()
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
         name: results-windows
         path: C:\results-*.json
@@ -196,7 +196,7 @@ jobs:
 
     - name: Results
       if: always()
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
         name: results-linux
         path: /tmp/results-linux.json

.github/workflows/release_version_bump.yml

@@ -1,17 +1,4 @@
-name: "release: bump version and cut the release"
-
-# One entry point for a SeaweedFS release:
-#   1. bump MAJOR/MINOR in constants.go and the Helm Chart.yaml, commit to master
-#   2. push the <appVersion> tag, which fans out to the workflows that trigger on
-#      `push: tags` (binaries_release*, container_release_unified, helm_manual_release)
-#   3. create the GitHub release, with GitHub's generated notes
-#   4. dispatch "Prepare release" in seaweedfs-csi-driver and seaweedfs-operator,
-#      which pick up the new master through `go get -u`, and wait for both
-#
-# Events raised by the default GITHUB_TOKEN do not start other workflows, and it
-# cannot reach the other two repositories at all. Add a repo secret RELEASE_PAT
-# with `contents: write` here and `actions: write` on the csi-driver and operator
-# repos. Without it the tag is still pushed, but nothing downstream of it runs.
+name: "release: bump version"
 
 on:
   workflow_dispatch:
@@ -27,37 +14,15 @@ on:
         description: "Explicit MAJOR.MINOR to set, e.g. 4.36 (overrides 'bump')"
         type: string
         required: false
-      downstream:
-        description: "Also release the CSI driver and the operator"
-        type: boolean
-        default: true
-      dry_run:
-        description: "Show the version bump, but change nothing"
-        type: boolean
-        default: false
+
+permissions:
+  contents: write
 
 jobs:
-  release:
+  bump-version:
     runs-on: ubuntu-latest
-    permissions:
-      contents: write
-    outputs:
-      app_version: ${{ steps.compute.outputs.app_version }}
-      sha: ${{ steps.tag.outputs.sha }}
     steps:
       - uses: actions/checkout@v7
-        with:
-          ref: master
-          fetch-depth: 0
-          token: ${{ secrets.RELEASE_PAT || secrets.GITHUB_TOKEN }}
-
-      - name: Check the release token
-        env:
-          HAS_PAT: ${{ secrets.RELEASE_PAT != '' }}
-        run: |
-          if [ "$HAS_PAT" != "true" ]; then
-            echo "::warning::RELEASE_PAT is not set. The tag will be pushed with GITHUB_TOKEN, so the binary, container and helm workflows will not start on their own."
-          fi
 
       - name: Compute new version
         id: compute
@@ -134,103 +99,17 @@ jobs:
           sed -i -E "s/^version:.*/version: ${CHART_VERSION}/" "$CHART"
           cat "$CHART"
 
-      - name: Commit, and push the tag
-        id: tag
+      - name: Commit and push
         env:
-          TAG: ${{ steps.compute.outputs.app_version }}
-          DRY_RUN: ${{ inputs.dry_run }}
+          APP_VERSION: ${{ steps.compute.outputs.app_version }}
         run: |
           set -euo pipefail
-          if git ls-remote --exit-code --tags origin "refs/tags/${TAG}" >/dev/null 2>&1; then
-            echo "::error::Tag ${TAG} already exists."
-            exit 1
-          fi
-          if [ "$DRY_RUN" = "true" ]; then
-            git --no-pager diff --stat
-            echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
+          if git diff --quiet; then
+            echo "No version change to commit."
             exit 0
           fi
-
           git config user.name "github-actions[bot]"
           git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
-          if git diff --quiet; then
-            echo "::warning::Version files are already at ${TAG}; tagging the current HEAD."
-          else
-            git add weed/util/version/constants.go k8s/charts/seaweedfs/Chart.yaml
-            git commit -m "${TAG}"
-            git push
-          fi
-
-          # Push the tag with git so the `push: tags` triggers fire. Creating the
-          # tag through the release API alone would only emit a `create` event.
-          git tag "$TAG"
-          git push origin "$TAG"
-          echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
-
-      - name: Create the release
-        if: ${{ !inputs.dry_run }}
-        env:
-          GH_TOKEN: ${{ secrets.RELEASE_PAT || secrets.GITHUB_TOKEN }}
-          TAG: ${{ steps.compute.outputs.app_version }}
-        run: gh release create "$TAG" --title "$TAG" --generate-notes --verify-tag
-
-  downstream:
-    needs: release
-    if: ${{ inputs.downstream && !inputs.dry_run }}
-    runs-on: ubuntu-latest
-    permissions: {}
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-          - repo: seaweedfs/seaweedfs-csi-driver
-            workflow: prepare_release.yaml
-          - repo: seaweedfs/seaweedfs-operator
-            workflow: prepare_release.yml
-    steps:
-      - name: Release ${{ matrix.repo }}
-        env:
-          GH_TOKEN: ${{ secrets.RELEASE_PAT }}
-          REPO: ${{ matrix.repo }}
-          WORKFLOW: ${{ matrix.workflow }}
-          SHA: ${{ needs.release.outputs.sha }}
-          MODULE: github.com/seaweedfs/seaweedfs
-        run: |
-          set -euo pipefail
-          if [ -z "${GH_TOKEN}" ]; then
-            echo "::error::RELEASE_PAT with actions:write on ${REPO} is required to release it"
-            exit 1
-          fi
-
-          # The dispatched workflow pins seaweedfs with `go get -u ...@latest`, so
-          # wait until the proxy serves the release commit as the tip. Asking for
-          # the commit by name is what makes the proxy fetch it.
-          for _ in $(seq 30); do
-            curl -sf "https://proxy.golang.org/${MODULE}/@v/${SHA}.info" >/dev/null || true
-            TIP=$(curl -sf "https://proxy.golang.org/${MODULE}/@latest" | jq -r '.Origin.Hash // ""' || true)
-            [ "$TIP" = "$SHA" ] && break
-            sleep 10
-          done
-          if [ "$TIP" != "$SHA" ]; then
-            echo "::error::the module proxy still serves ${TIP} as the tip, so ${REPO} would pin a pre-release commit. Run ${WORKFLOW} there once it catches up."
-            exit 1
-          fi
-
-          # Wait on the release the dispatched workflow publishes, not on the run
-          # that publishes it: a dispatch cannot be told apart from a concurrent
-          # one through the API, and the release is what we are here for.
-          released() { gh api "repos/${REPO}/releases?per_page=30" --jq '[.[].tag_name]'; }
-
-          BEFORE=$(released)
-          gh workflow run -R "$REPO" "$WORKFLOW" --ref master -f bump=patch -f update_seaweedfs=true
-
-          for _ in $(seq 80); do
-            sleep 15
-            NEW=$(released | jq -c --argjson before "$BEFORE" '. - $before')
-            [ "$(jq length <<<"$NEW")" -gt 0 ] && break
-          done
-          if [ "$(jq length <<<"$NEW")" -eq 0 ]; then
-            echo "::error::${REPO} published no release within 20 minutes; see https://github.com/${REPO}/actions/workflows/${WORKFLOW}"
-            exit 1
-          fi
-          echo "${REPO} released $(jq -r 'join(", ")' <<<"$NEW")"
+          git add weed/util/version/constants.go k8s/charts/seaweedfs/Chart.yaml
+          git commit -m "${APP_VERSION}"
+          git push

.github/workflows/rust-worker-tests.yml

@@ -1,82 +0,0 @@
-name: "Rust Plugin Worker Tests"
-
-on:
-  pull_request:
-    branches: [ master ]
-    paths:
-      - 'seaweed-worker/**'
-      - 'weed/pb/plugin.proto'
-      - '.github/workflows/rust-worker-tests.yml'
-  push:
-    branches: [ master, main ]
-    paths:
-      - 'seaweed-worker/**'
-      - 'weed/pb/plugin.proto'
-      - '.github/workflows/rust-worker-tests.yml'
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
-  cancel-in-progress: true
-
-permissions:
-  contents: read
-
-jobs:
-  rust-worker-build:
-    name: Rust Plugin Worker Build and Unit Tests
-    runs-on: ubuntu-22.04
-    timeout-minutes: 45
-
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v7
-        with:
-          persist-credentials: false
-
-      - name: Install Rust toolchain
-        uses: dtolnay/rust-toolchain@stable
-
-      # cargo tracks its own inputs but not the runner's C toolchain, so a cached
-      # target/ can carry C objects built against a different glibc than we link against.
-      - name: Fingerprint build toolchain
-        id: toolchain
-        run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
-
-      - name: Cache cargo registry and target
-        uses: actions/cache@v6
-        with:
-          path: |
-            ~/.cargo/registry
-            ~/.cargo/git
-            seaweed-worker/target/release
-          key: rust-worker-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-worker/Cargo.lock') }}
-          restore-keys: |
-            rust-worker-${{ steps.toolchain.outputs.fingerprint }}-
-
-      # lance's build scripts compile their own protos and look for a protoc.
-      # Point them at the one protoc-bin-vendored ships, which seaweed-worker's
-      # own build already uses, so no job depends on a system package and every
-      # build sees the same version.
-      - name: Use the vendored protoc
-        run: |
-          cd seaweed-worker
-          cargo fetch
-          # The version from the lock, not whatever else a restored cache holds.
-          version=$(awk '/^name = "protoc-bin-vendored-linux-x86_64"$/{found=1; next} found && /^version = /{gsub(/"/,"",$3); print $3; exit}' Cargo.lock)
-          test -n "$version" || { echo "protoc-bin-vendored-linux-x86_64 is not in Cargo.lock" >&2; exit 1; }
-          protoc=$(find ~/.cargo/registry/src -path "*protoc-bin-vendored-linux-x86_64-$version/bin/protoc" | head -1)
-          test -x "$protoc" || { echo "no vendored protoc $version in the registry" >&2; exit 1; }
-          echo "PROTOC=$protoc" >> "$GITHUB_ENV"
-
-      # The release profile is what ships, and it is where the release and the
-      # container builds would otherwise discover a break for the first time.
-      - name: Build the plugin workers
-        run: cd seaweed-worker && cargo build --release
-
-      # The tests that need a live gateway skip themselves without one, the way
-      # the Go integration tests skip without Docker; the lifecycle suite in
-      # test/s3tables/lifecycle is what runs them against a real cluster.
-      # Release, so this reuses the build above rather than compiling lance,
-      # arrow and datafusion a second time in another profile.
-      - name: Run unit tests
-        run: cd seaweed-worker && cargo test --release --workspace

.github/workflows/rust_binaries_release.yml

@@ -1,4 +1,4 @@
-name: "rust: build versioned binaries"
+name: "rust: build versioned volume server binaries"
 
 on:
   push:
@@ -113,102 +113,6 @@ jobs:
             weed-volume_${{ matrix.asset_suffix }}.tar.gz
             weed-volume_${{ matrix.asset_suffix }}.tar.gz.md5
 
-  # The Rust maintenance worker: Linux only, because it runs beside the cluster
-  # it maintains rather than on a laptop, and its dependency tree (lance, arrow,
-  # datafusion) makes every extra target an expensive build.
-  build-rust-worker-linux:
-    permissions:
-      contents: write
-    runs-on: ubuntu-22.04
-    strategy:
-      matrix:
-        include:
-          - target: x86_64-unknown-linux-gnu
-            asset_suffix: linux_amd64
-          - target: aarch64-unknown-linux-gnu
-            asset_suffix: linux_arm64
-            cross: true
-
-    steps:
-      - uses: actions/checkout@v7
-        with:
-          # The upload step is handed a token explicitly; a cargo build script
-          # should not find another one sitting in the checkout's git config.
-          persist-credentials: false
-
-      - name: Install Rust toolchain
-        uses: dtolnay/rust-toolchain@stable
-        with:
-          targets: ${{ matrix.target }}
-
-      - name: Install cross-compilation tools
-        if: matrix.cross
-        run: |
-          sudo dpkg --add-architecture arm64
-          sudo sed -i 's/^deb /deb [arch=amd64] /' /etc/apt/sources.list
-          echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy main restricted universe multiverse" | sudo tee /etc/apt/sources.list.d/arm64.list
-          echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/arm64.list
-          sudo apt-get update
-          sudo apt-get install -y gcc-aarch64-linux-gnu
-          echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
-
-      - name: Cache cargo registry and target
-        uses: actions/cache@v6
-        with:
-          path: |
-            ~/.cargo/registry
-            ~/.cargo/git
-            seaweed-worker/target/${{ matrix.target }}/release
-          key: rust-worker-release-${{ matrix.target }}-${{ hashFiles('seaweed-worker/Cargo.lock') }}
-          restore-keys: |
-            rust-worker-release-${{ matrix.target }}-
-
-      # lance's build scripts compile their own protos and look for a protoc.
-      # Point them at the one protoc-bin-vendored ships, which seaweed-worker's
-      # own build already uses, so no job depends on a system package and every
-      # build sees the same version.
-      - name: Use the vendored protoc
-        run: |
-          cd seaweed-worker
-          cargo fetch
-          # The version from the lock, not whatever else a restored cache holds.
-          version=$(awk '/^name = "protoc-bin-vendored-linux-x86_64"$/{found=1; next} found && /^version = /{gsub(/"/,"",$3); print $3; exit}' Cargo.lock)
-          test -n "$version" || { echo "protoc-bin-vendored-linux-x86_64 is not in Cargo.lock" >&2; exit 1; }
-          protoc=$(find ~/.cargo/registry/src -path "*protoc-bin-vendored-linux-x86_64-$version/bin/protoc" | head -1)
-          test -x "$protoc" || { echo "no vendored protoc $version in the registry" >&2; exit 1; }
-          echo "PROTOC=$protoc" >> "$GITHUB_ENV"
-
-      - name: Build the Rust maintenance worker
-        run: |
-          cd seaweed-worker
-          cargo build --release -p weed-lance-worker --target ${{ matrix.target }}
-
-      - name: Package binary
-        run: |
-          cp seaweed-worker/target/${{ matrix.target }}/release/weed-worker weed-worker
-          tar czf weed-worker_${{ matrix.asset_suffix }}.tar.gz weed-worker
-          rm weed-worker
-          md5sum weed-worker_${{ matrix.asset_suffix }}.tar.gz > weed-worker_${{ matrix.asset_suffix }}.tar.gz.md5
-
-      - name: Upload release assets
-        if: startsWith(github.ref, 'refs/tags/')
-        uses: softprops/action-gh-release@v3
-        with:
-          files: |
-            weed-worker_${{ matrix.asset_suffix }}.tar.gz
-            weed-worker_${{ matrix.asset_suffix }}.tar.gz.md5
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-
-      - name: Upload artifacts
-        if: ${{ !startsWith(github.ref, 'refs/tags/') }}
-        uses: actions/upload-artifact@v7
-        with:
-          name: rust-worker-${{ matrix.asset_suffix }}
-          path: |
-            weed-worker_${{ matrix.asset_suffix }}.tar.gz
-            weed-worker_${{ matrix.asset_suffix }}.tar.gz.md5
-
   build-rust-volume-darwin:
     permissions:
       contents: write

.github/workflows/s3-example-integration-tests.yml

@@ -64,14 +64,6 @@ jobs:
           echo "=== Running S3 Empty Directory Marker Tests ==="
           go test -v -timeout=180s -run TestS3ListObjectsEmptyDirectoryMarkers ./...
 
-      - name: Run S3 Prefix Object Tests
-        timeout-minutes: 15
-        working-directory: test/s3/normal
-        run: |
-          set -x
-          echo "=== Running S3 Prefix Object Tests ==="
-          go test -v -timeout=180s -run TestS3PrefixObjectKeys ./...
-
       - name: Run IAM Integration Tests
         timeout-minutes: 15
         working-directory: test/s3/normal

.github/workflows/s3-tables-tests.yml

@@ -6,7 +6,6 @@ on:
       - 'weed/s3api/**'
       - 'weed/filer/**'
       - 'weed/server/**'
-      - 'weed/worker/tasks/iceberg/**'
       - 'test/s3tables/**'
       - 'go.mod'
       - 'go.sum'
@@ -875,293 +874,6 @@ jobs:
           path: test/s3tables/unity_catalog/test-output.log
           retention-days: 3
 
-  lancedb-namespace-tests:
-    name: LanceDB Namespace Integration Tests
-    runs-on: ubuntu-22.04
-    timeout-minutes: 30
-
-    steps:
-      - name: Check out code
-        uses: actions/checkout@v7
-
-      - name: Set up Go
-        uses: actions/setup-go@v7
-        with:
-          go-version-file: 'go.mod'
-        id: go
-
-      - name: Configure Docker Hub mirror
-        run: |
-          echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
-          sudo systemctl restart docker
-
-      - name: Pre-pull images
-        run: |
-          pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
-          pull python:3.11-slim
-
-      - name: Run go mod tidy
-        run: go mod tidy
-
-      - name: Build SeaweedFS
-        run: |
-          cd weed && go build -buildvcs=false .
-
-      - name: Run LanceDB Namespace Integration Tests
-        timeout-minutes: 25
-        working-directory: test/s3tables/catalog_lancedb
-        run: |
-          set -x
-          set -o pipefail
-          echo "=== System Information ==="
-          uname -a
-          free -h
-          df -h
-          docker info
-          echo "=== Starting LanceDB Namespace Tests ==="
-
-          go test -v -timeout 20m . 2>&1 | tee test-output.log || {
-            echo "LanceDB namespace integration tests failed"
-            exit 1
-          }
-
-      - name: Show test output on failure
-        if: failure()
-        working-directory: test/s3tables/catalog_lancedb
-        run: |
-          echo "=== Test Output ==="
-          if [ -f test-output.log ]; then
-            tail -200 test-output.log
-          fi
-
-          echo "=== Process information ==="
-          ps aux | grep -E "(weed|test|docker)" || true
-
-      - name: Upload test logs on failure
-        if: failure()
-        uses: actions/upload-artifact@v7
-        with:
-          name: lancedb-namespace-test-logs
-          path: test/s3tables/catalog_lancedb/test-output.log
-          retention-days: 3
-
-  table-lifecycle-tests:
-    name: Table Lifecycle Integration Tests
-    runs-on: ubuntu-22.04
-    timeout-minutes: 40
-
-    steps:
-      - name: Check out code
-        uses: actions/checkout@v7
-
-      - name: Set up Go
-        uses: actions/setup-go@v7
-        with:
-          go-version-file: 'go.mod'
-        id: go
-
-      - name: Configure Docker Hub mirror
-        run: |
-          echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
-          sudo systemctl restart docker
-
-      - name: Pre-pull images
-        run: |
-          pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
-          pull python:3.11-slim
-          pull duckdb/duckdb:latest
-
-      - name: Run go mod tidy
-        run: go mod tidy
-
-      - name: Build SeaweedFS
-        run: |
-          cd weed && go build -buildvcs=false .
-
-      - name: Run Table Lifecycle Integration Tests
-        timeout-minutes: 35
-        working-directory: test/s3tables/lifecycle
-        env:
-          # The Rust worker's own tests cover its handlers; a cold build of the
-          # lance crate costs more here than the layer it would be checking.
-          WEED_LANCE_MAINTENANCE: library
-        run: |
-          set -x
-          set -o pipefail
-          go test -v -timeout 30m . 2>&1 | tee test-output.log || {
-            echo "Table lifecycle integration tests failed"
-            exit 1
-          }
-
-      - name: Show test output on failure
-        if: failure()
-        working-directory: test/s3tables/lifecycle
-        run: |
-          echo "=== Test Output ==="
-          if [ -f test-output.log ]; then
-            tail -200 test-output.log
-          fi
-
-          echo "=== Process information ==="
-          ps aux | grep -E "(weed|test|docker)" || true
-
-      - name: Upload test logs on failure
-        if: failure()
-        uses: actions/upload-artifact@v7
-        with:
-          name: table-lifecycle-test-logs
-          path: test/s3tables/lifecycle/test-output.log
-          retention-days: 3
-
-  duckdb-lance-tests:
-    name: DuckDB Lance Integration Tests
-    runs-on: ubuntu-22.04
-    timeout-minutes: 30
-
-    steps:
-      - name: Check out code
-        uses: actions/checkout@v7
-        with:
-          # The job uploads a test log on failure; nothing here needs to push,
-          # so do not leave a token in the checkout for it to pick up.
-          persist-credentials: false
-
-      - name: Set up Go
-        uses: actions/setup-go@v7
-        with:
-          go-version-file: 'go.mod'
-        id: go
-
-      - name: Configure Docker Hub mirror
-        run: |
-          echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
-          sudo systemctl restart docker
-
-      - name: Pre-pull images
-        run: |
-          pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
-          pull duckdb/duckdb:latest
-          pull python:3.11-slim
-
-      - name: Run go mod tidy
-        run: go mod tidy
-
-      - name: Build SeaweedFS
-        run: |
-          cd weed && go build -buildvcs=false .
-
-      - name: Run DuckDB Lance Integration Tests
-        timeout-minutes: 25
-        working-directory: test/s3tables/catalog_duckdb_lance
-        run: |
-          set -x
-          set -o pipefail
-          echo "=== System Information ==="
-          uname -a
-          free -h
-          df -h
-          docker info
-          echo "=== Starting DuckDB Lance Tests ==="
-
-          go test -v -timeout 20m . 2>&1 | tee test-output.log || {
-            echo "DuckDB Lance integration tests failed"
-            exit 1
-          }
-
-      - name: Show test output on failure
-        if: failure()
-        working-directory: test/s3tables/catalog_duckdb_lance
-        run: |
-          echo "=== Test Output ==="
-          if [ -f test-output.log ]; then
-            tail -200 test-output.log
-          fi
-
-          echo "=== Process information ==="
-          ps aux | grep -E "(weed|test|docker|duckdb)" || true
-
-      - name: Upload test logs on failure
-        if: failure()
-        uses: actions/upload-artifact@v7
-        with:
-          name: duckdb-lance-test-logs
-          path: test/s3tables/catalog_duckdb_lance/test-output.log
-          retention-days: 3
-
-  spark-lance-namespace-tests:
-    name: Spark Lance Namespace Integration Tests
-    runs-on: ubuntu-22.04
-    timeout-minutes: 40
-
-    steps:
-      - name: Check out code
-        uses: actions/checkout@v7
-        with:
-          # The job uploads a test log on failure; nothing here needs to push,
-          # so do not leave a token in the checkout for it to pick up.
-          persist-credentials: false
-
-      - name: Set up Go
-        uses: actions/setup-go@v7
-        with:
-          go-version-file: 'go.mod'
-        id: go
-
-      - name: Configure Docker Hub mirror
-        run: |
-          echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
-          sudo systemctl restart docker
-
-      - name: Pre-pull images
-        run: |
-          pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
-          pull apache/spark:3.5.1
-
-      - name: Run go mod tidy
-        run: go mod tidy
-
-      - name: Build SeaweedFS
-        run: |
-          cd weed && go build -buildvcs=false .
-
-      - name: Run Spark Lance Namespace Integration Tests
-        timeout-minutes: 35
-        working-directory: test/s3tables/catalog_spark_lance
-        run: |
-          set -x
-          set -o pipefail
-          echo "=== System Information ==="
-          uname -a
-          free -h
-          df -h
-          docker info
-          echo "=== Starting Spark Lance Namespace Tests ==="
-
-          go test -v -timeout 30m . 2>&1 | tee test-output.log || {
-            echo "Spark Lance namespace integration tests failed"
-            exit 1
-          }
-
-      - name: Show test output on failure
-        if: failure()
-        working-directory: test/s3tables/catalog_spark_lance
-        run: |
-          echo "=== Test Output ==="
-          if [ -f test-output.log ]; then
-            tail -200 test-output.log
-          fi
-
-          echo "=== Process information ==="
-          ps aux | grep -E "(weed|test|docker|spark)" || true
-
-      - name: Upload test logs on failure
-        if: failure()
-        uses: actions/upload-artifact@v7
-        with:
-          name: spark-lance-namespace-test-logs
-          path: test/s3tables/catalog_spark_lance/test-output.log
-          retention-days: 3
-
   s3-tables-build-verification:
     name: S3 Tables Build Verification
     runs-on: ubuntu-22.04

.github/workflows/spark-integration-tests.yml

@@ -34,7 +34,7 @@ jobs:
       uses: actions/checkout@v7
 
     - name: Set up JDK 11
-      uses: actions/setup-java@v6
+      uses: actions/setup-java@v5
       with:
         java-version: '11'
         distribution: 'temurin'

README.md

@@ -90,7 +90,7 @@ S3_BUCKET=my-bucket \
 ./weed mini -dir=/data
 ```
 
-That's it — the S3 endpoint is at http://localhost:8333, `my-bucket` already exists, and `admin`/`secret` are valid credentials. `S3_BUCKET` accepts a comma-separated list (e.g. `raw,processed`); use `S3_TABLE_BUCKET` for S3 Tables buckets, each `name` or `name:FORMAT` where the format is `ICEBERG` (the default) or `LANCE`. Drop any of the env vars to skip that piece (no AWS keys → S3 runs in unauthenticated "Allow All" mode for development).
+That's it — the S3 endpoint is at http://localhost:8333, `my-bucket` already exists, and `admin`/`secret` are valid credentials. `S3_BUCKET` accepts a comma-separated list (e.g. `raw,processed`); use `S3_TABLE_BUCKET` for S3 Tables (Iceberg) buckets. Drop any of the env vars to skip that piece (no AWS keys → S3 runs in unauthenticated "Allow All" mode for development).
 
 The same command starts everything else too:
 - **S3 Endpoint**: http://localhost:8333

docker/Dockerfile.go_build

@@ -29,7 +29,6 @@ FROM alpine:3.23 as rust_builder
 ARG TARGETARCH
 ARG TAGS
 COPY weed-volume-prebuilt/ /prebuilt/
-COPY weed-worker-prebuilt/ /prebuilt-worker/
 COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/seaweed-volume /build/seaweed-volume
 COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/weed /build/weed
 WORKDIR /build/seaweed-volume
@@ -48,30 +47,16 @@ RUN if [ -f "/prebuilt/weed-volume-${TARGETARCH}" ]; then \
       echo "Skipping Rust build for $TARGETARCH (unsupported)" && \
       touch /weed-volume; \
     fi
-# The Rust maintenance worker is taken pre-built or not at all: the lance jobs
-# it carries pull in arrow and datafusion, a far larger dependency tree than
-# the image build can carry, so an architecture CI did not build for gets the
-# same empty placeholder the entrypoint refuses to exec.
-RUN if [ -f "/prebuilt-worker/weed-worker-${TARGETARCH}" ]; then \
-      echo "Using pre-built Rust worker for ${TARGETARCH}" && \
-      cp "/prebuilt-worker/weed-worker-${TARGETARCH}" /weed-worker; \
-    else \
-      echo "No pre-built Rust worker for ${TARGETARCH}" && \
-      touch /weed-worker; \
-    fi
-
 # Pre-built binaries arrive via GitHub Actions artifacts, which drop the
 # executable bit, so the copied file is 0644 and exec fails with "Permission
-# denied". Restore it (no-op for the empty placeholders, which stay size 0).
-RUN chmod 0755 /weed-volume /weed-worker
+# denied". Restore it (no-op for the empty placeholder, which stays size 0).
+RUN chmod 0755 /weed-volume
 
 FROM alpine AS final
 LABEL author="Chris Lu"
 COPY --from=builder /go/bin/weed /usr/bin/
 # Copy Rust volume server binary (real binary on amd64/arm64, empty placeholder on other platforms)
 COPY --from=rust_builder /weed-volume /usr/bin/weed-volume
-# Same for the Rust maintenance worker, which serves Lance table buckets
-COPY --from=rust_builder /weed-worker /usr/bin/weed-worker
 RUN mkdir -p /etc/seaweedfs
 COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/docker/filer.toml /etc/seaweedfs/filer.toml
 COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/docker/entrypoint.sh /entrypoint.sh

docker/entrypoint.sh

@@ -90,16 +90,6 @@ case "$1" in
   	exec /usr/bin/weed-volume $ARGS $@
 	;;
 
-  'worker-rust')
-  	shift
-  	if [ ! -s /usr/bin/weed-worker ]; then
-  	  echo "Error: Rust maintenance worker is not available on this platform ($(uname -m))." >&2
-  	  echo "Use 'worker' for the Go maintenance worker instead." >&2
-  	  exit 1
-  	fi
-  	exec /usr/bin/weed-worker "$@"
-	;;
-
   'server')
   	ARGS="-dir=/data -volume.max=0 -master.volumeSizeLimitMB=1024"
   	if isArgPassed "-volume.max" "$@"; then

go.mod

@@ -4,7 +4,7 @@ go 1.26
 
 require (
 	cloud.google.com/go v0.123.0 // indirect
-	cloud.google.com/go/pubsub v1.51.1
+	cloud.google.com/go/pubsub v1.51.0
 	cloud.google.com/go/storage v1.64.0
 	github.com/Shopify/sarama v1.38.1
 	github.com/aws/aws-sdk-go v1.55.8
@@ -32,7 +32,7 @@ require (
 	github.com/google/btree v1.1.3
 	github.com/google/uuid v1.6.0
 	github.com/google/wire v0.7.0 // indirect
-	github.com/googleapis/gax-go/v2 v2.24.0 // indirect
+	github.com/googleapis/gax-go/v2 v2.23.0 // indirect
 	github.com/gorilla/mux v1.8.1
 	github.com/hashicorp/errwrap v1.1.0 // indirect
 	github.com/hashicorp/go-multierror v1.1.1 // indirect
@@ -64,7 +64,7 @@ require (
 	github.com/prometheus/procfs v0.21.1
 	github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
-	github.com/seaweedfs/goexif v2.0.0+incompatible
+	github.com/seaweedfs/goexif v1.0.3
 	github.com/seaweedfs/raft v1.2.0
 	github.com/sirupsen/logrus v1.9.4 // indirect
 	github.com/spf13/afero v1.15.0 // indirect
@@ -99,15 +99,15 @@ require (
 	golang.org/x/text v0.41.0 // indirect
 	golang.org/x/tools v0.48.0 // indirect
 	golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
-	google.golang.org/api v0.294.0
-	google.golang.org/genproto v0.0.0-20260715232425-e75dac1f907d // indirect
-	google.golang.org/grpc v1.85.0-dev
-	google.golang.org/protobuf v1.36.12
+	google.golang.org/api v0.293.0
+	google.golang.org/genproto v0.0.0-20260519071638-aa98bba5eb94 // indirect
+	google.golang.org/grpc v1.84.0-dev.0.20260723093437-b6eac429d7b6
+	google.golang.org/protobuf v1.36.11
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	modernc.org/b v1.0.0 // indirect
 	modernc.org/mathutil v1.7.1 // indirect
 	modernc.org/memory v1.11.0 // indirect
-	modernc.org/sqlite v1.57.0
+	modernc.org/sqlite v1.56.0
 )
 
 require (
@@ -122,14 +122,14 @@ require (
 	github.com/apple/foundationdb/bindings/go v0.0.0-20250911184653-27f7192f47c3
 	github.com/arangodb/go-driver v1.6.9
 	github.com/armon/go-metrics v0.4.1
-	github.com/aws/aws-sdk-go-v2 v1.45.1
+	github.com/aws/aws-sdk-go-v2 v1.43.5
 	github.com/aws/aws-sdk-go-v2/config v1.32.33
 	github.com/aws/aws-sdk-go-v2/credentials v1.19.34
 	github.com/aws/aws-sdk-go-v2/service/s3 v1.105.2
 	github.com/cespare/xxhash/v2 v2.3.0
 	github.com/cognusion/imaging v1.0.4
 	github.com/fluent/fluent-logger-golang v1.10.1
-	github.com/getsentry/sentry-go v0.48.0
+	github.com/getsentry/sentry-go v0.44.1
 	github.com/go-ldap/ldap/v3 v3.4.13
 	github.com/golang-jwt/jwt/v5 v5.3.1
 	github.com/google/flatbuffers/go v0.0.0-20230108230133-3b8644d32c50
@@ -141,22 +141,22 @@ require (
 	github.com/linkedin/goavro/v2 v2.15.0
 	github.com/minio/crc64nvme v1.1.1
 	github.com/orcaman/concurrent-map/v2 v2.0.1
-	github.com/parquet-go/parquet-go v0.32.0
+	github.com/parquet-go/parquet-go v0.30.1
 	github.com/pkg/sftp v1.13.11
-	github.com/rabbitmq/amqp091-go v1.14.0
+	github.com/rabbitmq/amqp091-go v1.13.0
 	github.com/rclone/rclone v1.75.0
 	github.com/rdleal/intervalst v1.5.0
 	github.com/redis/go-redis/v9 v9.21.0
 	github.com/schollz/progressbar/v3 v3.19.1
 	github.com/seaweedfs/go-fuse/v2 v2.9.4
-	github.com/shirou/gopsutil/v4 v4.26.7
+	github.com/shirou/gopsutil/v4 v4.26.6
 	github.com/tarantool/go-option v1.1.0
-	github.com/tarantool/go-tarantool/v3 v3.0.1
+	github.com/tarantool/go-tarantool/v3 v3.0.0
 	github.com/testcontainers/testcontainers-go v0.43.0
 	github.com/tikv/client-go/v2 v2.0.7
 	github.com/xeipuuv/gojsonschema v1.2.0
 	github.com/ydb-platform/ydb-go-sdk-auth-environ v0.5.2
-	github.com/ydb-platform/ydb-go-sdk/v3 v3.151.1
+	github.com/ydb-platform/ydb-go-sdk/v3 v3.147.1
 	go.etcd.io/etcd/client/pkg/v3 v3.7.1
 	go.uber.org/atomic v1.11.0
 	golang.org/x/sync v0.22.0
@@ -193,7 +193,7 @@ require (
 	github.com/cenkalti/backoff/v5 v5.0.3 // indirect
 	github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
 	github.com/cockroachdb/apd/v3 v3.2.1 // indirect
-	github.com/cockroachdb/errors v1.14.0 // indirect
+	github.com/cockroachdb/errors v1.11.3 // indirect
 	github.com/cockroachdb/logtags v0.0.0-20241215232642-bb51bb14a506 // indirect
 	github.com/cockroachdb/redact v1.1.5 // indirect
 	github.com/cockroachdb/version v0.0.0-20250314144055-3860cd14adf2 // indirect
@@ -238,12 +238,12 @@ require (
 	github.com/lpar/calendar v0.2.0 // indirect
 	github.com/magiconair/properties v1.8.10 // indirect
 	github.com/moby/docker-image-spec v1.3.1 // indirect
-	github.com/moby/go-archive v0.3.0 // indirect
+	github.com/moby/go-archive v0.2.0 // indirect
 	github.com/moby/moby/api v1.54.2 // indirect
 	github.com/moby/moby/client v0.4.0 // indirect
 	github.com/moby/patternmatcher v0.6.1 // indirect
-	github.com/moby/sys/sequential v0.7.0 // indirect
-	github.com/moby/sys/user v0.4.1 // indirect
+	github.com/moby/sys/sequential v0.6.0 // indirect
+	github.com/moby/sys/user v0.4.0 // indirect
 	github.com/moby/sys/userns v0.1.0 // indirect
 	github.com/moby/term v0.5.2 // indirect
 	github.com/oklog/ulid/v2 v2.1.1 // indirect
@@ -260,7 +260,6 @@ require (
 	github.com/rclone/Proton-API-Bridge v1.0.4 // indirect
 	github.com/rclone/go-proton-api v1.0.3 // indirect
 	github.com/rogpeppe/go-internal v1.15.0 // indirect
-	github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd // indirect
 	github.com/ryanuber/go-glob v1.0.0 // indirect
 	github.com/sasha-s/go-deadlock v0.3.1 // indirect
 	github.com/smarty/assertions v1.15.0 // indirect
@@ -292,11 +291,11 @@ require (
 
 require (
 	cel.dev/expr v0.25.2 // indirect
-	cloud.google.com/go/auth v0.23.2 // indirect
+	cloud.google.com/go/auth v0.23.0 // indirect
 	cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
 	cloud.google.com/go/compute/metadata v0.9.0 // indirect
-	cloud.google.com/go/iam v1.12.0 // indirect
-	cloud.google.com/go/monitoring v1.30.0 // indirect
+	cloud.google.com/go/iam v1.11.0 // indirect
+	cloud.google.com/go/monitoring v1.29.0 // indirect
 	filippo.io/edwards25519 v1.2.0 // indirect
 	github.com/Azure/azure-sdk-for-go/sdk/azcore v1.22.0
 	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.0
@@ -337,7 +336,7 @@ require (
 	github.com/aws/aws-sdk-go-v2/service/sso v1.33.4 // indirect
 	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.4 // indirect
 	github.com/aws/aws-sdk-go-v2/service/sts v1.45.4
-	github.com/aws/smithy-go v1.28.1
+	github.com/aws/smithy-go v1.27.7
 	github.com/boltdb/bolt v1.3.1 // indirect
 	github.com/bradenaw/juniper v0.15.3 // indirect
 	github.com/buengese/sgzip v0.1.1 // indirect
@@ -355,7 +354,7 @@ require (
 	github.com/d4l3k/messagediff v1.2.1 // indirect
 	github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 // indirect
 	github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.4.0 // indirect
-	github.com/ebitengine/purego v0.10.2 // indirect
+	github.com/ebitengine/purego v0.10.1 // indirect
 	github.com/elastic/gosigar v0.14.3 // indirect
 	github.com/emersion/go-message v0.18.2 // indirect
 	github.com/emersion/go-vcard v0.0.0-20260618161152-d854b7e0e2d3 // indirect
@@ -475,7 +474,7 @@ require (
 	github.com/winfsp/cgofuse v1.6.1-0.20260126094232-f2c4fccdb286
 	github.com/xanzy/ssh-agent v0.3.3 // indirect
 	github.com/yandex-cloud/go-genproto v0.0.0-20211115083454-9ca41db5ed9e // indirect
-	github.com/ydb-platform/ydb-go-genproto v0.0.0-20260810122915-65bfd5c4b705 // indirect
+	github.com/ydb-platform/ydb-go-genproto v0.0.0-20260428144813-1c07baab7f7b // indirect
 	github.com/ydb-platform/ydb-go-yc v0.12.1 // indirect
 	github.com/ydb-platform/ydb-go-yc-metadata v0.6.1 // indirect
 	github.com/yunify/qingstor-sdk-go/v3 v3.2.0 // indirect
@@ -497,8 +496,8 @@ require (
 	go.uber.org/zap v1.27.1 // indirect
 	golang.org/x/term v0.45.0
 	golang.org/x/time v0.15.0
-	google.golang.org/genproto/googleapis/api v0.0.0-20260715232425-e75dac1f907d // indirect
-	google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20260706201446-f0a921348800 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20260807164820-c8921c73eeea // indirect
 	gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
 	gopkg.in/validator.v2 v2.0.1 // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect

go.sum

@@ -94,8 +94,8 @@ cloud.google.com/go/assuredworkloads v1.7.0/go.mod h1:z/736/oNmtGAyU47reJgGN+KVo
 cloud.google.com/go/assuredworkloads v1.8.0/go.mod h1:AsX2cqyNCOvEQC8RMPnoc0yEarXQk6WEKkxYfL6kGIo=
 cloud.google.com/go/assuredworkloads v1.9.0/go.mod h1:kFuI1P78bplYtT77Tb1hi0FMxM0vVpRC7VVoJC3ZoT0=
 cloud.google.com/go/assuredworkloads v1.10.0/go.mod h1:kwdUQuXcedVdsIaKgKTp9t0UJkE5+PAVNhdQm4ZVq2E=
-cloud.google.com/go/auth v0.23.2 h1:pxSCpfiji41hpzpPdMCftEUCezpgpqmmDdYiAjCKXxo=
-cloud.google.com/go/auth v0.23.2/go.mod h1:4DhBRcqvtljQN3dJ57qtqbib5ZGCYE5f2crfiiC2EM0=
+cloud.google.com/go/auth v0.23.0 h1:6Gg1CMgpgubRG7DGz5Vf1pcoNo8RfiRiRAPS4crTp54=
+cloud.google.com/go/auth v0.23.0/go.mod h1:4DhBRcqvtljQN3dJ57qtqbib5ZGCYE5f2crfiiC2EM0=
 cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIiLpZnkHRbnc=
 cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c=
 cloud.google.com/go/automl v1.5.0/go.mod h1:34EjfoFGMZ5sgJ9EoLsRtdPSNZLcfflJR39VbVNS2M0=
@@ -283,8 +283,8 @@ cloud.google.com/go/iam v0.7.0/go.mod h1:H5Br8wRaDGNc8XP3keLc4unfUUZeyH3Sfl9XpQE
 cloud.google.com/go/iam v0.8.0/go.mod h1:lga0/y3iH6CX7sYqypWJ33hf7kkfXJag67naqGESjkE=
 cloud.google.com/go/iam v0.11.0/go.mod h1:9PiLDanza5D+oWFZiH1uG+RnRCfEGKoyl6yo4cgWZGY=
 cloud.google.com/go/iam v0.12.0/go.mod h1:knyHGviacl11zrtZUoDuYpDgLjvr28sLQaG0YB2GYAY=
-cloud.google.com/go/iam v1.12.0 h1:Aki3bX9aHUDKPHfnRJfDcTdVedvy6quGBQcTqx3DRXk=
-cloud.google.com/go/iam v1.12.0/go.mod h1:FEZ4lXpADAC2AIpQY7LANNjjwyQ2jK439CI2VaD+sLY=
+cloud.google.com/go/iam v1.11.0 h1:KieQ9Pb+LLPak1O3Rv3GgCxhnmkYf7Xyh0P5HfF1jFM=
+cloud.google.com/go/iam v1.11.0/go.mod h1:KP+nKGugNJW4LcLx1uEZcq1ok5sQHFaQehQNl4QDgV4=
 cloud.google.com/go/iap v1.4.0/go.mod h1:RGFwRJdihTINIe4wZ2iCP0zF/qu18ZwyKxrhMhygBEc=
 cloud.google.com/go/iap v1.5.0/go.mod h1:UH/CGgKd4KyohZL5Pt0jSKE4m3FR51qg6FKQ/z/Ix9A=
 cloud.google.com/go/iap v1.6.0/go.mod h1:NSuvI9C/j7UdjGjIde7t7HBz+QTwBcapPE07+sSRcLk=
@@ -310,8 +310,8 @@ cloud.google.com/go/lifesciences v0.6.0/go.mod h1:ddj6tSX/7BOnhxCSd3ZcETvtNr8NZ6
 cloud.google.com/go/lifesciences v0.8.0/go.mod h1:lFxiEOMqII6XggGbOnKiyZ7IBwoIqA84ClvoezaA/bo=
 cloud.google.com/go/logging v1.6.1/go.mod h1:5ZO0mHHbvm8gEmeEUHrmDlTDSu5imF6MUP9OfilNXBw=
 cloud.google.com/go/logging v1.7.0/go.mod h1:3xjP2CjkM3ZkO73aj4ASA5wRPGGCRrPIAeNqVNkzY8M=
-cloud.google.com/go/logging v1.19.0 h1:NCqhdVUg3wQ8Cobdf16FDSuTGi3+6+hdSBHrY5TsR6Q=
-cloud.google.com/go/logging v1.19.0/go.mod h1:i40NZCHC9Gqvod4yE+yQfDWwlgwW/SrshkkGibCHxcA=
+cloud.google.com/go/logging v1.18.0 h1:KhzZq+1cSkPH9YUaKLLhLtQxIHitVayBmk0sGfoM9+k=
+cloud.google.com/go/logging v1.18.0/go.mod h1:ZGKnpBaURITh+g/uom2VhbiFoFWvejcrHPDhxFtU/gI=
 cloud.google.com/go/longrunning v0.1.1/go.mod h1:UUFxuDWkv22EuY93jjmDMFT5GPQKeFVJBIF6QlTqdsE=
 cloud.google.com/go/longrunning v0.3.0/go.mod h1:qth9Y41RRSUE69rDcOn6DdK3HfQfsUI0YSmW3iIlLJc=
 cloud.google.com/go/longrunning v0.4.1/go.mod h1:4iWDqhBZ70CvZ6BfETbvam3T8FMvLK+eFj0E6AaRQTo=
@@ -338,8 +338,8 @@ cloud.google.com/go/metastore v1.10.0/go.mod h1:fPEnH3g4JJAk+gMRnrAnoqyv2lpUCqJP
 cloud.google.com/go/monitoring v1.7.0/go.mod h1:HpYse6kkGo//7p6sT0wsIC6IBDET0RhIsnmlA53dvEk=
 cloud.google.com/go/monitoring v1.8.0/go.mod h1:E7PtoMJ1kQXWxPjB6mv2fhC5/15jInuulFdYYtlcvT4=
 cloud.google.com/go/monitoring v1.12.0/go.mod h1:yx8Jj2fZNEkL/GYZyTLS4ZtZEZN8WtDEiEqG4kLK50w=
-cloud.google.com/go/monitoring v1.30.0 h1:r/d+JUbyKmJ8b07iznuKfzVzrIXTWxHQ3lBRm3x2LlY=
-cloud.google.com/go/monitoring v1.30.0/go.mod h1:htlUR0QWVMrjFzZmN4LGnMAve9xB/eduwjmINxVZ8RM=
+cloud.google.com/go/monitoring v1.29.0 h1:AHhDsFaSax1/4k+qlIDX/SDGe6hggnfXJ9dkgD9qBPY=
+cloud.google.com/go/monitoring v1.29.0/go.mod h1:72NOVjJXHY/HBfoLT0+qlCZBT059+9VXLeAnL2PeeVM=
 cloud.google.com/go/networkconnectivity v1.4.0/go.mod h1:nOl7YL8odKyAOtzNX73/M5/mGZgqqMeryi6UPZTk/rA=
 cloud.google.com/go/networkconnectivity v1.5.0/go.mod h1:3GzqJx7uhtlM3kln0+x5wyFvuVH1pIBJjhCpjzSt75o=
 cloud.google.com/go/networkconnectivity v1.6.0/go.mod h1:OJOoEXW+0LAxHh89nXd64uGG+FbQoeH8DtxCHVOMlaM=
@@ -391,8 +391,8 @@ cloud.google.com/go/pubsub v1.3.1/go.mod h1:i+ucay31+CNRpDW4Lu78I4xXG+O1r/MAHgjp
 cloud.google.com/go/pubsub v1.26.0/go.mod h1:QgBH3U/jdJy/ftjPhTkyXNj543Tin1pRYcdcPRnFIRI=
 cloud.google.com/go/pubsub v1.27.1/go.mod h1:hQN39ymbV9geqBnfQq6Xf63yNhUAhv9CZhzp5O6qsW0=
 cloud.google.com/go/pubsub v1.28.0/go.mod h1:vuXFpwaVoIPQMGXqRyUQigu/AX1S3IWugR9xznmcXX8=
-cloud.google.com/go/pubsub v1.51.1 h1:R3G1wCOxBO7jRpL8x2pdZMv1GAJDF6ax/m2zPOtvTNE=
-cloud.google.com/go/pubsub v1.51.1/go.mod h1:y2T0IKtW1iWwVvazYaRpqOAFO4gy2+O7dTDt9TWY/5U=
+cloud.google.com/go/pubsub v1.51.0 h1:XOaCejsqX7EEtUdQz+WPag66wWsUUGliyCOfGPKfo90=
+cloud.google.com/go/pubsub v1.51.0/go.mod h1:NERXf11sd82UV3VnflcUj8POIyQUXT/QwrKlxD8di/I=
 cloud.google.com/go/pubsub/v2 v2.6.0 h1:8pjR0id+GTB+krKx5G6AGJoYrHog58w2Q89PCOrfM64=
 cloud.google.com/go/pubsub/v2 v2.6.0/go.mod h1:4anqvV/w8Pcgu2tO0qr2XgsF3GXHowzryfQ5gOnVmWY=
 cloud.google.com/go/pubsublite v1.5.0/go.mod h1:xapqNQ1CuLfGi23Yda/9l4bBCKz/wC3KIJ5gKcxveZg=
@@ -707,8 +707,8 @@ github.com/armon/go-metrics v0.4.1/go.mod h1:E6amYzXo6aW1tqzoZGT755KkbgrJsSdpwZ+
 github.com/atomicgo/cursor v0.0.1/go.mod h1:cBON2QmmrysudxNBFthvMtN32r3jxVRIvzkUiF/RuIk=
 github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
 github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
-github.com/aws/aws-sdk-go-v2 v1.45.1 h1:iIoG3NaLhV6UZpPXyPXlDj2I9oS8tV/nMcMnITCC6Ks=
-github.com/aws/aws-sdk-go-v2 v1.45.1/go.mod h1:bttEH6JqnUL8LepvDVfdrds/fZ5bCIxzpe3abyUrhDU=
+github.com/aws/aws-sdk-go-v2 v1.43.5 h1:yKT5GYnFWhuDo+DqKvE5ZPwVn3RjC4MAeBtZGlh6AVM=
+github.com/aws/aws-sdk-go-v2 v1.43.5/go.mod h1:wZjAJppCntyOGgVSmgVTfDyRJK5PHOasO6Wsy8U7Axk=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 h1:3IZY0XAJquT3aHzbkHfPzy4ACPcEjVG0x87KOwtpqGY=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14/go.mod h1:zwM6veDkhGgQFqkBy+uT28AAYpLu+uFMlPl+rCg/73E=
 github.com/aws/aws-sdk-go-v2/config v1.32.33 h1:M1m/Q6f0OKDEDGwhiNOqx1OjTdrewe3v+GDbHmKczWk=
@@ -749,8 +749,8 @@ github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.4 h1:AsbZcJAQPRmHDJG8K1N0pof/
 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.4/go.mod h1:6imqztH0//t0mKbl6yWl7swSEl7F/w32oAmqB3vP1ag=
 github.com/aws/aws-sdk-go-v2/service/sts v1.45.4 h1:w/AryDYMjSUANSQ2uoZxJovUsMTwWJNTv3IMex30Y+4=
 github.com/aws/aws-sdk-go-v2/service/sts v1.45.4/go.mod h1:WeBiAa67azG7Su9Vf+ChGDBLiAozJCXzdjXiPBUwtbc=
-github.com/aws/smithy-go v1.28.1 h1:R/nXH00c8qcfCzQVELtRw+eLQWtzv+VAIEFJ1/xxXlQ=
-github.com/aws/smithy-go v1.28.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
+github.com/aws/smithy-go v1.27.7 h1:Zgj5z4LfcDYoQIVk+n/yGdTkP/2y6ZT5vYxe0fp7bqE=
+github.com/aws/smithy-go v1.27.7/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
 github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
 github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
 github.com/bazelbuild/rules_go v0.46.0 h1:CTefzjN/D3Cdn3rkrM6qMWuQj59OBcuOjyIp3m4hZ7s=
@@ -848,8 +848,8 @@ github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 h1:aBangftG7EVZoUb69Os
 github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2/go.mod h1:qwXFYgsP6T7XnJtbKlf1HP8AjxZZyzxMmc+Lq5GjlU4=
 github.com/cockroachdb/apd/v3 v3.2.1 h1:U+8j7t0axsIgvQUqthuNm82HIrYXodOV2iWLWtEaIwg=
 github.com/cockroachdb/apd/v3 v3.2.1/go.mod h1:klXJcjp+FffLTHlhIG69tezTDvdP065naDsHzKhYSqc=
-github.com/cockroachdb/errors v1.14.0 h1:EfdVEJpN3z8rPMo43Yit59LxoiIa470fSXpZXuEs+ZI=
-github.com/cockroachdb/errors v1.14.0/go.mod h1:xRa70jZ9sNBQmISt5KmJmAD++E4dQHm89oCRiZGEdq0=
+github.com/cockroachdb/errors v1.11.3 h1:5bA+k2Y6r+oz/6Z/RFlNeVCesGARKuC6YymtcDrbC/I=
+github.com/cockroachdb/errors v1.11.3/go.mod h1:m4UIW4CDjx+R5cybPsNrRbreomiFqt8o1h1wUVazSd8=
 github.com/cockroachdb/logtags v0.0.0-20241215232642-bb51bb14a506 h1:ASDL+UJcILMqgNeV5jiqR4j+sTuvQNHdf2chuKj1M5k=
 github.com/cockroachdb/logtags v0.0.0-20241215232642-bb51bb14a506/go.mod h1:Mw7HqKr2kdtu6aYGn3tPmAftiP3QPX63LdK/zcariIo=
 github.com/cockroachdb/redact v1.1.5 h1:u1PMllDkdFfPWaNGMyLD1+so+aq3uUItthCFqzwPJ30=
@@ -956,8 +956,8 @@ github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 h1:Oy0F4A
 github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3/go.mod h1:YvSRo5mw33fLEx1+DlK6L2VV43tJt5Eyel9n9XBcR+0=
 github.com/eapache/queue v1.1.0 h1:YOEu7KNc61ntiQlcEeUIoDTJ2o8mQznoNvUhiigpIqc=
 github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
-github.com/ebitengine/purego v0.10.2 h1:W809HbnvzAxgdm+aOvlSekrM16wGCdT/e76+9tS7gzE=
-github.com/ebitengine/purego v0.10.2/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
+github.com/ebitengine/purego v0.10.1 h1:dewVBCBT2GaMu1SrNTYxQhgQBethzfhiwvZiLGP/qyY=
+github.com/ebitengine/purego v0.10.1/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
 github.com/eiannone/keyboard v0.0.0-20220611211555-0d226195f203 h1:XBBHcIb256gUJtLmY22n99HaZTz+r2Z51xUPi01m3wg=
 github.com/eiannone/keyboard v0.0.0-20220611211555-0d226195f203/go.mod h1:E1jcSv8FaEny+OP/5k9UxZVw9YFWGj7eI4KR/iOBqCg=
 github.com/elastic/gosigar v0.14.3 h1:xwkKwPia+hSfg9GqrCUKYdId102m9qTJIIr7egmK/uo=
@@ -1031,8 +1031,8 @@ github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9
 github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
 github.com/geoffgarside/ber v1.2.0 h1:/loowoRcs/MWLYmGX9QtIAbA+V/FrnVLsMMPhwiRm64=
 github.com/geoffgarside/ber v1.2.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNecGT85ZCc=
-github.com/getsentry/sentry-go v0.48.0 h1:FRZNr7Uk1C86ev1bSJmYlUkL9oyivQA6YOcdYfaaMmY=
-github.com/getsentry/sentry-go v0.48.0/go.mod h1:E5UkA5wp1qR2+MDydNYlVeUiNN2xEdjYMidkgf0Qoss=
+github.com/getsentry/sentry-go v0.44.1 h1:/cPtrA5qB7uMRrhgSn9TYtcEF36auGP3Y6+ThvD/yaI=
+github.com/getsentry/sentry-go v0.44.1/go.mod h1:XDotiNZbgf5U8bPDUAfvcFmOnMQQceESxyKaObSssW0=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/gin-contrib/sse v1.1.0 h1:n0w2GMuUpWDVp7qSpvze6fAu9iRxJY4Hmj6AmBOU05w=
 github.com/gin-contrib/sse v1.1.0/go.mod h1:hxRZ5gVpWMT7Z0B0gSNYqqsSCNIJMjzvm6fqCz9vjwM=
@@ -1267,8 +1267,8 @@ github.com/googleapis/gax-go/v2 v2.4.0/go.mod h1:XOTVJ59hdnfJLIP/dh8n5CGryZR2LxK
 github.com/googleapis/gax-go/v2 v2.5.1/go.mod h1:h6B0KMMFNtI2ddbGJn3T3ZbwkeT6yqEF02fYlzkUCyo=
 github.com/googleapis/gax-go/v2 v2.6.0/go.mod h1:1mjbznJAPHFpesgE5ucqfYEscaz5kMdcIDwU/6+DDoY=
 github.com/googleapis/gax-go/v2 v2.7.0/go.mod h1:TEop28CZZQ2y+c0VxMUmu1lV+fQx57QpBWsYpwqHJx8=
-github.com/googleapis/gax-go/v2 v2.24.0 h1:myMaPYyF9MecEmvQqMqomIwn9t/4KCZN9qnwsS76wlg=
-github.com/googleapis/gax-go/v2 v2.24.0/go.mod h1:IaTHBDd7NHxSCiu0vEs8pQZu4dGZrWwuSoxCnk16OFM=
+github.com/googleapis/gax-go/v2 v2.23.0 h1:Tchl7qkvE7Ip3y+ztvNufYFvkfqTe7NfLTYGIdJRLuE=
+github.com/googleapis/gax-go/v2 v2.23.0/go.mod h1:rBQKOVJCdb8IFEzg+FCwlt1LP/xMDGuqUXhUG+XMXEg=
 github.com/googleapis/go-type-adapters v1.0.0/go.mod h1:zHW75FOG2aur7gAO2B+MLby+cLsWGBF62rFAi7WjWO4=
 github.com/googleapis/google-cloud-go-testing v0.0.0-20200911160855-bcd43fbb19e8/go.mod h1:dvDLG8qkwmyD9a/MJJN3XJcT3xFxOKAvTZGvuZmac9g=
 github.com/gookit/assert v0.1.1 h1:lh3GcawXe/p+cU7ESTZ5Ui3Sm/x8JWpIis4/1aF0mY0=
@@ -1543,8 +1543,8 @@ github.com/moby/buildkit v0.29.0 h1:wxLEFbCOJntEDjSNNN2YWd8zxltZxT5muDQ0LzpbtpU=
 github.com/moby/buildkit v0.29.0/go.mod h1:Dmv2FeDe34t75QuzeU87rBoZpAAkcpT5zeu4hXzmASc=
 github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
 github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
-github.com/moby/go-archive v0.3.0 h1:nos4BtzzUIqB406BgQnWGMI4qib9BZ8XUHU+ucv/n1c=
-github.com/moby/go-archive v0.3.0/go.mod h1:Npdv43fFqlhZW7Xo8fbm3ZMYFvAGNviUPqX21VERbcE=
+github.com/moby/go-archive v0.2.0 h1:zg5QDUM2mi0JIM9fdQZWC7U8+2ZfixfTYoHL7rWUcP8=
+github.com/moby/go-archive v0.2.0/go.mod h1:mNeivT14o8xU+5q1YnNrkQVpK+dnNe/K6fHqnTg4qPU=
 github.com/moby/locker v1.0.1 h1:fOXqR41zeveg4fFODix+1Ch4mj/gT0NE1XJbp/epuBg=
 github.com/moby/locker v1.0.1/go.mod h1:S7SDdo5zpBK84bzzVlKr2V0hz+7x9hWbYC/kq7oQppc=
 github.com/moby/moby/api v1.54.2 h1:wiat9QAhnDQjA7wk1kh/TqHz2I1uUA7M7t9SAl/JNXg=
@@ -1559,14 +1559,14 @@ github.com/moby/sys/capability v0.4.0 h1:4D4mI6KlNtWMCM1Z/K0i7RV1FkX+DBDHKVJpCnd
 github.com/moby/sys/capability v0.4.0/go.mod h1:4g9IK291rVkms3LKCDOoYlnV8xKwoDTpIrNEE35Wq0I=
 github.com/moby/sys/mountinfo v0.7.2 h1:1shs6aH5s4o5H2zQLn796ADW1wMrIwHsyJ2v9KouLrg=
 github.com/moby/sys/mountinfo v0.7.2/go.mod h1:1YOa8w8Ih7uW0wALDUgT1dTTSBrZ+HiBLGws92L2RU4=
-github.com/moby/sys/sequential v0.7.0 h1:ASQNGNROJSuOO6LL6bPHbKvuZu6NU8P4ldPWk31zj/8=
-github.com/moby/sys/sequential v0.7.0/go.mod h1:NfSTAp6V3fw4tmkD62PEcOKeZKquXT8VKCkf7aVR79o=
+github.com/moby/sys/sequential v0.6.0 h1:qrx7XFUd/5DxtqcoH1h438hF5TmOvzC/lspjy7zgvCU=
+github.com/moby/sys/sequential v0.6.0/go.mod h1:uyv8EUTrca5PnDsdMGXhZe6CCe8U/UiTWd+lL+7b/Ko=
 github.com/moby/sys/signal v0.7.1 h1:PrQxdvxcGijdo6UXXo/lU/TvHUWyPhj7UOpSo8tuvk0=
 github.com/moby/sys/signal v0.7.1/go.mod h1:Se1VGehYokAkrSQwL4tDzHvETwUZlnY7S5XtQ50mQp8=
 github.com/moby/sys/symlink v0.3.0 h1:GZX89mEZ9u53f97npBy4Rc3vJKj7JBDj/PN2I22GrNU=
 github.com/moby/sys/symlink v0.3.0/go.mod h1:3eNdhduHmYPcgsJtZXW1W4XUJdZGBIkttZ8xKqPUJq0=
-github.com/moby/sys/user v0.4.1 h1:RgjRlaDKi/Xmyrz4t8lyzXT6v2ooFeO/7xtchmhVWE0=
-github.com/moby/sys/user v0.4.1/go.mod h1:E9QsW5WRe1kUAf7kW8hXKwu1uhsZEAdPLYHYSDudF4Y=
+github.com/moby/sys/user v0.4.0 h1:jhcMKit7SA80hivmFJcbB1vqmw//wU61Zdui2eQXuMs=
+github.com/moby/sys/user v0.4.0/go.mod h1:bG+tYYYJgaMtRKgEmuueC0hJEAZWwtIbZTB+85uoHjs=
 github.com/moby/sys/userns v0.1.0 h1:tVLXkFOxVu9A64/yh59slHVv9ahO9UIev4JZusOLG/g=
 github.com/moby/sys/userns v0.1.0/go.mod h1:IHUYgu/kao6N8YZlp9Cf444ySSvCmDlmzUcYfDHOl28=
 github.com/moby/term v0.5.2 h1:6qk3FJAFDs6i/q3W/pQ97SX192qKfZgGjCQqfCJkgzQ=
@@ -1636,8 +1636,8 @@ github.com/parquet-go/bitpack v1.0.0 h1:AUqzlKzPPXf2bCdjfj4sTeacrUwsT7NlcYDMUQxP
 github.com/parquet-go/bitpack v1.0.0/go.mod h1:XnVk9TH+O40eOOmvpAVZ7K2ocQFrQwysLMnc6M/8lgs=
 github.com/parquet-go/jsonlite v1.0.0 h1:87QNdi56wOfsE5bdgas0vRzHPxfJgzrXGml1zZdd7VU=
 github.com/parquet-go/jsonlite v1.0.0/go.mod h1:nDjpkpL4EOtqs6NQugUsi0Rleq9sW/OtC1NnZEnxzF0=
-github.com/parquet-go/parquet-go v0.32.0 h1:NWDqTUHfrCS4cJP/Fj2HlxvqsrVedWG3sayMkf+znzM=
-github.com/parquet-go/parquet-go v0.32.0/go.mod h1:navtkAYr2LGoJVp141oXPlO/sxLvaOe3la2JEoD8+rg=
+github.com/parquet-go/parquet-go v0.30.1 h1:Oy6ganNrAdFiVwy7wNmWagfPTWA2X9Z3tVHBc7JtuX8=
+github.com/parquet-go/parquet-go v0.30.1/go.mod h1:navtkAYr2LGoJVp141oXPlO/sxLvaOe3la2JEoD8+rg=
 github.com/pascaldekloe/goe v0.1.0 h1:cBOtyMzM9HTpWjXfbbunk26uA6nG3a8n06Wieeh0MwY=
 github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
 github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
@@ -1749,8 +1749,8 @@ github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
 github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
 github.com/quic-go/quic-go v0.59.0 h1:OLJkp1Mlm/aS7dpKgTc6cnpynnD2Xg7C1pwL6vy/SAw=
 github.com/quic-go/quic-go v0.59.0/go.mod h1:upnsH4Ju1YkqpLXC305eW3yDZ4NfnNbmQRCMWS58IKU=
-github.com/rabbitmq/amqp091-go v1.14.0 h1:RSaT7aOKt/OrkVUyswPDW29lnRz9psuGmfZFBmLqLek=
-github.com/rabbitmq/amqp091-go v1.14.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
+github.com/rabbitmq/amqp091-go v1.13.0 h1:L8NA1WtF76C6KA3LAoufjfLgbist/If1UQYcsOjtxXA=
+github.com/rabbitmq/amqp091-go v1.13.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
 github.com/rclone/Proton-API-Bridge v1.0.4 h1:uGQJRjQC1hVLd5kqLsXc6CWO6oqrVeLoKQYoHapEZDg=
 github.com/rclone/Proton-API-Bridge v1.0.4/go.mod h1:VTPBYZotKAeDLlAzxU2O/s14NXk9FxUt9hn1jhH2iY8=
 github.com/rclone/go-proton-api v1.0.3 h1:3gBTzR+j0dYiTwtj9yKIdN/aV3W2a8KIPKp0GArojyQ=
@@ -1790,8 +1790,6 @@ github.com/rs/zerolog v1.34.0 h1:k43nTLIwcTVQAncfCw4KZ2VY6ukYoZaBPNOE8txlOeY=
 github.com/rs/zerolog v1.34.0/go.mod h1:bJsvje4Z08ROH4Nhs5iH600c3IkWhwp44iRc54W6wYQ=
 github.com/ruudk/golang-pdf417 v0.0.0-20181029194003-1af4ab5afa58/go.mod h1:6lfFZQK844Gfx8o5WFuvpxWRwnSoipWe/p622j1v06w=
 github.com/ruudk/golang-pdf417 v0.0.0-20201230142125-a7e3863a1245/go.mod h1:pQAZKsJ8yyVxGRWYNEm9oFB8ieLgKFnamEyDmSA0BRk=
-github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd h1:CmH9+J6ZSsIjUK3dcGsnCnO41eRBOnY12zwkn5qVwgc=
-github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd/go.mod h1:hPqNNc0+uJM6H+SuU8sEs5K5IQeKccPqeSjfgcKGgPk=
 github.com/ryanuber/go-glob v1.0.0 h1:iQh3xXAumdQ+4Ufa5b25cRpC5TYKlno6hsv6Cb3pkBk=
 github.com/ryanuber/go-glob v1.0.0/go.mod h1:807d1WSdnB0XRJzKNil9Om6lcp/3a0v4qIHxIXzX/Yc=
 github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 h1:OkMGxebDjyw0ULyrTYWeN0UNCCkmCWfjPnIA2W6oviI=
@@ -1810,8 +1808,8 @@ github.com/seaweedfs/cockroachdb-parser v0.0.0-20260225204133-2f342c5ea564 h1:Tg
 github.com/seaweedfs/cockroachdb-parser v0.0.0-20260225204133-2f342c5ea564/go.mod h1:JSKCh6uCHBz91lQYFYHCyTrSVIPge4SUFVn28iwMNB0=
 github.com/seaweedfs/go-fuse/v2 v2.9.4 h1:ACyloiuopdhRSjdLLeSWbsVaemMPskORaRF01TY6GyM=
 github.com/seaweedfs/go-fuse/v2 v2.9.4/go.mod h1:zABdmWEa6A0bwaBeEOBUeUkGIZlxUhcdv+V1Dcc/U/I=
-github.com/seaweedfs/goexif v2.0.0+incompatible h1:x8pckiT12QQhifwhDQpeISgDfsqmQ6VR4LFPQ64JRps=
-github.com/seaweedfs/goexif v2.0.0+incompatible/go.mod h1:Oni780Z236sXpIQzk1XoJlTwqrJ02smEin9zQeff7Fk=
+github.com/seaweedfs/goexif v1.0.3 h1:ve/OjI7dxPW8X9YQsv3JuVMaxEyF9Rvfd04ouL+Bz30=
+github.com/seaweedfs/goexif v1.0.3/go.mod h1:Oni780Z236sXpIQzk1XoJlTwqrJ02smEin9zQeff7Fk=
 github.com/seaweedfs/raft v1.2.0 h1:Ez4Hw9ifBbTT7wg54DvGHBjw1vRlTb4roH0TKl0Oj9Y=
 github.com/seaweedfs/raft v1.2.0/go.mod h1:fgs/rAVEzjQ7e04XMzG3eJhwZZRmBW+2uRtjakeCGeU=
 github.com/secure-systems-lab/go-securesystemslib v0.10.0 h1:l+H5ErcW0PAehBNrBxoGv1jjNpGYdZ9RcheFkB2WI14=
@@ -1822,8 +1820,8 @@ github.com/sergi/go-diff v1.2.0 h1:XU+rvMAioB0UC3q1MFrIQy4Vo5/4VsRDQQXHsEya6xQ=
 github.com/sergi/go-diff v1.2.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
 github.com/shibumi/go-pathspec v1.3.0 h1:QUyMZhFo0Md5B8zV8x2tesohbb5kfbpTi9rBnKh5dkI=
 github.com/shibumi/go-pathspec v1.3.0/go.mod h1:Xutfslp817l2I1cZvgcfeMQJG5QnU2lh5tVaaMCl3jE=
-github.com/shirou/gopsutil/v4 v4.26.7 h1:IXzpHz/dkMRYAhKkOXr1HB6SuzWU3eoyyeWe7g3bNZc=
-github.com/shirou/gopsutil/v4 v4.26.7/go.mod h1:5O9FjBiXoTDFatIWjZZosqj4pV0DRtLx598xGbBehzM=
+github.com/shirou/gopsutil/v4 v4.26.6 h1:Mzr/npDtQC/xpeEuQKHZt8Zo9CmPvhTj8nkR8w5TLDs=
+github.com/shirou/gopsutil/v4 v4.26.6/go.mod h1:LZ6ewCSkBqUpvSOf+LsTGnRinC6iaNUNMGBtDkJBaLQ=
 github.com/sigstore/sigstore v1.10.4 h1:ytOmxMgLdcUed3w1SbbZOgcxqwMG61lh1TmZLN+WeZE=
 github.com/sigstore/sigstore v1.10.4/go.mod h1:tDiyrdOref3q6qJxm2G+JHghqfmvifB7hw+EReAfnbI=
 github.com/sigstore/sigstore-go v1.1.4 h1:wTTsgCHOfqiEzVyBYA6mDczGtBkN7cM8mPpjJj5QvMg=
@@ -1909,8 +1907,8 @@ github.com/tarantool/go-iproto v1.1.0 h1:HULVOIHsiehI+FnHfM7wMDntuzUddO09DKqu2Wn
 github.com/tarantool/go-iproto v1.1.0/go.mod h1:LNCtdyZxojUed8SbOiYHoc3v9NvaZTB7p96hUySMlIo=
 github.com/tarantool/go-option v1.1.0 h1:ShoOhNsdL41sRpm4hXCRDjV8H0WzPkd4UnKhLKbW//w=
 github.com/tarantool/go-option v1.1.0/go.mod h1:hMr9z2JXOWlgdCBpCPSL2nwp8718GKYvNBJ+ZuzJbCo=
-github.com/tarantool/go-tarantool/v3 v3.0.1 h1:vaUX4xmVmXh2dIJ/LqlX1MXK3iYqAqV6YiE54Wwl/qg=
-github.com/tarantool/go-tarantool/v3 v3.0.1/go.mod h1:TXxLWhUCgdxXFfelnTSkq+goKRTRj660zxq4/WXPe8k=
+github.com/tarantool/go-tarantool/v3 v3.0.0 h1:zsIXS4nvSSXqZWtN/f1Bfuq973j6J85O5U/8RYQCRcE=
+github.com/tarantool/go-tarantool/v3 v3.0.0/go.mod h1:TXxLWhUCgdxXFfelnTSkq+goKRTRj660zxq4/WXPe8k=
 github.com/testcontainers/testcontainers-go v0.43.0 h1:oEQx5MW2DGd9z3AeEQfB2lPM0eLs7ztyaGRu75bFo5A=
 github.com/testcontainers/testcontainers-go v0.43.0/go.mod h1:+VxkT2NQnKOZPKi6praMuMKYHYyOGXr0XSBSlSMCzFo=
 github.com/testcontainers/testcontainers-go/modules/compose v0.42.0 h1:+t1ZN31TD36cwxmeLqGwe7wIdvblBm0Z+vlj4SX8Mv0=
@@ -2032,14 +2030,14 @@ github.com/yandex-cloud/go-genproto v0.0.0-20211115083454-9ca41db5ed9e h1:9LPdmD
 github.com/yandex-cloud/go-genproto v0.0.0-20211115083454-9ca41db5ed9e/go.mod h1:HEUYX/p8966tMUHHT+TsS0hF/Ca/NYwqprC5WXSDMfE=
 github.com/ydb-platform/ydb-go-genproto v0.0.0-20221215182650-986f9d10542f/go.mod h1:Er+FePu1dNUieD+XTMDduGpQuCPssK5Q4BjF+IIXJ3I=
 github.com/ydb-platform/ydb-go-genproto v0.0.0-20230528143953-42c825ace222/go.mod h1:Er+FePu1dNUieD+XTMDduGpQuCPssK5Q4BjF+IIXJ3I=
-github.com/ydb-platform/ydb-go-genproto v0.0.0-20260810122915-65bfd5c4b705 h1:7VKlOrBIQ8L8acJ9wFCt6lWzLDbOhc05VLpL31743tU=
-github.com/ydb-platform/ydb-go-genproto v0.0.0-20260810122915-65bfd5c4b705/go.mod h1:Er+FePu1dNUieD+XTMDduGpQuCPssK5Q4BjF+IIXJ3I=
+github.com/ydb-platform/ydb-go-genproto v0.0.0-20260428144813-1c07baab7f7b h1:xeiobG1riqe6dTMZuTcOvwOY0BbFidSk3gRLyVeLfJA=
+github.com/ydb-platform/ydb-go-genproto v0.0.0-20260428144813-1c07baab7f7b/go.mod h1:Er+FePu1dNUieD+XTMDduGpQuCPssK5Q4BjF+IIXJ3I=
 github.com/ydb-platform/ydb-go-sdk-auth-environ v0.5.2 h1:e2nGQPGC5OEPBWlMnLPFpdgyqNA8iTOPiZShCrv6944=
 github.com/ydb-platform/ydb-go-sdk-auth-environ v0.5.2/go.mod h1:9YzkhlIymWaJGX6KMU3vh5sOf3UKbCXkG/ZdjaI3zNM=
 github.com/ydb-platform/ydb-go-sdk/v3 v3.44.0/go.mod h1:oSLwnuilwIpaF5bJJMAofnGgzPJusoI3zWMNb8I+GnM=
 github.com/ydb-platform/ydb-go-sdk/v3 v3.47.3/go.mod h1:bWnOIcUHd7+Sl7DN+yhyY1H/I61z53GczvwJgXMgvj0=
-github.com/ydb-platform/ydb-go-sdk/v3 v3.151.1 h1:T+fB2ZDHpYIGC7DWjK+rzZHLoexBF0zG/n3Q9DGNskY=
-github.com/ydb-platform/ydb-go-sdk/v3 v3.151.1/go.mod h1:dJXJ1u00IqO8Vsph8fWmYj8b1E7jyphnvJPqA69emBY=
+github.com/ydb-platform/ydb-go-sdk/v3 v3.147.1 h1:WRxyl1UdFD6EmFlujwrHm4sX9ktYb/adZ/SW+WWNOEg=
+github.com/ydb-platform/ydb-go-sdk/v3 v3.147.1/go.mod h1:b9NEO6mgaiqsnOMkS003uS82XsKh6GL+ZTFfPqXWz+c=
 github.com/ydb-platform/ydb-go-yc v0.12.1 h1:qw3Fa+T81+Kpu5Io2vYHJOwcrYrVjgJlT6t/0dOXJrA=
 github.com/ydb-platform/ydb-go-yc v0.12.1/go.mod h1:t/ZA4ECdgPWjAb4jyDe8AzQZB5dhpGbi3iCahFaNwBY=
 github.com/ydb-platform/ydb-go-yc-metadata v0.6.1 h1:9E5q8Nsy2RiJMZDNVy0A3KUrIMBPakJ2VgloeWbcI84=
@@ -2659,8 +2657,8 @@ google.golang.org/api v0.106.0/go.mod h1:2Ts0XTHNVWxypznxWOYUeI4g3WdP9Pk2Qk58+a/
 google.golang.org/api v0.107.0/go.mod h1:2Ts0XTHNVWxypznxWOYUeI4g3WdP9Pk2Qk58+a/O9MY=
 google.golang.org/api v0.108.0/go.mod h1:2Ts0XTHNVWxypznxWOYUeI4g3WdP9Pk2Qk58+a/O9MY=
 google.golang.org/api v0.110.0/go.mod h1:7FC4Vvx1Mooxh8C5HWjzZHcavuS2f6pmJpZx60ca7iI=
-google.golang.org/api v0.294.0 h1:8gASjJxdtcIieB3OqbkLcF0FfbXVNqKtU5iozD1ssvA=
-google.golang.org/api v0.294.0/go.mod h1:02qB8+Ox1ZFzcaKFMguy1nQLJmSIyvV6Ff4txJEXtl4=
+google.golang.org/api v0.293.0 h1:p9XIWOf63U4OgYx120ZwVU8+vl4XTPmWfgVPnmOAS9w=
+google.golang.org/api v0.293.0/go.mod h1:6n5tjEB1gzwniZTepZ0g5u+wM7Bof5GeULCx/zh8ZE0=
 google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
 google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
@@ -2794,12 +2792,12 @@ google.golang.org/genproto v0.0.0-20230209215440-0dfe4f8abfcc/go.mod h1:RGgjbofJ
 google.golang.org/genproto v0.0.0-20230216225411-c8e22ba71e44/go.mod h1:8B0gmkoRebU8ukX6HP+4wrVQUY1+6PkQ44BSyIlflHA=
 google.golang.org/genproto v0.0.0-20230222225845-10f96fb3dbec/go.mod h1:3Dl5ZL0q0isWJt+FVcfpQyirqemEuLAK/iFvg1UP1Hw=
 google.golang.org/genproto v0.0.0-20230306155012-7f2fa6fef1f4/go.mod h1:NWraEVixdDnqcqQ30jipen1STv2r/n24Wb7twVTGR4s=
-google.golang.org/genproto v0.0.0-20260715232425-e75dac1f907d h1:C9v1o0/4quuhOAfmRXA2j+we0PqZIp8traLdeogF3Ms=
-google.golang.org/genproto v0.0.0-20260715232425-e75dac1f907d/go.mod h1:Wz2wFJntZFmLGo7pLDXZ3wYk5hyc0Mb+SkHhDDXT+lU=
-google.golang.org/genproto/googleapis/api v0.0.0-20260715232425-e75dac1f907d h1:QwnJwPte4XXAkhPu26LTDIahnsMSUV0kK8HkxbC+Pc4=
-google.golang.org/genproto/googleapis/api v0.0.0-20260715232425-e75dac1f907d/go.mod h1:WRrQ7/7N19PypuT0fxLOL5Lq0waoiRri4FbtHDEKrGE=
-google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 h1:cYNAzI2sUwhmCcoj9TxvihSrqsxt6uIkj3rDRhSDmW4=
-google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA=
+google.golang.org/genproto v0.0.0-20260519071638-aa98bba5eb94 h1:YJjbgu+dkp5kUJLfpMyCLfBIWZb/FcJyuLeo1gVBOuo=
+google.golang.org/genproto v0.0.0-20260519071638-aa98bba5eb94/go.mod h1:RRHjglSYABVCWpQ7USCpdfhcd9t4PkajvVwyynZizTc=
+google.golang.org/genproto/googleapis/api v0.0.0-20260706201446-f0a921348800 h1:admdQBe8jR3VWhBsUrAOaF2Qw6K/+p5pSm1GN8+6Fw4=
+google.golang.org/genproto/googleapis/api v0.0.0-20260706201446-f0a921348800/go.mod h1:FPk7EXUKMtImne7AmknoYjT4QXqKIzzRbeQIXzLk6fQ=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260807164820-c8921c73eeea h1:kVhQEPTpKQahD5+JSBTfBB19wcgQTTjAIn45MBqnyHk=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260807164820-c8921c73eeea/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
 google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
 google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
@@ -2840,8 +2838,8 @@ google.golang.org/grpc v1.51.0/go.mod h1:wgNDFcnuBGmxLKI/qn4T+m5BtEBYXJPvibbUPsA
 google.golang.org/grpc v1.52.0/go.mod h1:pu6fVzoFb+NBYNAvQL08ic+lvB2IojljRYuun5vorUY=
 google.golang.org/grpc v1.53.0/go.mod h1:OnIrk0ipVdj4N5d9IUoFUx72/VlD7+jUsHwZgwSMQpw=
 google.golang.org/grpc v1.55.0/go.mod h1:iYEXKGkEBhg1PjZQvoYEVPTDkHo1/bjTnfwTeGONTY8=
-google.golang.org/grpc v1.85.0-dev h1:HxkDyKIIZPpFnroC56tQv5gNuKTmVvi0t7TzOf5zt7g=
-google.golang.org/grpc v1.85.0-dev/go.mod h1:ljCht0DrxQrXBDRTZp52Qxh3Ffk8CdYm2sj4O2QN2C0=
+google.golang.org/grpc v1.84.0-dev.0.20260723093437-b6eac429d7b6 h1:HfjjkdGIa8u9sP9EW5WCygy0kQDuTI/Tax4j//t24Fo=
+google.golang.org/grpc v1.84.0-dev.0.20260723093437-b6eac429d7b6/go.mod h1:ljCht0DrxQrXBDRTZp52Qxh3Ffk8CdYm2sj4O2QN2C0=
 google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0/go.mod h1:6Kw0yEErY5E/yWrBtf03jp27GLLJujG4z/JK95pnjjw=
 google.golang.org/grpc/examples v0.0.0-20250407062114-b368379ef8f6 h1:ExN12ndbJ608cboPYflpTny6mXSzPrDLh0iTaVrRrds=
 google.golang.org/grpc/examples v0.0.0-20250407062114-b368379ef8f6/go.mod h1:6ytKWczdvnpnO+m+JiG9NjEDzR1FJfsnmJdG7B8QVZ8=
@@ -2863,8 +2861,8 @@ google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQ
 google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
 google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
 google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
-google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
-google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
+google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
+google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@@ -2964,8 +2962,8 @@ modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
 modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
 modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
 modernc.org/sqlite v1.18.1/go.mod h1:6ho+Gow7oX5V+OiOQ6Tr4xeqbx13UZ6t+Fw9IRUG4d4=
-modernc.org/sqlite v1.57.0 h1:qNQP6xnx5M0ISNtlnxoOX0+cD5bJ0/gr9aMmndFczzg=
-modernc.org/sqlite v1.57.0/go.mod h1:yCJ2cmAaIkHQ25oXWrF8H4O1lIfPYPR26yCEDj2P3pQ=
+modernc.org/sqlite v1.56.0 h1:/D8e2RfFqoy/Zc6PuC76U28zFwmI/sYx1Kjm4yEn9e0=
+modernc.org/sqlite v1.56.0/go.mod h1:yCJ2cmAaIkHQ25oXWrF8H4O1lIfPYPR26yCEDj2P3pQ=
 modernc.org/strutil v1.1.0/go.mod h1:lstksw84oURvj9y3tn8lGvRxyRC1S2+g5uuIzNfIOBs=
 modernc.org/strutil v1.1.1/go.mod h1:DE+MQQ/hjKBZS2zNInV5hhcipt5rLPWkmpbGeW5mmdw=
 modernc.org/strutil v1.1.3/go.mod h1:MEHNA7PdEnEwLvspRMtWTNnp2nnyvMfkimT1NKNAGbw=

install.sh

@@ -9,7 +9,7 @@
 #   curl -fsSL ... | bash -s -- --version 4.34 --dir /usr/local/bin
 #
 # Options:
-#   --component COMP   Which binary to install: weed, volume-rust, worker-rust, all (default: weed)
+#   --component COMP   Which binary to install: weed, volume-rust, all (default: weed)
 #   --version VER      Release version tag (default: latest)
 #   --large-disk       Use large disk variant (5-byte offset, 8TB max volume)
 #   --dir DIR          Installation directory (default: /usr/local/bin)
@@ -22,7 +22,6 @@ COMPONENT="weed"
 VERSION=""
 LARGE_DISK=false
 INSTALL_DIR="/usr/local/bin"
-WORKER_INSTALLED=false
 
 # Colors (if terminal supports them)
 if [ -t 1 ]; then
@@ -129,33 +128,6 @@ rust_asset_name() {
     fi
 }
 
-# Does a release carry this asset? Answers 0 for yes and 1 for a 404, and stops
-# the installer on anything else: a rate limit or a network blip must not read
-# as "this release predates the binary" and quietly skip it.
-asset_exists() {
-    local url="$1" code=""
-    if command -v curl &>/dev/null; then
-        code="$(curl -sL -o /dev/null -I -w '%{http_code}' "$url" || true)"
-    elif command -v wget &>/dev/null; then
-        # --spider's exit status folds 404 in with every other server error, so
-        # read the status line itself; -S prints one per redirect hop.
-        code="$(wget -S --spider -q -O /dev/null "$url" 2>&1 | awk '/^ *HTTP\// {c=$2} END {print c}')"
-    fi
-
-    case "$code" in
-        200) return 0 ;;
-        404) return 1 ;;
-        *)   error "Could not check ${url} (HTTP ${code:-none}). Retry, or install components one at a time." ;;
-    esac
-}
-
-# Build Rust maintenance worker asset name. No large-disk variant: the worker
-# maintains tables through the namespace and never opens a volume file.
-worker_asset_name() {
-    local os="$1" arch="$2"
-    echo "weed-worker_${os}_${arch}.tar.gz"
-}
-
 # Install a single component
 install_component() {
     local component="$1" os="$2" arch="$3"
@@ -232,48 +204,10 @@ install_component() {
             ok "Installed weed-volume to ${INSTALL_DIR}/${dest_name}"
             ;;
 
-        worker-rust)
-            # Published for linux only: the worker runs beside the cluster it
-            # maintains, and its dependency tree makes every extra target an
-            # expensive build.
-            case "$os" in
-                linux) ;;
-                *) error "Rust maintenance worker is not available for ${os}. Supported: linux" ;;
-            esac
-            case "$arch" in
-                amd64|arm64) ;;
-                *) error "Rust maintenance worker is not available for ${arch}. Supported: amd64, arm64" ;;
-            esac
-
-            asset_name="$(worker_asset_name "$os" "$arch")"
-            download_url="https://github.com/${REPO}/releases/download/${VERSION}/${asset_name}"
-            download "$download_url" "${tmpdir}/${asset_name}"
-
-            info "Extracting ${asset_name}..."
-            tar xzf "${tmpdir}/${asset_name}" -C "$tmpdir"
-
-            local worker_bin
-            worker_bin="$(find "$tmpdir" -name 'weed-worker' -type f | head -1)"
-            if [ -z "$worker_bin" ]; then
-                error "Could not find weed-worker binary in archive"
-            fi
-
-            chmod +x "$worker_bin"
-            install_binary "$worker_bin" "weed-worker"
-            WORKER_INSTALLED=true
-            ok "Installed weed-worker to ${INSTALL_DIR}/weed-worker"
-            ;;
-
         *)
-            error "Unknown component: ${component}. Use: weed, volume-rust, worker-rust, all"
+            error "Unknown component: ${component}. Use: weed, volume-rust, all"
             ;;
     esac
-
-    # The trap is per-process, so a later component's would replace this one and
-    # leave the earlier extraction behind. Clean up here and hand the trap back;
-    # the error paths above exit, which still fires it.
-    rm -rf "$tmpdir"
-    trap - EXIT
 }
 
 # Copy binary to install dir, using sudo if needed
@@ -317,16 +251,6 @@ main() {
         all)
             install_component "weed" "$os" "$arch"
             install_component "volume-rust" "$os" "$arch"
-            # The worker is published for linux amd64/arm64 only, and only by
-            # releases new enough to carry it; skip either case rather than fail
-            # an install that has already put two binaries in place.
-            if [ "$os" != "linux" ] || { [ "$arch" != "amd64" ] && [ "$arch" != "arm64" ]; }; then
-                warn "Skipping the Rust maintenance worker: no build for ${os}/${arch}"
-            elif ! asset_exists "https://github.com/${REPO}/releases/download/${VERSION}/$(worker_asset_name "$os" "$arch")"; then
-                warn "Skipping the Rust maintenance worker: ${VERSION} does not carry one"
-            else
-                install_component "worker-rust" "$os" "$arch"
-            fi
             ;;
         *)
             install_component "$COMPONENT" "$os" "$arch"
@@ -341,17 +265,11 @@ main() {
     if [ "$COMPONENT" = "volume-rust" ] || [ "$COMPONENT" = "all" ]; then
         info "  weed-volume:  ${INSTALL_DIR}/weed-volume"
     fi
-    if [ "$WORKER_INSTALLED" = true ]; then
-        info "  weed-worker:     ${INSTALL_DIR}/weed-worker"
-    fi
     echo ""
     info "Quick start:"
     info "  weed master                          # Start master server"
     info "  weed volume -mserver=localhost:9333   # Start Go volume server"
     info "  weed-volume -mserver localhost:9333 # Start Rust volume server"
-    if [ "$WORKER_INSTALLED" = true ]; then
-        info "  weed-worker --admin localhost:23646  # Start the Rust maintenance worker"
-    fi
 }
 
 main

k8s/charts/seaweedfs/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: v1
 description: SeaweedFS
 name: seaweedfs
-appVersion: "4.44"
+appVersion: "4.42"
 # Dev note: Trigger a helm chart release by `git tag -a helm-<version>`
-version: 4.44.0
+version: 4.42.0

k8s/charts/seaweedfs/README.md

@@ -22,8 +22,8 @@ helm install --values=values.yaml seaweedfs seaweedfs/seaweedfs
 ## Info:
 * master/filer/volume are stateful sets with anti-affinity on the hostname,
 so your deployment will be spread/HA.
-* leveldb2 is the default filer backend; a mysql-compatible database (memsql, ...) enables HA (multiple filer instances) and the backup/HA it can provide.
-* with `filer.extraEnvironmentVars.WEED_MYSQL_ENABLED` set to `"true"`, mysql user/password are created in a k8s secret (default: `<release>-seaweedfs-db-secret`) and injected to the filer with ENV. On any other store neither the secret nor the `WEED_MYSQL_*` env, plain or secret-backed, is rendered.
+* chart is using memsql(mysql) as the filer backend to enable HA (multiple filer instances) and backup/HA memsql can provide.
+* mysql user/password are created in a k8s secret (default: `<release>-seaweedfs-db-secret`) and injected to the filer with ENV.
 * cert config exists and can be enabled, but not been tested, requires cert-manager to be installed.
 
 ## Prerequisites
@@ -289,32 +289,6 @@ stringData:
   seaweedfs_s3_config: '{"identities":[{"name":"anvAdmin","credentials":[{"accessKey":"snu8yoP6QAlY0ne4","secretKey":"PNzBcmeLNEdR0oviwm04NQAicOrDH1Km"}],"actions":["Admin","Read","Write"]},{"name":"anvReadOnly","credentials":[{"accessKey":"SCigFee6c5lbi04A","secretKey":"kgFhbT38R8WUYVtiFQ1OiSVOrYr3NKku"}],"actions":["Read"]}]}'
 ```
 
-#### Source S3 credentials from an existing Secret
-
-To keep the keys out of `values.yaml` while still letting the chart generate the
-identities file, point an identity at an existing Secret:
-
-```yaml
-s3:
-  enabled: true
-  enableAuth: true
-  credentials:
-    admin:
-      existingSecret: minio-root
-      accessKeyKey: root-user
-      secretKeyKey: root-password
-```
-
-`accessKeyKey` and `secretKeyKey` default to the chart's own key names
-(`admin_access_key_id`, `admin_secret_access_key`, and the `read_` pair). The
-generated `seaweedfs_s3_config` references the keys as `${SEAWEEDFS_S3_ADMIN_ACCESS_KEY_ID}`
-and the gateway resolves them from the environment, which the chart wires up
-from the Secret. Nothing is read from the cluster at render time, so
-`helm template`, `--dry-run` and an Argo CD diff all render what an install
-applies. Rotating a key in the Secret takes effect on the next pod restart, as
-with any other environment variable. The COSI driver parses the config itself
-and does not resolve these references.
-
 ## Admin Component
 
 The admin component provides a modern web-based administration interface for managing SeaweedFS clusters. It includes:

k8s/charts/seaweedfs/templates/admin/admin-service.yaml

@@ -21,7 +21,6 @@ spec:
   {{- $nodePorts = .Values.admin.service.nodePorts | default dict }}
   {{- end }}
   type: {{ $serviceType }}
-  {{- include "seaweedfs.service.loadBalancerFields" .Values.admin.service }}
   ports:
   - name: "http"
     port: {{ .Values.admin.port }}

k8s/charts/seaweedfs/templates/all-in-one/all-in-one-deployment.yaml

@@ -84,9 +84,6 @@ spec:
           imagePullPolicy: {{ default "IfNotPresent" .Values.global.seaweedfs.imagePullPolicy }}
           env:
             {{- include "seaweedfs.licenseEnv" . | nindent 12 }}
-            {{- if and .Values.allInOne.s3.enabled (or .Values.allInOne.s3.enableAuth .Values.s3.enableAuth .Values.filer.s3.enableAuth) }}
-            {{- include "seaweedfs.s3.credentialEnv" . | nindent 12 }}
-            {{- end }}
             {{- /* Determine default cluster alias and the corresponding env var keys to avoid conflicts */}}
             {{- $mergedExtraEnvironmentVars := dict }}
             {{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global.seaweedfs "component" .Values.allInOne "target" $mergedExtraEnvironmentVars) }}

k8s/charts/seaweedfs/templates/all-in-one/all-in-one-service.yml

@@ -21,7 +21,6 @@ spec:
   {{- $nodePorts = .Values.allInOne.service.nodePorts | default dict }}
   {{- end }}
   type: {{ $serviceType }}
-  {{- include "seaweedfs.service.loadBalancerFields" .Values.allInOne.service }}
   internalTrafficPolicy: {{ .Values.allInOne.service.internalTrafficPolicy | default "Cluster" }}
   {{- if and (semverCompare ">=1.31-0" .Capabilities.KubeVersion.GitVersion) .Values.allInOne.s3.trafficDistribution }}
   trafficDistribution: {{ include "seaweedfs.trafficDistribution" (dict "value" .Values.allInOne.s3.trafficDistribution "Capabilities" .Capabilities) }}

k8s/charts/seaweedfs/templates/filer/filer-statefulset.yaml

@@ -92,7 +92,6 @@ spec:
           image: {{ template "seaweedfs.filer.image" . }}
           imagePullPolicy: {{ default "IfNotPresent" .Values.global.seaweedfs.imagePullPolicy }}
           env:
-            {{- $mysqlEnabled := include "seaweedfs.filer.mysqlEnabled" . }}
             - name: POD_IP
               valueFrom:
                 fieldRef:
@@ -105,7 +104,6 @@ spec:
               valueFrom:
                 fieldRef:
                   fieldPath: metadata.namespace
-            {{- if $mysqlEnabled }}
             - name: WEED_MYSQL_USERNAME
               valueFrom:
                 secretKeyRef:
@@ -118,21 +116,10 @@ spec:
                   name: {{ include "seaweedfs.fullname" . }}-db-secret
                   key: password
                   optional: true
-            {{- end }}
             - name: SEAWEEDFS_FULLNAME
               value: "{{ include "seaweedfs.fullname" . }}"
-            {{- if and .Values.filer.s3.enabled .Values.filer.s3.enableAuth }}
-            {{- include "seaweedfs.s3.credentialEnv" . | nindent 12 }}
-            {{- end }}
             {{- $mergedExtraEnvironmentVars := dict }}
             {{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global.seaweedfs "component" .Values.filer "target" $mergedExtraEnvironmentVars) }}
-            {{- if not $mysqlEnabled }}
-            {{- range $key := keys $mergedExtraEnvironmentVars }}
-            {{- if hasPrefix "WEED_MYSQL_" $key }}
-            {{- $_ := unset $mergedExtraEnvironmentVars $key }}
-            {{- end }}
-            {{- end }}
-            {{- end }}
             {{- range $key := keys $mergedExtraEnvironmentVars | sortAlpha }}
             {{- $value := index $mergedExtraEnvironmentVars $key }}
             - name: {{ $key }}
@@ -145,12 +132,10 @@ spec:
             {{- end }}
             {{- if .Values.filer.secretExtraEnvironmentVars }}
             {{- range $key, $value := .Values.filer.secretExtraEnvironmentVars }}
-            {{- if or $mysqlEnabled (not (hasPrefix "WEED_MYSQL_" $key)) }}
             - name: {{ $key }}
               valueFrom: {{ toYaml $value | nindent 16 }}
             {{- end }}
             {{- end }}
-            {{- end }}
           command:
             - "/bin/sh"
             - "-ec"

k8s/charts/seaweedfs/templates/s3/s3-deployment.yaml

@@ -94,9 +94,6 @@ spec:
                   fieldPath: metadata.namespace
             - name: SEAWEEDFS_FULLNAME
               value: "{{ include "seaweedfs.fullname" . }}"
-            {{- if .Values.s3.enableAuth }}
-            {{- include "seaweedfs.s3.credentialEnv" . | nindent 12 }}
-            {{- end }}
             {{- $mergedExtraEnvironmentVars := dict }}
             {{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global.seaweedfs "component" .Values.s3 "target" $mergedExtraEnvironmentVars) }}
             {{- range $key := keys $mergedExtraEnvironmentVars | sortAlpha }}
@@ -149,8 +146,6 @@ spec:
               {{- if .Values.s3.icebergPort }}
               -port.iceberg={{ .Values.s3.icebergPort }} \
               {{- end }}
-              {{- /* rendered even when 0: an if would drop the flag and weed would serve its default */}}
-              -port.lance={{ .Values.s3.lancePort | default 0 }} \
               {{- range .Values.s3.extraArgs }}
               {{ . }} \
               {{- end }}
@@ -200,10 +195,6 @@ spec:
             - containerPort: {{ .Values.s3.icebergPort }}
               name: swfs-iceberg
             {{- end }}
-            {{- if .Values.s3.lancePort }}
-            - containerPort: {{ .Values.s3.lancePort }}
-              name: swfs-lance
-            {{- end }}
             {{- if .Values.s3.metricsPort }}
             - containerPort: {{ .Values.s3.metricsPort }}
               name: metrics

k8s/charts/seaweedfs/templates/s3/s3-lance-ingress.yaml

@@ -1,61 +0,0 @@
-{{- define "seaweedfs.s3.lance.ingress.paths" -}}
-paths:
-- path: {{ .Values.s3.lanceIngress.path | quote }}
-  pathType: {{ .Values.s3.lanceIngress.pathType | quote }}
-  backend:
-    {{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion }}
-    service:
-      name: {{ include "seaweedfs.componentName" (list . "s3") }}
-      port:
-        number: {{ .Values.s3.lancePort }}
-    {{- else }}
-    serviceName: {{ include "seaweedfs.componentName" (list . "s3") }}
-    servicePort: {{ .Values.s3.lancePort }}
-    {{- end }}
-{{- end -}}
-{{- if and .Values.s3.enabled .Values.s3.lancePort .Values.s3.lanceIngress.enabled }}
-{{- $hosts := list }}
-{{- if kindIs "slice" .Values.s3.lanceIngress.host }}
-  {{- $hosts = .Values.s3.lanceIngress.host }}
-{{- else if .Values.s3.lanceIngress.host }}
-  {{- $hosts = list .Values.s3.lanceIngress.host }}
-{{- end }}
-{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion }}
-apiVersion: networking.k8s.io/v1
-{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion }}
-apiVersion: networking.k8s.io/v1beta1
-{{- else }}
-apiVersion: extensions/v1beta1
-{{- end }}
-kind: Ingress
-metadata:
-  name: ingress-{{ include "seaweedfs.fullname" . }}-s3-lance
-  namespace: {{ .Release.Namespace }}
-  {{- with .Values.s3.lanceIngress.annotations }}
-  annotations:
-    {{- toYaml . | nindent 4 }}
-  {{- end }}
-  labels:
-    app.kubernetes.io/name: {{ template "seaweedfs.name" . }}
-    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
-    app.kubernetes.io/managed-by: {{ .Release.Service }}
-    app.kubernetes.io/instance: {{ .Release.Name }}
-    app.kubernetes.io/component: s3-lance
-spec:
-  {{- if .Values.s3.lanceIngress.className }}
-  ingressClassName: {{ .Values.s3.lanceIngress.className | quote }}
-  {{- end }}
-  tls:
-    {{ .Values.s3.lanceIngress.tls | default list | toYaml | nindent 6}}
-  rules:
-{{- if $hosts }}
-{{- range $host := $hosts }}
-  - host: {{ $host | quote }}
-    http:
-      {{- include "seaweedfs.s3.lance.ingress.paths" $ | nindent 6 }}
-{{- end }}
-{{- else }}
-  - http:
-      {{- include "seaweedfs.s3.lance.ingress.paths" . | nindent 4 }}
-{{- end }}
-{{- end }}

k8s/charts/seaweedfs/templates/s3/s3-secret.yaml

@@ -1,7 +1,4 @@
-{{- /* Mirrors the condition the all-in-one deployment mounts this secret under,
-       so the flags that make it mount are exactly the flags that create it. */}}
-{{- $allInOneAuth := and .Values.allInOne.enabled .Values.allInOne.s3.enabled (or .Values.allInOne.s3.enableAuth .Values.s3.enableAuth .Values.filer.s3.enableAuth) (not (or .Values.allInOne.s3.existingConfigSecret .Values.s3.existingConfigSecret .Values.filer.s3.existingConfigSecret)) }}
-{{- if or (and (or .Values.s3.enabled .Values.allInOne.enabled) .Values.s3.enableAuth (not .Values.s3.existingConfigSecret)) (and .Values.filer.s3.enabled .Values.filer.s3.enableAuth (not .Values.filer.s3.existingConfigSecret)) $allInOneAuth }}
+{{- if or (and (or .Values.s3.enabled .Values.allInOne.enabled) .Values.s3.enableAuth (not .Values.s3.existingConfigSecret)) (and .Values.filer.s3.enabled .Values.filer.s3.enableAuth (not .Values.filer.s3.existingConfigSecret)) }}
 {{- $secretName := printf "%s-s3-secret" (include "seaweedfs.fullname" .) }}
 {{- $legacySecretName := "seaweedfs-s3-secret" }}
 {{- $lookupName := $secretName }}
@@ -17,20 +14,14 @@
 {{- $adminCreds := $creds.admin | default dict -}}
 {{- $access_key_admin := $adminCreds.accessKey -}}
 {{- $secret_key_admin := $adminCreds.secretKey -}}
-{{- if $adminCreds.existingSecret -}}
-  {{- $access_key_admin = printf "${%s}" (include "seaweedfs.s3.credentialEnvName" (list "admin" "accessKey")) -}}
-  {{- $secret_key_admin = printf "${%s}" (include "seaweedfs.s3.credentialEnvName" (list "admin" "secretKey")) -}}
-{{- else if not (and $access_key_admin $secret_key_admin) -}}
+{{- if not (and $access_key_admin $secret_key_admin) -}}
   {{- $access_key_admin = include "seaweedfs.getOrGeneratePassword" (dict "namespace" .Release.Namespace "secretName" $secretName "key" "admin_access_key_id" "length" 20 "existingSecret" (ternary $existingSecret nil $reuse)) -}}
   {{- $secret_key_admin = include "seaweedfs.getOrGeneratePassword" (dict "namespace" .Release.Namespace "secretName" $secretName "key" "admin_secret_access_key" "length" 40 "existingSecret" (ternary $existingSecret nil $reuse)) -}}
 {{- end -}}
 {{- $readCreds := $creds.read | default dict -}}
 {{- $access_key_read := $readCreds.accessKey -}}
 {{- $secret_key_read := $readCreds.secretKey -}}
-{{- if $readCreds.existingSecret -}}
-  {{- $access_key_read = printf "${%s}" (include "seaweedfs.s3.credentialEnvName" (list "read" "accessKey")) -}}
-  {{- $secret_key_read = printf "${%s}" (include "seaweedfs.s3.credentialEnvName" (list "read" "secretKey")) -}}
-{{- else if not (and $access_key_read $secret_key_read) -}}
+{{- if not (and $access_key_read $secret_key_read) -}}
   {{- $access_key_read = include "seaweedfs.getOrGeneratePassword" (dict "namespace" .Release.Namespace "secretName" $secretName "key" "read_access_key_id" "length" 20 "existingSecret" (ternary $existingSecret nil $reuse)) -}}
   {{- $secret_key_read = include "seaweedfs.getOrGeneratePassword" (dict "namespace" .Release.Namespace "secretName" $secretName "key" "read_secret_access_key" "length" 40 "existingSecret" (ternary $existingSecret nil $reuse)) -}}
 {{- end -}}
@@ -50,16 +41,10 @@ metadata:
     app.kubernetes.io/instance: {{ .Release.Name }}
     app.kubernetes.io/component: s3
 stringData:
-  {{- /* An identity read from an existing Secret keeps its keys there; the
-         config below names the environment variables carrying them. */}}
-  {{- if not $adminCreds.existingSecret }}
   admin_access_key_id: {{ $access_key_admin }}
   admin_secret_access_key: {{ $secret_key_admin }}
-  {{- end }}
-  {{- if not $readCreds.existingSecret }}
   read_access_key_id: {{ $access_key_read }}
   read_secret_access_key: {{ $secret_key_read }}
-  {{- end }}
   seaweedfs_s3_config: '{"identities":[{"name":"anvAdmin","credentials":[{"accessKey":"{{ $access_key_admin }}","secretKey":"{{ $secret_key_admin }}"}],"actions":["Admin","Read","Write"]},{"name":"anvReadOnly","credentials":[{"accessKey":"{{ $access_key_read }}","secretKey":"{{ $secret_key_read }}"}],"actions":["Read"]}]}'
   {{- if .Values.filer.s3.auditLogConfig }}
   filer_s3_auditLogConfig.json: |

k8s/charts/seaweedfs/templates/s3/s3-service.yaml

@@ -21,7 +21,6 @@ spec:
   {{- $nodePorts = .Values.s3.service.nodePorts | default dict }}
   {{- end }}
   type: {{ $serviceType }}
-  {{- include "seaweedfs.service.loadBalancerFields" .Values.s3.service }}
   internalTrafficPolicy: {{ .Values.s3.internalTrafficPolicy | default "Cluster" }}
   {{- $td := .Values.s3.trafficDistribution | default .Values.filer.s3.trafficDistribution }}
   {{- if and (semverCompare ">=1.31-0" .Capabilities.KubeVersion.GitVersion) $td }}
@@ -44,15 +43,6 @@ spec:
 {{- end }}
     protocol: TCP
 {{- end }}
-{{- if and .Values.s3.enabled .Values.s3.lancePort }}
-  - name: "swfs-lance"
-    port: {{ .Values.s3.lancePort }}
-    targetPort: {{ .Values.s3.lancePort }}
-{{- if $nodePorts.lance }}
-    nodePort: {{ $nodePorts.lance }}
-{{- end }}
-    protocol: TCP
-{{- end }}
 {{- if and .Values.s3.enabled .Values.s3.httpsPort }}
   - name: "swfs-s3-tls"
     port: {{ .Values.s3.httpsPort }}

k8s/charts/seaweedfs/templates/sftp/sftp-service.yaml

@@ -21,7 +21,6 @@ spec:
   {{- $nodePorts = .Values.sftp.service.nodePorts | default dict }}
   {{- end }}
   type: {{ $serviceType }}
-  {{- include "seaweedfs.service.loadBalancerFields" .Values.sftp.service }}
   internalTrafficPolicy: {{ .Values.sftp.internalTrafficPolicy | default "Cluster" }}
   ports:
   - name: "swfs-sftp"

k8s/charts/seaweedfs/templates/shared/_helpers.tpl

@@ -76,18 +76,6 @@ Inject extra environment vars in the format key:value, if populated
 {{- end }}
 {{- end -}}
 
-{{/* Whether the mysql filer store is selected; a flag the chart cannot read counts as selected. */}}
-{{- define "seaweedfs.filer.mysqlEnabled" -}}
-{{- $merged := dict -}}
-{{- $_ := include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global.seaweedfs "component" .Values.filer "target" $merged) -}}
-{{- $enabled := index $merged "WEED_MYSQL_ENABLED" -}}
-{{- if or (kindIs "map" $enabled) (hasKey (.Values.filer.secretExtraEnvironmentVars | default dict) "WEED_MYSQL_ENABLED") -}}
-true
-{{- else if and $enabled (eq (lower (toString $enabled)) "true") -}}
-true
-{{- end -}}
-{{- end -}}
-
 {{/* Return the proper filer image */}}
 {{- define "seaweedfs.filer.image" -}}
 {{- if .Values.filer.imageOverride -}}
@@ -148,15 +136,6 @@ true
 {{- end -}}
 {{- end -}}
 
-{{/* Lance namespace URL the worker's Lance container maintains; empty when unreachable */}}
-{{- define "seaweedfs.worker.lanceNamespaceUrl" -}}
-{{- if .Values.worker.namespaceUrl -}}
-{{- .Values.worker.namespaceUrl -}}
-{{- else if and .Values.s3.enabled .Values.s3.lancePort -}}
-{{- printf "http://%s.%s:%d" (include "seaweedfs.componentName" (list . "s3")) .Release.Namespace (int .Values.s3.lancePort) -}}
-{{- end -}}
-{{- end -}}
-
 {{/* Return the proper volume image */}}
 {{- define "seaweedfs.volume.image" -}}
 {{- if .Values.volume.imageOverride -}}
@@ -517,39 +496,6 @@ true
 {{- end }}
 {{- end -}}
 
-{{/* Name of the environment variable carrying one generated S3 credential
-     field, e.g. SEAWEEDFS_S3_ADMIN_ACCESS_KEY_ID. The generated identities file
-     names it in place of the key when the key lives in an existing Secret.
-     Usage: include "seaweedfs.s3.credentialEnvName" (list "admin" "accessKey") */}}
-{{- define "seaweedfs.s3.credentialEnvName" -}}
-{{- $identity := index . 0 -}}
-{{- $field := index . 1 -}}
-{{- printf "SEAWEEDFS_S3_%s_%s" (upper $identity) (ternary "ACCESS_KEY_ID" "SECRET_ACCESS_KEY" (eq $field "accessKey")) -}}
-{{- end -}}
-
-{{/* Environment for the S3 identities the chart generates from an existing
-     Secret. The gateway resolves the ${VAR} references the identities file
-     carries, so the keys never enter the rendered manifests and a dry run
-     renders the same as an install. */}}
-{{- define "seaweedfs.s3.credentialEnv" -}}
-{{- $creds := $.Values.s3.credentials | default dict -}}
-{{- range $identity := list "admin" "read" -}}
-{{- $identityCreds := index $creds $identity | default dict -}}
-{{- if $identityCreds.existingSecret }}
-- name: {{ include "seaweedfs.s3.credentialEnvName" (list $identity "accessKey") }}
-  valueFrom:
-    secretKeyRef:
-      name: {{ $identityCreds.existingSecret | quote }}
-      key: {{ default (printf "%s_access_key_id" $identity) $identityCreds.accessKeyKey | quote }}
-- name: {{ include "seaweedfs.s3.credentialEnvName" (list $identity "secretKey") }}
-  valueFrom:
-    secretKeyRef:
-      name: {{ $identityCreds.existingSecret | quote }}
-      key: {{ default (printf "%s_secret_access_key" $identity) $identityCreds.secretKeyKey | quote }}
-{{- end -}}
-{{- end -}}
-{{- end -}}
-
 {{/* Generate a compatible trafficDistribution value due to "PreferClose" fast deprecation in k8s v1.35.
      Accepts a dict with "value" (the trafficDistribution string) and "Capabilities". */}}
 {{- define "seaweedfs.trafficDistribution" -}}
@@ -557,23 +503,3 @@ true
 {{- and (eq .value "PreferClose") (semverCompare ">=1.35-0" .Capabilities.KubeVersion.GitVersion) | ternary "PreferSameZone" .value -}}
 {{- end -}}
 {{- end -}}
-
-{{/*
-Render LoadBalancer-specific service fields (loadBalancerClass, loadBalancerIP,
-loadBalancerSourceRanges), only when the service type is LoadBalancer.
-Usage: {{ include "seaweedfs.service.loadBalancerFields" .Values.s3.service }}
-*/}}
-{{- define "seaweedfs.service.loadBalancerFields" -}}
-{{- if eq (.type | default "ClusterIP") "LoadBalancer" }}
-{{- with .loadBalancerClass }}
-  loadBalancerClass: {{ . }}
-{{- end }}
-{{- with .loadBalancerIP }}
-  loadBalancerIP: {{ . }}
-{{- end }}
-{{- with .loadBalancerSourceRanges }}
-  loadBalancerSourceRanges:
-    {{- toYaml . | nindent 4 }}
-{{- end }}
-{{- end }}
-{{- end -}}

k8s/charts/seaweedfs/templates/shared/networkpolicy.yaml

@@ -73,9 +73,6 @@
 {{-   if .Values.s3.icebergPort }}
 {{-     $ports = append $ports .Values.s3.icebergPort }}
 {{-   end }}
-{{-   if .Values.s3.lancePort }}
-{{-     $ports = append $ports .Values.s3.lancePort }}
-{{-   end }}
 {{-   if .Values.s3.metricsPort }}
 {{-     $ports = append $ports .Values.s3.metricsPort }}
 {{-   end }}
@@ -100,9 +97,6 @@
 {{-   if .Values.worker.metricsPort }}
 {{-     $ports = append $ports .Values.worker.metricsPort }}
 {{-   end }}
-{{-   if and .Values.worker.lanceMetricsPort (include "seaweedfs.worker.lanceNamespaceUrl" .) }}
-{{-     $ports = append $ports .Values.worker.lanceMetricsPort }}
-{{-   end }}
 {{-   $targets = append $targets (dict "component" "worker" "ports" $ports) }}
 {{- end }}
 

k8s/charts/seaweedfs/templates/shared/secret-seaweedfs-db.yaml

@@ -1,4 +1,4 @@
-{{- if and .Values.filer.enabled (include "seaweedfs.filer.mysqlEnabled" .) }}
+{{- if .Values.filer.enabled }}
 apiVersion: v1
 kind: Secret
 type: Opaque

k8s/charts/seaweedfs/templates/worker/worker-deployment.yaml

@@ -223,81 +223,6 @@ spec:
           {{- if .Values.worker.containerSecurityContext.enabled }}
           securityContext: {{- omit .Values.worker.containerSecurityContext "enabled" | toYaml | nindent 12 }}
           {{- end }}
-        {{- if include "seaweedfs.worker.lanceNamespaceUrl" . }}
-        - name: worker-lance
-          image: {{ template "seaweedfs.worker.image" . }}
-          imagePullPolicy: {{ default "IfNotPresent" .Values.global.seaweedfs.imagePullPolicy }}
-          env:
-            - name: POD_NAME
-              valueFrom:
-                fieldRef:
-                  fieldPath: metadata.name
-            {{- /* the URL crosses a shell line; metacharacters must arrive as data, not syntax */}}
-            - name: LANCE_NAMESPACE_URL
-              value: {{ include "seaweedfs.worker.lanceNamespaceUrl" . | quote }}
-          command:
-            - "/bin/sh"
-            - "-ec"
-            - |
-              {{- /* the armv7/386 placeholder is empty; exec of it becomes the shell and exits 0 */}}
-              if [ ! -s /usr/bin/weed-worker ]; then
-                echo "the Rust worker is not available on this platform ($(uname -m)); it ships for amd64 and arm64" >&2
-                exit 1
-              fi
-              exec /usr/bin/weed-worker \
-              --id="$POD_NAME" \
-              {{- if .Values.worker.adminServer }}
-              --admin={{ .Values.worker.adminServer }} \
-              {{- else }}
-              --admin={{ template "seaweedfs.fullname" . }}-admin.{{ .Release.Namespace }}:{{ .Values.admin.port }}{{ if .Values.admin.grpcPort }}.{{ .Values.admin.grpcPort }}{{ end }} \
-              {{- end }}
-              --namespace="$LANCE_NAMESPACE_URL" \
-              {{- if .Values.global.seaweedfs.enableSecurity }}
-              --tls-ca=/usr/local/share/ca-certificates/ca/tls.crt \
-              --tls-cert=/usr/local/share/ca-certificates/worker/tls.crt \
-              --tls-key=/usr/local/share/ca-certificates/worker/tls.key \
-              {{- end }}
-              {{- if .Values.worker.lanceMetricsPort }}
-              --metrics-port={{ .Values.worker.lanceMetricsPort }} \
-              --metrics-ip=0.0.0.0 \
-              {{- end }}
-              --max-concurrency={{ .Values.worker.maxExecute }}
-          {{- if .Values.global.seaweedfs.enableSecurity }}
-          volumeMounts:
-            - name: ca-cert
-              readOnly: true
-              mountPath: /usr/local/share/ca-certificates/ca/
-            - name: worker-cert
-              readOnly: true
-              mountPath: /usr/local/share/ca-certificates/worker/
-          {{- end }}
-          {{- if .Values.worker.lanceMetricsPort }}
-          ports:
-            - containerPort: {{ .Values.worker.lanceMetricsPort }}
-              name: lance-metrics
-          livenessProbe:
-            httpGet:
-              path: /health
-              port: lance-metrics
-            initialDelaySeconds: 30
-            periodSeconds: 60
-            successThreshold: 1
-            failureThreshold: 5
-            timeoutSeconds: 10
-          readinessProbe:
-            httpGet:
-              path: /ready
-              port: lance-metrics
-            initialDelaySeconds: 20
-            periodSeconds: 15
-            successThreshold: 1
-            failureThreshold: 3
-            timeoutSeconds: 10
-          {{- end }}
-          {{- if .Values.worker.containerSecurityContext.enabled }}
-          securityContext: {{- omit .Values.worker.containerSecurityContext "enabled" | toYaml | nindent 12 }}
-          {{- end }}
-        {{- end }}
       {{- if .Values.worker.sidecars }}
       {{- include "seaweedfs.tplvalues.render" (dict "value" .Values.worker.sidecars "context" $) | nindent 8 }}
       {{- end }}

k8s/charts/seaweedfs/templates/worker/worker-service.yaml

@@ -12,22 +12,13 @@ metadata:
     app.kubernetes.io/component: worker
 spec:
   clusterIP: None  # Headless service
-  {{- $lanceMetrics := and .Values.worker.lanceMetricsPort (include "seaweedfs.worker.lanceNamespaceUrl" .) }}
-  {{- if or .Values.worker.metricsPort $lanceMetrics }}
-  ports:
   {{- if .Values.worker.metricsPort }}
+  ports:
   - name: "metrics"
     port: {{ .Values.worker.metricsPort }}
     targetPort: {{ .Values.worker.metricsPort }}
     protocol: TCP
   {{- end }}
-  {{- if $lanceMetrics }}
-  - name: "lance-metrics"
-    port: {{ .Values.worker.lanceMetricsPort }}
-    targetPort: {{ .Values.worker.lanceMetricsPort }}
-    protocol: TCP
-  {{- end }}
-  {{- end }}
   selector:
     app.kubernetes.io/name: {{ template "seaweedfs.name" . }}
     app.kubernetes.io/instance: {{ .Release.Name }}

k8s/charts/seaweedfs/templates/worker/worker-servicemonitor.yaml

@@ -1,7 +1,6 @@
 {{- include "seaweedfs.compat" . -}}
 {{- if .Values.worker.enabled }}
-{{- $lanceMetrics := and .Values.worker.lanceMetricsPort (include "seaweedfs.worker.lanceNamespaceUrl" .) }}
-{{- if or .Values.worker.metricsPort $lanceMetrics }}
+{{- if .Values.worker.metricsPort }}
 {{- if .Values.global.seaweedfs.monitoring.enabled }}
 apiVersion: monitoring.coreos.com/v1
 kind: ServiceMonitor
@@ -23,16 +22,9 @@ metadata:
 {{- end }}
 spec:
   endpoints:
-    {{- if .Values.worker.metricsPort }}
     - interval: 30s
       port: metrics
       scrapeTimeout: 5s
-    {{- end }}
-    {{- if $lanceMetrics }}
-    - interval: 30s
-      port: lance-metrics
-      scrapeTimeout: 5s
-    {{- end }}
   selector:
     matchLabels:
       app.kubernetes.io/name: {{ template "seaweedfs.name" . }}

k8s/charts/seaweedfs/values.yaml

@@ -897,7 +897,6 @@ filer:
 
   # extraEnvVars is a list of extra environment variables to set with the stateful set.
   extraEnvironmentVars:
-    # the WEED_MYSQL_* keys and the db credential secret only render while this is "true"
     WEED_MYSQL_ENABLED: "false"
     WEED_MYSQL_HOSTNAME: "mysql-db-host"
     WEED_MYSQL_PORT: "3306"
@@ -1006,9 +1005,6 @@ s3:
   # Iceberg catalog REST port (Apache Iceberg REST Catalog API)
   # Set to a port number to enable, or 0/null to disable
   icebergPort: null
-  # Lance Namespace port; weed serves 9101 by default, 0 disables it
-  # (and, unless worker.namespaceUrl points elsewhere, the worker's Lance container)
-  lancePort: 9101
   loggingOverrideLevel: null
   # enable user & permission to s3 (need to inject to all services)
   enableAuth: false
@@ -1018,24 +1014,13 @@ s3:
   # Optionally provide explicit credentials for the S3 gateway.
   # When set, these are used in the generated s3 secret instead of
   # auto-generating random credentials.
-  # An identity may instead name an existing Secret to read its keys from. The
-  # generated config then references the keys through environment variables, so
-  # nothing is looked up at render time and a dry run renders what an install
-  # applies. Note the COSI driver parses the config itself and does not resolve
-  # those references.
   # credentials:
   #   admin:
   #     accessKey: ""
   #     secretKey: ""
-  #     existingSecret: ""
-  #     accessKeyKey: admin_access_key_id
-  #     secretKeyKey: admin_secret_access_key
   #   read:
   #     accessKey: ""
   #     secretKey: ""
-  #     existingSecret: ""
-  #     accessKeyKey: read_access_key_id
-  #     secretKeyKey: read_secret_access_key
   auditLogConfig: {}
   # You may specify buckets to be created during the install or upgrade process.
   # Buckets may be exposed publicly by setting `anonymousRead` to `true`
@@ -1187,16 +1172,11 @@ s3:
   # Service settings
   service:
     type: ClusterIP
-    # used only when type is LoadBalancer
-    loadBalancerClass: ""
-    loadBalancerIP: ""
-    loadBalancerSourceRanges: []
     # fixed nodePorts, used only when type is NodePort or LoadBalancer
     nodePorts:
       http: null
       https: null
       iceberg: null
-      lance: null
       metrics: null
 
   icebergIngress:
@@ -1208,15 +1188,6 @@ s3:
     annotations: {}
     tls: []
 
-  lanceIngress:
-    enabled: false
-    className: ""
-    host: "seaweedfs-lance.cluster.local"
-    path: "/"
-    pathType: Prefix
-    annotations: {}
-    tls: []
-
 sftp:
   enabled: false
   imageOverride: null
@@ -1305,10 +1276,6 @@ sftp:
   # Service settings
   service:
     type: ClusterIP
-    # used only when type is LoadBalancer
-    loadBalancerClass: ""
-    loadBalancerIP: ""
-    loadBalancerSourceRanges: []
     # fixed nodePorts, used only when type is NodePort or LoadBalancer
     nodePorts:
       sftp: null
@@ -1451,10 +1418,6 @@ admin:
   service:
     type: ClusterIP
     annotations: {}
-    # used only when type is LoadBalancer
-    loadBalancerClass: ""
-    loadBalancerIP: ""
-    loadBalancerSourceRanges: []
     # fixed nodePorts, used only when type is NodePort or LoadBalancer
     nodePorts:
       http: null
@@ -1473,15 +1436,6 @@ worker:
   metricsPort: 9327
   metricsIp: ""  # If empty, defaults to 0.0.0.0
 
-  # The lance_* jobs run in their own container, /usr/bin/weed-worker.
-  # amd64/arm64 only; pin mixed clusters with worker.affinity/nodeSelector.
-
-  # Lance namespace URL override; empty derives it from s3.lancePort.
-  namespaceUrl: ""
-
-  # Metrics port for the Lance worker container; the Go worker keeps metricsPort
-  lanceMetricsPort: 9328
-
   # Admin server to connect to
   adminServer: ""
 
@@ -1688,10 +1642,6 @@ allInOne:
     annotations: {}  # Annotations for the service
     type: ClusterIP  # Service type (ClusterIP, NodePort, LoadBalancer)
     internalTrafficPolicy: Cluster  # Internal traffic policy
-    # used only when type is LoadBalancer
-    loadBalancerClass: ""
-    loadBalancerIP: ""
-    loadBalancerSourceRanges: []
     # fixed nodePorts, used only when type is NodePort or LoadBalancer
     nodePorts:
       master: null

seaweed-volume/proto/master.proto

@@ -27,8 +27,6 @@ service Seaweed {
   }
   rpc VolumeList (VolumeListRequest) returns (VolumeListResponse) {
   }
-  rpc VolumeListStream (VolumeListRequest) returns (stream VolumeListStreamResponse) {
-  }
   rpc LookupEcVolume (LookupEcVolumeRequest) returns (LookupEcVolumeResponse) {
   }
   rpc VacuumVolume (VacuumVolumeRequest) returns (VacuumVolumeResponse) {
@@ -408,44 +406,12 @@ message TopologyInfo {
   map<string, DiskInfo> diskInfos = 3;
 }
 message VolumeListRequest {
-  // Empty and zero take everything. Only the volumes and ec shards listed
-  // under a disk are selected; the topology and its disk counters are always
-  // reported in full.
-  string collection = 1;
-  repeated uint32 volume_ids = 2;
-  // The one collection the empty string cannot name. A named collection wins.
-  bool default_collection_only = 3;
-  // Empty and zero take everything. Wildcards are supported.
-  string remote_storage_name = 4;
-  bool local_volume_only = 5;
-  // The topology, its disks and their counters alone, without the volumes
-  // and ec shards. Selecting volumes above contradicts this and is refused.
-  // A master that predates this field ignores it and answers in full.
-  bool topology_only = 6;
 }
 message VolumeListResponse {
   TopologyInfo topology_info = 1;
   uint64 volume_size_limit_mb = 2;
 }
 
-// VolumeListStream answers the same request as VolumeList without either end
-// holding every volume in the cluster at once. At 800k volumes the reply is
-// 36MB on the wire but 305MB as messages, which the master built in full
-// before sending any of it.
-message VolumeListStreamResponse {
-  // Sent once, first, listing no volumes: the topology, its disks and their
-  // counters. Every message after carries volumes for one of those disks.
-  VolumeListResponse header = 1;
-  // Which disk this batch is from. A disk arrives over as many batches as it
-  // takes, so append rather than assign.
-  string data_center = 2;
-  string rack = 3;
-  string data_node = 4;
-  string disk_type = 5;
-  repeated VolumeInformationMessage volume_infos = 6;
-  repeated VolumeEcShardInformationMessage ec_shard_infos = 7;
-}
-
 message LookupEcVolumeRequest {
   uint32 volume_id = 1;
 }

seaweed-volume/proto/volume_server.proto

@@ -539,7 +539,6 @@ message VolumeEcShardsInfoResponse {
     uint64 volume_size = 2;
     uint64 file_count = 3;
     uint64 file_deleted_count = 4;
-    EcShardConfig ec_shard_config = 10;  // the layout this holder serves reads through; a binary predating a field reports it as unset
 }
 
 message EcShardInfo {
@@ -613,7 +612,6 @@ message EcShardConfig {
     uint32 data_shards = 1;   // Number of data shards (e.g., 10)
     uint32 parity_shards = 2; // Number of parity shards (e.g., 4)
     int64 encode_ts_ns = 3;   // encode time (unix nanos); a read served from a shard of a different encode run is rejected
-    int64 block_size = 4;     // uniform block layout: each shard is a single contiguous block of this many bytes; 0 = legacy 1GiB/1MiB two-tier layout
 }
 // EcBitrotProtection is the entire content of a bitrot checksum sidecar
 // (<base>.ecsum for the legacy generation, <base>.ecsum.v<N> for vacuum
@@ -716,7 +714,6 @@ enum VolumeScrubMode {
     FULL = 2;
     LOCAL = 3;
     CHECKSUM = 4; // EC only: verify each local shard's raw bytes against the bitrot checksum sidecar
-    READS = 5;    // like FULL, but EC intervals no shard can serve are reconstructed from parity
 }
 
 message ScrubVolumeRequest {

seaweed-volume/src/metrics.rs

@@ -330,16 +330,6 @@ pub fn delete_collection_metrics(collection: &str) {
     delete_partial_match_collection(&DISK_SIZE_GAUGE, collection);
 }
 
-/// Drop a collection's volume server series once its last volume leaves this
-/// server. These gauges are only ever set for collections still present, so the
-/// values from the heartbeat that saw the last volume would otherwise stand
-/// until the process restarts.
-pub fn delete_volume_server_collection_metrics(collection: &str) {
-    let _ = DISK_SIZE_GAUGE.remove_label_values(&[collection, DISK_SIZE_LABEL_NORMAL]);
-    let _ = DISK_SIZE_GAUGE.remove_label_values(&[collection, DISK_SIZE_LABEL_DELETED_BYTES]);
-    delete_partial_match_collection(&READ_ONLY_VOLUME_GAUGE, collection);
-}
-
 /// Remove all metric entries from a GaugeVec where the "collection" label matches.
 /// This emulates Go's `DeletePartialMatch(prometheus.Labels{"collection": collection})`.
 fn delete_partial_match_collection(gauge: &GaugeVec, collection: &str) {

seaweed-volume/src/server/grpc_server.rs

@@ -38,7 +38,6 @@ fn scrub_mode_label(mode: i32) -> &'static str {
         2 => "FULL",
         3 => "LOCAL",
         4 => "CHECKSUM",
-        5 => "READS",
         _ => "UNKNOWN",
     }
 }
@@ -92,57 +91,6 @@ pub fn load_state_file(
     volume_server_pb::VolumeServerState::decode(data.as_slice()).ok()
 }
 
-/// One disk location's stake in an EC volume, as seen by the rebuild handler.
-struct LocInfo {
-    dir: String,
-    idx_dir: String,
-    shard_count: usize,
-    has_ecx: bool,
-}
-
-/// Picks the location a rebuild should write into — the one holding an `.ecx`
-/// and the most shards — and returns every other directory it may have to read
-/// from. Shards are only half of what the rebuild needs: a split
-/// `-dir`/`-dir.idx` layout keeps `.ecx`/`.ecj`/`.vif` with the INDEX, and on a
-/// multi-disk server the chosen disk may hold nothing but shards while this
-/// volume's `.vif` or generation-0 `.ecsum` sits on a sibling. Miss those and
-/// the layout resolution falls back to 10+4 with the legacy striping and
-/// reconstructs through the wrong matrix, so both directories of every other
-/// location are listed. The rebuild's own two are passed separately by the
-/// caller and dropped here, along with empties and duplicates.
-///
-/// Returns `None` when no location holds an `.ecx`, i.e. there is nothing to
-/// rebuild from.
-fn select_rebuild_location(loc_infos: &[LocInfo]) -> Option<(usize, Vec<String>)> {
-    let mut rebuild_loc_idx: Option<usize> = None;
-    let mut other_dirs: Vec<String> = Vec::new();
-
-    for (i, info) in loc_infos.iter().enumerate() {
-        let better = info.has_ecx
-            && rebuild_loc_idx
-                .is_none_or(|prev| info.shard_count > loc_infos[prev].shard_count);
-        if better {
-            if let Some(prev) = rebuild_loc_idx {
-                other_dirs.push(loc_infos[prev].dir.clone());
-                other_dirs.push(loc_infos[prev].idx_dir.clone());
-            }
-            rebuild_loc_idx = Some(i);
-        } else {
-            other_dirs.push(info.dir.clone());
-            other_dirs.push(info.idx_dir.clone());
-        }
-    }
-
-    let rebuild_loc_idx = rebuild_loc_idx?;
-    let rebuild_dir = &loc_infos[rebuild_loc_idx].dir;
-    let rebuild_idx_dir = &loc_infos[rebuild_loc_idx].idx_dir;
-    other_dirs.retain(|d| !d.is_empty() && d != rebuild_dir && d != rebuild_idx_dir);
-    other_dirs.sort();
-    other_dirs.dedup();
-
-    Some((rebuild_loc_idx, other_dirs))
-}
-
 struct WriteThrottler {
     bytes_per_second: i64,
     last_size_counter: i64,
@@ -1820,14 +1768,11 @@ impl VolumeServer for VolumeGrpcService {
                                 }
                                 Some(store.locations[info.disk_id as usize].directory.clone())
                             } else {
-                                // The mounted-volume refusal above means no disk
-                                // holds an in-memory claim here.
                                 store
                                     .find_ec_shard_target_location(
                                         &info.collection,
                                         vid,
                                         DATA_SHARDS_COUNT as u32,
-                                        &[],
                                     )
                                     .map(|i| store.locations[i].directory.clone())
                             };
@@ -2437,21 +2382,13 @@ impl VolumeServer for VolumeGrpcService {
             )
         };
 
-        // Check existing .vif for EC shard config (matching Go's MaybeLoadVolumeInfo).
-        // The block size is recomputed by the encode for the current .dat, so
-        // only the ratio is carried over from a prior config.
-        let (data_shards, parity_shards, _) =
+        // Check existing .vif for EC shard config (matching Go's MaybeLoadVolumeInfo)
+        let (data_shards, parity_shards) =
             crate::storage::erasure_coding::ec_volume::read_ec_shard_config(
                 &dir, &idx_dir, collection, vid,
-            )
-            .map_err(|e| {
-                tonic::Status::internal(format!(
-                    "read ec shard config for volume {}: {}",
-                    vid.0, e
-                ))
-            })?;
+            );
 
-        let block_size = match crate::storage::erasure_coding::ec_encoder::write_ec_files(
+        if let Err(e) = crate::storage::erasure_coding::ec_encoder::write_ec_files(
             &dir,
             &idx_dir,
             collection,
@@ -2459,19 +2396,16 @@ impl VolumeServer for VolumeGrpcService {
             data_shards as usize,
             parity_shards as usize,
         ) {
-            Ok(block_size) => block_size,
-            Err(e) => {
-                // Cleanup partially-created .ecNN and .ecx files on failure (matching Go defer)
-                let base = crate::storage::volume::volume_file_name(&dir, collection, vid);
-                let total_shards = data_shards + parity_shards;
-                for i in 0..total_shards {
-                    let shard_path = format!("{}.ec{:02}", base, i);
-                    let _ = std::fs::remove_file(&shard_path);
-                }
-                let _ = std::fs::remove_file(format!("{}.ecx", base));
-                return Err(Status::internal(e.to_string()));
+            // Cleanup partially-created .ecNN and .ecx files on failure (matching Go defer)
+            let base = crate::storage::volume::volume_file_name(&dir, collection, vid);
+            let total_shards = data_shards + parity_shards;
+            for i in 0..total_shards {
+                let shard_path = format!("{}.ec{:02}", base, i);
+                let _ = std::fs::remove_file(&shard_path);
             }
-        };
+            let _ = std::fs::remove_file(format!("{}.ecx", base));
+            return Err(Status::internal(e.to_string()));
+        }
 
         // Write .vif file with EC shard metadata
         {
@@ -2490,7 +2424,6 @@ impl VolumeServer for VolumeGrpcService {
                         .duration_since(std::time::UNIX_EPOCH)
                         .unwrap_or_default()
                         .as_nanos() as i64,
-                    block_size,
                 }),
                 ..Default::default()
             };
@@ -2524,6 +2457,13 @@ impl VolumeServer for VolumeGrpcService {
             format!("{}_{}", collection, vid.0)
         };
 
+        struct LocInfo {
+            dir: String,
+            idx_dir: String,
+            shard_count: usize,
+            has_ecx: bool,
+        }
+
         let store = self.state.store.read().unwrap();
         let mut loc_infos: Vec<LocInfo> = Vec::new();
 
@@ -2571,8 +2511,26 @@ impl VolumeServer for VolumeGrpcService {
             ));
         }
 
-        let (rebuild_loc_idx, other_dirs) = match select_rebuild_location(&loc_infos) {
-            Some(picked) => picked,
+        // Pick rebuild location: has .ecx and most shards
+        let mut rebuild_loc_idx: Option<usize> = None;
+        let mut other_dirs: Vec<String> = Vec::new();
+
+        for (i, info) in loc_infos.iter().enumerate() {
+            if info.has_ecx
+                && (rebuild_loc_idx.is_none()
+                    || info.shard_count > loc_infos[rebuild_loc_idx.unwrap()].shard_count)
+            {
+                if let Some(prev) = rebuild_loc_idx {
+                    other_dirs.push(loc_infos[prev].dir.clone());
+                }
+                rebuild_loc_idx = Some(i);
+            } else {
+                other_dirs.push(info.dir.clone());
+            }
+        }
+
+        let rebuild_loc_idx = match rebuild_loc_idx {
+            Some(i) => i,
             None => {
                 return Ok(Response::new(
                     volume_server_pb::VolumeEcShardsRebuildResponse {
@@ -2586,37 +2544,13 @@ impl VolumeServer for VolumeGrpcService {
         let rebuild_idx_dir = loc_infos[rebuild_loc_idx].idx_dir.clone();
 
         // Determine data/parity shard config from rebuild dir
-        // The encode-time .dat size resolves the row count the ecx rebuild
-        // de-stripes with; 0 leaves it to infer from the padded shard extent.
-        // Both lookups search the sibling disks too: the rebuild writes into one
-        // location, but a multi-disk server may keep this volume's .vif or its
-        // generation-0 .ecsum on another, and defaulting to 10+4 with the
-        // legacy layout would reconstruct through the wrong matrix.
-        let dat_file_size = crate::storage::erasure_coding::ec_volume::load_vif_info_across_dirs(
-            &rebuild_dir,
-            &rebuild_idx_dir,
-            &other_dirs,
-            collection,
-            vid,
-        )
-        .ok()
-        .flatten()
-        .map(|(v, _)| v.dat_file_size)
-        .unwrap_or(0);
-        let (data_shards, parity_shards, block_size) =
-            crate::storage::erasure_coding::ec_volume::read_ec_shard_config_across_dirs(
+        let (data_shards, parity_shards) =
+            crate::storage::erasure_coding::ec_volume::read_ec_shard_config(
                 &rebuild_dir,
                 &rebuild_idx_dir,
-                &other_dirs,
                 collection,
                 vid,
-            )
-            .map_err(|e| {
-                tonic::Status::internal(format!(
-                    "read ec shard config for volume {}: {}",
-                    vid.0, e
-                ))
-            })?;
+            );
         let total_shards = data_shards + parity_shards;
 
         // Check which shards are missing (check rebuild dir and all other dirs)
@@ -2655,17 +2589,8 @@ impl VolumeServer for VolumeGrpcService {
 
         // Rebuild missing shards, searching all locations for input shards.
         // Pass other_dirs so shards on sibling disks are found even when the
-        // primary rebuild dir doesn't hold them. This one takes a single flat
-        // list — the shape Go's RebuildEcFiles uses — so unlike the resolvers
-        // above it cannot be handed the rebuild's own index directory
-        // separately, and a split -dir/-dir.idx location keeps its .ecx and
-        // .vif there. Go's additionalDirs carries that directory for the same
-        // reason.
-        let mut rebuild_search_dirs: Vec<String> = other_dirs.clone();
-        if !rebuild_idx_dir.is_empty() && rebuild_idx_dir != rebuild_dir {
-            rebuild_search_dirs.push(rebuild_idx_dir.clone());
-        }
-        let other_dir_refs: Vec<&str> = rebuild_search_dirs.iter().map(|s| s.as_str()).collect();
+        // primary rebuild dir doesn't hold them.
+        let other_dir_refs: Vec<&str> = other_dirs.iter().map(|s| s.as_str()).collect();
         crate::storage::erasure_coding::ec_encoder::rebuild_ec_files(
             &rebuild_dir,
             collection,
@@ -2703,8 +2628,6 @@ impl VolumeServer for VolumeGrpcService {
             collection,
             vid,
             data_shards as usize,
-            block_size,
-            dat_file_size,
             &ecx_dir_refs,
         )
         .map_err(|e| Status::internal(format!("RebuildEcxFile: {}", e)))?;
@@ -2729,15 +2652,13 @@ impl VolumeServer for VolumeGrpcService {
         //   When disk_id > 0: use that specific location.
         //   When disk_id == 0 (unset): auto-select via
         //   find_ec_shard_target_location, which prefers a disk that
-        //   already owns one of the shards being copied (a retried move
-        //   must overwrite in place, not leave two disks of this server
-        //   claiming the same shard), then a disk that already has the EC
-        //   volume mounted, then a disk that owns the .ecx on disk (volume
-        //   not yet mounted — relevant for ec.rebuild, where only the
-        //   first shard carries .ecx and subsequent shards must land on
-        //   the same disk; see #9212), then any HDD, then any disk. Pass
-        //   the build's default data-shard count; the helper takes it as a
-        //   parameter so custom-ratio builds can swap it.
+        //   already has the EC volume mounted, then a disk that owns the
+        //   .ecx on disk (volume not yet mounted — relevant for
+        //   ec.rebuild, where only the first shard carries .ecx and
+        //   subsequent shards must land on the same disk; see #9212),
+        //   then any HDD, then any disk. Pass the build's default
+        //   data-shard count; the helper takes it as a parameter so
+        //   custom-ratio builds can swap it.
         let (dest_dir, dest_idx_dir) = {
             let store = self.state.store.read().unwrap();
             let count = store.locations.len();
@@ -2753,27 +2674,10 @@ impl VolumeServer for VolumeGrpcService {
                 let loc = &store.locations[req.disk_id as usize];
                 (loc.directory.clone(), loc.idx_directory.clone())
             } else {
-                // A batch whose requested shards are already owned by
-                // different local disks has no single correct destination:
-                // writing them all to one disk would duplicate the other
-                // disks' claims. Refuse so the caller splits the batch per
-                // shard (or chooses explicitly via disk_id).
-                let owners = store.ec_shard_owner_disks(vid, &req.shard_ids);
-                if owners.len() > 1 {
-                    let dirs: Vec<&str> = owners
-                        .iter()
-                        .map(|&i| store.locations[i].directory.as_str())
-                        .collect();
-                    return Err(Status::failed_precondition(format!(
-                        "volume {} shards {:?} are already owned by multiple local disks {:?}: no single destination; copy per shard or pass disk_id",
-                        req.volume_id, req.shard_ids, dirs
-                    )));
-                }
                 match store.find_ec_shard_target_location(
                     &req.collection,
                     vid,
                     DATA_SHARDS_COUNT as u32,
-                    &req.shard_ids,
                 ) {
                     Some(i) => {
                         let loc = &store.locations[i];
@@ -3110,27 +3014,6 @@ impl VolumeServer for VolumeGrpcService {
                     Status::internal(format!("mount {}.{}: {}", req.volume_id, shard_id, e))
                 })?;
         }
-        // A delivery can bring the checksum manifest alongside the shards, but
-        // the receive path only writes the file. When this server already had
-        // the volume mounted, the EcVolume in memory keeps whatever protection
-        // state it resolved at mount — off, for a volume whose sidecar arrives
-        // now — until a remount. Re-resolve it here, where the shards it
-        // describes have just been added.
-        //
-        // Every per-disk runtime, not just the first: a vid mounts as one
-        // EcVolume per disk, the delivery lands the .ecsum on one of them, and
-        // the first-match lookup would leave the siblings reporting no
-        // protection. Each re-resolves against its own data and index
-        // directories, so a shared -dir.idx reaches all of them.
-        // Resolving across every EC metadata directory is what makes that
-        // reload mean something: startup mirroring gives each shard-bearing
-        // disk its own .ecx/.ecj/.vif but deliberately not the sidecar, so a
-        // runtime restricted to its own two directories would find nothing
-        // however often it reloaded. One delivered copy, reachable from all.
-        let ec_metadata_dirs = store.ec_metadata_dirs();
-        for ec_vol in store.find_all_ec_volumes_mut(vid) {
-            ec_vol.reload_bitrot_sidecar(&ec_metadata_dirs);
-        }
         drop(store);
         self.state.volume_state_notify.notify_one();
 
@@ -3465,8 +3348,6 @@ impl VolumeServer for VolumeGrpcService {
         let ecx_dir = ec_vol.ecx_actual_dir().to_string();
         let collection = ec_vol.collection.clone();
         let vif_dat_file_size = ec_vol.dat_file_size;
-        let (large_block_size, small_block_size) =
-            (ec_vol.large_block_size(), ec_vol.small_block_size());
         // shard_dirs[i] is guaranteed Some for i in 0..data_shards by
         // the check above; collect concrete dirs for the decoder.
         let per_shard_dirs: Vec<String> = shard_dirs[..data_shards]
@@ -3500,8 +3381,6 @@ impl VolumeServer for VolumeGrpcService {
             vif_dat_file_size,
             data_shards,
             &per_shard_dirs,
-            large_block_size as usize,
-            small_block_size as usize,
         )
         .map_err(|e| Status::internal(format!("WriteDatFile: {}", e)))?;
 
@@ -3560,24 +3439,12 @@ impl VolumeServer for VolumeGrpcService {
             .walk_ecx_stats()
             .map_err(|e| Status::internal(e.to_string()))?;
 
-        // The layout this holder serves reads through, as Go reports it: a
-        // coordinator cannot otherwise tell a holder that understands the
-        // uniform block layout from one that dropped the unknown .vif field
-        // and mounted the volume as legacy.
-        let ec_shard_config = Some(volume_server_pb::EcShardConfig {
-            data_shards: ec_vol.data_shards,
-            parity_shards: ec_vol.parity_shards,
-            encode_ts_ns: 0,
-            block_size: ec_vol.block_size,
-        });
-
         Ok(Response::new(
             volume_server_pb::VolumeEcShardsInfoResponse {
                 ec_shard_infos: shard_infos,
                 volume_size,
                 file_count,
                 file_deleted_count,
-                ec_shard_config,
             },
         ))
     }
@@ -3708,12 +3575,7 @@ impl VolumeServer for VolumeGrpcService {
                             modified_time: dat_modified_secs,
                             extension: ".dat".to_string(),
                         });
-                        vol.refresh_remote_write_mode().map_err(|e| {
-                            Status::internal(format!(
-                                "volume {} failed to refresh write mode: {}",
-                                vid, e
-                            ))
-                        })?;
+                        vol.refresh_remote_write_mode();
 
                         if let Err(e) = vol.save_volume_info() {
                             return Err(Status::internal(format!(
@@ -3899,40 +3761,10 @@ impl VolumeServer for VolumeGrpcService {
                         )));
                     }
 
-                    // Snapshot the remote reference before dropping it: the
-                    // refresh below can fail, and a half-applied transition
-                    // leaves the volume claiming local while the remote backend
-                    // is still attached and the on-disk .vif still says remote
-                    // — a state a retry reads as "already on local disk" and
-                    // refuses to finish.
-                    let removed_remote = if vol.volume_info.files.is_empty() {
-                        None
-                    } else {
-                        Some(vol.volume_info.files.remove(0))
-                    };
-                    // Swaps the read-only sorted map out before the volume is
-                    // published as writable; without it the first write would
-                    // append to the local .dat and then fail to index.
-                    if let Err(e) = vol.refresh_remote_write_mode() {
-                        if let Some(remote) = removed_remote {
-                            vol.volume_info.files.insert(0, remote);
-                        }
-                        // Put the derived flags and the needle map back where
-                        // the restored reference says they belong. Best effort:
-                        // if even this fails the volume stays pinned read-only,
-                        // which is the safe end of the transition.
-                        if let Err(restore_err) = vol.refresh_remote_write_mode() {
-                            tracing::warn!(
-                                volume_id = vid.0,
-                                error = %restore_err,
-                                "tier-down rollback could not restore the remote write mode",
-                            );
-                        }
-                        return Err(Status::internal(format!(
-                            "volume {} failed to refresh write mode: {}",
-                            vid, e
-                        )));
+                    if !vol.volume_info.files.is_empty() {
+                        vol.volume_info.files.remove(0);
                     }
+                    vol.refresh_remote_write_mode();
 
                     if let Err(e) = vol.save_volume_info() {
                         return Err(Status::internal(format!(
@@ -4230,7 +4062,7 @@ impl VolumeServer for VolumeGrpcService {
         // Validate mode
         let mode = req.mode;
         match mode {
-            1 | 2 | 3 | 5 => {} // INDEX=1, FULL=2, LOCAL=3, READS=5 (FULL for regular volumes)
+            1 | 2 | 3 => {} // INDEX=1, FULL=2, LOCAL=3
             _ => {
                 return Err(Status::invalid_argument(format!(
                     "unsupported volume scrub mode {}",
@@ -4330,7 +4162,7 @@ impl VolumeServer for VolumeGrpcService {
         // Validate mode
         let mode = req.mode;
         match mode {
-            1 | 2 | 3 | 4 | 5 => {} // INDEX=1, FULL=2, LOCAL=3, CHECKSUM=4, READS=5
+            1 | 2 | 3 | 4 => {} // INDEX=1, FULL=2, LOCAL=3, CHECKSUM=4
             _ => {
                 return Err(Status::invalid_argument(format!(
                     "unsupported EC volume scrub mode {}",
@@ -4339,14 +4171,6 @@ impl VolumeServer for VolumeGrpcService {
             }
         }
 
-        // Only the modes that walk needles can be strict about deleted ones.
-        let force_deleted_needles_check = req.force_deleted_needles_check;
-        if force_deleted_needles_check && mode != 2 && mode != 5 {
-            return Err(Status::invalid_argument(
-                "deleted needle checks are only supported for FULL and READS scrubs",
-            ));
-        }
-
         // Collect the volume ids under a brief lock, then release it: FULL (mode 2)
         // reads remote shards and must not hold the !Send store guard across .await.
         let vids: Vec<VolumeId> = {
@@ -4388,8 +4212,8 @@ impl VolumeServer for VolumeGrpcService {
                         }
                     }
                 }
-                2 | 5 => {
-                    // FULL/READS: Go-parity per-needle local+remote walk, PLUS a TEMPORARY
+                2 => {
+                    // FULL: Go-parity per-needle local+remote walk, PLUS a TEMPORARY
                     // local Reed-Solomon parity check. The needle walk only reads
                     // DATA-shard intervals of LIVE needles, so on its own it can't
                     // catch silent bitrot in a PARITY shard or an unwalked cold
@@ -4421,13 +4245,8 @@ impl VolumeServer for VolumeGrpcService {
 
                     // (1) Per-needle local+remote walk (Go ScrubEcVolume parity).
                     let (files, mut shard_infos, mut errs) =
-                        crate::server::store_ec::scrub_ec_volume_distributed(
-                            &self.state,
-                            vid,
-                            force_deleted_needles_check,
-                            mode == 5,
-                        )
-                        .await;
+                        crate::server::store_ec::scrub_ec_volume_distributed(&self.state, vid, false)
+                            .await;
                     total_files += files as u64; // count comes from the needle walk only
 
                     // (2) Local parity check, gated on all-shards-local. Blocking RS
@@ -5218,110 +5037,6 @@ mod tests {
     use tempfile::TempDir;
     use tokio_stream::StreamExt;
 
-    fn loc(dir: &str, idx_dir: &str, shard_count: usize, has_ecx: bool) -> LocInfo {
-        LocInfo {
-            dir: dir.to_string(),
-            idx_dir: idx_dir.to_string(),
-            shard_count,
-            has_ecx,
-        }
-    }
-
-    // The rebuild reads its shards from one directory but resolves the volume's
-    // layout -- ratio and uniform block size -- from the .vif or the
-    // generation-0 .ecsum, which on a multi-disk server may sit anywhere. Every
-    // directory that could hold one has to be in the search list, or the
-    // resolution silently falls back to 10+4 with the legacy striping and
-    // reconstructs through the wrong matrix.
-    // The rebuild's own data and index directories are handed to the resolvers
-    // as their own arguments, so they are deliberately absent from this list --
-    // unlike Go, whose resolver takes a single directory list and therefore
-    // carries the rebuild's index directory inside it.
-    #[test]
-    fn select_rebuild_location_excludes_the_rebuilds_own_dirs() {
-        for infos in [
-            vec![loc("/data1", "/idx1", 3, true)],
-            vec![loc("/data1", "/data1", 3, true)],
-        ] {
-            let (idx, others) = select_rebuild_location(&infos).expect("a location with .ecx");
-            assert_eq!(idx, 0);
-            assert!(others.is_empty(), "got {:?}", others);
-        }
-    }
-
-    // The case two reviewers flagged: a sibling holding only shards while its
-    // index directory holds this volume's .vif.
-    #[test]
-    fn select_rebuild_location_searches_a_siblings_index_dir_not_just_its_data_dir() {
-        let infos = vec![
-            loc("/data1", "/data1", 5, true),
-            loc("/data2", "/idx2", 2, false),
-        ];
-        let (idx, others) = select_rebuild_location(&infos).expect("a location with .ecx");
-        assert_eq!(idx, 0);
-        assert_eq!(others, vec!["/data2".to_string(), "/idx2".to_string()]);
-    }
-
-    // Several disks pointed at one index directory is a normal -dir.idx
-    // deployment; the shared directory is worth searching but only once.
-    #[test]
-    fn select_rebuild_location_lists_a_shared_index_dir_once() {
-        let infos = vec![
-            loc("/data1", "/data1", 5, true),
-            loc("/data2", "/shared-idx", 2, false),
-            loc("/data3", "/shared-idx", 1, false),
-        ];
-        let (_, others) = select_rebuild_location(&infos).expect("a location with .ecx");
-        assert_eq!(
-            others,
-            vec![
-                "/data2".to_string(),
-                "/data3".to_string(),
-                "/shared-idx".to_string()
-            ]
-        );
-    }
-
-    // When the shared index directory is the rebuild's own it drops out, since
-    // the caller passes it separately.
-    #[test]
-    fn select_rebuild_location_omits_a_shared_index_dir_it_rebuilds_into() {
-        let infos = vec![
-            loc("/data1", "/shared-idx", 5, true),
-            loc("/data2", "/shared-idx", 2, false),
-        ];
-        let (_, others) = select_rebuild_location(&infos).expect("a location with .ecx");
-        assert_eq!(others, vec!["/data2".to_string()]);
-    }
-
-    // The winner moves as a fuller location turns up; the one it displaces
-    // still has to be searched, index directory included.
-    #[test]
-    fn select_rebuild_location_keeps_the_displaced_winners_dirs() {
-        let infos = vec![
-            loc("/data1", "/idx1", 2, true),
-            loc("/data2", "/idx2", 9, true),
-        ];
-        let (idx, others) = select_rebuild_location(&infos).expect("a location with .ecx");
-        assert_eq!(idx, 1, "the fuller location wins");
-        assert_eq!(others, vec!["/data1".to_string(), "/idx1".to_string()]);
-    }
-
-    #[test]
-    fn select_rebuild_location_drops_empty_dirs() {
-        let infos = vec![loc("/data1", "", 3, true), loc("/data2", "", 1, false)];
-        let (_, others) = select_rebuild_location(&infos).expect("a location with .ecx");
-        assert_eq!(others, vec!["/data2".to_string()]);
-    }
-
-    // Nothing carries an .ecx: there is no index to rebuild the shards against,
-    // so the caller answers with an empty rebuild rather than guessing.
-    #[test]
-    fn select_rebuild_location_is_none_without_an_ecx() {
-        let infos = vec![loc("/data1", "/idx1", 3, false)];
-        assert!(select_rebuild_location(&infos).is_none());
-    }
-
     #[test]
     fn test_parse_grpc_address_with_explicit_grpc_port() {
         // Format: "ip:port.grpcPort" — used by SeaweedFS for source_data_node
@@ -6338,89 +6053,4 @@ mod tests {
         assert!(vif.expire_at_sec >= before + ttl.to_seconds());
         assert!(vif.expire_at_sec <= before + ttl.to_seconds() + 5);
     }
-
-    async fn scrub_ec_volume_1(
-        service: &VolumeGrpcService,
-        mode: i32,
-    ) -> volume_server_pb::ScrubEcVolumeResponse {
-        service
-            .scrub_ec_volume(Request::new(volume_server_pb::ScrubEcVolumeRequest {
-                mode,
-                volume_ids: vec![1],
-                force_deleted_needles_check: false,
-            }))
-            .await
-            .unwrap()
-            .into_inner()
-    }
-
-    #[tokio::test]
-    async fn test_scrub_ec_volume_reads_mode_reconstructs_missing_shard() {
-        let (service, _tmp) = make_local_service_with_volume("", None);
-        service
-            .volume_ec_shards_generate(Request::new(
-                volume_server_pb::VolumeEcShardsGenerateRequest {
-                    volume_id: 1,
-                    collection: String::new(),
-                },
-            ))
-            .await
-            .unwrap();
-        service
-            .volume_ec_shards_mount(Request::new(
-                volume_server_pb::VolumeEcShardsMountRequest {
-                    volume_id: 1,
-                    collection: String::new(),
-                    shard_ids: (0..14).collect(),
-                    source_disk_type: String::new(),
-                    recover_missing_index: false,
-                },
-            ))
-            .await
-            .unwrap();
-
-        // Seed the shard-location cache so the scrub skips the master lookup.
-        // The explicit-gRPC-port form targets port 1, so remote reads fail fast
-        // with connection-refused instead of resolving to a live local port.
-        {
-            let store = service.state.store.read().unwrap();
-            let ecv = store.find_ec_volume(VolumeId(1)).unwrap();
-            let mut locs = ecv.shard_locations.write().unwrap();
-            for sid in 0u8..14 {
-                locs.insert(sid, vec!["127.0.0.1:255.1".to_string()]);
-            }
-            *ecv.shard_locations_refresh_time.lock().unwrap() =
-                Some(std::time::Instant::now());
-        }
-
-        // All shards local: FULL is clean.
-        let resp = scrub_ec_volume_1(&service, 2).await;
-        assert!(resp.broken_volume_ids.is_empty(), "{:?}", resp.details);
-        assert_eq!(resp.total_files, 1);
-
-        service
-            .volume_ec_shards_unmount(Request::new(
-                volume_server_pb::VolumeEcShardsUnmountRequest {
-                    volume_id: 1,
-                    shard_ids: vec![0],
-                    encode_ts_ns: 0,
-                },
-            ))
-            .await
-            .unwrap();
-
-        // With a shard unreadable, FULL flags it AND fails the needle...
-        let resp = scrub_ec_volume_1(&service, 2).await;
-        assert_eq!(resp.broken_volume_ids, vec![1]);
-        assert!(resp.broken_shard_infos.iter().any(|s| s.shard_id == 0));
-        assert!(!resp.details.is_empty());
-
-        // ...while READS still reports the shard broken but reconstructs the
-        // interval from the local survivors, so no needle errors surface.
-        let resp = scrub_ec_volume_1(&service, 5).await;
-        assert_eq!(resp.broken_volume_ids, vec![1]);
-        assert!(resp.broken_shard_infos.iter().any(|s| s.shard_id == 0));
-        assert!(resp.details.is_empty(), "{:?}", resp.details);
-        assert_eq!(resp.total_files, 1);
-    }
 }

seaweed-volume/src/server/handlers.rs

@@ -1589,15 +1589,12 @@ async fn get_or_head_handler_inner(
 }
 
 /// Handle HTTP Range requests. Returns 206 Partial Content or 416 Range Not Satisfiable.
-#[derive(Clone, Copy, Debug)]
+#[derive(Clone, Copy)]
 struct HttpRange {
     start: i64,
     length: i64,
 }
 
-// Returned when the first-byte-pos of every byte-range-spec is at or past the content size.
-const RANGE_NO_OVERLAP: &str = "invalid range: failed to overlap";
-
 fn parse_range_header(s: &str, size: i64) -> Result<Vec<HttpRange>, &'static str> {
     if s.is_empty() {
         return Ok(Vec::new());
@@ -1607,7 +1604,6 @@ fn parse_range_header(s: &str, size: i64) -> Result<Vec<HttpRange>, &'static str
         return Err("invalid range");
     }
     let mut ranges = Vec::new();
-    let mut no_overlap = false;
     for part in s[PREFIX.len()..].split(',') {
         let part = part.trim();
         if part.is_empty() {
@@ -1631,13 +1627,9 @@ fn parse_range_header(s: &str, size: i64) -> Result<Vec<HttpRange>, &'static str
             r.length = size - r.start;
         } else {
             let i = start_str.parse::<i64>().map_err(|_| "invalid range")?;
-            if i < 0 {
+            if i > size || i < 0 {
                 return Err("invalid range");
             }
-            if i >= size {
-                no_overlap = true;
-                continue;
-            }
             r.start = i;
             if end_str.is_empty() {
                 r.length = size - r.start;
@@ -1654,9 +1646,6 @@ fn parse_range_header(s: &str, size: i64) -> Result<Vec<HttpRange>, &'static str
         }
         ranges.push(r);
     }
-    if no_overlap && ranges.is_empty() {
-        return Err(RANGE_NO_OVERLAP);
-    }
     Ok(ranges)
 }
 
@@ -1690,15 +1679,7 @@ fn handle_range_request(
     let total = data.len() as i64;
     let ranges = match parse_range_header(range_str, total) {
         Ok(r) => r,
-        Err(msg) => {
-            if msg == RANGE_NO_OVERLAP {
-                headers.insert(
-                    "Content-Range",
-                    format!("bytes */{}", total).parse().unwrap(),
-                );
-            }
-            return range_error_response(headers, msg);
-        }
+        Err(msg) => return range_error_response(headers, msg),
     };
 
     // Go's ProcessRangeRequest returns nil (empty body) for empty or oversized ranges
@@ -1781,15 +1762,7 @@ fn handle_range_request_from_source(
     let total = info.data_size as i64;
     let ranges = match parse_range_header(range_str, total) {
         Ok(r) => r,
-        Err(msg) => {
-            if msg == RANGE_NO_OVERLAP {
-                headers.insert(
-                    "Content-Range",
-                    format!("bytes */{}", total).parse().unwrap(),
-                );
-            }
-            return range_error_response(headers, msg);
-        }
+        Err(msg) => return range_error_response(headers, msg),
     };
 
     if ranges.is_empty() {
@@ -3939,25 +3912,6 @@ mod tests {
         assert!(parse_url_path("").is_none());
     }
 
-    #[test]
-    fn test_parse_range_header_no_overlap() {
-        assert_eq!(
-            parse_range_header("bytes=10-", 10).unwrap_err(),
-            RANGE_NO_OVERLAP
-        );
-        assert_eq!(
-            parse_range_header("bytes=100-", 10).unwrap_err(),
-            RANGE_NO_OVERLAP
-        );
-        // 416 only when every range fails to overlap
-        let ranges = parse_range_header("bytes=10-,0-1", 10).unwrap();
-        assert_eq!(ranges.len(), 1);
-        assert_eq!((ranges[0].start, ranges[0].length), (0, 2));
-        // an end past the size is clamped, still satisfiable
-        let ranges = parse_range_header("bytes=5-100", 10).unwrap();
-        assert_eq!((ranges[0].start, ranges[0].length), (5, 5));
-    }
-
     #[test]
     fn test_extract_jwt_bearer() {
         let mut headers = HeaderMap::new();

seaweed-volume/src/server/heartbeat.rs

@@ -398,7 +398,8 @@ async fn do_heartbeat(
 
     // Keep track of what we sent, to generate delta updates
     let (initial_hb, initial_volumes) = collect_heartbeat_with_snapshot(config, state);
-    let mut last_volumes: HashMap<u32, VolumeIdentity> = volume_identities(&initial_volumes);
+    let mut last_volumes: HashMap<u32, master_pb::VolumeInformationMessage> =
+        initial_volumes.iter().map(|v| (v.id, v.clone())).collect();
     let mut last_ec_shards = {
         let store = state.store.read().unwrap();
         collect_ec_shard_delta_messages(&store)
@@ -465,7 +466,8 @@ async fn do_heartbeat(
                         if changed {
                             let (adjusted_hb, adjusted_volumes) =
                                 collect_heartbeat_with_snapshot(config, state);
-                            last_volumes = volume_identities(&adjusted_volumes);
+                            last_volumes =
+                                adjusted_volumes.iter().map(|v| (v.id, v.clone())).collect();
                             last_ec_shards = {
                                 let store = state.store.read().unwrap();
                                 collect_ec_shard_delta_messages(&store)
@@ -499,7 +501,7 @@ async fn do_heartbeat(
                     s.maybe_adjust_volume_max();
                 }
                 let (current_hb, current_volumes) = collect_heartbeat_with_snapshot(config, state);
-                last_volumes = volume_identities(&current_volumes);
+                last_volumes = current_volumes.iter().map(|v| (v.id, v.clone())).collect();
                 last_ec_shards = {
                     let store = state.store.read().unwrap();
                     collect_ec_shard_delta_messages(&store)
@@ -528,7 +530,7 @@ async fn do_heartbeat(
                     return Ok(None);
                 }
                 let held_volumes = collect_volume_snapshot(config, state);
-                let current_volumes = volume_identities(&held_volumes);
+                let current_volumes: HashMap<u32, _> = held_volumes.iter().map(|v| (v.id, v.clone())).collect();
                 let current_ec_shards = {
                     let store = state.store.read().unwrap();
                     collect_ec_shard_delta_messages(&store)
@@ -539,13 +541,29 @@ async fn do_heartbeat(
 
                 for (id, vol) in &current_volumes {
                     if !last_volumes.contains_key(id) {
-                        new_vols.push(vol.to_short_message(*id));
+                        new_vols.push(master_pb::VolumeShortInformationMessage {
+                            id: *id,
+                            collection: vol.collection.clone(),
+                            version: vol.version,
+                            replica_placement: vol.replica_placement,
+                            ttl: vol.ttl,
+                            disk_type: vol.disk_type.clone(),
+                            disk_id: vol.disk_id,
+                        });
                     }
                 }
 
                 for (id, vol) in &last_volumes {
                     if !current_volumes.contains_key(id) {
-                        del_vols.push(vol.to_short_message(*id));
+                        del_vols.push(master_pb::VolumeShortInformationMessage {
+                            id: *id,
+                            collection: vol.collection.clone(),
+                            version: vol.version,
+                            replica_placement: vol.replica_placement,
+                            ttl: vol.ttl,
+                            disk_type: vol.disk_type.clone(),
+                            disk_id: vol.disk_id,
+                        });
                     }
                 }
 
@@ -726,54 +744,6 @@ fn parse_bool_property(value: Option<&String>) -> bool {
         .unwrap_or(true)
 }
 
-/// What a mount or unmount delta has to name, which is far less than the
-/// information message the heartbeat carries. A server holding millions of
-/// volumes cannot keep a whole message for each just to notice one leave; the
-/// Go report state keeps the same fields for the same reason.
-#[derive(Clone)]
-struct VolumeIdentity {
-    collection: String,
-    disk_type: String,
-    version: u32,
-    replica_placement: u32,
-    ttl: u32,
-    disk_id: u32,
-}
-
-impl VolumeIdentity {
-    fn of(v: &master_pb::VolumeInformationMessage) -> Self {
-        Self {
-            collection: v.collection.clone(),
-            disk_type: v.disk_type.clone(),
-            version: v.version,
-            replica_placement: v.replica_placement,
-            ttl: v.ttl,
-            disk_id: v.disk_id,
-        }
-    }
-
-    fn to_short_message(&self, id: u32) -> master_pb::VolumeShortInformationMessage {
-        master_pb::VolumeShortInformationMessage {
-            id,
-            collection: self.collection.clone(),
-            version: self.version,
-            replica_placement: self.replica_placement,
-            ttl: self.ttl,
-            disk_type: self.disk_type.clone(),
-            disk_id: self.disk_id,
-        }
-    }
-}
-
-fn volume_identities(
-    volumes: &[master_pb::VolumeInformationMessage],
-) -> HashMap<u32, VolumeIdentity> {
-    volumes
-        .iter()
-        .map(|v| (v.id, VolumeIdentity::of(v)))
-        .collect()
-}
-
 /// Collect volume information into a Heartbeat message.
 fn collect_heartbeat_with_snapshot(
     config: &HeartbeatConfig,
@@ -872,7 +842,8 @@ fn build_heartbeat_with_ec_status(
     // master can tell whether applying what it was sent leaves it current.
     // Volumes skipped below -- quarantined, phantom, expired -- are in neither.
     let mut volume_digest: u64 = 0;
-    let (send_full_list, report_generation, report_pass) = store.volume_report.begin();
+    let (send_full_list, report_generation) = store.volume_report.begin();
+    let mut reported_hashes: HashMap<VolumeReportKey, u64> = HashMap::new();
     let mut changed_volumes = Vec::new();
     let mut max_file_key = NeedleId(0);
     let mut max_volume_counts: HashMap<String, u32> = HashMap::new();
@@ -970,14 +941,8 @@ fn build_heartbeat_with_ec_status(
                 let hash = report_hash(&volume_message);
                 volume_digest ^= hash;
                 let key: VolumeReportKey = (volume_message.disk_id, volume_message.id);
-                // A snapshot must leave the reporting state as it found it, so
-                // it asks rather than marks.
-                let is_news = if commit_report {
-                    store.volume_report.record(key, hash, report_pass)
-                } else {
-                    store.volume_report.changed(key, hash)
-                };
-                if send_full_list || is_news {
+                reported_hashes.insert(key, hash);
+                if send_full_list || store.volume_report.changed(key, hash) {
                     changed_volumes.push(volume_message.clone());
                 }
                 volumes.push(volume_message);
@@ -986,30 +951,24 @@ fn build_heartbeat_with_ec_status(
                 should_delete_volume = true;
             }
 
-            // Track disk size by collection. A volume on its way out is left
-            // out: an entry here is also what says the collection is still on
-            // this server.
+            // Track disk size by collection
+            let entry = disk_sizes.entry(vol.collection.clone()).or_insert((0, 0));
             if !should_delete_volume {
-                let entry = disk_sizes.entry(vol.collection.clone()).or_insert((0, 0));
                 entry.0 += volume_size;
                 entry.1 += vol.deleted_size();
             }
 
-            // An entry here is what says the collection is still on this
-            // server, so a volume on its way out must not make one.
-            if !should_delete_volume {
-                let read_only = ro_counts.entry(vol.collection.clone()).or_default();
-                if vol.is_read_only() {
-                    read_only.is_read_only += 1;
-                    if vol.is_no_write_or_delete() {
-                        read_only.no_write_or_delete += 1;
-                    }
-                    if vol.is_no_write_can_delete() {
-                        read_only.no_write_can_delete += 1;
-                    }
-                    if loc.is_disk_space_low.load(Ordering::Relaxed) {
-                        read_only.is_disk_space_low += 1;
-                    }
+            let read_only = ro_counts.entry(vol.collection.clone()).or_default();
+            if !should_delete_volume && vol.is_read_only() {
+                read_only.is_read_only += 1;
+                if vol.is_no_write_or_delete() {
+                    read_only.no_write_or_delete += 1;
+                }
+                if vol.is_no_write_can_delete() {
+                    read_only.no_write_can_delete += 1;
+                }
+                if loc.is_disk_space_low.load(Ordering::Relaxed) {
+                    read_only.is_disk_space_low += 1;
                 }
             }
 
@@ -1043,17 +1002,6 @@ fn build_heartbeat_with_ec_status(
             .with_label_values(&[col, crate::metrics::READ_ONLY_LABEL_IS_DISK_SPACE_LOW])
             .set(counts.is_disk_space_low as f64);
     }
-    // ro_counts has an entry for every collection that kept a volume through
-    // this pass, including the ones counting zero read-only volumes.
-    {
-        let mut reported = store.reported_collections.lock().unwrap();
-        for col in reported.iter() {
-            if !ro_counts.contains_key(col) {
-                crate::metrics::delete_volume_server_collection_metrics(col);
-            }
-        }
-        *reported = ro_counts.keys().cloned().collect();
-    }
     // Update max volumes gauge
     let total_max: i64 = max_volume_counts.values().map(|v| *v as i64).sum();
     crate::metrics::MAX_VOLUMES.set(total_max);
@@ -1061,7 +1009,7 @@ fn build_heartbeat_with_ec_status(
     // Only when this heartbeat is going to be sent: marking volumes reported
     // and then discarding the message would leave the master never told.
     if commit_report {
-        store.volume_report.commit(report_pass, report_generation);
+        store.volume_report.commit(reported_hashes, report_generation);
     }
 
     // has_no_volumes says the server holds nothing, so it may only be derived
@@ -1129,14 +1077,6 @@ fn collect_live_ec_shards(
                 .with_label_values(&[col, crate::metrics::DISK_SIZE_LABEL_EC])
                 .set(*size as f64);
         }
-        let mut reported = store.reported_ec_collections.lock().unwrap();
-        for col in reported.iter() {
-            if !ec_sizes.contains_key(col) {
-                let _ = crate::metrics::DISK_SIZE_GAUGE
-                    .remove_label_values(&[col, crate::metrics::DISK_SIZE_LABEL_EC]);
-            }
-        }
-        *reported = ec_sizes.keys().cloned().collect();
     }
 
     ec_shards
@@ -1488,10 +1428,10 @@ mod tests {
         store.volume_report.accept_deltas();
         build_heartbeat(&test_config(), &mut store);
 
-        let (full, generation, pass) = store.volume_report.begin();
+        let (full, generation) = store.volume_report.begin();
         assert!(!full);
         store.volume_report.request_full_list();
-        store.volume_report.commit(pass, generation);
+        store.volume_report.commit(HashMap::new(), generation);
 
         let heartbeat = build_heartbeat(&test_config(), &mut store);
         assert_eq!(heartbeat.volumes.len(), 2);
@@ -1608,7 +1548,7 @@ mod tests {
             let (_, volume) = store.find_volume_mut(VolumeId(17)).unwrap();
             volume.set_read_only().unwrap();
             volume.volume_info.files.push(Default::default());
-            volume.refresh_remote_write_mode().unwrap();
+            volume.refresh_remote_write_mode();
         }
 
         let heartbeat = build_heartbeat(&test_config(), &mut store);
@@ -1656,82 +1596,6 @@ mod tests {
         );
     }
 
-    fn collection_series(gauge: &prometheus::GaugeVec, collection: &str) -> usize {
-        use prometheus::core::Collector;
-        gauge
-            .collect()
-            .iter()
-            .flat_map(|family| family.get_metric().to_vec())
-            .filter(|metric| {
-                metric
-                    .get_label()
-                    .iter()
-                    .any(|label| label.get_name() == "collection" && label.get_value() == collection)
-            })
-            .count()
-    }
-
-    // The per-collection gauges are only ever set for collections the heartbeat
-    // still finds on this server. A volume.balance that moves a collection's
-    // last volume off a server used to leave its read-only count - marked
-    // read-only for the move, moments before it went - standing on that server
-    // until a restart, with nothing in volume.list to match it.
-    #[test]
-    fn test_build_heartbeat_clears_metrics_of_departed_collection() {
-        let temp_dir = tempfile::tempdir().unwrap();
-        let dir = temp_dir.path().to_str().unwrap();
-        let collection = "heartbeat_departed_case";
-
-        let mut store = Store::new(NeedleMapKind::InMemory);
-        store
-            .add_location(
-                dir,
-                dir,
-                8,
-                DiskType::HardDrive,
-                MinFreeSpace::Percent(1.0),
-                Vec::new(),
-            )
-            .unwrap();
-        store
-            .add_volume(
-                VolumeId(21),
-                collection,
-                None,
-                None,
-                0,
-                DiskType::HardDrive,
-                Version::current(),
-            )
-            .unwrap();
-        {
-            let (_, volume) = store.find_volume_mut(VolumeId(21)).unwrap();
-            volume.set_read_only().unwrap();
-        }
-
-        build_heartbeat(&test_config(), &mut store);
-        assert_eq!(
-            READ_ONLY_VOLUME_GAUGE
-                .with_label_values(&[collection, READ_ONLY_LABEL_IS_READ_ONLY])
-                .get(),
-            1.0
-        );
-
-        assert!(store.unmount_volume(VolumeId(21)));
-        build_heartbeat(&test_config(), &mut store);
-
-        assert_eq!(
-            collection_series(&READ_ONLY_VOLUME_GAUGE, collection),
-            0,
-            "read-only series left after the collection left the server"
-        );
-        assert_eq!(
-            collection_series(&DISK_SIZE_GAUGE, collection),
-            0,
-            "disk size series left after the collection left the server"
-        );
-    }
-
     #[test]
     fn test_build_heartbeat_reports_disk_bytes() {
         let temp_dir = tempfile::tempdir().unwrap();
@@ -1974,7 +1838,7 @@ mod tests {
             key: "volumes/71.dat".to_string(),
             ..Default::default()
         });
-        volume.refresh_remote_write_mode().unwrap();
+        volume.refresh_remote_write_mode();
 
         let heartbeat = build_heartbeat(&test_config(), &mut store);
 

seaweed-volume/src/server/store_ec.rs

@@ -33,9 +33,7 @@ use std::sync::Arc;
 use std::time::{Duration, Instant};
 
 use futures::future::join_all;
-use futures::stream::{self, StreamExt};
 use reed_solomon_erasure::galois_8::ReedSolomon;
-use tokio::sync::Semaphore;
 use tonic::Request;
 
 use crate::pb::master_pb::{self, seaweed_client::SeaweedClient, LookupEcVolumeRequest};
@@ -51,19 +49,6 @@ use crate::storage::store_ec_reconcile::EcVolumeMissingIndex;
 use crate::storage::types::*;
 use crate::storage::volume::volume_file_name;
 
-/// Bounds the fan-out of a single needle read. Mirrors Go's
-/// `ecIntervalReadConcurrency`.
-const INTERVAL_READ_CONCURRENCY: usize = 8;
-
-/// Bounds the bytes EC recovery holds in flight across every concurrent read.
-/// Recovery is the one read path that multiplies the served bytes — it keeps an
-/// interval-sized buffer per shard alive until Reed-Solomon runs — and a peer
-/// that is slow to fail holds each of them for the whole gRPC timeout, so a
-/// burst of reads during a network blip walked the server into an OOM. Mirrors
-/// Go's `ecRecoverBudget`.
-const EC_RECOVER_BUDGET: usize = 256 << 20;
-static EC_RECOVER_SEM: Semaphore = Semaphore::const_new(EC_RECOVER_BUDGET);
-
 /// One interval's data after Phase A.
 enum IntervalResult {
     /// Already read from a locally-mounted shard.
@@ -105,7 +90,7 @@ pub async fn read_ec_shard_needle_distributed(
     // intervals, and read any locally-mounted shard intervals. We must
     // not `.await` while holding this guard (std::sync::RwLockReadGuard
     // is !Send).
-    let mut snapshot = match snapshot_under_lock(state, vid, needle_id)? {
+    let snapshot = match snapshot_under_lock(state, vid, needle_id)? {
         Some(s) => s,
         None => return Ok(None),
     };
@@ -121,9 +106,7 @@ pub async fn read_ec_shard_needle_distributed(
 
     let mut shard_locations = snapshot.cached_locations.clone();
     if any_remote
-        && claim_shard_locations_refresh(
-            state,
-            vid,
+        && needs_refresh(
             &shard_locations,
             snapshot.cache_refreshed_at,
             snapshot.data_shards as usize,
@@ -134,21 +117,17 @@ pub async fn read_ec_shard_needle_distributed(
             Ok(fresh) => {
                 // A complete reply merges into the cache; an incomplete one
                 // (< data_shards) is left unwritten — keep the prior cache.
-                match write_back_shard_locations(state, vid, fresh, snapshot.data_shards as usize)
+                if let Some(merged) =
+                    write_back_shard_locations(state, vid, fresh, snapshot.data_shards as usize)
                 {
-                    Some(merged) => shard_locations = merged,
-                    // An incomplete reply leaves the cache unwritten and its refresh
-                    // time unadvanced, so the mark this refresh consumed goes back.
-                    None => mark_shard_locations_stale(state, vid),
+                    shard_locations = merged;
                 }
             }
             Err(e) => {
                 // Lookup failed — proceed with cached values. If cache
                 // is empty, the remote fetch below will fail and we
                 // surface a NotFound (matching Go's behavior when no
-                // locations are known). The mark goes back: nothing
-                // answered for it, and the map stays disproved.
-                mark_shard_locations_stale(state, vid);
+                // locations are known).
                 tracing::warn!(
                     "ec lookup failed for volume {}: {} — using cached locations ({} entries)",
                     vid.0,
@@ -159,56 +138,39 @@ pub async fn read_ec_shard_needle_distributed(
         }
     }
 
-    // Phase C — fetch missing intervals, reconstructing when the direct peer
-    // read fails. Blocks that follow each other in the .dat live on different
-    // shards, so a needle spanning several of them costs one round trip per
-    // block when fetched in sequence; `buffered` keeps the order while letting
-    // INTERVAL_READ_CONCURRENCY of them fly at once.
-    let data_shards = snapshot.data_shards as usize;
-    let parity_shards = snapshot.parity_shards as usize;
-    let encode_ts_ns = snapshot.encode_ts_ns;
-    let intervals = std::mem::take(&mut snapshot.intervals);
-    let fetched: Vec<io::Result<(Vec<u8>, bool)>> = stream::iter(intervals.into_iter().map(|res| {
-        let shard_locations = &shard_locations;
-        async move {
-            match res {
-                IntervalResult::Local(buf) => Ok((buf, false)),
-                IntervalResult::NeedRemote {
+    // Phase C — fetch missing intervals, reconstructing when the
+    // direct peer read fails.
+    let mut assembled: Vec<Vec<u8>> = Vec::with_capacity(snapshot.intervals.len());
+    for res in snapshot.intervals {
+        match res {
+            IntervalResult::Local(buf) => assembled.push(buf),
+            IntervalResult::NeedRemote {
+                shard_id,
+                shard_offset,
+                size,
+            } => {
+                let (buf, is_deleted) = fetch_one_interval(
+                    state,
+                    vid,
+                    needle_id,
                     shard_id,
                     shard_offset,
                     size,
-                } => {
-                    fetch_one_interval(
-                        state,
-                        vid,
-                        needle_id,
-                        shard_id,
-                        shard_offset,
-                        size,
-                        shard_locations,
-                        data_shards,
-                        parity_shards,
-                        encode_ts_ns,
-                    )
-                    .await
+                    &shard_locations,
+                    snapshot.data_shards as usize,
+                    snapshot.parity_shards as usize,
+                    snapshot.encode_ts_ns,
+                )
+                .await?;
+                // A peer reports the needle deleted (a cross-server window where the
+                // local index still shows it live): treat as not-found rather than
+                // serving zeros, mirroring Go's ErrorDeleted.
+                if is_deleted {
+                    return Ok(None);
                 }
+                assembled.push(buf);
             }
         }
-    }))
-    .buffered(INTERVAL_READ_CONCURRENCY)
-    .collect()
-    .await;
-
-    let mut assembled: Vec<Vec<u8>> = Vec::with_capacity(fetched.len());
-    for res in fetched {
-        let (buf, is_deleted) = res?;
-        // A peer reports the needle deleted (a cross-server window where the
-        // local index still shows it live): treat as not-found rather than
-        // serving zeros, mirroring Go's ErrorDeleted.
-        if is_deleted {
-            return Ok(None);
-        }
-        assembled.push(buf);
     }
 
     // Phase D — assemble and parse the Needle. Mirrors the tail of
@@ -246,9 +208,7 @@ pub async fn read_ec_shard_needle_distributed(
 /// without decoding (so genuine shard faults are reported rather than healed).
 /// Mirrors Go's `Store.ScrubEcVolume`. Returns (rows walked, broken shards,
 /// errors). `force_deleted_needles_check` disables the benign delete-state
-/// size-mismatch suppression. `recover_unreadable` (READS mode) rebuilds an
-/// unreadable interval from the surviving shards: the same shards are reported
-/// broken, but only needles parity can no longer recover become errors.
+/// size-mismatch suppression.
 ///
 /// Shard locations are refreshed once up front. Each needle is then processed via
 /// `scrub_snapshot_under_lock` + lock-drop + no-reconstruct `read_remote_ec_shard_interval`,
@@ -257,19 +217,10 @@ pub async fn scrub_ec_volume_distributed(
     state: &Arc<VolumeServerState>,
     vid: VolumeId,
     force_deleted_needles_check: bool,
-    recover_unreadable: bool,
 ) -> (i64, Vec<crate::pb::volume_server_pb::EcShardInfo>, Vec<String>) {
     // Phase A — under the Store read lock, run the index scrub and grab the
     // paths/scalars + shard-location staleness; release the lock before any await.
-    let (
-        ecx_path,
-        collection,
-        seed_errs,
-        cached_locations,
-        cache_refreshed_at,
-        data_shards,
-        total_shards,
-    ) = {
+    let (ecx_path, collection, seed_errs, cached_locations, cache_refreshed_at, data_shards, total_shards) = {
         let store = state.store.read().unwrap();
         let ecv = match store.find_ec_volume(vid) {
             Some(v) => v,
@@ -304,18 +255,10 @@ pub async fn scrub_ec_volume_distributed(
     // cachedLookupEcShardLocations). A partial reply (< data_shards locations, a
     // master mid-recovery) or a failed lookup is a hard, retryable error — never
     // overwrite a good cache with a partial map or storm a down master per needle.
-    if claim_shard_locations_refresh(
-        state,
-        vid,
-        &cached_locations,
-        cache_refreshed_at,
-        data_shards,
-        total_shards,
-    ) {
+    if needs_refresh(&cached_locations, cache_refreshed_at, data_shards, total_shards) {
         match cached_lookup_ec_shard_locations(state, vid).await {
             Ok(fresh) => {
                 if write_back_shard_locations(state, vid, fresh, data_shards).is_none() {
-                    mark_shard_locations_stale(state, vid);
                     return (
                         0,
                         Vec::new(),
@@ -327,12 +270,11 @@ pub async fn scrub_ec_volume_distributed(
                 }
             }
             Err(e) => {
-                mark_shard_locations_stale(state, vid);
                 return (
                     0,
                     Vec::new(),
                     vec![format!("failed to locate shard via master grpc: {}", e)],
-                );
+                )
             }
         }
     }
@@ -398,9 +340,8 @@ pub async fn scrub_ec_volume_distributed(
             }
         };
 
-        // Read each interval local-then-remote. Neither read decodes: the point is to
-        // find shards that are themselves broken, not to heal around them. READS then
-        // rebuilds what it could not read. Locations refreshed above.
+        // Read each interval local-then-remote WITHOUT reconstructing: we verify
+        // the shards are valid, we do not heal them. Locations refreshed above.
         let n_intervals = snapshot.intervals.len();
         let mut data: Vec<u8> = Vec::with_capacity(snapshot.actual_size);
         for (i, res) in snapshot.intervals.iter().enumerate() {
@@ -430,9 +371,11 @@ pub async fn scrub_ec_volume_distributed(
                         // -> the delete-state suppression (mirrors Go's pre-zeroed buffer).
                         Ok((_, true)) => data.resize(data.len() + *ssize, 0),
                         Ok((buf, false)) => data.extend_from_slice(&buf),
-                        Err(read_err) => {
-                            // The shard is broken whether or not the needle survives it,
-                            // so report it either way.
+                        Err(_) => {
+                            errs.push(format!(
+                                "failed to read EC shard {} for needle {} on volume {} (interval {}/{})",
+                                shard_id, id.0, vid.0, i + 1, n_intervals
+                            ));
                             broken_shards.insert(
                                 *shard_id,
                                 crate::pb::volume_server_pb::EcShardInfo {
@@ -443,41 +386,7 @@ pub async fn scrub_ec_volume_distributed(
                                     ..Default::default()
                                 },
                             );
-                            if !recover_unreadable {
-                                errs.push(format!(
-                                    "failed to read EC shard {} for needle {} on volume {} (interval {}/{}): {}",
-                                    shard_id, id.0, vid.0, i + 1, n_intervals, read_err
-                                ));
-                                break;
-                            }
-                            match recover_one_remote_ec_shard_interval(
-                                state,
-                                vid,
-                                id,
-                                *shard_id,
-                                *shard_offset,
-                                *ssize,
-                                &locations,
-                                data_shards,
-                                total_shards - data_shards,
-                                snapshot.encode_ts_ns,
-                            )
-                            .await
-                            {
-                                // Same as the direct read above: a holder reporting the
-                                // needle deleted is authoritative and answers with no
-                                // bytes, so zero-fill and let the delete-state
-                                // suppression have it.
-                                Ok((_, true)) => data.resize(data.len() + *ssize, 0),
-                                Ok((buf, false)) => data.extend_from_slice(&buf),
-                                Err(e) => {
-                                    errs.push(format!(
-                                        "failed to recover EC shard {} for needle {} on volume {} (interval {}/{}): {}",
-                                        shard_id, id.0, vid.0, i + 1, n_intervals, e
-                                    ));
-                                    break;
-                                }
-                            }
+                            break;
                         }
                     }
                 }
@@ -599,7 +508,7 @@ fn read_local_intervals(
 ) -> Vec<IntervalResult> {
     let mut interval_results = Vec::with_capacity(intervals.len());
     for interval in intervals {
-        let (shard_id, shard_offset) = ecv.interval_to_shard_id_and_offset(interval);
+        let (shard_id, shard_offset) = interval.to_shard_id_and_offset(ecv.data_shards);
         let buf_size = interval.size as usize;
         let local = ecv.shards.get(shard_id as usize).and_then(|s| s.as_ref());
         match local {
@@ -662,7 +571,6 @@ fn build_snapshot(
 fn needs_refresh(
     locations: &HashMap<ShardId, Vec<String>>,
     refreshed_at: Option<Instant>,
-    stale: bool,
     data_shards: usize,
     total_shards: usize,
 ) -> bool {
@@ -672,53 +580,16 @@ fn needs_refresh(
         None => return true,
     };
     let shard_count = locations.len();
-    // A complete map is trusted longest. One short of data_shards, or one a
-    // failed read has just disproved, is re-checked promptly: until it is, reads
-    // keep aiming at a location the shard has left.
-    let ttl = if stale || shard_count < data_shards {
-        Duration::from_secs(11)
-    } else if shard_count == total_shards {
-        Duration::from_secs(37 * 60)
-    } else {
-        Duration::from_secs(7 * 60)
-    };
-    age >= ttl
-}
-
-/// Mark the cached shard map for a prompt re-check after a read failed against
-/// one of its locations. Go drops the entry outright in `forgetShardId`, which
-/// costs it the direct read until the map is re-learned; here the entry stays
-/// (a dead peer just fails fast on the next attempt) and only the freshness
-/// window is cut, so a shard that has moved is picked up in seconds either way.
-fn mark_shard_locations_stale(state: &Arc<VolumeServerState>, vid: VolumeId) {
-    let store = state.store.read().unwrap();
-    if let Some(ecv) = store.find_ec_volume(vid) {
-        *ecv.shard_locations_stale.lock().unwrap() = true;
+    if shard_count < data_shards && age < Duration::from_secs(11) {
+        return false;
     }
-}
-
-/// Decide whether the cached map is due a master lookup and, when it is, consume
-/// its stale mark in the same critical section. A mark raised from here on
-/// belongs to the next refresh: the read that raised it has disproved the map
-/// this lookup is about to install.
-fn claim_shard_locations_refresh(
-    state: &Arc<VolumeServerState>,
-    vid: VolumeId,
-    locations: &HashMap<ShardId, Vec<String>>,
-    refreshed_at: Option<Instant>,
-    data_shards: usize,
-    total_shards: usize,
-) -> bool {
-    let store = state.store.read().unwrap();
-    let Some(ecv) = store.find_ec_volume(vid) else {
-        return needs_refresh(locations, refreshed_at, false, data_shards, total_shards);
-    };
-    let mut stale = ecv.shard_locations_stale.lock().unwrap();
-    let refresh = needs_refresh(locations, refreshed_at, *stale, data_shards, total_shards);
-    if refresh {
-        *stale = false;
+    if shard_count == total_shards && age < Duration::from_secs(37 * 60) {
+        return false;
     }
-    refresh
+    if shard_count >= data_shards && age < Duration::from_secs(7 * 60) {
+        return false;
+    }
+    true
 }
 
 async fn cached_lookup_ec_shard_locations(
@@ -854,9 +725,6 @@ async fn fetch_one_interval(
                         sources,
                         e
                     );
-                    // Reconstruction below skips this very shard, so nothing else
-                    // invalidates the location that just failed.
-                    mark_shard_locations_stale(state, vid);
                 }
             }
         }
@@ -864,7 +732,7 @@ async fn fetch_one_interval(
 
     // Reconstruct: fan-out reads to every other shard at the same
     // (shard_offset, size). Mirrors `recoverOneRemoteEcShardInterval`.
-    recover_one_remote_ec_shard_interval(
+    let buf = recover_one_remote_ec_shard_interval(
         state,
         vid,
         needle_id,
@@ -876,7 +744,8 @@ async fn fetch_one_interval(
         parity_shards,
         expected_encode_ts_ns,
     )
-    .await
+    .await?;
+    Ok((buf, false))
 }
 
 async fn read_remote_ec_shard_interval(
@@ -1036,7 +905,7 @@ async fn recover_one_remote_ec_shard_interval(
     data_shards: usize,
     parity_shards: usize,
     expected_encode_ts_ns: i64,
-) -> io::Result<(Vec<u8>, bool)> {
+) -> io::Result<Vec<u8>> {
     let total_shards = data_shards + parity_shards;
     let rs = ReedSolomon::new(data_shards, parity_shards).map_err(|e| {
         io::Error::new(
@@ -1045,136 +914,89 @@ async fn recover_one_remote_ec_shard_interval(
         )
     })?;
 
-    // Charge the buffers this recovery is about to hold against the budget, so a
-    // burst of them queues here rather than on the heap. An interval whose
-    // fan-out outgrows the whole budget takes all of it and so runs alone,
-    // rather than waiting on permits that can never be granted.
-    let _permit = EC_RECOVER_SEM
-        .acquire_many((size * data_shards).min(EC_RECOVER_BUDGET) as u32)
-        .await
-        .map_err(|e| {
-            io::Error::new(
-                io::ErrorKind::Other,
-                format!(
-                    "ec recover budget for shard {}.{}: {}",
-                    vid.0, shard_id_to_recover, e
-                ),
-            )
-        })?;
-
     let mut bufs: Vec<Option<Vec<u8>>> = vec![None; total_shards];
 
     // Phase 0: seed bufs from LOCALLY mounted shards. If this node
     // already holds enough sibling shards, reconstruction completes
     // without any peer fan-out — and even with a cold/incomplete
     // shard_locations cache or a failed master lookup, local
-    // survivors still contribute.
-    let mut available = 0usize;
+    // survivors still contribute. Mirrors Go's
+    // recoverOneRemoteEcShardInterval behaviour, which is implicitly
+    // local-aware because the Store fan-out targets ALL known
+    // locations (including the caller's own server address); the
+    // Rust port had been remote-only, so reconstructing with a cold
+    // cache failed even when enough siblings were on disk.
     {
         let store = state.store.read().unwrap();
-        for sid in 0..total_shards {
-            if available >= data_shards {
-                break;
-            }
-            if sid as ShardId == shard_id_to_recover {
-                continue;
-            }
-            // Resolve the shard together with the EcVolume on the disk that owns
-            // it: a reconciled volume has its shards split across data dirs. A
-            // shard from a different encode run must not be fed to Reed-Solomon;
-            // lenient only when the caller carries no identity (pre-upgrade).
-            // Mirrors Go's `readLocalEcShardInterval`.
-            let owner = match store.find_ec_volume_with_shard(vid, sid as u32) {
-                Some(ecv) if expected_encode_ts_ns == 0 || ecv.encode_ts_ns == expected_encode_ts_ns => ecv,
-                _ => continue,
-            };
-            if let Some(Some(shard)) = owner.shards.get(sid) {
-                let mut buf = vec![0u8; size];
-                if shard.read_at(&mut buf, shard_offset as u64).map(|n| n == size).unwrap_or(false) {
-                    bufs[sid] = Some(buf);
-                    available += 1;
+        if let Some(ecv) = store.find_ec_volume(vid) {
+            for sid in 0..total_shards {
+                if sid as ShardId == shard_id_to_recover {
+                    continue;
                 }
-            }
-        }
-    }
-
-    // Phase 1: remote fan-out over the shard locations we DON'T already have
-    // locally and DON'T need to recover. Reconstruction consumes data_shards
-    // shards, so reading every remaining one holds a third more buffers than
-    // that and asks a third more of peers that may already be struggling: fetch
-    // what is still missing, and widen only if some of those reads fail.
-    let mut candidates: Vec<(ShardId, Vec<String>)> = shard_locations
-        .iter()
-        .filter(|(sid, locs)| {
-            **sid != shard_id_to_recover
-                && (**sid as usize) < total_shards
-                && !locs.is_empty()
-                && bufs[**sid as usize].is_none()
-        })
-        .map(|(sid, locs)| (*sid, locs.clone()))
-        .collect();
-
-    let mut any_deleted = false;
-    while available < data_shards && !candidates.is_empty() {
-        let rest = candidates.split_off((data_shards - available).min(candidates.len()));
-        let wave = std::mem::replace(&mut candidates, rest);
-        let results = join_all(wave.into_iter().map(|(sid, locs)| {
-            let state = state.clone();
-            async move {
-                let res = read_remote_ec_shard_interval(
-                    &state,
-                    &locs,
-                    vid,
-                    needle_id,
-                    sid,
-                    shard_offset,
-                    size,
-                    expected_encode_ts_ns,
-                )
-                .await;
-                (sid, res)
-            }
-        }))
-        .await;
-
-        for (sid, res) in results {
-            match res {
-                // Exclude a deleted shard from reconstruction (Go gates on a full
-                // read): feeding the empty/zero buffer into Reed-Solomon would
-                // corrupt the recovered shard.
-                Ok((buf, is_deleted)) => {
-                    if is_deleted {
-                        any_deleted = true;
-                        continue;
+                if let Some(Some(shard)) = ecv.shards.get(sid) {
+                    let mut buf = vec![0u8; size];
+                    if shard.read_at(&mut buf, shard_offset as u64).map(|n| n == size).unwrap_or(false) {
+                        bufs[sid] = Some(buf);
                     }
-                    bufs[sid as usize] = Some(buf);
-                    available += 1;
-                }
-                Err(e) => {
-                    tracing::debug!(
-                        "recover: read {}.{} for needle {} failed: {}",
-                        vid.0,
-                        sid,
-                        needle_id,
-                        e
-                    );
-                    mark_shard_locations_stale(state, vid);
                 }
             }
         }
-        if any_deleted {
-            // every shard of a deleted needle answers deleted, so another wave cannot help
-            break;
+    }
+
+    // Phase 1: remote fan-out — one task per known shard location
+    // we DON'T already have locally and DON'T need to recover.
+    let mut tasks = Vec::new();
+    for (sid, locs) in shard_locations {
+        if *sid == shard_id_to_recover || locs.is_empty() {
+            continue;
+        }
+        if bufs[*sid as usize].is_some() {
+            continue;
+        }
+        let sid = *sid;
+        let locs = locs.clone();
+        let state = state.clone();
+        tasks.push(async move {
+            let res = read_remote_ec_shard_interval(
+                &state,
+                &locs,
+                vid,
+                needle_id,
+                sid,
+                shard_offset,
+                size,
+                expected_encode_ts_ns,
+            )
+            .await;
+            (sid, res)
+        });
+    }
+    let results = join_all(tasks).await;
+
+    for (sid, res) in results {
+        match res {
+            // Exclude a deleted shard from reconstruction (Go gates on a full
+            // read): feeding the empty/zero buffer into Reed-Solomon would
+            // corrupt the recovered shard.
+            Ok((buf, is_deleted)) => {
+                if !is_deleted && (sid as usize) < total_shards {
+                    bufs[sid as usize] = Some(buf);
+                }
+            }
+            Err(e) => {
+                tracing::debug!(
+                    "recover: read {}.{} for needle {} failed: {}",
+                    vid.0,
+                    sid,
+                    needle_id,
+                    e
+                );
+            }
         }
     }
 
+    let available = bufs.iter().filter(|b| b.is_some()).count();
     if available < data_shards {
-        // A holder reporting the needle deleted is authoritative -- deletes are
-        // never invented and never undone -- so answer that rather than the
-        // failure to gather shards of a needle that is gone.
-        if any_deleted {
-            return Ok((Vec::new(), true));
-        }
         return Err(io::Error::new(
             io::ErrorKind::Other,
             format!(
@@ -1195,7 +1017,7 @@ async fn recover_one_remote_ec_shard_interval(
     })?;
 
     match bufs.into_iter().nth(shard_id_to_recover as usize).flatten() {
-        Some(buf) => Ok((buf, any_deleted)),
+        Some(buf) => Ok(buf),
         None => Err(io::Error::new(
             io::ErrorKind::Other,
             format!(
@@ -1433,34 +1255,3 @@ async fn drain_copy_stream(
     }
     Ok(())
 }
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    fn locations(count: usize) -> HashMap<ShardId, Vec<String>> {
-        (0..count)
-            .map(|sid| (sid as ShardId, vec!["127.0.0.1:8080".to_string()]))
-            .collect()
-    }
-
-    #[test]
-    fn needs_refresh_re_checks_a_map_a_failed_read_disproved() {
-        let just_now = Some(Instant::now());
-        let aged = Some(Instant::now() - Duration::from_secs(12));
-
-        // A complete map is trusted for a long time, and one shard short still
-        // outlasts a 12-second gap.
-        assert!(!needs_refresh(&locations(14), aged, false, 10, 14));
-        assert!(!needs_refresh(&locations(13), aged, false, 10, 14));
-        // Disproved by a read, the same maps are re-checked within seconds.
-        assert!(needs_refresh(&locations(14), aged, true, 10, 14));
-        assert!(needs_refresh(&locations(13), aged, true, 10, 14));
-        // But the mark buys one prompt re-check, not a lookup per read.
-        assert!(!needs_refresh(&locations(14), just_now, true, 10, 14));
-        // A map short of the data shards is re-checked promptly regardless.
-        assert!(needs_refresh(&locations(9), aged, false, 10, 14));
-        // An unrefreshed cache always looks up.
-        assert!(needs_refresh(&locations(0), None, false, 10, 14));
-    }
-}

seaweed-volume/src/storage/disk_location.rs

@@ -843,8 +843,7 @@ impl DiskLocation {
         // .ecx open error, .ecj create error, malformed .vif) would
         // have to panic via unwrap(). Build the EcVolume up front and
         // propagate the error to the caller.
-        let created = !self.ec_volumes.contains_key(&vid);
-        if created {
+        if !self.ec_volumes.contains_key(&vid) {
             let ec_vol = EcVolume::new(&dir, idx_dir, collection, vid)
                 .map_err(VolumeError::Io)?;
             self.ec_volumes.insert(vid, ec_vol);
@@ -867,27 +866,9 @@ impl DiskLocation {
         }
 
         for &shard_id in shard_ids {
-            // A mount retry re-listing a shard this volume already holds:
-            // keep the existing registration (mirrors Go's AddEcVolumeShard
-            // added=false) — re-adding would replace a serving fd and bump
-            // the ec_shards gauge without growing the mounted count.
-            if ec_vol.has_shard(shard_id as u8) {
-                continue;
-            }
             let mut shard = EcVolumeShard::new(&dir, collection, vid, shard_id as u8);
             shard.disk_type = ec_vol.disk_type.clone();
-            if let Err(e) = ec_vol.add_shard(shard) {
-                // The shard was dropped (its descriptors closed) inside the
-                // failed add. If this call just created the EcVolume and it
-                // holds nothing, remove it too — a zero-shard registration
-                // would advertise a mount that serves no data while pinning
-                // its descriptors.
-                let now_empty = ec_vol.shard_count() == 0;
-                if created && now_empty {
-                    self.ec_volumes.remove(&vid);
-                }
-                return Err(VolumeError::Io(e));
-            }
+            ec_vol.add_shard(shard).map_err(VolumeError::Io)?;
             crate::metrics::VOLUME_GAUGE
                 .with_label_values(&[collection, "ec_shards"])
                 .inc();
@@ -940,32 +921,27 @@ impl DiskLocation {
         // double-count for those filenames.
         let mut seen: HashSet<String> = HashSet::new();
         let mut entries: Vec<String> = Vec::new();
-        // Keep only the shard and index files this scan acts on: a disk of
-        // regular volumes has millions of .dat/.idx/.vif names that would
-        // otherwise each cost a String here and a slot in the sort below.
-        let mut collect = |dir: &str| -> io::Result<()> {
-            for ent in fs::read_dir(dir)? {
+        for ent in fs::read_dir(&self.directory)? {
+            let ent = ent?;
+            if ent.file_type().map(|ft| ft.is_dir()).unwrap_or(false) {
+                continue;
+            }
+            let name = ent.file_name().to_string_lossy().into_owned();
+            if seen.insert(name.clone()) {
+                entries.push(name);
+            }
+        }
+        if self.idx_directory != self.directory {
+            for ent in fs::read_dir(&self.idx_directory)? {
                 let ent = ent?;
                 if ent.file_type().map(|ft| ft.is_dir()).unwrap_or(false) {
                     continue;
                 }
                 let name = ent.file_name().to_string_lossy().into_owned();
-                let Some(dot) = name.rfind('.') else {
-                    continue;
-                };
-                let ext = &name[dot..];
-                if parse_ec_shard_extension(ext).is_none() && ext != ".ecx" {
-                    continue;
-                }
                 if seen.insert(name.clone()) {
                     entries.push(name);
                 }
             }
-            Ok(())
-        };
-        collect(&self.directory)?;
-        if self.idx_directory != self.directory {
-            collect(&self.idx_directory)?;
         }
         entries.sort();
 
@@ -1798,94 +1774,6 @@ mod tests {
         }
     }
 
-    /// A refused shard (a 0-byte file beside an index with entries) on the
-    /// FIRST mount of a volume must not leave the just-created zero-shard
-    /// EcVolume registered — it would advertise a mount serving no data,
-    /// pin the .ecx/.ecj descriptors, and make placement's mounted tier
-    /// prefer this disk. A volume that already holds shards keeps them
-    /// (the mount RPC's pre-existing first-error-aborts contract).
-    #[test]
-    fn test_mount_ec_shards_refused_shard_removes_created_empty_volume() {
-        let tmp = TempDir::new().unwrap();
-        let dir = tmp.path().to_str().unwrap();
-        let mut loc = DiskLocation::new(
-            dir,
-            dir,
-            10,
-            DiskType::HardDrive,
-            MinFreeSpace::Percent(1.0),
-            Vec::new(),
-        )
-        .unwrap();
-
-        // An index with one 16-byte entry and a 0-byte shard: the mount is
-        // refused, and the EcVolume created for it must be unregistered.
-        std::fs::write(format!("{}/pics_9.ecx", dir), [0u8; 16]).unwrap();
-        std::fs::write(format!("{}/pics_9.ec00", dir), b"").unwrap();
-        let err = loc
-            .mount_ec_shards(VolumeId(9), "pics", &[0], "")
-            .expect_err("a 0-byte shard beside an index with entries must refuse the mount");
-        assert!(
-            err.to_string().contains("empty (0 bytes)"),
-            "want the empty-shard refusal, got: {}",
-            err
-        );
-        assert!(
-            loc.find_ec_volume(VolumeId(9)).is_none(),
-            "a refused first mount must not leave a zero-shard EcVolume registered",
-        );
-
-        // With a valid shard mounted, a later refused shard keeps the
-        // existing registration intact.
-        std::fs::write(format!("{}/pics_9.ec01", dir), b"good bytes").unwrap();
-        loc.mount_ec_shards(VolumeId(9), "pics", &[1], "").unwrap();
-        loc.mount_ec_shards(VolumeId(9), "pics", &[0], "")
-            .expect_err("the 0-byte shard stays refused");
-        assert_eq!(
-            loc.find_ec_volume(VolumeId(9)).map(|v| v.shard_count()),
-            Some(1),
-            "an existing volume keeps its valid shards when a later shard is refused",
-        );
-    }
-
-    /// A mount retry re-listing an already mounted shard must keep the
-    /// existing registration and not bump the ec_shards gauge — the Rust
-    /// twin of Go's AddEcVolumeShard added=false handling.
-    #[test]
-    fn test_mount_ec_shards_duplicate_keeps_registration_and_gauge() {
-        let tmp = TempDir::new().unwrap();
-        let dir = tmp.path().to_str().unwrap();
-        let mut loc = DiskLocation::new(
-            dir,
-            dir,
-            10,
-            DiskType::HardDrive,
-            MinFreeSpace::Percent(1.0),
-            Vec::new(),
-        )
-        .unwrap();
-
-        // A collection name unique to this test: the gauge is process-global
-        // and sibling tests running in parallel touch other labels.
-        std::fs::write(format!("{}/dupmount_11.ec00", dir), b"shard bytes").unwrap();
-        let gauge = crate::metrics::VOLUME_GAUGE.with_label_values(&["dupmount", "ec_shards"]);
-        let before = gauge.get();
-
-        loc.mount_ec_shards(VolumeId(11), "dupmount", &[0], "").unwrap();
-        loc.mount_ec_shards(VolumeId(11), "dupmount", &[0], "")
-            .expect("a duplicate mount must succeed as a no-op");
-
-        assert_eq!(
-            loc.find_ec_volume(VolumeId(11)).map(|v| v.shard_count()),
-            Some(1),
-        );
-        assert_eq!(
-            gauge.get(),
-            before + 1.0,
-            "the duplicate mount must not bump the ec_shards gauge",
-        );
-    }
-
     #[test]
     fn test_disk_location_persists_directory_uuid_and_tags() {
         let tmp = TempDir::new().unwrap();

seaweed-volume/src/storage/erasure_coding/ec_bitrot.rs

@@ -465,28 +465,6 @@ pub fn resolve_status(
     }
 }
 
-/// Whether a generation-matching sidecar agrees with the geometry the volume is
-/// mounted with. Both files record the layout the generation was encoded with,
-/// so a disagreement means one of them is wrong and reads through the other
-/// would land at the wrong shard offsets — the caller fails the mount rather
-/// than merely dropping protection. A sidecar that records no EC config has
-/// nothing to contradict.
-pub fn geometry_matches(
-    prot: &EcBitrotProtection,
-    data_shards: usize,
-    parity_shards: usize,
-    block_size: i64,
-) -> bool {
-    match &prot.ec_shard_config {
-        None => true,
-        Some(cfg) => {
-            cfg.data_shards as usize == data_shards
-                && cfg.parity_shards as usize == parity_shards
-                && cfg.block_size == block_size
-        }
-    }
-}
-
 /// Returns the [`EcShardChecksums`] entry for a shard id, or `None`.
 pub fn shard_checksums(prot: &EcBitrotProtection, shard_id: u32) -> Option<&EcShardChecksums> {
     prot.shards.iter().find(|s| s.shard_id == shard_id)
@@ -561,12 +539,11 @@ fn read_full_at(f: &File, buf: &mut [u8], offset: u64) -> io::Result<()> {
 
 /// Builds the `EcShardConfig` proto for the given layout. The bitrot sidecar
 /// carries its own top-level encode_uuid, so the nested config leaves it empty.
-pub fn ec_shard_config(data_shards: u32, parity_shards: u32, block_size: i64) -> EcShardConfig {
+pub fn ec_shard_config(data_shards: u32, parity_shards: u32) -> EcShardConfig {
     EcShardConfig {
         data_shards,
         parity_shards,
         encode_ts_ns: 0,
-        block_size,
     }
 }
 
@@ -590,7 +567,6 @@ mod tests {
                 data_shards: 10,
                 parity_shards: 4,
                 encode_ts_ns: 0,
-                block_size: 0,
             }),
             shards: vec![
                 EcShardChecksums {
@@ -736,7 +712,7 @@ mod tests {
             algorithm: ChecksumAlgorithm::ChecksumCrc32c as i32,
             block_size: DEFAULT_BITROT_BLOCK_SIZE as u32,
             generation: 0,
-            ec_shard_config: Some(ec_shard_config(10, 4, 0)),
+            ec_shard_config: Some(ec_shard_config(10, 4)),
             shards: vec![EcShardChecksums {
                 shard_id: 0,
                 covered_size: covered,
@@ -774,7 +750,7 @@ mod tests {
             algorithm: ChecksumAlgorithm::ChecksumCrc32c as i32,
             block_size: DEFAULT_BITROT_BLOCK_SIZE as u32,
             generation: 0,
-            ec_shard_config: Some(ec_shard_config(10, 4, 0)),
+            ec_shard_config: Some(ec_shard_config(10, 4)),
             shards: vec![EcShardChecksums {
                 shard_id: 0,
                 covered_size: 5,
@@ -818,7 +794,7 @@ mod tests {
             algorithm: ChecksumAlgorithm::ChecksumCrc32c as i32,
             block_size: DEFAULT_BITROT_BLOCK_SIZE as u32,
             generation: 0,
-            ec_shard_config: Some(ec_shard_config(10, 4, 0)),
+            ec_shard_config: Some(ec_shard_config(10, 4)),
             shards,
             encode_uuid: vec![0u8; 16],
         }

seaweed-volume/src/storage/erasure_coding/ec_decoder.rs

@@ -80,8 +80,6 @@ pub fn write_dat_file_from_shards(
     dat_file_size: i64,
     encoded_dat_file_size: i64,
     data_shards: usize,
-    large_block_size: usize,
-    small_block_size: usize,
 ) -> io::Result<()> {
     let dirs: Vec<String> = (0..data_shards).map(|_| dir.to_string()).collect();
     write_dat_file_from_shards_with_dirs(
@@ -92,8 +90,6 @@ pub fn write_dat_file_from_shards(
         encoded_dat_file_size,
         data_shards,
         &dirs,
-        large_block_size,
-        small_block_size,
     )
 }
 
@@ -117,10 +113,7 @@ pub fn write_dat_file_from_shards(
 /// boundary, and deriving the layout from the shrunk extent would read
 /// the shards in the wrong block order. Pass zero when the .vif does
 /// not record the encode-time size to infer the layout from the shard
-/// size. `large_block_size`/`small_block_size` are the volume's shard
-/// block layout, e.g. `EcVolume::large_block_size()` /
-/// `small_block_size()` from its .vif EC config.
-#[allow(clippy::too_many_arguments)]
+/// size.
 pub fn write_dat_file_from_shards_with_dirs(
     dat_dir: &str,
     collection: &str,
@@ -129,8 +122,6 @@ pub fn write_dat_file_from_shards_with_dirs(
     encoded_dat_file_size: i64,
     data_shards: usize,
     shard_dirs: &[String],
-    large_block_size: usize,
-    small_block_size: usize,
 ) -> io::Result<()> {
     write_dat_file(
         dat_dir,
@@ -140,8 +131,8 @@ pub fn write_dat_file_from_shards_with_dirs(
         encoded_dat_file_size,
         data_shards,
         shard_dirs,
-        large_block_size,
-        small_block_size,
+        ERASURE_CODING_LARGE_BLOCK_SIZE,
+        ERASURE_CODING_SMALL_BLOCK_SIZE,
     )
 }
 
@@ -421,9 +412,7 @@ mod tests {
         // Encode to EC
         let data_shards = 10;
         let parity_shards = 4;
-        let block_size =
-            ec_encoder::write_ec_files(dir, dir, "", VolumeId(1), data_shards, parity_shards)
-                .unwrap();
+        ec_encoder::write_ec_files(dir, dir, "", VolumeId(1), data_shards, parity_shards).unwrap();
 
         // Delete original .dat and .idx
         std::fs::remove_file(format!("{}/1.dat", dir)).unwrap();
@@ -437,8 +426,6 @@ mod tests {
             original_dat_size as i64,
             original_dat_size as i64,
             data_shards,
-            block_size as usize,
-            block_size as usize,
         )
         .unwrap();
         write_idx_file_from_ec_index(dir, "", VolumeId(1)).unwrap();
@@ -485,16 +472,7 @@ mod tests {
         let dir = tmp.path().to_str().unwrap();
         // No shard files exist, so de-striping must fail and publish nothing:
         // neither the final .dat nor a partial .dat.tmp may remain.
-        let res = write_dat_file_from_shards(
-            dir,
-            "",
-            VolumeId(7),
-            100,
-            100,
-            10,
-            ERASURE_CODING_LARGE_BLOCK_SIZE,
-            ERASURE_CODING_SMALL_BLOCK_SIZE,
-        );
+        let res = write_dat_file_from_shards(dir, "", VolumeId(7), 100, 100, 10);
         assert!(res.is_err());
         assert!(!std::path::Path::new(&format!("{}/7.dat", dir)).exists());
         assert!(!std::path::Path::new(&format!("{}/7.dat.tmp", dir)).exists());
@@ -547,7 +525,6 @@ mod tests {
                 &mut builders,
                 data_shards,
                 parity_shards,
-                SMALL,
                 LARGE,
                 SMALL,
             )
@@ -639,7 +616,6 @@ mod tests {
             &mut builders,
             data_shards,
             parity_shards,
-            SMALL,
             LARGE,
             SMALL,
         )

seaweed-volume/src/storage/erasure_coding/ec_encoder.rs

@@ -25,10 +25,6 @@ use crate::storage::volume::volume_file_name;
 ///
 /// Creates .ec00-.ec13 files in the same directory.
 /// Also creates a sorted .ecx index from the .idx file.
-///
-/// Always encodes with the uniform block layout, sized for this .dat, and
-/// returns the block size so the caller can persist it to .vif. Mirrors Go's
-/// WriteEcFiles.
 pub fn write_ec_files(
     dir: &str,
     idx_dir: &str,
@@ -36,7 +32,7 @@ pub fn write_ec_files(
     volume_id: VolumeId,
     data_shards: usize,
     parity_shards: usize,
-) -> io::Result<i64> {
+) -> io::Result<()> {
     let base = volume_file_name(dir, collection, volume_id);
     let dat_path = format!("{}.dat", base);
     let idx_base = volume_file_name(idx_dir, collection, volume_id);
@@ -70,7 +66,7 @@ pub fn write_ec_files(
         .map(|_| ShardChecksumBuilder::new(DEFAULT_BITROT_BLOCK_SIZE as i64))
         .collect();
 
-    let block_size = uniform_block_size(dat_size, data_shards);
+    // Encode in large blocks, then small blocks
     encode_dat_file(
         &dat_file,
         dat_size,
@@ -79,9 +75,8 @@ pub fn write_ec_files(
         &mut builders,
         data_shards,
         parity_shards,
-        ENCODE_BUFFER_SIZE,
-        block_size as usize,
-        block_size as usize,
+        ERASURE_CODING_LARGE_BLOCK_SIZE,
+        ERASURE_CODING_SMALL_BLOCK_SIZE,
     )?;
 
     // Close all shards
@@ -108,7 +103,6 @@ pub fn write_ec_files(
         ec_shard_config: Some(ec_bitrot::ec_shard_config(
             data_shards as u32,
             parity_shards as u32,
-            block_size,
         )),
         shards: shard_checksums,
         encode_uuid: ec_bitrot::new_encode_uuid(),
@@ -126,19 +120,7 @@ pub fn write_ec_files(
         );
     }
 
-    Ok(block_size)
-}
-
-/// uniform_block_size returns the per-shard block size of the uniform layout
-/// for a .dat of the given size: ceil(dat_file_size/data_shards) rounded up to
-/// a whole small block. For every input this equals the legacy layout's padded
-/// shard size, so only the byte placement differs between the two layouts,
-/// never the shard length. Mirrors Go's UniformBlockSize.
-pub fn uniform_block_size(dat_file_size: i64, data_shards: usize) -> i64 {
-    let small = ERASURE_CODING_SMALL_BLOCK_SIZE as i64;
-    let per_shard = (dat_file_size + data_shards as i64 - 1) / data_shards as i64;
-    let blocks = ((per_shard + small - 1) / small).max(1);
-    blocks * small
+    Ok(())
 }
 
 /// Rebuild missing EC shard files from existing shards using Reed-Solomon reconstruct.
@@ -388,7 +370,7 @@ pub fn verify_ec_shards(
 }
 
 /// Write sorted .ecx index from .idx file.
-pub(crate) fn write_sorted_ecx_from_idx(idx_path: &str, ecx_path: &str) -> io::Result<()> {
+fn write_sorted_ecx_from_idx(idx_path: &str, ecx_path: &str) -> io::Result<()> {
     if !std::path::Path::new(idx_path).exists() {
         return Err(io::Error::new(
             io::ErrorKind::NotFound,
@@ -439,8 +421,6 @@ pub fn rebuild_ecx_file(
     collection: &str,
     volume_id: VolumeId,
     data_shards: usize,
-    block_size: i64,
-    dat_file_size: i64,
     additional_dirs: &[&str],
 ) -> io::Result<()> {
     use crate::storage::needle::needle::get_actual_size;
@@ -483,42 +463,10 @@ pub fn rebuild_ecx_file(
     // Determine total logical data size from shard sizes
     let shard_size = shards.iter().map(|s| s.file_size()).max().unwrap_or(0);
     let total_data_size = shard_size as i64 * data_shards as i64;
-    // The volume's shard block layout: the .vif-recorded uniform block size,
-    // or the legacy two-tier sizes when 0. The row count comes from the shard
-    // length; -1 disambiguates a legacy shard that is an exact large-block
-    // multiple (mirrors the ecdFileSize-1 fallback in the read path).
-    let (large_block, small_block) = if block_size > 0 {
-        (block_size, block_size)
-    } else {
-        (
-            ERASURE_CODING_LARGE_BLOCK_SIZE as i64,
-            ERASURE_CODING_SMALL_BLOCK_SIZE as i64,
-        )
-    };
-    // The row count the de-stripe walks with. The encode-time .dat size is the
-    // authority — the same value the read path divides by data_shards — and the
-    // padded extent is only a fallback: under the legacy layout a shard that is
-    // an exact large-block multiple reads as one row too many, which
-    // re-interprets its last large row as small blocks and scrambles the
-    // recovered offsets. Subtracting one keeps that fallback on the safe side of
-    // the boundary, exactly as the read path's own fallback does.
-    let locate_shard_size = if dat_file_size > 0 {
-        dat_file_size / data_shards as i64
-    } else {
-        (shard_size as i64 - 1).max(0)
-    };
 
     // Read version from superblock (first byte of logical data)
     let mut sb_buf = [0u8; SUPER_BLOCK_SIZE];
-    read_from_data_shards(
-        &shards,
-        &mut sb_buf,
-        0,
-        data_shards,
-        locate_shard_size,
-        large_block,
-        small_block,
-    )?;
+    read_from_data_shards(&shards, &mut sb_buf, 0, data_shards)?;
     let version = Version(sb_buf[0]);
 
     // Walk needles starting after superblock
@@ -527,30 +475,10 @@ pub fn rebuild_ecx_file(
     let mut entries: Vec<(NeedleId, Offset, Size)> = Vec::new();
 
     while offset + header_size as i64 <= total_data_size {
-        // Read needle header (cookie + needle_id + size = 16 bytes).
-        // A read failure is NOT the end of the data — every offset in
-        // range maps into the shards, so an error means a truncated or
-        // unreadable shard. Publishing the entries collected so far as
-        // a successful .ecx would hand out a silently incomplete
-        // recovery index; propagate instead. (The scan still ends
-        // normally on the zero-cookie tail below.)
+        // Read needle header (cookie + needle_id + size = 16 bytes)
         let mut header_buf = [0u8; NEEDLE_HEADER_SIZE];
-        if let Err(e) = read_from_data_shards(
-            &shards,
-            &mut header_buf,
-            offset as u64,
-            data_shards,
-            locate_shard_size,
-            large_block,
-            small_block,
-        ) {
-            for s in &mut shards {
-                s.close();
-            }
-            return Err(io::Error::new(
-                e.kind(),
-                format!("scan needle header at offset {}: {}", offset, e),
-            ));
+        if read_from_data_shards(&shards, &mut header_buf, offset as u64, data_shards).is_err() {
+            break;
         }
 
         let cookie = Cookie::from_bytes(&header_buf[..COOKIE_SIZE]);
@@ -604,83 +532,58 @@ pub fn rebuild_ecx_file(
     Ok(())
 }
 
-/// Read bytes from EC data shards at a logical offset in the .dat file,
-/// resolving the shard/offset through the volume's block layout via
-/// locate_data — the same mapping the read path uses.
-#[allow(clippy::too_many_arguments)]
+/// Read bytes from EC data shards at a logical offset in the .dat file.
 fn read_from_data_shards(
     shards: &[EcVolumeShard],
     buf: &mut [u8],
     logical_offset: u64,
     data_shards: usize,
-    locate_shard_size: i64,
-    large_block_size: i64,
-    small_block_size: i64,
 ) -> io::Result<()> {
-    let intervals = crate::storage::erasure_coding::ec_locate::locate_data(
-        logical_offset as i64,
-        Size(buf.len() as i32),
-        locate_shard_size,
-        data_shards as u32,
-        large_block_size,
-        small_block_size,
-    );
-    let mut bytes_read = 0usize;
-    for interval in &intervals {
-        let (shard_id, shard_offset) =
-            interval.to_shard_id_and_offset(data_shards as u32, large_block_size, small_block_size);
-        if shard_id as usize >= data_shards {
+    let small_block = ERASURE_CODING_SMALL_BLOCK_SIZE as u64;
+    let data_shards_u64 = data_shards as u64;
+
+    let mut bytes_read = 0u64;
+    let mut remaining = buf.len() as u64;
+    let mut current_offset = logical_offset;
+
+    while remaining > 0 {
+        // Determine which shard and at what shard-offset this logical offset maps to.
+        // The data is interleaved: large blocks first, then small blocks.
+        // For simplicity, use the small block size for all calculations since
+        // large blocks are multiples of small blocks.
+        let row_size = small_block * data_shards_u64;
+        let row_index = current_offset / row_size;
+        let row_offset = current_offset % row_size;
+        let shard_index = (row_offset / small_block) as usize;
+        let shard_offset = row_index * small_block + (row_offset % small_block);
+
+        if shard_index >= data_shards {
             return Err(io::Error::new(
                 io::ErrorKind::InvalidInput,
                 "shard index out of range",
             ));
         }
-        let to_read = interval.size as usize;
-        let dest = &mut buf[bytes_read..bytes_read + to_read];
-        // Exact-read semantics: read_at may legally return fewer bytes
-        // than requested, and treating a short read as complete leaves
-        // the tail of `dest` as whatever the buffer held before. Loop
-        // until filled; zero bytes inside the mapped range means the
-        // shard is truncated — an error, not an end.
-        let mut filled = 0usize;
-        while filled < to_read {
-            let n = shards[shard_id as usize]
-                .read_at(&mut dest[filled..], shard_offset as u64 + filled as u64)?;
-            if n == 0 {
-                return Err(io::Error::new(
-                    io::ErrorKind::UnexpectedEof,
-                    format!(
-                        "short read from data shard {}: {} of {} bytes at offset {}",
-                        shard_id, filled, to_read, shard_offset
-                    ),
-                ));
-            }
-            filled += n;
-        }
-        bytes_read += to_read;
-    }
-    if bytes_read != buf.len() {
-        return Err(io::Error::new(
-            io::ErrorKind::UnexpectedEof,
-            "short read from data shards",
-        ));
+
+        // How many bytes can we read from this position in this shard block
+        let bytes_left_in_block = small_block - (row_offset % small_block);
+        let to_read = remaining.min(bytes_left_in_block) as usize;
+
+        let dest = &mut buf[bytes_read as usize..bytes_read as usize + to_read];
+        shards[shard_index].read_at(dest, shard_offset)?;
+
+        bytes_read += to_read as u64;
+        remaining -= to_read as u64;
+        current_offset += to_read as u64;
     }
+
     Ok(())
 }
 
-/// Buffer size for one encode sub-batch per shard, mirroring Go's 256KB
-/// bufferSize in WriteEcFiles. A block is processed in block_size/buffer_size
-/// sub-batches, so memory stays at total_shards * 256KB no matter how large
-/// the uniform block is.
-const ENCODE_BUFFER_SIZE: usize = 256 * 1024;
-
 /// Encode the .dat file data into shard files.
 ///
 /// Uses a two-phase approach matching Go's ec_encoder.go:
-/// 1. Process as many large blocks as possible
-/// 2. Process remaining data with small blocks
-///
-/// `buffer_size` must divide both block sizes.
+/// 1. Process as many large blocks (1GB) as possible
+/// 2. Process remaining data with small blocks (1MB)
 #[allow(clippy::too_many_arguments)]
 pub(crate) fn encode_dat_file(
     dat_file: &File,
@@ -690,50 +593,44 @@ pub(crate) fn encode_dat_file(
     builders: &mut [ShardChecksumBuilder],
     data_shards: usize,
     parity_shards: usize,
-    buffer_size: usize,
     large_block_size: usize,
     small_block_size: usize,
 ) -> io::Result<()> {
-    let total_shards = data_shards + parity_shards;
-    let mut buffers: Vec<Vec<u8>> = (0..total_shards)
-        .map(|_| vec![0u8; buffer_size])
-        .collect();
-
     let mut remaining = dat_size;
     let mut offset: u64 = 0;
 
-    // Phase 1: process whole large-block rows while enough data remains
+    // Phase 1: Process large blocks (1GB each) while enough data remains
     let large_row_size = large_block_size * data_shards;
 
     while remaining >= large_row_size as i64 {
-        encode_data(
+        encode_one_batch(
             dat_file,
             offset,
             large_block_size,
             rs,
-            &mut buffers,
             shards,
             builders,
             data_shards,
+            parity_shards,
         )?;
         offset += large_row_size as u64;
         remaining -= large_row_size as i64;
     }
 
-    // Phase 2: process remaining data with small blocks
+    // Phase 2: Process remaining data with small blocks (1MB each)
     let small_row_size = small_block_size * data_shards;
 
     while remaining > 0 {
         let to_process = remaining.min(small_row_size as i64);
-        encode_data(
+        encode_one_batch(
             dat_file,
             offset,
             small_block_size,
             rs,
-            &mut buffers,
             shards,
             builders,
             data_shards,
+            parity_shards,
         )?;
         offset += to_process as u64;
         remaining -= to_process;
@@ -742,71 +639,61 @@ pub(crate) fn encode_dat_file(
     Ok(())
 }
 
-/// Encode one row of blocks, streaming it in ENCODE_BUFFER_SIZE sub-batches so
-/// arbitrarily large blocks never require block-sized allocations. Mirrors
-/// Go's encodeData.
-#[allow(clippy::too_many_arguments)]
-fn encode_data(
-    dat_file: &File,
-    row_offset: u64,
-    block_size: usize,
-    rs: &ReedSolomon,
-    buffers: &mut [Vec<u8>],
-    shards: &mut [EcVolumeShard],
-    builders: &mut [ShardChecksumBuilder],
-    data_shards: usize,
-) -> io::Result<()> {
-    let buffer_size = buffers[0].len();
-    if block_size % buffer_size != 0 {
-        return Err(io::Error::new(
-            io::ErrorKind::InvalidInput,
-            format!(
-                "unexpected block size {} buffer size {}",
-                block_size, buffer_size
-            ),
-        ));
-    }
-    let batch_count = block_size / buffer_size;
-    for b in 0..batch_count {
-        encode_one_batch(
-            dat_file,
-            row_offset + (b * buffer_size) as u64,
-            block_size,
-            rs,
-            buffers,
-            shards,
-            builders,
-            data_shards,
-        )?;
-    }
-    Ok(())
-}
-
-/// Encode one sub-batch: the same buffer-sized slice of every shard's block in
-/// this row. Mirrors Go's encodeDataOneBatch.
-#[allow(clippy::too_many_arguments)]
+/// Encode one batch (row) of data.
 fn encode_one_batch(
     dat_file: &File,
     offset: u64,
     block_size: usize,
     rs: &ReedSolomon,
-    buffers: &mut [Vec<u8>],
     shards: &mut [EcVolumeShard],
     builders: &mut [ShardChecksumBuilder],
     data_shards: usize,
+    parity_shards: usize,
 ) -> io::Result<()> {
-    // Read data shards from the .dat file, zero-filling past EOF — the buffers
-    // are reused across batches, so the tail must be cleared explicitly.
+    let total_shards = data_shards + parity_shards;
+    // Each batch allocates block_size * total_shards bytes.
+    // With large blocks (1 GiB) this is 14 GiB -- guard against OOM.
+    let total_alloc = block_size.checked_mul(total_shards).ok_or_else(|| {
+        io::Error::new(
+            io::ErrorKind::InvalidInput,
+            "block_size * shard count overflows usize",
+        )
+    })?;
+    // Large-block encoding uses 1 GiB * 14 shards = 14 GiB; allow up to 16 GiB.
+    const MAX_BATCH_ALLOC: usize = 16 * 1024 * 1024 * 1024; // 16 GiB safety limit
+    if total_alloc > MAX_BATCH_ALLOC {
+        return Err(io::Error::new(
+            io::ErrorKind::InvalidInput,
+            format!(
+                "batch allocation too large ({} bytes, limit {} bytes); block_size={} shards={}",
+                total_alloc, MAX_BATCH_ALLOC, block_size, total_shards,
+            ),
+        ));
+    }
+
+    // Allocate buffers for all shards
+    let mut buffers: Vec<Vec<u8>> = (0..total_shards).map(|_| vec![0u8; block_size]).collect();
+
+    // Read data shards from .dat file
     for i in 0..data_shards {
         let read_offset = offset + (i * block_size) as u64;
-        let n = read_at_most(dat_file, &mut buffers[i], read_offset)?;
-        for b in buffers[i][n..].iter_mut() {
-            *b = 0;
+
+        #[cfg(unix)]
+        {
+            use std::os::unix::fs::FileExt;
+            dat_file.read_at(&mut buffers[i], read_offset)?;
+        }
+
+        #[cfg(not(unix))]
+        {
+            let mut f = dat_file.try_clone()?;
+            f.seek(SeekFrom::Start(read_offset))?;
+            f.read(&mut buffers[i])?;
         }
     }
 
     // Encode parity shards
-    rs.encode(&mut *buffers).map_err(|e| {
+    rs.encode(&mut buffers).map_err(|e| {
         io::Error::new(
             io::ErrorKind::Other,
             format!("reed-solomon encode: {:?}", e),
@@ -823,29 +710,6 @@ fn encode_one_batch(
     Ok(())
 }
 
-/// Read into `buf` at `offset` until it is full or EOF; returns bytes read.
-fn read_at_most(dat_file: &File, buf: &mut [u8], offset: u64) -> io::Result<usize> {
-    let mut n = 0;
-    while n < buf.len() {
-        #[cfg(unix)]
-        let r = {
-            use std::os::unix::fs::FileExt;
-            dat_file.read_at(&mut buf[n..], offset + n as u64)?
-        };
-        #[cfg(not(unix))]
-        let r = {
-            let mut f = dat_file.try_clone()?;
-            f.seek(SeekFrom::Start(offset + n as u64))?;
-            f.read(&mut buf[n..])?
-        };
-        if r == 0 {
-            break;
-        }
-        n += r;
-    }
-    Ok(n)
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -1156,7 +1020,7 @@ mod tests {
 
         // Without additional_dirs, rebuild must fail: shards 1, 3, 6 are not
         // in primary and the full logical .dat content can't be reconstructed.
-        let res = rebuild_ecx_file(&primary, "", VolumeId(1), 10, 0, 0, &[]);
+        let res = rebuild_ecx_file(&primary, "", VolumeId(1), 10, &[]);
         assert!(
             res.is_err(),
             "ecx rebuild without additional_dirs must fail when data shards are on another disk"
@@ -1167,7 +1031,7 @@ mod tests {
         );
 
         // With additional_dirs pointing at the secondary, the rebuild must succeed.
-        rebuild_ecx_file(&primary, "", VolumeId(1), 10, 0, 0, &[secondary.as_str()]).unwrap();
+        rebuild_ecx_file(&primary, "", VolumeId(1), 10, &[secondary.as_str()]).unwrap();
 
         assert!(
             std::path::Path::new(&ecx_path).exists(),
@@ -1179,118 +1043,6 @@ mod tests {
         );
     }
 
-    // A uniform-layout volume (block size > 1MiB) must have its .ecx rebuilt
-    // through the recorded geometry; the legacy 1MiB mapping would scan
-    // garbage past the first block boundary.
-    #[test]
-    fn test_rebuild_ecx_file_uniform_layout() {
-        use crate::storage::needle_map::NeedleMapKind;
-        use crate::storage::volume::Volume;
-        let tmp = TempDir::new().unwrap();
-        let dir = tmp.path().to_str().unwrap().to_string();
-        let mut v = Volume::new(
-            &dir,
-            &dir,
-            "",
-            VolumeId(2),
-            NeedleMapKind::InMemory,
-            None,
-            None,
-            0,
-            Version::current(),
-        )
-        .unwrap();
-        for i in 1u64..=12 {
-            let data: Vec<u8> = (0..2 << 20)
-                .map(|b| ((b as u64).wrapping_mul(2654435761).wrapping_add(i) >> 8) as u8)
-                .collect();
-            let mut n = Needle {
-                id: NeedleId(i),
-                cookie: Cookie(i as u32),
-                data: data.clone(),
-                data_size: data.len() as u32,
-                ..Needle::default()
-            };
-            v.write_needle(&mut n, true, false).unwrap();
-        }
-        v.sync_to_disk().unwrap();
-        v.close();
-        let block_size = write_ec_files(&dir, &dir, "", VolumeId(2), 10, 4).unwrap();
-        assert!(
-            block_size > ERASURE_CODING_SMALL_BLOCK_SIZE as i64,
-            "fixture must diverge from the legacy layout"
-        );
-
-        let ecx_path = format!("{}/2.ecx", dir);
-        let canonical = std::fs::read(&ecx_path).unwrap();
-        std::fs::remove_file(&ecx_path).unwrap();
-
-        rebuild_ecx_file(&dir, "", VolumeId(2), 10, block_size, 0, &[]).unwrap();
-        let rebuilt = std::fs::read(&ecx_path).unwrap();
-        assert_eq!(canonical, rebuilt, "rebuilt .ecx must match the encode-time .ecx");
-    }
-
-    // A truncated data shard must FAIL the .ecx rebuild, not publish the
-    // entries scanned so far as a successful (silently incomplete) index.
-    #[test]
-    fn test_rebuild_ecx_file_fails_on_truncated_shard() {
-        use crate::storage::needle_map::NeedleMapKind;
-        use crate::storage::volume::Volume;
-        let tmp = TempDir::new().unwrap();
-        let dir = tmp.path().to_str().unwrap().to_string();
-        let mut v = Volume::new(
-            &dir,
-            &dir,
-            "",
-            VolumeId(3),
-            NeedleMapKind::InMemory,
-            None,
-            None,
-            0,
-            Version::current(),
-        )
-        .unwrap();
-        for i in 1u64..=12 {
-            let data: Vec<u8> = (0..2 << 20)
-                .map(|b| ((b as u64).wrapping_mul(2654435761).wrapping_add(i) >> 8) as u8)
-                .collect();
-            let mut n = Needle {
-                id: NeedleId(i),
-                cookie: Cookie(i as u32),
-                data: data.clone(),
-                data_size: data.len() as u32,
-                ..Needle::default()
-            };
-            v.write_needle(&mut n, true, false).unwrap();
-        }
-        v.sync_to_disk().unwrap();
-        v.close();
-        let block_size = write_ec_files(&dir, &dir, "", VolumeId(3), 10, 4).unwrap();
-
-        let ecx_path = format!("{}/3.ecx", dir);
-        std::fs::remove_file(&ecx_path).unwrap();
-
-        // Truncate shard 0 to just the superblock: the scan's very first
-        // needle-header read (offset SUPER_BLOCK_SIZE, shard 0 under the
-        // uniform layout) lands in the missing region. The pre-fix code
-        // broke the scan there and published an EMPTY .ecx as success.
-        let shard_path = format!("{}/3.ec00", dir);
-        let f = std::fs::OpenOptions::new()
-            .write(true)
-            .open(&shard_path)
-            .unwrap();
-        f.set_len(crate::storage::super_block::SUPER_BLOCK_SIZE as u64)
-            .unwrap();
-        drop(f);
-
-        let res = rebuild_ecx_file(&dir, "", VolumeId(3), 10, block_size, 0, &[]);
-        assert!(res.is_err(), "rebuild over a truncated shard must fail");
-        assert!(
-            !std::path::Path::new(&ecx_path).exists(),
-            "a failed rebuild must not leave a partial .ecx behind"
-        );
-    }
-
     #[test]
     fn test_reed_solomon_basic() {
         let data_shards = 10;

seaweed-volume/src/storage/erasure_coding/ec_locate.rs

@@ -18,26 +18,21 @@ pub struct Interval {
 }
 
 impl Interval {
-    pub fn to_shard_id_and_offset(
-        &self,
-        data_shards: u32,
-        large_block_size: i64,
-        small_block_size: i64,
-    ) -> (ShardId, i64) {
+    pub fn to_shard_id_and_offset(&self, data_shards: u32) -> (ShardId, i64) {
         let data_shards_usize = data_shards as usize;
         let shard_id = (self.block_index % data_shards_usize) as ShardId;
         let row_index = self.block_index / data_shards_usize;
 
         let block_size = if self.is_large_block {
-            large_block_size
+            ERASURE_CODING_LARGE_BLOCK_SIZE as i64
         } else {
-            small_block_size
+            ERASURE_CODING_SMALL_BLOCK_SIZE as i64
         };
 
         let mut offset = row_index as i64 * block_size + self.inner_block_offset;
         if !self.is_large_block {
             // Small blocks come after large blocks in the shard file
-            offset += self.large_block_rows_count as i64 * large_block_size;
+            offset += self.large_block_rows_count as i64 * ERASURE_CODING_LARGE_BLOCK_SIZE as i64;
         }
 
         (shard_id, offset)
@@ -47,14 +42,7 @@ impl Interval {
 /// Locate the EC shard intervals needed to read data at the given offset and size.
 ///
 /// `shard_size` is the size of a single shard file.
-pub fn locate_data(
-    offset: i64,
-    size: Size,
-    shard_size: i64,
-    data_shards: u32,
-    large_block_size: i64,
-    small_block_size: i64,
-) -> Vec<Interval> {
+pub fn locate_data(offset: i64, size: Size, shard_size: i64, data_shards: u32) -> Vec<Interval> {
     let mut intervals = Vec::new();
     let data_size = size.0 as i64;
 
@@ -62,14 +50,17 @@ pub fn locate_data(
         return intervals;
     }
 
+    let large_block_size = ERASURE_CODING_LARGE_BLOCK_SIZE as i64;
+    let small_block_size = ERASURE_CODING_SMALL_BLOCK_SIZE as i64;
     let large_row_size = large_block_size * data_shards as i64;
     let small_row_size = small_block_size * data_shards as i64;
 
-    // Number of large block rows. Mirrors Go's shardDatSize/largeBlockLength:
-    // the caller's ecd-size fallback already subtracts 1 to disambiguate the
-    // exact-multiple case, so no further -1 here — a shard size that IS an
-    // exact multiple (dat_file_size path) means real full large rows.
-    let n_large_block_rows = (shard_size / large_block_size) as usize;
+    // Number of large block rows
+    let n_large_block_rows = if shard_size > 0 {
+        ((shard_size - 1) / large_block_size) as usize
+    } else {
+        0
+    };
     let large_section_size = n_large_block_rows as i64 * large_row_size;
 
     let mut remaining_offset = offset;
@@ -159,11 +150,7 @@ mod tests {
             is_large_block: true,
             large_block_rows_count: 1,
         };
-        let (shard_id, offset) = interval.to_shard_id_and_offset(
-            data_shards,
-            ERASURE_CODING_LARGE_BLOCK_SIZE as i64,
-            ERASURE_CODING_SMALL_BLOCK_SIZE as i64,
-        );
+        let (shard_id, offset) = interval.to_shard_id_and_offset(data_shards);
         assert_eq!(shard_id, 0);
         assert_eq!(offset, 100);
 
@@ -175,11 +162,7 @@ mod tests {
             is_large_block: true,
             large_block_rows_count: 1,
         };
-        let (shard_id, _offset) = interval.to_shard_id_and_offset(
-            data_shards,
-            ERASURE_CODING_LARGE_BLOCK_SIZE as i64,
-            ERASURE_CODING_SMALL_BLOCK_SIZE as i64,
-        );
+        let (shard_id, _offset) = interval.to_shard_id_and_offset(data_shards);
         assert_eq!(shard_id, 5);
 
         // Block index 12 (data_shards=10) → row_index 1, shard_id 2
@@ -190,11 +173,7 @@ mod tests {
             is_large_block: true,
             large_block_rows_count: 5,
         };
-        let (shard_id, offset) = interval.to_shard_id_and_offset(
-            data_shards,
-            ERASURE_CODING_LARGE_BLOCK_SIZE as i64,
-            ERASURE_CODING_SMALL_BLOCK_SIZE as i64,
-        );
+        let (shard_id, offset) = interval.to_shard_id_and_offset(data_shards);
         assert_eq!(shard_id, 2); // 12 % 10 = 2
         assert_eq!(offset, large_block_size + 200); // row 1 offset + inner_block_offset
 
@@ -206,11 +185,7 @@ mod tests {
             is_large_block: true,
             large_block_rows_count: 2,
         };
-        let (shard_id, offset) = interval.to_shard_id_and_offset(
-            data_shards,
-            ERASURE_CODING_LARGE_BLOCK_SIZE as i64,
-            ERASURE_CODING_SMALL_BLOCK_SIZE as i64,
-        );
+        let (shard_id, offset) = interval.to_shard_id_and_offset(data_shards);
         assert_eq!(shard_id, 0);
         assert_eq!(offset, ERASURE_CODING_LARGE_BLOCK_SIZE as i64); // row 1 offset
     }
@@ -218,14 +193,7 @@ mod tests {
     #[test]
     fn test_locate_data_small_file() {
         // Small file: 100 bytes at offset 50, shard size = 1MB
-        let intervals = locate_data(
-            50,
-            Size(100),
-            1024 * 1024,
-            10,
-            ERASURE_CODING_LARGE_BLOCK_SIZE as i64,
-            ERASURE_CODING_SMALL_BLOCK_SIZE as i64,
-        );
+        let intervals = locate_data(50, Size(100), 1024 * 1024, 10);
         assert!(!intervals.is_empty());
 
         // Should be a single small block interval (no large block rows for 1MB shard)
@@ -235,14 +203,7 @@ mod tests {
 
     #[test]
     fn test_locate_data_empty() {
-        let intervals = locate_data(
-            0,
-            Size(0),
-            1024 * 1024,
-            10,
-            ERASURE_CODING_LARGE_BLOCK_SIZE as i64,
-            ERASURE_CODING_SMALL_BLOCK_SIZE as i64,
-        );
+        let intervals = locate_data(0, Size(0), 1024 * 1024, 10);
         assert!(intervals.is_empty());
     }
 
@@ -255,11 +216,7 @@ mod tests {
             is_large_block: false,
             large_block_rows_count: 2,
         };
-        let (_shard_id, offset) = interval.to_shard_id_and_offset(
-            10,
-            ERASURE_CODING_LARGE_BLOCK_SIZE as i64,
-            ERASURE_CODING_SMALL_BLOCK_SIZE as i64,
-        );
+        let (_shard_id, offset) = interval.to_shard_id_and_offset(10);
         // Should be after 2 large block rows
         assert_eq!(offset, 2 * ERASURE_CODING_LARGE_BLOCK_SIZE as i64);
     }

seaweed-volume/src/storage/needle_map.rs

@@ -14,10 +14,7 @@ use std::path::Path;
 use std::sync::atomic::{AtomicI64, AtomicU64, Ordering};
 
 mod compact_map;
-pub mod file_pool;
-pub mod sorted_file;
 use compact_map::CompactMap;
-use sorted_file::SortedFileNeedleMap;
 
 use redb::{Database, Durability, ReadableDatabase, ReadableTable, TableDefinition};
 
@@ -753,12 +750,10 @@ impl RedbNeedleMap {
         Ok(())
     }
 
-    /// Look up a needle. A redb failure is an ERROR, not an absent needle:
-    /// answering "not found" would turn a database problem into a read miss
-    /// and let a delete report success without recording a tombstone.
-    pub fn get(&self, key: NeedleId) -> io::Result<Option<NeedleValue>> {
+    /// Look up a needle.
+    pub fn get(&self, key: NeedleId) -> Option<NeedleValue> {
         let key_u64: u64 = key.into();
-        self.get_internal(key_u64)
+        self.get_internal(key_u64).ok().flatten()
     }
 
     /// Internal get that returns io::Result for error propagation.
@@ -944,31 +939,33 @@ impl RedbNeedleMap {
     }
 
     /// Collect all entries as a Vec for iteration (used by volume.rs iter patterns).
-    pub fn collect_entries(&self) -> io::Result<Vec<(NeedleId, NeedleValue)>> {
+    pub fn collect_entries(&self) -> Vec<(NeedleId, NeedleValue)> {
         let mut result = Vec::new();
-        let txn: redb::ReadTransaction = self
-            .db
-            .begin_read()
-            .map_err(|e| io::Error::other(format!("redb begin_read: {e}")))?;
-        let table = txn
-            .open_table(NEEDLE_TABLE)
-            .map_err(|e| io::Error::other(format!("redb open_table: {e}")))?;
-        let iter = table
-            .iter()
-            .map_err(|e| io::Error::other(format!("redb iter: {e}")))?;
+        let txn: redb::ReadTransaction = match self.db.begin_read() {
+            Ok(t) => t,
+            Err(_) => return result,
+        };
+        let table = match txn.open_table(NEEDLE_TABLE) {
+            Ok(t) => t,
+            Err(_) => return result,
+        };
+        let iter = match table.iter() {
+            Ok(i) => i,
+            Err(_) => return result,
+        };
         for entry in iter {
-            let (key_guard, val_guard) =
-                entry.map_err(|e| io::Error::other(format!("redb entry: {e}")))?;
-            let key_u64: u64 = key_guard.value();
-            let bytes: &[u8] = val_guard.value();
-            if bytes.len() == PACKED_NEEDLE_VALUE_SIZE {
-                let mut arr = [0u8; PACKED_NEEDLE_VALUE_SIZE];
-                arr.copy_from_slice(bytes);
-                let nv = unpack_needle_value(&arr);
-                result.push((NeedleId(key_u64), nv));
+            if let Ok((key_guard, val_guard)) = entry {
+                let key_u64: u64 = key_guard.value();
+                let bytes: &[u8] = val_guard.value();
+                if bytes.len() == PACKED_NEEDLE_VALUE_SIZE {
+                    let mut arr = [0u8; PACKED_NEEDLE_VALUE_SIZE];
+                    arr.copy_from_slice(bytes);
+                    let nv = unpack_needle_value(&arr);
+                    result.push((NeedleId(key_u64), nv));
+                }
             }
         }
-        Ok(result)
+        result
     }
 }
 
@@ -980,10 +977,6 @@ impl RedbNeedleMap {
 pub enum NeedleMap {
     InMemory(CompactNeedleMap),
     Redb(RedbNeedleMap),
-    /// Read-only volumes — including every cloud-tiered one — search the sorted
-    /// `.sdx` on disk instead of holding an index in RAM. Mirrors Go's
-    /// `SortedFileNeedleMap`.
-    SortedFile(SortedFileNeedleMap),
 }
 
 impl NeedleMap {
@@ -992,18 +985,14 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.put(key, offset, size),
             NeedleMap::Redb(nm) => nm.put(key, offset, size),
-            NeedleMap::SortedFile(nm) => nm.put(key, offset, size),
         }
     }
 
-    /// Look up a needle. Disk- and database-backed maps report their own
-    /// failures rather than folding them into "not found" — see the notes on
-    /// `RedbNeedleMap::get` and `SortedFileNeedleMap::get`.
-    pub fn get(&self, key: NeedleId) -> io::Result<Option<NeedleValue>> {
+    /// Look up a needle.
+    pub fn get(&self, key: NeedleId) -> Option<NeedleValue> {
         match self {
-            NeedleMap::InMemory(nm) => Ok(nm.get(key)),
+            NeedleMap::InMemory(nm) => nm.get(key),
             NeedleMap::Redb(nm) => nm.get(key),
-            NeedleMap::SortedFile(nm) => nm.get(key),
         }
     }
 
@@ -1012,7 +1001,6 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.delete(key, offset),
             NeedleMap::Redb(nm) => nm.delete(key, offset),
-            NeedleMap::SortedFile(nm) => nm.delete(key, offset),
         }
     }
 
@@ -1021,9 +1009,6 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.set_idx_file(file, offset),
             NeedleMap::Redb(nm) => nm.set_idx_file(file, offset),
-            // The sorted map borrows its .idx per append, so there is no
-            // long-lived writer to install.
-            NeedleMap::SortedFile(_) => {}
         }
     }
 
@@ -1032,8 +1017,6 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.has_idx_writer(),
             NeedleMap::Redb(nm) => nm.has_idx_writer(),
-            // Appends open the .idx on demand, so one is always available.
-            NeedleMap::SortedFile(_) => true,
         }
     }
 
@@ -1042,7 +1025,6 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.content_size(),
             NeedleMap::Redb(nm) => nm.content_size(),
-            NeedleMap::SortedFile(nm) => nm.content_size(),
         }
     }
 
@@ -1051,7 +1033,6 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.deleted_size(),
             NeedleMap::Redb(nm) => nm.deleted_size(),
-            NeedleMap::SortedFile(nm) => nm.deleted_size(),
         }
     }
 
@@ -1060,7 +1041,6 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.file_count(),
             NeedleMap::Redb(nm) => nm.file_count(),
-            NeedleMap::SortedFile(nm) => nm.file_count(),
         }
     }
 
@@ -1069,7 +1049,6 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.deleted_count(),
             NeedleMap::Redb(nm) => nm.deleted_count(),
-            NeedleMap::SortedFile(nm) => nm.deleted_count(),
         }
     }
 
@@ -1078,7 +1057,6 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.max_file_key(),
             NeedleMap::Redb(nm) => nm.max_file_key(),
-            NeedleMap::SortedFile(nm) => nm.max_file_key(),
         }
     }
 
@@ -1089,7 +1067,6 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.max_needle_end(),
             NeedleMap::Redb(nm) => nm.max_needle_end(),
-            NeedleMap::SortedFile(nm) => nm.max_needle_end(),
         }
     }
 
@@ -1098,7 +1075,6 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.index_file_size(),
             NeedleMap::Redb(nm) => nm.index_file_size(),
-            NeedleMap::SortedFile(nm) => nm.index_file_size(),
         }
     }
 
@@ -1107,7 +1083,6 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.sync(),
             NeedleMap::Redb(nm) => nm.sync(),
-            NeedleMap::SortedFile(nm) => nm.sync(),
         }
     }
 
@@ -1116,7 +1091,6 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.close(),
             NeedleMap::Redb(nm) => nm.close(),
-            NeedleMap::SortedFile(nm) => nm.close(),
         }
     }
 
@@ -1125,7 +1099,6 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.save_to_idx(path),
             NeedleMap::Redb(nm) => nm.save_to_idx(path),
-            NeedleMap::SortedFile(nm) => nm.save_to_idx(path),
         }
     }
 
@@ -1137,28 +1110,22 @@ impl NeedleMap {
         match self {
             NeedleMap::InMemory(nm) => nm.ascending_visit(f),
             NeedleMap::Redb(nm) => nm.ascending_visit(f),
-            NeedleMap::SortedFile(nm) => nm.ascending_visit(f),
         }
     }
 
     /// Iterate all entries. Returns a Vec of (NeedleId, NeedleValue) pairs.
-    /// For InMemory this collects via ascending visit; the disk-backed maps read
-    /// it back off disk, so a truncated .sdx or a redb read fault surfaces here
-    /// as an error. Compaction treats the result as the complete live set, so a
-    /// partial scan must never be mistaken for an empty tail.
-    pub fn iter_entries(&self) -> io::Result<Vec<(NeedleId, NeedleValue)>> {
+    /// For InMemory this collects via ascending visit; for Redb it reads from disk.
+    pub fn iter_entries(&self) -> Vec<(NeedleId, NeedleValue)> {
         match self {
             NeedleMap::InMemory(nm) => {
                 let mut entries = Vec::new();
-                // The visitor never fails, so neither can this.
                 let _ = nm.ascending_visit(|id, nv| {
                     entries.push((id, *nv));
                     Ok(())
                 });
-                Ok(entries)
+                entries
             }
             NeedleMap::Redb(nm) => nm.collect_entries(),
-            NeedleMap::SortedFile(nm) => nm.iter_entries(),
         }
     }
 }
@@ -1313,13 +1280,13 @@ mod tests {
         nm.put(NeedleId(2), Offset::from_actual_offset(128), Size(200))
             .unwrap();
 
-        let v1 = nm.get(NeedleId(1)).unwrap().unwrap();
+        let v1 = nm.get(NeedleId(1)).unwrap();
         assert_eq!(v1.size, Size(100));
 
-        let v2 = nm.get(NeedleId(2)).unwrap().unwrap();
+        let v2 = nm.get(NeedleId(2)).unwrap();
         assert_eq!(v2.size, Size(200));
 
-        assert!(nm.get(NeedleId(99)).unwrap().is_none());
+        assert!(nm.get(NeedleId(99)).is_none());
     }
 
     #[test]
@@ -1344,7 +1311,7 @@ mod tests {
         assert_eq!(nm.deleted_size(), 100);
 
         // Deleted entry should have negated size
-        let nv = nm.get(NeedleId(1)).unwrap().unwrap();
+        let nv = nm.get(NeedleId(1)).unwrap();
         assert_eq!(nv.size, Size(-100));
     }
 
@@ -1417,9 +1384,9 @@ mod tests {
         let mut cursor = Cursor::new(idx_data);
         let nm = RedbNeedleMap::load_from_idx(db_path.to_str().unwrap(), &mut cursor, Version::current()).unwrap();
 
-        assert!(nm.get(NeedleId(1)).unwrap().is_some());
-        assert!(nm.get(NeedleId(2)).unwrap().is_none()); // deleted and removed
-        assert!(nm.get(NeedleId(3)).unwrap().is_some());
+        assert!(nm.get(NeedleId(1)).is_some());
+        assert!(nm.get(NeedleId(2)).is_none()); // deleted and removed
+        assert!(nm.get(NeedleId(3)).is_some());
         assert_eq!(nm.file_count(), 2);
     }
 
@@ -1536,7 +1503,7 @@ mod tests {
         let mut nm = NeedleMap::InMemory(CompactNeedleMap::new());
         nm.put(NeedleId(1), Offset::from_actual_offset(0), Size(100))
             .unwrap();
-        assert_eq!(nm.get(NeedleId(1)).unwrap().unwrap().size, Size(100));
+        assert_eq!(nm.get(NeedleId(1)).unwrap().size, Size(100));
         assert_eq!(nm.file_count(), 1);
     }
 
@@ -1547,7 +1514,7 @@ mod tests {
         let mut nm = NeedleMap::Redb(RedbNeedleMap::new(db_path.to_str().unwrap()).unwrap());
         nm.put(NeedleId(1), Offset::from_actual_offset(0), Size(100))
             .unwrap();
-        assert_eq!(nm.get(NeedleId(1)).unwrap().unwrap().size, Size(100));
+        assert_eq!(nm.get(NeedleId(1)).unwrap().size, Size(100));
         assert_eq!(nm.file_count(), 1);
     }
 }

seaweed-volume/src/storage/needle_map/file_pool.rs

@@ -1,247 +0,0 @@
-//! Bounded pool of open index-file descriptors.
-//!
-//! Read-only volumes — cloud-tiered ones above all — outnumber writable ones by
-//! orders of magnitude on a large server, and a volume that pins its `.idx` and
-//! `.sdx` for the life of the process costs two descriptors whether or not
-//! anybody reads it. At ~600K volumes per server that alone exhausts any fd
-//! limit. Neither file is needed except while a lookup is in flight, so
-//! [`SortedFileNeedleMap`](super::sorted_file::SortedFileNeedleMap) borrows them
-//! from this pool: an idle volume holds nothing, a busy one keeps its handles
-//! hot rather than paying an `open()` per needle.
-//!
-//! Mirrors Go's `weed/storage/needle_map_file_pool.go`. Handles are handed out
-//! as `Arc<File>`, so an eviction cannot close a descriptor a reader still
-//! holds — the file closes when the last borrower drops its `Arc`.
-
-use std::collections::{BTreeMap, HashMap};
-use std::fs::{File, OpenOptions};
-use std::io;
-use std::sync::{Arc, Mutex, OnceLock};
-
-/// Descriptors the pool keeps open. Matches Go's `maxPooledIndexFiles`.
-pub const MAX_POOLED_INDEX_FILES: usize = 1024;
-
-struct Entry {
-    file: Arc<File>,
-    tick: u64,
-}
-
-#[derive(Default)]
-struct Inner {
-    entries: HashMap<String, Entry>,
-    /// Recency order, oldest tick first, so eviction is a `pop_first`.
-    order: BTreeMap<u64, String>,
-    next_tick: u64,
-}
-
-pub struct IndexFilePool {
-    capacity: usize,
-    inner: Mutex<Inner>,
-}
-
-/// Writable and read-only handles for the same path are pooled separately so a
-/// read never depends on the file being openable for write — a volume served
-/// off a read-only mount still answers lookups.
-fn pool_key(path: &str, writable: bool) -> String {
-    if writable {
-        format!("{path}\0rw")
-    } else {
-        path.to_string()
-    }
-}
-
-impl IndexFilePool {
-    pub fn new(capacity: usize) -> Self {
-        IndexFilePool {
-            capacity: capacity.max(1),
-            inner: Mutex::new(Inner::default()),
-        }
-    }
-
-    /// Hand out an open handle for `path`, reusing the pooled one when there is
-    /// one. The descriptor lives as long as the returned `Arc`.
-    pub fn borrow(&self, path: &str, writable: bool) -> io::Result<Arc<File>> {
-        let key = pool_key(path, writable);
-        if let Some(file) = self.touch(&key) {
-            return Ok(file);
-        }
-
-        // Opened outside the lock: a cold open blocks on disk, and holding a
-        // process-wide mutex across it would serialize every volume's lookups.
-        let file = Arc::new(OpenOptions::new().read(true).write(writable).open(path)?);
-        Ok(self.insert(key, file))
-    }
-
-    /// Forget the pooled handles for `path`, so a later rename or delete of that
-    /// path cannot be served from a descriptor on the old inode.
-    pub fn discard(&self, path: &str) {
-        let mut inner = self.inner.lock().unwrap();
-        for key in [pool_key(path, false), pool_key(path, true)] {
-            if let Some(entry) = inner.entries.remove(&key) {
-                inner.order.remove(&entry.tick);
-            }
-        }
-    }
-
-    /// Descriptors currently pooled. Test-only visibility into the bound.
-    #[cfg(test)]
-    pub fn pooled_count(&self) -> usize {
-        self.inner.lock().unwrap().entries.len()
-    }
-
-    fn touch(&self, key: &str) -> Option<Arc<File>> {
-        let mut inner = self.inner.lock().unwrap();
-        let tick = inner.next_tick;
-        let entry = inner.entries.get_mut(key)?;
-        let file = entry.file.clone();
-        let old_tick = std::mem::replace(&mut entry.tick, tick);
-        inner.order.remove(&old_tick);
-        inner.order.insert(tick, key.to_string());
-        inner.next_tick += 1;
-        Some(file)
-    }
-
-    fn insert(&self, key: String, file: Arc<File>) -> Arc<File> {
-        let mut inner = self.inner.lock().unwrap();
-        if let Some(entry) = inner.entries.get(&key) {
-            // Another borrower opened the same path first; keep one descriptor.
-            return entry.file.clone();
-        }
-        let tick = inner.next_tick;
-        inner.next_tick += 1;
-        inner.order.insert(tick, key.clone());
-        inner.entries.insert(
-            key,
-            Entry {
-                file: file.clone(),
-                tick,
-            },
-        );
-        while inner.entries.len() > self.capacity {
-            let Some((_, oldest)) = inner.order.pop_first() else {
-                break;
-            };
-            inner.entries.remove(&oldest);
-        }
-        file
-    }
-}
-
-/// Process-wide pool shared by every read-only volume on this server.
-pub fn pooled_index_files() -> &'static IndexFilePool {
-    static POOL: OnceLock<IndexFilePool> = OnceLock::new();
-    POOL.get_or_init(|| IndexFilePool::new(MAX_POOLED_INDEX_FILES))
-}
-
-/// Descriptors this process holds on `.idx`/`.sdx` files under `dir`, read from
-/// `/proc/self/fd` where it exists and from `lsof` otherwise. `None` when
-/// neither is available, so a caller can skip rather than assert vacuously.
-#[cfg(test)]
-pub(crate) fn open_index_fds(dir: &std::path::Path) -> Option<usize> {
-    let prefix = std::fs::canonicalize(dir).unwrap_or_else(|_| dir.to_path_buf());
-    let is_index = |target: &std::path::Path| {
-        target.starts_with(&prefix)
-            && matches!(
-                target.extension().and_then(|e| e.to_str()),
-                Some("idx") | Some("sdx")
-            )
-    };
-
-    if let Ok(entries) = std::fs::read_dir("/proc/self/fd") {
-        return Some(
-            entries
-                .filter_map(|e| std::fs::read_link(e.ok()?.path()).ok())
-                .filter(|target| is_index(target))
-                .count(),
-        );
-    }
-
-    let out = std::process::Command::new("lsof")
-        .args(["-p", &std::process::id().to_string(), "-F", "n"])
-        .output()
-        .ok()?;
-    Some(
-        String::from_utf8_lossy(&out.stdout)
-            .lines()
-            .filter_map(|line| line.strip_prefix('n'))
-            .filter(|line| is_index(std::path::Path::new(line)))
-            .count(),
-    )
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use std::io::Write;
-
-    fn write_file(dir: &std::path::Path, name: &str, contents: &[u8]) -> String {
-        let path = dir.join(name);
-        let mut f = File::create(&path).unwrap();
-        f.write_all(contents).unwrap();
-        path.to_str().unwrap().to_string()
-    }
-
-    #[test]
-    fn evicted_handle_stays_usable_for_its_borrower() {
-        let dir = tempfile::tempdir().unwrap();
-        let first = write_file(dir.path(), "first", b"first");
-        let second = write_file(dir.path(), "second", b"second");
-
-        let pool = IndexFilePool::new(1);
-        let borrowed = pool.borrow(&first, false).unwrap();
-
-        // Pushes the single slot over, evicting the entry still in use.
-        let _other = pool.borrow(&second, false).unwrap();
-        assert_eq!(pool.pooled_count(), 1);
-
-        let mut buf = [0u8; 5];
-        #[cfg(unix)]
-        {
-            use std::os::unix::fs::FileExt;
-            borrowed.read_exact_at(&mut buf, 0).unwrap();
-        }
-        assert_eq!(&buf, b"first");
-    }
-
-    #[test]
-    fn borrow_reuses_the_pooled_handle() {
-        let dir = tempfile::tempdir().unwrap();
-        let path = write_file(dir.path(), "idx", b"x");
-
-        let pool = IndexFilePool::new(4);
-        let a = pool.borrow(&path, false).unwrap();
-        let b = pool.borrow(&path, false).unwrap();
-        assert!(Arc::ptr_eq(&a, &b));
-        assert_eq!(pool.pooled_count(), 1);
-
-        // A writable handle is pooled separately from the read-only one.
-        let w = pool.borrow(&path, true).unwrap();
-        assert!(!Arc::ptr_eq(&a, &w));
-        assert_eq!(pool.pooled_count(), 2);
-    }
-
-    #[test]
-    fn discard_drops_both_handles() {
-        let dir = tempfile::tempdir().unwrap();
-        let path = write_file(dir.path(), "idx", b"x");
-
-        let pool = IndexFilePool::new(4);
-        let _r = pool.borrow(&path, false).unwrap();
-        let _w = pool.borrow(&path, true).unwrap();
-        assert_eq!(pool.pooled_count(), 2);
-
-        pool.discard(&path);
-        assert_eq!(pool.pooled_count(), 0);
-    }
-
-    #[test]
-    fn pool_stays_within_capacity() {
-        let dir = tempfile::tempdir().unwrap();
-        let pool = IndexFilePool::new(3);
-        for i in 0..10 {
-            let path = write_file(dir.path(), &format!("f{i}"), b"x");
-            let _ = pool.borrow(&path, false).unwrap();
-        }
-        assert_eq!(pool.pooled_count(), 3);
-    }
-}

seaweed-volume/src/storage/store.rs

@@ -4,10 +4,8 @@
 //! It coordinates volume placement, lookup, and lifecycle operations.
 //! Matches Go's storage/store.go.
 
-use std::collections::HashSet;
 use std::io;
 use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
-use std::sync::Mutex;
 
 use crate::config::MinFreeSpace;
 use crate::pb::master_pb;
@@ -34,12 +32,6 @@ pub struct Store {
     pub data_center: String,
     pub rack: String,
     pub volume_report: crate::storage::volume_report::VolumeReportState,
-    /// Collections the last heartbeat set per-collection gauges for. Those
-    /// gauges are only ever set for collections still held here, so one whose
-    /// last volume leaves - moved away by volume.balance, say - would keep
-    /// reporting the heartbeat that saw it.
-    pub reported_collections: Mutex<HashSet<String>>,
-    pub reported_ec_collections: Mutex<HashSet<String>>,
 }
 
 impl Store {
@@ -57,8 +49,6 @@ impl Store {
             volume_report: Default::default(),
             data_center: String::new(),
             rack: String::new(),
-            reported_collections: Mutex::new(HashSet::new()),
-            reported_ec_collections: Mutex::new(HashSet::new()),
         }
     }
 
@@ -259,25 +249,11 @@ impl Store {
     /// Returns the index of the disk that should receive a new EC
     /// shard / index file for `(collection, vid)`. Selection order:
     ///
-    /// 0. a disk that already owns one of `shard_ids` (in-memory claim),
     /// 1. a disk that already has the EC volume mounted (in-memory state),
     /// 2. a disk that owns the `.ecx` file on disk (volume not yet mounted),
     /// 3. any HDD with free space,
     /// 4. any disk with free space.
     ///
-    /// Step 0 keeps the per-server invariant that a shard id is owned by
-    /// at most one disk. Steps 1-4 only know the volume, and a multi-disk
-    /// server legitimately mounts the same vid on several disks, so the
-    /// free-count tie-break alone can send a re-copy of a shard the server
-    /// already holds (a retried `ec.balance` / `ec.rebuild` move) to a
-    /// sibling disk. Both disks then claim the same (vid, shard) and
-    /// report it to the master from two disk ids, and which claimant
-    /// serves reads or survives a later unmount/delete of the shard id
-    /// becomes an accident of location order. Overwriting in place is what
-    /// the caller meant, so an owning disk wins ahead of the space filters
-    /// too — a re-copy needs no new shard slot, and a genuinely full disk
-    /// fails the write rather than silently splitting the claim.
-    ///
     /// Step 2 is the missing primitive that pinned subsequent shards to
     /// the first-shard disk during `ec.rebuild`: rebuild only sets
     /// `CopyEcxFile=true` on the first shard, then relies on auto-select
@@ -300,26 +276,20 @@ impl Store {
         collection: &str,
         vid: VolumeId,
         data_shard_count: u32,
-        shard_ids: &[u32],
     ) -> Option<usize> {
         const TIER_ANY_DISK: u8 = 1;
         const TIER_HDD: u8 = 2;
         const TIER_ECX_ON_DISK: u8 = 3;
         const TIER_MOUNTED: u8 = 4;
-        const TIER_OWNS_SHARD: u8 = 5;
 
-        // (index, tier, owned shard count, free shard slots)
-        let mut best: Option<(usize, u8, usize, i64)> = None;
+        let mut best: Option<(usize, u8, i64)> = None;
         for (i, loc) in self.locations.iter().enumerate() {
-            let owned = owned_ec_shard_count(loc, vid, shard_ids);
+            if loc.is_disk_space_low.load(Ordering::Relaxed) {
+                continue;
+            }
             let free = ec_free_shard_count(loc, data_shard_count);
-            if owned == 0 {
-                if loc.is_disk_space_low.load(Ordering::Relaxed) {
-                    continue;
-                }
-                if free <= 0 {
-                    continue;
-                }
+            if free <= 0 {
+                continue;
             }
             let mut tier = TIER_ANY_DISK;
             if loc.disk_type == DiskType::HardDrive {
@@ -331,41 +301,15 @@ impl Store {
             if loc.has_ec_volume(vid) {
                 tier = TIER_MOUNTED;
             }
-            if owned > 0 {
-                tier = TIER_OWNS_SHARD;
-            }
             let better = match best {
                 None => true,
-                // owned only separates disks inside TIER_OWNS_SHARD; it is 0
-                // everywhere else, so this falls through to the free-count
-                // tie-break for the other tiers.
-                Some((_, b_tier, b_owned, b_free)) => {
-                    tier > b_tier
-                        || (tier == b_tier
-                            && (owned > b_owned || (owned == b_owned && free > b_free)))
-                }
+                Some((_, b_tier, b_free)) => tier > b_tier || (tier == b_tier && free > b_free),
             };
             if better {
-                best = Some((i, tier, owned, free));
+                best = Some((i, tier, free));
             }
         }
-        best.map(|(i, _, _, _)| i)
-    }
-
-    /// Returns the distinct disk indexes that already own one of `shard_ids`
-    /// for `vid`, in location order. More than one owner means the batch has
-    /// no single correct destination — whichever disk receives it would
-    /// duplicate a sibling disk's claim — so batch callers must split by
-    /// owner (`volume_ec_shards_copy` refuses such a batch instead of
-    /// guessing). Mirrors `Store.EcShardOwnerDisks` in
-    /// `weed/storage/store_ec.go`.
-    pub fn ec_shard_owner_disks(&self, vid: VolumeId, shard_ids: &[u32]) -> Vec<usize> {
-        self.locations
-            .iter()
-            .enumerate()
-            .filter(|(_, loc)| owned_ec_shard_count(loc, vid, shard_ids) > 0)
-            .map(|(i, _)| i)
-            .collect()
+        best.map(|(i, _, _)| i)
     }
 
     /// Create a new volume, placing it on the location with the most free space.
@@ -850,37 +794,6 @@ impl Store {
     }
 
     /// Find an EC volume across all locations (mutable).
-    /// Every per-disk `EcVolume` this store maps for `vid`. A vid can mount on
-    /// N disks as N distinct runtimes, and the first-match `find_ec_volume_mut`
-    /// hides the siblings — so anything that has to reach the whole volume,
-    /// rather than any one runtime of it, iterates this instead.
-    /// Every directory on this server that could hold an EC volume's metadata —
-    /// each disk's data and index directory. Startup mirroring gives each
-    /// shard-bearing disk its own .ecx/.ecj/.vif, but the checksum sidecar is
-    /// not mirrored and a repair delivers exactly one copy, so a runtime looking
-    /// only at its own two directories cannot see it. Handing this list to the
-    /// sidecar resolution keeps one authoritative copy reachable from every
-    /// runtime rather than duplicating a file that is rewritten as shards are
-    /// repaired and generations published.
-    pub fn ec_metadata_dirs(&self) -> Vec<String> {
-        let mut dirs: Vec<String> = Vec::with_capacity(self.locations.len() * 2);
-        for loc in &self.locations {
-            for dir in [&loc.directory, &loc.idx_directory] {
-                if !dir.is_empty() && !dirs.iter().any(|d| d == dir) {
-                    dirs.push(dir.clone());
-                }
-            }
-        }
-        dirs
-    }
-
-    pub fn find_all_ec_volumes_mut(&mut self, vid: VolumeId) -> Vec<&mut EcVolume> {
-        self.locations
-            .iter_mut()
-            .filter_map(|loc| loc.find_ec_volume_mut(vid))
-            .collect()
-    }
-
     pub fn find_ec_volume_mut(&mut self, vid: VolumeId) -> Option<&mut EcVolume> {
         for loc in &mut self.locations {
             if let Some(ecv) = loc.find_ec_volume_mut(vid) {
@@ -1379,20 +1292,6 @@ fn ec_free_shard_count(loc: &DiskLocation, data_shard_count: u32) -> i64 {
     free
 }
 
-/// Reports how many of `shard_ids` this disk already claims for `vid`, per
-/// the in-memory registration the read path and heartbeats use.
-///
-/// Mirrors `ownedEcShardCount` in `weed/storage/store_ec.go`.
-fn owned_ec_shard_count(loc: &DiskLocation, vid: VolumeId, shard_ids: &[u32]) -> usize {
-    let Some(ecv) = loc.find_ec_volume(vid) else {
-        return 0;
-    };
-    shard_ids
-        .iter()
-        .filter(|&&shard_id| ecv.has_shard(shard_id as u8))
-        .count()
-}
-
 // ============================================================================
 // Tests
 // ============================================================================
@@ -1931,7 +1830,7 @@ mod tests {
         let base = volume_file_name(&store.locations[2].idx_directory, collection, vid);
         std::fs::write(format!("{}.ecx", base), vec![0u8; 20]).unwrap();
 
-        let got = store.find_ec_shard_target_location(collection, vid, 10, &[]);
+        let got = store.find_ec_shard_target_location(collection, vid, 10);
         assert_eq!(
             got,
             Some(2),
@@ -2073,7 +1972,7 @@ mod tests {
         let base = volume_file_name(&store.locations[2].idx_directory, collection, vid);
         std::fs::write(format!("{}.ecx", base), vec![0u8; 20]).unwrap();
 
-        let got = store.find_ec_shard_target_location(collection, vid, 10, &[]);
+        let got = store.find_ec_shard_target_location(collection, vid, 10);
         assert_eq!(got, Some(1), "expected the mounted disk to win; got {:?}", got);
     }
 
@@ -2082,7 +1981,7 @@ mod tests {
     #[test]
     fn test_find_ec_shard_target_location_falls_through_to_hdd_when_nothing_matches() {
         let (store, _tmp) = make_ec_target_test_store(2);
-        let got = store.find_ec_shard_target_location("grafana-loki", VolumeId(3333), 10, &[]);
+        let got = store.find_ec_shard_target_location("grafana-loki", VolumeId(3333), 10);
         assert!(got.is_some(), "expected an HDD fallback");
         assert_eq!(store.locations[got.unwrap()].disk_type, DiskType::HardDrive);
     }
@@ -2098,7 +1997,7 @@ mod tests {
             .max_volume_count
             .store(0, Ordering::Relaxed);
 
-        let got = store.find_ec_shard_target_location("grafana-loki", VolumeId(4444), 10, &[]);
+        let got = store.find_ec_shard_target_location("grafana-loki", VolumeId(4444), 10);
         assert_eq!(
             got,
             Some(0),
@@ -2136,7 +2035,7 @@ mod tests {
             .mount_ec_shards(vid, collection, &[0], "")
             .unwrap();
 
-        let got = store.find_ec_shard_target_location(collection, vid, 10, &[]);
+        let got = store.find_ec_shard_target_location(collection, vid, 10);
         assert_eq!(
             got,
             Some(1),
@@ -2144,130 +2043,4 @@ mod tests {
             got,
         );
     }
-
-    /// Per-server invariant: a shard id is owned by at most one disk.
-    ///
-    /// A multi-disk server legitimately mounts one vid on several disks, each
-    /// holding a disjoint subset of the shards, so the mounted tier ties and
-    /// the free-count tie-break decides — and it points at whichever disk
-    /// happens to be emptier, not at the disk that already has this shard. A
-    /// re-copy of a shard the server already holds (a retried `ec.balance` /
-    /// `ec.rebuild` move) then lands a second copy on the sibling disk, and
-    /// both disks register the same shard id.
-    #[test]
-    fn test_find_ec_shard_target_location_pins_to_the_disk_owning_the_shard() {
-        let (mut store, _tmp) = make_ec_target_test_store(2);
-        let collection = "grafana-loki";
-        let vid = VolumeId(8888);
-
-        // Disk 0 owns shards 0 and 1, disk 1 owns shard 2 — so disk 1 is the
-        // emptier of the two and wins the free-count tie-break.
-        let base0 = volume_file_name(&store.locations[0].directory, collection, vid);
-        std::fs::write(format!("{}.ec00", base0), b"x").unwrap();
-        std::fs::write(format!("{}.ec01", base0), b"x").unwrap();
-        store.locations[0]
-            .mount_ec_shards(vid, collection, &[0, 1], "")
-            .unwrap();
-
-        let base1 = volume_file_name(&store.locations[1].directory, collection, vid);
-        std::fs::write(format!("{}.ec02", base1), b"x").unwrap();
-        store.locations[1]
-            .mount_ec_shards(vid, collection, &[2], "")
-            .unwrap();
-
-        assert_eq!(
-            store.find_ec_shard_target_location(collection, vid, 10, &[0]),
-            Some(0),
-            "a copy of shard 0 left its owning disk",
-        );
-        assert_eq!(
-            store.find_ec_shard_target_location(collection, vid, 10, &[2]),
-            Some(1),
-            "a copy of shard 2 left its owning disk",
-        );
-        // A shard no disk owns yet is placed by the unchanged waterfall: both
-        // disks have it mounted, so the emptier one wins.
-        assert_eq!(
-            store.find_ec_shard_target_location(collection, vid, 10, &[7]),
-            Some(1),
-            "placement of an unclaimed shard changed",
-        );
-    }
-
-    /// The second half of the invariant: a disk with no free shard slots
-    /// still wins for a shard it already owns. Re-copying that shard
-    /// overwrites bytes the disk is already accounted for, while routing to a
-    /// sibling splits the claim across two disks.
-    #[test]
-    fn test_find_ec_shard_target_location_owning_disk_wins_when_full() {
-        let (mut store, _tmp) = make_ec_target_test_store(2);
-        store.locations[0]
-            .max_volume_count
-            .store(1, Ordering::Relaxed);
-
-        let collection = "grafana-loki";
-        let vid = VolumeId(9999);
-
-        let base = volume_file_name(&store.locations[0].directory, collection, vid);
-        std::fs::write(format!("{}.ec00", base), b"x").unwrap();
-        store.locations[0]
-            .mount_ec_shards(vid, collection, &[0], "")
-            .unwrap();
-
-        // Fill disk 0 past its shard-slot budget so ec_free_shard_count is 0.
-        let filler = VolumeId(10000);
-        let filler_base = volume_file_name(&store.locations[0].directory, collection, filler);
-        let filler_shards: Vec<u32> = (0..10).collect();
-        for shard_id in &filler_shards {
-            std::fs::write(format!("{}.ec{:02}", filler_base, shard_id), b"x").unwrap();
-        }
-        store.locations[0]
-            .mount_ec_shards(filler, collection, &filler_shards, "")
-            .unwrap();
-
-        assert_eq!(
-            store.find_ec_shard_target_location(collection, vid, 10, &[0]),
-            Some(0),
-            "a copy of shard 0 left its owning disk when the disk was full",
-        );
-        // A shard it does not own still respects the space filter.
-        assert_eq!(
-            store.find_ec_shard_target_location(collection, vid, 10, &[7]),
-            Some(1),
-            "an unclaimed shard should go to the disk with free slots",
-        );
-    }
-
-    /// Mixed-owner batch contract: a batch whose requested shards are
-    /// already owned by different disks reports every owner, so
-    /// `volume_ec_shards_copy` can refuse it rather than rank the owners
-    /// into one destination and duplicate the loser's claim.
-    #[test]
-    fn test_ec_shard_owner_disks() {
-        let (mut store, _tmp) = make_ec_target_test_store(3);
-        let collection = "grafana-loki";
-        let vid = VolumeId(11111);
-
-        let base0 = volume_file_name(&store.locations[0].directory, collection, vid);
-        std::fs::write(format!("{}.ec00", base0), b"x").unwrap();
-        std::fs::write(format!("{}.ec01", base0), b"x").unwrap();
-        store.locations[0]
-            .mount_ec_shards(vid, collection, &[0, 1], "")
-            .unwrap();
-        let base1 = volume_file_name(&store.locations[1].directory, collection, vid);
-        std::fs::write(format!("{}.ec02", base1), b"x").unwrap();
-        store.locations[1]
-            .mount_ec_shards(vid, collection, &[2], "")
-            .unwrap();
-
-        assert_eq!(
-            store.ec_shard_owner_disks(vid, &[0, 2]),
-            vec![0, 1],
-            "a batch owned by two disks must report both owners",
-        );
-        // A batch on one disk, with or without unowned extras, has one owner.
-        assert_eq!(store.ec_shard_owner_disks(vid, &[0, 1, 7]), vec![0]);
-        // A wholly unowned batch reports none — fresh placement stays allowed.
-        assert!(store.ec_shard_owner_disks(vid, &[7, 8]).is_empty());
-    }
 }

seaweed-volume/src/storage/volume_report.rs

@@ -1,6 +1,7 @@
 //! Mirror of `weed/storage/store_volume_report.go`.
 
 use std::collections::HashMap;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
 use std::sync::Mutex;
 
 /// Identifies one reported copy. Keyed by disk as well as id because a volume
@@ -8,34 +9,6 @@ use std::sync::Mutex;
 /// other's changes untold.
 pub type VolumeReportKey = (u32, u32);
 
-/// What the master was told about one volume copy: the hash that detects
-/// change, and the heartbeat pass that last found the copy held.
-#[derive(Clone, Copy)]
-struct ReportedVolume {
-    hash: u64,
-    pass: u64,
-}
-
-/// Everything a heartbeat reads and writes about what the master was told,
-/// under one lock. The full-list flag and the generation that answers it have
-/// to move together: split across atomics, a request landing between two of
-/// them is answered by a heartbeat that never carried a list. Go holds a single
-/// mutex over the same fields.
-#[derive(Default)]
-struct ReportState {
-    /// Set once the master says it compares digests. Until then the whole list
-    /// goes every time, which is what an older master needs.
-    deltas_accepted: bool,
-    full_list_needed: bool,
-    /// Counts requests for the whole list, so one arriving while a heartbeat is
-    /// being built is not marked satisfied by it.
-    full_list_generation: u64,
-    /// Numbers heartbeats, so one can mark the copies it finds held without
-    /// building a second map of them.
-    pass: u64,
-    last_reported: HashMap<VolumeReportKey, ReportedVolume>,
-}
-
 /// Remembers what the master was last told about each volume, so a heartbeat
 /// can carry only what moved since.
 ///
@@ -45,86 +18,56 @@ struct ReportState {
 /// until one accepts changes.
 #[derive(Default)]
 pub struct VolumeReportState {
-    state: Mutex<ReportState>,
+    /// Set once the master says it compares digests. Until then the whole list
+    /// goes every time, which is what an older master needs.
+    deltas_accepted: AtomicBool,
+    full_list_needed: AtomicBool,
+    /// Counts requests for the whole list, so one arriving while a heartbeat is
+    /// being built is not marked satisfied by it.
+    full_list_generation: AtomicU64,
+    last_reported: Mutex<HashMap<VolumeReportKey, u64>>,
 }
 
 impl VolumeReportState {
     /// Drops everything known about the master's view.
     pub fn reset(&self) {
-        let mut state = self.state.lock().unwrap();
-        state.deltas_accepted = false;
-        state.full_list_needed = true;
-        state.full_list_generation += 1;
-        state.last_reported.clear();
+        self.deltas_accepted.store(false, Ordering::Relaxed);
+        self.full_list_needed.store(true, Ordering::Relaxed);
+        self.full_list_generation.fetch_add(1, Ordering::Relaxed);
+        self.last_reported.lock().unwrap().clear();
     }
 
     pub fn accept_deltas(&self) {
-        self.state.lock().unwrap().deltas_accepted = true;
+        self.deltas_accepted.store(true, Ordering::Relaxed);
     }
 
     pub fn request_full_list(&self) {
-        let mut state = self.state.lock().unwrap();
-        state.full_list_needed = true;
-        state.full_list_generation += 1;
+        self.full_list_needed.store(true, Ordering::Relaxed);
+        self.full_list_generation.fetch_add(1, Ordering::Relaxed);
     }
 
-    /// Opens a heartbeat: whether it must carry the whole list, the request it
-    /// answers, and the pass number that marks the copies it finds still held.
-    pub fn begin(&self) -> (bool, u64, u64) {
-        let mut state = self.state.lock().unwrap();
-        state.pass += 1;
-        (
-            state.full_list_needed || !state.deltas_accepted,
-            state.full_list_generation,
-            state.pass,
-        )
+    /// Reports whether this heartbeat must carry the whole list, and the
+    /// request it answers.
+    pub fn begin(&self) -> (bool, u64) {
+        let full = self.full_list_needed.load(Ordering::Relaxed)
+            || !self.deltas_accepted.load(Ordering::Relaxed);
+        (full, self.full_list_generation.load(Ordering::Relaxed))
     }
 
     /// Reports whether the master needs telling about this volume, given what
-    /// it was last told. For a caller that is only taking a snapshot and so
-    /// must leave the reporting state alone; a heartbeat calls `record`.
+    /// it was last told.
     pub fn changed(&self, key: VolumeReportKey, hash: u64) -> bool {
-        self.state
-            .lock()
-            .unwrap()
-            .last_reported
-            .get(&key)
-            .is_none_or(|previous| previous.hash != hash)
+        self.last_reported.lock().unwrap().get(&key) != Some(&hash)
     }
 
-    /// Marks one volume copy as held by the heartbeat being built, and reports
-    /// whether the master needs telling about it. It updates the entry already
-    /// held rather than build a second map beside it, so a server whose volumes
-    /// are quiet allocates nothing per volume per heartbeat.
-    pub fn record(&self, key: VolumeReportKey, hash: u64, pass: u64) -> bool {
-        let mut state = self.state.lock().unwrap();
-        match state.last_reported.get_mut(&key) {
-            Some(previous) => {
-                let changed = previous.hash != hash;
-                previous.hash = hash;
-                previous.pass = pass;
-                changed
-            }
-            None => {
-                state
-                    .last_reported
-                    .insert(key, ReportedVolume { hash, pass });
-                true
-            }
-        }
-    }
-
-    /// Closes the heartbeat. Copies this pass did not find are forgotten, so one
-    /// that comes back is reported again.
-    pub fn commit(&self, pass: u64, generation: u64) {
-        let mut state = self.state.lock().unwrap();
-        state
-            .last_reported
-            .retain(|_, reported| reported.pass == pass);
+    /// Records what this heartbeat told the master. Volumes absent from
+    /// `reported` are forgotten, so one that comes back is reported again.
+    pub fn commit(&self, reported: HashMap<VolumeReportKey, u64>, generation: u64) {
+        *self.last_reported.lock().unwrap() = reported;
         // A request that arrived while this heartbeat was being built asked
         // about a later state than it carries, so it stands.
-        if state.full_list_generation == generation {
-            state.full_list_needed = false;
+        if self.full_list_generation.load(Ordering::Relaxed) == generation {
+            self.full_list_needed.store(false, Ordering::Relaxed);
         }
     }
 }

seaweed-worker/Cargo.lock

@@ -4666,70 +4666,6 @@ dependencies = [
  "prost 0.14.4",
 ]
 
-[[package]]
-name = "protoc-bin-vendored"
-version = "3.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d1c381df33c98266b5f08186583660090a4ffa0889e76c7e9a5e175f645a67fa"
-dependencies = [
- "protoc-bin-vendored-linux-aarch_64",
- "protoc-bin-vendored-linux-ppcle_64",
- "protoc-bin-vendored-linux-s390_64",
- "protoc-bin-vendored-linux-x86_32",
- "protoc-bin-vendored-linux-x86_64",
- "protoc-bin-vendored-macos-aarch_64",
- "protoc-bin-vendored-macos-x86_64",
- "protoc-bin-vendored-win32",
-]
-
-[[package]]
-name = "protoc-bin-vendored-linux-aarch_64"
-version = "3.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c350df4d49b5b9e3ca79f7e646fde2377b199e13cfa87320308397e1f37e1a4c"
-
-[[package]]
-name = "protoc-bin-vendored-linux-ppcle_64"
-version = "3.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a55a63e6c7244f19b5c6393f025017eb5d793fd5467823a099740a7a4222440c"
-
-[[package]]
-name = "protoc-bin-vendored-linux-s390_64"
-version = "3.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1dba5565db4288e935d5330a07c264a4ee8e4a5b4a4e6f4e83fad824cc32f3b0"
-
-[[package]]
-name = "protoc-bin-vendored-linux-x86_32"
-version = "3.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8854774b24ee28b7868cd71dccaae8e02a2365e67a4a87a6cd11ee6cdbdf9cf5"
-
-[[package]]
-name = "protoc-bin-vendored-linux-x86_64"
-version = "3.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b38b07546580df720fa464ce124c4b03630a6fb83e05c336fea2a241df7e5d78"
-
-[[package]]
-name = "protoc-bin-vendored-macos-aarch_64"
-version = "3.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "89278a9926ce312e51f1d999fee8825d324d603213344a9a706daa009f1d8092"
-
-[[package]]
-name = "protoc-bin-vendored-macos-x86_64"
-version = "3.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "81745feda7ccfb9471d7a4de888f0652e806d5795b61480605d4943176299756"
-
-[[package]]
-name = "protoc-bin-vendored-win32"
-version = "3.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "95067976aca6421a523e491fce939a3e65249bac4b977adee0ee9771568e8aa3"
-
 [[package]]
 name = "quick-xml"
 version = "0.39.4"
@@ -5443,7 +5379,6 @@ dependencies = [
  "prometheus",
  "prost 0.13.5",
  "prost-types 0.13.5",
- "protoc-bin-vendored",
  "tokio",
  "tokio-stream",
  "tonic",

seaweed-worker/README.md

@@ -12,17 +12,6 @@ one.
 `core` knows nothing about any job. A second worker is a new crate beside
 `lance` that depends on it, not a fork of the protocol.
 
-## Building
-
-`core` compiles `plugin.proto` with the protoc that protoc-bin-vendored ships,
-the way seaweed-volume does, so it needs no system install.
-
-The lance crates compile protos of their own, in their own build-script
-processes, which nothing our build script sets can reach. They need a protoc of
-their own: either one on PATH — `brew install protobuf`, `apt install
-protobuf-compiler` — or `PROTOC` naming one. CI points it at the vendored
-binary for the runner's platform, resolved from the version in `Cargo.lock`.
-
 ## Running
 
     cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646
@@ -31,22 +20,6 @@ The admin's *HTTP* address is what an operator has; the gRPC port is derived
 from it the way the Go side does. Dialling the HTTP port fails as "frame with
 invalid size", which reads like a protocol bug rather than a wrong port.
 
-The binary is `weed-worker`, not `weed-lance-worker`: it is the Rust side of
-`weed worker`, and lance is the first family of jobs it carries rather than the
-only one it ever will.
-
-Released builds do not need a toolchain. The worker ships inside the SeaweedFS
-image, beside the Rust volume server, under the verb that mirrors
-`volume-rust`:
-
-    docker run chrislusf/seaweedfs worker-rust --admin admin:23646
-
-and as `weed-worker_linux_{amd64,arm64}.tar.gz` on each GitHub release. Both are
-linux amd64/arm64 only — lance, arrow and datafusion make every extra target an
-expensive build, and the worker runs beside the cluster it maintains. On an
-architecture without a build the image carries an empty placeholder and the
-entrypoint says so rather than failing as "not found".
-
 ## Metrics
 
     cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646 --metrics-port 9328

seaweed-worker/crates/core/Cargo.toml

@@ -21,7 +21,3 @@ tracing.workspace = true
 
 [build-dependencies]
 tonic-build.workspace = true
-# Ships protoc with the build so neither CI nor a developer needs a system
-# install, and so the version is pinned rather than whatever the platform's
-# package manager happens to carry. The same crate seaweed-volume uses.
-protoc-bin-vendored = "3"

seaweed-worker/crates/core/build.rs

@@ -1,12 +1,4 @@
 fn main() -> Result<(), Box<dyn std::error::Error>> {
-    // Use the protoc that ships with protoc-bin-vendored rather than a system
-    // one, so the build needs no package manager and always sees the same
-    // version. An explicit PROTOC still wins, for packagers supplying their own
-    // and for the lance crates, whose own build scripts read the same variable.
-    if std::env::var_os("PROTOC").is_none() {
-        std::env::set_var("PROTOC", protoc_bin_vendored::protoc_bin_path()?);
-    }
-
     // Compiled straight out of the Go tree, the way seaweed-volume already reads
     // filer.proto, so the contract cannot drift from a vendored copy.
     tonic_build::configure()

seaweed-worker/crates/lance/Cargo.toml

@@ -7,11 +7,8 @@ description = "SeaweedFS maintenance worker for Lance tables"
 [lib]
 name = "weed_lance_worker"
 
-# The binary is not named for lance: it is the Rust side of `weed worker`, and
-# the job families it registers will outgrow this crate. When a second one
-# arrives the bin target moves to a crate of its own under the same name.
 [[bin]]
-name = "weed-worker"
+name = "weed-lance-worker"
 path = "src/main.rs"
 
 [dependencies]

seaweed-worker/crates/lance/src/main.rs

@@ -13,8 +13,8 @@ use weed_lance_worker::metrics::LanceMetrics;
 /// language and an operator should not have to learn a second set of names.
 #[derive(Parser, Debug)]
 #[command(
-    name = "weed-worker",
-    about = "SeaweedFS maintenance worker"
+    name = "weed-lance-worker",
+    about = "SeaweedFS maintenance worker for Lance tables"
 )]
 struct Args {
     /// Admin server gRPC address.

seaweed-worker/crates/lance/tests/common/mod.rs

@@ -1,97 +0,0 @@
-//! What every integration test in this crate needs to drive a handler: a sender
-//! that keeps what the handler sent, and the two settings that make a handler
-//! usable against a test gateway.
-//!
-//! Each test binary compiles this module on its own and uses part of it.
-#![allow(dead_code)]
-
-use std::collections::HashMap;
-use std::sync::Mutex;
-
-use anyhow::Result;
-use seaweed_worker_core::pb::{
-    config_value::Kind, ActivityEvent, ConfigValue, DetectionComplete, DetectionProposals,
-    JobCompleted, JobProgressUpdate, JobProposal, ObjectObservation, WorkerObservations,
-};
-use seaweed_worker_core::{DetectionSender, ExecutionSender};
-
-/// Keeps everything a handler reports, so a test can assert on it afterwards.
-#[derive(Default)]
-pub struct Recorder {
-    pub proposals: Mutex<Vec<JobProposal>>,
-    pub observations: Mutex<Vec<ObjectObservation>>,
-    pub completed: Mutex<Vec<JobCompleted>>,
-}
-
-impl DetectionSender for Recorder {
-    fn send_proposals(&self, proposals: DetectionProposals) -> Result<()> {
-        self.proposals.lock().unwrap().extend(proposals.proposals);
-        Ok(())
-    }
-    fn send_complete(&self, _complete: DetectionComplete) -> Result<()> {
-        Ok(())
-    }
-    fn send_activity(&self, _activity: ActivityEvent) -> Result<()> {
-        Ok(())
-    }
-    fn send_observations(&self, observations: WorkerObservations) -> Result<()> {
-        self.observations
-            .lock()
-            .unwrap()
-            .extend(observations.observations);
-        Ok(())
-    }
-}
-
-impl ExecutionSender for Recorder {
-    fn send_progress(&self, _progress: JobProgressUpdate) -> Result<()> {
-        Ok(())
-    }
-    fn send_completed(&self, completed: JobCompleted) -> Result<()> {
-        self.completed.lock().unwrap().push(completed);
-        Ok(())
-    }
-}
-
-/// The gateway to run against, or nothing, in which case the caller skips:
-/// these handlers rewrite real files, and there is nothing to learn from them
-/// against a fake.
-pub fn namespace_url() -> Option<String> {
-    std::env::var("WEED_LANCE_NAMESPACE")
-        .ok()
-        .filter(|s| !s.is_empty())
-}
-
-pub fn int_config(name: &str, value: i64) -> HashMap<String, ConfigValue> {
-    let mut values = HashMap::new();
-    values.insert(
-        name.to_string(),
-        ConfigValue {
-            kind: Some(Kind::Int64Value(value)),
-        },
-    );
-    values
-}
-
-/// What to reach the data with where the namespace vends nothing of its own -
-/// a gateway without STS vends no credentials, and one bound to a wildcard
-/// address vends no endpoint either. Taken from the environment so a harness
-/// can point these tests at a gateway that does check what it is given.
-pub fn fallback() -> weed_lance_worker::dataset::FallbackOptions {
-    let mut options = weed_lance_worker::dataset::FallbackOptions::new();
-    let from_env =
-        |name: &str, default: &str| std::env::var(name).unwrap_or_else(|_| default.to_string());
-    options.insert(
-        "aws_access_key_id".to_string(),
-        from_env("AWS_ACCESS_KEY_ID", "any"),
-    );
-    options.insert(
-        "aws_secret_access_key".to_string(),
-        from_env("AWS_SECRET_ACCESS_KEY", "any"),
-    );
-    if let Ok(endpoint) = std::env::var("AWS_ENDPOINT_URL") {
-        options.insert("aws_endpoint".to_string(), endpoint);
-        options.insert("allow_http".to_string(), "true".to_string());
-    }
-    options
-}

seaweed-worker/crates/lance/tests/compaction.rs

@@ -5,26 +5,82 @@
 //! nothing to learn from it against a fake.
 
 use std::collections::HashMap;
+use std::sync::Mutex;
 
 use anyhow::Result;
 use seaweed_worker_core::pb::{
-    config_value::Kind, ExecuteJobRequest, JobSpec, RunDetectionRequest,
+    config_value::Kind, ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest,
+    JobCompleted, JobProgressUpdate, JobProposal, JobSpec, RunDetectionRequest,
 };
-use seaweed_worker_core::{JobHandler, PreviewProvider};
+use seaweed_worker_core::{DetectionSender, ExecutionSender, JobHandler, PreviewProvider};
 use weed_lance_worker::catalog::NamespaceClient;
 use weed_lance_worker::jobs::cleanup::CleanupVersionsHandler;
 use weed_lance_worker::jobs::compact::{CompactHandler, JOB_TYPE};
 use weed_lance_worker::jobs::indices::OptimizeIndicesHandler;
 use weed_lance_worker::preview::LancePreview;
 
-mod common;
-use common::{fallback, int_config, namespace_url, Recorder};
+#[derive(Default)]
+struct Recorder {
+    proposals: Mutex<Vec<JobProposal>>,
+    observations: Mutex<Vec<seaweed_worker_core::pb::ObjectObservation>>,
+    completed: Mutex<Vec<JobCompleted>>,
+}
+
+impl DetectionSender for Recorder {
+    fn send_proposals(&self, proposals: DetectionProposals) -> Result<()> {
+        self.proposals.lock().unwrap().extend(proposals.proposals);
+        Ok(())
+    }
+    fn send_complete(&self, _complete: DetectionComplete) -> Result<()> {
+        Ok(())
+    }
+    fn send_activity(&self, _activity: seaweed_worker_core::pb::ActivityEvent) -> Result<()> {
+        Ok(())
+    }
+    fn send_observations(
+        &self,
+        observations: seaweed_worker_core::pb::WorkerObservations,
+    ) -> Result<()> {
+        self.observations
+            .lock()
+            .unwrap()
+            .extend(observations.observations);
+        Ok(())
+    }
+}
+
+impl ExecutionSender for Recorder {
+    fn send_progress(&self, _progress: JobProgressUpdate) -> Result<()> {
+        Ok(())
+    }
+    fn send_completed(&self, completed: JobCompleted) -> Result<()> {
+        self.completed.lock().unwrap().push(completed);
+        Ok(())
+    }
+}
 
 /// These tests drive one live gateway and one shared catalog: `list_all_tables`
 /// sweeps everything, so a table another test is writing shows up in this test's
 /// detection. Rust runs a binary's tests concurrently, so take a lock.
 static GATEWAY: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
 
+fn namespace_url() -> Option<String> {
+    std::env::var("WEED_LANCE_NAMESPACE")
+        .ok()
+        .filter(|s| !s.is_empty())
+}
+
+fn int_config(name: &str, value: i64) -> HashMap<String, ConfigValue> {
+    let mut values = HashMap::new();
+    values.insert(
+        name.to_string(),
+        ConfigValue {
+            kind: Some(Kind::Int64Value(value)),
+        },
+    );
+    values
+}
+
 /// Declares a table through the namespace and writes `fragments` one-row
 /// appends into it, so a test brings its own state instead of depending on
 /// whatever a previous run left behind.
@@ -149,12 +205,15 @@ async fn compacts_a_fragmented_table() {
         eprintln!("WEED_LANCE_NAMESPACE is unset, skipping");
         return;
     };
+    let mut fallback = weed_lance_worker::dataset::FallbackOptions::new();
+    fallback.insert("aws_access_key_id".to_string(), "any".to_string());
+    fallback.insert("aws_secret_access_key".to_string(), "any".to_string());
     // Seeded here rather than by a script, so the test is repeatable: a previous
     // run compacts the table it depended on.
     let encoded = seed_fragmented_table(&url, "compactme", 12)
         .await
         .expect("seed a fragmented table");
-    let handler = CompactHandler::new(url).with_fallback(fallback());
+    let handler = CompactHandler::new(url).with_fallback(fallback);
     let recorder = Recorder::default();
 
     let request = RunDetectionRequest {
@@ -229,6 +288,13 @@ async fn compacts_a_fragmented_table() {
     eprintln!("compaction result: {summary}");
 }
 
+fn fallback() -> weed_lance_worker::dataset::FallbackOptions {
+    let mut options = weed_lance_worker::dataset::FallbackOptions::new();
+    options.insert("aws_access_key_id".to_string(), "any".to_string());
+    options.insert("aws_secret_access_key".to_string(), "any".to_string());
+    options
+}
+
 /// A table with more versions than the floor is proposed, and running the job
 /// reports what it removed. The compaction test above leaves one behind.
 #[tokio::test]

seaweed-worker/crates/lance/tests/lifecycle.rs

@@ -1,123 +0,0 @@
-//! Maintains one table someone else wrote, and nothing else.
-//!
-//! The Go suite in test/s3tables/lifecycle drives a table through its whole
-//! life - declared through the namespace, written by a Lance client, maintained,
-//! read again, dropped - and this is the maintenance step of it, so that step
-//! goes through the handlers a deployed worker runs rather than through the two
-//! lance calls they wrap. The table to work on comes in as WEED_LANCE_TABLE
-//! because the table is the Go test's, seeded and checked there.
-//!
-//! Skipped, like the rest of this crate's integration tests, unless
-//! WEED_LANCE_NAMESPACE names a live gateway.
-
-use std::collections::HashMap;
-
-use seaweed_worker_core::pb::{ConfigValue, ExecuteJobRequest, JobSpec, RunDetectionRequest};
-use seaweed_worker_core::JobHandler;
-use weed_lance_worker::jobs::cleanup::{self, CleanupVersionsHandler};
-use weed_lance_worker::jobs::compact::{CompactHandler, JOB_TYPE as COMPACT_JOB_TYPE};
-
-mod common;
-use common::{fallback, int_config, namespace_url, Recorder};
-
-fn table() -> Option<String> {
-    std::env::var("WEED_LANCE_TABLE")
-        .ok()
-        .filter(|s| !s.is_empty())
-}
-
-/// Compacts the named table's fragments, then drops the versions compaction
-/// left behind. Both go through the handler's own detect-then-execute path: a
-/// proposal the worker would not have made is not one worth running.
-#[tokio::test]
-async fn maintains_the_named_table() {
-    let (Some(url), Some(name)) = (namespace_url(), table()) else {
-        eprintln!("WEED_LANCE_NAMESPACE or WEED_LANCE_TABLE is unset, skipping");
-        return;
-    };
-
-    // A table written as a fragment per append is the case this exists for, so
-    // anything above one fragment is worth merging here.
-    let mut config = int_config("min_fragments", 2);
-    config.extend(int_config("target_rows_per_fragment", 1_048_576));
-    let compact = CompactHandler::new(url.clone()).with_fallback(fallback());
-    run(&compact, COMPACT_JOB_TYPE, &name, config, true).await;
-
-    // Keep the current version and retain nothing else, so the versions
-    // compaction superseded are actually removed rather than counted.
-    let mut config = int_config("min_versions_to_keep", 1);
-    config.extend(int_config("retain_hours", 0));
-    let cleanup = CleanupVersionsHandler::new(url).with_fallback(fallback());
-    run(&cleanup, cleanup::JOB_TYPE, &name, config, true).await;
-}
-
-/// Detects, finds the proposal for `name`, and executes it. The config carries
-/// every key both halves read; each ignores what it does not know.
-async fn run<H: JobHandler>(
-    handler: &H,
-    job_type: &str,
-    name: &str,
-    config: HashMap<String, ConfigValue>,
-    required: bool,
-) {
-    let recorder = Recorder::default();
-    handler
-        .detect(
-            &RunDetectionRequest {
-                request_id: format!("detect-{job_type}"),
-                job_type: job_type.to_string(),
-                worker_config_values: config.clone(),
-                ..Default::default()
-            },
-            &recorder,
-        )
-        .await
-        .unwrap_or_else(|err| panic!("{job_type} detection failed: {err}"));
-
-    let proposals = recorder.proposals.lock().unwrap().clone();
-    let Some(proposal) = proposals.iter().find(|p| p.summary.contains(name)) else {
-        assert!(
-            !required,
-            "{job_type} proposed nothing for {name}, out of {} proposals",
-            proposals.len()
-        );
-        eprintln!("{job_type}: nothing to do for {name}");
-        return;
-    };
-
-    handler
-        .execute(
-            &ExecuteJobRequest {
-                request_id: format!("execute-{job_type}"),
-                job: Some(JobSpec {
-                    job_id: format!("job-{job_type}"),
-                    job_type: job_type.to_string(),
-                    parameters: proposal.parameters.clone(),
-                    ..Default::default()
-                }),
-                worker_config_values: config,
-                ..Default::default()
-            },
-            &recorder,
-        )
-        .await
-        .unwrap_or_else(|err| panic!("{job_type} execution failed: {err}"));
-
-    let completed = recorder.completed.lock().unwrap().clone();
-    let result = completed
-        .last()
-        .unwrap_or_else(|| panic!("{job_type} reported no completion"));
-    assert!(
-        result.success,
-        "{job_type} reported failure: {}",
-        result.error_message
-    );
-    eprintln!(
-        "{job_type}: {}",
-        result
-            .result
-            .as_ref()
-            .map(|r| r.summary.clone())
-            .unwrap_or_default()
-    );
-}

telemetry/README.md

@@ -139,7 +139,7 @@ The telemetry server exposes these Prometheus metrics:
 ### Cluster Metrics
 - `seaweedfs_telemetry_total_clusters`: Total unique clusters (30 days)
 - `seaweedfs_telemetry_active_clusters`: Active clusters (7 days)
-- `seaweedfs_telemetry_confirmed_clusters`: Active clusters seen on 7+ distinct days — one-shot reports don't count, and the version/OS distributions in `/api/stats` are computed over these
+- `seaweedfs_telemetry_confirmed_clusters`: Active clusters seen on 2+ distinct days — one-shot reports don't count, and the version/OS distributions in `/api/stats` are computed over these
 
 ### Per-Cluster Metrics
 - `seaweedfs_telemetry_volume_servers{cluster_id}`: Volume servers per cluster

telemetry/server/api/handlers_test.go

@@ -2,7 +2,6 @@ package api
 
 import (
 	"bytes"
-	"encoding/json"
 	"net/http"
 	"net/http/httptest"
 	"testing"
@@ -12,9 +11,6 @@ import (
 	protobuf "google.golang.org/protobuf/proto"
 )
 
-// promauto registers on the global registry: one storage per test binary.
-var testHandler = NewHandler(storage.NewPrometheusStorage())
-
 func validReport() *proto.TelemetryData {
 	return &proto.TelemetryData{
 		TopologyId:        "38422678-6a0d-4482-aa33-65b90010ac47",
@@ -47,7 +43,8 @@ func marshalReport(t *testing.T, data *proto.TelemetryData) []byte {
 }
 
 func TestCollectTelemetryValidation(t *testing.T) {
-	h := testHandler
+	// promauto registers on the global registry: one storage per test binary.
+	h := NewHandler(storage.NewPrometheusStorage())
 
 	t.Run("valid report accepted", func(t *testing.T) {
 		if w := postCollect(t, h, marshalReport(t, validReport()), "application/x-protobuf"); w.Code != http.StatusOK {
@@ -99,42 +96,3 @@ func TestCollectTelemetryValidation(t *testing.T) {
 		}
 	})
 }
-
-// The dashboard looks confirmation windows up by the select's string value,
-// so the stats JSON must key confirmed_by_days by decimal strings and carry
-// unmet thresholds as zeros.
-func TestStatsSerializedThresholds(t *testing.T) {
-	data := validReport()
-	data.TopologyId = "49533789-7b1e-4593-bb44-76ca1121bd58"
-	if w := postCollect(t, testHandler, marshalReport(t, data), "application/x-protobuf"); w.Code != http.StatusOK {
-		t.Fatalf("collect: got %d: %s", w.Code, w.Body.String())
-	}
-
-	req := httptest.NewRequest(http.MethodGet, "/api/stats", nil)
-	w := httptest.NewRecorder()
-	testHandler.GetStats(w, req)
-	if w.Code != http.StatusOK {
-		t.Fatalf("stats: got %d", w.Code)
-	}
-
-	var stats struct {
-		ConfirmedByDays map[string]int `json:"confirmed_by_days"`
-	}
-	if err := json.Unmarshal(w.Body.Bytes(), &stats); err != nil {
-		t.Fatalf("decode: %v", err)
-	}
-	if len(stats.ConfirmedByDays) != 5 {
-		t.Fatalf("confirmed_by_days = %v, want the 5 thresholds", stats.ConfirmedByDays)
-	}
-	for _, key := range []string{"1", "3", "7", "14", "30"} {
-		if _, ok := stats.ConfirmedByDays[key]; !ok {
-			t.Errorf("confirmed_by_days missing %q: %v", key, stats.ConfirmedByDays)
-		}
-	}
-	if stats.ConfirmedByDays["1"] < 1 {
-		t.Errorf("fresh cluster missing from the 1-day count: %v", stats.ConfirmedByDays)
-	}
-	if stats.ConfirmedByDays["30"] != 0 {
-		t.Errorf("unmet threshold not zero: %v", stats.ConfirmedByDays)
-	}
-}

telemetry/server/dashboard/dashboard.go

@@ -57,15 +57,6 @@ func (h *Handler) ServeIndex(w http.ResponseWriter, r *http.Request) {
             color: #666;
             margin-top: 5px;
         }
-        .stat-label select {
-            border: none;
-            background: none;
-            color: inherit;
-            font: inherit;
-            padding: 0;
-            cursor: pointer;
-            text-decoration: underline dotted;
-        }
         .chart-container {
             background: white;
             padding: 20px;
@@ -138,14 +129,7 @@ func (h *Handler) ServeIndex(w http.ResponseWriter, r *http.Request) {
                 </div>
                 <div class="stat-card">
                     <div class="stat-value" id="confirmedInstances">-</div>
-                    <div class="stat-label">Confirmed Clusters
-                        (<select id="confirmDays" onchange="updateConfirmed()">
-                            <option value="1">1+</option>
-                            <option value="3">3+</option>
-                            <option value="7" selected>7+</option>
-                            <option value="14">14+</option>
-                            <option value="30">30+</option>
-                        </select> days)</div>
+                    <div class="stat-label">Confirmed Clusters (2+ days)</div>
                 </div>
                 <div class="stat-card">
                     <div class="stat-value" id="totalVersions">-</div>
@@ -245,26 +229,14 @@ func (h *Handler) ServeIndex(w http.ResponseWriter, r *http.Request) {
             }
         }
 
-        let latestStats = {};
-
         function updateStats(stats) {
-            latestStats = stats;
             document.getElementById('totalInstances').textContent = stats.total_instances || 0;
             document.getElementById('activeInstances').textContent = stats.active_instances || 0;
-            updateConfirmed();
+            document.getElementById('confirmedInstances').textContent = stats.confirmed_instances || 0;
             document.getElementById('totalVersions').textContent = Object.keys(stats.versions || {}).length;
             document.getElementById('totalOS').textContent = Object.keys(stats.os_distribution || {}).length;
         }
 
-        // Servers from before confirmed_by_days fall back to the fixed
-        // 7-day count.
-        function updateConfirmed() {
-            const days = document.getElementById('confirmDays').value;
-            const byDays = latestStats.confirmed_by_days || {};
-            const count = byDays[days] !== undefined ? byDays[days] : latestStats.confirmed_instances;
-            document.getElementById('confirmedInstances').textContent = count || 0;
-        }
-
         function updateCharts(stats) {
             createPieChart('versionChart', 'Version Distribution', stats.versions || {});
             createPieChart('osChart', 'Operating System Distribution', stats.os_distribution || {});

telemetry/server/go.mod

@@ -1,6 +1,6 @@
 module github.com/seaweedfs/seaweedfs/telemetry/server
 
-go 1.26
+go 1.25.8
 
 require (
 	github.com/prometheus/client_golang v1.24.1

telemetry/server/storage/confirmed_test.go

@@ -44,18 +44,13 @@ func TestConfirmedClusters(t *testing.T) {
 		t.Fatalf("fallback distribution missing active cluster: %v", v)
 	}
 
-	// Give cluster A samples from the six previous days: now seen on 7
-	// distinct days.
+	// Give cluster A a sample from yesterday: now seen on 2 distinct days.
 	s.mu.Lock()
 	id := "aaaaaaaa-0000-0000-0000-000000000001"
-	var older []HistorySample
-	for offset := -6; offset < 0; offset++ {
-		older = append(older, HistorySample{
-			Ts:             time.Now().AddDate(0, 0, offset).Unix(),
-			TotalDiskBytes: 50,
-		})
-	}
-	s.histories[id] = append(older, s.histories[id]...)
+	s.histories[id] = append([]HistorySample{{
+		Ts:             time.Now().AddDate(0, 0, -1).Unix(),
+		TotalDiskBytes: 50,
+	}}, s.histories[id]...)
 	s.mu.Unlock()
 
 	// A one-shot cluster B arrives (like an injected report): it counts as
@@ -65,7 +60,7 @@ func TestConfirmedClusters(t *testing.T) {
 	}
 	stats = statsOf(t, s)
 	if stats["active_instances"] != 2 || stats["confirmed_instances"] != 1 {
-		t.Fatalf("day seven: active=%v confirmed=%v, want 2/1", stats["active_instances"], stats["confirmed_instances"])
+		t.Fatalf("day two: active=%v confirmed=%v, want 2/1", stats["active_instances"], stats["confirmed_instances"])
 	}
 	v := stats["versions"].(map[string]int)
 	if v["4.40"] != 1 {
@@ -74,14 +69,4 @@ func TestConfirmedClusters(t *testing.T) {
 	if _, ok := v["9.99"]; ok {
 		t.Errorf("one-shot cluster polluted the distribution: %v", v)
 	}
-
-	// Cluster A meets the 1/3/7-day thresholds, B only the 1-day one, and
-	// unmet thresholds are present as zero so the dashboard can show them.
-	byDays := stats["confirmed_by_days"].(map[int]int)
-	want := map[int]int{1: 2, 3: 1, 7: 1, 14: 0, 30: 0}
-	for threshold, expected := range want {
-		if got, ok := byDays[threshold]; !ok || got != expected {
-			t.Errorf("confirmed_by_days[%d] = %v (present=%v), want %d", threshold, got, ok, expected)
-		}
-	}
 }

telemetry/server/storage/history.go

@@ -8,11 +8,7 @@ import (
 
 // confirmDays is how many distinct UTC days a cluster must have reported
 // on before it counts as confirmed in the aggregated stats.
-const confirmDays = 7
-
-// confirmThresholds are the confirmation windows the dashboard lets the
-// viewer pick between; confirmDays is the one everything else is built on.
-var confirmThresholds = []int{1, 3, 7, 14, 30}
+const confirmDays = 2
 
 // activeDays is how recently a cluster must have reported to count as active.
 const activeDays = 7
@@ -47,9 +43,9 @@ func (s *PrometheusStorage) appendHistory(data *proto.TelemetryData, receivedAt
 }
 
 // seriesHistories picks the clusters the fleet-wide series are built from: the
-// confirmed ones. A cluster that reported for less than a week is usually a CI
-// or test cluster, and those arrive faster than they age out, so counting them
-// makes every fleet total climb forever. Falls back to
+// confirmed ones. A cluster that only ever reported on one day is usually a CI
+// or test cluster that lived for a minute, and those arrive faster than they
+// age out, so counting them makes every fleet total climb forever. Falls back to
 // all clusters while none is confirmed yet, so a fresh server still draws its
 // charts. Callers must hold s.mu.
 func (s *PrometheusStorage) seriesHistories() map[string][]HistorySample {

telemetry/server/storage/metrics_test.go

@@ -15,8 +15,7 @@ func TestGetMetricsSumsEachDay(t *testing.T) {
 	seedSamples(s, "daily", HistorySample{TotalDiskBytes: 300, VolumeServerCount: 3},
 		-9, -8, -7, -6, -5, -4, -3, -2, -1, 0)
 	// Stopped reporting past the active window: counts on its own days only.
-	seedSamples(s, "gone", HistorySample{TotalDiskBytes: 900, VolumeServerCount: 9},
-		-14, -13, -12, -11, -10, -9, -8)
+	seedSamples(s, "gone", HistorySample{TotalDiskBytes: 900, VolumeServerCount: 9}, -9, -8)
 
 	metrics, err := s.GetMetrics(10)
 	if err != nil {
@@ -41,8 +40,7 @@ func TestGetMetricsSumsEachDay(t *testing.T) {
 func TestGetMetricsCarriesSkippedDaysForward(t *testing.T) {
 	s := newPrometheusStorage(prometheus.NewRegistry())
 
-	seedSamples(s, "gappy", HistorySample{TotalDiskBytes: 500, VolumeServerCount: 5},
-		-8, -7, -6, -5, -4, -3, -1)
+	seedSamples(s, "gappy", HistorySample{TotalDiskBytes: 500, VolumeServerCount: 5}, -3, -1)
 
 	metrics, err := s.GetMetrics(4)
 	if err != nil {
@@ -58,23 +56,21 @@ func TestGetMetricsCarriesSkippedDaysForward(t *testing.T) {
 // out of nothing on its first day of data.
 func TestGetMetricsWindowStartsAtOldestSample(t *testing.T) {
 	s := newPrometheusStorage(prometheus.NewRegistry())
-	seedSamples(s, "recent", HistorySample{TotalDiskBytes: 100, VolumeServerCount: 1},
-		-6, -5, -4, -3, -2, -1, 0)
+	seedSamples(s, "recent", HistorySample{TotalDiskBytes: 100, VolumeServerCount: 1}, -2, -1, 0)
 
 	metrics, err := s.GetMetrics(30)
 	if err != nil {
 		t.Fatal(err)
 	}
-	if got := metrics["dates"].([]string); len(got) != 7 {
-		t.Errorf("dates = %v, want the 7 days with history, not 30", got)
+	if got := metrics["dates"].([]string); len(got) != 3 {
+		t.Errorf("dates = %v, want the 3 days with history, not 30", got)
 	}
-	if got := metrics["disk_usage"].([]uint64); !equal(got, []uint64{100, 100, 100, 100, 100, 100, 100}) {
+	if got := metrics["disk_usage"].([]uint64); !equal(got, []uint64{100, 100, 100}) {
 		t.Errorf("disk_usage = %v, want no leading zero days", got)
 	}
 
 	// History reaching past the requested window still clips to the window.
-	seedSamples(s, "old", HistorySample{TotalDiskBytes: 50, VolumeServerCount: 1},
-		-45, -44, -43, -42, -41, -40, -39)
+	seedSamples(s, "old", HistorySample{TotalDiskBytes: 50, VolumeServerCount: 1}, -40, -39)
 	metrics, err = s.GetMetrics(10)
 	if err != nil {
 		t.Fatal(err)
@@ -90,10 +86,8 @@ func TestGetMetricsAgreesWithClusterSizes(t *testing.T) {
 	s := newPrometheusStorage(prometheus.NewRegistry())
 	seedSamples(s, "daily", HistorySample{TotalDiskBytes: 300, VolumeServerCount: 3},
 		-9, -8, -7, -6, -5, -4, -3, -2, -1, 0)
-	seedSamples(s, "lagging", HistorySample{TotalDiskBytes: 200, VolumeServerCount: 2},
-		-8, -7, -6, -5, -4, -3, -2)
-	seedSamples(s, "gone", HistorySample{TotalDiskBytes: 900, VolumeServerCount: 9},
-		-14, -13, -12, -11, -10, -9, -8)
+	seedSamples(s, "lagging", HistorySample{TotalDiskBytes: 200, VolumeServerCount: 2}, -3, -2)
+	seedSamples(s, "gone", HistorySample{TotalDiskBytes: 900, VolumeServerCount: 9}, -9, -8)
 
 	metrics, err := s.GetMetrics(10)
 	if err != nil {
@@ -117,8 +111,7 @@ func TestGetMetricsAgreesWithClusterSizes(t *testing.T) {
 func TestGetMetricsExcludesUnconfirmedClusters(t *testing.T) {
 	s := newPrometheusStorage(prometheus.NewRegistry())
 
-	seedSamples(s, "real", HistorySample{TotalDiskBytes: 300, VolumeServerCount: 3},
-		-6, -5, -4, -3, -2, -1, 0)
+	seedSamples(s, "real", HistorySample{TotalDiskBytes: 300, VolumeServerCount: 3}, -3, -2, -1, 0)
 	for _, id := range []string{"ci-1", "ci-2", "ci-3"} {
 		seedSamples(s, id, HistorySample{TotalDiskBytes: 5, VolumeServerCount: 14}, -1)
 	}
@@ -135,7 +128,7 @@ func TestGetMetricsExcludesUnconfirmedClusters(t *testing.T) {
 	}
 }
 
-// Until any cluster has a week of history the charts fall back to every
+// Until any cluster has two days of history the charts fall back to every
 // cluster, so a fresh server doesn't serve empty series.
 func TestGetMetricsFallsBackWhenNoneConfirmed(t *testing.T) {
 	s := newPrometheusStorage(prometheus.NewRegistry())

telemetry/server/storage/prometheus.go

@@ -53,7 +53,7 @@ func newPrometheusStorage(reg prometheus.Registerer) *PrometheusStorage {
 		}),
 		confirmedClusters: promauto.NewGauge(prometheus.GaugeOpts{
 			Name: "seaweedfs_telemetry_confirmed_clusters",
-			Help: "Active clusters seen on at least 7 distinct days (last 7 days)",
+			Help: "Active clusters seen on at least 2 distinct days (last 7 days)",
 		}),
 		volumeServerCount: promauto.NewGaugeVec(prometheus.GaugeOpts{
 			Name: "seaweedfs_telemetry_volume_servers",
@@ -207,10 +207,6 @@ func (s *PrometheusStorage) updateStats() {
 	totalInstances := 0
 	activeInstances := 0
 	confirmedInstances := 0
-	confirmedByDays := make(map[int]int, len(confirmThresholds))
-	for _, threshold := range confirmThresholds {
-		confirmedByDays[threshold] = 0
-	}
 	versionsAll := make(map[string]int)
 	osAll := make(map[string]int)
 	versionsConfirmed := make(map[string]int)
@@ -224,16 +220,10 @@ func (s *PrometheusStorage) updateStats() {
 			activeInstances++
 			versionsAll[instance.TelemetryData.Version]++
 			osAll[instance.TelemetryData.Os]++
-			// A cluster is confirmed once seen on confirmDays distinct UTC
-			// days (histories hold one sample per day), so short-lived
-			// clusters can't skew the distributions below.
-			daysSeen := len(s.histories[instance.TelemetryData.TopologyId])
-			for _, threshold := range confirmThresholds {
-				if daysSeen >= threshold {
-					confirmedByDays[threshold]++
-				}
-			}
-			if daysSeen >= confirmDays {
+			// A cluster is confirmed once seen on >=2 distinct UTC days
+			// (histories hold one sample per day), so one-shot reports
+			// can't skew the distributions below.
+			if len(s.histories[instance.TelemetryData.TopologyId]) >= confirmDays {
 				confirmedInstances++
 				versionsConfirmed[instance.TelemetryData.Version]++
 				osConfirmed[instance.TelemetryData.Os]++
@@ -241,7 +231,7 @@ func (s *PrometheusStorage) updateStats() {
 		}
 	}
 
-	// Before any cluster has a week of history (fresh server with no
+	// Before any cluster has two days of history (fresh server with no
 	// prior state), fall back to all active clusters so the dashboard
 	// distributions aren't empty.
 	versions, osDistribution := versionsConfirmed, osConfirmed
@@ -259,7 +249,6 @@ func (s *PrometheusStorage) updateStats() {
 		"total_instances":     totalInstances,
 		"active_instances":    activeInstances,
 		"confirmed_instances": confirmedInstances,
-		"confirmed_by_days":   confirmedByDays,
 		"versions":            versions,
 		"os_distribution":     osDistribution,
 	}

telemetry/server/storage/sizes_test.go

@@ -25,13 +25,11 @@ func TestClusterSizeSeries(t *testing.T) {
 	// Reported every day of the window.
 	seedSamples(s, "daily", HistorySample{TotalDiskBytes: 300, VolumeServerCount: 3},
 		-9, -8, -7, -6, -5, -4, -3, -2, -1, 0)
-	// Stopped reporting two days ago: still active, so its size is held to
-	// the right edge instead of dropping out of the stack.
-	seedSamples(s, "lagging", HistorySample{TotalDiskBytes: 200, VolumeServerCount: 2},
-		-8, -7, -6, -5, -4, -3, -2)
+	// Reported two days ago and not since: still active, so its size is held
+	// to the right edge instead of dropping out of the stack.
+	seedSamples(s, "lagging", HistorySample{TotalDiskBytes: 200, VolumeServerCount: 2}, -3, -2)
 	// Stopped reporting past the active window: its own days only.
-	seedSamples(s, "gone", HistorySample{TotalDiskBytes: 900, VolumeServerCount: 9},
-		-14, -13, -12, -11, -10, -9, -8)
+	seedSamples(s, "gone", HistorySample{TotalDiskBytes: 900, VolumeServerCount: 9}, -9, -8)
 	// One day of history only: unconfirmed, so it stays out of the stack.
 	seedSamples(s, "oneshot", HistorySample{TotalDiskBytes: 400, VolumeServerCount: 4}, -1)
 
@@ -50,7 +48,7 @@ func TestClusterSizeSeries(t *testing.T) {
 	if got := byId["daily"].Disk; !equal(got, []uint64{300, 300, 300, 300, 300, 300, 300, 300, 300, 300}) {
 		t.Errorf("daily = %v, want 300 every day", got)
 	}
-	if got := byId["lagging"].Disk; !equal(got, []uint64{0, 200, 200, 200, 200, 200, 200, 200, 200, 200}) {
+	if got := byId["lagging"].Disk; !equal(got, []uint64{0, 0, 0, 0, 0, 0, 200, 200, 200, 200}) {
 		t.Errorf("lagging = %v, want its size carried to the right edge", got)
 	}
 	if got := byId["gone"].Disk; !equal(got, []uint64{900, 900, 0, 0, 0, 0, 0, 0, 0, 0}) {
@@ -64,7 +62,7 @@ func TestClusterSizeSeries(t *testing.T) {
 	if got := byId["daily"].Servers; !equal(got, []uint64{3, 3, 3, 3, 3, 3, 3, 3, 3, 3}) {
 		t.Errorf("daily servers = %v, want 3 every day", got)
 	}
-	if got := byId["lagging"].Servers; !equal(got, []uint64{0, 2, 2, 2, 2, 2, 2, 2, 2, 2}) {
+	if got := byId["lagging"].Servers; !equal(got, []uint64{0, 0, 0, 0, 0, 0, 2, 2, 2, 2}) {
 		t.Errorf("lagging servers = %v, want carried to the right edge", got)
 	}
 	if got := byId["gone"].Servers; !equal(got, []uint64{9, 9, 0, 0, 0, 0, 0, 0, 0, 0}) {
@@ -91,10 +89,10 @@ func TestClusterSizeSeries(t *testing.T) {
 	if series.Other == nil || series.Other.Count != 2 {
 		t.Fatalf("other = %+v, want 2 clusters", series.Other)
 	}
-	if !equal(series.Other.Disk, []uint64{900, 1100, 200, 200, 200, 200, 200, 200, 200, 200}) {
+	if !equal(series.Other.Disk, []uint64{900, 900, 0, 0, 0, 0, 200, 200, 200, 200}) {
 		t.Errorf("other = %v, want lagging+gone summed per day", series.Other.Disk)
 	}
-	if !equal(series.Other.Servers, []uint64{9, 11, 2, 2, 2, 2, 2, 2, 2, 2}) {
+	if !equal(series.Other.Servers, []uint64{9, 9, 0, 0, 0, 0, 2, 2, 2, 2}) {
 		t.Errorf("other servers = %v, want lagging+gone summed per day", series.Other.Servers)
 	}
 	if series.ClusterCount != 3 || series.TotalDisk != 500 || series.TotalServers != 5 {

telemetry/server/storage/versions_test.go

@@ -10,31 +10,31 @@ func TestVersionSeries(t *testing.T) {
 	s := newPrometheusStorage(prometheus.NewRegistry())
 
 	// Upgraded mid-window: its band leaves the old version for the new one.
-	seedSamples(s, "upgraded", HistorySample{Version: "4.39"}, -6, -5, -4, -3)
+	seedSamples(s, "upgraded", HistorySample{Version: "4.39"}, -4, -3)
 	seedSamples(s, "upgraded", HistorySample{Version: "4.40"}, -2, -1, 0)
 	// Reported every day on the same version.
-	seedSamples(s, "steady", HistorySample{Version: "4.40"}, -6, -5, -4, -3, -2, -1, 0)
+	seedSamples(s, "steady", HistorySample{Version: "4.40"}, -4, -3, -2, -1, 0)
 	// One day of history only: unconfirmed, so it stays out of the stack.
 	seedSamples(s, "oneshot", HistorySample{Version: "4.40"}, -1)
 	// Confirmed, but its samples predate versions being recorded.
-	seedSamples(s, "versionless", HistorySample{}, -13, -12, -11, -10, -9, -8, -7)
+	seedSamples(s, "versionless", HistorySample{}, -9, -8)
 
 	series := s.GetVersionSeries(10, 0)
 
 	// The versionless days are dropped before the axis is built, so the chart
 	// spans the days a version is known for instead of climbing out of blanks.
-	if len(series.Dates) != 7 {
-		t.Fatalf("dates = %v, want the 7 days with versions", series.Dates)
+	if len(series.Dates) != 5 {
+		t.Fatalf("dates = %v, want the 5 days with versions", series.Dates)
 	}
 	if len(series.Versions) != 2 ||
 		series.Versions[0].Version != "4.39" || series.Versions[1].Version != "4.40" {
 		t.Fatalf("versions = %+v, want 4.39 then 4.40", series.Versions)
 	}
-	if got := series.Versions[0].Clusters; !equal(got, []uint64{1, 1, 1, 1, 0, 0, 0}) {
-		t.Errorf("4.39 = %v, want the upgraded cluster's first four days", got)
+	if got := series.Versions[0].Clusters; !equal(got, []uint64{1, 1, 0, 0, 0}) {
+		t.Errorf("4.39 = %v, want the upgraded cluster's first two days", got)
 	}
-	if got := series.Versions[1].Clusters; !equal(got, []uint64{1, 1, 1, 1, 2, 2, 2}) {
-		t.Errorf("4.40 = %v, want steady plus upgraded from day 5", got)
+	if got := series.Versions[1].Clusters; !equal(got, []uint64{1, 1, 2, 2, 2}) {
+		t.Errorf("4.40 = %v, want steady plus upgraded from day 3", got)
 	}
 	if series.TotalClusters != 2 {
 		t.Errorf("total_clusters = %d, want 2", series.TotalClusters)
@@ -48,12 +48,12 @@ func TestVersionSeriesHoldsForwardAndLimits(t *testing.T) {
 	s := newPrometheusStorage(prometheus.NewRegistry())
 
 	// Stopped reporting past the active window: its own days only.
-	seedSamples(s, "gone", HistorySample{Version: "3.97"}, -14, -13, -12, -11, -10, -9, -8)
+	seedSamples(s, "gone", HistorySample{Version: "3.97"}, -9, -8)
 	// Reported every day of the window.
 	seedSamples(s, "daily", HistorySample{Version: "4.40"}, -9, -8, -7, -6, -5, -4, -3, -2, -1, 0)
-	// Stopped reporting two days ago: still active, so it holds its version
-	// to the right edge instead of dropping out of the stack.
-	seedSamples(s, "lagging", HistorySample{Version: "4.30"}, -8, -7, -6, -5, -4, -3, -2)
+	// Reported three days ago and not since: still active, so it holds its
+	// version to the right edge instead of dropping out of the stack.
+	seedSamples(s, "lagging", HistorySample{Version: "4.30"}, -3, -2)
 
 	series := s.GetVersionSeries(10, 0)
 	if len(series.Dates) != 10 {
@@ -66,7 +66,7 @@ func TestVersionSeriesHoldsForwardAndLimits(t *testing.T) {
 	if got := byVersion["3.97"]; !equal(got, []uint64{1, 1, 0, 0, 0, 0, 0, 0, 0, 0}) {
 		t.Errorf("3.97 = %v, want nothing after its last report", got)
 	}
-	if got := byVersion["4.30"]; !equal(got, []uint64{0, 1, 1, 1, 1, 1, 1, 1, 1, 1}) {
+	if got := byVersion["4.30"]; !equal(got, []uint64{0, 0, 0, 0, 0, 0, 1, 1, 1, 1}) {
 		t.Errorf("4.30 = %v, want carried to the right edge", got)
 	}
 	if got := byVersion["4.40"]; !equal(got, []uint64{1, 1, 1, 1, 1, 1, 1, 1, 1, 1}) {
@@ -85,7 +85,7 @@ func TestVersionSeriesHoldsForwardAndLimits(t *testing.T) {
 	if series.Other == nil || series.Other.Count != 2 {
 		t.Fatalf("other = %+v, want 2 versions", series.Other)
 	}
-	if !equal(series.Other.Clusters, []uint64{1, 2, 1, 1, 1, 1, 1, 1, 1, 1}) {
+	if !equal(series.Other.Clusters, []uint64{1, 1, 0, 0, 0, 0, 1, 1, 1, 1}) {
 		t.Errorf("other = %v, want 3.97+4.30 summed per day", series.Other.Clusters)
 	}
 	if series.TotalClusters != 2 {

test/erasure_coding/chaos_lifecycle_test.go

@@ -933,7 +933,7 @@ func copyFileContents(src, dst string) error {
 // chaosDataNodes lists the data nodes from a fresh master topology snapshot.
 func chaosDataNodes(commandEnv *shell.CommandEnv) []*master_pb.DataNodeInfo {
 	var resp *master_pb.VolumeListResponse
-	err := commandEnv.MasterClient.WithClient(context.Background(), false, func(client master_pb.SeaweedClient) error {
+	err := commandEnv.MasterClient.WithClient(false, func(client master_pb.SeaweedClient) error {
 		var e error
 		resp, e = client.VolumeList(context.Background(), &master_pb.VolumeListRequest{})
 		return e
@@ -955,7 +955,7 @@ func chaosDataNodes(commandEnv *shell.CommandEnv) []*master_pb.DataNodeInfo {
 func masterEcGenerations(commandEnv *shell.CommandEnv, volumeId uint32) map[int64]bool {
 	generations := map[int64]bool{}
 	var resp *master_pb.VolumeListResponse
-	err := commandEnv.MasterClient.WithClient(context.Background(), false, func(client master_pb.SeaweedClient) error {
+	err := commandEnv.MasterClient.WithClient(false, func(client master_pb.SeaweedClient) error {
 		var e error
 		resp, e = client.VolumeList(context.Background(), &master_pb.VolumeListRequest{})
 		return e

test/erasure_coding/multidisk_shardloss_test.go

@@ -200,7 +200,7 @@ func disksWithShards(testDir string, volumeId uint32) int {
 func nodeVolumeDiskCounts(t *testing.T, commandEnv *shell.CommandEnv) map[string]int {
 	t.Helper()
 	var resp *master_pb.VolumeListResponse
-	err := commandEnv.MasterClient.WithClient(context.Background(), false, func(client master_pb.SeaweedClient) error {
+	err := commandEnv.MasterClient.WithClient(false, func(client master_pb.SeaweedClient) error {
 		var e error
 		resp, e = client.VolumeList(context.Background(), &master_pb.VolumeListRequest{})
 		return e

test/erasure_coding/multidisk_shell_lifecycle_test.go

@@ -370,7 +370,7 @@ func removeTwoShardFiles(t *testing.T, testDir string, volumeId uint32) []int {
 func masterEcShardIds(commandEnv *shell.CommandEnv, volumeId uint32) map[int]bool {
 	ids := map[int]bool{}
 	var resp *master_pb.VolumeListResponse
-	err := commandEnv.MasterClient.WithClient(context.Background(), false, func(client master_pb.SeaweedClient) error {
+	err := commandEnv.MasterClient.WithClient(false, func(client master_pb.SeaweedClient) error {
 		var e error
 		resp, e = client.VolumeList(context.Background(), &master_pb.VolumeListRequest{})
 		return e

test/fuse_dlm/framework_test.go

@@ -8,7 +8,6 @@ import (
 	"os/exec"
 	"path/filepath"
 	"strconv"
-	"strings"
 	"sync"
 	"syscall"
 	"testing"
@@ -91,8 +90,6 @@ func startDLMTestCluster(t testing.TB) *dlmTestCluster {
 	require.NoError(t, c.startVolume(configDir))
 	require.NoError(t, c.waitForTCP(fmt.Sprintf("127.0.0.1:%d", c.volumePort), 30*time.Second),
 		"volume not ready\n%s", c.tailLog("volume"))
-	require.NoError(t, c.waitForVolumeRegistered(30*time.Second),
-		"volume server registration\n%s", c.tailLog("master"))
 
 	// Start 2 filers
 	for i := 0; i < 2; i++ {
@@ -264,10 +261,7 @@ func (c *dlmTestCluster) tailLog(name string) string {
 }
 
 func (c *dlmTestCluster) copyLogsForCI() {
-	// One subdirectory per test: a flat layout lets every teardown overwrite
-	// the previous test's logs, so the CI artifact only ever shows the last
-	// cluster, never the failing one.
-	ciLogDir := filepath.Join("/tmp/seaweedfs-fuse-dlm-logs", strings.ReplaceAll(c.t.Name(), "/", "_"))
+	ciLogDir := "/tmp/seaweedfs-fuse-dlm-logs"
 	os.MkdirAll(ciLogDir, 0755)
 	logsDir := filepath.Join(c.baseDir, "logs")
 	entries, err := os.ReadDir(logsDir)
@@ -354,44 +348,6 @@ func (c *dlmTestCluster) waitForFilerCount(expected int, timeout time.Duration)
 	return fmt.Errorf("timed out waiting for %d filers in group %q", expected, filerGroup)
 }
 
-// waitForVolumeRegistered waits until the volume server shows up in the master
-// topology with its slots reported. An open volume port only means the process
-// is listening: until the master has elected itself and accepted a heartbeat,
-// an assign fails and the first write on a fresh mount surfaces it as ENOSPC.
-func (c *dlmTestCluster) waitForVolumeRegistered(timeout time.Duration) error {
-	addr := fmt.Sprintf("127.0.0.1:%d", c.masterGrpcPort)
-	conn, err := grpc.NewClient(addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
-	if err != nil {
-		return err
-	}
-	defer conn.Close()
-
-	client := master_pb.NewSeaweedClient(conn)
-	deadline := time.Now().Add(timeout)
-	for time.Now().Before(deadline) {
-		ctx, cancel := context.WithTimeout(context.Background(), time.Second)
-		resp, err := client.VolumeList(ctx, &master_pb.VolumeListRequest{})
-		cancel()
-		if err == nil && resp.TopologyInfo != nil {
-			var slots int64
-			for _, dc := range resp.TopologyInfo.DataCenterInfos {
-				for _, rack := range dc.RackInfos {
-					for _, dn := range rack.DataNodeInfos {
-						for _, disk := range dn.DiskInfos {
-							slots += disk.MaxVolumeCount
-						}
-					}
-				}
-			}
-			if slots > 0 {
-				return nil
-			}
-		}
-		time.Sleep(200 * time.Millisecond)
-	}
-	return fmt.Errorf("volume server not registered with the master within %v", timeout)
-}
-
 // waitForLockRingConverged verifies that both filers have a consistent view of
 // the lock ring by acquiring the same lock through each filer and checking
 // mutual exclusion. Adapted from test/s3/distributed_lock/.

test/fuse_failover/framework_test.go

@@ -109,7 +109,6 @@ func startFailoverCluster(t testing.TB, numVolumes, numMounts int) *failoverClus
 		require.NoError(t, c.waitForMount(mp, 30*time.Second),
 			"mount %d not ready\n%s", i, c.tailLog(fmt.Sprintf("mount%d", i)))
 	}
-	require.NoError(t, c.WaitForVolumeServers(60*time.Second))
 	return c
 }
 
@@ -345,62 +344,6 @@ func (c *failoverCluster) WaitForHolders(vid uint32, count int, timeout time.Dur
 	}
 }
 
-// WaitForVolumeServers blocks until the master's topology lists every volume
-// server. A volume server accepts connections well before the master knows it
-// exists: a lone master only elects itself once its bootstrap check expires,
-// and heartbeats sent before that are refused and retried with backoff, so
-// registration lands seconds after the process is up. Until it does the
-// topology has no data node at all, an assign fails with "no free volumes
-// left", and the mount reports that as ENOSPC instead of retrying.
-func (c *failoverCluster) WaitForVolumeServers(timeout time.Duration) error {
-	deadline := time.Now().Add(timeout)
-	for {
-		registered := c.registeredVolumeServers()
-		var missing []string
-		for i := range c.volumePorts {
-			if address := c.VolumeServerAddress(i); !registered[address] {
-				missing = append(missing, address)
-			}
-		}
-		if len(missing) == 0 {
-			return nil
-		}
-		if time.Now().After(deadline) {
-			return fmt.Errorf("volume servers %v not registered with the master within %v\n%s",
-				missing, timeout, c.MasterGet("/dir/status?pretty=y"))
-		}
-		time.Sleep(200 * time.Millisecond)
-	}
-}
-
-// registeredVolumeServers is the set of volume server addresses the master
-// currently holds in its topology.
-func (c *failoverCluster) registeredVolumeServers() map[string]bool {
-	var status struct {
-		Topology struct {
-			DataCenters []struct {
-				Racks []struct {
-					DataNodes []struct {
-						Url string `json:"Url"`
-					} `json:"DataNodes"`
-				} `json:"Racks"`
-			} `json:"DataCenters"`
-		} `json:"Topology"`
-	}
-	if err := json.Unmarshal([]byte(c.MasterGet("/dir/status")), &status); err != nil {
-		return nil
-	}
-	registered := make(map[string]bool)
-	for _, dc := range status.Topology.DataCenters {
-		for _, rack := range dc.Racks {
-			for _, dn := range rack.DataNodes {
-				registered[dn.Url] = true
-			}
-		}
-	}
-	return registered
-}
-
 // volumeIndexOf maps a server address back to its index, or -1.
 func (c *failoverCluster) volumeIndexOf(address string) int {
 	for i := range c.volumePorts {

test/fuse_failover/volume_failover_test.go

@@ -190,9 +190,7 @@ func TestReadWithVolumeServerDown(t *testing.T) {
 		t.Logf("read %s (held by volume%d) with volume%d down in %v", name, victim, victim, elapsed)
 
 		require.NoError(t, c.StartVolume(victim))
-		// The next iteration picks its victim from the master's placement view,
-		// so wait for the restarted server to be in it again.
-		require.NoError(t, c.WaitForVolumeServers(60*time.Second))
+		time.Sleep(3 * time.Second) // let the master see the heartbeat again
 	}
 }
 

test/fuse_p2p/framework_test.go

@@ -3,14 +3,12 @@
 package fuse_p2p
 
 import (
-	"context"
 	"fmt"
 	"net"
 	"os"
 	"os/exec"
 	"path/filepath"
 	"strconv"
-	"strings"
 	"sync"
 	"syscall"
 	"testing"
@@ -18,10 +16,7 @@ import (
 
 	"github.com/seaweedfs/seaweedfs/test/testutil"
 	"github.com/seaweedfs/seaweedfs/weed/pb"
-	"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
 	"github.com/stretchr/testify/require"
-	"google.golang.org/grpc"
-	"google.golang.org/grpc/credentials/insecure"
 )
 
 // p2pTestCluster manages a minimal SeaweedFS cluster exercising the peer
@@ -105,8 +100,6 @@ func startP2PTestCluster(t testing.TB, numMounts int) *p2pTestCluster {
 	require.NoError(t, c.startVolume(configDir))
 	require.NoError(t, c.waitForTCP(c.volumeCmd, "volume",
 		fmt.Sprintf("127.0.0.1:%d", c.volumePort), 30*time.Second))
-	require.NoError(t, c.waitForVolumeRegistered(30*time.Second),
-		"volume server registration\n%s", c.tailLog("master"))
 
 	require.NoError(t, c.startFiler(configDir))
 	require.NoError(t, c.waitForTCP(c.filerCmd, "filer",
@@ -279,10 +272,7 @@ func (c *p2pTestCluster) tailLogFull(name string) string {
 }
 
 func (c *p2pTestCluster) copyLogsForCI() {
-	// One subdirectory per test: a flat layout lets every teardown overwrite
-	// the previous test's logs, so the CI artifact only ever shows the last
-	// cluster, never the failing one.
-	ciLogDir := filepath.Join("/tmp/seaweedfs-fuse-p2p-logs", strings.ReplaceAll(c.t.Name(), "/", "_"))
+	ciLogDir := "/tmp/seaweedfs-fuse-p2p-logs"
 	os.MkdirAll(ciLogDir, 0755)
 	logsDir := filepath.Join(c.baseDir, "logs")
 	entries, err := os.ReadDir(logsDir)
@@ -298,44 +288,6 @@ func (c *p2pTestCluster) copyLogsForCI() {
 	}
 }
 
-// waitForVolumeRegistered waits until the volume server shows up in the master
-// topology with its slots reported. An open volume port only means the process
-// is listening: until the master has elected itself and accepted a heartbeat,
-// an assign fails and the first write on a fresh mount surfaces it as ENOSPC.
-func (c *p2pTestCluster) waitForVolumeRegistered(timeout time.Duration) error {
-	addr := fmt.Sprintf("127.0.0.1:%d", c.masterGrpcPort)
-	conn, err := grpc.NewClient(addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
-	if err != nil {
-		return err
-	}
-	defer conn.Close()
-
-	client := master_pb.NewSeaweedClient(conn)
-	deadline := time.Now().Add(timeout)
-	for time.Now().Before(deadline) {
-		ctx, cancel := context.WithTimeout(context.Background(), time.Second)
-		resp, err := client.VolumeList(ctx, &master_pb.VolumeListRequest{})
-		cancel()
-		if err == nil && resp.TopologyInfo != nil {
-			var slots int64
-			for _, dc := range resp.TopologyInfo.DataCenterInfos {
-				for _, rack := range dc.RackInfos {
-					for _, dn := range rack.DataNodeInfos {
-						for _, disk := range dn.DiskInfos {
-							slots += disk.MaxVolumeCount
-						}
-					}
-				}
-			}
-			if slots > 0 {
-				return nil
-			}
-		}
-		time.Sleep(200 * time.Millisecond)
-	}
-	return fmt.Errorf("volume server not registered with the master within %v", timeout)
-}
-
 // waitForTCP polls addr until it accepts a connection, OR the supplied
 // subprocess exits — whichever comes first. Short-circuiting on child
 // exit turns a 30 s-spin-on-dead-process into an immediate failure with

test/multi_master/chaos_test.go

@@ -1,186 +0,0 @@
-package multi_master
-
-import (
-	"fmt"
-	"math/rand/v2"
-	"os"
-	"strconv"
-	"testing"
-	"time"
-)
-
-// chaosRounds is one stop or start per round, enough to walk in and out of
-// quorum several times without turning this into a soak test.
-const chaosRounds = 12
-
-// TestRandomStartStopElection bounces masters at random and holds the election
-// to the two things it must never get wrong: two masters claiming leadership at
-// once, and a quorum that comes back without agreeing on one. The cluster's
-// identity has to survive the whole walk — a master that re-mints a TopologyId
-// here is the split brain SetTopologyId kills its peers over.
-func TestRandomStartStopElection(t *testing.T) {
-	for _, impl := range raftImplementations {
-		t.Run(impl.name, func(t *testing.T) {
-			seed := chaosSeed(t)
-			t.Logf("seed %d — replay this walk with MULTI_MASTER_IT_SEED=%d", seed, seed)
-			rng := rand.New(rand.NewPCG(seed, seed))
-
-			mc := NewMasterCluster(t, impl.raftHashicorp)
-			for i := range 3 {
-				mc.StartNode(i)
-			}
-			if err := mc.WaitForLeader(waitTimeout); err != nil {
-				mc.DumpLogs()
-				t.Fatalf("cluster did not elect a leader: %v", err)
-			}
-			topologyId, err := mc.WaitForTopologyId(waitTimeout)
-			if err != nil {
-				mc.DumpLogs()
-				t.Fatalf("no initial TopologyId: %v", err)
-			}
-
-			for round := 1; round <= chaosRounds; round++ {
-				target := rng.IntN(3)
-				if mc.IsNodeRunning(target) {
-					mc.StopNode(target)
-					t.Logf("round %d: stopped master %d, %d left running", round, target, runningMasters(mc))
-				} else {
-					mc.StartNode(target)
-					t.Logf("round %d: started master %d, %d now running", round, target, runningMasters(mc))
-				}
-
-				if err := waitForSettledElection(mc, leaderElectionTimeout); err != nil {
-					mc.DumpLogs()
-					t.Fatalf("round %d (seed %d): %v", round, seed, err)
-				}
-
-				// /dir/status proxies to the leader, so this reads the value the
-				// cluster as a whole is carrying, not any one master's copy.
-				if runningMasters(mc) >= 2 {
-					id, err := mc.WaitForTopologyId(leaderElectionTimeout)
-					if err != nil {
-						mc.DumpLogs()
-						t.Fatalf("round %d (seed %d): %v", round, seed, err)
-					}
-					if id != topologyId {
-						mc.DumpLogs()
-						t.Fatalf("round %d (seed %d): TopologyId changed from %s to %s", round, seed, topologyId, id)
-					}
-				}
-			}
-
-			// Everything back up, so the walk ends on a full cluster.
-			for i := range 3 {
-				mc.StartNode(i)
-			}
-			if _, err := waitForCommonLeader(mc, leaderElectionTimeout); err != nil {
-				mc.DumpLogs()
-				t.Fatalf("cluster did not recover after the walk (seed %d): %v", seed, err)
-			}
-			for i := range 3 {
-				if err := waitForPeerCount(mc, i, 2, leaderElectionTimeout); err != nil {
-					mc.DumpLogs()
-					t.Fatalf("master %d does not see the full cluster (seed %d): %v", i, seed, err)
-				}
-			}
-			id, err := mc.WaitForTopologyId(leaderElectionTimeout)
-			if err != nil {
-				mc.DumpLogs()
-				t.Fatalf("no TopologyId after the walk (seed %d): %v", seed, err)
-			}
-			if id != topologyId {
-				mc.DumpLogs()
-				t.Fatalf("TopologyId changed from %s to %s over the walk (seed %d)", topologyId, id, seed)
-			}
-		})
-	}
-}
-
-// waitForSettledElection waits for a quorum to agree on one leader, and fails
-// if two masters claim leadership across consecutive polls — anything shorter
-// than that is a master on its way down.
-//
-// Below a quorum there is nothing to wait for: the walk moves on. A master that
-// has lost its quorum can keep claiming leadership for tens of seconds under
-// goraft, and it cannot commit anything in that window, so stepping down is a
-// liveness question rather than a safety one. TestTwoMastersDownAndRestart
-// holds that direction to account.
-func waitForSettledElection(mc *MasterCluster, timeout time.Duration) error {
-	if runningMasters(mc) < 2 {
-		return nil
-	}
-
-	var lastErr error
-	splitPolls := 0
-	deadline := time.Now().Add(timeout)
-	for time.Now().Before(deadline) {
-		claims, err := leaderClaims(mc)
-		if err != nil {
-			lastErr = err
-		}
-		if len(claims) > 1 {
-			splitPolls++
-			if splitPolls > 1 {
-				return fmt.Errorf("masters %v all claim leadership", claims)
-			}
-			time.Sleep(waitTick)
-			continue
-		}
-		splitPolls = 0
-
-		if _, err := commonLeader(mc); err == nil {
-			return nil
-		} else {
-			lastErr = err
-		}
-		time.Sleep(waitTick)
-	}
-	if lastErr == nil {
-		lastErr = fmt.Errorf("cluster did not settle within %v", timeout)
-	}
-	return lastErr
-}
-
-// leaderClaims returns the running masters that call themselves leader. The
-// error reports masters that did not answer at all, which is a reason to keep
-// waiting rather than a verdict.
-func leaderClaims(mc *MasterCluster) (claims []int, err error) {
-	for i := range 3 {
-		if !mc.IsNodeRunning(i) {
-			continue
-		}
-		cs, statusErr := mc.GetClusterStatus(i)
-		if statusErr != nil {
-			err = fmt.Errorf("master %d: %w", i, statusErr)
-			continue
-		}
-		if cs.IsLeader {
-			claims = append(claims, i)
-		}
-	}
-	return claims, err
-}
-
-func runningMasters(mc *MasterCluster) int {
-	count := 0
-	for i := range 3 {
-		if mc.IsNodeRunning(i) {
-			count++
-		}
-	}
-	return count
-}
-
-// chaosSeed picks the walk. It is random by default and always logged, so a
-// failure names the seed that reproduces it.
-func chaosSeed(t *testing.T) uint64 {
-	t.Helper()
-	if v := os.Getenv("MULTI_MASTER_IT_SEED"); v != "" {
-		seed, err := strconv.ParseUint(v, 10, 64)
-		if err != nil {
-			t.Fatalf("MULTI_MASTER_IT_SEED %q: %v", v, err)
-		}
-		return seed
-	}
-	return uint64(time.Now().UnixNano())
-}

test/multi_master/cluster.go

@@ -10,7 +10,6 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
-	"regexp"
 	"runtime"
 	"strconv"
 	"strings"
@@ -34,12 +33,6 @@ type masterNode struct {
 	cmd      *exec.Cmd
 	logFile  string
 	stopped  bool
-	// peersStr overrides the cluster-wide peer list for this node, so a test
-	// can start a master that only knows about a subset of the cluster.
-	peersStr string
-	// raftBootstrap starts the master with -raftBootstrap, the way the helm
-	// chart renders master.raftBootstrap on every master, every restart.
-	raftBootstrap bool
 }
 
 // MasterCluster manages a 3-node master raft cluster for integration tests.
@@ -55,9 +48,6 @@ type MasterCluster struct {
 
 	// peers string shared by all nodes, e.g. "127.0.0.1:9333,127.0.0.1:9334,127.0.0.1:9335"
 	peersStr string
-
-	// raftHashicorp starts the masters with -raftHashicorp
-	raftHashicorp bool
 }
 
 // clusterStatus is the JSON returned by /cluster/status.
@@ -71,35 +61,6 @@ type clusterStatus struct {
 func StartMasterCluster(t testing.TB) *MasterCluster {
 	t.Helper()
 
-	mc := NewMasterCluster(t, false)
-	for i := range 3 {
-		mc.StartNode(i)
-	}
-
-	if err := mc.WaitForLeader(waitTimeout); err != nil {
-		mc.DumpLogs()
-		mc.StopAll()
-		t.Fatalf("cluster did not elect a leader: %v", err)
-	}
-
-	// Wait for TopologyId to be generated and propagated. This is async
-	// after leader election, and we need it committed before tests can
-	// reliably stop/restart nodes.
-	if _, err := mc.WaitForTopologyId(waitTimeout); err != nil {
-		mc.DumpLogs()
-		mc.StopAll()
-		t.Fatalf("TopologyId not generated: %v", err)
-	}
-
-	return mc
-}
-
-// NewMasterCluster allocates ports and data directories for a 3-node master
-// cluster without starting anything, so a test can choose what each node comes
-// up with.
-func NewMasterCluster(t testing.TB, raftHashicorp bool) *MasterCluster {
-	t.Helper()
-
 	weedBinary, err := findOrBuildWeedBinary()
 	if err != nil {
 		t.Fatalf("resolve weed binary: %v", err)
@@ -133,14 +94,32 @@ func NewMasterCluster(t testing.TB, raftHashicorp bool) *MasterCluster {
 	}
 
 	mc := &MasterCluster{
-		t:             t,
-		weedBinary:    weedBinary,
-		baseDir:       baseDir,
-		logsDir:       logsDir,
-		keepLogs:      keepLogs,
-		nodes:         nodes,
-		peersStr:      strings.Join(peerParts, ","),
-		raftHashicorp: raftHashicorp,
+		t:          t,
+		weedBinary: weedBinary,
+		baseDir:    baseDir,
+		logsDir:    logsDir,
+		keepLogs:   keepLogs,
+		nodes:      nodes,
+		peersStr:   strings.Join(peerParts, ","),
+	}
+
+	for i := range 3 {
+		mc.StartNode(i)
+	}
+
+	if err := mc.WaitForLeader(waitTimeout); err != nil {
+		mc.DumpLogs()
+		mc.StopAll()
+		t.Fatalf("cluster did not elect a leader: %v", err)
+	}
+
+	// Wait for TopologyId to be generated and propagated. This is async
+	// after leader election, and we need it committed before tests can
+	// reliably stop/restart nodes.
+	if err := mc.WaitForTopologyId(waitTimeout); err != nil {
+		mc.DumpLogs()
+		mc.StopAll()
+		t.Fatalf("TopologyId not generated: %v", err)
 	}
 
 	t.Cleanup(func() {
@@ -149,21 +128,6 @@ func NewMasterCluster(t testing.TB, raftHashicorp bool) *MasterCluster {
 	return mc
 }
 
-// SetNodePeers narrows the peer list node i starts with, mirroring a
-// StatefulSet whose replica count changed under a running master.
-func (mc *MasterCluster) SetNodePeers(i int, peers string) {
-	mc.mu.Lock()
-	defer mc.mu.Unlock()
-	mc.nodes[i].peersStr = peers
-}
-
-// SetRaftBootstrap makes node i start with -raftBootstrap.
-func (mc *MasterCluster) SetRaftBootstrap(i int) {
-	mc.mu.Lock()
-	defer mc.mu.Unlock()
-	mc.nodes[i].raftBootstrap = true
-}
-
 // StartNode starts the master process at the given index (0–2).
 func (mc *MasterCluster) StartNode(i int) {
 	mc.t.Helper()
@@ -180,27 +144,17 @@ func (mc *MasterCluster) StartNode(i int) {
 		mc.t.Fatalf("create log for node %d: %v", i, err)
 	}
 
-	peersStr := n.peersStr
-	if peersStr == "" {
-		peersStr = mc.peersStr
-	}
 	args := []string{
 		"master",
 		"-ip=127.0.0.1",
 		"-port=" + strconv.Itoa(n.port),
 		"-port.grpc=" + strconv.Itoa(n.grpcPort),
 		"-mdir=" + n.dataDir,
-		"-peers=" + peersStr,
+		"-peers=" + mc.peersStr,
 		"-electionTimeout=3s",
 		"-volumeSizeLimitMB=32",
 		"-defaultReplication=000",
 	}
-	if mc.raftHashicorp {
-		args = append(args, "-raftHashicorp")
-	}
-	if n.raftBootstrap {
-		args = append(args, "-raftBootstrap")
-	}
 
 	n.cmd = exec.Command(mc.weedBinary, args...)
 	n.cmd.Dir = mc.baseDir
@@ -338,24 +292,6 @@ func (mc *MasterCluster) WaitForLeader(timeout time.Duration) error {
 	return fmt.Errorf("no leader elected within %v", timeout)
 }
 
-// WaitForNoLeader waits until no running master claims leadership. goraft only
-// checks whether it still has a quorum on an election-timeout ticker, and needs
-// its peers to go quiet for a full timeout first, so a master that has lost its
-// quorum can keep claiming leadership for tens of seconds. It cannot commit
-// anything in that window.
-func (mc *MasterCluster) WaitForNoLeader(timeout time.Duration) error {
-	deadline := time.Now().Add(timeout)
-	for time.Now().Before(deadline) {
-		idx, _ := mc.FindLeader()
-		if idx < 0 {
-			return nil
-		}
-		time.Sleep(waitTick)
-	}
-	idx, addr := mc.FindLeader()
-	return fmt.Errorf("master %d at %s still claims leadership after %v", idx, addr, timeout)
-}
-
 // WaitForNewLeader waits for a leader that is different from the given address.
 func (mc *MasterCluster) WaitForNewLeader(oldLeaderAddr string, timeout time.Duration) (int, string, error) {
 	deadline := time.Now().Add(timeout)
@@ -369,20 +305,18 @@ func (mc *MasterCluster) WaitForNewLeader(oldLeaderAddr string, timeout time.Dur
 	return -1, "", fmt.Errorf("no new leader (different from %s) within %v", oldLeaderAddr, timeout)
 }
 
-// WaitForTopologyId waits until the leader reports a non-empty TopologyId, and
-// returns it. It is only readable once the leader has applied the raft entry
-// carrying it, which lands after the election it won.
-func (mc *MasterCluster) WaitForTopologyId(timeout time.Duration) (string, error) {
+// WaitForTopologyId waits until the leader reports a non-empty TopologyId.
+func (mc *MasterCluster) WaitForTopologyId(timeout time.Duration) error {
 	deadline := time.Now().Add(timeout)
 	for time.Now().Before(deadline) {
 		if idx, _ := mc.FindLeader(); idx >= 0 {
 			if id, err := mc.GetTopologyId(idx); err == nil && id != "" {
-				return id, nil
+				return nil
 			}
 		}
 		time.Sleep(waitTick)
 	}
-	return "", fmt.Errorf("TopologyId not available within %v", timeout)
+	return fmt.Errorf("TopologyId not available within %v", timeout)
 }
 
 // WaitForNodeReady waits for node i to respond to HTTP.
@@ -400,57 +334,6 @@ func (mc *MasterCluster) WaitForNodeReady(i int, timeout time.Duration) error {
 	return fmt.Errorf("node %d not ready within %v", i, timeout)
 }
 
-// LogContains reports whether node i's log holds the given text.
-func (mc *MasterCluster) LogContains(i int, text string) bool {
-	b, err := os.ReadFile(mc.nodes[i].logFile)
-	if err != nil {
-		return false
-	}
-	return strings.Contains(string(b), text)
-}
-
-// topologyIdLine matches every line a master logs when it learns a TopologyId,
-// whichever raft implementation applied it.
-var topologyIdLine = regexp.MustCompile(`TopologyId[^:]*: ([0-9a-f-]{36})`)
-
-// NodeTopologyIds returns the TopologyIds node i has logged. /dir/status is
-// proxied to the leader, so a master's own view of the cluster identity is only
-// visible in its log, and that is where a fork shows up.
-func (mc *MasterCluster) NodeTopologyIds(i int) []string {
-	b, err := os.ReadFile(mc.nodes[i].logFile)
-	if err != nil {
-		return nil
-	}
-	var ids []string
-	for _, m := range topologyIdLine.FindAllStringSubmatch(string(b), -1) {
-		ids = append(ids, m[1])
-	}
-	return ids
-}
-
-// WaitForNodeTopologyIds waits until every master has logged a TopologyId and
-// returns what each one saw.
-func (mc *MasterCluster) WaitForNodeTopologyIds(timeout time.Duration) ([3][]string, error) {
-	var ids [3][]string
-	deadline := time.Now().Add(timeout)
-	for {
-		missing := -1
-		for i := range 3 {
-			ids[i] = mc.NodeTopologyIds(i)
-			if len(ids[i]) == 0 {
-				missing = i
-			}
-		}
-		if missing < 0 {
-			return ids, nil
-		}
-		if !time.Now().Before(deadline) {
-			return ids, fmt.Errorf("master %d logged no TopologyId within %v", missing, timeout)
-		}
-		time.Sleep(waitTick)
-	}
-}
-
 // DumpLogs prints the tail of all master logs.
 func (mc *MasterCluster) DumpLogs() {
 	for i := range 3 {

test/multi_master/failover_test.go

@@ -9,12 +9,7 @@ import (
 
 const (
 	// Election timeout is 3s in our cluster config; allow generous margin.
-	leaderElectionTimeout = 30 * time.Second
-
-	// Losing a quorum is much slower to show than winning one. goraft only
-	// notices on an election-timeout ticker, after its peers have been quiet
-	// for a full timeout, and has been measured taking over 30s to step down.
-	leaderStepDownTimeout = 60 * time.Second
+	leaderElectionTimeout = 20 * time.Second
 )
 
 // TestLeaderDownAndRecoverQuickly verifies that when the leader is stopped and
@@ -154,11 +149,20 @@ func TestTwoMastersDownAndRestart(t *testing.T) {
 	mc.StopNode(down1)
 	mc.StopNode(down2)
 
-	// The surviving node alone cannot form a quorum, so it has to give up the
-	// leadership it is still claiming.
-	if err := mc.WaitForNoLeader(leaderStepDownTimeout); err != nil {
+	// The surviving node alone cannot form a quorum — no leader expected.
+	// Wait long enough for any stale leadership to expire (election timeout
+	// is 3s in our config, quorum check fires every election timeout).
+	time.Sleep(5 * time.Second)
+	soloLeaderIdx, _ := mc.FindLeader()
+	if soloLeaderIdx >= 0 {
+		// It's possible the survivor briefly thinks it's leader before stepping down.
+		// Give it time to realize it lost quorum.
+		time.Sleep(5 * time.Second)
+		soloLeaderIdx, _ = mc.FindLeader()
+	}
+	if soloLeaderIdx >= 0 {
 		mc.DumpLogs()
-		t.Fatalf("expected no leader with only 1 of 3 nodes: %v", err)
+		t.Fatalf("expected no leader with only 1 of 3 nodes, but node %d claims leadership", soloLeaderIdx)
 	}
 
 	// Restart both downed nodes.

test/multi_master/join_test.go

@@ -1,212 +0,0 @@
-package multi_master
-
-import (
-	"fmt"
-	"testing"
-	"time"
-
-	"github.com/seaweedfs/seaweedfs/weed/pb"
-)
-
-var raftImplementations = []struct {
-	name          string
-	raftHashicorp bool
-}{
-	{"goraft", false},
-	{"hashicorp", true},
-}
-
-// TestFreshClusterFormsWithoutALeader covers the other half of the bootstrap
-// decision: with no leader anywhere, three masters starting together still have
-// to mint a cluster between them.
-func TestFreshClusterFormsWithoutALeader(t *testing.T) {
-	for _, impl := range raftImplementations {
-		t.Run(impl.name, func(t *testing.T) {
-			mc := NewMasterCluster(t, impl.raftHashicorp)
-			for i := range 3 {
-				mc.StartNode(i)
-			}
-
-			if _, err := waitForCommonLeader(mc, waitTimeout); err != nil {
-				mc.DumpLogs()
-				t.Fatalf("fresh cluster did not converge: %v", err)
-			}
-			for i := range 3 {
-				if err := waitForPeerCount(mc, i, 2, waitTimeout); err != nil {
-					mc.DumpLogs()
-					t.Fatalf("master %d does not see the full cluster: %v", i, err)
-				}
-			}
-		})
-	}
-}
-
-// TestScaleUpOntoExistingLeader mirrors a Kubernetes master StatefulSet whose
-// replica count goes back from one to three: master 0 keeps running as the
-// leader of a single-peer cluster while two fresh masters come up pointing at
-// all three. The newcomers start with an empty raft log, so neither raft
-// implementation lets them campaign — the sitting leader has to admit them.
-func TestScaleUpOntoExistingLeader(t *testing.T) {
-	for _, impl := range raftImplementations {
-		t.Run(impl.name, func(t *testing.T) {
-			mc := NewMasterCluster(t, impl.raftHashicorp)
-
-			// Master 0 is alone in its peer list, the way the operator renders
-			// -peers when spec.master.replicas is 1.
-			mc.SetNodePeers(0, mc.NodeAddress(0))
-			mc.StartNode(0)
-			if err := mc.WaitForLeader(waitTimeout); err != nil {
-				mc.DumpLogs()
-				t.Fatalf("single master did not become leader: %v", err)
-			}
-
-			// Scale up. These two carry the full peer list; master 0 still
-			// runs with the old one and has never heard of them.
-			mc.StartNode(1)
-			mc.StartNode(2)
-
-			leader, err := waitForCommonLeader(mc, waitTimeout)
-			if err != nil {
-				mc.DumpLogs()
-				t.Fatalf("masters did not converge after scaling up: %v", err)
-			}
-			if leader != mc.NodeAddress(0) {
-				t.Fatalf("leader moved to %s, want the sitting leader %s", leader, mc.NodeAddress(0))
-			}
-
-			if err := waitForPeerCount(mc, 0, 2, waitTimeout); err != nil {
-				mc.DumpLogs()
-				t.Fatalf("leader did not admit both new masters: %v", err)
-			}
-		})
-	}
-}
-
-// waitForCommonLeader waits until every running master names the same leader,
-// and returns it.
-func waitForCommonLeader(mc *MasterCluster, timeout time.Duration) (string, error) {
-	var lastErr error
-	deadline := time.Now().Add(timeout)
-	for time.Now().Before(deadline) {
-		leader, err := commonLeader(mc)
-		if err == nil {
-			return leader, nil
-		}
-		lastErr = err
-		time.Sleep(waitTick)
-	}
-	return "", lastErr
-}
-
-func commonLeader(mc *MasterCluster) (string, error) {
-	agreed := ""
-	for i := range 3 {
-		if !mc.IsNodeRunning(i) {
-			continue
-		}
-		cs, err := mc.GetClusterStatus(i)
-		if err != nil {
-			return "", err
-		}
-		leader := pb.ServerAddress(cs.Leader).ToHttpAddress()
-		if leader == "" {
-			return "", fmt.Errorf("master %d has no leader", i)
-		}
-		if agreed == "" {
-			agreed = leader
-		} else if agreed != leader {
-			return "", fmt.Errorf("masters disagree on the leader: %s and %s", agreed, leader)
-		}
-	}
-	if agreed == "" {
-		return "", fmt.Errorf("no master is running")
-	}
-	return agreed, nil
-}
-
-// waitForPeerCount waits until node i reports the given number of raft peers.
-// The count excludes the node itself.
-func waitForPeerCount(mc *MasterCluster, i, want int, timeout time.Duration) error {
-	got := -1
-	deadline := time.Now().Add(timeout)
-	for time.Now().Before(deadline) {
-		cs, err := mc.GetClusterStatus(i)
-		if err == nil {
-			got = peerCountExcludingSelf(cs.Peers, mc.NodeAddress(i))
-			if got == want {
-				return nil
-			}
-		}
-		time.Sleep(waitTick)
-	}
-	return fmt.Errorf("master %d reports %d peers, want %d", i, got, want)
-}
-
-func peerCountExcludingSelf(peers []string, self string) int {
-	count := 0
-	for _, peer := range peers {
-		if pb.ServerAddress(peer).ToHttpAddress() != self {
-			count++
-		}
-	}
-	return count
-}
-
-// TestRaftBootstrapKeepsExistingCluster covers a master restarting under
-// -raftBootstrap, the way the helm chart renders it on every master on every
-// roll. Bootstrapping is genesis: seeding a second cluster over committed raft
-// state mints a rival TopologyId, and the split-brain guard then Fatals every
-// master that still holds the first one.
-func TestRaftBootstrapKeepsExistingCluster(t *testing.T) {
-	for _, impl := range raftImplementations {
-		t.Run(impl.name, func(t *testing.T) {
-			mc := NewMasterCluster(t, impl.raftHashicorp)
-			for i := range 3 {
-				mc.SetRaftBootstrap(i)
-				mc.StartNode(i)
-			}
-			before, err := mc.WaitForTopologyId(waitTimeout)
-			if err != nil {
-				mc.DumpLogs()
-				t.Fatalf("cluster did not mint a TopologyId: %v", err)
-			}
-
-			for i := range 3 {
-				mc.StopNode(i)
-			}
-			for i := range 3 {
-				mc.StartNode(i)
-			}
-
-			after, err := mc.WaitForTopologyId(waitTimeout)
-			if err != nil {
-				mc.DumpLogs()
-				t.Fatalf("cluster did not come back after a restart: %v", err)
-			}
-			if after != before {
-				mc.DumpLogs()
-				t.Fatalf("-raftBootstrap re-seeded the cluster: TopologyId %s became %s", before, after)
-			}
-
-			// The leader answers for the whole cluster, so a follower that
-			// forked is only visible in its own log.
-			seen, err := mc.WaitForNodeTopologyIds(waitTimeout)
-			if err != nil {
-				mc.DumpLogs()
-				t.Fatal(err)
-			}
-			for i, ids := range seen {
-				for _, id := range ids {
-					if id != before {
-						mc.DumpLogs()
-						t.Fatalf("master %d saw TopologyId %s, want %s", i, id, before)
-					}
-				}
-				if mc.LogContains(i, "Split-brain detected") {
-					mc.DumpLogs()
-					t.Fatalf("master %d hit the split-brain guard", i)
-				}
-			}
-		})
-	}
-}

test/plugin_workers/fake_volume_server.go

@@ -15,7 +15,6 @@ import (
 	"github.com/seaweedfs/seaweedfs/weed/operation"
 	"github.com/seaweedfs/seaweedfs/weed/pb"
 	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
-	"github.com/seaweedfs/seaweedfs/weed/storage/volume_info"
 	"google.golang.org/grpc"
 	"google.golang.org/grpc/credentials/insecure"
 )
@@ -340,14 +339,7 @@ func (v *VolumeServer) VolumeEcShardsInfo(ctx context.Context, req *volume_serve
 		}
 	}
 
-	// Answer with the layout out of the .vif that was actually delivered here,
-	// the way a real holder answers from the context it mounted the shards
-	// with. A coordinator uses this to tell a server that understands the
-	// shard block layout from one that never knew the field, so a fake that
-	// always reported "unset" would look like a pre-upgrade server.
-	resp := &volume_server_pb.VolumeEcShardsInfoResponse{
-		EcShardConfig: v.ecShardConfigFromVif(req.VolumeId),
-	}
+	resp := &volume_server_pb.VolumeEcShardsInfoResponse{}
 	prefix := fmt.Sprintf("%d.ec", req.VolumeId)
 	entries, _ := os.ReadDir(v.baseDir)
 	for _, entry := range entries {
@@ -380,16 +372,6 @@ func (v *VolumeServer) VolumeEcShardsInfo(ctx context.Context, req *volume_serve
 	return resp, nil
 }
 
-// ecShardConfigFromVif reads the EC layout out of the .vif this server was
-// given, which distribution ships to every holder alongside its shards.
-func (v *VolumeServer) ecShardConfigFromVif(volumeID uint32) *volume_server_pb.EcShardConfig {
-	vi, _, found, err := volume_info.MaybeLoadVolumeInfo(v.filePath(volumeID, ".vif"))
-	if err != nil || !found {
-		return nil
-	}
-	return vi.GetEcShardConfig()
-}
-
 func (v *VolumeServer) VolumeDelete(ctx context.Context, req *volume_server_pb.VolumeDeleteRequest) (*volume_server_pb.VolumeDeleteResponse, error) {
 	v.mu.Lock()
 	v.deleteRequests = append(v.deleteRequests, req)

test/s3/copying/s3_rename_test.go

@@ -156,9 +156,8 @@ func TestRenameObjectSourceIfMatch(t *testing.T) {
 	assert.True(t, objectExists(t, client, bucketName, "target.txt"))
 }
 
-// TestRenameObjectOntoDirectory: S3 keys are flat, so a key that other keys are
-// nested under is still a key of its own. The rename writes it without disturbing
-// them - it does not replace the directory, it stores the object on it.
+// TestRenameObjectOntoDirectory: a key that already holds other objects is a
+// directory, and an object must not be allowed to replace one.
 func TestRenameObjectOntoDirectory(t *testing.T) {
 	client := getS3Client(t)
 	bucketName := getNewBucketName()
@@ -173,10 +172,9 @@ func TestRenameObjectOntoDirectory(t *testing.T) {
 		Key:          aws.String("target"),
 		RenameSource: aws.String(createRenameSource("source.txt")),
 	})
-	require.NoError(t, err)
-	assert.False(t, objectExists(t, client, bucketName, "source.txt"))
-	assert.Equal(t, "content", getObjectBody(t, getObject(t, client, bucketName, "target")))
-	assert.Equal(t, "child", getObjectBody(t, getObject(t, client, bucketName, "target/child.txt")))
+	requireRenameStatus(t, err, 409)
+	assert.True(t, objectExists(t, client, bucketName, "source.txt"))
+	assert.True(t, objectExists(t, client, bucketName, "target/child.txt"))
 }
 
 // TestRenameObjectDirectorySource: a directory can be named without a trailing

test/s3/normal/s3_prefix_object_test.go

@@ -1,546 +0,0 @@
-package example
-
-import (
-	"bytes"
-	"io"
-	"net/http"
-	"sort"
-	"strings"
-	"testing"
-	"time"
-
-	"github.com/aws/aws-sdk-go/aws"
-	"github.com/aws/aws-sdk-go/aws/awserr"
-	v1credentials "github.com/aws/aws-sdk-go/aws/credentials"
-	v1signer "github.com/aws/aws-sdk-go/aws/signer/v4"
-	"github.com/aws/aws-sdk-go/service/s3"
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-)
-
-// TestS3PrefixObjectKeys covers keys that are a strict prefix of other keys: S3's
-// namespace is flat, so "collision/foo" and "collision/foo/bar" are independent
-// objects that coexist in either write order.
-func TestS3PrefixObjectKeys(t *testing.T) {
-	if testing.Short() {
-		t.Skip("Skipping integration test in short mode")
-	}
-
-	cluster, err := startMiniCluster(t)
-	require.NoError(t, err)
-	defer cluster.Stop()
-
-	put := func(t *testing.T, bucket, key string, body []byte) {
-		t.Helper()
-		_, err := cluster.s3Client.PutObject(&s3.PutObjectInput{
-			Bucket: aws.String(bucket),
-			Key:    aws.String(key),
-			Body:   bytes.NewReader(body),
-		})
-		require.NoError(t, err, "put %s", key)
-	}
-	// read checks both paths a client reaches an object by, since a directory entry
-	// carrying an object is served by neither the directory nor the plain object path
-	// alone.
-	read := func(t *testing.T, bucket, key string, want []byte) {
-		t.Helper()
-		head, err := cluster.s3Client.HeadObject(&s3.HeadObjectInput{
-			Bucket: aws.String(bucket),
-			Key:    aws.String(key),
-		})
-		require.NoError(t, err, "head %s", key)
-		assert.Equal(t, int64(len(want)), aws.Int64Value(head.ContentLength), "head %s", key)
-
-		get, err := cluster.s3Client.GetObject(&s3.GetObjectInput{
-			Bucket: aws.String(bucket),
-			Key:    aws.String(key),
-		})
-		require.NoError(t, err, "get %s", key)
-		defer get.Body.Close()
-		got, err := io.ReadAll(get.Body)
-		require.NoError(t, err, "read %s", key)
-		assert.Equal(t, want, got, "get %s", key)
-	}
-	// gone checks the key answers as absent rather than lingering on a directory
-	// entry that outlived the object.
-	gone := func(t *testing.T, bucket, key string) {
-		t.Helper()
-		_, err := cluster.s3Client.GetObject(&s3.GetObjectInput{
-			Bucket: aws.String(bucket),
-			Key:    aws.String(key),
-		})
-		var missing awserr.RequestFailure
-		require.ErrorAs(t, err, &missing, "get %s", key)
-		assert.Equal(t, http.StatusNotFound, missing.StatusCode(), "get %s", key)
-
-		_, err = cluster.s3Client.HeadObject(&s3.HeadObjectInput{
-			Bucket: aws.String(bucket),
-			Key:    aws.String(key),
-		})
-		require.ErrorAs(t, err, &missing, "head %s", key)
-		assert.Equal(t, http.StatusNotFound, missing.StatusCode(), "head %s", key)
-	}
-	listKeys := func(t *testing.T, bucket string) []string {
-		t.Helper()
-		resp, err := cluster.s3Client.ListObjectsV2(&s3.ListObjectsV2Input{Bucket: aws.String(bucket)})
-		require.NoError(t, err)
-		keys := collectKeys(resp.Contents)
-		sort.Strings(keys)
-		return keys
-	}
-
-	body := []byte("prefix object")
-	// Distinct bodies, so a read that resolves to the wrong entry cannot pass.
-	nested := []byte("nested under the prefix object")
-
-	// The reported order: the nested key is written first, so the prefix key has to
-	// land on a path the filer already holds a directory at.
-	t.Run("ChildFirst", func(t *testing.T) {
-		bucket := createTestBucket(t, cluster, "test-prefix-child-first-")
-		put(t, bucket, "collision/foo/bar", nested)
-		put(t, bucket, "collision/foo", body)
-
-		assert.Equal(t, []string{"collision/foo", "collision/foo/bar"}, listKeys(t, bucket))
-		read(t, bucket, "collision/foo", body)
-		read(t, bucket, "collision/foo/bar", nested)
-	})
-
-	// The opposite order used to keep the prefix key's data but hide the key.
-	t.Run("PrefixFirst", func(t *testing.T) {
-		bucket := createTestBucket(t, cluster, "test-prefix-first-")
-		put(t, bucket, "collision/foo", body)
-		put(t, bucket, "collision/foo/bar", nested)
-
-		assert.Equal(t, []string{"collision/foo", "collision/foo/bar"}, listKeys(t, bucket))
-		read(t, bucket, "collision/foo", body)
-		read(t, bucket, "collision/foo/bar", nested)
-	})
-
-	// An empty object leaves no chunks, content or mime behind, so it is the case a
-	// promoted directory carries no other trace of.
-	t.Run("EmptyObject", func(t *testing.T) {
-		bucket := createTestBucket(t, cluster, "test-prefix-empty-")
-		put(t, bucket, "a/foo/bar", nil)
-		put(t, bucket, "a/foo", nil)
-		put(t, bucket, "b/foo", nil)
-		put(t, bucket, "b/foo/bar", nil)
-
-		assert.Equal(t, []string{"a/foo", "a/foo/bar", "b/foo", "b/foo/bar"}, listKeys(t, bucket))
-		read(t, bucket, "a/foo", []byte{})
-		read(t, bucket, "b/foo", []byte{})
-	})
-
-	// The key has no trailing slash, and the keys nested under it still roll up into
-	// their own CommonPrefix.
-	t.Run("Delimiter", func(t *testing.T) {
-		bucket := createTestBucket(t, cluster, "test-prefix-delimiter-")
-		put(t, bucket, "collision/foo/bar", nested)
-		put(t, bucket, "collision/foo", body)
-		put(t, bucket, "collision/other", body)
-
-		resp, err := cluster.s3Client.ListObjectsV2(&s3.ListObjectsV2Input{
-			Bucket:    aws.String(bucket),
-			Prefix:    aws.String("collision/"),
-			Delimiter: aws.String("/"),
-		})
-		require.NoError(t, err)
-
-		keys := collectKeys(resp.Contents)
-		sort.Strings(keys)
-		assert.Equal(t, []string{"collision/foo", "collision/other"}, keys)
-		assert.Equal(t, []string{"collision/foo/"}, collectPrefixes(resp.CommonPrefixes))
-
-		// Listing the prefix itself names only what is under it.
-		resp, err = cluster.s3Client.ListObjectsV2(&s3.ListObjectsV2Input{
-			Bucket:    aws.String(bucket),
-			Prefix:    aws.String("collision/foo/"),
-			Delimiter: aws.String("/"),
-		})
-		require.NoError(t, err)
-		assert.Equal(t, []string{"collision/foo/bar"}, collectKeys(resp.Contents))
-		assert.Empty(t, collectPrefixes(resp.CommonPrefixes))
-	})
-
-	// The key and the CommonPrefix its nested keys fold into come off one filer
-	// entry, so a page boundary must not drop either of them.
-	t.Run("Paged", func(t *testing.T) {
-		bucket := createTestBucket(t, cluster, "test-prefix-paged-")
-		for _, key := range []string{"foo", "foo/bar", "foobar", "other", "zed", "zed/a"} {
-			put(t, bucket, key, body)
-		}
-
-		for _, maxKeys := range []int64{1, 2, 3, 4, 5} {
-			var keys, prefixes []string
-			var token *string
-			for page := 0; page < 12; page++ {
-				resp, err := cluster.s3Client.ListObjectsV2(&s3.ListObjectsV2Input{
-					Bucket:            aws.String(bucket),
-					Delimiter:         aws.String("/"),
-					MaxKeys:           aws.Int64(maxKeys),
-					ContinuationToken: token,
-				})
-				require.NoError(t, err)
-				// One entry over the budget is the documented same-entry exception;
-				// anything more means the unsigned budget wrapped.
-				assert.LessOrEqual(t, int64(len(resp.Contents)+len(resp.CommonPrefixes)), maxKeys+1,
-					"maxKeys=%d page %d", maxKeys, page)
-				keys = append(keys, collectKeys(resp.Contents)...)
-				prefixes = append(prefixes, collectPrefixes(resp.CommonPrefixes)...)
-				if !aws.BoolValue(resp.IsTruncated) {
-					token = nil
-					break
-				}
-				token = resp.NextContinuationToken
-				require.NotNil(t, token, "a truncated page must name where to resume")
-			}
-			require.Nil(t, token, "maxKeys=%d did not finish", maxKeys)
-			assert.Equal(t, []string{"foo", "foobar", "other", "zed"}, keys, "maxKeys=%d", maxKeys)
-			assert.Equal(t, []string{"foo/", "zed/"}, prefixes, "maxKeys=%d", maxKeys)
-		}
-	})
-
-	// Versioning reaches a prefix object from two directions: a suspended bucket
-	// writes the null version at the key's own path, and a bucket versioned later
-	// finds one already sitting there. Both leave a key that is a directory with
-	// version history beside it.
-	t.Run("Versioned", func(t *testing.T) {
-		setVersioning := func(t *testing.T, bucket, status string) {
-			t.Helper()
-			_, err := cluster.s3Client.PutBucketVersioning(&s3.PutBucketVersioningInput{
-				Bucket:                  aws.String(bucket),
-				VersioningConfiguration: &s3.VersioningConfiguration{Status: aws.String(status)},
-			})
-			require.NoError(t, err)
-		}
-
-		// Both write orders, in a bucket that is versioned and in one where versioning
-		// was suspended - the suspended one is the case that writes at the key's path.
-		for _, state := range []string{"Enabled", "Suspended"} {
-			bucket := createTestBucket(t, cluster, "test-prefix-"+strings.ToLower(state)+"-")
-			setVersioning(t, bucket, "Enabled")
-			if state == "Suspended" {
-				setVersioning(t, bucket, "Suspended")
-			}
-			put(t, bucket, "child/foo/bar", nested)
-			put(t, bucket, "child/foo", body)
-			put(t, bucket, "prefix/foo", body)
-			put(t, bucket, "prefix/foo/bar", nested)
-
-			assert.Equal(t, []string{"child/foo", "child/foo/bar", "prefix/foo", "prefix/foo/bar"},
-				listKeys(t, bucket), state)
-			for _, key := range []string{"child/foo", "prefix/foo"} {
-				read(t, bucket, key, body)
-			}
-			for _, key := range []string{"child/foo/bar", "prefix/foo/bar"} {
-				read(t, bucket, key, nested)
-			}
-		}
-
-		// A prefix object written before versioning is the key's null version. Removing
-		// that version by id must not take the keys nested under it with it.
-		bucket := createTestBucket(t, cluster, "test-prefix-nullversion-")
-		put(t, bucket, "collision/foo/bar", nested)
-		put(t, bucket, "collision/foo", body)
-		setVersioning(t, bucket, "Enabled")
-		newer := []byte("written after versioning was enabled")
-		versioned, err := cluster.s3Client.PutObject(&s3.PutObjectInput{
-			Bucket: aws.String(bucket),
-			Key:    aws.String("collision/foo"),
-			Body:   bytes.NewReader(newer),
-		})
-		require.NoError(t, err)
-
-		for _, v := range []struct {
-			id   string
-			want []byte
-		}{{"null", body}, {aws.StringValue(versioned.VersionId), newer}} {
-			got, err := cluster.s3Client.GetObject(&s3.GetObjectInput{
-				Bucket:    aws.String(bucket),
-				Key:       aws.String("collision/foo"),
-				VersionId: aws.String(v.id),
-			})
-			require.NoError(t, err, "get version %s", v.id)
-			body, err := io.ReadAll(got.Body)
-			require.NoError(t, err)
-			got.Body.Close()
-			assert.Equal(t, v.want, body, "get version %s", v.id)
-		}
-
-		_, err = cluster.s3Client.DeleteObject(&s3.DeleteObjectInput{
-			Bucket:    aws.String(bucket),
-			Key:       aws.String("collision/foo"),
-			VersionId: aws.String("null"),
-		})
-		require.NoError(t, err, "the null version sits on a directory other keys live in")
-
-		read(t, bucket, "collision/foo", newer)
-		read(t, bucket, "collision/foo/bar", nested)
-		remaining, err := cluster.s3Client.ListObjectVersions(&s3.ListObjectVersionsInput{
-			Bucket: aws.String(bucket),
-			Prefix: aws.String("collision/foo"),
-		})
-		require.NoError(t, err)
-		for _, v := range remaining.Versions {
-			if aws.StringValue(v.Key) != "collision/foo" {
-				// collision/foo/bar predates versioning too, and keeps its null version.
-				continue
-			}
-			assert.NotEqual(t, "null", aws.StringValue(v.VersionId), "the null version was deleted")
-		}
-	})
-
-	// The two listings walk the tree differently, and a prefix object is the entry
-	// they disagree about: it is a directory the version listing descends through and
-	// a key at the same time. They have to name the same keys and the same prefixes.
-	t.Run("VersionListingMatchesObjectListing", func(t *testing.T) {
-		bucket := createTestBucket(t, cluster, "test-prefix-parity-")
-		for _, key := range []string{"foo", "foo/bar", "other", "a/foo", "a/foo/bar", "a/z"} {
-			put(t, bucket, key, body)
-		}
-
-		for _, q := range []struct{ prefix, delimiter string }{
-			{"", ""},
-			{"foo/", ""},
-			{"a/", ""},
-			{"a/foo/", ""},
-			{"", "/"},
-			{"a/", "/"},
-		} {
-			name := "prefix=" + q.prefix + " delimiter=" + q.delimiter
-
-			objects, err := cluster.s3Client.ListObjectsV2(&s3.ListObjectsV2Input{
-				Bucket:    aws.String(bucket),
-				Prefix:    aws.String(q.prefix),
-				Delimiter: aws.String(q.delimiter),
-			})
-			require.NoError(t, err, name)
-
-			versions, err := cluster.s3Client.ListObjectVersions(&s3.ListObjectVersionsInput{
-				Bucket:    aws.String(bucket),
-				Prefix:    aws.String(q.prefix),
-				Delimiter: aws.String(q.delimiter),
-			})
-			require.NoError(t, err, name)
-
-			versionKeys := make([]string, 0, len(versions.Versions))
-			for _, v := range versions.Versions {
-				versionKeys = append(versionKeys, aws.StringValue(v.Key))
-			}
-			versionPrefixes := make([]string, 0, len(versions.CommonPrefixes))
-			for _, p := range versions.CommonPrefixes {
-				versionPrefixes = append(versionPrefixes, aws.StringValue(p.Prefix))
-			}
-			sort.Strings(versionKeys)
-			sort.Strings(versionPrefixes)
-
-			objectKeys := collectKeys(objects.Contents)
-			objectPrefixes := collectPrefixes(objects.CommonPrefixes)
-			sort.Strings(objectKeys)
-			sort.Strings(objectPrefixes)
-
-			assert.Equal(t, objectKeys, versionKeys, "keys, %s", name)
-			assert.Equal(t, objectPrefixes, versionPrefixes, "prefixes, %s", name)
-		}
-	})
-
-	// A prefix object written before versioning is the key's null version, so the
-	// version written after it has to take the latest flag off it.
-	t.Run("VersionedAfterPrefixObject", func(t *testing.T) {
-		bucket := createTestBucket(t, cluster, "test-prefix-versioned-")
-		put(t, bucket, "collision/foo/bar", nested)
-		put(t, bucket, "collision/foo", body)
-		_, err := cluster.s3Client.PutBucketVersioning(&s3.PutBucketVersioningInput{
-			Bucket:                  aws.String(bucket),
-			VersioningConfiguration: &s3.VersioningConfiguration{Status: aws.String("Enabled")},
-		})
-		require.NoError(t, err)
-		newer := []byte("written after versioning was enabled")
-		put(t, bucket, "collision/foo", newer)
-
-		resp, err := cluster.s3Client.ListObjectVersions(&s3.ListObjectVersionsInput{Bucket: aws.String(bucket)})
-		require.NoError(t, err)
-
-		latest := map[string]int{}
-		var nullSize int64 = -1
-		for _, v := range resp.Versions {
-			if aws.BoolValue(v.IsLatest) {
-				latest[aws.StringValue(v.Key)]++
-			}
-			if aws.StringValue(v.Key) == "collision/foo" && aws.StringValue(v.VersionId) == "null" {
-				nullSize = aws.Int64Value(v.Size)
-				assert.False(t, aws.BoolValue(v.IsLatest), "the newer version is the latest one")
-			}
-		}
-		assert.Equal(t, 1, latest["collision/foo"], "exactly one version of a key is the latest")
-		assert.Equal(t, int64(len(body)), nullSize, "the null version keeps the prefix object's size")
-		read(t, bucket, "collision/foo", newer)
-		read(t, bucket, "collision/foo/bar", nested)
-	})
-
-	// A key that other keys are nested under is a copy source and a copy destination
-	// like any other. The keys nested under either end are not part of the copy.
-	t.Run("Copy", func(t *testing.T) {
-		bucket := createTestBucket(t, cluster, "test-prefix-copy-")
-		copyObject := func(t *testing.T, src, dst string) {
-			t.Helper()
-			_, err := cluster.s3Client.CopyObject(&s3.CopyObjectInput{
-				Bucket:     aws.String(bucket),
-				Key:        aws.String(dst),
-				CopySource: aws.String(bucket + "/" + src),
-			})
-			require.NoError(t, err, "copy %s to %s", src, dst)
-		}
-
-		put(t, bucket, "collision/foo", body)
-		put(t, bucket, "collision/foo/bar", nested)
-
-		// Out of a prefix object, into a key of its own.
-		copyObject(t, "collision/foo", "plain")
-		read(t, bucket, "plain", body)
-		read(t, bucket, "collision/foo", body)
-		read(t, bucket, "collision/foo/bar", nested)
-
-		// Into a key that other keys are nested under.
-		put(t, bucket, "target/child", nested)
-		copyObject(t, "plain", "target")
-		read(t, bucket, "target", body)
-		read(t, bucket, "target/child", nested)
-
-		// And between two of them.
-		put(t, bucket, "other", []byte("copied between prefix keys"))
-		copyObject(t, "other", "collision/foo")
-		read(t, bucket, "collision/foo", []byte("copied between prefix keys"))
-		read(t, bucket, "collision/foo/bar", nested)
-
-		assert.Equal(t, []string{"collision/foo", "collision/foo/bar", "other", "plain", "target", "target/child"},
-			listKeys(t, bucket))
-	})
-
-	// Rename moves the object off the key without moving the keys nested under it,
-	// which is not what the filer's atomic rename of a directory would do.
-	t.Run("Rename", func(t *testing.T) {
-		bucket := createTestBucket(t, cluster, "test-prefix-rename-")
-		renameObject := func(t *testing.T, src, dst string) {
-			t.Helper()
-			req, _ := http.NewRequest(http.MethodPut, cluster.s3Endpoint+"/"+bucket+"/"+dst+"?renameObject=", nil)
-			req.Header.Set("x-amz-rename-source", "/"+bucket+"/"+src)
-			signer := v1signer.NewSigner(v1credentials.NewStaticCredentials(testAccessKey, testSecretKey, ""))
-			_, err := signer.Sign(req, nil, "s3", testRegion, time.Now())
-			require.NoError(t, err)
-
-			resp, err := (&http.Client{Timeout: 20 * time.Second}).Do(req)
-			require.NoError(t, err, "rename %s to %s", src, dst)
-			defer resp.Body.Close()
-			io.Copy(io.Discard, resp.Body)
-			require.Equal(t, http.StatusOK, resp.StatusCode, "rename %s to %s", src, dst)
-		}
-
-		put(t, bucket, "collision/foo", body)
-		put(t, bucket, "collision/foo/bar", nested)
-
-		// Off a prefix object: the key goes, the keys under it stay.
-		renameObject(t, "collision/foo", "moved")
-		read(t, bucket, "moved", body)
-		read(t, bucket, "collision/foo/bar", nested)
-		gone(t, bucket, "collision/foo")
-
-		// Onto a key other keys are nested under.
-		put(t, bucket, "target/child", nested)
-		renameObject(t, "moved", "target")
-		read(t, bucket, "target", body)
-		read(t, bucket, "target/child", nested)
-		gone(t, bucket, "moved")
-
-		assert.Equal(t, []string{"collision/foo/bar", "target", "target/child"}, listKeys(t, bucket))
-	})
-
-	// A directory SeaweedFS keeps its own state in is not a prefix a key can be
-	// stored on: the object would replace that state with its own.
-	t.Run("ReservedDirectory", func(t *testing.T) {
-		bucket := createTestBucket(t, cluster, "test-prefix-reserved-")
-		_, err := cluster.s3Client.PutBucketVersioning(&s3.PutBucketVersioningInput{
-			Bucket:                  aws.String(bucket),
-			VersioningConfiguration: &s3.VersioningConfiguration{Status: aws.String("Enabled")},
-		})
-		require.NoError(t, err)
-		put(t, bucket, "foo", body)
-		_, err = cluster.s3Client.PutBucketVersioning(&s3.PutBucketVersioningInput{
-			Bucket:                  aws.String(bucket),
-			VersioningConfiguration: &s3.VersioningConfiguration{Status: aws.String("Suspended")},
-		})
-		require.NoError(t, err)
-
-		_, err = cluster.s3Client.PutObject(&s3.PutObjectInput{
-			Bucket: aws.String(bucket),
-			Key:    aws.String("foo.versions"),
-			Body:   bytes.NewReader(body),
-		})
-		// Not just any error: a permanently impossible write must not come back as a
-		// 500 the SDK retries.
-		var refused awserr.RequestFailure
-		require.ErrorAs(t, err, &refused, "the version history of foo is not a prefix of foo.versions")
-		assert.Equal(t, http.StatusConflict, refused.StatusCode())
-		assert.Equal(t, "ExistingObjectIsDirectory", refused.Code())
-
-		versions, err := cluster.s3Client.ListObjectVersions(&s3.ListObjectVersionsInput{Bucket: aws.String(bucket)})
-		require.NoError(t, err)
-		require.Len(t, versions.Versions, 1)
-		assert.Equal(t, "foo", aws.StringValue(versions.Versions[0].Key))
-		read(t, bucket, "foo", body)
-
-		// The multipart staging folder is the other one, and an in-flight upload has
-		// to survive the attempt.
-		staging := createTestBucket(t, cluster, "test-prefix-uploads-")
-		created, err := cluster.s3Client.CreateMultipartUpload(&s3.CreateMultipartUploadInput{
-			Bucket: aws.String(staging),
-			Key:    aws.String("mp.bin"),
-		})
-		require.NoError(t, err)
-
-		_, err = cluster.s3Client.PutObject(&s3.PutObjectInput{
-			Bucket: aws.String(staging),
-			Key:    aws.String(".uploads"),
-			Body:   bytes.NewReader(body),
-		})
-		require.ErrorAs(t, err, &refused, "the multipart staging folder is not a prefix of .uploads")
-		assert.Equal(t, http.StatusConflict, refused.StatusCode())
-		assert.Equal(t, "ExistingObjectIsDirectory", refused.Code())
-
-		uploads, err := cluster.s3Client.ListMultipartUploads(&s3.ListMultipartUploadsInput{Bucket: aws.String(staging)})
-		require.NoError(t, err)
-		require.Len(t, uploads.Uploads, 1)
-		assert.Equal(t, "mp.bin", aws.StringValue(uploads.Uploads[0].Key))
-		_, err = cluster.s3Client.AbortMultipartUpload(&s3.AbortMultipartUploadInput{
-			Bucket:   aws.String(staging),
-			Key:      aws.String("mp.bin"),
-			UploadId: created.UploadId,
-		})
-		require.NoError(t, err)
-	})
-
-	// Either key can be deleted without touching the other.
-	t.Run("Delete", func(t *testing.T) {
-		bucket := createTestBucket(t, cluster, "test-prefix-delete-")
-		put(t, bucket, "collision/foo/bar", nested)
-		put(t, bucket, "collision/foo", body)
-
-		_, err := cluster.s3Client.DeleteObject(&s3.DeleteObjectInput{
-			Bucket: aws.String(bucket),
-			Key:    aws.String("collision/foo"),
-		})
-		require.NoError(t, err)
-		assert.Equal(t, []string{"collision/foo/bar"}, listKeys(t, bucket))
-		read(t, bucket, "collision/foo/bar", nested)
-		gone(t, bucket, "collision/foo")
-
-		put(t, bucket, "collision/foo", body)
-		_, err = cluster.s3Client.DeleteObject(&s3.DeleteObjectInput{
-			Bucket: aws.String(bucket),
-			Key:    aws.String("collision/foo/bar"),
-		})
-		require.NoError(t, err)
-		assert.Equal(t, []string{"collision/foo"}, listKeys(t, bucket))
-		read(t, bucket, "collision/foo", body)
-		gone(t, bucket, "collision/foo/bar")
-	})
-}

test/s3/proxy_signature/README.md

@@ -26,16 +26,16 @@ nginx (:9000)
     v
 SeaweedFS S3 (:8333, -s3.externalUrl=http://localhost:9000)
     |  externalHost = "localhost:9000" (parsed at startup)
-    |  extractHostHeaderCandidates() tries "localhost:9000" first
+    |  extractHostHeader() returns "localhost:9000"
     |  Matches what AWS CLI signed with
     v
 Signature verification succeeds
 ```
 
-**Note:** `-s3.externalUrl` is tried first, not exclusively. A client that
-dials the backend port (8333) directly still verifies against the host it
-actually signed, so a mixed topology of proxied and in-cluster clients works
-with the flag set.
+**Note:** When `-s3.externalUrl` is configured, direct access to the backend
+port (8333) will fail signature verification because the client signs with a
+different Host header than what `externalUrl` specifies. This is expected —
+all S3 traffic should go through the proxy.
 
 ## Prerequisites
 

test/s3/retention/s3_object_lock_directory_marker_test.go

@@ -1,231 +0,0 @@
-package retention
-
-import (
-	"context"
-	"errors"
-	"strings"
-	"testing"
-	"time"
-
-	"github.com/aws/aws-sdk-go-v2/aws"
-	"github.com/aws/aws-sdk-go-v2/service/s3"
-	"github.com/aws/aws-sdk-go-v2/service/s3/types"
-	"github.com/aws/smithy-go"
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-)
-
-func requireAccessDenied(t *testing.T, err error, msg string) {
-	t.Helper()
-	require.Error(t, err, msg)
-	var apiErr smithy.APIError
-	require.True(t, errors.As(err, &apiErr), "expected an API error, got %T", err)
-	assert.Equal(t, "AccessDenied", apiErr.ErrorCode(), msg)
-}
-
-// A key ending in "/" is stored as the filer directory rather than as an object
-// beside it, and is deleted the unversioned way. Object Lock still covers it: the
-// gateway lists it as an object and serves retention set on it.
-func TestObjectLockDirectoryMarker(t *testing.T) {
-	client := getS3Client(t)
-	bucketName := getNewBucketName()
-
-	createBucketWithObjectLock(t, client, bucketName)
-	defer deleteBucket(t, client, bucketName)
-
-	retainUntil := time.Now().Add(24 * time.Hour)
-
-	t.Run("retention headers are honored, not dropped", func(t *testing.T) {
-		key := "records/evidence/"
-		_, err := client.PutObject(context.TODO(), &s3.PutObjectInput{
-			Bucket:                    aws.String(bucketName),
-			Key:                       aws.String(key),
-			Body:                      strings.NewReader("marker"),
-			ObjectLockMode:            types.ObjectLockModeCompliance,
-			ObjectLockRetainUntilDate: aws.Time(retainUntil),
-		})
-		require.NoError(t, err)
-
-		_, err = client.DeleteObject(context.TODO(), &s3.DeleteObjectInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-		})
-		requireAccessDenied(t, err, "a retained marker must not be deletable")
-
-		_, err = client.HeadObject(context.TODO(), &s3.HeadObjectInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-		})
-		assert.NoError(t, err, "the marker must still be there after the refused delete")
-	})
-
-	t.Run("retention set through PutObjectRetention is honored", func(t *testing.T) {
-		key := "records/ledger/"
-		_, err := client.PutObject(context.TODO(), &s3.PutObjectInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-			Body:   strings.NewReader("marker"),
-		})
-		require.NoError(t, err)
-
-		_, err = client.PutObjectRetention(context.TODO(), &s3.PutObjectRetentionInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-			Retention: &types.ObjectLockRetention{
-				Mode:            types.ObjectLockRetentionModeCompliance,
-				RetainUntilDate: aws.Time(retainUntil),
-			},
-		})
-		require.NoError(t, err)
-
-		_, err = client.DeleteObject(context.TODO(), &s3.DeleteObjectInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-		})
-		requireAccessDenied(t, err, "retention the gateway serves back must also block the delete")
-
-		_, err = client.HeadObject(context.TODO(), &s3.HeadObjectInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-		})
-		assert.NoError(t, err, "the marker must still be there after the refused delete")
-	})
-
-	t.Run("multi-object delete is refused too", func(t *testing.T) {
-		key := "records/batch/"
-		_, err := client.PutObject(context.TODO(), &s3.PutObjectInput{
-			Bucket:                    aws.String(bucketName),
-			Key:                       aws.String(key),
-			Body:                      strings.NewReader("marker"),
-			ObjectLockMode:            types.ObjectLockModeCompliance,
-			ObjectLockRetainUntilDate: aws.Time(retainUntil),
-		})
-		require.NoError(t, err)
-
-		resp, err := client.DeleteObjects(context.TODO(), &s3.DeleteObjectsInput{
-			Bucket: aws.String(bucketName),
-			Delete: &types.Delete{Objects: []types.ObjectIdentifier{{Key: aws.String(key)}}},
-		})
-		require.NoError(t, err)
-		assert.Empty(t, resp.Deleted, "a retained marker must not be reported deleted")
-		require.Len(t, resp.Errors, 1)
-		assert.Equal(t, key, aws.ToString(resp.Errors[0].Key))
-		assert.Equal(t, "AccessDenied", aws.ToString(resp.Errors[0].Code))
-
-		_, err = client.HeadObject(context.TODO(), &s3.HeadObjectInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-		})
-		assert.NoError(t, err, "the marker must survive the batch delete")
-	})
-
-	t.Run("invalid lock headers are rejected, as on a regular key", func(t *testing.T) {
-		_, err := client.PutObject(context.TODO(), &s3.PutObjectInput{
-			Bucket:                    aws.String(bucketName),
-			Key:                       aws.String("records/bad-mode/"),
-			Body:                      strings.NewReader("marker"),
-			ObjectLockMode:            "INVALID_MODE",
-			ObjectLockRetainUntilDate: aws.Time(retainUntil),
-		})
-		require.Error(t, err)
-
-		_, err = client.PutObject(context.TODO(), &s3.PutObjectInput{
-			Bucket:         aws.String(bucketName),
-			Key:            aws.String("records/no-date/"),
-			Body:           strings.NewReader("marker"),
-			ObjectLockMode: types.ObjectLockModeGovernance,
-		})
-		require.Error(t, err)
-	})
-
-	// Past 1KiB a trailing-slash key is a real versioned object, and the delete
-	// takes the whole history at once, so an older retained version has to block
-	// it even when the version on top carries no retention of its own.
-	t.Run("a retained version under an unretained one still blocks", func(t *testing.T) {
-		key := "records/history/"
-		body := strings.Repeat("x", 2048)
-
-		first, err := client.PutObject(context.TODO(), &s3.PutObjectInput{
-			Bucket:                    aws.String(bucketName),
-			Key:                       aws.String(key),
-			Body:                      strings.NewReader(body),
-			ObjectLockMode:            types.ObjectLockModeCompliance,
-			ObjectLockRetainUntilDate: aws.Time(retainUntil),
-		})
-		require.NoError(t, err)
-		require.NotNil(t, first.VersionId)
-
-		_, err = client.PutObject(context.TODO(), &s3.PutObjectInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-			Body:   strings.NewReader(body),
-		})
-		require.NoError(t, err)
-
-		_, err = client.DeleteObject(context.TODO(), &s3.DeleteObjectInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-		})
-		requireAccessDenied(t, err, "the retained version underneath must block the delete")
-
-		_, err = client.HeadObject(context.TODO(), &s3.HeadObjectInput{
-			Bucket:    aws.String(bucketName),
-			Key:       aws.String(key),
-			VersionId: first.VersionId,
-		})
-		assert.NoError(t, err, "the retained version must survive")
-	})
-
-	// mkdir replaces the marker entry outright, taking its lock metadata with it,
-	// so a plain PUT over a retained marker has to be refused - including once the
-	// key has grown a version history that the latest-version lookup would find
-	// instead of the marker.
-	t.Run("a plain PUT cannot replace a retained marker", func(t *testing.T) {
-		key := "records/overwrite/"
-		_, err := client.PutObject(context.TODO(), &s3.PutObjectInput{
-			Bucket:                    aws.String(bucketName),
-			Key:                       aws.String(key),
-			Body:                      strings.NewReader("marker"),
-			ObjectLockMode:            types.ObjectLockModeCompliance,
-			ObjectLockRetainUntilDate: aws.Time(retainUntil),
-		})
-		require.NoError(t, err)
-
-		_, err = client.PutObject(context.TODO(), &s3.PutObjectInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-			Body:   strings.NewReader("replacement"),
-		})
-		requireAccessDenied(t, err, "a retained marker must not be replaceable")
-
-		_, err = client.PutObject(context.TODO(), &s3.PutObjectInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-			Body:   strings.NewReader(strings.Repeat("x", 2048)),
-		})
-		require.NoError(t, err, "a versioned write of the same key adds a version")
-
-		_, err = client.PutObject(context.TODO(), &s3.PutObjectInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-			Body:   strings.NewReader("replacement"),
-		})
-		requireAccessDenied(t, err, "the history must not hide the marker's own lock")
-	})
-
-	t.Run("an unretained marker still deletes", func(t *testing.T) {
-		key := "records/plain/"
-		_, err := client.PutObject(context.TODO(), &s3.PutObjectInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-			Body:   strings.NewReader("marker"),
-		})
-		require.NoError(t, err)
-
-		_, err = client.DeleteObject(context.TODO(), &s3.DeleteObjectInput{
-			Bucket: aws.String(bucketName),
-			Key:    aws.String(key),
-		})
-		require.NoError(t, err)
-	})
-}

test/s3/versioning/s3_conditional_reads_test.go

@@ -1,130 +0,0 @@
-package s3api
-
-import (
-	"context"
-	"errors"
-	"testing"
-	"time"
-
-	"github.com/aws/aws-sdk-go-v2/aws"
-	"github.com/aws/aws-sdk-go-v2/service/s3"
-	"github.com/aws/smithy-go"
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-)
-
-func requireAPIErrorCode(t *testing.T, err error, expected string) {
-	t.Helper()
-	require.Error(t, err)
-	var apiErr smithy.APIError
-	require.True(t, errors.As(err, &apiErr), "expected a smithy.APIError, got %T: %v", err, err)
-	assert.Equal(t, expected, apiErr.ErrorCode())
-}
-
-// TestConditionalReadsOfMissingObject verifies that a missing key stays a missing key
-// under If-Match and If-Unmodified-Since instead of surfacing as 412.
-// reproduces issue #10984
-func TestConditionalReadsOfMissingObject(t *testing.T) {
-	client := getS3Client(t)
-	bucketName := getNewBucketName()
-
-	createBucket(t, client, bucketName)
-	defer deleteBucket(t, client, bucketName)
-
-	existing := putObject(t, client, bucketName, "etag-source", "content")
-	require.NotNil(t, existing.ETag)
-	future := aws.Time(time.Now().Add(24 * time.Hour))
-	missing := aws.String("conditional-missing")
-
-	t.Run("HeadObject If-Match", func(t *testing.T) {
-		_, err := client.HeadObject(context.TODO(), &s3.HeadObjectInput{
-			Bucket: aws.String(bucketName), Key: missing, IfMatch: existing.ETag,
-		})
-		requireAPIErrorCode(t, err, "NotFound")
-	})
-
-	t.Run("HeadObject If-Unmodified-Since", func(t *testing.T) {
-		_, err := client.HeadObject(context.TODO(), &s3.HeadObjectInput{
-			Bucket: aws.String(bucketName), Key: missing, IfUnmodifiedSince: future,
-		})
-		requireAPIErrorCode(t, err, "NotFound")
-	})
-
-	t.Run("GetObject If-Match", func(t *testing.T) {
-		_, err := client.GetObject(context.TODO(), &s3.GetObjectInput{
-			Bucket: aws.String(bucketName), Key: missing, IfMatch: existing.ETag,
-		})
-		requireAPIErrorCode(t, err, "NoSuchKey")
-	})
-
-	t.Run("GetObject If-Unmodified-Since", func(t *testing.T) {
-		_, err := client.GetObject(context.TODO(), &s3.GetObjectInput{
-			Bucket: aws.String(bucketName), Key: missing, IfUnmodifiedSince: future,
-		})
-		requireAPIErrorCode(t, err, "NoSuchKey")
-	})
-
-	t.Run("GetObject stale If-Match on a live object stays 412", func(t *testing.T) {
-		_, err := client.GetObject(context.TODO(), &s3.GetObjectInput{
-			Bucket: aws.String(bucketName), Key: aws.String("etag-source"),
-			IfMatch: aws.String(`"0000000000000000000000000000dead"`),
-		})
-		requireAPIErrorCode(t, err, "PreconditionFailed")
-	})
-}
-
-// TestConditionalReadsOfNamedVersion verifies that a conditional GET or HEAD of an
-// explicit versionId is evaluated against that version rather than the latest one,
-// including when the latest version is a delete marker.
-func TestConditionalReadsOfNamedVersion(t *testing.T) {
-	client := getS3Client(t)
-	bucketName := getNewBucketName()
-
-	createBucket(t, client, bucketName)
-	defer deleteBucket(t, client, bucketName)
-	enableVersioning(t, client, bucketName)
-
-	key := "conditional-read-version"
-	v1 := putObject(t, client, bucketName, key, "content-v1")
-	require.NotNil(t, v1.ETag)
-	require.NotNil(t, v1.VersionId)
-	v2 := putObject(t, client, bucketName, key, "content-v2")
-	require.NotNil(t, v2.ETag)
-	require.NotEqual(t, *v1.ETag, *v2.ETag)
-
-	t.Run("If-Match matches the named version, not the latest", func(t *testing.T) {
-		_, err := client.GetObject(context.TODO(), &s3.GetObjectInput{
-			Bucket: aws.String(bucketName), Key: aws.String(key),
-			VersionId: v1.VersionId, IfMatch: v1.ETag,
-		})
-		require.NoError(t, err)
-	})
-
-	t.Run("If-Match against the latest ETag fails on the named version", func(t *testing.T) {
-		_, err := client.GetObject(context.TODO(), &s3.GetObjectInput{
-			Bucket: aws.String(bucketName), Key: aws.String(key),
-			VersionId: v1.VersionId, IfMatch: v2.ETag,
-		})
-		requireAPIErrorCode(t, err, "PreconditionFailed")
-	})
-
-	_, err := client.DeleteObject(context.TODO(), &s3.DeleteObjectInput{
-		Bucket: aws.String(bucketName), Key: aws.String(key),
-	})
-	require.NoError(t, err)
-
-	t.Run("named version survives a delete marker on the latest", func(t *testing.T) {
-		_, err := client.HeadObject(context.TODO(), &s3.HeadObjectInput{
-			Bucket: aws.String(bucketName), Key: aws.String(key),
-			VersionId: v1.VersionId, IfMatch: v1.ETag,
-		})
-		require.NoError(t, err)
-	})
-
-	t.Run("delete marker latest is a missing object", func(t *testing.T) {
-		_, err := client.GetObject(context.TODO(), &s3.GetObjectInput{
-			Bucket: aws.String(bucketName), Key: aws.String(key), IfMatch: v1.ETag,
-		})
-		requireAPIErrorCode(t, err, "NoSuchKey")
-	})
-}

test/s3tables/catalog_duckdb_lance/Dockerfile.seed

@@ -1,15 +0,0 @@
-# Seeds a Lance dataset so DuckDB has something to read. The catalog records
-# where a table lives and does not carry its data, so the writing half is
-# pylance; DuckDB then reads it over S3 with no catalog involved.
-FROM python:3.11-slim
-
-WORKDIR /app
-
-RUN pip install --no-cache-dir \
-    "lance-namespace==0.8.6" \
-    "pylance==10.0.0" \
-    "pyarrow==25.0.1"
-
-COPY seed_table.py /app/
-
-CMD ["python3", "/app/seed_table.py", "--help"]

test/s3tables/catalog_duckdb_lance/README.md

@@ -1,66 +0,0 @@
-# DuckDB Lance Integration Test
-
-Reads SeaweedFS Lance tables from DuckDB's `lance` core extension, the
-counterpart of the DuckDB Iceberg tests in `../catalog/`.
-
-## What this one proves that the others do not
-
-The LanceDB and Spark suites go through the catalog. DuckDB does not: it reaches
-the data over S3 with no namespace involved. That exercises the other half of the
-design — a table bucket's layout is a valid Lance dataset directory, so a table
-stays readable when the catalog is not in the path.
-
-## What it does
-
-`TestDuckDBLance`:
-
-1. Starts a `weed mini` cluster with S3 and the Lance Namespace enabled.
-2. Creates a table bucket declared `LANCE`, then declares a table through the
-   namespace and writes a dataset into it with pylance.
-3. Runs `duckdb_lance_ops.sql` in `duckdb/duckdb:latest`.
-
-| Step | What it proves |
-| --- | --- |
-| `__lance_scan(s3://…)` | DuckDB reads a table this catalog created |
-| `DESCRIBE` | the schema survived, vector column included |
-| `WHERE id < 5` | the filter path |
-| `lance_vector_search` | vector search over data behind SeaweedFS |
-| a `.lance` path | the replacement scan works on a suffixed path |
-| a suffix-less path | and does **not** see one without the suffix |
-
-## The `.lance` suffix
-
-DuckDB's replacement scan — `SELECT * FROM 's3://…'` — recognises a Lance dataset
-by a `.lance` path suffix. Tables created through this catalog deliberately have
-none: the catalog entry *is* the dataset directory, table names may not contain a
-dot, and a suffix would leak into ARNs and policies.
-
-So from DuckDB, a table in a SeaweedFS table bucket is read with
-`__lance_scan('s3://bucket/namespace/table')` rather than the bare `FROM 's3://…'`
-form. The test asserts both halves, so if the extension ever recognises a
-suffix-less directory the test fails and tells us to update the documentation.
-
-## Credentials
-
-```sql
-CREATE SECRET seaweedfs (
-    TYPE lance,
-    ACCESS_KEY_ID '…', SECRET_ACCESS_KEY '…',
-    REGION 'us-east-1',
-    ENDPOINT 'http://seaweed:8333',
-    ALLOW_HTTP true,
-    VIRTUAL_HOSTED_STYLE_REQUEST false
-);
-```
-
-Those are object_store's key names, the same as everywhere else Lance touches
-storage.
-
-## Running it
-
-    cd test/s3tables/catalog_duckdb_lance
-    (cd ../../../weed && go build .)     # the harness runs this binary
-    go test -run TestDuckDBLance -v -timeout 30m .
-
-Skipped without Docker, in `-short` mode, and if the DuckDB image cannot load the
-extension.

test/s3tables/catalog_duckdb_lance/duckdb_lance_ops.sql

@@ -1,48 +0,0 @@
--- Read a SeaweedFS Lance table from DuckDB.
---
--- The lance extension reaches the data over S3 rather than through the
--- namespace, so what is exercised here is the layout on the S3 door and the
--- credentials, not the catalog protocol. That is the point: this is the client
--- that proves a Lance table stays readable without a catalog at all.
---
--- Placeholders are substituted by the Go harness: __ENDPOINT__, __KEY__,
--- __SECRET__, __TABLE__ (s3://bucket/ns/table), __SUFFIXED__ (the same data at a
--- path ending in .lance).
-
-INSTALL lance;
-LOAD lance;
-
-CREATE SECRET seaweedfs (
-    TYPE lance,
-    ACCESS_KEY_ID '__KEY__',
-    SECRET_ACCESS_KEY '__SECRET__',
-    REGION 'us-east-1',
-    ENDPOINT '__ENDPOINT__',
-    ALLOW_HTTP true,
-    VIRTUAL_HOSTED_STYLE_REQUEST false
-);
-
--- 1. The table the catalog created, read by URI. A table bucket's layout is a
---    valid Lance dataset directory, which is what makes this possible.
-SELECT 'scan_rows=' || count(*) FROM __lance_scan('__TABLE__');
-
--- 2. Its schema survived, vector column included. Ordered, because string_agg
---    over an unordered relation is free to return the names in any order.
-SELECT 'scan_columns=' || string_agg(column_name, ',' ORDER BY column_name)
-FROM (DESCRIBE SELECT * FROM __lance_scan('__TABLE__'));
-
--- 3. A filter, so it is not only a full scan.
-SELECT 'filtered_rows=' || count(*) FROM __lance_scan('__TABLE__') WHERE id < 5;
-
--- 4. Vector search, which is what the format is for. No index is built here, so
---    this is a brute-force search; the ids nearest the query are what matters.
-SELECT 'nearest=' || string_agg(id::VARCHAR, ',' ORDER BY _distance ASC, id ASC)
-FROM lance_vector_search('__TABLE__', 'vector',
-                         [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0], k := 3);
-
--- 5. DuckDB's replacement scan recognises a path by its .lance suffix. Tables
---    this catalog creates deliberately have no suffix - the name is the table's
---    name, and a suffix would leak into ARNs and policies - so the bare
---    SELECT ... FROM 's3://…' form does not see them, and __lance_scan is the
---    way in. This asserts both halves so that a change upstream is noticed.
-SELECT 'suffixed_rows=' || count(*) FROM '__SUFFIXED__';

test/s3tables/catalog_duckdb_lance/duckdb_lance_test.go

@@ -1,390 +0,0 @@
-// Package duckdblance reads SeaweedFS Lance tables from DuckDB's lance
-// extension, the counterpart of the DuckDB Iceberg tests next door.
-//
-// DuckDB reaches the data over S3 rather than through the namespace, so what
-// this proves is the other half of the design: a table bucket's layout is a
-// valid Lance dataset directory, and a table stays readable with no catalog in
-// the path at all. It also pins the one place that costs us - DuckDB's
-// replacement scan recognises a dataset by its .lance suffix, which tables
-// created through this catalog deliberately do not have.
-package duckdblance
-
-import (
-	"context"
-	"fmt"
-	"math/rand"
-	"net/http"
-	"os"
-	"os/exec"
-	"path/filepath"
-	"strings"
-	"testing"
-	"time"
-
-	"github.com/seaweedfs/seaweedfs/test/testutil"
-)
-
-const (
-	// Tracks whatever DuckDB users are actually running; the lance extension is
-	// a core extension, so there is nothing to pin beyond the image.
-	duckDBImage = "duckdb/duckdb:latest"
-	// Seeds a dataset for DuckDB to read.
-	seedImage      = "seaweedfs-lance-seed"
-	startupTimeout = 60 * time.Second
-	clientTimeout  = 15 * time.Minute
-	seededRows     = 128
-)
-
-// TestDuckDBLance reads a table this catalog created, from DuckDB.
-func TestDuckDBLance(t *testing.T) {
-	if testing.Short() {
-		t.Skip("Skipping integration test in short mode")
-	}
-	if !hasDocker() {
-		t.Skip("Docker not available, skipping DuckDB Lance integration test")
-	}
-
-	env := newEnvironment(t)
-	defer env.cleanup()
-
-	env.start(t)
-
-	bucket := "duckdb-" + randomSuffix()
-	env.createTableBucket(t, bucket)
-
-	requireDuckDBLance(t)
-	buildSeedImage(t)
-	env.seedTable(t, bucket)
-	env.runDuckDB(t, bucket)
-}
-
-type environment struct {
-	weedBinary string
-	dataDir    string
-	bindIP     string
-
-	masterPort     int
-	masterGrpcPort int
-	volumePort     int
-	volumeGrpcPort int
-	filerPort      int
-	filerGrpcPort  int
-	s3Port         int
-	s3GrpcPort     int
-	lancePort      int
-
-	accessKey string
-	secretKey string
-
-	weedCancel context.CancelFunc
-	weedCmd    *exec.Cmd
-}
-
-func newEnvironment(t *testing.T) *environment {
-	t.Helper()
-
-	wd, err := os.Getwd()
-	if err != nil {
-		t.Fatalf("get working directory: %v", err)
-	}
-	seaweedDir := wd
-	for i := 0; i < 5; i++ {
-		if _, err := os.Stat(filepath.Join(seaweedDir, "go.mod")); err == nil {
-			break
-		}
-		seaweedDir = filepath.Dir(seaweedDir)
-	}
-
-	weedBinary := filepath.Join(seaweedDir, "weed", "weed")
-	if info, statErr := os.Stat(weedBinary); statErr == nil && !info.IsDir() {
-		// `make test` builds first; a plain `go test` will otherwise drive a
-		// binary from days ago and report a pass for code it never ran.
-		t.Logf("using %s, built %s", weedBinary, info.ModTime().Format(time.RFC3339))
-	} else {
-		weedBinary = "weed"
-		if _, err := exec.LookPath(weedBinary); err != nil {
-			t.Skip("weed binary not found, skipping integration test")
-		}
-	}
-
-	dataDir, err := os.MkdirTemp("", "seaweed-duckdb-lance-test-*")
-	if err != nil {
-		t.Fatalf("create temp dir: %v", err)
-	}
-
-	ports := testutil.MustAllocatePorts(t, 9)
-	return &environment{
-		weedBinary:     weedBinary,
-		dataDir:        dataDir,
-		bindIP:         testutil.FindBindIP(),
-		masterPort:     ports[0],
-		masterGrpcPort: ports[1],
-		volumePort:     ports[2],
-		volumeGrpcPort: ports[3],
-		filerPort:      ports[4],
-		filerGrpcPort:  ports[5],
-		s3Port:         ports[6],
-		s3GrpcPort:     ports[7],
-		lancePort:      ports[8],
-		accessKey:      "AKIAIOSFODNN7EXAMPLE",
-		secretKey:      "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
-	}
-}
-
-func (env *environment) start(t *testing.T) {
-	t.Helper()
-
-	iamConfigPath, err := testutil.WriteIAMConfig(env.dataDir, env.accessKey, env.secretKey)
-	if err != nil {
-		t.Fatalf("write IAM config: %v", err)
-	}
-
-	ctx, cancel := context.WithCancel(context.Background())
-	env.weedCancel = cancel
-
-	cmd := exec.CommandContext(ctx, env.weedBinary, "mini",
-		"-master.port", fmt.Sprintf("%d", env.masterPort),
-		"-master.port.grpc", fmt.Sprintf("%d", env.masterGrpcPort),
-		"-volume.port", fmt.Sprintf("%d", env.volumePort),
-		"-volume.port.grpc", fmt.Sprintf("%d", env.volumeGrpcPort),
-		"-filer.port", fmt.Sprintf("%d", env.filerPort),
-		"-filer.port.grpc", fmt.Sprintf("%d", env.filerGrpcPort),
-		"-s3.port", fmt.Sprintf("%d", env.s3Port),
-		"-s3.port.grpc", fmt.Sprintf("%d", env.s3GrpcPort),
-		"-s3.port.lance", fmt.Sprintf("%d", env.lancePort),
-		"-s3.config", iamConfigPath,
-		"-ip", env.bindIP,
-		"-ip.bind", "0.0.0.0",
-		"-dir", env.dataDir,
-	)
-	cmd.Dir = env.dataDir
-	cmd.Stdout = os.Stdout
-	cmd.Stderr = os.Stderr
-	cmd.Env = append(os.Environ(),
-		"AWS_ACCESS_KEY_ID="+env.accessKey,
-		"AWS_SECRET_ACCESS_KEY="+env.secretKey,
-	)
-
-	if err := cmd.Start(); err != nil {
-		t.Fatalf("start SeaweedFS: %v", err)
-	}
-	env.weedCmd = cmd
-
-	// The namespace answers /v1/table once it is serving, which is a cheaper
-	// readiness check than waiting on a bucket that does not exist yet.
-	url := fmt.Sprintf("http://%s:%d/v1/table", env.bindIP, env.lancePort)
-	if !waitForHTTP(url, startupTimeout) {
-		t.Fatalf("the Lance namespace did not become ready at %s", url)
-	}
-}
-
-// waitForHTTP polls until the URL answers at all. An auth refusal counts: it
-// means the server is up, which is the only thing being waited on.
-func waitForHTTP(url string, timeout time.Duration) bool {
-	client := &http.Client{Timeout: 2 * time.Second}
-	deadline := time.Now().Add(timeout)
-	for time.Now().Before(deadline) {
-		resp, err := client.Get(url)
-		if err != nil {
-			time.Sleep(500 * time.Millisecond)
-			continue
-		}
-		status := resp.StatusCode
-		resp.Body.Close()
-		if status < 500 {
-			return true
-		}
-		time.Sleep(500 * time.Millisecond)
-	}
-	return false
-}
-
-func (env *environment) cleanup() {
-	if env.weedCancel != nil {
-		env.weedCancel()
-	}
-	if env.weedCmd != nil {
-		_ = env.weedCmd.Wait()
-	}
-	if env.dataDir != "" {
-		_ = os.RemoveAll(env.dataDir)
-	}
-}
-
-// createTableBucket makes the bucket LanceDB will read through, declared LANCE
-// so the catalog refuses anything of another format in it.
-func (env *environment) createTableBucket(t *testing.T, bucket string) {
-	t.Helper()
-
-	ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
-	defer cancel()
-
-	cmd := exec.CommandContext(ctx, env.weedBinary, "shell",
-		fmt.Sprintf("-master=%s:%d.%d", env.bindIP, env.masterPort, env.masterGrpcPort),
-	)
-	cmd.Stdin = strings.NewReader(fmt.Sprintf(
-		"s3tables.bucket -create -name %s -format LANCE -account 000000000000\nexit\n", bucket))
-	out, err := cmd.CombinedOutput()
-	if err != nil {
-		t.Fatalf("create table bucket %s: %v\n%s", bucket, err, out)
-	}
-	// weed shell reports a command's own failure on stdout and still exits 0, so
-	// the exit code alone would let a missing bucket through and turn a setup
-	// failure into a confusing engine failure later.
-	if !env.tableBucketExists(t, bucket) {
-		t.Fatalf("table bucket %s was not created:\n%s", bucket, out)
-	}
-	t.Logf("created LANCE table bucket %s", bucket)
-}
-
-// tableBucketExists asks the namespace, which lists table buckets at its root.
-func (env *environment) tableBucketExists(t *testing.T, bucket string) bool {
-	t.Helper()
-
-	url := fmt.Sprintf("http://%s:%d/v1/namespace/%s/exists", env.bindIP, env.lancePort, bucket)
-	resp, err := http.Post(url, "application/json", strings.NewReader("{}"))
-	if err != nil {
-		t.Fatalf("ask the namespace whether %s exists: %v", bucket, err)
-	}
-	defer resp.Body.Close()
-	return resp.StatusCode == http.StatusOK
-}
-
-// runSpark runs the SQL driver inside the stock Spark image. The connector is
-// pulled from Maven at submit time, the way the Iceberg Spark suite pulls its
-// runtime, so nothing has to be built here.
-// requireDuckDBLance skips rather than fails when the image cannot load the
-// extension, so a DuckDB build without it reads as "not applicable" rather than
-// as a broken catalog.
-func requireDuckDBLance(t *testing.T) {
-	t.Helper()
-
-	const ready = "lance extension ready"
-	cmd := exec.Command("docker", "run", "--rm", "--entrypoint", "duckdb", duckDBImage,
-		"-c", fmt.Sprintf("INSTALL lance; LOAD lance; SELECT '%s' AS marker;", ready))
-	out, err := cmd.CombinedOutput()
-	if err != nil || !strings.Contains(string(out), ready) {
-		t.Skipf("DuckDB image cannot load the lance extension: %v\n%s", err, out)
-	}
-}
-
-func buildSeedImage(t *testing.T) {
-	t.Helper()
-
-	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
-	defer cancel()
-
-	cmd := exec.CommandContext(ctx, "docker", "build", "-t", seedImage, "-f", "Dockerfile.seed", ".")
-	if out, err := cmd.CombinedOutput(); err != nil {
-		t.Fatalf("build the seed image: %v\n%s", err, out)
-	}
-}
-
-// seedTable declares a table through the namespace and writes a dataset into
-// it, which is the split the catalog serves: it records where a table lives and
-// does not carry its data.
-func (env *environment) seedTable(t *testing.T, bucket string) {
-	t.Helper()
-
-	ctx, cancel := context.WithTimeout(context.Background(), clientTimeout)
-	defer cancel()
-
-	cmd := exec.CommandContext(ctx, "docker", "run", "--rm",
-		"--add-host", "host.docker.internal:host-gateway",
-		"-e", "AWS_ACCESS_KEY_ID="+env.accessKey,
-		"-e", "AWS_SECRET_ACCESS_KEY="+env.secretKey,
-		"-e", "AWS_REGION=us-east-1",
-		seedImage,
-		"python3", "/app/seed_table.py",
-		"--namespace-url", fmt.Sprintf("http://host.docker.internal:%d", env.lancePort),
-		"--s3-endpoint", fmt.Sprintf("http://host.docker.internal:%d", env.s3Port),
-		"--bucket", bucket,
-		"--rows", fmt.Sprintf("%d", seededRows),
-		"--access-key", env.accessKey,
-		"--secret-key", env.secretKey,
-	)
-	out, err := cmd.CombinedOutput()
-	t.Logf("seeder:\n%s", out)
-	if err != nil {
-		t.Fatalf("seeding the table failed: %v", err)
-	}
-}
-
-// runDuckDB substitutes the endpoint and paths into the SQL and runs it.
-func (env *environment) runDuckDB(t *testing.T, bucket string) {
-	t.Helper()
-
-	sqlBytes, err := os.ReadFile("duckdb_lance_ops.sql")
-	if err != nil {
-		t.Fatalf("read the SQL: %v", err)
-	}
-	table := fmt.Sprintf("s3://%s/ml/embeddings", bucket)
-	sql := strings.NewReplacer(
-		"__ENDPOINT__", fmt.Sprintf("http://host.docker.internal:%d", env.s3Port),
-		"__KEY__", env.accessKey,
-		"__SECRET__", env.secretKey,
-		"__TABLE__", table,
-		"__SUFFIXED__", table+"-direct.lance",
-	).Replace(string(sqlBytes))
-
-	ctx, cancel := context.WithTimeout(context.Background(), clientTimeout)
-	defer cancel()
-
-	cmd := exec.CommandContext(ctx, "docker", "run", "--rm", "-i",
-		"--add-host", "host.docker.internal:host-gateway",
-		"--entrypoint", "duckdb", duckDBImage, "-c", sql)
-	out, err := cmd.CombinedOutput()
-	t.Logf("DuckDB output:\n%s", out)
-	if err != nil {
-		t.Fatalf("the DuckDB query failed: %v", err)
-	}
-
-	output := string(out)
-	for _, want := range []string{
-		fmt.Sprintf("scan_rows=%d", seededRows),
-		"scan_columns=id,title,vector",
-		"filtered_rows=5",
-		"nearest=1,0,2",
-		fmt.Sprintf("suffixed_rows=%d", seededRows),
-	} {
-		if !strings.Contains(output, want) {
-			t.Fatalf("DuckDB did not report %q", want)
-		}
-	}
-
-	// The other half of the suffix rule: a table this catalog created has no
-	// .lance suffix, so DuckDB's replacement scan does not see it and
-	// __lance_scan is the way in. If this ever starts working, the docs saying
-	// otherwise are wrong.
-	bare := exec.CommandContext(ctx, "docker", "run", "--rm", "-i",
-		"--add-host", "host.docker.internal:host-gateway",
-		"--entrypoint", "duckdb", duckDBImage,
-		"-c", fmt.Sprintf("INSTALL lance; LOAD lance; SELECT count(*) FROM '%s';", table))
-	bareOut, bareErr := bare.CombinedOutput()
-	// DuckDB exits nonzero for any error, so the exit status alone does not tell
-	// "the replacement scan refused the path" from "the query never ran": require
-	// the catalog error either way.
-	if !strings.Contains(string(bareOut), "does not exist") {
-		t.Fatalf("a suffix-less path did not fail the way the docs say it does (%v); "+
-			"if the replacement scan now reads it, update them:\n%s", bareErr, bareOut)
-	}
-	t.Logf("a suffix-less path is not seen by the replacement scan, as expected")
-}
-
-// hasDocker reports whether a Docker daemon answers. Bounded, because an
-// unhealthy daemon makes `docker version` hang, and this runs before the test
-// has a timeout of its own: better to skip than to eat the whole budget.
-func hasDocker() bool {
-	ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
-	defer cancel()
-	return exec.CommandContext(ctx, "docker", "version").Run() == nil
-}
-
-func randomSuffix() string {
-	const charset = "abcdefghijklmnopqrstuvwxyz0123456789"
-	suffix := make([]byte, 8)
-	for i := range suffix {
-		suffix[i] = charset[rand.Intn(len(charset))]
-	}
-	return string(suffix)
-}

test/s3tables/catalog_duckdb_lance/seed_table.py

@@ -1,79 +0,0 @@
-#!/usr/bin/env python3
-"""Declare a table through the namespace and write a dataset into it.
-
-Also writes the same rows to a second path whose name ends in `.lance`, because
-DuckDB's replacement scan recognises a dataset by that suffix and tables created
-through this catalog do not have one. The test asserts both behaviours so a
-change upstream is noticed rather than silently making the docs wrong.
-"""
-
-import argparse
-import sys
-import warnings
-
-warnings.filterwarnings("ignore")
-
-import lance
-import lance_namespace as ln
-import pyarrow as pa
-
-DIM = 8
-
-
-def rows(count):
-    return pa.table(
-        {
-            "id": pa.array(list(range(count)), type=pa.int64()),
-            "title": pa.array([f"row-{i}" for i in range(count)]),
-            "vector": pa.array(
-                [[float(i) + d for d in range(DIM)] for i in range(count)],
-                type=pa.list_(pa.float32(), DIM),
-            ),
-        }
-    )
-
-
-def main():
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("--namespace-url", required=True)
-    parser.add_argument("--s3-endpoint", required=True)
-    parser.add_argument("--bucket", required=True)
-    parser.add_argument("--namespace", default="ml")
-    parser.add_argument("--table", default="embeddings")
-    parser.add_argument("--rows", type=int, default=128)
-    parser.add_argument("--access-key", default="any")
-    parser.add_argument("--secret-key", default="any")
-    args = parser.parse_args()
-
-    storage = {
-        "aws_endpoint": args.s3_endpoint,
-        "allow_http": "true",
-        "aws_access_key_id": args.access_key,
-        "aws_secret_access_key": args.secret_key,
-        "aws_region": "us-east-1",
-    }
-
-    ns = ln.connect("rest", {"uri": args.namespace_url})
-    ns.create_namespace(ln.CreateNamespaceRequest(id=[args.bucket], mode="EXIST_OK"))
-    ns.create_namespace(
-        ln.CreateNamespaceRequest(id=[args.bucket, args.namespace], mode="EXIST_OK")
-    )
-    declared = ns.declare_table(
-        ln.DeclareTableRequest(id=[args.bucket, args.namespace, args.table])
-    )
-    lance.write_dataset(rows(args.rows), declared.location, storage_options=storage,
-                        mode="overwrite")
-    print(f"seeded {args.rows} rows at {declared.location}")
-
-    # The same data under a name DuckDB's replacement scan recognises. Written
-    # directly rather than declared, because a table name containing a dot is
-    # not a valid catalog name.
-    suffixed = f"s3://{args.bucket}/{args.namespace}/{args.table}-direct.lance"
-    lance.write_dataset(rows(args.rows), suffixed, storage_options=storage,
-                        mode="overwrite")
-    print(f"seeded {args.rows} rows at {suffixed}")
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())