Compare commits

..
Author SHA1 Message Date
Chris LuandGitHub 4d0afa3286 operation: fold extra dial options into WithVolumeServerClient (#11392)
WithVolumeServerClientOptions duplicated the existing helper only to
append extra dial options. Trail the extras as a variadic parameter on
WithVolumeServerClient instead, keeping the (dialOption, fn) argument
order so all existing callers keep working. TailVolumeFromSource gets
the same treatment.
2026-09-18 13:26:31 -07:00
Chris LuandGitHub 09f835842c Merge branch 'master' into fix/volume-copy-source-guard 2026-09-18 13:24:53 -07:00
Chris Lu 934b9b4daf admin: only claim fallback master leadership on an empty raft response
A nonempty RaftListClusterServers response whose entries were all
rejected left masterMap empty, so the fallback marked the reachable
current master as leader the same way a genuinely empty (non-raft)
response does. Track whether the successful response returned zero
servers and only promote the fallback master then.
2026-09-18 12:22:39 -07:00
Chris Lu bd41ce39f7 test: opt erasure-coding loopback clusters out of the remote endpoint guard
The erasure-coding suites drive VolumeEcShardsCopy / VolumeCopy between
volume servers bound to 127.0.0.1, which the copy/tail source guard now
rejects by default. Pass -volume.allowUntrustedRemoteEndpoints to the
test volume launches, matching what the volume_server framework
harnesses already do.
2026-09-18 11:34:05 -07:00
Chris Lu 0d6024e2e0 pb: return empty server address for malformed grpc addresses
GrpcAddressToServerAddress used to return the unparseable input on a
hostAndPort failure, so a malformed raft address (e.g. "host:abc")
flowed into admin dashboard master maps unchanged. Return an empty
string instead, skip empty conversions at the two raft-cluster merge
sites, and drop the now-stale comment about the fatal exit the earlier
commit removed.
2026-09-18 11:34:05 -07:00
Chris Lu 0ae7874ed9 volume: pin validated copy/tail source addresses at dial time
validateReplicaTarget resolves the source hostname once, but the gRPC
client resolved it again at connect, leaving a DNS-rebinding window for
hostname sources. The copy and tail source dials now run through the
same guardedDialerPolicy the remote-storage path uses, so every resolved
address is re-checked against the replica deny list (private peers
allowed) immediately before the TCP connect. guardedDialerPolicy also
moves to util.OutboundDialContext so the guarded path keeps the -ip.bind
source binding the default gRPC dialer had.

The Rust volume server mirrors this with connect_guarded, a tonic
connector that resolves, re-checks each address, and connects to the
first passing IP; handlers use it whenever the untrusted-endpoint
opt-out is off. A handler-level test now exercises the enabled
validation branches for all three source-taking RPCs.
2026-09-18 11:32:28 -07:00
Chris Lu 81778defa1 rust volume: validate copy and tail source addresses before dialing
Mirror the Go guard on the Rust volume server: volume_copy,
volume_ec_shards_copy and volume_tail_receiver dial a caller-supplied
source address, so run it through validate_replica_target first (bare
host:port; no loopback, link-local or unspecified hosts; private peers
stay allowed). --volume.allowUntrustedRemoteEndpoints opts out; the test
fixture and the Rust test-cluster launcher set it so loopback sources in
tests keep working.
2026-09-18 10:57:18 -07:00
Chris Lu 39a8d3253c volume: validate copy and tail source addresses before dialing
VolumeCopy, VolumeEcShardsCopy and VolumeTailReceiver dial a
caller-supplied source address (SourceDataNode / SourceVolumeServer)
with no endpoint validation, so an anonymous caller could aim the volume
server at loopback, link-local (cloud metadata) or other unintended
destinations and read dial behavior back as a connectivity oracle.

Apply the same peer-target deny list FetchAndWriteNeedle uses for
replica targets: the source must be a bare host:port whose host is not
loopback, link-local or unspecified; cluster peers stay reachable on
private networks, and -volume.allowUntrustedRemoteEndpoints opts out.
The loopback-using copy tests set the flag to keep exercising the copy
path in process.
2026-09-18 10:57:18 -07:00
Chris Lu 47f323bbb3 pb: stop exiting the process on malformed server addresses
ServerToGrpcAddress and GrpcAddressToServerAddress called glog.Fatalf
when hostAndPort could not parse the port, which os.Exit(255)ed the whole
process. A caller-supplied copy or tail source address reached this path
synchronously in the serving goroutine, so one anonymous VolumeCopy with
a non-numeric port terminated the volume server.

Log the parse error and return the input unchanged instead: the dial or
request that consumes the address then fails as an ordinary error.
2026-09-18 10:57:14 -07:00
112 changed files with 2497 additions and 7121 deletions
+3 -3
View File
@@ -27,7 +27,7 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4.38.1
uses: github/codeql-action/init@v4.38.0
# Override language selection by uncommenting this and choosing your languages
with:
languages: go
@@ -35,7 +35,7 @@ jobs:
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below).
- name: Autobuild
uses: github/codeql-action/autobuild@v4.38.1
uses: github/codeql-action/autobuild@v4.38.0
# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -49,4 +49,4 @@ jobs:
# make release
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4.38.1
uses: github/codeql-action/analyze@v4.38.0
+1 -2
View File
@@ -6,7 +6,6 @@ on:
paths:
- 'weed/**'
- 'seaweed-volume/**'
- 'seaweed-common/**'
- 'seaweed-worker/**'
- 'docker/**'
- 'go.mod'
@@ -153,7 +152,7 @@ jobs:
org.opencontainers.image.vendor=Chris Lu
- name: Set up QEMU
uses: docker/setup-qemu-action@v4.4.0
uses: docker/setup-qemu-action@v4.3.0
- name: Create BuildKit config
run: |
@@ -129,7 +129,7 @@ jobs:
echo "seaweedfs_ref=$seaweed" >> "$GITHUB_OUTPUT"
- name: Set up QEMU
uses: docker/setup-qemu-action@v4.4.0
uses: docker/setup-qemu-action@v4.3.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
+2 -2
View File
@@ -236,7 +236,7 @@ jobs:
org.opencontainers.image.vendor=Chris Lu
- name: Set up QEMU
if: matrix.platform != 'amd64'
uses: docker/setup-qemu-action@v4.4.0
uses: docker/setup-qemu-action@v4.3.0
- name: Create BuildKit config
run: |
cat > /tmp/buildkitd.toml <<EOF
@@ -405,7 +405,7 @@ jobs:
output: trivy-results.sarif
exit-code: '0'
- name: Upload Trivy scan results to GitHub Security
uses: github/codeql-action/upload-sarif@v4.38.1
uses: github/codeql-action/upload-sarif@v4.38.0
if: always()
with:
sarif_file: trivy-results.sarif
@@ -46,7 +46,7 @@ jobs:
org.opencontainers.image.vendor=Chris Lu
-
name: Set up QEMU
uses: docker/setup-qemu-action@v4.4.0
uses: docker/setup-qemu-action@v4.3.0
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
@@ -231,7 +231,7 @@ jobs:
- name: Set up QEMU
if: (github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant) && matrix.qemu
uses: docker/setup-qemu-action@v4.4.0
uses: docker/setup-qemu-action@v4.3.0
- name: Create BuildKit config
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
@@ -456,7 +456,7 @@ jobs:
- name: Upload Trivy scan results to GitHub Security
if: always()
uses: github/codeql-action/upload-sarif@v4.38.1
uses: github/codeql-action/upload-sarif@v4.38.0
with:
sarif_file: trivy-results.sarif
category: trivy-${{ matrix.variant }}
@@ -85,7 +85,7 @@ jobs:
echo "seaweedfs_ref=$seaweed" >> "$GITHUB_OUTPUT"
- name: Set up QEMU
uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v1
uses: docker/setup-qemu-action@1f40c72289eff860ee54a304f1438e3cff362e0a # v1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v1
-14
View File
@@ -116,20 +116,6 @@ jobs:
grep -q "security-config" /tmp/security.yaml
echo "Security configuration renders correctly"
echo ""
echo "=== Testing admin.allowInsecureBind satisfies the admin auth render guard ==="
helm template test $CHART_DIR --set admin.enabled=true --set admin.allowInsecureBind=true \
> /tmp/admin-allow-insecure-bind.yaml
grep -q -- "-allowInsecureBind" /tmp/admin-allow-insecure-bind.yaml
echo "admin.allowInsecureBind renders -allowInsecureBind and passes the render guard"
if helm template test $CHART_DIR --set admin.enabled=true > /tmp/admin-no-auth.yaml 2>/tmp/admin-no-auth.err; then
echo "FAIL: admin.enabled=true with no auth configured should fail to render"
exit 1
fi
grep -q "admin.allowInsecureBind" /tmp/admin-no-auth.err
echo "admin with no auth configured still fails the render guard, and the guard mentions admin.allowInsecureBind"
echo ""
echo "=== Testing JWT expiration overrides ==="
helm template test $CHART_DIR \
-1
View File
@@ -8,7 +8,6 @@ on:
- 'go.mod'
- 'go.sum'
- 'seaweed-volume/**'
- 'seaweed-common/**'
- 'test/perf/**'
- '.github/workflows/performance.yml'
workflow_dispatch:
+4 -16
View File
@@ -5,7 +5,6 @@ on:
branches: [ master ]
paths:
- 'seaweed-volume/**'
- 'seaweed-common/**'
- 'test/volume_server/**'
- 'weed/pb/volume_server.proto'
- 'weed/pb/volume_server_pb/**'
@@ -14,7 +13,6 @@ on:
branches: [ master, main ]
paths:
- 'seaweed-volume/**'
- 'seaweed-common/**'
- 'test/volume_server/**'
- 'weed/pb/volume_server.proto'
- 'weed/pb/volume_server_pb/**'
@@ -45,7 +43,7 @@ jobs:
- name: Filter changed paths
id: filter
uses: dorny/paths-filter@v4
uses: dorny/paths-filter@v3
with:
filters: |
rust:
@@ -77,7 +75,7 @@ jobs:
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-common/Cargo.lock') }}
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
@@ -95,16 +93,6 @@ jobs:
# - name: Check formatting
# run: cd seaweed-volume && cargo fmt --check
# seaweed-common is a path dependency of this crate, not a member of its
# workspace, so the run below does not reach its own tests. It builds into
# this job's cached target directory, and the cache key above covers the
# shared crate's lock, so the aws-lc-sys that rustls pulls in is restored
# with the cache instead of compiled from scratch on every run.
- name: Run shared-crate unit tests
env:
CARGO_TARGET_DIR: ${{ github.workspace }}/seaweed-volume/target
run: cd seaweed-common && cargo test
- name: Run Rust unit tests
run: cd seaweed-volume && cargo test
@@ -180,7 +168,7 @@ jobs:
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-common/Cargo.lock') }}
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
@@ -262,7 +250,7 @@ jobs:
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-common/Cargo.lock') }}
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
+1 -14
View File
@@ -5,14 +5,12 @@ on:
branches: [ master ]
paths:
- 'seaweed-worker/**'
- 'seaweed-common/**'
- 'weed/pb/plugin.proto'
- '.github/workflows/rust-worker-tests.yml'
push:
branches: [ master, main ]
paths:
- 'seaweed-worker/**'
- 'seaweed-common/**'
- 'weed/pb/plugin.proto'
- '.github/workflows/rust-worker-tests.yml'
@@ -51,7 +49,7 @@ jobs:
~/.cargo/registry
~/.cargo/git
seaweed-worker/target/release
key: rust-worker-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-worker/Cargo.lock', 'seaweed-common/Cargo.lock') }}
key: rust-worker-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-worker/Cargo.lock') }}
restore-keys: |
rust-worker-${{ steps.toolchain.outputs.fingerprint }}-
@@ -86,17 +84,6 @@ jobs:
# - name: Check formatting
# run: cd seaweed-worker && cargo fmt --all --check
# seaweed-common is a path dependency of core and lance, not a member of
# this workspace, so `--workspace` below does not reach its own tests.
# Release and this job's cached target directory, and the cache key above
# covers the shared crate's lock. That lock pins the same rustls and
# aws-lc-sys this workspace resolves, so the release build above has
# already paid for them.
- name: Run shared-crate unit tests
env:
CARGO_TARGET_DIR: ${{ github.workspace }}/seaweed-worker/target
run: cd seaweed-common && cargo test --release
# The tests that need a live gateway skip themselves without one, the way
# the Go integration tests skip without Docker; the lifecycle suite in
# test/s3tables/lifecycle is what runs them against a real cluster.
-1
View File
@@ -5,7 +5,6 @@ on:
branches: [ master ]
paths:
- 'seaweed-volume/**'
- 'seaweed-common/**'
- '.github/workflows/rust_binaries_dev.yml'
permissions:
-106
View File
@@ -1,106 +0,0 @@
name: "Snowflake S3Compat API tests"
on:
push:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'weed/iam/**'
- 'weed/command/**'
- 'weed/storage/**'
- 'weed/operation/**'
- 'weed/wdclient/**'
- 'weed/cluster/**'
- 'weed/pb/**'
- 'test/s3/snowflake/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/s3-snowflake-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'weed/iam/**'
- 'weed/command/**'
- 'weed/storage/**'
- 'weed/operation/**'
- 'weed/wdclient/**'
- 'weed/cluster/**'
- 'weed/pb/**'
- 'test/s3/snowflake/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/s3-snowflake-tests.yml'
concurrency:
group: ${{ github.event.pull_request.number || github.ref }}/s3-snowflake-tests
cancel-in-progress: true
permissions:
contents: read
jobs:
snowflake-s3compat-tests:
name: Snowflake S3Compat API tests
runs-on: ubuntu-22.04
timeout-minutes: 30
env:
WORK_DIR: /tmp/seaweedfs-snowflake-tests
steps:
- name: Check out code
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
id: go
- name: Set up Java
uses: actions/setup-java@v6
with:
java-version: '17'
distribution: 'temurin'
cache: 'maven'
- name: Install SeaweedFS
run: |
cd weed
go install -buildvcs=false
weed version
- name: Run Snowflake S3Compat API tests
timeout-minutes: 20
run: |
# Starts weed server, creates the buckets/objects the suite needs,
# clones the upstream suite, and runs mvn -Dtest=S3CompatApiTest.
bash test/s3/snowflake/run.sh
- name: Show logs on failure
if: failure()
run: |
echo "=== SeaweedFS Server Log ==="
tail -200 "$WORK_DIR/weed.log" || echo "No server log"
echo ""
echo "=== Surefire results ==="
cat "$WORK_DIR"/snowflake-s3compat-api-test-suite/s3compatapi/target/surefire-reports/*.txt 2>/dev/null || echo "No surefire reports"
- name: Upload test results
if: always()
uses: actions/upload-artifact@v7
with:
name: snowflake-s3compat-surefire-reports
path: /tmp/seaweedfs-snowflake-tests/snowflake-s3compat-api-test-suite/s3compatapi/target/surefire-reports/
retention-days: 14
- name: Cleanup
if: always()
run: |
pkill -9 -f "weed server" || true
rm -rf "$WORK_DIR" || true
-6
View File
@@ -14,9 +14,6 @@ RUN cd /go/src/github.com/seaweedfs/seaweedfs && \
git checkout $BRANCH) || \
(echo "ERROR: Branch/commit $BRANCH not found in repository" && \
echo "Available branches:" && git branch -a && exit 1))
# seaweed-common only exists on revisions that have it; a BRANCH predating it
# still needs the directory so the COPY into rust_builder below never fails.
RUN mkdir -p /go/src/github.com/seaweedfs/seaweedfs/seaweed-common
ARG TARGETOS TARGETARCH TARGETVARIANT
RUN cd /go/src/github.com/seaweedfs/seaweedfs/weed \
&& export LDFLAGS="-X github.com/seaweedfs/seaweedfs/weed/util/version.COMMIT=$(git rev-parse --short HEAD)" \
@@ -34,9 +31,6 @@ ARG TAGS
COPY weed-volume-prebuilt/ /prebuilt/
COPY weed-worker-prebuilt/ /prebuilt-worker/
COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/seaweed-volume /build/seaweed-volume
# seaweed-common is a path dependency of seaweed-volume that lives beside it,
# so the source build below needs it in the same relative position.
COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/seaweed-common /build/seaweed-common
COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/weed /build/weed
WORKDIR /build/seaweed-volume
RUN if [ -f "/prebuilt/weed-volume-${TARGETARCH}" ]; then \
+8 -8
View File
@@ -70,7 +70,7 @@ require (
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/viper v1.21.0
github.com/stretchr/testify v1.12.1
github.com/stretchr/testify v1.11.1
github.com/stvp/tempredis v0.0.0-20181119212430-b82af8480203
github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965
github.com/tidwall/gjson v1.18.0
@@ -90,7 +90,7 @@ require (
gocloud.dev v0.46.0
gocloud.dev/pubsub/natspubsub v0.46.0
gocloud.dev/pubsub/rabbitpubsub v0.46.0
golang.org/x/crypto v0.57.0
golang.org/x/crypto v0.56.0
golang.org/x/exp v0.0.0-20260709172345-9ea1abe57597
golang.org/x/image v0.46.0
golang.org/x/net v0.58.0
@@ -111,7 +111,7 @@ require (
)
require (
cloud.google.com/go/kms v1.34.0
cloud.google.com/go/kms v1.33.0
github.com/Azure/azure-sdk-for-go/sdk/keyvault/azkeys v0.10.0
github.com/DATA-DOG/go-sqlmock v1.5.2
github.com/Jille/raft-grpc-transport v1.6.1
@@ -151,7 +151,7 @@ require (
github.com/seaweedfs/go-fuse/v2 v2.9.4
github.com/shirou/gopsutil/v4 v4.26.7
github.com/tarantool/go-option v1.1.0
github.com/tarantool/go-tarantool/v3 v3.0.2
github.com/tarantool/go-tarantool/v3 v3.0.1
github.com/testcontainers/testcontainers-go v0.44.0
github.com/tikv/client-go/v2 v2.0.7
github.com/twmb/avro v1.8.0
@@ -159,7 +159,7 @@ require (
github.com/ydb-platform/ydb-go-sdk-auth-environ v0.5.2
github.com/ydb-platform/ydb-go-sdk/v3 v3.151.1
go.etcd.io/etcd/client/pkg/v3 v3.7.1
go.uber.org/atomic v1.12.0
go.uber.org/atomic v1.11.0
golang.org/x/sync v0.23.0
golang.org/x/tools/godoc v0.1.0-deprecated
google.golang.org/grpc/security/advancedtls v1.0.0
@@ -289,7 +289,7 @@ require (
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
go.uber.org/mock v0.5.2 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/mod v0.41.0 // indirect
gonum.org/v1/gonum v0.17.0 // indirect
)
@@ -340,7 +340,7 @@ require (
github.com/aws/aws-sdk-go-v2/service/sqs v1.42.24 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.38.0 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.51.0
github.com/aws/aws-sdk-go-v2/service/sts v1.50.0
github.com/aws/smithy-go v1.28.1
github.com/boltdb/bolt v1.3.1 // indirect
github.com/bradenaw/juniper v0.15.3 // indirect
@@ -499,7 +499,7 @@ require (
go.opentelemetry.io/otel/trace v1.45.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.1 // indirect
golang.org/x/term v0.46.0
golang.org/x/term v0.45.0
golang.org/x/time v0.15.0
google.golang.org/genproto/googleapis/api v0.0.0-20260817212433-ac3dfec99bb1 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
+16 -16
View File
@@ -298,8 +298,8 @@ cloud.google.com/go/kms v1.4.0/go.mod h1:fajBHndQ+6ubNw6Ss2sSd+SWvjL26RNo/dr7uxs
cloud.google.com/go/kms v1.5.0/go.mod h1:QJS2YY0eJGBg3mnDfuaCyLauWwBJiHRboYxJ++1xJNg=
cloud.google.com/go/kms v1.6.0/go.mod h1:Jjy850yySiasBUDi6KFUwUv2n1+o7QZFyuUJg6OgjA0=
cloud.google.com/go/kms v1.9.0/go.mod h1:qb1tPTgfF9RQP8e1wq4cLFErVuTJv7UsSC915J8dh3w=
cloud.google.com/go/kms v1.34.0 h1:mxWcXEiyjxwFH5gclulLx+B8Y2OEpKJRZ5FOF78c2XE=
cloud.google.com/go/kms v1.34.0/go.mod h1:FbxZWUiihmyjxlaBha84OK5+fmJHPrS6F5/mBFdJk6A=
cloud.google.com/go/kms v1.33.0 h1:pG0X78m212b2pv9N4fdMoUO69LuZGQ9kSvn8sHBOFAo=
cloud.google.com/go/kms v1.33.0/go.mod h1:CSGvW6GnMQbY+1nOHcIzhMtHSbExXlOmCKjWtYVjcpA=
cloud.google.com/go/language v1.4.0/go.mod h1:F9dRpNFQmJbkaop6g0JhSBXCNlO90e1KWx5iDdxbWic=
cloud.google.com/go/language v1.6.0/go.mod h1:6dJ8t3B+lUYfStgls25GusK04NLh3eDLQnWM3mdEbhI=
cloud.google.com/go/language v1.7.0/go.mod h1:DJ6dYN/W+SQOjF8e1hLQXMF21AkH2w9wiPzPCJa2MIE=
@@ -750,8 +750,8 @@ github.com/aws/aws-sdk-go-v2/service/sso v1.38.0 h1:JGeeBcMlhg1xtOXYpeCaTQBZObtX
github.com/aws/aws-sdk-go-v2/service/sso v1.38.0/go.mod h1:XwteswG9EOMRFm73UT0t+MbTwyLxMrEXkU6e+v92Lzo=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.0 h1:obhahQXDEdVEv8y5bTKXR30LVaxYe1kyYM0L7l2Iq+k=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.0/go.mod h1:6twZZ/aXHNy1vXUO8koUbp++MYzMASkOgEBdkbJYmO0=
github.com/aws/aws-sdk-go-v2/service/sts v1.51.0 h1:Zpnqa6XtrNzXZnwbdCqHOXpXhMsa01ql/pcRQ1sb4hk=
github.com/aws/aws-sdk-go-v2/service/sts v1.51.0/go.mod h1:/8JRcdTt//hG0Q4BTmGbuOplT7ABe+5rdtqUHqXvYIM=
github.com/aws/aws-sdk-go-v2/service/sts v1.50.0 h1:khXV3+K5D3f4e8xtplaRdSFn1bEg3gj5EBHQvbCOZbQ=
github.com/aws/aws-sdk-go-v2/service/sts v1.50.0/go.mod h1:/8JRcdTt//hG0Q4BTmGbuOplT7ABe+5rdtqUHqXvYIM=
github.com/aws/smithy-go v1.28.1 h1:R/nXH00c8qcfCzQVELtRw+eLQWtzv+VAIEFJ1/xxXlQ=
github.com/aws/smithy-go v1.28.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
@@ -1897,8 +1897,8 @@ github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/stvp/tempredis v0.0.0-20181119212430-b82af8480203 h1:QVqDTf3h2WHt08YuiTGPZLls0Wq99X9bWd0Q5ZSBesM=
github.com/stvp/tempredis v0.0.0-20181119212430-b82af8480203/go.mod h1:oqN97ltKNihBbwlX8dLpwxCl3+HnXKV/R0e+sRLd9C8=
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
@@ -1918,8 +1918,8 @@ github.com/tarantool/go-iproto v1.1.0 h1:HULVOIHsiehI+FnHfM7wMDntuzUddO09DKqu2Wn
github.com/tarantool/go-iproto v1.1.0/go.mod h1:LNCtdyZxojUed8SbOiYHoc3v9NvaZTB7p96hUySMlIo=
github.com/tarantool/go-option v1.1.0 h1:ShoOhNsdL41sRpm4hXCRDjV8H0WzPkd4UnKhLKbW//w=
github.com/tarantool/go-option v1.1.0/go.mod h1:hMr9z2JXOWlgdCBpCPSL2nwp8718GKYvNBJ+ZuzJbCo=
github.com/tarantool/go-tarantool/v3 v3.0.2 h1:9ZtHllun80QX7KS9tqd3dXa+QAu2BfqFtX1ZWcNnaW8=
github.com/tarantool/go-tarantool/v3 v3.0.2/go.mod h1:TXxLWhUCgdxXFfelnTSkq+goKRTRj660zxq4/WXPe8k=
github.com/tarantool/go-tarantool/v3 v3.0.1 h1:vaUX4xmVmXh2dIJ/LqlX1MXK3iYqAqV6YiE54Wwl/qg=
github.com/tarantool/go-tarantool/v3 v3.0.1/go.mod h1:TXxLWhUCgdxXFfelnTSkq+goKRTRj660zxq4/WXPe8k=
github.com/testcontainers/testcontainers-go v0.44.0 h1:/Fwh6HY1mIikhnm9e7HwoxGycx0lzRAE0f5VQpjFxzI=
github.com/testcontainers/testcontainers-go v0.44.0/go.mod h1:IcnwQrYTO86xHXu5bvMaBH7ATlbS3Qn1M1QWW3c66rE=
github.com/testcontainers/testcontainers-go/modules/compose v0.44.0 h1:8YcW51jhgpkkiRVe10Wj9TCBthJmoNpU2fK5WSf7TQ8=
@@ -2145,8 +2145,8 @@ go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT7
go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
go.uber.org/atomic v1.12.0 h1:BvcXdFKuviU4fTL/f+SxdQ5qJX/Jix8pAkgdUcb3XOE=
go.uber.org/atomic v1.12.0/go.mod h1:I6c4cg+6HCxRjfjSsYtApoFILnpc0CGUdGkXVqbYVNk=
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
go.uber.org/goleak v1.1.10/go.mod h1:8a7PlsEVH3e/a/GLqe5IIrQx6GzcnRmZEufDUTk4A7A=
go.uber.org/goleak v1.1.12/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
@@ -2163,8 +2163,8 @@ go.uber.org/zap v1.27.1 h1:08RqriUEv8+ArZRYSTXy1LeBScaMpVSTBhCeaZYfMYc=
go.uber.org/zap v1.27.1/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v4 v4.0.0-rc.6 h1:1h7H1ohdUh93/FyE4YaDa1Zh64K6VVbjF4K6WUxMtH4=
go.yaml.in/yaml/v4 v4.0.0-rc.6/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
gocloud.dev v0.46.0 h1:niIuZwSjMtBx8K+ITB2s5kZullB13PGOS2ZoQPZxQ4Q=
@@ -2193,8 +2193,8 @@ golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58
golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU=
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf4=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/crypto v0.56.0 h1:GUh5Ii4J5jtcseSMiRqr1jXCNHoxjeV9Fmekc2oLy6Y=
golang.org/x/crypto v0.56.0/go.mod h1:OMW5y6CY9l38uPLmxU6l6pwcXp1obtLo3e6gT7gQR2I=
golang.org/x/exp v0.0.0-20180321215751-8460e604b9de/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20180807140117-3d87b88a115f/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
@@ -2491,8 +2491,8 @@ golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
golang.org/x/term v0.13.0/go.mod h1:LTmsnFJwVN6bCy1rVCoS+qHT1HhALEFxKncY3WNNh4U=
golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+1 -4
View File
@@ -376,10 +376,7 @@ start on a non-loopback address without `-adminPassword`, so the chart fails at
render time if `admin.ip` is non-loopback and authentication is not configured via
`admin.secret.adminPassword`, `admin.secret.existingSecret`, or
`WEED_ADMIN_PASSWORD` supplied through `admin.extraEnvironmentVars` /
`admin.secretExtraEnvironmentVars`. Setting `admin.allowInsecureBind` renders
`-allowInsecureBind` and bypasses this guard; it leaves the admin API
unauthenticated on the network, so use it only when access is otherwise
restricted (e.g. network policies). The whole `127.0.0.0/8` range and `::1` are
`admin.secretExtraEnvironmentVars`. The whole `127.0.0.0/8` range and `::1` are
treated as loopback (matching `weed admin`); `localhost` is treated as
non-loopback. Set `admin.ip` to a loopback address only if you also replace the
httpGet probes (e.g. with an `exec` probe that checks `127.0.0.1`).
@@ -9,7 +9,7 @@
{{- $adminAuthEnabled := include "seaweedfs.admin.authEnabled" . }}
{{- $adminIp := .Values.admin.ip | default "0.0.0.0" }}
{{- if and (not (include "seaweedfs.admin.isLoopbackIp" $adminIp)) (ne $adminAuthEnabled "true") }}
{{- fail (printf "admin.ip is set to %q (non-loopback) but admin authentication is not configured. Since `weed admin` 4.46 refuses to bind a non-loopback address without authentication, the admin container would exit on startup. Set admin.secret.adminPassword or admin.secret.existingSecret, or supply WEED_ADMIN_PASSWORD via admin.extraEnvironmentVars / admin.secretExtraEnvironmentVars, or set admin.ip to a loopback address such as 127.0.0.1 (note: a loopback bind makes the chart's httpGet readiness/liveness probes fail), or set admin.allowInsecureBind to true to opt out via -allowInsecureBind (INSECURE: exposes the admin API unauthenticated on the network)." $adminIp) -}}
{{- fail (printf "admin.ip is set to %q (non-loopback) but admin authentication is not configured. Since `weed admin` 4.46 refuses to bind a non-loopback address without authentication, the admin container would exit on startup. Set admin.secret.adminPassword or admin.secret.existingSecret, or supply WEED_ADMIN_PASSWORD via admin.extraEnvironmentVars / admin.secretExtraEnvironmentVars, or set admin.ip to a loopback address such as 127.0.0.1 (note: a loopback bind makes the chart's httpGet readiness/liveness probes fail)." $adminIp) -}}
{{- end }}
apiVersion: apps/v1
kind: StatefulSet
@@ -176,17 +176,14 @@ spec:
-dataDir={{ .Values.admin.dataDir }} \
{{- end }}
{{- if .Values.admin.masters }}
-masters={{ .Values.admin.masters }} \
-masters={{ .Values.admin.masters }}{{- if or $urlPrefix .Values.admin.extraArgs }} \{{ end }}
{{- else if .Values.global.seaweedfs.masterServer }}
-masters={{ .Values.global.seaweedfs.masterServer }} \
-masters={{ .Values.global.seaweedfs.masterServer }}{{- if or $urlPrefix .Values.admin.extraArgs }} \{{ end }}
{{- else }}
-masters={{ range $index := until (.Values.master.replicas | int) }}${SEAWEEDFS_FULLNAME}-master-{{ $index }}.${SEAWEEDFS_FULLNAME}-master.{{ $.Release.Namespace }}:{{ $.Values.master.port }}{{ if lt $index (sub ($.Values.master.replicas | int) 1) }},{{ end }}{{ end }} \
-masters={{ range $index := until (.Values.master.replicas | int) }}${SEAWEEDFS_FULLNAME}-master-{{ $index }}.${SEAWEEDFS_FULLNAME}-master.{{ $.Release.Namespace }}:{{ $.Values.master.port }}{{ if lt $index (sub ($.Values.master.replicas | int) 1) }},{{ end }}{{ end }}{{- if or $urlPrefix .Values.admin.extraArgs }} \{{ end }}
{{- end }}
{{- if $urlPrefix }}
-urlPrefix={{ $urlPrefix }} \
{{- end }}
{{- if .Values.admin.allowInsecureBind }}
-allowInsecureBind \
-urlPrefix={{ $urlPrefix }}{{- if .Values.admin.extraArgs }} \{{ end }}
{{- end }}
{{- range $index, $arg := .Values.admin.extraArgs }}
{{ $arg }}{{- if lt $index (sub (len $.Values.admin.extraArgs) 1) }} \{{ end }}
@@ -105,13 +105,13 @@ true
{{- end -}}
{{- end -}}
{{/* Whether the admin non-loopback bind guard is satisfied: admin.secret
(adminPassword or existingSecret), WEED_ADMIN_PASSWORD via
extraEnvironmentVars / secretExtraEnvironmentVars, or
admin.allowInsecureBind. A secret-backed entry counts as enabled even
though the chart cannot read its value. */}}
{{/* Whether admin authentication is enabled from any supported source:
admin.secret (adminPassword or existingSecret), or WEED_ADMIN_PASSWORD
supplied via extraEnvironmentVars / secretExtraEnvironmentVars (which
weed admin picks up through viper's AutomaticEnv). A secret-backed
entry counts as enabled even though the chart cannot read its value. */}}
{{- define "seaweedfs.admin.authEnabled" -}}
{{- if or .Values.admin.secret.existingSecret .Values.admin.secret.adminPassword .Values.admin.allowInsecureBind -}}
{{- if or .Values.admin.secret.existingSecret .Values.admin.secret.adminPassword -}}
true
{{- else -}}
{{- $merged := dict -}}
+3 -7
View File
@@ -1339,11 +1339,10 @@ admin:
# kubelet's httpGet readiness/liveness probes (which dial the pod IP) to ever
# succeed. "0.0.0.0" restores the pre-4.46 behaviour of listening on all
# interfaces. A non-loopback address requires authentication: set
# admin.secret.adminPassword or admin.secret.existingSecret, supply
# admin.secret.adminPassword or admin.secret.existingSecret, or supply
# WEED_ADMIN_PASSWORD via admin.extraEnvironmentVars /
# admin.secretExtraEnvironmentVars, or opt out with admin.allowInsecureBind;
# otherwise the admin container will exit with a clear error rather than
# silently staying unready. The whole
# admin.secretExtraEnvironmentVars; otherwise the admin container will exit
# with a clear error rather than silently staying unready. The whole
# 127.0.0.0/8 range and ::1 are treated as loopback (matching weed admin).
# Set to a loopback address only if you also replace the httpGet probes.
# Note: the -ip flag requires SeaweedFS 4.46 or newer; pinning
@@ -1351,9 +1350,6 @@ admin:
ip: "0.0.0.0"
loggingOverrideLevel: null
# INSECURE: allow binding a non-loopback ip without authentication.
allowInsecureBind: false
# Admin authentication
secret:
# Name of an existing secret containing admin credentials. If set, adminUser and adminPassword below are ignored.
+1026 -1016
View File
File diff suppressed because it is too large Load Diff

Before

Width:  |  Height:  |  Size: 53 KiB

After

Width:  |  Height:  |  Size: 53 KiB

-293
View File
@@ -1,293 +0,0 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "aws-lc-rs"
version = "1.18.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ce2b2dcc879c3bae0d371e77c99f2238400ef24ec001394befa67b6e543add9e"
dependencies = [
"aws-lc-sys",
"zeroize",
]
[[package]]
name = "aws-lc-sys"
version = "0.44.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f09fae7be8bb3174e05c6afdb34199e6dc0c7c04ba9fa237b1967adfbde27483"
dependencies = [
"cc",
"cmake",
"dunce",
"fs_extra",
"pkg-config",
]
[[package]]
name = "cc"
version = "1.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "509591b7bcd67f4ef775afad7662703b4935daaa6ec0e5605cfb1090b32a2b6d"
dependencies = [
"find-msvc-tools",
"jobserver",
"libc",
"shlex",
]
[[package]]
name = "cfg-if"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "cmake"
version = "0.1.58"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678"
dependencies = [
"cc",
]
[[package]]
name = "dunce"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813"
[[package]]
name = "find-msvc-tools"
version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d45db016d36b838f563236e9193d0ee6ce38f3f68b6c94e914b4929c96bbb890"
[[package]]
name = "fs_extra"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
[[package]]
name = "getrandom"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
dependencies = [
"cfg-if",
"libc",
"wasi",
]
[[package]]
name = "getrandom"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099"
dependencies = [
"cfg-if",
"libc",
"r-efi",
]
[[package]]
name = "jobserver"
version = "0.1.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3"
dependencies = [
"getrandom 0.4.3",
"libc",
]
[[package]]
name = "libc"
version = "0.2.189"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"
[[package]]
name = "log"
version = "0.4.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"
[[package]]
name = "once_cell"
version = "1.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
[[package]]
name = "pkg-config"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
[[package]]
name = "r-efi"
version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
[[package]]
name = "ring"
version = "0.17.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
dependencies = [
"cc",
"cfg-if",
"getrandom 0.2.17",
"libc",
"untrusted",
"windows-sys",
]
[[package]]
name = "rustls"
version = "0.23.43"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0283386ce02abc0151e1761d08802dfe86c173b0b494af5cbc086574e453da06"
dependencies = [
"aws-lc-rs",
"log",
"once_cell",
"rustls-pki-types",
"rustls-webpki",
"subtle",
"zeroize",
]
[[package]]
name = "rustls-pki-types"
version = "1.15.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96"
dependencies = [
"zeroize",
]
[[package]]
name = "rustls-webpki"
version = "0.103.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0527518605e68109d875e248ea259b6758801cf165e4b2c2733ae3b51f12535a"
dependencies = [
"aws-lc-rs",
"ring",
"rustls-pki-types",
"untrusted",
]
[[package]]
name = "seaweed-common"
version = "0.1.0"
dependencies = [
"rustls",
]
[[package]]
name = "shlex"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
[[package]]
name = "subtle"
version = "2.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
[[package]]
name = "untrusted"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
[[package]]
name = "wasi"
version = "0.11.1+wasi-snapshot-preview1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
[[package]]
name = "windows-sys"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
dependencies = [
"windows-targets",
]
[[package]]
name = "windows-targets"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
dependencies = [
"windows_aarch64_gnullvm",
"windows_aarch64_msvc",
"windows_i686_gnu",
"windows_i686_gnullvm",
"windows_i686_msvc",
"windows_x86_64_gnu",
"windows_x86_64_gnullvm",
"windows_x86_64_msvc",
]
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
[[package]]
name = "windows_aarch64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
[[package]]
name = "windows_i686_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
[[package]]
name = "windows_i686_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
[[package]]
name = "windows_i686_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
[[package]]
name = "windows_x86_64_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
[[package]]
name = "windows_x86_64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
[[package]]
name = "zeroize"
version = "1.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e"
-27
View File
@@ -1,27 +0,0 @@
[package]
name = "seaweed-common"
version = "0.1.0"
edition = "2024"
# The lower of the two consumers' floors (seaweed-volume 1.91.1,
# seaweed-worker 1.94.1), so depending on this crate cannot raise either
# tree's MSRV. Verified with `cargo +1.91.1 check --all-targets`.
rust-version = "1.91.1"
description = "Helpers shared by the SeaweedFS Rust volume server and the Rust plugin workers"
# There is no root manifest: seaweed-volume and seaweed-worker are separate
# cargo trees with their own lockfiles, and this crate is a path dependency of
# both rather than a member of either. Keeping the lint policy identical in all
# three manifests is what stops them drifting.
[lints.clippy]
# Protobuf message literals keep `..Default::default()` on purpose: it is
# what lets a proto gain a field without touching every constructor.
needless_update = "allow"
[dependencies]
# The same requirement both consumers already write. Cargo unifies all
# semver-compatible `rustls = "0.23"` requirements into one crate per binary,
# which is what makes `install_default_crypto_provider` write the same
# process-wide static the consuming crate reads. rustls is already in both
# trees (the volume server directly, seaweed-worker-core through tonic's
# `tls-aws-lc`), so this adds no crate to either graph.
rustls = "0.23"
-307
View File
@@ -1,307 +0,0 @@
//! SeaweedFS server addresses, the way the Go tree does them.
//!
//! An operator gives a SeaweedFS process an HTTP address and the gRPC port is
//! derived from it rather than asked for separately: `host:port` means gRPC on
//! `port + 10000`, and the explicit `host:port.grpcPort` form names it outright.
//! Dialling the HTTP port by mistake fails as "frame with invalid size", which
//! reads like a protocol bug rather than a wrong port, so the rule is worth its
//! own module. Mirrors `pb.ServerToGrpcAddress` in
//! `weed/pb/grpc_client_server.go`.
//!
//! The volume server and the workers each had their own copy of this and the
//! copies had drifted: the worker's bracketed IPv6 literals and the volume
//! server's did not, so `::1:19333` produced `::1:29333`, which the HTTP
//! authority parser rejects. One implementation, two thin wrappers.
use std::fmt;
use std::num::ParseIntError;
/// SeaweedFS's HTTP↔gRPC port-offset convention.
pub const GRPC_PORT_OFFSET: u16 = 10000;
/// Why an address could not be turned into a gRPC address.
///
/// The `Display` text is the volume server's original wording, because its
/// `parse_grpc_address` wrapper hands it straight to callers that put it in a
/// `Status` or an `io::Error`.
#[derive(Debug, Clone, PartialEq, Eq)]
#[non_exhaustive]
pub enum AddressError {
/// No `:` at all, so there is no port to translate.
MissingPort(String),
/// The HTTP port of the `host:port.grpcPort` form is not a `u16`. It is
/// validated even though it is then discarded, so that a malformed address
/// is rejected here instead of failing later as an opaque connect error.
InvalidHttpPort { port: String, source: ParseIntError },
/// The gRPC port of the `host:port.grpcPort` form is not a `u16`.
InvalidGrpcPort { port: String, source: ParseIntError },
/// The port of the `host:port` form is not a `u16`.
InvalidPort { port: String, source: ParseIntError },
/// `port + GRPC_PORT_OFFSET` leaves the TCP port range, e.g. `host:60000`.
/// Without the check the cast would wrap silently.
ImplicitGrpcPortOutOfRange(u16),
}
impl fmt::Display for AddressError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::MissingPort(address) => write!(f, "cannot parse address: {address}"),
Self::InvalidHttpPort { port, source } => {
write!(f, "invalid http port {port:?}: {source}")
}
Self::InvalidGrpcPort { port, source } => {
write!(f, "invalid grpc port {port:?}: {source}")
}
Self::InvalidPort { port, source } => write!(f, "invalid port {port:?}: {source}"),
Self::ImplicitGrpcPortOutOfRange(port) => write!(
f,
"implicit grpc port out of range: {port} + {GRPC_PORT_OFFSET} = {}",
u32::from(*port) + u32::from(GRPC_PORT_OFFSET)
),
}
}
}
impl std::error::Error for AddressError {
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
match self {
Self::InvalidHttpPort { source, .. }
| Self::InvalidGrpcPort { source, .. }
| Self::InvalidPort { source, .. } => Some(source),
Self::MissingPort(_) | Self::ImplicitGrpcPortOutOfRange(_) => None,
}
}
}
/// Turn a SeaweedFS server address (`"host:port.grpcPort"` or `"host:port"`)
/// into the `host:grpcPort` form the endpoint builders expect.
///
/// With the trailing `.grpcPort` segment that segment *is* the gRPC port;
/// without it the gRPC port is `port + GRPC_PORT_OFFSET`. An unbracketed IPv6
/// literal comes back bracketed, because otherwise the port reads as part of
/// the address.
pub fn to_grpc_address(server: &str) -> Result<String, AddressError> {
// rfind, not find: an IPv6 literal is full of colons and the port is after
// the last one.
let colon_idx = server
.rfind(':')
.ok_or_else(|| AddressError::MissingPort(server.to_string()))?;
let host = &server[..colon_idx];
let port_part = &server[colon_idx + 1..];
// rfind again rather than split_once: the host may be an IPv4 address, and
// only the part after the last colon is being split here anyway.
if let Some(dot_idx) = port_part.rfind('.') {
let http_port = &port_part[..dot_idx];
let grpc_port = &port_part[dot_idx + 1..];
http_port
.parse::<u16>()
.map_err(|source| AddressError::InvalidHttpPort {
port: http_port.to_string(),
source,
})?;
let grpc_port =
grpc_port
.parse::<u16>()
.map_err(|source| AddressError::InvalidGrpcPort {
port: grpc_port.to_string(),
source,
})?;
return Ok(join_host_port(host, grpc_port));
}
let port: u16 = port_part
.parse()
.map_err(|source| AddressError::InvalidPort {
port: port_part.to_string(),
source,
})?;
let grpc_port = port
.checked_add(GRPC_PORT_OFFSET)
.ok_or(AddressError::ImplicitGrpcPortOutOfRange(port))?;
Ok(join_host_port(host, grpc_port))
}
/// Join a host and a port, bracketing an IPv6 literal that is not bracketed
/// already. Public because the address rule is not the only place that has to
/// put a host and a port back together.
pub fn join_host_port(host: &str, port: u16) -> String {
// An IPv6 literal has to keep its brackets or the port reads as part of it.
if host.contains(':') && !host.starts_with('[') {
format!("[{host}]:{port}")
} else {
format!("{host}:{port}")
}
}
#[cfg(test)]
mod tests {
use super::{AddressError, GRPC_PORT_OFFSET, join_host_port, to_grpc_address};
// ---- the volume server's cases -------------------------------------
#[test]
fn dotted_form_states_the_grpc_port() {
assert_eq!(
to_grpc_address("127.0.0.1:8080.18080").unwrap(),
"127.0.0.1:18080"
);
assert_eq!(
to_grpc_address("192.168.1.66:8080.18080").unwrap(),
"192.168.1.66:18080"
);
}
#[test]
fn implicit_form_adds_the_offset() {
assert_eq!(
to_grpc_address("127.0.0.1:8080").unwrap(),
"127.0.0.1:18080"
);
assert_eq!(
to_grpc_address("192.168.1.66:8080").unwrap(),
"192.168.1.66:18080"
);
assert_eq!(
to_grpc_address("localhost:9333").unwrap(),
"localhost:19333"
);
}
#[test]
fn the_dotted_grpc_port_comes_back_normalised() {
// The volume server's copy validated this segment as a u16 and then
// emitted the original text, so a padded or signed port produced an
// authority the URI parser rejects. The parsed value is emitted now.
assert_eq!(to_grpc_address("host:8080.018080").unwrap(), "host:18080");
assert_eq!(to_grpc_address("host:8080.+18080").unwrap(), "host:18080");
}
#[test]
fn an_ipv4_host_is_not_confused_with_the_dotted_port() {
// Regression: a naive split on '.' breaks on IP addresses.
assert_eq!(
to_grpc_address("10.0.0.1:8080.18080").unwrap(),
"10.0.0.1:18080"
);
assert_eq!(to_grpc_address("10.0.0.1:8080").unwrap(), "10.0.0.1:18080");
}
#[test]
fn rejects_a_non_numeric_http_port_in_the_dotted_form() {
let err = to_grpc_address("host:abc.18080").unwrap_err();
assert!(
matches!(err, AddressError::InvalidHttpPort { .. }),
"{err:?}"
);
assert!(err.to_string().contains("invalid http port"), "{err}");
}
#[test]
fn rejects_a_non_numeric_grpc_port_in_the_dotted_form() {
let err = to_grpc_address("host:8080.xyz").unwrap_err();
assert!(
matches!(err, AddressError::InvalidGrpcPort { .. }),
"{err:?}"
);
assert!(err.to_string().contains("invalid grpc port"), "{err}");
}
#[test]
fn rejects_an_implicit_port_that_leaves_the_tcp_range() {
let err = to_grpc_address("127.0.0.1:60000").unwrap_err();
assert!(
matches!(err, AddressError::ImplicitGrpcPortOutOfRange(60000)),
"{err:?}"
);
assert!(err.to_string().contains("out of range"), "{err}");
}
#[test]
fn the_messages_are_the_volume_servers_wording_verbatim() {
// parse_grpc_address hands these straight to callers that put them in a
// Status or an io::Error, so the whole string is the contract, not just
// the substring the older tests match on. Only the two variants whose
// text is entirely ours are pinned exactly; the other three end in a
// std ParseIntError message, which is std's to reword.
assert_eq!(
to_grpc_address("127.0.0.1:60000").unwrap_err().to_string(),
"implicit grpc port out of range: 60000 + 10000 = 70000"
);
assert_eq!(
to_grpc_address("hostname").unwrap_err().to_string(),
"cannot parse address: hostname"
);
}
#[test]
fn rejects_an_address_without_a_port() {
for source in ["hostname", "no-colon", "localhost"] {
let err = to_grpc_address(source).unwrap_err();
assert!(matches!(err, AddressError::MissingPort(_)), "{err:?}");
assert!(err.to_string().contains("cannot parse"), "{err}");
}
}
// ---- the worker's cases --------------------------------------------
#[test]
fn derives_the_grpc_port() {
assert_eq!(
to_grpc_address("localhost:23646").unwrap(),
"localhost:33646"
);
assert_eq!(
to_grpc_address("127.0.0.1:9333").unwrap(),
"127.0.0.1:19333"
);
}
#[test]
fn honours_an_explicit_grpc_port() {
assert_eq!(
to_grpc_address("localhost:23646.33999").unwrap(),
"localhost:33999"
);
}
#[test]
fn rejects_what_it_cannot_parse() {
let err = to_grpc_address("localhost:notaport").unwrap_err();
assert!(matches!(err, AddressError::InvalidPort { .. }), "{err:?}");
assert!(err.to_string().contains("invalid port"), "{err}");
}
// ---- IPv6, which only the worker's copy handled --------------------
#[test]
fn brackets_ipv6_literals() {
assert_eq!(to_grpc_address("::1:23646").unwrap(), "[::1]:33646");
assert_eq!(to_grpc_address("::1:9333").unwrap(), "[::1]:19333");
assert_eq!(
to_grpc_address("fe80::1:9333.19333").unwrap(),
"[fe80::1]:19333"
);
}
#[test]
fn leaves_an_already_bracketed_literal_alone() {
assert_eq!(to_grpc_address("[::1]:9333").unwrap(), "[::1]:19333");
assert_eq!(to_grpc_address("[::1]:9333.19333").unwrap(), "[::1]:19333");
}
#[test]
fn join_host_port_brackets_only_unbracketed_literals() {
assert_eq!(join_host_port("127.0.0.1", 19333), "127.0.0.1:19333");
assert_eq!(join_host_port("localhost", 19333), "localhost:19333");
assert_eq!(join_host_port("::1", 19333), "[::1]:19333");
assert_eq!(join_host_port("[::1]", 19333), "[::1]:19333");
}
// ---- the offset itself ---------------------------------------------
#[test]
fn the_offset_is_the_seaweedfs_convention() {
assert_eq!(GRPC_PORT_OFFSET, 10000);
}
}
-11
View File
@@ -1,11 +0,0 @@
//! Helpers the SeaweedFS Rust volume server and the Rust plugin workers both need.
//!
//! `seaweed-volume` and `seaweed-worker` are separate cargo trees with separate
//! lockfiles and no root manifest, so anything both of them need was, until this
//! crate existed, written twice. The two things in here are the ones where a
//! second copy is a correctness risk rather than a typing cost: the HTTP↔gRPC
//! address rule, which two copies had already drifted on, and the process-wide
//! rustls provider, which only works if every binary installs the same one.
pub mod address;
pub mod tls;
-28
View File
@@ -1,28 +0,0 @@
//! The process-wide rustls crypto provider.
//!
//! Both binaries link aws-lc-rs and ring transitively — in the volume server
//! through the AWS SDK and reqwest, in the lance worker through lance's `aws`
//! backend and reqwest — so rustls cannot auto-select a provider and tonic's
//! client TLS panics on first use. Each binary has to pin one, and it has to be
//! the same one, which is why the choice lives here rather than in either tree.
use rustls::crypto::aws_lc_rs;
/// Pin rustls's process-wide default provider to aws-lc-rs, matching the
/// volume server's TLS config. Idempotent: the first call wins and every
/// later one is a no-op, so callers do not have to coordinate.
pub fn install_default_crypto_provider() {
let _ = aws_lc_rs::default_provider().install_default();
}
#[cfg(test)]
mod tests {
use super::install_default_crypto_provider;
#[test]
fn installing_is_idempotent_and_leaves_a_default_behind() {
install_default_crypto_provider();
install_default_crypto_provider();
assert!(rustls::crypto::CryptoProvider::get_default().is_some());
}
}
-8
View File
@@ -3487,13 +3487,6 @@ version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
[[package]]
name = "seaweed-common"
version = "0.1.0"
dependencies = [
"rustls",
]
[[package]]
name = "sec1"
version = "0.3.0"
@@ -4591,7 +4584,6 @@ dependencies = [
"rustls",
"rustls-pemfile",
"rusty-leveldb",
"seaweed-common",
"serde",
"serde_json",
"serde_urlencoded",
-7
View File
@@ -27,15 +27,8 @@ redb-experimental-cursor = ["redb/experimental_cursor"]
# Protobuf message literals keep `..Default::default()` on purpose: it is
# what lets a proto gain a field without touching every constructor.
needless_update = "allow"
# Every `unsafe` block states its precondition, right above the block.
undocumented_unsafe_blocks = "warn"
[dependencies]
# Helpers the Rust plugin workers (seaweed-worker) need as well. A path
# dependency because the two trees are separate cargo workspaces with no
# common root manifest.
seaweed-common = { path = "../seaweed-common" }
# Async runtime
tokio = { version = "1", features = ["full"] }
tokio-stream = { version = "0.1", features = ["net"] }
+203 -143
View File
@@ -1,31 +1,9 @@
use clap::Parser;
use std::ffi::OsString;
use std::net::UdpSocket;
use std::path::{Path, PathBuf};
use crate::security::tls::TlsPolicy;
/// How this module reads the environment.
///
/// Production passes [`process_env`]; tests pass a map. Injecting the lookup
/// is what lets the tests cover `HOME` and the `WEED_*` overrides without
/// `std::env::set_var`, which is `unsafe` in Rust 2024 because it races with
/// every other thread in the process — and the lib test binary runs tokio
/// runtimes, TLS stacks and AWS clients that read the environment lazily on
/// threads no test mutex can reach.
type EnvLookup<'a> = &'a dyn Fn(&str) -> Option<OsString>;
/// The real process environment, as an [`EnvLookup`].
fn process_env(key: &str) -> Option<OsString> {
std::env::var_os(key)
}
/// `std::env::var`'s view of an [`EnvLookup`]: unset and non-UTF-8 both read
/// as absent, exactly as `std::env::var(key).ok()` does.
fn env_string(env: EnvLookup<'_>, key: &str) -> Option<String> {
env(key).and_then(|value| value.into_string().ok())
}
/// SeaweedFS Volume Server (Rust implementation)
///
/// Start a volume server to provide storage spaces.
@@ -356,7 +334,7 @@ pub fn parse_cli() -> VolumeServerConfig {
let normalized = normalize_args_vec(args);
let merged = merge_options_file(normalized);
let cli = Cli::parse_from(merged);
resolve_config_with_env(cli, &process_env)
resolve_config(cli)
}
/// Find `-options`/`--options` in args, parse the referenced file, and inject
@@ -606,7 +584,7 @@ fn parse_volume_tags(tags_arg: &str, folder_count: usize) -> Vec<Vec<String>> {
folder_tags
}
fn resolve_config_with_env(cli: Cli, env: EnvLookup<'_>) -> VolumeServerConfig {
fn resolve_config(cli: Cli) -> VolumeServerConfig {
// Backward compatibility: --mserver overrides --master
let master_string = if !cli.mserver.is_empty() {
&cli.mserver
@@ -761,7 +739,7 @@ fn resolve_config_with_env(cli: Cli, env: EnvLookup<'_>) -> VolumeServerConfig {
};
// Parse security config from TOML file
let sec = parse_security_config_with_env(&cli.security_file, env);
let sec = parse_security_config(&cli.security_file);
// Parse whitelist: merge CLI --whiteList with guard.white_list from security.toml
let mut white_list: Vec<String> = cli
@@ -837,8 +815,9 @@ fn resolve_config_with_env(cli: Cli, env: EnvLookup<'_>) -> VolumeServerConfig {
grpc_allowed_wildcard_domain: sec.grpc_allowed_wildcard_domain,
grpc_volume_allowed_common_names: sec.grpc_volume_allowed_common_names,
tls_policy: sec.tls_policy,
enable_write_queue: env_string(env, "SEAWEED_WRITE_QUEUE")
.is_some_and(|v| v == "1" || v == "true"),
enable_write_queue: std::env::var("SEAWEED_WRITE_QUEUE")
.map(|v| v == "1" || v == "true")
.unwrap_or(false),
security_file: cli.security_file,
allow_untrusted_remote_endpoints: cli.allow_untrusted_remote_endpoints,
}
@@ -916,13 +895,9 @@ const SECURITY_CONFIG_FILE_NAME: &str = "security.toml";
/// allowed_commonNames = "volume-a.internal,volume-b.internal"
/// ```
pub fn parse_security_config(path: &str) -> SecurityConfig {
parse_security_config_with_env(path, &process_env)
}
fn parse_security_config_with_env(path: &str, env: EnvLookup<'_>) -> SecurityConfig {
let Some(config_path) = resolve_security_config_path(path, env) else {
let Some(config_path) = resolve_security_config_path(path) else {
let mut cfg = SecurityConfig::default();
apply_env_overrides(&mut cfg, env);
apply_env_overrides(&mut cfg);
return cfg;
};
@@ -930,7 +905,7 @@ fn parse_security_config_with_env(path: &str, env: EnvLookup<'_>) -> SecurityCon
Ok(c) => c,
Err(_) => {
let mut cfg = SecurityConfig::default();
apply_env_overrides(&mut cfg, env);
apply_env_overrides(&mut cfg);
return cfg;
}
};
@@ -1086,19 +1061,19 @@ fn parse_security_config_with_env(path: &str, env: EnvLookup<'_>) -> SecurityCon
// Override with WEED_ environment variables (matches Go's Viper convention:
// prefix WEED_, uppercase, replace . with _).
// e.g. WEED_JWT_SIGNING_KEY overrides [jwt.signing] key
apply_env_overrides(&mut cfg, env);
apply_env_overrides(&mut cfg);
cfg
}
fn resolve_security_config_path(path: &str, env: EnvLookup<'_>) -> Option<PathBuf> {
fn resolve_security_config_path(path: &str) -> Option<PathBuf> {
if !path.is_empty() {
return Some(PathBuf::from(path));
}
default_security_config_candidates(
std::env::current_dir().ok().as_deref(),
home_dir_from_env(env).as_deref(),
home_dir_from_env().as_deref(),
)
.into_iter()
.find(|candidate| candidate.is_file())
@@ -1120,12 +1095,12 @@ fn default_security_config_candidates(
candidates
}
fn home_dir_from_env(env: EnvLookup<'_>) -> Option<PathBuf> {
env("HOME")
fn home_dir_from_env() -> Option<PathBuf> {
std::env::var_os("HOME")
.filter(|v| !v.is_empty())
.map(PathBuf::from)
.or_else(|| {
env("USERPROFILE")
std::env::var_os("USERPROFILE")
.filter(|v| !v.is_empty())
.map(PathBuf::from)
})
@@ -1133,80 +1108,80 @@ fn home_dir_from_env(env: EnvLookup<'_>) -> Option<PathBuf> {
/// Apply WEED_ environment variable overrides to a SecurityConfig.
/// Matches Go's Viper convention: WEED_ prefix, uppercase, dots replaced with underscores.
fn apply_env_overrides(cfg: &mut SecurityConfig, env: EnvLookup<'_>) {
if let Some(v) = env_string(env, "WEED_JWT_SIGNING_KEY") {
fn apply_env_overrides(cfg: &mut SecurityConfig) {
if let Ok(v) = std::env::var("WEED_JWT_SIGNING_KEY") {
cfg.jwt_signing_key = v.into_bytes();
}
if let Some(v) = env_string(env, "WEED_JWT_SIGNING_EXPIRES_AFTER_SECONDS") {
if let Ok(v) = std::env::var("WEED_JWT_SIGNING_EXPIRES_AFTER_SECONDS") {
cfg.jwt_signing_expires = v.parse().unwrap_or(cfg.jwt_signing_expires);
}
if let Some(v) = env_string(env, "WEED_JWT_SIGNING_READ_KEY") {
if let Ok(v) = std::env::var("WEED_JWT_SIGNING_READ_KEY") {
cfg.jwt_read_signing_key = v.into_bytes();
}
if let Some(v) = env_string(env, "WEED_JWT_SIGNING_READ_EXPIRES_AFTER_SECONDS") {
if let Ok(v) = std::env::var("WEED_JWT_SIGNING_READ_EXPIRES_AFTER_SECONDS") {
cfg.jwt_read_signing_expires = v.parse().unwrap_or(cfg.jwt_read_signing_expires);
}
if let Some(v) = env_string(env, "WEED_HTTPS_VOLUME_CERT") {
if let Ok(v) = std::env::var("WEED_HTTPS_VOLUME_CERT") {
cfg.https_cert_file = v;
}
if let Some(v) = env_string(env, "WEED_HTTPS_VOLUME_KEY") {
if let Ok(v) = std::env::var("WEED_HTTPS_VOLUME_KEY") {
cfg.https_key_file = v;
}
if let Some(v) = env_string(env, "WEED_HTTPS_VOLUME_CA") {
if let Ok(v) = std::env::var("WEED_HTTPS_VOLUME_CA") {
cfg.https_ca_file = v;
}
if let Some(v) = env_string(env, "WEED_HTTPS_CLIENT_ENABLED") {
if let Ok(v) = std::env::var("WEED_HTTPS_CLIENT_ENABLED") {
cfg.https_client_enabled = v == "true" || v == "1";
}
if let Some(v) = env_string(env, "WEED_HTTPS_CLIENT_CERT") {
if let Ok(v) = std::env::var("WEED_HTTPS_CLIENT_CERT") {
cfg.https_client_cert_file = v;
}
if let Some(v) = env_string(env, "WEED_HTTPS_CLIENT_KEY") {
if let Ok(v) = std::env::var("WEED_HTTPS_CLIENT_KEY") {
cfg.https_client_key_file = v;
}
if let Some(v) = env_string(env, "WEED_HTTPS_CLIENT_CA") {
if let Ok(v) = std::env::var("WEED_HTTPS_CLIENT_CA") {
cfg.https_client_ca_file = v;
}
if let Some(v) = env_string(env, "WEED_GRPC_VOLUME_CERT") {
if let Ok(v) = std::env::var("WEED_GRPC_VOLUME_CERT") {
cfg.grpc_cert_file = v;
}
if let Some(v) = env_string(env, "WEED_GRPC_VOLUME_KEY") {
if let Ok(v) = std::env::var("WEED_GRPC_VOLUME_KEY") {
cfg.grpc_key_file = v;
}
if let Some(v) = env_string(env, "WEED_GRPC_VOLUME_CLIENT_CERT") {
if let Ok(v) = std::env::var("WEED_GRPC_VOLUME_CLIENT_CERT") {
cfg.grpc_client_cert_file = v;
}
if let Some(v) = env_string(env, "WEED_GRPC_VOLUME_CLIENT_KEY") {
if let Ok(v) = std::env::var("WEED_GRPC_VOLUME_CLIENT_KEY") {
cfg.grpc_client_key_file = v;
}
if let Some(v) = env_string(env, "WEED_GRPC_CA") {
if let Ok(v) = std::env::var("WEED_GRPC_CA") {
cfg.grpc_ca_file = v;
} else if let Some(v) = env_string(env, "WEED_GRPC_VOLUME_CA") {
} else if let Ok(v) = std::env::var("WEED_GRPC_VOLUME_CA") {
cfg.grpc_ca_file = v;
}
if let Some(v) = env_string(env, "WEED_GRPC_ALLOWED_WILDCARD_DOMAIN") {
if let Ok(v) = std::env::var("WEED_GRPC_ALLOWED_WILDCARD_DOMAIN") {
cfg.grpc_allowed_wildcard_domain = v;
}
if let Some(v) = env_string(env, "WEED_GRPC_VOLUME_ALLOWED_COMMONNAMES") {
if let Ok(v) = std::env::var("WEED_GRPC_VOLUME_ALLOWED_COMMONNAMES") {
cfg.grpc_volume_allowed_common_names = v.split(',').map(|name| name.to_string()).collect();
}
if let Some(v) = env_string(env, "WEED_TLS_MIN_VERSION") {
if let Ok(v) = std::env::var("WEED_TLS_MIN_VERSION") {
cfg.tls_policy.min_version = v;
}
if let Some(v) = env_string(env, "WEED_TLS_MAX_VERSION") {
if let Ok(v) = std::env::var("WEED_TLS_MAX_VERSION") {
cfg.tls_policy.max_version = v;
}
if let Some(v) = env_string(env, "WEED_TLS_CIPHER_SUITES") {
if let Ok(v) = std::env::var("WEED_TLS_CIPHER_SUITES") {
cfg.tls_policy.cipher_suites = v;
}
if let Some(v) = env_string(env, "WEED_GUARD_WHITE_LIST") {
if let Ok(v) = std::env::var("WEED_GUARD_WHITE_LIST") {
cfg.guard_white_list = v
.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect();
}
if let Some(v) = env_string(env, "WEED_ACCESS_UI") {
if let Ok(v) = std::env::var("WEED_ACCESS_UI") {
cfg.access_ui = v == "true" || v == "1";
}
}
@@ -1227,21 +1202,41 @@ fn detect_host_address() -> String {
#[cfg(test)]
mod tests {
use super::*;
use std::collections::HashMap;
use std::ffi::OsString;
use std::sync::{Mutex, MutexGuard, OnceLock};
/// Serialises the tests that move the process working directory against
/// the ones that read it. `set_current_dir` is safe, but it is still
/// process-global: without this, a `parse_security_config("")` in one test
/// would search another test's temporary directory.
///
/// Nothing here guards the environment any more — tests inject an
/// [`EnvLookup`] instead of mutating the real one.
fn process_state_lock() -> MutexGuard<'static, ()> {
static LOCK: OnceLock<Mutex<()>> = OnceLock::new();
LOCK.get_or_init(|| Mutex::new(())).lock().unwrap()
}
// SAFETY (all env mutation in this module): `set_var`/`remove_var` are
// unsafe as of Rust 2024 because they race with concurrent readers in
// other threads. Every test that reaches these helpers holds
// `process_state_lock()` for the duration, so only one test at a time
// touches the environment and none observes another's edit.
fn with_temp_env_var<F: FnOnce()>(key: &str, value: Option<&str>, f: F) {
let previous = std::env::var_os(key);
unsafe {
match value {
Some(v) => std::env::set_var(key, v),
None => std::env::remove_var(key),
}
}
f();
restore_env_var(key, previous);
}
fn restore_env_var(key: &str, value: Option<OsString>) {
unsafe {
if let Some(value) = value {
std::env::set_var(key, value);
} else {
std::env::remove_var(key);
}
}
}
fn with_temp_current_dir<F: FnOnce()>(dir: &Path, f: F) {
let previous = std::env::current_dir().unwrap();
std::env::set_current_dir(dir).unwrap();
@@ -1249,14 +1244,51 @@ mod tests {
std::env::set_current_dir(previous).unwrap();
}
/// The whole environment a test wants the config layer to see, ready to
/// pass as `&env`. `&|_| None` is an environment with nothing set at all.
fn fake_env(pairs: &[(&str, &str)]) -> impl Fn(&str) -> Option<OsString> + use<> {
let map: HashMap<String, OsString> = pairs
fn with_cleared_security_env<F: FnOnce()>(f: F) {
const KEYS: &[&str] = &[
"WEED_JWT_SIGNING_KEY",
"WEED_JWT_SIGNING_EXPIRES_AFTER_SECONDS",
"WEED_JWT_SIGNING_READ_KEY",
"WEED_JWT_SIGNING_READ_EXPIRES_AFTER_SECONDS",
"WEED_HTTPS_VOLUME_CERT",
"WEED_HTTPS_VOLUME_KEY",
"WEED_HTTPS_VOLUME_CA",
"WEED_HTTPS_CLIENT_ENABLED",
"WEED_HTTPS_CLIENT_CERT",
"WEED_HTTPS_CLIENT_KEY",
"WEED_HTTPS_CLIENT_CA",
"WEED_GRPC_VOLUME_CERT",
"WEED_GRPC_VOLUME_KEY",
"WEED_GRPC_VOLUME_CLIENT_CERT",
"WEED_GRPC_VOLUME_CLIENT_KEY",
"WEED_GRPC_CA",
"WEED_GRPC_VOLUME_CA",
"WEED_GRPC_ALLOWED_WILDCARD_DOMAIN",
"WEED_GRPC_VOLUME_ALLOWED_COMMONNAMES",
"WEED_TLS_MIN_VERSION",
"WEED_TLS_MAX_VERSION",
"WEED_TLS_CIPHER_SUITES",
"WEED_GUARD_WHITE_LIST",
"WEED_ACCESS_UI",
];
let previous: Vec<(&str, Option<OsString>)> = KEYS
.iter()
.map(|(key, value)| ((*key).to_string(), OsString::from(*value)))
.map(|key| (*key, std::env::var_os(key)))
.collect();
move |key| map.get(key).cloned()
for key in KEYS {
// SAFETY: as above — the caller holds `process_state_lock()`.
unsafe {
std::env::remove_var(key);
}
}
f();
for (key, value) in previous {
restore_env_var(key, value);
}
}
#[test]
@@ -1393,17 +1425,17 @@ mod tests {
#[test]
fn test_resolve_config_defaults_dir_to_platform_temp_dir() {
// With no --securityFile, the security-config search reads the process
// working directory, so this has to hold the lock against
// `with_temp_current_dir`. The environment is injected, not shared.
// resolve_config reads HOME/USERPROFILE and the WEED_* set, so it has to
// hold the same lock the mutation helpers take — a concurrent set_var
// during this read is exactly what makes those calls unsafe.
let _guard = process_state_lock();
let cfg = resolve_config_with_env(Cli::parse_from(["bin"]), &|_| None);
let cfg = resolve_config(Cli::parse_from(["bin"]));
assert_eq!(cfg.folders, vec![default_volume_dir()]);
}
#[test]
fn test_resolve_config_index_accepts_redb_and_leveldb_aliases() {
// As above: the security-config search reads the working directory.
// As above: resolve_config reads the environment.
let _guard = process_state_lock();
let pairs = [
("memory", NeedleMapKind::InMemory),
@@ -1415,14 +1447,14 @@ mod tests {
("leveldbLarge", NeedleMapKind::RedbLarge),
];
for (input, expected) in pairs {
let cfg =
resolve_config_with_env(Cli::parse_from(["bin", "--index", input]), &|_| None);
let cfg = resolve_config(Cli::parse_from(["bin", "--index", input]));
assert_eq!(cfg.index_type, expected, "input={}", input);
}
}
#[test]
fn test_parse_security_config_access_ui() {
let _guard = process_state_lock();
let tmp = tempfile::NamedTempFile::new().unwrap();
std::fs::write(
tmp.path(),
@@ -1436,9 +1468,11 @@ ui = true
)
.unwrap();
let cfg = parse_security_config_with_env(tmp.path().to_str().unwrap(), &|_| None);
assert_eq!(cfg.jwt_signing_key, b"secret");
assert!(cfg.access_ui);
with_cleared_security_env(|| {
let cfg = parse_security_config(tmp.path().to_str().unwrap());
assert_eq!(cfg.jwt_signing_key, b"secret");
assert!(cfg.access_ui);
});
}
#[test]
@@ -1455,8 +1489,10 @@ key = "cwd-secret"
.unwrap();
with_temp_current_dir(tmp.path(), || {
let cfg = parse_security_config_with_env("", &|_| None);
assert_eq!(cfg.jwt_signing_key, b"cwd-secret");
with_temp_env_var("WEED_JWT_SIGNING_KEY", None, || {
let cfg = parse_security_config("");
assert_eq!(cfg.jwt_signing_key, b"cwd-secret");
});
});
}
@@ -1476,15 +1512,19 @@ key = "home-secret"
)
.unwrap();
let env = fake_env(&[("HOME", home_dir.path().to_str().unwrap())]);
with_temp_current_dir(current_dir.path(), || {
let cfg = parse_security_config_with_env("", &env);
assert_eq!(cfg.jwt_signing_key, b"home-secret");
with_temp_env_var("WEED_JWT_SIGNING_KEY", None, || {
with_temp_env_var("HOME", Some(home_dir.path().to_str().unwrap()), || {
let cfg = parse_security_config("");
assert_eq!(cfg.jwt_signing_key, b"home-secret");
});
});
});
}
#[test]
fn test_parse_security_config_uses_grpc_root_ca() {
let _guard = process_state_lock();
let tmp = tempfile::NamedTempFile::new().unwrap();
std::fs::write(
tmp.path(),
@@ -1499,14 +1539,17 @@ key = "/etc/seaweedfs/volume-key.pem"
)
.unwrap();
let cfg = parse_security_config_with_env(tmp.path().to_str().unwrap(), &|_| None);
assert_eq!(cfg.grpc_ca_file, "/etc/seaweedfs/grpc-ca.pem");
assert_eq!(cfg.grpc_cert_file, "/etc/seaweedfs/volume-cert.pem");
assert_eq!(cfg.grpc_key_file, "/etc/seaweedfs/volume-key.pem");
with_cleared_security_env(|| {
let cfg = parse_security_config(tmp.path().to_str().unwrap());
assert_eq!(cfg.grpc_ca_file, "/etc/seaweedfs/grpc-ca.pem");
assert_eq!(cfg.grpc_cert_file, "/etc/seaweedfs/volume-cert.pem");
assert_eq!(cfg.grpc_key_file, "/etc/seaweedfs/volume-key.pem");
});
}
#[test]
fn test_parse_security_config_uses_grpc_volume_client_cert() {
let _guard = process_state_lock();
let tmp = tempfile::NamedTempFile::new().unwrap();
std::fs::write(
tmp.path(),
@@ -1520,19 +1563,22 @@ client_key = "/etc/seaweedfs/volume-client-key.pem"
)
.unwrap();
let cfg = parse_security_config_with_env(tmp.path().to_str().unwrap(), &|_| None);
assert_eq!(
cfg.grpc_client_cert_file,
"/etc/seaweedfs/volume-client-cert.pem"
);
assert_eq!(
cfg.grpc_client_key_file,
"/etc/seaweedfs/volume-client-key.pem"
);
with_cleared_security_env(|| {
let cfg = parse_security_config(tmp.path().to_str().unwrap());
assert_eq!(
cfg.grpc_client_cert_file,
"/etc/seaweedfs/volume-client-cert.pem"
);
assert_eq!(
cfg.grpc_client_key_file,
"/etc/seaweedfs/volume-client-key.pem"
);
});
}
#[test]
fn test_parse_security_config_uses_grpc_peer_name_policy() {
let _guard = process_state_lock();
let tmp = tempfile::NamedTempFile::new().unwrap();
std::fs::write(
tmp.path(),
@@ -1546,19 +1592,22 @@ allowed_commonNames = "volume-a.internal,volume-b.internal"
)
.unwrap();
let cfg = parse_security_config_with_env(tmp.path().to_str().unwrap(), &|_| None);
assert_eq!(cfg.grpc_allowed_wildcard_domain, ".example.com");
assert_eq!(
cfg.grpc_volume_allowed_common_names,
vec![
String::from("volume-a.internal"),
String::from("volume-b.internal")
]
);
with_cleared_security_env(|| {
let cfg = parse_security_config(tmp.path().to_str().unwrap());
assert_eq!(cfg.grpc_allowed_wildcard_domain, ".example.com");
assert_eq!(
cfg.grpc_volume_allowed_common_names,
vec![
String::from("volume-a.internal"),
String::from("volume-b.internal")
]
);
});
}
#[test]
fn test_parse_security_config_uses_https_client_settings() {
let _guard = process_state_lock();
let tmp = tempfile::NamedTempFile::new().unwrap();
std::fs::write(
tmp.path(),
@@ -1572,15 +1621,18 @@ ca = "/etc/seaweedfs/client-ca.pem"
)
.unwrap();
let cfg = parse_security_config_with_env(tmp.path().to_str().unwrap(), &|_| None);
assert!(cfg.https_client_enabled);
assert_eq!(cfg.https_client_cert_file, "/etc/seaweedfs/client-cert.pem");
assert_eq!(cfg.https_client_key_file, "/etc/seaweedfs/client-key.pem");
assert_eq!(cfg.https_client_ca_file, "/etc/seaweedfs/client-ca.pem");
with_cleared_security_env(|| {
let cfg = parse_security_config(tmp.path().to_str().unwrap());
assert!(cfg.https_client_enabled);
assert_eq!(cfg.https_client_cert_file, "/etc/seaweedfs/client-cert.pem");
assert_eq!(cfg.https_client_key_file, "/etc/seaweedfs/client-key.pem");
assert_eq!(cfg.https_client_ca_file, "/etc/seaweedfs/client-ca.pem");
});
}
#[test]
fn test_parse_security_config_uses_tls_policy_settings() {
let _guard = process_state_lock();
let tmp = tempfile::NamedTempFile::new().unwrap();
std::fs::write(
tmp.path(),
@@ -1593,13 +1645,15 @@ cipher_suites = "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
)
.unwrap();
let cfg = parse_security_config_with_env(tmp.path().to_str().unwrap(), &|_| None);
assert_eq!(cfg.tls_policy.min_version, "TLS 1.2");
assert_eq!(cfg.tls_policy.max_version, "TLS 1.3");
assert_eq!(
cfg.tls_policy.cipher_suites,
"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
);
with_cleared_security_env(|| {
let cfg = parse_security_config(tmp.path().to_str().unwrap());
assert_eq!(cfg.tls_policy.min_version, "TLS 1.2");
assert_eq!(cfg.tls_policy.max_version, "TLS 1.3");
assert_eq!(
cfg.tls_policy.cipher_suites,
"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
);
});
}
#[test]
@@ -1700,15 +1754,16 @@ cipher_suites = "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
#[test]
fn test_env_override_jwt_signing_key() {
// No file path, so the search reads the working directory.
let _guard = process_state_lock();
let env = fake_env(&[("WEED_JWT_SIGNING_KEY", "env-secret")]);
let cfg = parse_security_config_with_env("", &env);
assert_eq!(cfg.jwt_signing_key, b"env-secret");
with_temp_env_var("WEED_JWT_SIGNING_KEY", Some("env-secret"), || {
let cfg = parse_security_config("");
assert_eq!(cfg.jwt_signing_key, b"env-secret");
});
}
#[test]
fn test_env_override_takes_precedence_over_file() {
let _guard = process_state_lock();
let tmp = tempfile::NamedTempFile::new().unwrap();
std::fs::write(
tmp.path(),
@@ -1719,26 +1774,31 @@ key = "file-secret"
)
.unwrap();
let env = fake_env(&[("WEED_JWT_SIGNING_KEY", "env-secret")]);
let cfg = parse_security_config_with_env(tmp.path().to_str().unwrap(), &env);
assert_eq!(cfg.jwt_signing_key, b"env-secret");
with_temp_env_var("WEED_JWT_SIGNING_KEY", Some("env-secret"), || {
let cfg = parse_security_config(tmp.path().to_str().unwrap());
assert_eq!(cfg.jwt_signing_key, b"env-secret");
});
}
#[test]
fn test_env_override_guard_white_list() {
// No file path, so the search reads the working directory.
let _guard = process_state_lock();
let env = fake_env(&[("WEED_GUARD_WHITE_LIST", "10.0.0.0/8, 192.168.1.0/24")]);
let cfg = parse_security_config_with_env("", &env);
assert_eq!(cfg.guard_white_list, vec!["10.0.0.0/8", "192.168.1.0/24"]);
with_temp_env_var(
"WEED_GUARD_WHITE_LIST",
Some("10.0.0.0/8, 192.168.1.0/24"),
|| {
let cfg = parse_security_config("");
assert_eq!(cfg.guard_white_list, vec!["10.0.0.0/8", "192.168.1.0/24"]);
},
);
}
#[test]
fn test_env_override_access_ui() {
// No file path, so the search reads the working directory.
let _guard = process_state_lock();
let env = fake_env(&[("WEED_ACCESS_UI", "true")]);
let cfg = parse_security_config_with_env("", &env);
assert!(cfg.access_ui);
with_temp_env_var("WEED_ACCESS_UI", Some("true"), || {
let cfg = parse_security_config("");
assert!(cfg.access_ui);
});
}
}
+8 -8
View File
@@ -12,10 +12,7 @@ use seaweed_volume::security::tls::{
use seaweed_volume::security::{Guard, SigningKey};
#[cfg(unix)]
use seaweed_volume::server::debug::build_debug_router;
use seaweed_volume::server::grpc_client::{
GRPC_INITIAL_WINDOW_SIZE, GRPC_KEEPALIVE_INTERVAL, GRPC_KEEPALIVE_TIMEOUT,
GRPC_MAX_MESSAGE_SIZE, load_outgoing_grpc_tls,
};
use seaweed_volume::server::grpc_client::load_outgoing_grpc_tls;
use seaweed_volume::server::grpc_server::VolumeGrpcService;
#[cfg(unix)]
use seaweed_volume::server::profiling::CpuProfileSession;
@@ -34,10 +31,10 @@ type CpuProfileParam = Option<CpuProfileSession>;
#[cfg(not(unix))]
type CpuProfileParam = Option<()>;
// The two settings that only make sense for the inbound server. The rest of
// this server's HTTP/2 tuning — keepalive, window sizes, message size — is
// imported from `server::grpc_client` above, which is also what the outgoing
// clients dial with, so the two directions cannot drift apart.
const GRPC_MAX_MESSAGE_SIZE: usize = 1 << 30;
const GRPC_KEEPALIVE_INTERVAL: std::time::Duration = std::time::Duration::from_secs(60);
const GRPC_KEEPALIVE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(20);
const GRPC_INITIAL_WINDOW_SIZE: u32 = 16 * 1024 * 1024;
const GRPC_MAX_HEADER_LIST_SIZE: u32 = 8 * 1024 * 1024;
const GRPC_MAX_CONCURRENT_STREAMS: u32 = 1000;
@@ -346,6 +343,9 @@ async fn run(
pre_stop_seconds: config.pre_stop_seconds,
volume_state_notify: tokio::sync::Notify::new(),
write_queue: std::sync::OnceLock::new(),
s3_tier_registry: std::sync::RwLock::new(
seaweed_volume::remote_storage::s3_tier::S3TierRegistry::new(),
),
read_mode: config.read_mode,
allow_untrusted_remote_endpoints: config.allow_untrusted_remote_endpoints,
master_url,
+3 -1
View File
@@ -7,7 +7,9 @@ pub mod endpoint_guard;
pub mod s3;
pub mod s3_tier;
pub use endpoint_guard::{guarded_tcp_connect, validate_remote_endpoint, validate_replica_target};
pub use endpoint_guard::{
guarded_tcp_connect, validate_remote_endpoint, validate_replica_target,
};
use crate::pb::remote_pb::{RemoteConf, RemoteStorageLocation};
+4 -5
View File
@@ -120,11 +120,10 @@ impl ClientCertVerifier for CommonNameVerifier {
// aws-lc-rs and ring both get linked transitively, so rustls can't auto-select
// a provider and tonic's client TLS panics on first use. Pin the default to
// aws-lc-rs, matching the server config. Idempotent. The body lives in
// seaweed-common so this binary and the Rust plugin workers cannot end up
// installing different providers; re-exported here so callers keep their
// import path.
pub use seaweed_common::tls::install_default_crypto_provider;
// aws-lc-rs, matching the server config. Idempotent.
pub fn install_default_crypto_provider() {
let _ = aws_lc_rs::default_provider().install_default();
}
pub fn build_rustls_server_config(
cert_path: &str,
+44 -317
View File
@@ -1,42 +1,16 @@
//! Construction of the volume server's *outgoing* gRPC clients: TLS material,
//! endpoint tuning, dial bounds, and the three client constructors every call
//! site goes through.
//!
//! The keepalive, window-size and message-size constants below are shared with
//! the *inbound* server built in `main.rs`, which imports them from here rather
//! than declaring its own. Changing one therefore changes both directions at
//! once, which is deliberate: a volume server talks to its peers with the same
//! HTTP/2 settings it offers them.
use std::error::Error;
use std::fmt;
use std::time::Duration;
use hyper::http::Uri;
use tonic::service::interceptor::InterceptedService;
use tonic::transport::{Certificate, Channel, ClientTlsConfig, Endpoint, Identity};
use tonic::{Request, Status};
use crate::config::VolumeServerConfig;
use crate::pb::filer_pb::seaweed_filer_client::SeaweedFilerClient;
use crate::pb::master_pb::seaweed_client::SeaweedClient;
use crate::pb::volume_server_pb::volume_server_client::VolumeServerClient;
use crate::server::request_id::outgoing_request_id_interceptor;
pub const GRPC_MAX_MESSAGE_SIZE: usize = 1 << 30;
pub const GRPC_KEEPALIVE_INTERVAL: Duration = Duration::from_secs(60);
pub const GRPC_KEEPALIVE_TIMEOUT: Duration = Duration::from_secs(20);
pub const GRPC_INITIAL_WINDOW_SIZE: u32 = 16 * 1024 * 1024;
/// Bound on the TCP connect of every outgoing dial. `build_grpc_endpoint` is
/// private and `connect_channel` is the only way out of this module, so every
/// call site picks this up whether it thinks about timeouts or not.
///
/// It bounds the TCP handshake only — tonic hands it to
/// `HttpConnector::set_connect_timeout`. A peer that completes the handshake
/// and then stalls in the TLS or HTTP/2 exchange is not covered; callers that
/// need that bound wrap the whole dial (see `connect_ping_target`).
const GRPC_CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
const GRPC_KEEPALIVE_INTERVAL: Duration = Duration::from_secs(60);
const GRPC_KEEPALIVE_TIMEOUT: Duration = Duration::from_secs(20);
const GRPC_INITIAL_WINDOW_SIZE: u32 = 16 * 1024 * 1024;
#[derive(Clone, Debug)]
pub struct OutgoingGrpcTlsConfig {
@@ -107,7 +81,7 @@ pub fn grpc_endpoint_uri(grpc_host_port: &str, tls: Option<&OutgoingGrpcTlsConfi
format!("{}://{}", scheme, grpc_host_port)
}
fn build_grpc_endpoint(
pub fn build_grpc_endpoint(
grpc_host_port: &str,
tls: Option<&OutgoingGrpcTlsConfig>,
) -> Result<Endpoint, GrpcClientError> {
@@ -175,152 +149,6 @@ pub async fn connect_guarded(
.map_err(|e| GrpcClientError(format!("connect {} failed: {}", target, e)))
}
/// How a dial is bounded.
///
/// `connect_timeout` is handed to the TCP connector. `request_timeout` becomes
/// [`Endpoint::timeout`], which tonic installs as a `GrpcTimeout` layer in
/// front of *every* request the resulting channel carries — it is not a
/// property of one call.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct GrpcDialOptions {
/// Bound on establishing the connection to the peer.
pub connect_timeout: Duration,
/// Deadline applied to each RPC on the channel, or `None` to leave them
/// unbounded.
pub request_timeout: Option<Duration>,
}
impl GrpcDialOptions {
/// A short request/response call: connect within 5 s, answer within 10 s.
pub fn unary() -> Self {
Self {
connect_timeout: GRPC_CONNECT_TIMEOUT,
request_timeout: Some(Duration::from_secs(10)),
}
}
/// A call the peer may take a while to answer: connect within 5 s, answer
/// within 30 s.
pub fn long() -> Self {
Self {
connect_timeout: GRPC_CONNECT_TIMEOUT,
request_timeout: Some(Duration::from_secs(30)),
}
}
/// A bounded connect with no deadline on the RPCs themselves.
///
/// `request_timeout` must stay `None` here. [`Endpoint::timeout`] is not a
/// transfer budget: tonic layers it as a `GrpcTimeout` around the
/// response future, which resolves when the server's *first response
/// headers* arrive, so it bounds how long the peer may take to start
/// answering — per request, for every request the channel carries. A 10 s
/// value picked to suit one short call would therefore also be the header
/// deadline for the `VolumeCopy` that shares the dial, and a busy source
/// that takes longer than that to open its file would lose the whole copy.
/// `VolumeCopy`, `VolumeTailSender` and `VolumeEcShardsCopy` have never
/// carried one.
pub fn stream() -> Self {
Self {
connect_timeout: GRPC_CONNECT_TIMEOUT,
request_timeout: None,
}
}
}
/// Dial a peer and return a connected channel.
///
/// The error carries only the transport failure: every caller already wraps it
/// with the address and the operation it was attempting.
pub async fn connect_channel(
grpc_host_port: &str,
tls: Option<&OutgoingGrpcTlsConfig>,
opts: GrpcDialOptions,
) -> Result<Channel, GrpcClientError> {
let mut endpoint =
build_grpc_endpoint(grpc_host_port, tls)?.connect_timeout(opts.connect_timeout);
if let Some(request_timeout) = opts.request_timeout {
endpoint = endpoint.timeout(request_timeout);
}
endpoint
.connect()
.await
.map_err(|e| GrpcClientError(e.to_string()))
}
/// Dial a copy/tail source and return a connected channel, re-validating every
/// resolved address at connect time.
///
/// The guarded equivalent of [`connect_channel`]: same `opts` bounds, but the
/// dial goes through [`connect_guarded`] so a source address that passed
/// validation cannot be re-pointed by DNS between the check and the connect.
/// The bounds are applied to the endpoint *before* delegating, so the
/// `allow_untrusted` opt-out is timed too.
///
/// `target` is the caller-facing source address (the unparsed
/// `"ip:port.grpcPort"` form), which is what the guard pins against; the error
/// carries only the transport failure, as every caller already wraps it with
/// the address and the operation it was attempting.
pub async fn connect_channel_guarded(
grpc_host_port: &str,
target: &str,
tls: Option<&OutgoingGrpcTlsConfig>,
opts: GrpcDialOptions,
allow_untrusted: bool,
) -> Result<Channel, GrpcClientError> {
let mut endpoint =
build_grpc_endpoint(grpc_host_port, tls)?.connect_timeout(opts.connect_timeout);
if let Some(request_timeout) = opts.request_timeout {
endpoint = endpoint.timeout(request_timeout);
}
connect_guarded(endpoint, target, allow_untrusted).await
}
/// The outgoing request-id interceptor as a concrete type, so the client
/// aliases below can name it.
pub type RequestIdInterceptor = fn(Request<()>) -> Result<Request<()>, Status>;
/// A volume-server client with the request-id interceptor attached.
pub type VolumeServerGrpcClient =
VolumeServerClient<InterceptedService<Channel, RequestIdInterceptor>>;
/// A master client with the request-id interceptor attached.
pub type MasterGrpcClient = SeaweedClient<InterceptedService<Channel, RequestIdInterceptor>>;
/// A filer client with the request-id interceptor attached.
pub type FilerGrpcClient = SeaweedFilerClient<InterceptedService<Channel, RequestIdInterceptor>>;
/// Wrap a connected channel in a volume-server client that forwards the
/// current request id and lifts both message-size limits.
pub fn volume_server_client(channel: Channel) -> VolumeServerGrpcClient {
VolumeServerClient::with_interceptor(
channel,
outgoing_request_id_interceptor as RequestIdInterceptor,
)
.max_decoding_message_size(GRPC_MAX_MESSAGE_SIZE)
.max_encoding_message_size(GRPC_MAX_MESSAGE_SIZE)
}
/// Wrap a connected channel in a master client that forwards the current
/// request id and lifts both message-size limits.
pub fn master_client(channel: Channel) -> MasterGrpcClient {
SeaweedClient::with_interceptor(
channel,
outgoing_request_id_interceptor as RequestIdInterceptor,
)
.max_decoding_message_size(GRPC_MAX_MESSAGE_SIZE)
.max_encoding_message_size(GRPC_MAX_MESSAGE_SIZE)
}
/// Wrap a connected channel in a filer client that forwards the current
/// request id and lifts both message-size limits.
pub fn filer_client(channel: Channel) -> FilerGrpcClient {
SeaweedFilerClient::with_interceptor(
channel,
outgoing_request_id_interceptor as RequestIdInterceptor,
)
.max_decoding_message_size(GRPC_MAX_MESSAGE_SIZE)
.max_encoding_message_size(GRPC_MAX_MESSAGE_SIZE)
}
/// Parse a SeaweedFS server address (`"ip:port.grpcPort"` or
/// `"ip:port"`) into the `host:grpcPort` form `build_grpc_endpoint`
/// expects. With the trailing `.grpcPort` segment, that segment IS
@@ -330,27 +158,53 @@ pub fn filer_client(channel: Channel) -> FilerGrpcClient {
/// Shared between `grpc_server.rs` and the distributed-EC-read path
/// in `store_ec.rs` — keep this as the single source of truth so the
/// HTTP↔gRPC port translation can't drift between callers.
///
/// The rule itself lives in `seaweed_common::address`, which the Rust
/// plugin workers share; this wrapper only flattens the typed error
/// back to the `String` its callers already handle. Unbracketed IPv6
/// literals come back bracketed, which this copy used to get wrong.
pub fn parse_grpc_address(source: &str) -> Result<String, String> {
seaweed_common::address::to_grpc_address(source).map_err(|e| e.to_string())
let colon_idx = source
.rfind(':')
.ok_or_else(|| format!("cannot parse address: {}", source))?;
let host = &source[..colon_idx];
let port_part = &source[colon_idx + 1..];
if let Some(dot_idx) = port_part.rfind('.') {
// Format: "ip:port.grpcPort". Validate BOTH ports as u16
// so a malformed HTTP port (e.g. `host:abc.18080`) is
// rejected here rather than tripping a downstream
// `build_grpc_endpoint` URI parse failure with a less
// useful error.
let http_port = &port_part[..dot_idx];
let grpc_port = &port_part[dot_idx + 1..];
http_port
.parse::<u16>()
.map_err(|e| format!("invalid http port {:?}: {}", http_port, e))?;
grpc_port
.parse::<u16>()
.map_err(|e| format!("invalid grpc port {:?}: {}", grpc_port, e))?;
return Ok(format!("{}:{}", host, grpc_port));
}
// Format: "ip:port" → grpc = port + 10000. Reject inputs whose
// implicit grpc port would overflow the TCP port range (e.g.
// `host:60000` produces 70000 — invalid). Without this check
// the cast silently wraps and the endpoint call later fails
// with an opaque connection error.
let port: u16 = port_part
.parse()
.map_err(|e| format!("invalid port {:?}: {}", port_part, e))?;
let grpc_port = port as u32 + 10000;
if grpc_port > u16::MAX as u32 {
return Err(format!(
"implicit grpc port out of range: {} + 10000 = {}",
port, grpc_port
));
}
Ok(format!("{}:{}", host, grpc_port))
}
#[cfg(test)]
mod tests {
use super::{
GrpcDialOptions, build_grpc_endpoint, connect_channel, grpc_endpoint_uri,
load_outgoing_grpc_tls, volume_server_client,
};
use super::{build_grpc_endpoint, grpc_endpoint_uri, load_outgoing_grpc_tls};
use crate::config::{NeedleMapKind, ReadMode, VolumeServerConfig};
use crate::pb::volume_server_pb;
use crate::security::tls::TlsPolicy;
use crate::server::request_id::scope_request_id;
use std::sync::{Arc, Mutex};
use std::time::Duration;
const TEST_CERT_PEM: &str = "-----BEGIN CERTIFICATE-----\nMIIBPDCB76ADAgECAhRuRPQgeAu43BT/M7EfAWSdapVdYDAFBgMrZXAwFDESMBAG\nA1UEAwwJbG9jYWxob3N0MB4XDTI2MDcwNTE2MTUyOVoXDTM2MDcwMjE2MTUyOVow\nFDESMBAGA1UEAwwJbG9jYWxob3N0MCowBQYDK2VwAyEAr/3bNIFI+8V32oCiY6y+\nXRFmZpdNQ2g//VtRkT+nQg+jUzBRMB0GA1UdDgQWBBTsy9tLf1zPiXCQfgci6zNi\ndEzRSjAfBgNVHSMEGDAWgBTsy9tLf1zPiXCQfgci6zNidEzRSjAPBgNVHRMBAf8E\nBTADAQH/MAUGAytlcANBAIvsdw0IbvOBBkb9cd7BfMJfIP9pQQrAL03pCRWJFnFh\nSysaLVgFXI4T078IiaM874oO+iB+5vNbWEpc7CkGow4=\n-----END CERTIFICATE-----\n";
const TEST_KEY_PEM: &str = "-----BEGIN PRIVATE KEY-----\nMC4CAQAwBQYDK2VwBCIEIHbyn71Kk+Y7KT3sBctit7uZpErpoH6qDbFj6P8qGaZH\n-----END PRIVATE KEY-----\n";
@@ -545,131 +399,4 @@ mod tests {
let err = parse_grpc_address("hostname").unwrap_err();
assert!(err.contains("cannot parse"), "{}", err);
}
#[test]
fn test_parse_grpc_address_brackets_ipv6_literals() {
use super::parse_grpc_address;
// This used to come back as `::1:29333`, which is not a valid
// authority: `build_grpc_endpoint` reads the last colon as the port
// separator and rejects the rest.
assert_eq!(parse_grpc_address("::1:19333").unwrap(), "[::1]:29333");
assert_eq!(parse_grpc_address("::1:9333.19333").unwrap(), "[::1]:19333");
// Already bracketed, so it is left alone.
assert_eq!(parse_grpc_address("[::1]:9333").unwrap(), "[::1]:19333");
}
#[test]
fn test_build_grpc_endpoint_accepts_an_ipv6_master_address() {
use super::parse_grpc_address;
let endpoint = build_grpc_endpoint(&parse_grpc_address("::1:9333").unwrap(), None).unwrap();
assert_eq!(endpoint.uri().port_u16(), Some(19333));
}
/// A minimal HTTP/2 server that records the gRPC request headers it is
/// sent and answers every call with a trailers-only `unimplemented`. It is
/// enough to prove what a helper-built client puts on the wire, without
/// standing up the whole `VolumeServer` service behind a tonic server.
async fn serve_header_capture() -> (u16, Arc<Mutex<Option<String>>>) {
use hyper::service::service_fn;
use hyper_util::rt::{TokioExecutor, TokioIo};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = listener.local_addr().unwrap().port();
let seen: Arc<Mutex<Option<String>>> = Arc::new(Mutex::new(None));
let captured = Arc::clone(&seen);
tokio::spawn(async move {
while let Ok((stream, _)) = listener.accept().await {
let captured = Arc::clone(&captured);
tokio::spawn(async move {
let _ = hyper::server::conn::http2::Builder::new(TokioExecutor::new())
.serve_connection(
TokioIo::new(stream),
service_fn(move |req: hyper::Request<hyper::body::Incoming>| {
let captured = Arc::clone(&captured);
async move {
let value = req
.headers()
.get("x-amz-request-id")
.and_then(|v| v.to_str().ok())
.map(str::to_string);
*captured.lock().unwrap() = value;
Ok::<_, std::convert::Infallible>(
hyper::http::Response::builder()
.status(200)
.header("content-type", "application/grpc")
.header("grpc-status", "12")
.body(tonic::body::Body::empty())
.unwrap(),
)
}
}),
)
.await;
});
}
});
(port, seen)
}
#[tokio::test]
async fn test_helper_built_client_sends_the_scoped_request_id() {
let (port, seen) = serve_header_capture().await;
let channel = connect_channel(
&format!("127.0.0.1:{}", port),
None,
GrpcDialOptions::unary(),
)
.await
.expect("dial the header-capturing server");
let mut client = volume_server_client(channel);
// The interceptor has a request id to forward only inside a scope, so
// the call has to run inside one for this to test anything.
let _ = scope_request_id("REQUEST-ID-ON-THE-WIRE".to_string(), async move {
client
.ping(volume_server_pb::PingRequest {
target: String::new(),
target_type: String::new(),
})
.await
})
.await;
assert_eq!(
seen.lock().unwrap().as_deref(),
Some("REQUEST-ID-ON-THE-WIRE"),
"a client built by volume_server_client must carry the outgoing request id"
);
}
#[test]
fn test_dial_presets_match_the_call_sites_they_replace() {
assert_eq!(
GrpcDialOptions::unary().connect_timeout,
Duration::from_secs(5)
);
assert_eq!(
GrpcDialOptions::unary().request_timeout,
Some(Duration::from_secs(10))
);
assert_eq!(
GrpcDialOptions::long().connect_timeout,
Duration::from_secs(5)
);
assert_eq!(
GrpcDialOptions::long().request_timeout,
Some(Duration::from_secs(30))
);
assert_eq!(
GrpcDialOptions::stream().connect_timeout,
Duration::from_secs(5)
);
assert_eq!(
GrpcDialOptions::stream().request_timeout,
None,
"a streaming dial must not put a per-request deadline on the channel"
);
}
}
File diff suppressed because it is too large Load Diff
+74 -118
View File
@@ -15,8 +15,7 @@ use axum::http::{HeaderMap, Method, Request, StatusCode, header};
use axum::response::{IntoResponse, Response};
use serde::{Deserialize, Serialize};
use super::absolute_display_path;
use super::grpc_client::{GrpcDialOptions, connect_channel, volume_server_client};
use super::grpc_client::{GRPC_MAX_MESSAGE_SIZE, build_grpc_endpoint};
use super::volume_server::{VolumeServerState, normalize_outgoing_http_url, to_http_address};
use crate::config::ReadMode;
use crate::metrics;
@@ -541,16 +540,19 @@ async fn batch_delete_file_ids(
}
for (grpc_addr, batch) in server_to_file_ids {
// BatchDelete is unary, but `stream()` is still right: this fan-out has
// no per-request deadline today, and `unary()` would add a 10 s one.
let channel = connect_channel(
&grpc_addr,
state.outgoing_grpc_tls.as_ref(),
GrpcDialOptions::stream(),
)
.await
.map_err(|e| format!("batch delete {}: {}", grpc_addr, e))?;
let mut client = volume_server_client(channel);
let endpoint = build_grpc_endpoint(&grpc_addr, state.outgoing_grpc_tls.as_ref())
.map_err(|e| format!("batch delete {}: {}", grpc_addr, e))?;
let channel = endpoint
.connect()
.await
.map_err(|e| format!("batch delete {}: {}", grpc_addr, e))?;
let mut client =
volume_server_pb::volume_server_client::VolumeServerClient::with_interceptor(
channel,
super::request_id::outgoing_request_id_interceptor,
)
.max_decoding_message_size(GRPC_MAX_MESSAGE_SIZE)
.max_encoding_message_size(GRPC_MAX_MESSAGE_SIZE);
let response = client
.batch_delete(volume_server_pb::BatchDeleteRequest {
@@ -1351,7 +1353,9 @@ async fn get_or_head_handler_inner(
// H6: Determine Content-Type: filter application/octet-stream, use mime_guess
// For chunk manifests, skip extension-based MIME override — use stored MIME as-is (Go parity)
let content_type = if n.is_chunk_manifest() {
let content_type = if let Some(ref ct) = query.response_content_type {
Some(ct.clone())
} else if n.is_chunk_manifest() {
// Chunk manifests: use stored MIME but filter application/octet-stream (Go L334)
if !n.mime.is_empty() {
let mt = String::from_utf8_lossy(&n.mime).to_string();
@@ -1400,50 +1404,27 @@ async fn get_or_head_handler_inner(
}
}
};
// Every value below can come straight from the query string, so none of
// them may be unwrapped: `?response-cache-control=%0Aevil` decodes to a
// value with a newline, `HeaderValue::from_str` rejects it, and the unwrap
// would panic the connection task. An invalid `response-content-type`
// falls back to the needle MIME rather than dropping Content-Type.
if let Some(hval) = query
.response_content_type
.as_ref()
.and_then(|ct| ct.parse::<header::HeaderValue>().ok())
{
response_headers.insert(header::CONTENT_TYPE, hval);
} else if let Some(ref ct) = content_type
&& let Ok(hval) = ct.parse()
{
response_headers.insert(header::CONTENT_TYPE, hval);
if let Some(ref ct) = content_type {
response_headers.insert(header::CONTENT_TYPE, ct.parse().unwrap());
}
// Cache-Control override from query param
if let Some(ref cc) = query.response_cache_control
&& let Ok(hval) = cc.parse()
{
response_headers.insert(header::CACHE_CONTROL, hval);
if let Some(ref cc) = query.response_cache_control {
response_headers.insert(header::CACHE_CONTROL, cc.parse().unwrap());
}
// S3 response passthrough headers
if let Some(ref ce) = query.response_content_encoding
&& let Ok(hval) = ce.parse()
{
response_headers.insert(header::CONTENT_ENCODING, hval);
if let Some(ref ce) = query.response_content_encoding {
response_headers.insert(header::CONTENT_ENCODING, ce.parse().unwrap());
}
if let Some(ref exp) = query.response_expires
&& let Ok(hval) = exp.parse()
{
response_headers.insert(header::EXPIRES, hval);
if let Some(ref exp) = query.response_expires {
response_headers.insert(header::EXPIRES, exp.parse().unwrap());
}
if let Some(ref cl) = query.response_content_language
&& let Ok(hval) = cl.parse()
{
response_headers.insert("Content-Language", hval);
if let Some(ref cl) = query.response_content_language {
response_headers.insert("Content-Language", cl.parse().unwrap());
}
if let Some(ref cd) = query.response_content_disposition
&& let Ok(hval) = cd.parse()
{
response_headers.insert(header::CONTENT_DISPOSITION, hval);
if let Some(ref cd) = query.response_content_disposition {
response_headers.insert(header::CONTENT_DISPOSITION, cd.parse().unwrap());
}
// Last-Modified
@@ -2808,23 +2789,15 @@ pub async fn delete_handler(
{
let has_ec = state.store.read().unwrap().has_ec_volume(vid);
if has_ec {
// Step 1: Read the EC needle to get its size and validate cookie.
//
// This must go through the *distributed* reader, as the GET path
// does. The local-only `EcVolume::read_ec_shard_needle` errors
// "ec shard N not available locally" for any interval that lives on
// a peer, so on a standard 10+4 spread every HTTP DELETE of an EC
// needle failed 500 and never appended to `.ecj`. The distributed
// reader does a local-first pass in its snapshot phase, so the
// all-shards-local case costs the same as before.
//
// No store guard is held across this call: the reader takes and
// releases its own, and `RwLockReadGuard` is `!Send`.
let ec_read_result =
crate::server::store_ec::read_ec_shard_needle_distributed(&state, vid, needle_id)
.await;
// Step 1: Read the EC needle to get its size and validate cookie
let ec_read_result = {
let store = state.store.read().unwrap();
store
.find_ec_volume(vid)
.map(|ecv| ecv.read_ec_shard_needle(needle_id))
};
match ec_read_result {
Ok(Some(ec_needle)) => {
Some(Ok(Some(ec_needle))) => {
// Step 2: Validate cookie (Go: cookie != 0 && cookie != n.Cookie)
if cookie.0 != 0 && ec_needle.cookie != cookie {
return json_error_with_query(
@@ -2834,50 +2807,26 @@ pub async fn delete_handler(
);
}
let count = ec_needle.data_size as i64;
// Step 3: Journal the delete on a holder of the needle's
// primary data shard — Go's DeleteEcShardNeedle forwards a
// VolumeEcBlobDelete there rather than journaling locally,
// so exactly one node carries the tombstone and
// delete_count stays consistent across replicas.
match crate::server::store_ec::delete_ec_shard_needle_distributed(
&state, vid, needle_id,
)
.await
// Step 3: Journal the delete
let mut store = state.store.write().unwrap();
if let Some(ecv) = store.find_ec_volume_mut(vid)
&& let Err(e) = ecv.journal_delete(needle_id)
{
Ok(()) => {
let result = DeleteResult { size: count };
return json_response_with_params(
StatusCode::ACCEPTED,
&result,
Some(&del_params),
);
}
// Unmounted between the read and the append, or the
// needle went away in the same window: nothing was
// journalled, so answering 202 would lose the delete
// while reporting success.
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
let result = DeleteResult { size: 0 };
return json_response_with_params(
StatusCode::NOT_FOUND,
&result,
Some(&del_params),
);
}
Err(e) => {
return json_error_with_query(
StatusCode::INTERNAL_SERVER_ERROR,
format!("Deletion Failed: {}", e),
Some(&del_query),
);
}
return json_error_with_query(
StatusCode::INTERNAL_SERVER_ERROR,
format!("Deletion Failed: {}", e),
Some(&del_query),
);
}
let result = DeleteResult { size: count };
return json_response_with_params(
StatusCode::ACCEPTED,
&result,
Some(&del_params),
);
}
Ok(None) => {
// Needle not in the EC index, or the volume disappeared
// between the `has_ec` check and the snapshot — the
// distributed reader reports both as `Ok(None)`, and both
// mean the same thing to a deleter.
Some(Ok(None)) => {
// Needle not found in EC volume
let result = DeleteResult { size: 0 };
return json_response_with_params(
StatusCode::NOT_FOUND,
@@ -2885,25 +2834,22 @@ pub async fn delete_handler(
Some(&del_params),
);
}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
// A shard or interval the read needed is gone rather than
// merely remote. The GET path answers 404 here; answering
// 500 (as this handler did for every error) told callers to
// retry a delete that can never succeed.
let result = DeleteResult { size: 0 };
return json_response_with_params(
StatusCode::NOT_FOUND,
&result,
Some(&del_params),
);
}
Err(e) => {
Some(Err(e)) => {
return json_error_with_query(
StatusCode::INTERNAL_SERVER_ERROR,
format!("Deletion Failed: {}", e),
Some(&del_query),
);
}
None => {
// EC volume disappeared between has_ec check and find
let result = DeleteResult { size: 0 };
return json_response_with_params(
StatusCode::NOT_FOUND,
&result,
Some(&del_params),
);
}
}
}
}
@@ -3675,6 +3621,16 @@ async fn read_remote_chunk_needle(
// Helpers
// ============================================================================
fn absolute_display_path(path: &str) -> String {
let p = std::path::Path::new(path);
if p.is_absolute() {
return path.to_string();
}
std::env::current_dir()
.map(|cwd| cwd.join(p).to_string_lossy().to_string())
.unwrap_or_else(|_| path.to_string())
}
fn build_disk_statuses(store: &crate::storage::store::Store) -> Vec<serde_json::Value> {
let mut disk_statuses = Vec::new();
for loc in &store.locations {
+88 -61
View File
@@ -12,9 +12,10 @@ use std::time::{Duration, SystemTime, UNIX_EPOCH};
use tokio::sync::broadcast;
use tracing::{error, info, warn};
use super::grpc_client::{GrpcDialOptions, connect_channel, master_client};
use super::grpc_client::{GRPC_MAX_MESSAGE_SIZE, build_grpc_endpoint};
use super::volume_server::VolumeServerState;
use crate::pb::master_pb;
use crate::pb::master_pb::seaweed_client::SeaweedClient;
use crate::pb::volume_server_pb;
use crate::remote_storage::s3_tier::{S3TierBackend, S3TierConfig};
use crate::storage::store::Store;
@@ -23,6 +24,7 @@ use crate::storage::volume_report::VolumeReportKey;
use crate::storage::volume_report_hash::report_hash;
const DUPLICATE_UUID_RETRY_MESSAGE: &str = "duplicate UUIDs detected, retrying connection";
const VOLUME_IO_ERROR_TOLERANCE: i32 = 3;
const MAX_DUPLICATE_UUID_RETRIES: u32 = 3;
/// Configuration for the heartbeat client.
@@ -220,7 +222,7 @@ async fn check_with_master(config: &HeartbeatConfig, state: &Arc<VolumeServerSta
if changed {
state.metrics_notify.notify_waiters();
}
apply_storage_backends(&resp.storage_backends);
apply_storage_backends(state, &resp.storage_backends);
info!(
"Got master configuration from {}: metrics_address={}, metrics_interval={}s",
master_addr, resp.metrics_address, resp.metrics_interval_seconds
@@ -244,8 +246,17 @@ pub async fn try_get_master_configuration(
grpc_addr: &str,
tls: Option<&super::grpc_client::OutgoingGrpcTlsConfig>,
) -> Result<master_pb::GetMasterConfigurationResponse, Box<dyn std::error::Error>> {
let channel = connect_channel(grpc_addr, tls, GrpcDialOptions::unary()).await?;
let mut client = master_client(channel);
let channel = build_grpc_endpoint(grpc_addr, tls)?
.connect_timeout(Duration::from_secs(5))
.timeout(Duration::from_secs(10))
.connect()
.await?;
let mut client = SeaweedClient::with_interceptor(
channel,
super::request_id::outgoing_request_id_interceptor,
)
.max_decoding_message_size(GRPC_MAX_MESSAGE_SIZE)
.max_encoding_message_size(GRPC_MAX_MESSAGE_SIZE);
let resp = client
.get_master_configuration(master_pb::GetMasterConfigurationRequest {})
.await?;
@@ -380,14 +391,18 @@ async fn do_heartbeat(
pulse: Duration,
shutdown_rx: &mut broadcast::Receiver<()>,
) -> Result<Option<String>, Box<dyn std::error::Error>> {
let channel = connect_channel(
grpc_addr,
state.outgoing_grpc_tls.as_ref(),
GrpcDialOptions::long(),
)
.await?;
let channel = build_grpc_endpoint(grpc_addr, state.outgoing_grpc_tls.as_ref())?
.connect_timeout(Duration::from_secs(5))
.timeout(Duration::from_secs(30))
.connect()
.await?;
let mut client = master_client(channel);
let mut client = SeaweedClient::with_interceptor(
channel,
super::request_id::outgoing_request_id_interceptor,
)
.max_decoding_message_size(GRPC_MAX_MESSAGE_SIZE)
.max_encoding_message_size(GRPC_MAX_MESSAGE_SIZE);
let (tx, rx) = tokio::sync::mpsc::channel::<master_pb::Heartbeat>(32);
@@ -659,15 +674,16 @@ fn apply_metrics_push_settings(
true
}
/// Registers the master's S3 storage backends in the process-wide tier
/// registry, the single place both the tier-move handlers and `Volume` itself
/// resolve a backend from.
fn apply_storage_backends(storage_backends: &[master_pb::StorageBackend]) {
fn apply_storage_backends(
state: &VolumeServerState,
storage_backends: &[master_pb::StorageBackend],
) {
if storage_backends.is_empty() {
return;
}
let mut registry = crate::remote_storage::s3_tier::global_s3_tier_registry()
let mut registry = state.s3_tier_registry.write().unwrap();
let mut global_registry = crate::remote_storage::s3_tier::global_s3_tier_registry()
.write()
.unwrap();
for backend in storage_backends {
@@ -698,6 +714,7 @@ fn apply_storage_backends(storage_backends: &[master_pb::StorageBackend]) {
backend.id.as_str()
};
register_s3_backend(&mut registry, backend, backend_id, &config);
register_s3_backend(&mut global_registry, backend, backend_id, &config);
}
}
@@ -937,8 +954,8 @@ fn build_heartbeat_with_ec_status(
let volume_size = vol.dat_file_size().unwrap_or(0);
let mut should_delete_volume = false;
if vol.should_quarantine() {
let (_, io_count, io_quarantined) = vol.get_io_error_state();
let (_, io_count, io_quarantined) = vol.get_io_error_state();
if io_quarantined || io_count >= VOLUME_IO_ERROR_TOLERANCE {
if !io_quarantined {
vol.mark_io_quarantined();
warn!(
@@ -956,7 +973,7 @@ fn build_heartbeat_with_ec_status(
// whose .dat legitimately lives in cloud storage. Only a present .dat is
// cached for 30s; a missing one is re-checked every heartbeat so the volume
// stays suppressed until the file returns. See issues/10004
if vol.file_count() > 0 && !vol.has_remote_file() {
if vol.file_count() > 0 && !vol.has_remote_file {
const DISK_CHECK_INTERVAL_NS: i64 = 30 * 1_000_000_000;
let now_ns = SystemTime::now()
.duration_since(UNIX_EPOCH)
@@ -1233,6 +1250,7 @@ mod tests {
READ_ONLY_LABEL_NO_WRITE_CAN_DELETE, READ_ONLY_LABEL_NO_WRITE_OR_DELETE,
READ_ONLY_VOLUME_GAUGE,
};
use crate::remote_storage::s3_tier::S3TierRegistry;
use crate::security::{Guard, SigningKey};
use crate::storage::needle_map::NeedleMapKind;
use crate::storage::types::{DiskType, VolumeId};
@@ -1284,6 +1302,7 @@ mod tests {
pre_stop_seconds: 0,
volume_state_notify: tokio::sync::Notify::new(),
write_queue: std::sync::OnceLock::new(),
s3_tier_registry: std::sync::RwLock::new(S3TierRegistry::new()),
read_mode: ReadMode::Local,
allow_untrusted_remote_endpoints: false,
master_url: String::new(),
@@ -1669,9 +1688,8 @@ mod tests {
{
let (_, volume) = store.find_volume_mut(VolumeId(17)).unwrap();
volume.set_read_only().unwrap();
volume
.update_remote_files(|files| files.push(Default::default()))
.unwrap();
volume.volume_info.files.push(Default::default());
volume.refresh_remote_write_mode().unwrap();
}
let heartbeat = build_heartbeat(&test_config(), &mut store);
@@ -2036,15 +2054,15 @@ mod tests {
.unwrap();
let (_, volume) = store.find_volume_mut(VolumeId(71)).unwrap();
volume
.update_remote_files(|files| {
files.push(crate::storage::volume::PbRemoteFile {
backend_type: "s3".to_string(),
backend_id: "archive".to_string(),
key: "volumes/71.dat".to_string(),
..Default::default()
})
})
.unwrap();
.volume_info
.files
.push(crate::storage::volume::PbRemoteFile {
backend_type: "s3".to_string(),
backend_id: "archive".to_string(),
key: "volumes/71.dat".to_string(),
..Default::default()
});
volume.refresh_remote_write_mode().unwrap();
let heartbeat = build_heartbeat(&test_config(), &mut store);
@@ -2053,56 +2071,65 @@ mod tests {
assert_eq!(heartbeat.volumes[0].remote_storage_key, "volumes/71.dat");
}
// Not hermetic, and cannot be made so cheaply: `register_s3_backend` skips
// a name that is already registered, so had another test put `s3` or
// `s3.default` in the process-wide registry first, this would pass without
// proving that this call registered anything. Removing them afterwards is
// no better — unlike the tier tests' unique ids, the bare `s3` alias is the
// production one. Nothing else in the tree registers those two names.
#[test]
fn test_apply_storage_backends_registers_s3_default_aliases() {
let state = test_state_with_store(Store::new(NeedleMapKind::InMemory));
// Do not call clear() on the global registry — other tests may be
// running concurrently. Just register our entries and verify them.
apply_storage_backends(&[master_pb::StorageBackend {
r#type: "s3".to_string(),
id: "default".to_string(),
properties: std::collections::HashMap::from([
("aws_access_key_id".to_string(), "access".to_string()),
("aws_secret_access_key".to_string(), "secret".to_string()),
("bucket".to_string(), "bucket-a".to_string()),
("region".to_string(), "us-west-2".to_string()),
("endpoint".to_string(), "http://127.0.0.1:8333".to_string()),
("storage_class".to_string(), "STANDARD".to_string()),
("force_path_style".to_string(), "false".to_string()),
]),
}]);
apply_storage_backends(
&state,
&[master_pb::StorageBackend {
r#type: "s3".to_string(),
id: "default".to_string(),
properties: std::collections::HashMap::from([
("aws_access_key_id".to_string(), "access".to_string()),
("aws_secret_access_key".to_string(), "secret".to_string()),
("bucket".to_string(), "bucket-a".to_string()),
("region".to_string(), "us-west-2".to_string()),
("endpoint".to_string(), "http://127.0.0.1:8333".to_string()),
("storage_class".to_string(), "STANDARD".to_string()),
("force_path_style".to_string(), "false".to_string()),
]),
}],
);
let registry = crate::remote_storage::s3_tier::global_s3_tier_registry()
.read()
.unwrap();
let registry = state.s3_tier_registry.read().unwrap();
assert!(registry.get("s3.default").is_some());
assert!(registry.get("s3").is_some());
let global_registry = crate::remote_storage::s3_tier::global_s3_tier_registry()
.read()
.unwrap();
assert!(global_registry.get("s3.default").is_some());
assert!(global_registry.get("s3").is_some());
}
#[test]
fn test_apply_storage_backends_ignores_unsupported_types() {
let state = test_state_with_store(Store::new(NeedleMapKind::InMemory));
// Do not call clear() on the global registry — other tests may be
// running concurrently.
apply_storage_backends(&[master_pb::StorageBackend {
r#type: "rclone".to_string(),
id: "default".to_string(),
properties: std::collections::HashMap::new(),
}]);
apply_storage_backends(
&state,
&[master_pb::StorageBackend {
r#type: "rclone".to_string(),
id: "default".to_string(),
properties: std::collections::HashMap::new(),
}],
);
// The per-state registry is freshly created and should have no entries
// since "rclone" is unsupported.
let registry = state.s3_tier_registry.read().unwrap();
assert!(registry.names().is_empty());
// Only check that the unsupported type was not added to the global
// registry. Other tests may have their own entries present.
let registry = crate::remote_storage::s3_tier::global_s3_tier_registry()
let global_registry = crate::remote_storage::s3_tier::global_s3_tier_registry()
.read()
.unwrap();
assert!(registry.get("rclone.default").is_none());
assert!(registry.get("rclone").is_none());
assert!(global_registry.get("rclone.default").is_none());
assert!(global_registry.get("rclone").is_none());
}
#[test]
+9 -13
View File
@@ -36,20 +36,16 @@ pub fn collect_mem_status() -> volume_server_pb::MemStatus {
#[cfg(target_os = "linux")]
fn get_system_memory_linux() -> Option<(u64, u64)> {
// SAFETY: `libc::sysinfo` is plain data — integers and trailing padding,
// no pointers and no restricted niches — so the all-zero value is a valid
// one for the kernel to overwrite.
let mut info: libc::sysinfo = unsafe { std::mem::zeroed() };
// SAFETY: `&mut info` is a live, aligned, exclusive pointer to a
// `sysinfo` that the kernel only writes through, and its fields are read
// below only after the call reports success.
if unsafe { libc::sysinfo(&mut info) } != 0 {
return None;
unsafe {
let mut info: libc::sysinfo = std::mem::zeroed();
if libc::sysinfo(&mut info) == 0 {
let unit = info.mem_unit as u64;
let total = info.totalram as u64 * unit;
let free = info.freeram as u64 * unit;
return Some((total, free));
}
}
let unit = info.mem_unit as u64;
let total = info.totalram as u64 * unit;
let free = info.freeram as u64 * unit;
Some((total, free))
None
}
#[cfg(target_os = "linux")]
-80
View File
@@ -1,7 +1,3 @@
use tonic::Status;
use crate::storage::volume::VolumeError;
#[cfg(unix)]
pub mod debug;
pub mod grpc_client;
@@ -17,79 +13,3 @@ pub mod store_ec;
pub mod ui;
pub mod volume_server;
pub mod write_queue;
/// Map a storage error onto the gRPC code that describes it.
impl From<VolumeError> for Status {
fn from(err: VolumeError) -> Self {
let message = err.to_string();
match err {
VolumeError::NotFound | VolumeError::VolumeNotFound(_) => Status::not_found(message),
VolumeError::ReadOnly | VolumeError::NotEmpty => Status::failed_precondition(message),
VolumeError::InsufficientSpace { .. } => Status::resource_exhausted(message),
VolumeError::AlreadyExists => Status::already_exists(message),
_ => Status::internal(message),
}
}
}
/// Same mapping, with the RPC's own context prefixed (`compact volume 7: ...`).
pub fn status_with_context(context: &str, err: VolumeError) -> Status {
let status = Status::from(err);
Status::new(status.code(), format!("{context}: {}", status.message()))
}
/// Render a configured disk directory as an absolute path for display, so the
/// status JSON and the UI show the same thing for a relative `-dir`. Falls
/// back to the configured spelling when the current directory cannot be read.
pub(crate) fn absolute_display_path(path: &str) -> String {
let p = std::path::Path::new(path);
if p.is_absolute() {
return path.to_string();
}
std::env::current_dir()
.map(|cwd| cwd.join(p).to_string_lossy().to_string())
.unwrap_or_else(|_| path.to_string())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::storage::types::VolumeId;
#[test]
fn test_volume_error_maps_to_grpc_code() {
use tonic::Code;
let code = |e: VolumeError| Status::from(e).code();
assert_eq!(
code(VolumeError::VolumeNotFound(VolumeId(7))),
Code::NotFound
);
assert_eq!(code(VolumeError::NotFound), Code::NotFound);
assert_eq!(code(VolumeError::ReadOnly), Code::FailedPrecondition);
assert_eq!(
code(VolumeError::InsufficientSpace {
vid: VolumeId(7),
required: 2,
free: 1,
}),
Code::ResourceExhausted
);
assert_eq!(code(VolumeError::AlreadyExists), Code::AlreadyExists);
assert_eq!(code(VolumeError::NotInitialized), Code::Internal);
let status = status_with_context(
"compact volume 7",
VolumeError::InsufficientSpace {
vid: VolumeId(7),
required: 2,
free: 1,
},
);
assert_eq!(status.code(), Code::ResourceExhausted);
assert_eq!(
status.message(),
"compact volume 7: not enough free space: required 2, free 1"
);
}
}
+54 -274
View File
@@ -26,7 +26,7 @@
//! cache write-back briefly reacquires the EcVolume's internal
//! `RwLock` so we do not contend with the Store-level lock at all.
use std::collections::{HashMap, HashSet};
use std::collections::HashMap;
use std::fs;
use std::io;
use std::sync::Arc;
@@ -38,13 +38,12 @@ use reed_solomon_erasure::galois_8::ReedSolomon;
use tokio::sync::Semaphore;
use tonic::Request;
use crate::pb::master_pb::{self, LookupEcVolumeRequest};
use crate::pb::master_pb::{self, LookupEcVolumeRequest, seaweed_client::SeaweedClient};
use crate::pb::volume_server_pb::{
CopyFileRequest, VolumeEcBlobDeleteRequest, VolumeEcShardReadRequest,
};
use crate::server::grpc_client::{
GrpcDialOptions, connect_channel, master_client, parse_grpc_address, volume_server_client,
CopyFileRequest, VolumeEcShardReadRequest, volume_server_client::VolumeServerClient,
};
use crate::server::grpc_client::{GRPC_MAX_MESSAGE_SIZE, build_grpc_endpoint, parse_grpc_address};
use crate::server::request_id::outgoing_request_id_interceptor;
use crate::server::volume_server::{VolumeServerState, to_http_address};
use crate::storage::erasure_coding::ec_shard::{ShardId, shard_id_try_from};
use crate::storage::needle::needle::{Needle, NeedleError, get_actual_size};
@@ -255,237 +254,6 @@ pub async fn read_ec_shard_needle_distributed(
Ok(Some(n))
}
/// What one EC delete RPC carries — `VolumeEcBlobDeleteRequest` minus tonic.
struct EcDeleteTarget<'a> {
vid: VolumeId,
collection: &'a str,
version: Version,
needle_id: NeedleId,
}
/// `Store.doDeleteNeedleFromAtLeastOneRemoteEcShards` in Go: journal the
/// tombstone on one holder of the needle's primary data shard, falling back
/// to any other shard holder when the primary has none. Exactly one node
/// journals — replicas of a shard hold identical .ecx copies, so journaling
/// on more than one would double the reported delete count.
///
/// `NotFound` means the volume or needle is gone; other errors mean every
/// reachable holder failed or no shard has a holder at all.
pub async fn delete_ec_shard_needle_distributed(
state: &Arc<VolumeServerState>,
vid: VolumeId,
needle_id: NeedleId,
) -> io::Result<()> {
let (
primary_shard_id,
collection,
version,
total_shards,
local_shards,
data_shards,
encode_ts_ns,
refreshed_at,
cached_locations,
) = {
let store = state.store.read().unwrap();
let ecv = store.find_ec_volume(vid).ok_or_else(|| {
io::Error::new(
io::ErrorKind::NotFound,
format!("ec volume {} not mounted", vid.0),
)
})?;
let (_, _, intervals) = ecv.locate_needle(needle_id)?.ok_or_else(|| {
io::Error::new(
io::ErrorKind::NotFound,
format!("needle {} not in ec volume {}", needle_id, vid.0),
)
})?;
let (shard_id, _) = intervals
.first()
.map(|i| ecv.interval_to_shard_id_and_offset(i))
.ok_or_else(|| io::Error::new(io::ErrorKind::NotFound, "no intervals for needle"))?;
let (cached_locations, refreshed_at) = ecv.shard_locations_snapshot();
(
shard_id,
ecv.collection.clone(),
ecv.version,
ecv.data_shards + ecv.parity_shards,
local_shard_ids(ecv),
ecv.data_shards as usize,
ecv.encode_ts_ns,
refreshed_at,
cached_locations,
)
};
let target = EcDeleteTarget {
vid,
collection: &collection,
version,
needle_id,
};
// Holder addresses come from the same staleness-gated cache as the read
// path: a master LookupEcVolume only when due, merged on a complete reply.
let mut locations = cached_locations;
if claim_shard_locations_refresh(
state,
vid,
&locations,
refreshed_at,
data_shards,
total_shards as usize,
) {
match cached_lookup_ec_shard_locations(state, vid).await {
Ok(fresh) => {
match write_back_shard_locations(state, vid, fresh, data_shards, encode_ts_ns) {
Some(merged) => locations = merged,
None => mark_shard_locations_stale(state, vid),
}
}
Err(_) => mark_shard_locations_stale(state, vid),
}
}
match delete_on_ec_shard_holders(state, &locations, &local_shards, primary_shard_id, &target)
.await
{
Ok(true) => return Ok(()),
Err(e) => return Err(e),
Ok(false) => {}
}
for shard_id in 0..total_shards {
let Ok(shard_id) = shard_id_try_from(shard_id) else {
continue;
};
if shard_id == primary_shard_id {
continue;
}
if let Ok(true) =
delete_on_ec_shard_holders(state, &locations, &local_shards, shard_id, &target).await
{
return Ok(());
}
}
Err(io::Error::other(format!(
"ec volume {}: no shard holder could journal the delete",
vid.0
)))
}
/// `doDeleteNeedleFromRemoteEcShardServers` in Go. `Ok(false)` is the
/// shard-missing signal — no live holder anywhere — that triggers the
/// caller's fallback walk over the remaining shards.
async fn delete_on_ec_shard_holders(
state: &Arc<VolumeServerState>,
locations: &HashMap<ShardId, Vec<String>>,
local_shards: &HashSet<ShardId>,
shard_id: ShardId,
target: &EcDeleteTarget<'_>,
) -> io::Result<bool> {
let addrs = locations.get(&shard_id);
if !local_shards.contains(&shard_id) && addrs.is_none_or(|a| a.is_empty()) {
return Ok(false);
}
let mut last_err = None;
if local_shards.contains(&shard_id) {
match journal_delete_local(state, target.vid, target.needle_id) {
Ok(()) => return Ok(true),
// Nothing was committed — the volume unmounted or remounted
// without the needle — so it is safe to fall back to other
// shard holders, unlike an RPC failure which may have landed.
Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(false),
Err(e) => last_err = Some(e),
}
}
if let Some(addrs) = addrs {
let self_http = to_http_address(&state.self_url);
for addr in addrs {
// A stale self entry: the loopback RPC would journal on this same
// volume, which the local attempt above already covered.
if to_http_address(addr).as_ref() == self_http.as_ref() {
continue;
}
match delete_on_remote_ec_shard(state, addr, target).await {
Ok(()) => return Ok(true),
Err(e) => last_err = Some(e),
}
}
}
match last_err {
Some(e) => Err(e),
None => Ok(false),
}
}
/// `doDeleteNeedleFromRemoteEcShard` in Go — one `VolumeEcBlobDelete` RPC.
async fn delete_on_remote_ec_shard(
state: &Arc<VolumeServerState>,
addr: &str,
target: &EcDeleteTarget<'_>,
) -> io::Result<()> {
let grpc_addr =
parse_grpc_address(addr).map_err(|e| io::Error::new(io::ErrorKind::InvalidInput, e))?;
let channel = connect_channel(
&grpc_addr,
state.outgoing_grpc_tls.as_ref(),
GrpcDialOptions::unary(),
)
.await
.map_err(|e| io::Error::other(format!("connect to {}: {}", addr, e)))?;
let mut client = volume_server_client(channel);
client
.volume_ec_blob_delete(Request::new(VolumeEcBlobDeleteRequest {
volume_id: target.vid.0,
collection: target.collection.to_string(),
file_key: target.needle_id.0,
version: target.version.0 as u32,
}))
.await
.map_err(|e| io::Error::other(format!("volume_ec_blob_delete on {}: {}", addr, e)))?;
Ok(())
}
/// Journals on the local volume — what the `VolumeEcBlobDelete` handler runs
/// when this server is the shard holder. An absent needle is an error, not a
/// no-op: `journal_delete` would accept it silently, but here it means the
/// volume remounted as a different generation mid-delete and the tombstone
/// should go to a replica that still has the needle.
fn journal_delete_local(
state: &Arc<VolumeServerState>,
vid: VolumeId,
needle_id: NeedleId,
) -> io::Result<()> {
let mut store = state.store.write().unwrap();
let ecv = store.find_ec_volume_mut(vid).ok_or_else(|| {
io::Error::new(
io::ErrorKind::NotFound,
format!("ec volume {} unmounted", vid.0),
)
})?;
match ecv.find_needle_from_ecx(needle_id)? {
None => {
return Err(io::Error::new(
io::ErrorKind::NotFound,
format!("needle {} not in local ecx", needle_id),
));
}
Some((_, size)) if size.is_deleted() => return Ok(()),
Some(_) => {}
}
ecv.journal_delete(needle_id)
}
fn local_shard_ids(ecv: &crate::storage::erasure_coding::EcVolume) -> HashSet<ShardId> {
ecv.shards
.iter()
.enumerate()
.filter_map(|(i, s)| s.as_ref().map(|_| i as ShardId))
.collect()
}
/// FULL EC scrub: verify every needle's bytes across local AND remote shards,
/// without decoding (so genuine shard faults are reported rather than healed).
/// Mirrors Go's `Store.ScrubEcVolume`. Returns (rows walked, broken shards,
@@ -556,9 +324,9 @@ pub async fn scrub_ec_volume_distributed(
// mounted volume's encode_ts_ns no longer matches, abort like a
// mid-scan unmount rather than mixing generations.
let encode_ts_ns = ecv.encode_ts_ns;
// One read section, so the map and the refresh time it is aged against
// describe the same lookup.
let (cached_locations, cache_refreshed_at) = ecv.shard_locations_snapshot();
// Bind to locals so the inner RwLock/Mutex guards drop before the block ends.
let cached_locations = ecv.shard_locations.read().unwrap().clone();
let cache_refreshed_at = *ecv.shard_locations_refresh_time.lock().unwrap();
let data_shards = ecv.data_shards as usize;
let total_shards = (ecv.data_shards + ecv.parity_shards) as usize;
(
@@ -660,7 +428,7 @@ pub async fn scrub_ec_volume_distributed(
);
}
};
ecv.shard_locations_snapshot().0
ecv.shard_locations.read().unwrap().clone()
};
// Walk the .ecx (private fd captured under the lock, no lock held) for the
@@ -1005,7 +773,8 @@ fn build_snapshot(
}
let actual = get_actual_size(size, ecv.version);
let interval_results = read_local_intervals(ecv, intervals);
let (cached_locations, cache_refreshed_at) = ecv.shard_locations_snapshot();
let cached_locations = ecv.shard_locations.read().unwrap().clone();
let cache_refreshed_at = *ecv.shard_locations_refresh_time.lock().unwrap();
Ok(Snapshot {
data_shards: ecv.data_shards,
@@ -1057,14 +826,14 @@ fn needs_refresh(
fn mark_shard_locations_stale(state: &Arc<VolumeServerState>, vid: VolumeId) {
let store = state.store.read().unwrap();
if let Some(ecv) = store.find_ec_volume(vid) {
ecv.mark_shard_locations_stale();
*ecv.shard_locations_stale.lock().unwrap() = true;
}
}
/// Decide whether the caller's snapshot is due a master lookup and, when it is,
/// consume the cache's stale mark in the same critical section. A mark raised
/// from here on belongs to the next refresh: the read that raised it has
/// disproved the map this lookup is about to install.
/// Decide whether the cached map is due a master lookup and, when it is, consume
/// its stale mark in the same critical section. A mark raised from here on
/// belongs to the next refresh: the read that raised it has disproved the map
/// this lookup is about to install.
fn claim_shard_locations_refresh(
state: &Arc<VolumeServerState>,
vid: VolumeId,
@@ -1077,9 +846,12 @@ fn claim_shard_locations_refresh(
let Some(ecv) = store.find_ec_volume(vid) else {
return needs_refresh(locations, refreshed_at, false, data_shards, total_shards);
};
ecv.claim_shard_locations_refresh(|stale| {
needs_refresh(locations, refreshed_at, stale, data_shards, total_shards)
})
let mut stale = ecv.shard_locations_stale.lock().unwrap();
let refresh = needs_refresh(locations, refreshed_at, *stale, data_shards, total_shards);
if refresh {
*stale = false;
}
refresh
}
async fn cached_lookup_ec_shard_locations(
@@ -1100,15 +872,18 @@ async fn cached_lookup_ec_shard_locations(
let grpc_addr =
parse_grpc_address(&master).map_err(|e| io::Error::new(io::ErrorKind::InvalidInput, e))?;
let channel = connect_channel(
&grpc_addr,
state.outgoing_grpc_tls.as_ref(),
GrpcDialOptions::unary(),
)
.await
.map_err(|e| io::Error::other(format!("master connect: {}", e)))?;
let endpoint = build_grpc_endpoint(&grpc_addr, state.outgoing_grpc_tls.as_ref())
.map_err(|e| io::Error::other(e.to_string()))?;
let channel = endpoint
.connect_timeout(Duration::from_secs(5))
.timeout(Duration::from_secs(10))
.connect()
.await
.map_err(|e| io::Error::other(format!("master connect: {}", e)))?;
let mut client = master_client(channel);
let mut client = SeaweedClient::with_interceptor(channel, outgoing_request_id_interceptor)
.max_decoding_message_size(GRPC_MAX_MESSAGE_SIZE)
.max_encoding_message_size(GRPC_MAX_MESSAGE_SIZE);
let resp = client
.lookup_ec_volume(Request::new(LookupEcVolumeRequest { volume_id: vid.0 }))
@@ -1289,13 +1064,14 @@ async fn do_read_remote_ec_shard_interval(
} = iv;
let grpc_addr =
parse_grpc_address(source).map_err(|e| io::Error::new(io::ErrorKind::InvalidInput, e))?;
let channel = connect_channel(
&grpc_addr,
state.outgoing_grpc_tls.as_ref(),
GrpcDialOptions::long(),
)
.await
.map_err(|e| io::Error::other(format!("connect to {}: {}", source, e)))?;
let endpoint = build_grpc_endpoint(&grpc_addr, state.outgoing_grpc_tls.as_ref())
.map_err(|e| io::Error::other(e.to_string()))?;
let channel = endpoint
.connect_timeout(Duration::from_secs(5))
.timeout(Duration::from_secs(30))
.connect()
.await
.map_err(|e| io::Error::other(format!("connect to {}: {}", source, e)))?;
// TODO(grpc-jwt): clusters with `jwt.signing.key` configured will
// reject peer-to-peer VolumeEcShardRead calls until the Rust
@@ -1305,7 +1081,9 @@ async fn do_read_remote_ec_shard_interval(
// here in isolation would split the credential plumbing across
// call sites. Re-visit when outgoing JWT signing lands as a
// server-wide helper.
let mut client = volume_server_client(channel);
let mut client = VolumeServerClient::with_interceptor(channel, outgoing_request_id_interceptor)
.max_decoding_message_size(GRPC_MAX_MESSAGE_SIZE)
.max_encoding_message_size(GRPC_MAX_MESSAGE_SIZE);
let req = VolumeEcShardReadRequest {
volume_id: vid.0,
@@ -1693,14 +1471,16 @@ async fn fetch_ec_index_from_one_peer(
) -> io::Result<()> {
let grpc_addr =
parse_grpc_address(peer).map_err(|e| io::Error::new(io::ErrorKind::InvalidInput, e))?;
let channel = connect_channel(
&grpc_addr,
state.outgoing_grpc_tls.as_ref(),
GrpcDialOptions::long(),
)
.await
.map_err(|e| io::Error::other(format!("connect {}: {}", peer, e)))?;
let mut client = volume_server_client(channel);
let channel = build_grpc_endpoint(&grpc_addr, state.outgoing_grpc_tls.as_ref())
.map_err(|e| io::Error::other(e.to_string()))?
.connect_timeout(Duration::from_secs(5))
.timeout(Duration::from_secs(30))
.connect()
.await
.map_err(|e| io::Error::other(format!("connect {}: {}", peer, e)))?;
let mut client = VolumeServerClient::with_interceptor(channel, outgoing_request_id_interceptor)
.max_decoding_message_size(GRPC_MAX_MESSAGE_SIZE)
.max_encoding_message_size(GRPC_MAX_MESSAGE_SIZE);
let copy_req = |ext: &str, ignore_not_found: bool| CopyFileRequest {
volume_id: m.vid.0,
+10 -1
View File
@@ -1,6 +1,5 @@
use std::fmt::Write as _;
use crate::server::absolute_display_path;
use crate::server::server_stats;
use crate::server::volume_server::VolumeServerState;
use crate::storage::store::Store;
@@ -451,6 +450,16 @@ fn collect_ui_data(
(disk_rows, volumes, remote_volumes, ec_volumes)
}
fn absolute_display_path(path: &str) -> String {
let p = std::path::Path::new(path);
if p.is_absolute() {
return path.to_string();
}
std::env::current_dir()
.map(|cwd| cwd.join(p).to_string_lossy().to_string())
.unwrap_or_else(|_| path.to_string())
}
fn join_i64(values: &[i64]) -> String {
values
.iter()
@@ -73,6 +73,8 @@ pub struct VolumeServerState {
pub volume_state_notify: tokio::sync::Notify,
/// Optional batched write queue for improved throughput under load.
pub write_queue: std::sync::OnceLock<WriteQueue>,
/// Registry of S3 tier backends for tiered storage operations.
pub s3_tier_registry: std::sync::RwLock<crate::remote_storage::s3_tier::S3TierRegistry>,
/// Read mode: local, proxy, or redirect for non-local volumes.
pub read_mode: ReadMode,
/// If true, FetchAndWriteNeedle skips remote S3 endpoint validation,
+3
View File
@@ -207,6 +207,9 @@ mod tests {
pre_stop_seconds: 0,
volume_state_notify: tokio::sync::Notify::new(),
write_queue: std::sync::OnceLock::new(),
s3_tier_registry: std::sync::RwLock::new(
crate::remote_storage::s3_tier::S3TierRegistry::new(),
),
read_mode: crate::config::ReadMode::Local,
allow_untrusted_remote_endpoints: false,
master_url: String::new(),
+8 -13
View File
@@ -1133,20 +1133,15 @@ pub fn get_disk_stats(path: &str) -> (u64, u64) {
Ok(p) => p,
Err(_) => return (0, 0),
};
// SAFETY: `libc::statvfs` is plain data — integers and reserved
// padding, no pointers and no restricted niches — so the all-zero
// value is a valid one for the call to overwrite.
let mut stat: libc::statvfs = unsafe { std::mem::zeroed() };
// SAFETY: `c_path` is a live NUL-terminated `CString` that outlives
// the call, and `&mut stat` is a live, aligned, exclusive pointer the
// kernel only writes through; the fields are read below only after
// the call reports success.
if unsafe { libc::statvfs(c_path.as_ptr(), &mut stat) } != 0 {
return (0, 0);
unsafe {
let mut stat: libc::statvfs = std::mem::zeroed();
if libc::statvfs(c_path.as_ptr(), &mut stat) == 0 {
let all = stat.f_blocks as u64 * stat.f_frsize as u64;
let free = stat.f_bavail as u64 * stat.f_frsize as u64;
return (all, free);
}
}
let all = stat.f_blocks as u64 * stat.f_frsize as u64;
let free = stat.f_bavail as u64 * stat.f_frsize as u64;
(all, free)
(0, 0)
}
#[cfg(windows)]
{
@@ -7,48 +7,19 @@ use std::collections::{HashMap, HashSet};
use std::fs::{self, File, OpenOptions};
use std::io::{self, Write};
use std::sync::RwLock;
use std::time::{Instant, SystemTime, UNIX_EPOCH};
use std::time::{SystemTime, UNIX_EPOCH};
use crate::pb::master_pb;
use crate::storage::erasure_coding::ec_locate;
use crate::storage::erasure_coding::ec_shard::*;
use crate::storage::io::read_exact_at;
use crate::storage::io_error::IoErrorTracker;
use crate::storage::needle::needle::{Needle, NeedleError, get_actual_size};
use crate::storage::types::*;
use crate::storage::volume_open::open_volume_file;
/// The shard-location cache: where each shard lives, when that was last learned
/// from the master, and whether a read has since disproved it. One struct behind
/// one lock, because the freshness heuristic judges all three together (the map
/// and its time from the reader's own snapshot, see
/// `EcVolume::claim_shard_locations_refresh`) — a map published ahead of its
/// refresh time reads as a fresh map stamped with the previous lookup, and a
/// half-populated map must never look fresh at all.
#[derive(Default)]
pub(crate) struct ShardLocationCache {
/// Maps shard ID -> list of server addresses where that shard exists.
/// Used for distributed EC reads across the cluster.
locations: HashMap<ShardId, Vec<String>>,
/// Wall-clock timestamp of the most recent successful `LookupEcVolume`
/// refresh of `locations`. `None` until the first refresh. Drives the
/// staleness heuristic in `cached_lookup_ec_shard_locations` (mirrors Go's
/// `ShardLocationsRefreshTime`).
refreshed_at: Option<Instant>,
/// Marks the map for a prompt re-check: a read that failed against a cached
/// location has disproved what the map claims, and the normal freshness
/// window is far too long to serve from a map known to be wrong. Mirrors the
/// invalidation Go's `forgetShardId` performs. A field under the map's lock
/// rather than an atomic so the refresh can consume the mark in one critical
/// section, and a mark raised meanwhile survives for the next refresh.
stale: bool,
}
/// Bytes read per positional read when seeding `deleted_needles` from `.ecj`.
/// A multiple of `NEEDLE_ID_SIZE`; 1 MiB is 131072 entries per syscall.
const ECJ_LOAD_CHUNK_BYTES: usize = 1 << 20;
/// An erasure-coded volume managing its local shards and index.
pub const IO_ERROR_TOLERANCE: i32 = 3;
pub struct EcVolume {
pub volume_id: VolumeId,
pub collection: String,
@@ -79,11 +50,25 @@ pub struct EcVolume {
pub disk_type: DiskType,
/// Directory where .ecx/.ecj were actually found (may differ from dir_idx after fallback).
ecx_actual_dir: String,
/// Where each shard lives and how far that is to be trusted, all under one
/// `RwLock` so the read path can refresh it (under master lookup) without
/// holding the Store write lock — mirrors Go's `ShardLocationsLock
/// sync.RWMutex` in `weed/storage/erasure_coding/ec_volume.go`.
shard_locations: RwLock<ShardLocationCache>,
/// Maps shard ID -> list of server addresses where that shard exists.
/// Used for distributed EC reads across the cluster. Wrapped in
/// `RwLock` so the read path can refresh the map (under master
/// lookup) without holding the Store write lock — mirrors Go's
/// `ShardLocationsLock sync.RWMutex` in `weed/storage/erasure_coding/ec_volume.go`.
pub shard_locations: std::sync::RwLock<HashMap<ShardId, Vec<String>>>,
/// Wall-clock timestamp of the most recent successful
/// `LookupEcVolume` refresh of `shard_locations`. `None` until the
/// first refresh. Drives the staleness heuristic in
/// `cached_lookup_ec_shard_locations` (mirrors Go's
/// `ShardLocationsRefreshTime`).
pub shard_locations_refresh_time: std::sync::Mutex<Option<std::time::Instant>>,
/// Marks the map for a prompt re-check: a read that failed against a cached
/// location has disproved what the map claims, and the normal freshness
/// window is far too long to serve from a map known to be wrong. Mirrors the
/// invalidation Go's `forgetShardId` performs. A mutex rather than an atomic
/// so the refresh can judge the map and consume the mark in one critical
/// section, and a mark raised meanwhile survives for the next refresh.
pub shard_locations_stale: std::sync::Mutex<bool>,
/// EC volume expiration time (unix epoch seconds), set during EC encode from TTL.
pub expire_at_sec: u64,
/// Encode-run identity (unix nanos) loaded from the .vif EcShardConfig. A read
@@ -107,9 +92,9 @@ pub struct EcVolume {
/// sidecar other than the one those two fields actually hold.
pub(crate) bitrot_source_dir: String,
/// Consecutive storage-media errors and the quarantine they lead to,
/// for EC volume health monitoring.
io_errors: IoErrorTracker,
io_error_count: std::sync::atomic::AtomicI32,
io_error_quarantined: std::sync::atomic::AtomicBool,
last_io_error: std::sync::Mutex<Option<String>>,
}
/// Locate the `.vif` for a (collection, vid) by preferring the data dir
@@ -456,13 +441,17 @@ impl EcVolume {
deleted_needles: RwLock::new(HashSet::new()),
disk_type: DiskType::default(),
ecx_actual_dir: dir_idx.to_string(),
shard_locations: RwLock::new(ShardLocationCache::default()),
shard_locations: std::sync::RwLock::new(HashMap::new()),
shard_locations_refresh_time: std::sync::Mutex::new(None),
shard_locations_stale: std::sync::Mutex::new(false),
expire_at_sec,
encode_ts_ns,
bitrot: None,
bitrot_status: crate::storage::erasure_coding::ec_bitrot::BitrotStatus::Off,
bitrot_source_dir: String::new(),
io_errors: IoErrorTracker::default(),
io_error_count: std::sync::atomic::AtomicI32::new(0),
io_error_quarantined: std::sync::atomic::AtomicBool::new(false),
last_io_error: std::sync::Mutex::new(None),
};
// Open .ecx file (sorted index) in read/write mode for in-place deletion marking.
@@ -498,45 +487,6 @@ impl EcVolume {
let ecj_base =
crate::storage::volume::volume_file_name(&vol.ecx_actual_dir, collection, volume_id);
let ecj_path = format!("{}.ecj", ecj_base);
// Repair a torn tail BEFORE the append handle exists.
//
// The file is a flat array of fixed-size records and the journal handle
// is in append mode, so every write lands at the physical end. A
// trailing partial record therefore knocks every later append out of
// alignment: the loader below skips the partial bytes, but the next
// mount decodes those bytes together with the leading bytes of a real
// entry, yielding one garbage id and silently dropping the delete that
// followed the tear. Truncating to a whole number of records costs at
// most one incomplete id that was never readable anyway.
//
// This deliberately uses its own read+write handle rather than the
// append handle opened below: on Windows, `append(true)` requests
// FILE_APPEND_DATA *without* FILE_WRITE_DATA (and `.write(true)` is
// subsumed by `.append(true)`), so SetEndOfFile through that handle
// fails with ERROR_ACCESS_DENIED. The handle is scoped so it is closed
// again before the append handle opens.
{
let repair = open_volume_file(
OpenOptions::new().read(true).write(true).create(true),
&ecj_path,
)?;
let on_disk = repair.metadata()?.len() as i64;
let ragged = on_disk % NEEDLE_ID_SIZE as i64;
if ragged != 0 {
let whole = on_disk - ragged;
tracing::warn!(
volume_id = volume_id.0,
collection = %collection,
on_disk_bytes = on_disk,
truncated_to = whole,
"truncating torn .ecj tail so later appends stay aligned",
);
repair.set_len(whole as u64)?;
repair.sync_all()?;
}
}
let ecj_file = open_volume_file(
OpenOptions::new()
.read(true)
@@ -736,14 +686,6 @@ impl EcVolume {
/// from `new()` under exclusive ownership of the just-constructed
/// EcVolume, so locking is not strictly required — but we take the
/// write lock anyway for symmetry with later mutations.
///
/// Read in large chunks. This previously issued one `NEEDLE_ID_SIZE`-byte
/// positional read per entry, which is fine for a healthy journal (a few
/// KB) and catastrophic for a bloated one: at the 1.51 TB seen in
/// production that is ~188e9 syscalls, so the server spun at 100% of one
/// core with a 31 MB RSS — the set stays small because the ids repeat —
/// and never opened its HTTP port, which made the master unregister every
/// volume it held. Chunked reads cut that by ~`ECJ_LOAD_CHUNK_BYTES / 8`.
fn load_deleted_needles_from_ecj(&mut self) -> io::Result<()> {
let ecj_file = match self.ecj_file.as_ref() {
Some(f) => f,
@@ -752,51 +694,35 @@ impl EcVolume {
if self.ecj_file_size < NEEDLE_ID_SIZE as i64 {
return Ok(());
}
// Build into a local set and merge at the end. The previous version
// held the `deleted_needles` write lock for the whole scan, which on a
// bloated journal is the entire (unbounded) startup.
let mut loaded: HashSet<NeedleId> = HashSet::new();
let mut buf = vec![0u8; ECJ_LOAD_CHUNK_BYTES];
let end = self.ecj_file_size as u64;
let mut off: u64 = 0;
while off + NEEDLE_ID_SIZE as u64 <= end {
// Whole entries only; a trailing partial record is ignored, as the
// per-entry loop did by construction.
let mut want = std::cmp::min(ECJ_LOAD_CHUNK_BYTES as u64, end - off) as usize;
want -= want % NEEDLE_ID_SIZE;
if want == 0 {
break;
}
let mut buf = [0u8; NEEDLE_ID_SIZE];
let mut set = self
.deleted_needles
.write()
.map_err(|_| io::Error::other("deleted_needles lock poisoned"))?;
let mut off: i64 = 0;
while off + NEEDLE_ID_SIZE as i64 <= self.ecj_file_size {
#[cfg(unix)]
{
use std::os::unix::fs::FileExt;
ecj_file.read_exact_at(&mut buf[..want], off)?;
ecj_file.read_exact_at(&mut buf, off as u64)?;
}
#[cfg(windows)]
{
// Positional read so concurrent readers of the shared .ecj
// handle can't interleave seek/read. Mirrors the
// read_exact_at helper at the bottom of this file.
read_exact_at(ecj_file, &mut buf[..want], off)?;
read_exact_at(ecj_file, &mut buf, off as u64)?;
}
#[cfg(not(any(unix, windows)))]
{
compile_error!("Platform not supported: only unix and windows are supported");
}
for entry in buf[..want].chunks_exact(NEEDLE_ID_SIZE) {
loaded.insert(NeedleId::from_bytes(entry));
}
off += want as u64;
set.insert(NeedleId::from_bytes(&buf));
off += NEEDLE_ID_SIZE as i64;
}
let mut set = self
.deleted_needles
.write()
.map_err(|_| io::Error::other("deleted_needles lock poisoned"))?;
set.extend(loaded);
Ok(())
}
/// Returns (file_count, delete_count) for this EC volume. Mirrors Go's
/// `EcVolume.FileAndDeleteCount`:
///
@@ -1012,6 +938,37 @@ impl EcVolume {
// ---- Shard locations (distributed tracking) ----
/// Set the list of server addresses for a single shard ID. Does
/// NOT touch `shard_locations_refresh_time` — a per-shard write
/// from inside a multi-shard population (e.g. iterating the
/// `LookupEcVolume` response shard-by-shard) would otherwise
/// flip the staleness flag while the map is still incomplete,
/// letting a concurrent reader observe `needs_refresh == false`
/// against a half-populated cache and return NotFound for the
/// not-yet-inserted shards.
///
/// Callers writing back a whole `LookupEcVolume` reply should use
/// [`Self::merge_shard_locations`] instead — it upserts the reply's
/// shards under the write lock and advances the refresh timestamp in
/// one step, retaining cached shards the reply omits.
pub fn set_shard_locations(&self, shard_id: ShardId, locations: Vec<String>) {
self.shard_locations
.write()
.unwrap()
.insert(shard_id, locations);
}
/// Atomically replace the entire shard-locations map and stamp
/// the refresh time. Used by the distributed-read path's
/// post-`LookupEcVolume` write-back so the cache transitions
/// from old → fresh in a single observable step — concurrent
/// readers either see the full prior map or the full new map,
/// never an intermediate state with the freshness flag flipped.
pub fn replace_shard_locations(&self, locations: HashMap<ShardId, Vec<String>>) {
*self.shard_locations.write().unwrap() = locations;
*self.shard_locations_refresh_time.lock().unwrap() = Some(std::time::Instant::now());
}
/// Merge a fresh `LookupEcVolume` reply into the shard-locations cache and
/// stamp the refresh time, returning a clone of the resulting map.
///
@@ -1020,76 +977,73 @@ impl EcVolume {
/// from the reply keep their previously-cached locations — so a reply that
/// passes the data-shard completeness guard but omits a shard already in cache
/// does not drop that shard's known location (unlike a full replace).
///
/// Map and refresh time advance in one write section, so no reader can pair
/// the merged map with the previous lookup's timestamp.
pub(crate) fn merge_shard_locations(
pub fn merge_shard_locations(
&self,
locations: HashMap<ShardId, Vec<String>>,
) -> HashMap<ShardId, Vec<String>> {
let mut cache = self.shard_locations.write().unwrap();
for (shard_id, addrs) in locations {
cache.locations.insert(shard_id, addrs);
}
cache.refreshed_at = Some(Instant::now());
cache.locations.clone()
let merged = {
let mut guard = self.shard_locations.write().unwrap();
for (shard_id, addrs) in locations {
guard.insert(shard_id, addrs);
}
guard.clone()
};
*self.shard_locations_refresh_time.lock().unwrap() = Some(std::time::Instant::now());
merged
}
/// The cached map and the time it was learned, read in one section. Both
/// halves describe the same refresh, which is what the freshness heuristic
/// assumes when it ages the map against the timestamp.
pub(crate) fn shard_locations_snapshot(
&self,
) -> (HashMap<ShardId, Vec<String>>, Option<Instant>) {
let cache = self.shard_locations.read().unwrap();
(cache.locations.clone(), cache.refreshed_at)
/// Get a cloned list of server addresses for a given shard ID.
pub fn get_shard_locations(&self, shard_id: ShardId) -> Vec<String> {
self.shard_locations
.read()
.unwrap()
.get(&shard_id)
.cloned()
.unwrap_or_default()
}
/// Mark the cached map for a prompt re-check after a read failed against one
/// of its locations.
pub(crate) fn mark_shard_locations_stale(&self) {
self.shard_locations.write().unwrap().stale = true;
}
/// Put the stale mark to `decide`, and clear it only if `decide` says the
/// master lookup is going ahead — both in one critical section, so the mark
/// is consumed exactly once by the refresh that answers for it and a mark
/// raised meanwhile survives for the next one. `decide` owns the rest of the
/// freshness rule; it judges the map its caller will actually read from,
/// which is not necessarily the one cached here by the time it runs.
///
/// `decide` runs under the cache write lock and must not acquire other locks:
/// every caller reaches this while already holding the store read lock, so a
/// cache-write → store-read inside `decide` would invert that order.
pub(crate) fn claim_shard_locations_refresh(&self, decide: impl FnOnce(bool) -> bool) -> bool {
let mut cache = self.shard_locations.write().unwrap();
let refresh = decide(cache.stale);
if refresh {
cache.stale = false;
}
refresh
}
// ---- I/O error tracking ----
// ---- I/O error tracking (mirrors Go's EcVolume IoErrorTracker) ----
pub fn check_read_write_error(&self, err: Option<&io::Error>) {
self.io_errors.check_read_write_error(err);
use std::sync::atomic::Ordering;
if let Some(e) = err
&& crate::storage::volume::is_storage_io_error(e)
{
self.io_error_count.fetch_add(1, Ordering::Relaxed);
if let Ok(mut guard) = self.last_io_error.lock() {
*guard = Some(e.to_string());
}
crate::metrics::STORAGE_IO_ERROR_COUNTER.inc();
return;
}
self.io_error_count.store(0, Ordering::Relaxed);
if let Ok(mut guard) = self.last_io_error.lock()
&& guard.is_some()
{
*guard = None;
}
}
pub fn get_io_error_state(&self) -> (Option<String>, i32, bool) {
self.io_errors.get_io_error_state()
}
pub fn should_quarantine(&self) -> bool {
self.io_errors.should_quarantine()
use std::sync::atomic::Ordering;
let err = self.last_io_error.lock().ok().and_then(|g| g.clone());
let count = self.io_error_count.load(Ordering::Relaxed);
let quarantined = self.io_error_quarantined.load(Ordering::Relaxed);
(err, count, quarantined)
}
pub fn mark_io_quarantined(&self) {
self.io_errors.mark_io_quarantined();
self.io_error_quarantined
.store(true, std::sync::atomic::Ordering::Relaxed);
}
pub fn reset_io_error_state(&self) {
self.io_errors.reset_io_error_state();
use std::sync::atomic::Ordering;
self.io_error_count.store(0, Ordering::Relaxed);
self.io_error_quarantined.store(false, Ordering::Relaxed);
if let Ok(mut guard) = self.last_io_error.lock() {
*guard = None;
}
}
// ---- Index operations ----
@@ -1620,21 +1574,9 @@ impl EcVolume {
// write_all may have extended the file on disk before
// sync_all failed; truncate back to the known-good size so
// the on-disk journal never drifts past `deleted_needles`.
// Uses its own write handle: on Windows the append handle
// lacks FILE_WRITE_DATA, so set_len through it fails with
// ERROR_ACCESS_DENIED and the rollback would silently not
// happen.
let ecj_path = format!(
"{}.ecj",
crate::storage::volume::volume_file_name(
&self.ecx_actual_dir,
&self.collection,
self.volume_id,
)
);
let rollback = open_volume_file(OpenOptions::new().write(true), &ecj_path)
.and_then(|f| f.set_len(prev_ecj_size as u64).and_then(|_| f.sync_all()));
if let Err(trunc_err) = rollback {
if let Some(ecj) = self.ecj_file.as_mut()
&& let Err(trunc_err) = ecj.set_len(prev_ecj_size as u64)
{
tracing::error!(
volume_id = self.volume_id.0,
needle_id = needle_id.0,
@@ -2412,122 +2354,6 @@ mod tests {
assert_eq!((fc, dc), (2, 2));
}
/// Write a raw `.ecj` containing `ids` repeated `repeats` times, i.e. the
/// shape the append paths produce when a peer's whole journal is
/// concatenated onto this one over and over.
fn write_bloated_ecj(
dir: &str,
collection: &str,
vid: VolumeId,
ids: &[NeedleId],
repeats: usize,
) {
let base = crate::storage::volume::volume_file_name(dir, collection, vid);
let mut one = vec![0u8; ids.len() * NEEDLE_ID_SIZE];
for (i, id) in ids.iter().enumerate() {
id.to_bytes(&mut one[i * NEEDLE_ID_SIZE..(i + 1) * NEEDLE_ID_SIZE]);
}
let mut f = File::create(format!("{}.ecj", base)).unwrap();
for _ in 0..repeats {
f.write_all(&one).unwrap();
}
f.sync_all().unwrap();
}
/// A journal whose length is not a whole number of entries must not lose
/// the entries that ARE complete, and must not panic.
#[test]
fn test_ecj_with_trailing_partial_entry() {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
write_ecx_file(dir, "", VolumeId(40), &[]);
let base = crate::storage::volume::volume_file_name(dir, "", VolumeId(40));
let ids: Vec<NeedleId> = (1..=3).map(NeedleId).collect();
let mut bytes = vec![0u8; ids.len() * NEEDLE_ID_SIZE];
for (i, id) in ids.iter().enumerate() {
id.to_bytes(&mut bytes[i * NEEDLE_ID_SIZE..(i + 1) * NEEDLE_ID_SIZE]);
}
bytes.extend_from_slice(&[0xAB, 0xCD, 0xEF]); // torn tail
std::fs::write(format!("{}.ecj", base), &bytes).unwrap();
let vol = EcVolume::new(dir, dir, "", VolumeId(40)).unwrap();
assert_eq!(vol.read_deleted_needles().unwrap(), ids);
}
/// A torn tail must be truncated at mount, not merely skipped. The handle
/// is in append mode, so leaving the partial bytes in place would push
/// every later append out of alignment: the delete taken after the tear
/// would decode as garbage on the next mount and be silently lost.
#[test]
fn test_torn_ecj_tail_is_repaired_so_later_deletes_survive() {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
// journal_delete only appends on a live->tombstone transition.
write_ecx_file(
dir,
"",
VolumeId(42),
&[(NeedleId(7), Offset::from_actual_offset(8), Size(10))],
);
let base = crate::storage::volume::volume_file_name(dir, "", VolumeId(42));
let ecj_path = format!("{}.ecj", base);
let ids: Vec<NeedleId> = (1..=3).map(NeedleId).collect();
let mut bytes = vec![0u8; ids.len() * NEEDLE_ID_SIZE];
for (i, id) in ids.iter().enumerate() {
id.to_bytes(&mut bytes[i * NEEDLE_ID_SIZE..(i + 1) * NEEDLE_ID_SIZE]);
}
bytes.extend_from_slice(&[0xAB, 0xCD, 0xEF]); // torn tail
std::fs::write(&ecj_path, &bytes).unwrap();
let mut vol = EcVolume::new(dir, dir, "", VolumeId(42)).unwrap();
// The tear is gone from disk, not just ignored in memory.
assert_eq!(
std::fs::metadata(&ecj_path).unwrap().len(),
(ids.len() * NEEDLE_ID_SIZE) as u64,
"torn tail should have been truncated at mount",
);
vol.journal_delete(NeedleId(7)).unwrap();
drop(vol);
// The delete taken after the repair must survive a remount.
let vol2 = EcVolume::new(dir, dir, "", VolumeId(42)).unwrap();
let deleted = vol2.read_deleted_needles().unwrap();
assert!(
deleted.contains(&NeedleId(7)),
"delete after a torn tail was lost: {:?}",
deleted,
);
assert_eq!(
deleted.len(),
ids.len() + 1,
"misaligned decode: {:?}",
deleted
);
}
/// A journal spanning several read chunks must load every entry — guards
/// the chunk-boundary arithmetic in the buffered loader.
#[test]
fn test_ecj_spanning_multiple_read_chunks() {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
write_ecx_file(dir, "", VolumeId(41), &[]);
// 2.5 chunks' worth of DISTINCT ids, so nothing is masked by dedup and
// the total is not a chunk multiple.
let n = (ECJ_LOAD_CHUNK_BYTES / NEEDLE_ID_SIZE) * 5 / 2;
let ids: Vec<NeedleId> = (1..=n as u64).map(NeedleId).collect();
write_bloated_ecj(dir, "", VolumeId(41), &ids, 1);
let vol = EcVolume::new(dir, dir, "", VolumeId(41)).unwrap();
let deleted = vol.read_deleted_needles().unwrap();
assert_eq!(deleted.len(), n, "entries lost across a chunk boundary");
assert_eq!(deleted, ids);
}
#[test]
fn test_journal_delete_wrong_cookie() {
let tmp = TempDir::new().unwrap();
@@ -3682,93 +3508,6 @@ mod tests {
errs
);
}
/// Mount a bare EC volume: an empty .ecx is enough to exercise the
/// shard-location cache, which is pure in-memory state.
fn mount_bare_ec_volume(dir: &str) -> EcVolume {
let base = crate::storage::volume::volume_file_name(dir, "", VolumeId(1));
std::fs::write(format!("{}.ecx", base), b"").unwrap();
EcVolume::new(dir, dir, "", VolumeId(1)).unwrap()
}
/// The map and the refresh time it was learned at must become visible in the
/// same step. A merge that published the map first and stamped the time
/// afterwards let a reader in between pair a fresh map with the previous
/// refresh time -- and the freshness heuristic judges exactly that pair, so
/// the reader re-looked-up a map that had just been refreshed.
#[test]
fn merge_shard_locations_publishes_map_and_time_together() {
let tmp = TempDir::new().unwrap();
let vol = mount_bare_ec_volume(tmp.path().to_str().unwrap());
let (locations, refreshed_at) = vol.shard_locations_snapshot();
assert!(locations.is_empty(), "a fresh mount knows no locations");
assert!(
refreshed_at.is_none(),
"a volume that never refreshed has no refresh time"
);
let reply: HashMap<ShardId, Vec<String>> =
HashMap::from([(0, vec!["a:1".to_string()]), (1, vec!["b:2".to_string()])]);
let merged = vol.merge_shard_locations(reply.clone());
assert_eq!(merged, reply, "merge returns the resulting map");
let (locations, refreshed_at) = vol.shard_locations_snapshot();
assert_eq!(locations, reply, "the snapshot reports the merged map");
assert!(
refreshed_at.is_some(),
"the merge that produced that map also stamped its refresh time"
);
// Upsert, not replace: a reply that omits a cached shard keeps it.
// `before` is taken outside the merge so the assertion fails if the
// second merge did not re-stamp: `later >= refreshed_at` would hold on
// the first merge's stamp alone.
let before = Instant::now();
let merged = vol.merge_shard_locations(HashMap::from([(1, vec!["c:3".to_string()])]));
assert_eq!(merged.get(&0), Some(&vec!["a:1".to_string()]));
assert_eq!(merged.get(&1), Some(&vec!["c:3".to_string()]));
let (locations, later) = vol.shard_locations_snapshot();
assert_eq!(locations, merged, "the snapshot reports the merged map");
assert!(
later.unwrap() >= before,
"the second merge re-stamped the refresh time"
);
}
/// The stale mark is consumed by the refresh that acts on it, exactly once:
/// otherwise every later read keeps re-looking-up a map no one has disproved
/// since. A claim that declines the refresh must leave the mark standing.
#[test]
fn claim_shard_locations_refresh_consumes_the_stale_mark_once() {
let tmp = TempDir::new().unwrap();
let vol = mount_bare_ec_volume(tmp.path().to_str().unwrap());
assert!(
!vol.claim_shard_locations_refresh(|stale| stale),
"an untouched cache carries no mark"
);
vol.mark_shard_locations_stale();
assert!(
vol.claim_shard_locations_refresh(|stale| stale),
"the mark is visible to the next claim"
);
assert!(
!vol.claim_shard_locations_refresh(|stale| stale),
"the claim that acted on the mark consumed it"
);
vol.mark_shard_locations_stale();
assert!(
!vol.claim_shard_locations_refresh(|_| false),
"the refresh decision stays the caller's"
);
assert!(
vol.claim_shard_locations_refresh(|stale| stale),
"a claim that did not refresh leaves the mark for the next one"
);
}
}
#[cfg(test)]
-202
View File
@@ -1,202 +0,0 @@
//! Consecutive storage-media error tracking shared by `Volume` and
//! `EcVolume`. Mirrors Go's `weed/storage/io_error.go`.
use std::io;
use std::sync::Mutex;
use std::sync::atomic::{AtomicBool, AtomicI32, Ordering};
/// Consecutive storage-media errors allowed before the volume is quarantined.
pub(crate) const IO_ERROR_TOLERANCE: i32 = 3;
/// Returns true for I/O errors that indicate faulty storage media, not
/// transient/network failures. On Unix this is EIO; on Windows it covers
/// ERROR_CRC and ERROR_IO_DEVICE, which the kernel returns for failing disks.
pub(crate) fn is_storage_io_error(e: &io::Error) -> bool {
#[cfg(unix)]
{
e.raw_os_error() == Some(libc::EIO)
}
#[cfg(windows)]
{
const ERROR_CRC: i32 = 23;
const ERROR_IO_DEVICE: i32 = 1117;
return e.raw_os_error() == Some(ERROR_CRC) || e.raw_os_error() == Some(ERROR_IO_DEVICE);
}
#[cfg(not(any(unix, windows)))]
{
false
}
}
/// Consecutive storage-media error state for one volume. `quarantined` is
/// sticky: once set it survives later successful I/O and is lifted only by
/// `reset_io_error_state`.
#[derive(Default)]
pub(crate) struct IoErrorTracker {
last: Mutex<Option<String>>,
count: AtomicI32,
quarantined: AtomicBool,
}
impl IoErrorTracker {
/// `Some(e)` records a failure, `None` a success. Only storage-media
/// failures count; every other outcome clears the count and last error.
pub(crate) fn check_read_write_error(&self, err: Option<&io::Error>) {
if let Some(e) = err
&& is_storage_io_error(e)
{
self.count.fetch_add(1, Ordering::Relaxed);
if let Ok(mut guard) = self.last.lock() {
*guard = Some(e.to_string());
}
crate::metrics::STORAGE_IO_ERROR_COUNTER.inc();
return;
}
self.count.store(0, Ordering::Relaxed);
if let Ok(mut guard) = self.last.lock()
&& guard.is_some()
{
*guard = None;
}
}
/// The last recorded error, the consecutive count, and the quarantine flag.
pub(crate) fn get_io_error_state(&self) -> (Option<String>, i32, bool) {
let err = self.last.lock().ok().and_then(|g| g.clone());
let count = self.count.load(Ordering::Relaxed);
let quarantined = self.quarantined.load(Ordering::Relaxed);
(err, count, quarantined)
}
pub(crate) fn should_quarantine(&self) -> bool {
self.quarantined.load(Ordering::Relaxed)
|| self.count.load(Ordering::Relaxed) >= IO_ERROR_TOLERANCE
}
pub(crate) fn mark_io_quarantined(&self) {
self.quarantined.store(true, Ordering::Relaxed);
}
pub(crate) fn reset_io_error_state(&self) {
self.count.store(0, Ordering::Relaxed);
self.quarantined.store(false, Ordering::Relaxed);
if let Ok(mut guard) = self.last.lock() {
*guard = None;
}
}
#[cfg(test)]
pub(crate) fn set_last_io_error_for_test(&self, err: Option<&str>) {
if let Ok(mut guard) = self.last.lock() {
*guard = err.map(|value| value.to_string());
}
if err.is_some() {
self.count.store(IO_ERROR_TOLERANCE, Ordering::Relaxed);
} else {
self.count.store(0, Ordering::Relaxed);
}
}
}
// The tracker only reacts to errors `is_storage_io_error` recognises, which is
// nothing at all on a platform that is neither Unix nor Windows.
#[cfg(all(test, any(unix, windows)))]
mod tests {
use super::*;
/// An OS error the platform reports for failing storage media.
#[cfg(unix)]
fn media_error() -> io::Error {
io::Error::from_raw_os_error(libc::EIO)
}
/// An OS error the platform reports for failing storage media.
#[cfg(windows)]
fn media_error() -> io::Error {
const ERROR_IO_DEVICE: i32 = 1117;
io::Error::from_raw_os_error(ERROR_IO_DEVICE)
}
#[test]
fn check_read_write_error_counts_consecutive_media_errors() {
let tracker = IoErrorTracker::default();
tracker.check_read_write_error(Some(&media_error()));
tracker.check_read_write_error(Some(&media_error()));
let (last, count, quarantined) = tracker.get_io_error_state();
assert_eq!(last, Some(media_error().to_string()));
assert_eq!(count, 2);
assert!(!quarantined);
}
#[test]
fn success_clears_the_count_and_the_last_error() {
let tracker = IoErrorTracker::default();
tracker.check_read_write_error(Some(&media_error()));
tracker.check_read_write_error(None);
assert_eq!(tracker.get_io_error_state(), (None, 0, false));
}
#[test]
fn non_media_error_clears_the_count() {
let tracker = IoErrorTracker::default();
tracker.check_read_write_error(Some(&media_error()));
tracker.check_read_write_error(Some(&io::Error::new(
io::ErrorKind::NotFound,
"no such file",
)));
assert_eq!(tracker.get_io_error_state(), (None, 0, false));
}
#[test]
fn should_quarantine_only_once_the_tolerance_is_reached() {
let tracker = IoErrorTracker::default();
for _ in 1..IO_ERROR_TOLERANCE {
tracker.check_read_write_error(Some(&media_error()));
assert!(!tracker.should_quarantine());
}
tracker.check_read_write_error(Some(&media_error()));
assert!(tracker.should_quarantine());
}
#[test]
fn quarantine_survives_later_successful_io() {
let tracker = IoErrorTracker::default();
tracker.mark_io_quarantined();
tracker.check_read_write_error(None);
assert_eq!(tracker.get_io_error_state(), (None, 0, true));
assert!(tracker.should_quarantine());
}
#[test]
fn reset_io_error_state_lifts_the_quarantine() {
let tracker = IoErrorTracker::default();
tracker.check_read_write_error(Some(&media_error()));
tracker.mark_io_quarantined();
tracker.reset_io_error_state();
assert_eq!(tracker.get_io_error_state(), (None, 0, false));
assert!(!tracker.should_quarantine());
}
#[test]
fn test_helper_arms_a_sustained_error() {
let tracker = IoErrorTracker::default();
tracker.set_last_io_error_for_test(Some("input/output error"));
assert!(tracker.should_quarantine());
assert_eq!(
tracker.get_io_error_state(),
(
Some("input/output error".to_string()),
IO_ERROR_TOLERANCE,
false
)
);
tracker.set_last_io_error_for_test(None);
assert_eq!(tracker.get_io_error_state(), (None, 0, false));
}
}
-1
View File
@@ -2,7 +2,6 @@ pub mod disk_location;
pub mod erasure_coding;
pub mod idx;
pub(crate) mod io;
pub(crate) mod io_error;
pub mod needle;
pub mod needle_map;
pub mod store;
@@ -749,14 +749,6 @@ pub fn parse_needle_id_cookie(s: &str) -> Result<(NeedleId, Cookie), String> {
(s, None)
};
// Every length check and the split below are in BYTES, so a multi-byte
// character would let `split` land inside one and panic the slice. Hex is
// ASCII by definition; reject anything else up front, as Go's ParseUint
// does a step later.
if !hex_part.is_ascii() {
return Err("KeyHash must be ASCII hex.".to_string());
}
// Go: len(key_hash_string) <= CookieSize*2 => error (must be > 8 hex chars)
if hex_part.len() <= COOKIE_SIZE * 2 {
return Err("KeyHash is too short.".to_string());
@@ -836,30 +828,6 @@ pub enum NeedleError {
mod tests {
use super::*;
/// A fid whose hex part carries multi-byte UTF-8 must be rejected, not
/// panic. `split` is a byte offset into `hex_part`; before the ASCII guard
/// `&hex_part[..split]` could land inside a character. `GET /3,ééééa` is
/// nine bytes, so it passes the length checks and splits at byte 1 —
/// halfway through the first `é`. Go's `ParseUint` just errors.
#[test]
fn parse_needle_id_cookie_rejects_non_ascii_instead_of_panicking() {
for s in ["ééééa", "ééééaaaaa", "0123456é9abc", "ééééa_1"] {
assert!(
parse_needle_id_cookie(s).is_err(),
"non-ASCII fid {:?} must be an error",
s
);
}
}
/// The ASCII guard must not change any accepted input.
#[test]
fn parse_needle_id_cookie_still_accepts_ascii_hex() {
let (id, cookie) = parse_needle_id_cookie("01637037d6").unwrap();
assert_eq!(id, NeedleId(0x01));
assert_eq!(cookie, Cookie(0x637037d6));
}
#[test]
fn test_parse_header() {
let mut buf = [0u8; NEEDLE_HEADER_SIZE];
-16
View File
@@ -80,12 +80,6 @@ impl TTL {
if s.is_empty() {
return Ok(TTL::EMPTY);
}
// The unit is read as the last BYTE and the count as everything before
// it, so a trailing multi-byte character would split inside itself and
// panic. A TTL is digits plus a one-letter unit; reject the rest.
if !s.is_ascii() {
return Err(format!("invalid TTL {:?}: must be ASCII", s));
}
let last_byte = s.as_bytes()[s.len() - 1];
let (num_str, unit_byte) = if last_byte.is_ascii_digit() {
// All digits — default to minutes (matching Go)
@@ -246,16 +240,6 @@ impl fmt::Display for TTL {
mod tests {
use super::*;
/// `?ttl=5%C3%A9` must be an error, not a panic. The unit is taken as the
/// last *byte*, so a trailing multi-byte character made `&s[..s.len()-1]`
/// split inside it.
#[test]
fn ttl_read_rejects_non_ascii_instead_of_panicking() {
for s in ["5é", "é", "3🦀", "12é"] {
assert!(TTL::read(s).is_err(), "non-ASCII TTL {:?} must error", s);
}
}
#[test]
fn test_ttl_parse() {
let ttl = TTL::read("3m").unwrap();
+9 -11
View File
@@ -417,9 +417,9 @@ impl CompactNeedleMap {
}
/// Visit all entries in ascending order by needle ID.
pub fn ascending_visit<F, E>(&self, f: F) -> Result<(), E>
pub fn ascending_visit<F>(&self, f: F) -> Result<(), String>
where
F: FnMut(NeedleId, &NeedleValue) -> Result<(), E>,
F: FnMut(NeedleId, &NeedleValue) -> Result<(), String>,
{
self.map.ascending_visit(f)
}
@@ -1203,10 +1203,9 @@ impl RedbNeedleMap {
}
/// Visit all entries in ascending order by needle ID.
pub fn ascending_visit<F, E>(&self, mut f: F) -> Result<(), E>
pub fn ascending_visit<F>(&self, mut f: F) -> Result<(), String>
where
F: FnMut(NeedleId, &NeedleValue) -> Result<(), E>,
E: From<String>,
F: FnMut(NeedleId, &NeedleValue) -> Result<(), String>,
{
let txn = self
.db_or_err()
@@ -1444,10 +1443,9 @@ impl NeedleMap {
}
/// Visit all entries in ascending order by needle ID.
pub fn ascending_visit<F, E>(&self, f: F) -> Result<(), E>
pub fn ascending_visit<F>(&self, f: F) -> Result<(), String>
where
F: FnMut(NeedleId, &NeedleValue) -> Result<(), E>,
E: From<String>,
F: FnMut(NeedleId, &NeedleValue) -> Result<(), String>,
{
match self {
NeedleMap::InMemory(nm) => nm.ascending_visit(f),
@@ -1468,7 +1466,7 @@ impl NeedleMap {
// The visitor never fails, so neither can this.
let _ = nm.ascending_visit(|id, nv| {
entries.push((id, *nv));
Ok::<(), std::convert::Infallible>(())
Ok(())
});
Ok(entries)
}
@@ -1840,7 +1838,7 @@ mod tests {
let mut live = 0u64;
nm.ascending_visit(|_, _| {
live += 1;
Ok::<(), String>(())
Ok(())
})
.unwrap();
assert_eq!(live, N - 1);
@@ -2053,7 +2051,7 @@ mod tests {
let mut visited = Vec::new();
nm.ascending_visit(|id, nv| {
visited.push((id, nv.size));
Ok::<(), String>(())
Ok(())
})
.unwrap();
@@ -317,11 +317,9 @@ impl SortedFileNeedleMap {
Ok(())
}
/// Visit all live entries in ascending order by needle ID.
pub fn ascending_visit<F, E>(&self, mut f: F) -> Result<(), E>
pub fn ascending_visit<F>(&self, mut f: F) -> Result<(), String>
where
F: FnMut(NeedleId, &NeedleValue) -> Result<(), E>,
E: From<String>,
F: FnMut(NeedleId, &NeedleValue) -> Result<(), String>,
{
let mut visit_error = None;
self.visit_live_entries(|id, nv| {
@@ -331,7 +329,7 @@ impl SortedFileNeedleMap {
}
Ok(())
})
.map_err(|e| visit_error.take().unwrap_or_else(|| E::from(e.to_string())))
.map_err(|e| visit_error.take().unwrap_or_else(|| e.to_string()))
}
pub fn iter_entries(&self) -> io::Result<Vec<(NeedleId, NeedleValue)>> {
@@ -1037,7 +1035,7 @@ mod tests {
let mut visited = Vec::new();
m.ascending_visit(|id, _| {
visited.push(id);
Ok::<(), String>(())
Ok(())
})
.unwrap();
assert_eq!(visited, vec![NeedleId(2)]);
+71 -120
View File
@@ -764,9 +764,10 @@ impl Store {
// ---- Collection operations ----
/// Delete all volumes in a collection.
pub fn delete_collection(&mut self, collection: &str) -> Result<(), VolumeError> {
pub fn delete_collection(&mut self, collection: &str) -> Result<(), String> {
for loc in &mut self.locations {
loc.delete_collection(collection)?;
loc.delete_collection(collection)
.map_err(|e| format!("delete collection {}: {}", collection, e))?;
}
crate::metrics::delete_collection_metrics(collection);
Ok(())
@@ -1144,11 +1145,16 @@ impl Store {
for (vid, ec_vol) in loc.ec_volumes() {
if ec_vol.is_time_to_destroy() {
expired_vids.push(*vid);
} else if ec_vol.should_quarantine() {
io_quarantined_vids.push(*vid);
} else {
ec_shards
.extend(ec_vol.to_volume_ec_shard_information_messages(disk_id as u32));
let (_, io_count, quarantined) = ec_vol.get_io_error_state();
if quarantined
|| io_count >= crate::storage::erasure_coding::ec_volume::IO_ERROR_TOLERANCE
{
io_quarantined_vids.push(*vid);
} else {
ec_shards
.extend(ec_vol.to_volume_ec_shard_information_messages(disk_id as u32));
}
}
}
@@ -1359,6 +1365,18 @@ impl Store {
// ---- Vacuum / Compaction ----
/// Check the garbage level of a volume.
pub fn check_compact_volume(&self, vid: VolumeId) -> Result<f64, String> {
if let Some((_, v)) = self.find_volume(vid) {
Ok(v.garbage_level())
} else {
Err(format!(
"volume id {} is not found during check compact",
vid.0
))
}
}
/// Compact a volume by rewriting only live needles.
pub fn compact_volume<F>(
&mut self,
@@ -1366,53 +1384,68 @@ impl Store {
preallocate: u64,
max_bytes_per_second: i64,
progress_fn: F,
) -> Result<(), VolumeError>
) -> Result<(), String>
where
F: Fn(i64) -> bool,
{
// Required space matches Go's CompactVolume check: the larger of the
// requested preallocation and the estimated volume size.
let (loc_idx, space_needed) = {
let (loc_idx, v) = self
.find_volume(vid)
.ok_or(VolumeError::VolumeNotFound(vid))?;
let estimated = v.dat_file_size().unwrap_or(0) + v.idx_file_size();
(loc_idx, std::cmp::max(preallocate, estimated))
};
let loc_idx = self
.find_volume(vid)
.map(|(i, _)| i)
.ok_or_else(|| format!("volume id {} is not found during compact", vid.0))?;
let dir = self.locations[loc_idx].directory.clone();
let (_, free) = crate::storage::disk_location::get_disk_stats(&dir);
// Compute required space: use the larger of preallocate or estimated volume size
// matching Go's CompactVolume space check
let space_needed = {
let (_, v) = self.find_volume(vid).unwrap();
let estimated = v.dat_file_size().unwrap_or(0) + v.idx_file_size();
std::cmp::max(preallocate, estimated)
};
if free < space_needed {
return Err(VolumeError::InsufficientSpace {
vid,
required: space_needed,
free,
});
return Err(format!(
"not enough free space to compact volume {}. Required: {}, Free: {}",
vid.0, space_needed, free
));
}
let (_, v) = self
.find_volume_mut(vid)
.ok_or(VolumeError::VolumeNotFound(vid))?;
v.compact_by_index(preallocate, max_bytes_per_second, progress_fn)
if let Some((_, v)) = self.find_volume_mut(vid) {
v.compact_by_index(preallocate, max_bytes_per_second, progress_fn)
.map_err(|e| format!("compact volume {}: {}", vid.0, e))
} else {
Err(format!("volume id {} is not found during compact", vid.0))
}
}
/// Commit a completed compaction: swap files and reload.
pub fn commit_compact_volume(&mut self, vid: VolumeId) -> Result<(bool, u64), VolumeError> {
let (_, v) = self
.find_volume_mut(vid)
.ok_or(VolumeError::VolumeNotFound(vid))?;
let is_read_only = v.is_read_only();
v.commit_compact()?;
let volume_size = v.dat_file_size().unwrap_or(0);
Ok((is_read_only, volume_size))
pub fn commit_compact_volume(&mut self, vid: VolumeId) -> Result<(bool, u64), String> {
if let Some((_, v)) = self.find_volume_mut(vid) {
let is_read_only = v.is_read_only();
v.commit_compact()
.map_err(|e| format!("commit compact volume {}: {}", vid.0, e))?;
let volume_size = v.dat_file_size().unwrap_or(0);
Ok((is_read_only, volume_size))
} else {
Err(format!(
"volume id {} is not found during commit compact",
vid.0
))
}
}
/// Clean up leftover compaction files.
pub fn cleanup_compact_volume(&mut self, vid: VolumeId) -> Result<(), VolumeError> {
let (_, v) = self
.find_volume_mut(vid)
.ok_or(VolumeError::VolumeNotFound(vid))?;
v.cleanup_compact()
pub fn cleanup_compact_volume(&mut self, vid: VolumeId) -> Result<(), String> {
if let Some((_, v)) = self.find_volume_mut(vid) {
v.cleanup_compact()
.map_err(|e| format!("cleanup volume {}: {}", vid.0, e))
} else {
Err(format!(
"volume id {} is not found during cleaning up",
vid.0
))
}
}
/// Close all locations and their volumes.
@@ -2291,88 +2324,6 @@ mod tests {
assert!(matches!(err, Err(VolumeError::NotFound)));
}
#[test]
fn test_compaction_of_missing_volume_is_volume_not_found() {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let mut store = make_test_store(&[dir]);
let missing = VolumeId(4242);
let compact = store.compact_volume(missing, 0, 0, |_| true);
assert!(
matches!(compact, Err(VolumeError::VolumeNotFound(v)) if v == missing),
"{compact:?}"
);
let commit = store.commit_compact_volume(missing);
assert!(
matches!(commit, Err(VolumeError::VolumeNotFound(v)) if v == missing),
"{commit:?}"
);
let cleanup = store.cleanup_compact_volume(missing);
assert!(
matches!(cleanup, Err(VolumeError::VolumeNotFound(v)) if v == missing),
"{cleanup:?}"
);
assert_eq!(
VolumeError::VolumeNotFound(missing).to_string(),
"volume id 4242 is not found"
);
}
#[test]
fn test_compact_then_commit_reclaims_a_deleted_needle() {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let mut store = make_test_store(&[dir]);
let vid = VolumeId(1);
store
.add_volume(vid, DiskType::HardDrive, &VolumeSpec::default())
.unwrap();
for i in 1..=3u64 {
let payload = format!("data-{i}").into_bytes();
let mut n = Needle {
id: NeedleId(i),
cookie: Cookie(i as u32),
data_size: payload.len() as u32,
data: payload,
..Needle::default()
};
store.write_volume_needle(vid, &mut n, true).unwrap();
}
let mut del = Needle {
id: NeedleId(2),
cookie: Cookie(2),
..Needle::default()
};
store.delete_volume_needle(vid, &mut del).unwrap();
let size_before = store.find_volume(vid).unwrap().1.dat_file_size().unwrap();
// preallocate 0, unthrottled, progress fn that never cancels.
store.compact_volume(vid, 0, 0, |_| true).unwrap();
let (is_read_only, volume_size) = store.commit_compact_volume(vid).unwrap();
assert!(!is_read_only);
assert!(
volume_size < size_before,
"compaction must drop the deleted needle: {volume_size} vs {size_before}"
);
let (_, v) = store
.find_volume(vid)
.expect("the volume stays mounted across a commit");
assert_eq!(v.file_count(), 2);
assert_eq!(v.deleted_count(), 0);
assert_eq!(v.dat_file_size().unwrap(), volume_size);
}
/// Build a Store with N HDD disk locations under a single TempDir.
/// Returns the store and the TempDir guard so callers keep the dirs
/// alive for the test's lifetime.
-29
View File
@@ -221,35 +221,6 @@ mod tests {
use super::*;
use crate::storage::types::*;
/// Multi-byte input must be an error, not a panic: `to_digit` on the
/// leading characters rejects it before `chars[2]` is ever indexed.
#[test]
fn replica_placement_rejects_non_ascii_instead_of_panicking() {
for s in ["é", "0é", "é0", "🦀", "ééé"] {
assert!(
ReplicaPlacement::from_string(s).is_err(),
"non-ASCII replication {:?} must error",
s
);
}
}
/// The ASCII guard must not change any accepted input, including the
/// zero-padding shorthands.
#[test]
fn replica_placement_still_accepts_ascii_shorthands() {
assert_eq!(
ReplicaPlacement::from_string("1").unwrap(),
ReplicaPlacement::from_string("001").unwrap()
);
assert_eq!(
ReplicaPlacement::from_string("01").unwrap(),
ReplicaPlacement::from_string("001").unwrap()
);
let rp = ReplicaPlacement::from_string("010").unwrap();
assert_eq!(rp.diff_rack_count, 1);
}
#[test]
fn test_super_block_round_trip() {
let sb = SuperBlock {
File diff suppressed because it is too large Load Diff
@@ -122,19 +122,7 @@ impl Volume {
} else {
found.remove(&id);
}
let record_size = NEEDLE_HEADER_SIZE as i64 + needle_body_length(size, version);
// A corrupt header can make the record length zero or negative;
// the scan cannot advance past it.
if record_size <= 0 {
return Err(VolumeError::Io(io::Error::new(
io::ErrorKind::InvalidData,
format!(
"corrupt needle header at offset {offset}: size {}, record length {record_size}",
size.0
),
)));
}
offset += record_size;
offset += NEEDLE_HEADER_SIZE as i64 + needle_body_length(size, version);
}
Ok((found, order))
@@ -438,29 +426,4 @@ mod tests {
"needle 1 should have been recovered"
);
}
#[test]
fn test_scan_dat_head_fails_at_a_header_it_cannot_advance_past() {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
write_test_volume(dir, 2);
let mut dat = OpenOptions::new()
.append(true)
.open(format!("{}/1.dat", dir))
.unwrap();
let mut corrupt = [0u8; NEEDLE_HEADER_SIZE];
NeedleId(99).to_bytes(&mut corrupt[4..12]);
Size(-100).to_bytes(&mut corrupt[12..16]);
dat.write_all(&corrupt).unwrap();
drop(dat);
let v = open_volume(dir);
let first = v.super_block.block_size() as i64;
let err = v.scan_dat_head(v.version(), first, 10).unwrap_err();
assert!(
matches!(&err, VolumeError::Io(e) if e.kind() == io::ErrorKind::InvalidData),
"expected a corrupt-data error, got {err:?}"
);
}
}
+3 -215
View File
@@ -112,6 +112,9 @@ fn build_test_state(
pre_stop_seconds: 0,
volume_state_notify: tokio::sync::Notify::new(),
write_queue: std::sync::OnceLock::new(),
s3_tier_registry: std::sync::RwLock::new(
seaweed_volume::remote_storage::s3_tier::S3TierRegistry::new(),
),
read_mode: seaweed_volume::config::ReadMode::Local,
allow_untrusted_remote_endpoints: false,
master_url,
@@ -1040,218 +1043,3 @@ async fn chunk_manifest_expands_chunk_stored_on_ec_volume() {
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(body_bytes(response).await, chunk_data);
}
// ============================================================================
// HTTP DELETE on an EC volume whose shards are not all mounted locally
//
// The delete handler used to validate the cookie with the local-only
// `EcVolume::read_ec_shard_needle`, which errors "ec shard N not available
// locally" for any interval held by a peer. Every such error was mapped to 500
// and no `.ecj` tombstone was written, so on a standard 10+4 spread across 14
// servers no HTTP delete of an EC needle could ever succeed.
//
// A single node can exercise that path without peers: mount 13 of the 14
// shards, leaving out the one holding the needle's interval. The distributed
// reader seeds its Reed-Solomon buffers from locally mounted siblings (Phase 0
// in `store_ec.rs`), so with >= 10 survivors it reconstructs without any peer
// fan-out — while the local-only read still fails outright.
// ============================================================================
#[tokio::test]
async fn delete_on_ec_volume_succeeds_when_the_needles_shard_is_not_mounted() {
use seaweed_volume::storage::erasure_coding::ec_encoder::write_ec_files;
use seaweed_volume::storage::erasure_coding::ec_shard::ShardId;
use seaweed_volume::storage::needle::needle::{FileId, Needle};
use seaweed_volume::storage::types::{Cookie, NeedleId};
use seaweed_volume::storage::volume::{Volume, VolumeSpec};
let (state, tmp) = test_state();
let dir = tmp.path().to_str().unwrap();
let data: Vec<u8> = (0..4096u32).map(|i| (i % 251) as u8).collect();
let nid = NeedleId(0x5c);
let cookie = Cookie(0x0badc0de);
// Build regular volume 4, then EC-encode it. The volume is standalone and
// never registered in the store, so afterwards it exists only as shards.
{
let mut v = Volume::new(
dir,
dir,
VolumeId(4),
NeedleMapKind::InMemory,
&VolumeSpec::default(),
)
.unwrap();
let mut n = Needle {
id: nid,
cookie,
data: data.clone(),
data_size: data.len() as u32,
..Needle::default()
};
v.write_needle(&mut n, true, false).unwrap();
v.sync_to_disk().unwrap();
v.close();
}
write_ec_files(dir, dir, "", VolumeId(4), 10, 4).unwrap();
// Mount shards 1..=13 only. A needle at .dat offset 0 lives in shard 0's
// first small block, so the interval this delete needs is deliberately the
// one shard that is absent.
{
let mut store = state.store.write().unwrap();
let shard_ids: Vec<ShardId> = (1..14).collect();
store.mount_ec_shards(VolumeId(4), "", &shard_ids).unwrap();
}
let fid = FileId::new(VolumeId(4), nid, cookie).to_string();
let app = build_admin_router(state.clone());
let response = app
.oneshot(
Request::builder()
.method("DELETE")
.uri(format!("/{}", fid))
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(
response.status(),
StatusCode::ACCEPTED,
"DELETE of an EC needle must reconstruct through the distributed \
reader instead of failing 500 on a non-local shard"
);
// The tombstone must actually have landed: a later GET is a 404.
let app = build_admin_router(state.clone());
let response = app
.oneshot(
Request::builder()
.uri(format!("/{}", fid))
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(
response.status(),
StatusCode::NOT_FOUND,
"the delete must have been journalled, not just answered 202"
);
}
// ============================================================================
// Hostile response-header override params must not panic the handler
//
// The `response-*` query params are attacker-controlled and were inserted with
// `parse().unwrap()`. `%0A` decodes to a newline, `HeaderValue::from_str`
// rejects it, and the unwrap panicked the connection task — unauthenticated.
// The override must simply be skipped.
// ============================================================================
#[tokio::test]
async fn hostile_response_header_overrides_are_skipped_not_panicked() {
let (state, _tmp) = test_state();
let uri = "/1,01637037d6";
let app = build_admin_router(state.clone());
let response = app
.oneshot(
Request::builder()
.method("POST")
.uri(uri)
.body(Body::from(b"payload".to_vec()))
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::CREATED);
// One request per override param, each carrying a raw newline.
for param in [
"response-cache-control",
"response-content-encoding",
"response-expires",
"response-content-language",
"response-content-disposition",
"response-content-type",
] {
let app = build_admin_router(state.clone());
let response = app
.oneshot(
Request::builder()
.uri(format!("{}?{}=%0Aevil", uri, param))
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(
response.status(),
StatusCode::OK,
"{} with a newline must be ignored, not panic",
param
);
assert_eq!(body_bytes(response).await, b"payload".to_vec());
}
}
// ============================================================================
// Non-ASCII in the fid and in ?ttl= must be rejected, not panic
//
// `parse_needle_id_cookie` split the hex by BYTE offset and `TTL::read` took
// the unit as the last BYTE, so a multi-byte character split inside itself.
// Both are reachable unauthenticated from the request line / query string.
// ============================================================================
#[tokio::test]
async fn non_ascii_fid_and_ttl_are_rejected_not_panicked() {
let (state, _tmp) = test_state();
// A fid whose hex part is multi-byte UTF-8.
let app = build_admin_router(state.clone());
let response = app
.oneshot(
Request::builder()
.uri("/1,%C3%A9%C3%A9%C3%A9%C3%A9a")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert!(
response.status().is_client_error() || response.status().is_server_error(),
"non-ASCII fid must produce an error status, got {}",
response.status()
);
// A TTL whose unit character is multi-byte. The upload path does
// `TTL::read(..).ok()`, so *any* unparseable TTL is simply dropped and the
// write succeeds — the point here is that a non-ASCII one now takes that
// same road instead of panicking. Assert it matches an ASCII-invalid TTL
// rather than inventing a stricter contract than the handler has.
let mut statuses = Vec::new();
// Distinct needle ids: reusing one id with a different cookie is a
// cookie-mismatch overwrite, which would mask what this test measures.
for (fid, ttl) in [("/1,03637037d7", "5%C3%A9"), ("/1,04637037d8", "5z")] {
let app = build_admin_router(state.clone());
let response = app
.oneshot(
Request::builder()
.method("POST")
.uri(format!("{}?ttl={}", fid, ttl))
.body(Body::from(b"x".to_vec()))
.unwrap(),
)
.await
.unwrap();
statuses.push(response.status());
}
assert_eq!(
statuses[0], statuses[1],
"a non-ASCII ttl must behave like any other invalid ttl, not panic"
);
}
+1 -9
View File
@@ -5329,13 +5329,6 @@ version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
[[package]]
name = "seaweed-common"
version = "0.1.0"
dependencies = [
"rustls",
]
[[package]]
name = "seaweed-worker-core"
version = "0.1.0"
@@ -5347,7 +5340,6 @@ dependencies = [
"prost",
"prost-types",
"protoc-bin-vendored",
"seaweed-common",
"tokio",
"tokio-stream",
"tonic",
@@ -6457,7 +6449,7 @@ dependencies = [
"lance-table",
"prometheus",
"reqwest 0.12.28",
"seaweed-common",
"rustls",
"seaweed-worker-core",
"seaweed-worker-sort",
"serde",
-2
View File
@@ -20,8 +20,6 @@ rust-version = "1.94.1"
# Protobuf message literals keep `..Default::default()` on purpose: it is
# what lets a proto gain a field without touching every constructor.
needless_update = "allow"
# Every `unsafe` block states its precondition, right above the block.
undocumented_unsafe_blocks = "warn"
[workspace.dependencies]
anyhow = "1"
-4
View File
@@ -9,10 +9,6 @@ one.
crates/core the contract: stream, handshake, heartbeat, registry, config forms
crates/lance maintenance jobs for Lance tables, and a binary
It also depends on `../seaweed-common`, a small crate outside this workspace
holding the few helpers the Rust volume server needs identically — the
HTTP<->gRPC address rule and the rustls provider choice.
`core` knows nothing about any job. A second worker is a new crate beside
`lance` that depends on it, not a fork of the protocol.
-3
View File
@@ -9,9 +9,6 @@ description = "SeaweedFS plugin.proto worker contract"
name = "seaweed_worker_core"
[dependencies]
# Helpers the Rust volume server needs as well. A path dependency because
# the two trees are separate cargo workspaces with no common root manifest.
seaweed-common = { path = "../../../seaweed-common" }
anyhow.workspace = true
async-trait.workspace = true
prost.workspace = true
+22 -12
View File
@@ -5,15 +5,32 @@
//! instead fails as "frame with invalid size", which reads like a protocol bug
//! rather than a wrong port, so getting this right is worth its own module.
//! Mirrors pb.ServerToGrpcAddress in weed/pb/grpc_client_server.go.
//!
//! The rule itself now lives in `seaweed_common::address`, shared with the Rust
//! volume server, which had its own copy of it. What stays here is the `Option`
//! shape this crate's callers expect, and the tests that pin it.
const GRPC_PORT_OFFSET: u16 = 10000;
/// Converts `host:port` to the gRPC address, and accepts the explicit
/// `host:port.grpcPort` form the Go side also understands.
pub fn server_to_grpc_address(server: &str) -> Option<String> {
seaweed_common::address::to_grpc_address(server).ok()
let (host, port_part) = server.rsplit_once(':')?;
// "port.grpcPort" states the gRPC port outright.
if let Some((_, grpc_port)) = port_part.split_once('.')
&& let Ok(port) = grpc_port.parse::<u16>()
{
return Some(join_host_port(host, port));
}
let port: u16 = port_part.parse().ok()?;
Some(join_host_port(host, port.checked_add(GRPC_PORT_OFFSET)?))
}
fn join_host_port(host: &str, port: u16) -> String {
// An IPv6 literal has to keep its brackets or the port reads as part of it.
if host.contains(':') && !host.starts_with('[') {
format!("[{host}]:{port}")
} else {
format!("{host}:{port}")
}
}
#[cfg(test)]
@@ -53,11 +70,4 @@ mod tests {
assert!(server_to_grpc_address("localhost").is_none());
assert!(server_to_grpc_address("localhost:notaport").is_none());
}
#[test]
fn rejects_a_dotted_form_whose_http_port_is_not_a_port() {
// Tightened by the move to seaweed-common: this copy used to ignore the
// HTTP port of the dotted form and answer Some("host:18080").
assert!(server_to_grpc_address("host:abc.18080").is_none());
}
}
+1 -4
View File
@@ -16,10 +16,6 @@ name = "weed-worker"
path = "src/main.rs"
[dependencies]
# Helpers the Rust volume server needs as well. A path dependency because
# the two trees are separate cargo workspaces with no common root manifest.
# It also carries the rustls requirement this crate used to state itself.
seaweed-common = { path = "../../../seaweed-common" }
seaweed-worker-core = { path = "../core" }
seaweed-worker-sort = { path = "../sort" }
prometheus.workspace = true
@@ -40,6 +36,7 @@ anyhow.workspace = true
async-trait.workspace = true
clap = { version = "4", features = ["derive", "env"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
rustls = "0.23"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tokio.workspace = true
+6 -4
View File
@@ -1,10 +1,12 @@
use rustls::crypto::aws_lc_rs;
// aws-lc-rs and ring both get linked transitively (lance's aws backend pulls
// aws-lc-rs, reqwest's rustls-tls pulls ring), so rustls can't auto-select a
// provider and tonic's client TLS panics on first use. Pin the default to
// aws-lc-rs. The body lives in seaweed-common so this binary and the Rust
// volume server cannot end up installing different providers; re-exported here
// so callers keep their import path.
pub use seaweed_common::tls::install_default_crypto_provider;
// aws-lc-rs, matching the Rust volume server. Idempotent.
pub fn install_default_crypto_provider() {
let _ = aws_lc_rs::default_provider().install_default();
}
#[cfg(test)]
mod tests {
-291
View File
@@ -1,291 +0,0 @@
package checksum_test
import (
"bytes"
"context"
"crypto/sha256"
"encoding/base64"
"errors"
"testing"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types"
"github.com/aws/smithy-go"
"github.com/stretchr/testify/require"
)
// newWhenRequiredChecksumClient returns a client that only sends checksum
// headers when the request asks for them, so tests can upload parts with no
// checksum headers at all.
func newWhenRequiredChecksumClient(t *testing.T) *s3.Client {
t.Helper()
cfg, err := config.LoadDefaultConfig(context.TODO(),
config.WithRegion(defaultConfig.Region),
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(
defaultConfig.AccessKey, defaultConfig.SecretKey, "")),
)
require.NoError(t, err)
return s3.NewFromConfig(cfg, func(o *s3.Options) {
o.UsePathStyle = true
o.BaseEndpoint = aws.String(defaultConfig.Endpoint)
o.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired
})
}
// UploadPart without checksum headers inherits the algorithm declared at
// CreateMultipartUpload, so the part still gets a checksum and the upload can
// be completed with it (AWS behavior).
func TestMultipartPartInheritsChecksumAlgorithm(t *testing.T) {
client := newWhenRequiredChecksumClient(t)
bucket := uniqueBucket()
createBucket(t, client, bucket)
defer cleanupBucket(t, client, bucket)
body := bytes.Repeat([]byte("y"), 1024)
key := "inherit-algo"
create, err := client.CreateMultipartUpload(context.Background(), &s3.CreateMultipartUploadInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
ChecksumAlgorithm: types.ChecksumAlgorithmSha256,
})
require.NoError(t, err)
part, err := client.UploadPart(context.Background(), &s3.UploadPartInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
UploadId: create.UploadId,
PartNumber: aws.Int32(1),
Body: bytes.NewReader(body),
})
require.NoError(t, err)
require.NotEmpty(t, aws.ToString(part.ChecksumSHA256))
done, err := client.CompleteMultipartUpload(context.Background(), &s3.CompleteMultipartUploadInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
UploadId: create.UploadId,
MultipartUpload: &types.CompletedMultipartUpload{Parts: []types.CompletedPart{{
ETag: part.ETag,
PartNumber: aws.Int32(1),
ChecksumSHA256: part.ChecksumSHA256,
}}},
})
require.NoError(t, err)
require.Equal(t, types.ChecksumTypeComposite, done.ChecksumType)
require.NotEmpty(t, aws.ToString(done.ChecksumSHA256))
}
// Mimics the .NET repro in https://github.com/seaweedfs/seaweedfs/issues/11401:
// UploadPart carries an explicit client-computed x-amz-checksum-sha256 value,
// CompleteMultipartUpload echoes it back per part, and HeadObject returns the
// object checksum when x-amz-checksum-mode: ENABLED is sent (same as AWS).
func TestIssue11401(t *testing.T) {
client := getS3Client(t)
bucket := uniqueBucket()
createBucket(t, client, bucket)
defer cleanupBucket(t, client, bucket)
body := bytes.Repeat([]byte("x"), 1024)
checksum := base64.StdEncoding.EncodeToString(func() []byte { s := sha256.Sum256(body); return s[:] }())
key := "issue-11401"
create, err := client.CreateMultipartUpload(context.Background(), &s3.CreateMultipartUploadInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
ContentType: aws.String("application/octet-stream"),
ChecksumAlgorithm: types.ChecksumAlgorithmSha256,
})
require.NoError(t, err)
part, err := client.UploadPart(context.Background(), &s3.UploadPartInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
UploadId: create.UploadId,
PartNumber: aws.Int32(1),
Body: bytes.NewReader(body),
ChecksumSHA256: aws.String(checksum),
})
require.NoError(t, err)
require.Equal(t, checksum, aws.ToString(part.ChecksumSHA256))
done, err := client.CompleteMultipartUpload(context.Background(), &s3.CompleteMultipartUploadInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
UploadId: create.UploadId,
MultipartUpload: &types.CompletedMultipartUpload{Parts: []types.CompletedPart{{
ETag: part.ETag,
PartNumber: aws.Int32(1),
ChecksumSHA256: part.ChecksumSHA256,
}}},
})
require.NoError(t, err)
require.Equal(t, types.ChecksumTypeComposite, done.ChecksumType)
require.NotEmpty(t, aws.ToString(done.ChecksumSHA256))
head, err := client.HeadObject(context.Background(), &s3.HeadObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
ChecksumMode: types.ChecksumModeEnabled,
})
require.NoError(t, err)
require.Equal(t, aws.ToString(done.ChecksumSHA256), aws.ToString(head.ChecksumSHA256))
headNoMode, err := client.HeadObject(context.Background(), &s3.HeadObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
})
require.NoError(t, err)
require.Empty(t, aws.ToString(headNoMode.ChecksumSHA256))
}
// CompleteMultipartUpload validates the per-part checksums it is given, like
// AWS: missing checksums fail with InvalidRequest and wrong ones with
// BadDigest.
func TestCompleteMultipartUploadValidatesPartChecksums(t *testing.T) {
client := newWhenRequiredChecksumClient(t)
bucket := uniqueBucket()
createBucket(t, client, bucket)
defer cleanupBucket(t, client, bucket)
body := bytes.Repeat([]byte("z"), 1024)
key := "validate-parts"
create, err := client.CreateMultipartUpload(context.Background(), &s3.CreateMultipartUploadInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
ChecksumAlgorithm: types.ChecksumAlgorithmSha256,
})
require.NoError(t, err)
part, err := client.UploadPart(context.Background(), &s3.UploadPartInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
UploadId: create.UploadId,
PartNumber: aws.Int32(1),
Body: bytes.NewReader(body),
})
require.NoError(t, err)
require.NotEmpty(t, aws.ToString(part.ChecksumSHA256))
complete := func(checksum *string) error {
_, err := client.CompleteMultipartUpload(context.Background(), &s3.CompleteMultipartUploadInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
UploadId: create.UploadId,
MultipartUpload: &types.CompletedMultipartUpload{Parts: []types.CompletedPart{{
ETag: part.ETag,
PartNumber: aws.Int32(1),
ChecksumSHA256: checksum,
}}},
})
return err
}
var apiErr smithy.APIError
err = complete(nil)
require.Error(t, err)
require.True(t, errors.As(err, &apiErr))
require.Equal(t, "InvalidRequest", apiErr.ErrorCode())
wrong := base64.StdEncoding.EncodeToString(func() []byte { s := sha256.Sum256([]byte("other")); return s[:] }())
err = complete(aws.String(wrong))
require.Error(t, err)
require.True(t, errors.As(err, &apiErr))
require.Equal(t, "BadDigest", apiErr.ErrorCode())
require.NoError(t, complete(part.ChecksumSHA256))
}
// FULL_OBJECT uploads need no per-part checksums in the complete request; the
// whole-object checksum travels in a request header and is validated against
// the computed value (BadDigest on mismatch), as on AWS.
func TestCompleteMultipartUploadFullObjectChecksum(t *testing.T) {
client := newWhenRequiredChecksumClient(t)
bucket := uniqueBucket()
createBucket(t, client, bucket)
defer cleanupBucket(t, client, bucket)
body := bytes.Repeat([]byte("w"), 1024)
key := "full-object"
create, err := client.CreateMultipartUpload(context.Background(), &s3.CreateMultipartUploadInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
ChecksumAlgorithm: types.ChecksumAlgorithmCrc64nvme,
})
require.NoError(t, err)
require.Equal(t, types.ChecksumTypeFullObject, create.ChecksumType)
part, err := client.UploadPart(context.Background(), &s3.UploadPartInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
UploadId: create.UploadId,
PartNumber: aws.Int32(1),
Body: bytes.NewReader(body),
})
require.NoError(t, err)
require.NotEmpty(t, aws.ToString(part.ChecksumCRC64NVME))
complete := func(objectChecksum *string) error {
_, err := client.CompleteMultipartUpload(context.Background(), &s3.CompleteMultipartUploadInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
UploadId: create.UploadId,
ChecksumCRC64NVME: objectChecksum,
MultipartUpload: &types.CompletedMultipartUpload{Parts: []types.CompletedPart{{
ETag: part.ETag,
PartNumber: aws.Int32(1),
}}},
})
return err
}
var apiErr smithy.APIError
wrong := base64.StdEncoding.EncodeToString(func() []byte { s := sha256.Sum256(body); return s[:] }())
err = complete(aws.String(wrong))
require.Error(t, err)
require.True(t, errors.As(err, &apiErr))
require.Equal(t, "BadDigest", apiErr.ErrorCode())
require.NoError(t, complete(part.ChecksumCRC64NVME))
}
// UploadPart with a checksum algorithm that conflicts with the one declared at
// CreateMultipartUpload is rejected, as on AWS.
func TestMultipartPartConflictingAlgorithm(t *testing.T) {
client := newWhenRequiredChecksumClient(t)
bucket := uniqueBucket()
createBucket(t, client, bucket)
defer cleanupBucket(t, client, bucket)
create, err := client.CreateMultipartUpload(context.Background(), &s3.CreateMultipartUploadInput{
Bucket: aws.String(bucket),
Key: aws.String("conflict"),
ChecksumAlgorithm: types.ChecksumAlgorithmSha256,
})
require.NoError(t, err)
_, err = client.UploadPart(context.Background(), &s3.UploadPartInput{
Bucket: aws.String(bucket),
Key: aws.String("conflict"),
UploadId: create.UploadId,
PartNumber: aws.Int32(1),
Body: bytes.NewReader([]byte("data")),
ChecksumAlgorithm: types.ChecksumAlgorithmCrc32,
})
var apiErr smithy.APIError
require.Error(t, err)
require.True(t, errors.As(err, &apiErr))
require.Equal(t, "InvalidRequest", apiErr.ErrorCode())
}
-33
View File
@@ -1,33 +0,0 @@
# Snowflake S3Compat API test suite
Integration tests that run the upstream
[Snowflake s3compat API test suite](https://github.com/snowflakedb/snowflake-s3compat-api-test-suite)
against SeaweedFS. The suite covers `getBucketLocation`, `getObject` (including
range reads), `getObjectMetadata`, `putObject` (including a 5 GB upload),
`listObjectsV2` (including paged listing of >1000 objects), `deleteObject`,
`deleteObjects`, `copyObject`, and `generatePresignedUrl`.
## Running locally
Requires `weed` (or `WEED_BIN`), the `aws` CLI, `mvn`, and JDK 11+ on `PATH`.
```sh
(cd weed && go install -buildvcs=false) # build weed first
bash test/s3/snowflake/run.sh
```
`run.sh` starts a `weed server` with S3 enabled, calls `prepare.sh` to create
the fixtures, clones the suite into a scratch dir, and runs
`mvn -Dtest=S3CompatApiTest`. Set `WORK_DIR` to keep the server log and suite
clone around, `SKIP_SERVER_START=1` with `ENDPOINT_URL` to run against an
already-running server, and see the top of `run.sh` for the other overrides.
## Notes
- The server runs with `-s3.autoCreateBucket=false` so PUTs to a missing bucket
return `NoSuchBucket` like AWS; the suite asserts this.
- The suite forces virtual-hosted-style bucket addressing, which requires
wildcard DNS that does not exist for a local endpoint. `run.sh` switches it
to path-style access with a `sed` patch.
- `NOT_ACCESSIBLE_BUCKET` is a real bucket carrying a deny-all bucket policy,
which is how the suite's `AccessDenied` negative tests are satisfied.
-81
View File
@@ -1,81 +0,0 @@
#!/usr/bin/env bash
#
# Prepares the fixtures required by the Snowflake s3compat API test suite
# (https://github.com/snowflakedb/snowflake-s3compat-api-test-suite) against a
# running SeaweedFS S3 endpoint:
#
# BUCKET_NAME_1 versioning-enabled bucket the suite writes to
# NOT_ACCESSIBLE_BUCKET bucket with a deny-all bucket policy; the suite
# expects 403 AccessDenied for every operation on it
# PREFIX_FOR_PAGE_LISTING prefix under BUCKET_NAME_1 holding more than 1000
# objects (the suite asserts paged listing works)
#
# Requires: aws CLI on PATH, AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY set.
set -euo pipefail
ENDPOINT_URL="${ENDPOINT_URL:-http://127.0.0.1:8333}"
BUCKET_NAME_1="${BUCKET_NAME_1:-sf-snowflake-test}"
NOT_ACCESSIBLE_BUCKET="${NOT_ACCESSIBLE_BUCKET:-sf-denied-bucket}"
PREFIX_FOR_PAGE_LISTING="${PREFIX_FOR_PAGE_LISTING:-test-suite/page-listing/}"
PAGE_LISTING_TOTAL_SIZE="${PAGE_LISTING_TOTAL_SIZE:-1100}"
aws="aws --endpoint-url $ENDPOINT_URL"
echo "Creating test bucket $BUCKET_NAME_1 with versioning enabled"
$aws s3api create-bucket --bucket "$BUCKET_NAME_1"
$aws s3api put-bucket-versioning --bucket "$BUCKET_NAME_1" --versioning-configuration Status=Enabled
$aws s3api get-bucket-versioning --bucket "$BUCKET_NAME_1"
echo "Creating not-accessible bucket $NOT_ACCESSIBLE_BUCKET with a deny-all bucket policy"
$aws s3api create-bucket --bucket "$NOT_ACCESSIBLE_BUCKET"
POLICY_FILE="$(mktemp)"
cat > "$POLICY_FILE" <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyAll",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::${NOT_ACCESSIBLE_BUCKET}",
"arn:aws:s3:::${NOT_ACCESSIBLE_BUCKET}/*"
]
}
]
}
EOF
$aws s3api put-bucket-policy --bucket "$NOT_ACCESSIBLE_BUCKET" --policy "file://$POLICY_FILE"
rm -f "$POLICY_FILE"
if OUT="$($aws s3api get-bucket-location --bucket "$NOT_ACCESSIBLE_BUCKET" 2>&1)"; then
echo "ERROR: expected AccessDenied on $NOT_ACCESSIBLE_BUCKET, got success" >&2
exit 1
elif ! echo "$OUT" | grep -q "AccessDenied"; then
echo "ERROR: expected AccessDenied on $NOT_ACCESSIBLE_BUCKET, got: $OUT" >&2
exit 1
fi
echo "Verified $NOT_ACCESSIBLE_BUCKET denies access"
if [ "$PAGE_LISTING_TOTAL_SIZE" -le 1000 ]; then
echo "ERROR: PAGE_LISTING_TOTAL_SIZE must be > 1000, got $PAGE_LISTING_TOTAL_SIZE" >&2
exit 1
fi
echo "Uploading $PAGE_LISTING_TOTAL_SIZE objects to s3://$BUCKET_NAME_1/$PREFIX_FOR_PAGE_LISTING"
WORKDIR="$(mktemp -d)"
for i in $(seq 1 "$PAGE_LISTING_TOTAL_SIZE"); do
echo "object-$i" > "$WORKDIR/file_$(printf %05d "$i").txt"
done
$aws s3 sync "$WORKDIR" "s3://$BUCKET_NAME_1/$PREFIX_FOR_PAGE_LISTING" --quiet
rm -rf "$WORKDIR"
COUNT=$($aws s3 ls "s3://$BUCKET_NAME_1/$PREFIX_FOR_PAGE_LISTING" | wc -l | tr -d ' ')
if [ "$COUNT" != "$PAGE_LISTING_TOTAL_SIZE" ]; then
echo "ERROR: expected $PAGE_LISTING_TOTAL_SIZE objects under $PREFIX_FOR_PAGE_LISTING, found $COUNT" >&2
exit 1
fi
echo "Verified $COUNT objects under s3://$BUCKET_NAME_1/$PREFIX_FOR_PAGE_LISTING"
echo "Fixtures ready."
-140
View File
@@ -1,140 +0,0 @@
#!/usr/bin/env bash
#
# Runs the Snowflake s3compat API test suite
# (https://github.com/snowflakedb/snowflake-s3compat-api-test-suite) against a
# locally-built SeaweedFS server.
#
# Required on PATH: weed (or WEED_BIN), aws, mvn, java, git.
#
# Env overrides:
# WEED_BIN path to the weed binary (default: weed)
# WORK_DIR scratch dir for data + suite clone (default: mktemp -d,
# removed on success, kept on failure)
# MASTER_PORT master http port (default: 9333)
# VOLUME_PORT volume http port (default: 8080)
# FILER_PORT filer http port (default: 8888)
# S3_PORT s3 endpoint port (default: 8333)
# METRICS_PORT metrics http port (default: 9324)
# SKIP_SERVER_START if set, do not start weed; prepare and run the suite
# against ENDPOINT_URL
# SUITE_REPO git url of the test suite (default: upstream)
# SUITE_REV suite commit to check out (default: pinned SHA)
#
# The suite env vars (BUCKET_NAME_1, PREFIX_FOR_PAGE_LISTING,
# PAGE_LISTING_TOTAL_SIZE, NOT_ACCESSIBLE_BUCKET) default to the same values
# prepare.sh uses.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
WEED_BIN="${WEED_BIN:-weed}"
MASTER_PORT="${MASTER_PORT:-9333}"
VOLUME_PORT="${VOLUME_PORT:-8080}"
FILER_PORT="${FILER_PORT:-8888}"
S3_PORT="${S3_PORT:-8333}"
METRICS_PORT="${METRICS_PORT:-9324}"
ENDPOINT_URL="${ENDPOINT_URL:-http://127.0.0.1:$S3_PORT}"
SUITE_REPO="${SUITE_REPO:-https://github.com/snowflakedb/snowflake-s3compat-api-test-suite.git}"
# Pinned upstream revision verified against SeaweedFS; bump deliberately.
SUITE_REV="${SUITE_REV:-8ae535b35fff0d8a72e21bba4e51281ac991cab9}"
WORK_DIR_CREATED=""
if [ -z "${WORK_DIR:-}" ]; then
WORK_DIR="$(mktemp -d)"
WORK_DIR_CREATED=1
fi
export BUCKET_NAME_1="${BUCKET_NAME_1:-sf-snowflake-test}"
export NOT_ACCESSIBLE_BUCKET="${NOT_ACCESSIBLE_BUCKET:-sf-denied-bucket}"
export PREFIX_FOR_PAGE_LISTING="${PREFIX_FOR_PAGE_LISTING:-test-suite/page-listing/}"
export PAGE_LISTING_TOTAL_SIZE="${PAGE_LISTING_TOTAL_SIZE:-1100}"
export ENDPOINT_URL
# The suite reads credentials from these variables.
export S3COMPAT_ACCESS_KEY="${S3COMPAT_ACCESS_KEY:-snowflake_compat_access}"
export S3COMPAT_SECRET_KEY="${S3COMPAT_SECRET_KEY:-snowflake_compat_secret}"
export AWS_ACCESS_KEY_ID="$S3COMPAT_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="$S3COMPAT_SECRET_KEY"
WEED_PID=""
cleanup() {
status=$?
if [ -n "$WEED_PID" ]; then
kill "$WEED_PID" 2>/dev/null || true
sleep 2
kill -9 "$WEED_PID" 2>/dev/null || true
fi
if [ -n "$WORK_DIR_CREATED" ]; then
if [ "$status" -eq 0 ]; then
rm -rf "$WORK_DIR"
else
echo "Work dir kept for debugging: $WORK_DIR" >&2
fi
fi
}
trap cleanup EXIT
wait_for_url() {
local url="$1" name="$2"
for i in $(seq 1 30); do
if curl -s "$url" > /dev/null 2>&1; then
echo "$name is ready"
return 0
fi
echo "Waiting for $name... ($i/30)"
sleep 2
done
echo "ERROR: $name did not become ready" >&2
return 1
}
if [ -z "${SKIP_SERVER_START:-}" ]; then
WEED_DATA_DIR="$WORK_DIR/data"
mkdir -p "$WEED_DATA_DIR"
echo "Starting SeaweedFS (data dir: $WEED_DATA_DIR)"
"$WEED_BIN" server -filer -filer.maxMB=64 -s3 -ip 127.0.0.1 -ip.bind 127.0.0.1 \
-dir="$WEED_DATA_DIR" \
-master.raftHashicorp -master.electionTimeout 1s -master.volumeSizeLimitMB=5000 \
-volume.max=4 -volume.preStopSeconds=1 \
-master.peers=none \
-master.port="$MASTER_PORT" -volume.port="$VOLUME_PORT" -filer.port="$FILER_PORT" -s3.port="$S3_PORT" \
-metricsPort="$METRICS_PORT" \
-s3.allowDeleteBucketNotEmpty=true \
-s3.autoCreateBucket=false \
-s3.port.iceberg=0 -s3.port.lance=0 \
-s3.config="$SCRIPT_DIR/s3.json" \
> "$WORK_DIR/weed.log" 2>&1 &
WEED_PID=$!
wait_for_url "http://127.0.0.1:$MASTER_PORT/cluster/status" "Master server"
wait_for_url "http://127.0.0.1:$VOLUME_PORT/status" "Volume server"
wait_for_url "http://127.0.0.1:$FILER_PORT/" "Filer"
wait_for_url "$ENDPOINT_URL/" "S3 API"
echo "All SeaweedFS components are ready"
fi
"$SCRIPT_DIR/prepare.sh"
SUITE_DIR="$WORK_DIR/snowflake-s3compat-api-test-suite"
if [ ! -d "$SUITE_DIR" ]; then
git init -q "$SUITE_DIR"
git -C "$SUITE_DIR" remote add origin "$SUITE_REPO"
git -C "$SUITE_DIR" fetch -q --depth 1 origin "$SUITE_REV"
git -C "$SUITE_DIR" checkout -q FETCH_HEAD
fi
# The suite forces virtual-hosted style bucket addressing, which needs wildcard
# DNS (<bucket>.<endpoint>) that does not exist for a local endpoint. Switch it
# to path-style access, which SeaweedFS supports.
STORAGE_CLIENT="$SUITE_DIR/s3compatapi/src/main/java/com/snowflake/s3compatapitestsuite/compatapi/S3CompatStorageClient.java"
sed -i.bak 's/setPathStyleAccess(false)/setPathStyleAccess(true)/' "$STORAGE_CLIENT"
rm -f "$STORAGE_CLIENT.bak"
grep -q 'setPathStyleAccess(true)' "$STORAGE_CLIENT"
cd "$SUITE_DIR/s3compatapi"
END_POINT="$ENDPOINT_URL" \
REGION_1=us-east-1 \
REGION_2=us-west-2 \
mvn -B test -Dtest=S3CompatApiTest
-20
View File
@@ -1,20 +0,0 @@
{
"identities": [
{
"name": "snowflake_compat_admin",
"credentials": [
{
"accessKey": "snowflake_compat_access",
"secretKey": "snowflake_compat_secret"
}
],
"actions": [
"Admin",
"Read",
"List",
"Tagging",
"Write"
]
}
]
}
+1 -36
View File
@@ -20,8 +20,6 @@ import (
"github.com/seaweedfs/seaweedfs/weed/replication/source"
"github.com/seaweedfs/seaweedfs/weed/util"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"google.golang.org/protobuf/proto"
)
@@ -419,49 +417,16 @@ func shouldSendToRemote(entry *filer_pb.Entry) bool {
return false
}
// updateLocalEntry stamps the entry an event described with its RemoteEntry.
// The write carries IF_ENTRY_EQUAL over the event's entry: the filer deletes
// every stored chunk absent from an updated entry, so a snapshot older than
// the live entry (the file was rewritten while its upload was in flight, or
// the event is a replay) would delete the live chunks. A failed precondition
// means the filer moved past this event; the event that superseded it follows
// in the log and stamps the current entry, so the stale stamp is skipped the
// same way a superseded upload is.
func updateLocalEntry(filerClient filer_pb.FilerClient, dir string, entry *filer_pb.Entry, remoteEntry *filer_pb.RemoteEntry) error {
remoteEntry.LastLocalSyncTsNs = time.Now().UnixNano()
expected := proto.Clone(entry).(*filer_pb.Entry)
entry.RemoteEntry = remoteEntry
err := filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
return filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
_, err := client.UpdateEntry(context.Background(), &filer_pb.UpdateEntryRequest{
Directory: dir,
Entry: entry,
Condition: ifEntryEqual(expected),
})
return err
})
if isFailedPrecondition(err) {
glog.Errorf("skipping stale stamp of %s: %v", util.NewFullPath(dir, entry.Name), err)
return nil
}
return err
}
// ifEntryEqual builds the precondition that the stored entry still equals the
// one the event described: chunk fids, inline content, and metadata alike.
func ifEntryEqual(entry *filer_pb.Entry) *filer_pb.WriteCondition {
return &filer_pb.WriteCondition{
Clauses: []*filer_pb.WriteCondition_Clause{{Kind: filer_pb.WriteCondition_IF_ENTRY_EQUAL, ExpectedEntry: entry}},
}
}
// isFailedPrecondition reports a write condition the filer refused, through
// any wrapping WithFilerClient added.
func isFailedPrecondition(err error) bool {
if err == nil {
return false
}
st, ok := status.FromError(err)
return ok && st.Code() == codes.FailedPrecondition
}
func isMultipartUploadFile(dir string, name string) bool {
@@ -16,8 +16,6 @@ import (
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/util"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"google.golang.org/protobuf/proto"
)
@@ -810,41 +808,3 @@ func TestRenameDeleteOldKeyNotFoundStillWrites(t *testing.T) {
t.Errorf("writes = %+v, want the new key %s written", remote.writes, remote_storage.FormatLocation(wantWrite))
}
}
func TestIfEntryEqualCarriesTheEventEntry(t *testing.T) {
entry := &filer_pb.Entry{
Name: "f",
Content: []byte("inline"),
Chunks: []*filer_pb.FileChunk{
{FileId: "3,01a"},
{Fid: &filer_pb.FileId{VolumeId: 4, FileKey: 0x2b, Cookie: 0x0c}},
},
}
cond := ifEntryEqual(entry)
if len(cond.Clauses) != 1 || cond.Clauses[0].Kind != filer_pb.WriteCondition_IF_ENTRY_EQUAL {
t.Fatalf("condition = %v, want one IF_ENTRY_EQUAL clause", cond)
}
if got := cond.Clauses[0].ExpectedEntry; !proto.Equal(got, entry) {
t.Fatalf("expected_entry = %v, want %v", got, entry)
}
}
func TestIsFailedPrecondition(t *testing.T) {
refused := status.Error(codes.FailedPrecondition, "precondition failed: /buckets/b/f")
cases := []struct {
name string
err error
want bool
}{
{"nil", nil, false},
{"status", refused, true},
{"wrapped status", fmt.Errorf("update entry: %w", refused), true},
{"message only", errors.New("rpc error: code = FailedPrecondition desc = precondition failed: /f"), false},
{"other", status.Error(codes.Unavailable, "filer down"), false},
}
for _, c := range cases {
if got := isFailedPrecondition(c.err); got != c.want {
t.Errorf("%s: isFailedPrecondition = %v, want %v", c.name, got, c.want)
}
}
}
+5 -9
View File
@@ -10,19 +10,15 @@ import (
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
)
func WithVolumeServerClient(streamingMode bool, volumeServer pb.ServerAddress, grpcDialOption grpc.DialOption, fn func(volume_server_pb.VolumeServerClient) error) error {
return WithVolumeServerClientOptions(streamingMode, volumeServer, fn, grpcDialOption)
}
// WithVolumeServerClientOptions is WithVolumeServerClient with extra dial
// options appended after the TLS option, so a caller dialing an untrusted
// source address can pin the validated endpoint at connect time.
func WithVolumeServerClientOptions(streamingMode bool, volumeServer pb.ServerAddress, fn func(volume_server_pb.VolumeServerClient) error, grpcDialOptions ...grpc.DialOption) error {
// WithVolumeServerClient dials with the TLS option plus any extra dial options
// appended after it, so a caller dialing an untrusted source address can pin
// the validated endpoint at connect time.
func WithVolumeServerClient(streamingMode bool, volumeServer pb.ServerAddress, grpcDialOption grpc.DialOption, fn func(volume_server_pb.VolumeServerClient) error, extraDialOptions ...grpc.DialOption) error {
return pb.WithGrpcClient(context.Background(), streamingMode, 0, func(grpcConnection *grpc.ClientConn) error {
client := volume_server_pb.NewVolumeServerClient(grpcConnection)
return fn(client)
}, volumeServer.ToGrpcAddress(), false, grpcDialOptions...)
}, volumeServer.ToGrpcAddress(), false, append([]grpc.DialOption{grpcDialOption}, extraDialOptions...)...)
}
+4 -4
View File
@@ -25,11 +25,11 @@ func TailVolume(masterFn GetMasterFn, grpcDialOption grpc.DialOption, vid needle
volumeServer := lookup.Locations[0].ServerAddress()
return TailVolumeFromSource(volumeServer, vid, sinceNs, timeoutSeconds, fn, grpcDialOption)
return TailVolumeFromSource(volumeServer, vid, sinceNs, timeoutSeconds, grpcDialOption, fn)
}
func TailVolumeFromSource(volumeServer pb.ServerAddress, vid needle.VolumeId, sinceNs uint64, idleTimeoutSeconds int, fn func(n *needle.Needle) error, grpcDialOptions ...grpc.DialOption) error {
return WithVolumeServerClientOptions(true, volumeServer, func(client volume_server_pb.VolumeServerClient) error {
func TailVolumeFromSource(volumeServer pb.ServerAddress, vid needle.VolumeId, sinceNs uint64, idleTimeoutSeconds int, grpcDialOption grpc.DialOption, fn func(n *needle.Needle) error, extraDialOptions ...grpc.DialOption) error {
return WithVolumeServerClient(true, volumeServer, grpcDialOption, func(client volume_server_pb.VolumeServerClient) error {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
@@ -90,5 +90,5 @@ func TailVolumeFromSource(volumeServer pb.ServerAddress, vid needle.VolumeId, si
}
return nil
}, grpcDialOptions...)
}, extraDialOptions...)
}
-3
View File
@@ -281,7 +281,6 @@ message WriteCondition {
IF_EXTENDED_NOT_EQUAL = 7; // fail if present and extended[ext_key] == ext_value
IF_EXTENDED_TIME_ELAPSED = 8; // fail if present and extended[ext_key] (unix seconds) is in the future
IF_CHUNKS_EQUAL = 9; // fail unless the stored chunk fid multiset equals fids (absent entry = no chunks)
IF_ENTRY_EQUAL = 10; // fail unless the stored entry equals expected_entry
}
// Clause is one primitive comparison. IF_ETAG_MATCH holds when the current
// entry's ETag equals any value in etags; IF_ETAG_NOT_MATCH holds when it
@@ -310,7 +309,6 @@ message WriteCondition {
string gate_key = 7; // IF_EXTENDED_TIME_ELAPSED: only enforce when extended[gate_key] == gate_value
string gate_value = 8; // gate value (e.g. retention mode COMPLIANCE for governance bypass)
repeated string fids = 9; // chunk fid strings for IF_CHUNKS_EQUAL
Entry expected_entry = 10; // whole-entry comparison for IF_ENTRY_EQUAL
}
repeated Clause clauses = 1; // all must hold (logical AND)
}
@@ -479,7 +477,6 @@ message UpdateEntryRequest {
// the write, under the filer's per-path lock. The caller must route the
// key's writes to this entry's owner filer for the check to be authoritative.
WriteCondition condition = 6;
bool is_moved = 7; // set on a forwarded request so the owner applies it locally
}
message UpdateEntryResponse {
SubscribeMetadataResponse metadata_event = 1;
+112 -137
View File
@@ -1,7 +1,7 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.36.6
// protoc v3.21.12
// protoc v7.35.0
// source: filer.proto
package filer_pb
@@ -197,33 +197,31 @@ func (PosixLockOp) EnumDescriptor() ([]byte, []int) {
type WriteCondition_Kind int32
const (
WriteCondition_NONE WriteCondition_Kind = 0 // unconditional
WriteCondition_IF_NOT_EXISTS WriteCondition_Kind = 1 // fail if the entry exists (If-None-Match: *)
WriteCondition_IF_EXISTS WriteCondition_Kind = 2 // fail if the entry is absent (If-Match: *)
WriteCondition_IF_ETAG_MATCH WriteCondition_Kind = 3 // fail if absent or etag matches none of the set (If-Match)
WriteCondition_IF_ETAG_NOT_MATCH WriteCondition_Kind = 4 // fail if present and etag matches any of the set (If-None-Match)
WriteCondition_IF_UNMODIFIED_SINCE WriteCondition_Kind = 5 // fail if present and mtime > unix_time
WriteCondition_IF_MODIFIED_SINCE WriteCondition_Kind = 6 // fail if present and mtime <= unix_time
WriteCondition_IF_EXTENDED_NOT_EQUAL WriteCondition_Kind = 7 // fail if present and extended[ext_key] == ext_value
WriteCondition_IF_EXTENDED_TIME_ELAPSED WriteCondition_Kind = 8 // fail if present and extended[ext_key] (unix seconds) is in the future
WriteCondition_IF_CHUNKS_EQUAL WriteCondition_Kind = 9 // fail unless the stored chunk fid multiset equals fids (absent entry = no chunks)
WriteCondition_IF_ENTRY_EQUAL WriteCondition_Kind = 10 // fail unless the stored entry equals expected_entry
WriteCondition_NONE WriteCondition_Kind = 0 // unconditional
WriteCondition_IF_NOT_EXISTS WriteCondition_Kind = 1 // fail if the entry exists (If-None-Match: *)
WriteCondition_IF_EXISTS WriteCondition_Kind = 2 // fail if the entry is absent (If-Match: *)
WriteCondition_IF_ETAG_MATCH WriteCondition_Kind = 3 // fail if absent or etag matches none of the set (If-Match)
WriteCondition_IF_ETAG_NOT_MATCH WriteCondition_Kind = 4 // fail if present and etag matches any of the set (If-None-Match)
WriteCondition_IF_UNMODIFIED_SINCE WriteCondition_Kind = 5 // fail if present and mtime > unix_time
WriteCondition_IF_MODIFIED_SINCE WriteCondition_Kind = 6 // fail if present and mtime <= unix_time
WriteCondition_IF_EXTENDED_NOT_EQUAL WriteCondition_Kind = 7 // fail if present and extended[ext_key] == ext_value
WriteCondition_IF_EXTENDED_TIME_ELAPSED WriteCondition_Kind = 8 // fail if present and extended[ext_key] (unix seconds) is in the future
WriteCondition_IF_CHUNKS_EQUAL WriteCondition_Kind = 9 // fail unless the stored chunk fid multiset equals fids (absent entry = no chunks)
)
// Enum value maps for WriteCondition_Kind.
var (
WriteCondition_Kind_name = map[int32]string{
0: "NONE",
1: "IF_NOT_EXISTS",
2: "IF_EXISTS",
3: "IF_ETAG_MATCH",
4: "IF_ETAG_NOT_MATCH",
5: "IF_UNMODIFIED_SINCE",
6: "IF_MODIFIED_SINCE",
7: "IF_EXTENDED_NOT_EQUAL",
8: "IF_EXTENDED_TIME_ELAPSED",
9: "IF_CHUNKS_EQUAL",
10: "IF_ENTRY_EQUAL",
0: "NONE",
1: "IF_NOT_EXISTS",
2: "IF_EXISTS",
3: "IF_ETAG_MATCH",
4: "IF_ETAG_NOT_MATCH",
5: "IF_UNMODIFIED_SINCE",
6: "IF_MODIFIED_SINCE",
7: "IF_EXTENDED_NOT_EQUAL",
8: "IF_EXTENDED_TIME_ELAPSED",
9: "IF_CHUNKS_EQUAL",
}
WriteCondition_Kind_value = map[string]int32{
"NONE": 0,
@@ -236,7 +234,6 @@ var (
"IF_EXTENDED_NOT_EQUAL": 7,
"IF_EXTENDED_TIME_ELAPSED": 8,
"IF_CHUNKS_EQUAL": 9,
"IF_ENTRY_EQUAL": 10,
}
)
@@ -2378,7 +2375,6 @@ type UpdateEntryRequest struct {
// the write, under the filer's per-path lock. The caller must route the
// key's writes to this entry's owner filer for the check to be authoritative.
Condition *WriteCondition `protobuf:"bytes,6,opt,name=condition,proto3" json:"condition,omitempty"`
IsMoved bool `protobuf:"varint,7,opt,name=is_moved,json=isMoved,proto3" json:"is_moved,omitempty"` // set on a forwarded request so the owner applies it locally
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -2455,13 +2451,6 @@ func (x *UpdateEntryRequest) GetCondition() *WriteCondition {
return nil
}
func (x *UpdateEntryRequest) GetIsMoved() bool {
if x != nil {
return x.IsMoved
}
return false
}
type UpdateEntryResponse struct {
state protoimpl.MessageState `protogen:"open.v1"`
MetadataEvent *SubscribeMetadataResponse `protobuf:"bytes,1,opt,name=metadata_event,json=metadataEvent,proto3" json:"metadata_event,omitempty"`
@@ -6867,15 +6856,14 @@ func (x *LookupDirectoryEntriesResponse) GetReadAuth() map[string]string {
type WriteCondition_Clause struct {
state protoimpl.MessageState `protogen:"open.v1"`
Kind WriteCondition_Kind `protobuf:"varint,1,opt,name=kind,proto3,enum=filer_pb.WriteCondition_Kind" json:"kind,omitempty"`
Etags []string `protobuf:"bytes,2,rep,name=etags,proto3" json:"etags,omitempty"` // ETag set for IF_ETAG_* kinds
UnixTime int64 `protobuf:"varint,3,opt,name=unix_time,json=unixTime,proto3" json:"unix_time,omitempty"` // bound (unix seconds) for IF_*_SINCE kinds
AllowWeak bool `protobuf:"varint,4,opt,name=allow_weak,json=allowWeak,proto3" json:"allow_weak,omitempty"` // compare ETags ignoring the weak (W/) marker
ExtKey string `protobuf:"bytes,5,opt,name=ext_key,json=extKey,proto3" json:"ext_key,omitempty"` // extended attribute name for IF_EXTENDED_* kinds
ExtValue string `protobuf:"bytes,6,opt,name=ext_value,json=extValue,proto3" json:"ext_value,omitempty"` // blocking value for IF_EXTENDED_NOT_EQUAL
GateKey string `protobuf:"bytes,7,opt,name=gate_key,json=gateKey,proto3" json:"gate_key,omitempty"` // IF_EXTENDED_TIME_ELAPSED: only enforce when extended[gate_key] == gate_value
GateValue string `protobuf:"bytes,8,opt,name=gate_value,json=gateValue,proto3" json:"gate_value,omitempty"` // gate value (e.g. retention mode COMPLIANCE for governance bypass)
Fids []string `protobuf:"bytes,9,rep,name=fids,proto3" json:"fids,omitempty"` // chunk fid strings for IF_CHUNKS_EQUAL
ExpectedEntry *Entry `protobuf:"bytes,10,opt,name=expected_entry,json=expectedEntry,proto3" json:"expected_entry,omitempty"` // whole-entry comparison for IF_ENTRY_EQUAL
Etags []string `protobuf:"bytes,2,rep,name=etags,proto3" json:"etags,omitempty"` // ETag set for IF_ETAG_* kinds
UnixTime int64 `protobuf:"varint,3,opt,name=unix_time,json=unixTime,proto3" json:"unix_time,omitempty"` // bound (unix seconds) for IF_*_SINCE kinds
AllowWeak bool `protobuf:"varint,4,opt,name=allow_weak,json=allowWeak,proto3" json:"allow_weak,omitempty"` // compare ETags ignoring the weak (W/) marker
ExtKey string `protobuf:"bytes,5,opt,name=ext_key,json=extKey,proto3" json:"ext_key,omitempty"` // extended attribute name for IF_EXTENDED_* kinds
ExtValue string `protobuf:"bytes,6,opt,name=ext_value,json=extValue,proto3" json:"ext_value,omitempty"` // blocking value for IF_EXTENDED_NOT_EQUAL
GateKey string `protobuf:"bytes,7,opt,name=gate_key,json=gateKey,proto3" json:"gate_key,omitempty"` // IF_EXTENDED_TIME_ELAPSED: only enforce when extended[gate_key] == gate_value
GateValue string `protobuf:"bytes,8,opt,name=gate_value,json=gateValue,proto3" json:"gate_value,omitempty"` // gate value (e.g. retention mode COMPLIANCE for governance bypass)
Fids []string `protobuf:"bytes,9,rep,name=fids,proto3" json:"fids,omitempty"` // chunk fid strings for IF_CHUNKS_EQUAL
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -6973,13 +6961,6 @@ func (x *WriteCondition_Clause) GetFids() []string {
return nil
}
func (x *WriteCondition_Clause) GetExpectedEntry() *Entry {
if x != nil {
return x.ExpectedEntry
}
return nil
}
// if found, send the exact address
// if not found, send the full list of existing brokers
type LocateBrokerResponse_Resource struct {
@@ -7320,9 +7301,9 @@ const file_filer_proto_rawDesc = "" +
"signatures\x12=\n" +
"\x1bskip_check_parent_directory\x18\x06 \x01(\bR\x18skipCheckParentDirectory\x126\n" +
"\tcondition\x18\a \x01(\v2\x18.filer_pb.WriteConditionR\tcondition\x12\x19\n" +
"\bis_moved\x18\b \x01(\bR\aisMoved\"\x88\x05\n" +
"\bis_moved\x18\b \x01(\bR\aisMoved\"\xbc\x04\n" +
"\x0eWriteCondition\x129\n" +
"\aclauses\x18\x01 \x03(\v2\x1f.filer_pb.WriteCondition.ClauseR\aclauses\x1a\xc9\x02\n" +
"\aclauses\x18\x01 \x03(\v2\x1f.filer_pb.WriteCondition.ClauseR\aclauses\x1a\x91\x02\n" +
"\x06Clause\x121\n" +
"\x04kind\x18\x01 \x01(\x0e2\x1d.filer_pb.WriteCondition.KindR\x04kind\x12\x14\n" +
"\x05etags\x18\x02 \x03(\tR\x05etags\x12\x1b\n" +
@@ -7334,9 +7315,7 @@ const file_filer_proto_rawDesc = "" +
"\bgate_key\x18\a \x01(\tR\agateKey\x12\x1d\n" +
"\n" +
"gate_value\x18\b \x01(\tR\tgateValue\x12\x12\n" +
"\x04fids\x18\t \x03(\tR\x04fids\x126\n" +
"\x0eexpected_entry\x18\n" +
" \x01(\v2\x0f.filer_pb.EntryR\rexpectedEntry\"\xee\x01\n" +
"\x04fids\x18\t \x03(\tR\x04fids\"\xda\x01\n" +
"\x04Kind\x12\b\n" +
"\x04NONE\x10\x00\x12\x11\n" +
"\rIF_NOT_EXISTS\x10\x01\x12\r\n" +
@@ -7347,9 +7326,7 @@ const file_filer_proto_rawDesc = "" +
"\x11IF_MODIFIED_SINCE\x10\x06\x12\x19\n" +
"\x15IF_EXTENDED_NOT_EQUAL\x10\a\x12\x1c\n" +
"\x18IF_EXTENDED_TIME_ELAPSED\x10\b\x12\x13\n" +
"\x0fIF_CHUNKS_EQUAL\x10\t\x12\x12\n" +
"\x0eIF_ENTRY_EQUAL\x10\n" +
"\"\xa2\x05\n" +
"\x0fIF_CHUNKS_EQUAL\x10\t\"\xa2\x05\n" +
"\x0eObjectMutation\x121\n" +
"\x04type\x18\x01 \x01(\x0e2\x1d.filer_pb.ObjectMutation.TypeR\x04type\x12\x1c\n" +
"\tdirectory\x18\x02 \x01(\tR\tdirectory\x12\x12\n" +
@@ -7437,7 +7414,7 @@ const file_filer_proto_rawDesc = "" +
"\n" +
"error_code\x18\x03 \x01(\x0e2\x14.filer_pb.FilerErrorR\terrorCode\x12\x1a\n" +
"\tlog_ts_ns\x18\x04 \x01(\x03R\alogTsNs\x12#\n" +
"\rlog_signature\x18\x05 \x01(\x05R\flogSignature\"\xa5\x03\n" +
"\rlog_signature\x18\x05 \x01(\x05R\flogSignature\"\x8a\x03\n" +
"\x12UpdateEntryRequest\x12\x1c\n" +
"\tdirectory\x18\x01 \x01(\tR\tdirectory\x12%\n" +
"\x05entry\x18\x02 \x01(\v2\x0f.filer_pb.EntryR\x05entry\x121\n" +
@@ -7446,8 +7423,7 @@ const file_filer_proto_rawDesc = "" +
"signatures\x18\x04 \x03(\x05R\n" +
"signatures\x12_\n" +
"\x11expected_extended\x18\x05 \x03(\v22.filer_pb.UpdateEntryRequest.ExpectedExtendedEntryR\x10expectedExtended\x126\n" +
"\tcondition\x18\x06 \x01(\v2\x18.filer_pb.WriteConditionR\tcondition\x12\x19\n" +
"\bis_moved\x18\a \x01(\bR\aisMoved\x1aC\n" +
"\tcondition\x18\x06 \x01(\v2\x18.filer_pb.WriteConditionR\tcondition\x1aC\n" +
"\x15ExpectedExtendedEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\fR\x05value:\x028\x01\"\xa2\x01\n" +
@@ -8096,84 +8072,83 @@ var file_filer_proto_depIdxs = []int32{
107, // 67: filer_pb.LookupDirectoryEntriesResponse.locations_map:type_name -> filer_pb.LookupDirectoryEntriesResponse.LocationsMapEntry
108, // 68: filer_pb.LookupDirectoryEntriesResponse.read_auth:type_name -> filer_pb.LookupDirectoryEntriesResponse.ReadAuthEntry
3, // 69: filer_pb.WriteCondition.Clause.kind:type_name -> filer_pb.WriteCondition.Kind
10, // 70: filer_pb.WriteCondition.Clause.expected_entry:type_name -> filer_pb.Entry
44, // 71: filer_pb.LookupVolumeResponse.LocationsMapEntry.value:type_name -> filer_pb.Locations
44, // 72: filer_pb.LookupDirectoryEntriesResponse.LocationsMapEntry.value:type_name -> filer_pb.Locations
5, // 73: filer_pb.SeaweedFiler.LookupDirectoryEntry:input_type -> filer_pb.LookupDirectoryEntryRequest
96, // 74: filer_pb.SeaweedFiler.LookupDirectoryEntries:input_type -> filer_pb.LookupDirectoryEntriesRequest
7, // 75: filer_pb.SeaweedFiler.ListEntries:input_type -> filer_pb.ListEntriesRequest
17, // 76: filer_pb.SeaweedFiler.CreateEntry:input_type -> filer_pb.CreateEntryRequest
29, // 77: filer_pb.SeaweedFiler.UpdateEntry:input_type -> filer_pb.UpdateEntryRequest
31, // 78: filer_pb.SeaweedFiler.TouchAccessTime:input_type -> filer_pb.TouchAccessTimeRequest
33, // 79: filer_pb.SeaweedFiler.AppendToEntry:input_type -> filer_pb.AppendToEntryRequest
35, // 80: filer_pb.SeaweedFiler.DeleteEntry:input_type -> filer_pb.DeleteEntryRequest
21, // 81: filer_pb.SeaweedFiler.ObjectTransaction:input_type -> filer_pb.ObjectTransactionRequest
26, // 82: filer_pb.SeaweedFiler.ObjectTransactionBatch:input_type -> filer_pb.ObjectTransactionBatchRequest
24, // 83: filer_pb.SeaweedFiler.PosixLock:input_type -> filer_pb.PosixLockRequest
37, // 84: filer_pb.SeaweedFiler.AtomicRenameEntry:input_type -> filer_pb.AtomicRenameEntryRequest
39, // 85: filer_pb.SeaweedFiler.StreamRenameEntry:input_type -> filer_pb.StreamRenameEntryRequest
89, // 86: filer_pb.SeaweedFiler.StreamMutateEntry:input_type -> filer_pb.StreamMutateEntryRequest
41, // 87: filer_pb.SeaweedFiler.AssignVolume:input_type -> filer_pb.AssignVolumeRequest
43, // 88: filer_pb.SeaweedFiler.LookupVolume:input_type -> filer_pb.LookupVolumeRequest
48, // 89: filer_pb.SeaweedFiler.CollectionList:input_type -> filer_pb.CollectionListRequest
50, // 90: filer_pb.SeaweedFiler.DeleteCollection:input_type -> filer_pb.DeleteCollectionRequest
52, // 91: filer_pb.SeaweedFiler.Statistics:input_type -> filer_pb.StatisticsRequest
54, // 92: filer_pb.SeaweedFiler.Ping:input_type -> filer_pb.PingRequest
56, // 93: filer_pb.SeaweedFiler.GetFilerConfiguration:input_type -> filer_pb.GetFilerConfigurationRequest
64, // 94: filer_pb.SeaweedFiler.TraverseBfsMetadata:input_type -> filer_pb.TraverseBfsMetadataRequest
58, // 95: filer_pb.SeaweedFiler.SubscribeMetadata:input_type -> filer_pb.SubscribeMetadataRequest
58, // 96: filer_pb.SeaweedFiler.SubscribeLocalMetadata:input_type -> filer_pb.SubscribeMetadataRequest
60, // 97: filer_pb.SeaweedFiler.ListMetadataSubscribers:input_type -> filer_pb.ListMetadataSubscribersRequest
71, // 98: filer_pb.SeaweedFiler.KvGet:input_type -> filer_pb.KvGetRequest
73, // 99: filer_pb.SeaweedFiler.KvPut:input_type -> filer_pb.KvPutRequest
76, // 100: filer_pb.SeaweedFiler.CacheRemoteObjectToLocalCluster:input_type -> filer_pb.CacheRemoteObjectToLocalClusterRequest
78, // 101: filer_pb.SeaweedFiler.DistributedLock:input_type -> filer_pb.LockRequest
80, // 102: filer_pb.SeaweedFiler.DistributedUnlock:input_type -> filer_pb.UnlockRequest
82, // 103: filer_pb.SeaweedFiler.FindLockOwner:input_type -> filer_pb.FindLockOwnerRequest
85, // 104: filer_pb.SeaweedFiler.TransferLocks:input_type -> filer_pb.TransferLocksRequest
87, // 105: filer_pb.SeaweedFiler.ReplicateLock:input_type -> filer_pb.ReplicateLockRequest
91, // 106: filer_pb.SeaweedFiler.MountRegister:input_type -> filer_pb.MountRegisterRequest
93, // 107: filer_pb.SeaweedFiler.MountList:input_type -> filer_pb.MountListRequest
6, // 108: filer_pb.SeaweedFiler.LookupDirectoryEntry:output_type -> filer_pb.LookupDirectoryEntryResponse
98, // 109: filer_pb.SeaweedFiler.LookupDirectoryEntries:output_type -> filer_pb.LookupDirectoryEntriesResponse
8, // 110: filer_pb.SeaweedFiler.ListEntries:output_type -> filer_pb.ListEntriesResponse
28, // 111: filer_pb.SeaweedFiler.CreateEntry:output_type -> filer_pb.CreateEntryResponse
30, // 112: filer_pb.SeaweedFiler.UpdateEntry:output_type -> filer_pb.UpdateEntryResponse
32, // 113: filer_pb.SeaweedFiler.TouchAccessTime:output_type -> filer_pb.TouchAccessTimeResponse
34, // 114: filer_pb.SeaweedFiler.AppendToEntry:output_type -> filer_pb.AppendToEntryResponse
36, // 115: filer_pb.SeaweedFiler.DeleteEntry:output_type -> filer_pb.DeleteEntryResponse
22, // 116: filer_pb.SeaweedFiler.ObjectTransaction:output_type -> filer_pb.ObjectTransactionResponse
27, // 117: filer_pb.SeaweedFiler.ObjectTransactionBatch:output_type -> filer_pb.ObjectTransactionBatchResponse
25, // 118: filer_pb.SeaweedFiler.PosixLock:output_type -> filer_pb.PosixLockResponse
38, // 119: filer_pb.SeaweedFiler.AtomicRenameEntry:output_type -> filer_pb.AtomicRenameEntryResponse
40, // 120: filer_pb.SeaweedFiler.StreamRenameEntry:output_type -> filer_pb.StreamRenameEntryResponse
90, // 121: filer_pb.SeaweedFiler.StreamMutateEntry:output_type -> filer_pb.StreamMutateEntryResponse
42, // 122: filer_pb.SeaweedFiler.AssignVolume:output_type -> filer_pb.AssignVolumeResponse
46, // 123: filer_pb.SeaweedFiler.LookupVolume:output_type -> filer_pb.LookupVolumeResponse
49, // 124: filer_pb.SeaweedFiler.CollectionList:output_type -> filer_pb.CollectionListResponse
51, // 125: filer_pb.SeaweedFiler.DeleteCollection:output_type -> filer_pb.DeleteCollectionResponse
53, // 126: filer_pb.SeaweedFiler.Statistics:output_type -> filer_pb.StatisticsResponse
55, // 127: filer_pb.SeaweedFiler.Ping:output_type -> filer_pb.PingResponse
57, // 128: filer_pb.SeaweedFiler.GetFilerConfiguration:output_type -> filer_pb.GetFilerConfigurationResponse
65, // 129: filer_pb.SeaweedFiler.TraverseBfsMetadata:output_type -> filer_pb.TraverseBfsMetadataResponse
59, // 130: filer_pb.SeaweedFiler.SubscribeMetadata:output_type -> filer_pb.SubscribeMetadataResponse
59, // 131: filer_pb.SeaweedFiler.SubscribeLocalMetadata:output_type -> filer_pb.SubscribeMetadataResponse
61, // 132: filer_pb.SeaweedFiler.ListMetadataSubscribers:output_type -> filer_pb.ListMetadataSubscribersResponse
72, // 133: filer_pb.SeaweedFiler.KvGet:output_type -> filer_pb.KvGetResponse
74, // 134: filer_pb.SeaweedFiler.KvPut:output_type -> filer_pb.KvPutResponse
77, // 135: filer_pb.SeaweedFiler.CacheRemoteObjectToLocalCluster:output_type -> filer_pb.CacheRemoteObjectToLocalClusterResponse
79, // 136: filer_pb.SeaweedFiler.DistributedLock:output_type -> filer_pb.LockResponse
81, // 137: filer_pb.SeaweedFiler.DistributedUnlock:output_type -> filer_pb.UnlockResponse
83, // 138: filer_pb.SeaweedFiler.FindLockOwner:output_type -> filer_pb.FindLockOwnerResponse
86, // 139: filer_pb.SeaweedFiler.TransferLocks:output_type -> filer_pb.TransferLocksResponse
88, // 140: filer_pb.SeaweedFiler.ReplicateLock:output_type -> filer_pb.ReplicateLockResponse
92, // 141: filer_pb.SeaweedFiler.MountRegister:output_type -> filer_pb.MountRegisterResponse
94, // 142: filer_pb.SeaweedFiler.MountList:output_type -> filer_pb.MountListResponse
108, // [108:143] is the sub-list for method output_type
73, // [73:108] is the sub-list for method input_type
73, // [73:73] is the sub-list for extension type_name
73, // [73:73] is the sub-list for extension extendee
0, // [0:73] is the sub-list for field type_name
44, // 70: filer_pb.LookupVolumeResponse.LocationsMapEntry.value:type_name -> filer_pb.Locations
44, // 71: filer_pb.LookupDirectoryEntriesResponse.LocationsMapEntry.value:type_name -> filer_pb.Locations
5, // 72: filer_pb.SeaweedFiler.LookupDirectoryEntry:input_type -> filer_pb.LookupDirectoryEntryRequest
96, // 73: filer_pb.SeaweedFiler.LookupDirectoryEntries:input_type -> filer_pb.LookupDirectoryEntriesRequest
7, // 74: filer_pb.SeaweedFiler.ListEntries:input_type -> filer_pb.ListEntriesRequest
17, // 75: filer_pb.SeaweedFiler.CreateEntry:input_type -> filer_pb.CreateEntryRequest
29, // 76: filer_pb.SeaweedFiler.UpdateEntry:input_type -> filer_pb.UpdateEntryRequest
31, // 77: filer_pb.SeaweedFiler.TouchAccessTime:input_type -> filer_pb.TouchAccessTimeRequest
33, // 78: filer_pb.SeaweedFiler.AppendToEntry:input_type -> filer_pb.AppendToEntryRequest
35, // 79: filer_pb.SeaweedFiler.DeleteEntry:input_type -> filer_pb.DeleteEntryRequest
21, // 80: filer_pb.SeaweedFiler.ObjectTransaction:input_type -> filer_pb.ObjectTransactionRequest
26, // 81: filer_pb.SeaweedFiler.ObjectTransactionBatch:input_type -> filer_pb.ObjectTransactionBatchRequest
24, // 82: filer_pb.SeaweedFiler.PosixLock:input_type -> filer_pb.PosixLockRequest
37, // 83: filer_pb.SeaweedFiler.AtomicRenameEntry:input_type -> filer_pb.AtomicRenameEntryRequest
39, // 84: filer_pb.SeaweedFiler.StreamRenameEntry:input_type -> filer_pb.StreamRenameEntryRequest
89, // 85: filer_pb.SeaweedFiler.StreamMutateEntry:input_type -> filer_pb.StreamMutateEntryRequest
41, // 86: filer_pb.SeaweedFiler.AssignVolume:input_type -> filer_pb.AssignVolumeRequest
43, // 87: filer_pb.SeaweedFiler.LookupVolume:input_type -> filer_pb.LookupVolumeRequest
48, // 88: filer_pb.SeaweedFiler.CollectionList:input_type -> filer_pb.CollectionListRequest
50, // 89: filer_pb.SeaweedFiler.DeleteCollection:input_type -> filer_pb.DeleteCollectionRequest
52, // 90: filer_pb.SeaweedFiler.Statistics:input_type -> filer_pb.StatisticsRequest
54, // 91: filer_pb.SeaweedFiler.Ping:input_type -> filer_pb.PingRequest
56, // 92: filer_pb.SeaweedFiler.GetFilerConfiguration:input_type -> filer_pb.GetFilerConfigurationRequest
64, // 93: filer_pb.SeaweedFiler.TraverseBfsMetadata:input_type -> filer_pb.TraverseBfsMetadataRequest
58, // 94: filer_pb.SeaweedFiler.SubscribeMetadata:input_type -> filer_pb.SubscribeMetadataRequest
58, // 95: filer_pb.SeaweedFiler.SubscribeLocalMetadata:input_type -> filer_pb.SubscribeMetadataRequest
60, // 96: filer_pb.SeaweedFiler.ListMetadataSubscribers:input_type -> filer_pb.ListMetadataSubscribersRequest
71, // 97: filer_pb.SeaweedFiler.KvGet:input_type -> filer_pb.KvGetRequest
73, // 98: filer_pb.SeaweedFiler.KvPut:input_type -> filer_pb.KvPutRequest
76, // 99: filer_pb.SeaweedFiler.CacheRemoteObjectToLocalCluster:input_type -> filer_pb.CacheRemoteObjectToLocalClusterRequest
78, // 100: filer_pb.SeaweedFiler.DistributedLock:input_type -> filer_pb.LockRequest
80, // 101: filer_pb.SeaweedFiler.DistributedUnlock:input_type -> filer_pb.UnlockRequest
82, // 102: filer_pb.SeaweedFiler.FindLockOwner:input_type -> filer_pb.FindLockOwnerRequest
85, // 103: filer_pb.SeaweedFiler.TransferLocks:input_type -> filer_pb.TransferLocksRequest
87, // 104: filer_pb.SeaweedFiler.ReplicateLock:input_type -> filer_pb.ReplicateLockRequest
91, // 105: filer_pb.SeaweedFiler.MountRegister:input_type -> filer_pb.MountRegisterRequest
93, // 106: filer_pb.SeaweedFiler.MountList:input_type -> filer_pb.MountListRequest
6, // 107: filer_pb.SeaweedFiler.LookupDirectoryEntry:output_type -> filer_pb.LookupDirectoryEntryResponse
98, // 108: filer_pb.SeaweedFiler.LookupDirectoryEntries:output_type -> filer_pb.LookupDirectoryEntriesResponse
8, // 109: filer_pb.SeaweedFiler.ListEntries:output_type -> filer_pb.ListEntriesResponse
28, // 110: filer_pb.SeaweedFiler.CreateEntry:output_type -> filer_pb.CreateEntryResponse
30, // 111: filer_pb.SeaweedFiler.UpdateEntry:output_type -> filer_pb.UpdateEntryResponse
32, // 112: filer_pb.SeaweedFiler.TouchAccessTime:output_type -> filer_pb.TouchAccessTimeResponse
34, // 113: filer_pb.SeaweedFiler.AppendToEntry:output_type -> filer_pb.AppendToEntryResponse
36, // 114: filer_pb.SeaweedFiler.DeleteEntry:output_type -> filer_pb.DeleteEntryResponse
22, // 115: filer_pb.SeaweedFiler.ObjectTransaction:output_type -> filer_pb.ObjectTransactionResponse
27, // 116: filer_pb.SeaweedFiler.ObjectTransactionBatch:output_type -> filer_pb.ObjectTransactionBatchResponse
25, // 117: filer_pb.SeaweedFiler.PosixLock:output_type -> filer_pb.PosixLockResponse
38, // 118: filer_pb.SeaweedFiler.AtomicRenameEntry:output_type -> filer_pb.AtomicRenameEntryResponse
40, // 119: filer_pb.SeaweedFiler.StreamRenameEntry:output_type -> filer_pb.StreamRenameEntryResponse
90, // 120: filer_pb.SeaweedFiler.StreamMutateEntry:output_type -> filer_pb.StreamMutateEntryResponse
42, // 121: filer_pb.SeaweedFiler.AssignVolume:output_type -> filer_pb.AssignVolumeResponse
46, // 122: filer_pb.SeaweedFiler.LookupVolume:output_type -> filer_pb.LookupVolumeResponse
49, // 123: filer_pb.SeaweedFiler.CollectionList:output_type -> filer_pb.CollectionListResponse
51, // 124: filer_pb.SeaweedFiler.DeleteCollection:output_type -> filer_pb.DeleteCollectionResponse
53, // 125: filer_pb.SeaweedFiler.Statistics:output_type -> filer_pb.StatisticsResponse
55, // 126: filer_pb.SeaweedFiler.Ping:output_type -> filer_pb.PingResponse
57, // 127: filer_pb.SeaweedFiler.GetFilerConfiguration:output_type -> filer_pb.GetFilerConfigurationResponse
65, // 128: filer_pb.SeaweedFiler.TraverseBfsMetadata:output_type -> filer_pb.TraverseBfsMetadataResponse
59, // 129: filer_pb.SeaweedFiler.SubscribeMetadata:output_type -> filer_pb.SubscribeMetadataResponse
59, // 130: filer_pb.SeaweedFiler.SubscribeLocalMetadata:output_type -> filer_pb.SubscribeMetadataResponse
61, // 131: filer_pb.SeaweedFiler.ListMetadataSubscribers:output_type -> filer_pb.ListMetadataSubscribersResponse
72, // 132: filer_pb.SeaweedFiler.KvGet:output_type -> filer_pb.KvGetResponse
74, // 133: filer_pb.SeaweedFiler.KvPut:output_type -> filer_pb.KvPutResponse
77, // 134: filer_pb.SeaweedFiler.CacheRemoteObjectToLocalCluster:output_type -> filer_pb.CacheRemoteObjectToLocalClusterResponse
79, // 135: filer_pb.SeaweedFiler.DistributedLock:output_type -> filer_pb.LockResponse
81, // 136: filer_pb.SeaweedFiler.DistributedUnlock:output_type -> filer_pb.UnlockResponse
83, // 137: filer_pb.SeaweedFiler.FindLockOwner:output_type -> filer_pb.FindLockOwnerResponse
86, // 138: filer_pb.SeaweedFiler.TransferLocks:output_type -> filer_pb.TransferLocksResponse
88, // 139: filer_pb.SeaweedFiler.ReplicateLock:output_type -> filer_pb.ReplicateLockResponse
92, // 140: filer_pb.SeaweedFiler.MountRegister:output_type -> filer_pb.MountRegisterResponse
94, // 141: filer_pb.SeaweedFiler.MountList:output_type -> filer_pb.MountListResponse
107, // [107:142] is the sub-list for method output_type
72, // [72:107] is the sub-list for method input_type
72, // [72:72] is the sub-list for extension type_name
72, // [72:72] is the sub-list for extension extendee
0, // [0:72] is the sub-list for field type_name
}
func init() { file_filer_proto_init() }
+1 -1
View File
@@ -1,7 +1,7 @@
// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
// versions:
// - protoc-gen-go-grpc v1.6.2
// - protoc v3.21.12
// - protoc v7.35.0
// source: filer.proto
package filer_pb
-83
View File
@@ -1135,16 +1135,6 @@ func (m *WriteCondition_Clause) MarshalToSizedBufferVT(dAtA []byte) (int, error)
i -= len(m.unknownFields)
copy(dAtA[i:], m.unknownFields)
}
if m.ExpectedEntry != nil {
size, err := m.ExpectedEntry.MarshalToSizedBufferVT(dAtA[:i])
if err != nil {
return 0, err
}
i -= size
i = protohelpers.EncodeVarint(dAtA, i, uint64(size))
i--
dAtA[i] = 0x52
}
if len(m.Fids) > 0 {
for iNdEx := len(m.Fids) - 1; iNdEx >= 0; iNdEx-- {
i -= len(m.Fids[iNdEx])
@@ -2097,16 +2087,6 @@ func (m *UpdateEntryRequest) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
i -= len(m.unknownFields)
copy(dAtA[i:], m.unknownFields)
}
if m.IsMoved {
i--
if m.IsMoved {
dAtA[i] = 1
} else {
dAtA[i] = 0
}
i--
dAtA[i] = 0x38
}
if m.Condition != nil {
size, err := m.Condition.MarshalToSizedBufferVT(dAtA[:i])
if err != nil {
@@ -6978,10 +6958,6 @@ func (m *WriteCondition_Clause) SizeVT() (n int) {
n += 1 + l + protohelpers.SizeOfVarint(uint64(l))
}
}
if m.ExpectedEntry != nil {
l = m.ExpectedEntry.SizeVT()
n += 1 + l + protohelpers.SizeOfVarint(uint64(l))
}
n += len(m.unknownFields)
return n
}
@@ -7356,9 +7332,6 @@ func (m *UpdateEntryRequest) SizeVT() (n int) {
l = m.Condition.SizeVT()
n += 1 + l + protohelpers.SizeOfVarint(uint64(l))
}
if m.IsMoved {
n += 2
}
n += len(m.unknownFields)
return n
}
@@ -12401,42 +12374,6 @@ func (m *WriteCondition_Clause) UnmarshalVT(dAtA []byte) error {
}
m.Fids = append(m.Fids, string(dAtA[iNdEx:postIndex]))
iNdEx = postIndex
case 10:
if wireType != 2 {
return fmt.Errorf("proto: wrong wireType = %d for field ExpectedEntry", wireType)
}
var msglen int
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return protohelpers.ErrIntOverflow
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
msglen |= int(b&0x7F) << shift
if b < 0x80 {
break
}
}
if msglen < 0 {
return protohelpers.ErrInvalidLength
}
postIndex := iNdEx + msglen
if postIndex < 0 {
return protohelpers.ErrInvalidLength
}
if postIndex > l {
return io.ErrUnexpectedEOF
}
if m.ExpectedEntry == nil {
m.ExpectedEntry = &Entry{}
}
if err := m.ExpectedEntry.UnmarshalVT(dAtA[iNdEx:postIndex]); err != nil {
return err
}
iNdEx = postIndex
default:
iNdEx = preIndex
skippy, err := protohelpers.Skip(dAtA[iNdEx:])
@@ -15130,26 +15067,6 @@ func (m *UpdateEntryRequest) UnmarshalVT(dAtA []byte) error {
return err
}
iNdEx = postIndex
case 7:
if wireType != 0 {
return fmt.Errorf("proto: wrong wireType = %d for field IsMoved", wireType)
}
var v int
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return protohelpers.ErrIntOverflow
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
v |= int(b&0x7F) << shift
if b < 0x80 {
break
}
}
m.IsMoved = bool(v != 0)
default:
iNdEx = preIndex
skippy, err := protohelpers.Skip(dAtA[iNdEx:])
+2 -34
View File
@@ -383,7 +383,7 @@ func applyMultipartSSES3HeadersFromUploadEntry(dst *filer_pb.Entry, sses3Info *m
}
}
func (s3a *S3ApiServer) prepareMultipartCompletionState(r *http.Request, input *s3.CompleteMultipartUploadInput, parts *CompleteMultipartUpload, uploadDirectory, entryName, dirName string, completedPartNumbers []int, completedPartMap map[int][]string, maxPartNo int) (*multipartCompletionState, *CompleteMultipartUploadResult, s3err.ErrorCode) {
func (s3a *S3ApiServer) prepareMultipartCompletionState(r *http.Request, input *s3.CompleteMultipartUploadInput, uploadDirectory, entryName, dirName string, completedPartNumbers []int, completedPartMap map[int][]string, maxPartNo int) (*multipartCompletionState, *CompleteMultipartUploadResult, s3err.ErrorCode) {
if entry, err := s3a.resolveObjectEntry(*input.Bucket, *input.Key, ""); err == nil && entry != nil && entry.Extended != nil {
if uploadId, ok := entry.Extended[s3_constants.SeaweedFSUploadId]; ok && *input.UploadId == string(uploadId) {
cleanupEntries, _, cleanupErr := s3a.list(uploadDirectory, "", "", false, s3_constants.MaxS3MultipartParts+1)
@@ -581,37 +581,8 @@ func (s3a *S3ApiServer) prepareMultipartCompletionState(r *http.Request, input *
glog.Errorf("completeMultipartUpload: %v", typeErr)
return nil, nil, s3err.ErrInvalidRequest
}
query := parseRequestQuery(r)
if completeType := lookupHeaderOrQuery(r, query, s3_constants.AmzChecksumType); completeType != "" && !strings.EqualFold(completeType, resolvedType) {
return nil, nil, s3err.ErrBadDigest
}
checksumType = resolvedType
provided := make(map[int]string, len(parts.Parts))
for _, part := range parts.Parts {
provided[part.PartNumber] = part.GetChecksum(checksumHeaderName)
}
for _, partNumber := range completedPartNumbers {
providedChecksum := provided[partNumber]
if providedChecksum == "" {
// COMPOSITE uploads must carry every part checksum in the
// request; FULL_OBJECT uploads instead carry the whole-object
// checksum in a request header.
if checksumType == s3_constants.ChecksumTypeComposite {
return nil, nil, s3err.ErrInvalidRequest
}
continue
}
raw, _, decodeErr := decodePartChecksum(partNumber, partEntries[partNumber], checksumHeaderName)
if decodeErr != nil {
glog.Errorf("completeMultipartUpload: %v", decodeErr)
return nil, nil, s3err.ErrInvalidPart
}
if providedChecksum != base64.StdEncoding.EncodeToString(raw) {
return nil, nil, s3err.ErrBadDigest
}
}
var checksumErr error
if checksumType == s3_constants.ChecksumTypeFullObject {
checksumValue, checksumErr = computeFullObjectChecksum(checksumHeaderName, partEntries, completedPartNumbers)
@@ -622,9 +593,6 @@ func (s3a *S3ApiServer) prepareMultipartCompletionState(r *http.Request, input *
glog.Errorf("completeMultipartUpload: %s checksum computation failed: %v", checksumType, checksumErr)
return nil, nil, s3err.ErrInvalidPart
}
if objectChecksum := lookupHeaderOrQuery(r, query, checksumHeaderName); objectChecksum != "" && objectChecksum != checksumValue {
return nil, nil, s3err.ErrBadDigest
}
}
// Fold after the boundaries captured byte offsets; finalParts is flat
@@ -688,7 +656,7 @@ func (s3a *S3ApiServer) completeMultipartUpload(r *http.Request, input *s3.Compl
routeKey := s3a.objectRouteKey(*input.Bucket, *input.Key)
completionBody := func() s3err.ErrorCode {
var prepCode s3err.ErrorCode
completionState, output, prepCode = s3a.prepareMultipartCompletionState(r, input, parts, uploadDirectory, entryName, dirName, completedPartNumbers, completedPartMap, maxPartNo)
completionState, output, prepCode = s3a.prepareMultipartCompletionState(r, input, uploadDirectory, entryName, dirName, completedPartNumbers, completedPartMap, maxPartNo)
if prepCode != s3err.ErrNone || output != nil {
return prepCode
}
+5 -11
View File
@@ -116,6 +116,11 @@ var bucketQueryActions = map[string]map[string]string{
http.MethodPut: s3_constants.S3_ACTION_PUT_BUCKET_OWNERSHIP_CONTROLS,
http.MethodDelete: s3_constants.S3_ACTION_PUT_BUCKET_OWNERSHIP_CONTROLS, // DELETE uses same permission as PUT
},
// SeaweedFS extension: bucket quota subresource
"seaweedfs-quota": {
http.MethodGet: s3_constants.S3_ACTION_GET_BUCKET_QUOTA,
http.MethodPut: s3_constants.S3_ACTION_PUT_BUCKET_QUOTA,
},
}
// resolveFromQueryParameters checks query parameters to determine specific S3 actions
@@ -262,17 +267,6 @@ func resolveFromQueryParameters(query url.Values, method string, hasObject bool)
return s3_constants.S3_ACTION_DELETE_OBJECT
}
// SeaweedFS extension: the quota routes are registered last among the
// bucket subresource routes, so resolve seaweedfs-quota last to match.
if !hasObject && query.Has("seaweedfs-quota") {
switch method {
case http.MethodGet:
return s3_constants.S3_ACTION_GET_BUCKET_QUOTA
case http.MethodPut:
return s3_constants.S3_ACTION_PUT_BUCKET_QUOTA
}
}
return ""
}
-27
View File
@@ -185,33 +185,6 @@ func TestResolveS3Action_ListType(t *testing.T) {
}
}
// The seaweedfs-quota routes are registered last among the bucket subresource
// routes, so the resolver must resolve it last as well: a request combining it
// with another selector is served by that selector's handler, and authorization
// must name the same operation.
func TestResolveS3Action_Quota(t *testing.T) {
tests := []struct {
name string
method string
query string
want string
}{
{"quota get", http.MethodGet, "seaweedfs-quota", s3_constants.S3_ACTION_GET_BUCKET_QUOTA},
{"quota put", http.MethodPut, "seaweedfs-quota", s3_constants.S3_ACTION_PUT_BUCKET_QUOTA},
{"policy wins over quota", http.MethodPut, "policy&seaweedfs-quota", s3_constants.S3_ACTION_PUT_BUCKET_POLICY},
{"quota after policy wins too", http.MethodPut, "seaweedfs-quota&policy", s3_constants.S3_ACTION_PUT_BUCKET_POLICY},
{"tagging wins over quota", http.MethodPut, "tagging&seaweedfs-quota", s3_constants.S3_ACTION_PUT_BUCKET_TAGGING},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r, _ := http.NewRequest(tt.method, "http://localhost/bucket?"+tt.query, nil)
if got := ResolveS3Action(r, s3_constants.ACTION_ADMIN, "bucket", ""); got != tt.want {
t.Errorf("ResolveS3Action() = %q, want %q", got, tt.want)
}
})
}
}
// A base action naming another service carries no S3 request shape, so a query
// parameter on the request must not redirect it to an S3 action.
func TestResolveS3ActionKeepsNonS3Service(t *testing.T) {
-1
View File
@@ -87,7 +87,6 @@ const (
AmzTrailer = "X-Amz-Trailer"
AmzSdkChecksumAlgorithm = "X-Amz-Sdk-Checksum-Algorithm"
AmzChecksumType = "X-Amz-Checksum-Type"
AmzChecksumMode = "X-Amz-Checksum-Mode"
// S3 checksum type values (x-amz-checksum-type). A COMPOSITE checksum is a
// checksum-of-per-part-checksums ("base64-N"); a FULL_OBJECT checksum is the
@@ -3,10 +3,8 @@ package s3api
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/gorilla/mux"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -29,7 +27,6 @@ func TestAmbiguousSubresource(t *testing.T) {
"uploads=&prefix=a&x-id=CreateMultipartUpload",
"list-type=2&prefix=a&continuation-token=x",
"acl=&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Signature=deadbeef",
"seaweedfs-quota=",
} {
req, _ := http.NewRequest("GET", "http://localhost/bucket/key?"+query, nil)
assert.False(t, hasAmbiguousSubresource(req.URL.Query()), "%q names one operation", query)
@@ -52,9 +49,6 @@ func TestAmbiguousSubresource(t *testing.T) {
"list-type=2&tagging=",
"ownershipControls=&list-type=2",
"list-type=2&versions=",
"policy=&seaweedfs-quota=",
"seaweedfs-quota=&policy=",
"seaweedfs-quota=&tagging=",
} {
req, _ := http.NewRequest("PUT", "http://localhost/bucket?"+query, nil)
assert.True(t, hasAmbiguousSubresource(req.URL.Query()), "%q names two operations", query)
@@ -107,36 +101,3 @@ func TestListTypeOwnershipControlsRejectedBeforeHandler(t *testing.T) {
handler.ServeHTTP(rec, req)
assert.True(t, served, "a plain list-type request must still be served")
}
// Every query key a route can be selected by must be classified: an operation
// subresource counted by the ambiguity guard, or a parameter that only
// modifies the operation it accompanies. A selector left out of both
// authorizes as one operation while the router serves another.
func TestRouteQueryKeysAreClassified(t *testing.T) {
// Action selects STS routes on the root router, id narrows a config
// subresource, partNumber accompanies uploadId; none select an S3
// operation on their own.
modifiers := map[string]bool{
"Action": true, "id": true, "partNumber": true,
}
router := mux.NewRouter()
setupRoutingTestServer(t).registerRouter(router)
err := router.Walk(func(route *mux.Route, _ *mux.Router, _ []*mux.Route) error {
templates, err := route.GetQueriesTemplates()
if err != nil {
return nil
}
for _, q := range templates {
key, _, _ := strings.Cut(q, "=")
if modifiers[key] {
continue
}
_, resolved := bucketQueryActions[key]
assert.True(t, operationSubresources[key] || resolved,
"route query key %q selects an operation but is not counted by hasAmbiguousSubresource", key)
}
return nil
})
require.NoError(t, err)
}
-5
View File
@@ -1251,11 +1251,6 @@ func (s3a *S3ApiServer) DeleteBucketLifecycleHandler(w http.ResponseWriter, r *h
func (s3a *S3ApiServer) GetBucketLocationHandler(w http.ResponseWriter, r *http.Request) {
bucket, _ := s3_constants.GetBucketAndObject(r)
if err := s3bucket.VerifyS3BucketName(bucket); err != nil {
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidBucketName)
return
}
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, err)
return
@@ -227,24 +227,6 @@ func TestGetBucketLogging(t *testing.T) {
}
}
func TestGetBucketLocationInvalidBucketName(t *testing.T) {
// AWS answers 400 InvalidBucketName for a malformed bucket name rather than
// the 404 NoSuchBucket an unknown-but-valid name gets.
s3a := &S3ApiServer{}
req := httptest.NewRequest(http.MethodGet, "/invalid%20bucket%20name?location=", nil)
req = mux.SetURLVars(req, map[string]string{"bucket": "invalid bucket name"})
rec := httptest.NewRecorder()
s3a.GetBucketLocationHandler(rec, req)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want %d, body=%s", rec.Code, http.StatusBadRequest, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), "InvalidBucketName") {
t.Fatalf("body = %s, want InvalidBucketName", rec.Body.String())
}
}
func TestHandleAutoCreateBucketDisabled(t *testing.T) {
s3a := &S3ApiServer{option: &S3ApiServerOption{}}
req := newBucketRequest(http.MethodPut, "test-bucket", "", "")
+1 -1
View File
@@ -2238,7 +2238,7 @@ func (s3a *S3ApiServer) setResponseHeaders(w http.ResponseWriter, r *http.Reques
// 2. The response covers the full object (no Range header, no partNumber)
// The stored checksum covers the full object; returning it for partial
// responses causes SDK checksum validation failures.
if r != nil && strings.EqualFold(lookupHeaderOrQuery(r, parseRequestQuery(r), s3_constants.AmzChecksumMode), "ENABLED") && r.Header.Get("Range") == "" && requestedPartNumber(r) == 0 {
if r != nil && r.Header.Get("X-Amz-Checksum-Mode") == "ENABLED" && r.Header.Get("Range") == "" && requestedPartNumber(r) == 0 {
if entry.Extended != nil {
if algoName, ok := entry.Extended[s3_constants.ExtChecksumAlgorithm]; ok {
if checksumVal, ok := entry.Extended[s3_constants.ExtChecksumValue]; ok {
@@ -445,21 +445,6 @@ func (s3a *S3ApiServer) PutObjectPartHandler(w http.ResponseWriter, r *http.Requ
}
}
// Parts inherit the checksum algorithm declared at CreateMultipartUpload
// when the request doesn't specify one; a conflicting one is rejected.
if headerName := string(uploadEntry.Extended[s3_constants.ExtChecksumAlgorithm]); headerName != "" {
if algo, reqHeaderName, code := detectRequestedChecksumAlgorithm(r); code == s3err.ErrNone {
if algo == ChecksumAlgorithmNone {
if name := checksumAlgorithmNameFromHeaderName(headerName); name != "" {
r.Header.Set(s3_constants.AmzChecksumAlgorithm, name)
}
} else if reqHeaderName != headerName {
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidRequest)
return
}
}
}
filePath := s3a.genPartUploadPath(bucket, uploadID, partID)
if partID == 1 && r.Header.Get("Content-Type") == "" {
@@ -650,7 +635,6 @@ type CompleteMultipartUpload struct {
type CompletedPart struct {
ETag string
PartNumber int
ChecksumResult
}
// handleSSES3MultipartHeaders handles SSE-S3 multipart upload header setup to reduce nesting complexity
-16
View File
@@ -1225,22 +1225,6 @@ func (c *ChecksumResult) SetChecksum(headerName, value string) {
}
}
func (c *ChecksumResult) GetChecksum(headerName string) string {
switch headerName {
case s3_constants.AmzChecksumCRC32:
return c.ChecksumCRC32
case s3_constants.AmzChecksumCRC32C:
return c.ChecksumCRC32C
case s3_constants.AmzChecksumCRC64NVME:
return c.ChecksumCRC64NVME
case s3_constants.AmzChecksumSHA1:
return c.ChecksumSHA1
case s3_constants.AmzChecksumSHA256:
return c.ChecksumSHA256
}
return ""
}
// lookupHeaderOrQuery returns the value of an x-amz-* parameter, checking the
// request headers first and falling back to the pre-parsed query values. AWS
// SDK presigners hoist headers such as x-amz-sdk-checksum-algorithm into the
+2 -5
View File
@@ -53,17 +53,14 @@ var operationSubresources = map[string]bool{
"location": true, "logging": true, "metrics": true, "notification": true,
"object-lock": true, "ownershipControls": true, "policy": true, "policyStatus": true,
"publicAccessBlock": true, "renameObject": true, "replication": true,
"requestPayment": true, "retention": true, "seaweedfs-quota": true,
"tagging": true, "uploadId": true,
"requestPayment": true, "retention": true, "tagging": true, "uploadId": true,
"uploads": true, "versioning": true, "versions": true, "website": true,
}
func hasAmbiguousSubresource(query url.Values) bool {
seen := 0
for key := range query {
// bucketQueryActions keys select an operation by definition, so they
// count even if operationSubresources was not updated for them.
if _, ok := bucketQueryActions[key]; !ok && !operationSubresources[key] {
if !operationSubresources[key] {
continue
}
if seen++; seen > 1 {
-26
View File
@@ -662,32 +662,6 @@ func (fs *FilerServer) UpdateEntry(ctx context.Context, req *filer_pb.UpdateEntr
fullpath := util.Join(req.Directory, req.Entry.Name)
// A conditional or preconditioned update is a read-then-write that the
// per-path lock below only makes atomic on this filer. Route it to the
// entry's ring owner so one filer's lock arbitrates every writer
// cluster-wide; is_moved bounds this to one hop.
if !req.IsMoved && (conditionIsSet(req.Condition) || len(req.ExpectedExtended) > 0) {
var ownerResp *filer_pb.UpdateEntryResponse
handled, forwardErr := fs.forwardToWriteOwner(ctx, entryRouteKey(util.FullPath(fullpath)), func(owner pb.ServerAddress) error {
glog.V(2).InfofCtx(ctx, "UpdateEntry %s: forwarding to owner %s", fullpath, owner)
req.IsMoved = true
return pb.WithFilerClient(false, 0, owner, fs.grpcDialOption, func(client filer_pb.SeaweedFilerClient) error {
forwarded, e := client.UpdateEntry(ctx, req)
if e != nil {
return e
}
ownerResp = forwarded
return nil
})
})
if handled {
if forwardErr != nil {
return &filer_pb.UpdateEntryResponse{}, forwardErr
}
return ownerResp, nil
}
}
// Serialize concurrent mutations to the same path on this filer so the
// read (preconditions, garbage diff) and the write are atomic. Callers
// route a key's writes to this owner filer, making this local lock
@@ -8,7 +8,6 @@ import (
"github.com/seaweedfs/seaweedfs/weed/filer"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"google.golang.org/protobuf/proto"
)
// conditionIsSet reports whether a condition asks for any check at all.
@@ -83,14 +82,6 @@ func clauseSatisfied(c *filer_pb.WriteCondition_Clause, current *filer.Entry) bo
return deadline <= time.Now().Unix()
case filer_pb.WriteCondition_IF_CHUNKS_EQUAL:
return chunkFidsEqual(current, c.Fids)
case filer_pb.WriteCondition_IF_ENTRY_EQUAL:
if !exists || c.ExpectedEntry == nil {
return !exists && c.ExpectedEntry == nil
}
// Normalize the expected entry the way FindEntry normalizes the stored
// one (e.g. FileSize grows to the chunk extent), or an unchanged entry
// can compare unequal.
return proto.Equal(current.ToProtoEntry(), filer.FromPbEntry("", c.ExpectedEntry).ToProtoEntry())
default:
// An unrecognized clause kind (e.g. from a newer client) must not be
// treated as satisfied, which would silently bypass the guard. Fail
@@ -193,34 +193,6 @@ func TestWriteConditionUnknownKindFailsClosed(t *testing.T) {
}
}
// IF_ENTRY_EQUAL compares the expected entry after the same normalization
// FindEntry applied to the stored one: a raw event entry whose FileSize is
// still zero must match a stored entry grown to its chunk extent.
func TestIfEntryEqualNormalizesExpected(t *testing.T) {
raw := &filer_pb.Entry{
Name: "f",
Attributes: &filer_pb.FuseAttributes{Mtime: 42},
Chunks: []*filer_pb.FileChunk{
{Fid: &filer_pb.FileId{VolumeId: 3, FileKey: 1, Cookie: 2}, Size: 100},
},
}
stored := filer.FromPbEntry("/d", raw)
if stored.FileSize != 100 {
t.Fatalf("stored FileSize = %d, want chunk extent 100", stored.FileSize)
}
cond := one(&filer_pb.WriteCondition_Clause{
Kind: filer_pb.WriteCondition_IF_ENTRY_EQUAL,
ExpectedEntry: raw,
})
if !writeConditionSatisfied(cond, stored) {
t.Error("raw expected entry must equal its normalized stored form")
}
raw.Attributes.Mtime = 43
if writeConditionSatisfied(cond, stored) {
t.Error("changed expected entry must not equal the stored entry")
}
}
// storedEntryETag prefers the stored Seaweed ETag attribute and falls back to
// the Md5-derived ETag, matching the S3 gateway.
func TestStoredEntryETag(t *testing.T) {
+5 -8
View File
@@ -82,9 +82,9 @@ func (vs *VolumeServer) DeleteCollection(ctx context.Context, req *volume_server
if err != nil {
glog.Errorf("delete collection %s: %v", req.Collection, err)
return resp, volumeStatusError(fmt.Errorf("delete collection %s: %w", req.Collection, err))
} else {
glog.V(2).Infof("delete collection %v", req)
}
glog.V(2).Infof("delete collection %v", req)
return resp, err
@@ -202,7 +202,7 @@ func (vs *VolumeServer) VolumeDelete(ctx context.Context, req *volume_server_pb.
if err != nil {
glog.Errorf("volume delete %v: %v", req, err)
return resp, volumeStatusError(err)
return resp, volumeDeleteStatusError(err)
} else {
// V(0) so destructive RPCs are always traceable.
glog.Infof("volume delete %v", req)
@@ -212,19 +212,16 @@ func (vs *VolumeServer) VolumeDelete(ctx context.Context, req *volume_server_pb.
}
// volumeStatusError keeps the store's message so callers matching on
// volumeDeleteStatusError keeps the store's message so callers matching on
// "not found" or "volume not empty" keep working, and adds the status code so
// new callers do not have to.
func volumeStatusError(err error) error {
func volumeDeleteStatusError(err error) error {
if errors.Is(err, storage.ErrVolumeNotFound) {
return status.Error(codes.NotFound, err.Error())
}
if errors.Is(err, storage.ErrVolumeNotEmpty) {
return status.Error(codes.FailedPrecondition, err.Error())
}
if errors.Is(err, storage.ErrInsufficientSpace) {
return status.Error(codes.ResourceExhausted, err.Error())
}
return err
}
+4 -8
View File
@@ -15,22 +15,18 @@ import (
"github.com/stretchr/testify/require"
)
func TestVolumeStatusErrorDistinguishesAbsentFromTransportFailure(t *testing.T) {
notFound := volumeStatusError(fmt.Errorf("delete volume 17 not found on disk: %w", storage.ErrVolumeNotFound))
func TestVolumeDeleteStatusErrorDistinguishesAbsentFromTransportFailure(t *testing.T) {
notFound := volumeDeleteStatusError(fmt.Errorf("delete volume 17 not found on disk: %w", storage.ErrVolumeNotFound))
assert.Equal(t, codes.NotFound, status.Code(notFound))
assert.Contains(t, notFound.Error(), "not found", "the store message must survive for callers that match on it")
transport := errors.New("connection reset")
require.ErrorIs(t, volumeStatusError(transport), transport)
require.ErrorIs(t, volumeDeleteStatusError(transport), transport)
assert.NotEqual(t, codes.NotFound, status.Code(transport))
notEmpty := volumeStatusError(storage.ErrVolumeNotEmpty)
notEmpty := volumeDeleteStatusError(storage.ErrVolumeNotEmpty)
assert.Equal(t, codes.FailedPrecondition, status.Code(notEmpty))
assert.Contains(t, notEmpty.Error(), "volume not empty")
noSpace := volumeStatusError(fmt.Errorf("compact volume 17: %w", storage.ErrInsufficientSpace))
assert.Equal(t, codes.ResourceExhausted, status.Code(noSpace))
assert.Contains(t, noSpace.Error(), "insufficient free space")
}
func TestVolumeDeleteMapsAbsentStoreVolumeToNotFound(t *testing.T) {
+2 -2
View File
@@ -59,7 +59,7 @@ func (vs *VolumeServer) VolumeCopy(req *volume_server_pb.VolumeCopyRequest, stre
var sourceVolumeStatusAfterCopy *volume_server_pb.VolumeStatusResponse
var dataBaseFileName, indexBaseFileName, idxFileName, datFileName string
var hasRemoteDatFile bool
err := operation.WithVolumeServerClientOptions(true, pb.ServerAddress(req.SourceDataNode), func(client volume_server_pb.VolumeServerClient) error {
err := operation.WithVolumeServerClient(true, pb.ServerAddress(req.SourceDataNode), vs.grpcDialOption, func(client volume_server_pb.VolumeServerClient) error {
var err error
sourceVolumeStatus, err = client.VolumeStatus(stream.Context(), &volume_server_pb.VolumeStatusRequest{
VolumeId: req.VolumeId,
@@ -214,7 +214,7 @@ func (vs *VolumeServer) VolumeCopy(req *volume_server_pb.VolumeCopyRequest, stre
}
return nil
}, vs.grpcDialOption, vs.guardedGrpcDialOption(req.SourceDataNode))
}, vs.guardedGrpcDialOption(req.SourceDataNode))
if err != nil {
return err
+2 -2
View File
@@ -392,7 +392,7 @@ func (vs *VolumeServer) VolumeEcShardsCopy(ctx context.Context, req *volume_serv
}
throttler := util.NewWriteThrottler(ioBytePerSecond)
err := operation.WithVolumeServerClientOptions(true, pb.ServerAddress(req.SourceDataNode), func(client volume_server_pb.VolumeServerClient) error {
err := operation.WithVolumeServerClient(true, pb.ServerAddress(req.SourceDataNode), vs.grpcDialOption, func(client volume_server_pb.VolumeServerClient) error {
// copy ec data slices
for _, shardId := range req.ShardIds {
@@ -460,7 +460,7 @@ func (vs *VolumeServer) VolumeEcShardsCopy(ctx context.Context, req *volume_serv
}
}
return nil
}, vs.grpcDialOption, vs.guardedGrpcDialOption(req.SourceDataNode))
}, vs.guardedGrpcDialOption(req.SourceDataNode))
if err != nil {
return nil, fmt.Errorf("VolumeEcShardsCopy volume %d: %v", req.VolumeId, err)
}
+2 -2
View File
@@ -100,10 +100,10 @@ func (vs *VolumeServer) VolumeTailReceiver(ctx context.Context, req *volume_serv
defer glog.V(1).Infof("receive tailing volume %d finished", v.Id)
return resp, operation.TailVolumeFromSource(pb.ServerAddress(req.SourceVolumeServer), v.Id, req.SinceNs, int(req.IdleTimeoutSeconds), func(n *needle.Needle) error {
return resp, operation.TailVolumeFromSource(pb.ServerAddress(req.SourceVolumeServer), v.Id, req.SinceNs, int(req.IdleTimeoutSeconds), vs.grpcDialOption, func(n *needle.Needle) error {
_, err := vs.store.WriteVolumeNeedle(v.Id, n, false, false)
return err
}, vs.grpcDialOption, vs.guardedGrpcDialOption(req.SourceVolumeServer))
}, vs.guardedGrpcDialOption(req.SourceVolumeServer))
}
+11 -13
View File
@@ -2,7 +2,6 @@ package weed_server
import (
"context"
"fmt"
"strconv"
"time"
@@ -28,10 +27,9 @@ func (vs *VolumeServer) VacuumVolumeCheck(ctx context.Context, req *volume_serve
if err != nil {
glog.V(3).Infof("check volume %d: %v", req.VolumeId, err)
return resp, volumeStatusError(err)
}
return resp, nil
return resp, err
}
@@ -72,7 +70,7 @@ func (vs *VolumeServer) VacuumVolumeCompact(req *volume_server_pb.VacuumVolumeCo
stats.VolumeServerVacuumingCompactCounter.WithLabelValues(strconv.FormatBool(err == nil && sendErr == nil)).Inc()
if err != nil {
glog.Errorf("failed compact volume %d: %v", req.VolumeId, err)
return volumeStatusError(fmt.Errorf("compact volume %d: %w", req.VolumeId, err))
return err
}
if sendErr != nil {
glog.Errorf("failed compact volume %d report progress: %v", req.VolumeId, sendErr)
@@ -101,15 +99,15 @@ func (vs *VolumeServer) VacuumVolumeCommit(ctx context.Context, req *volume_serv
readOnly, volumeSize, err := vs.store.CommitCompactVolume(needle.VolumeId(req.VolumeId))
if err != nil {
glog.Errorf("failed commit volume %d: %v", req.VolumeId, err)
} else {
glog.V(1).Infof("commit volume %d", req.VolumeId)
}
stats.VolumeServerVacuumingCommitCounter.WithLabelValues(strconv.FormatBool(err == nil)).Inc()
resp.IsReadOnly = readOnly
resp.VolumeSize = uint64(volumeSize)
if err != nil {
glog.Errorf("failed commit volume %d: %v", req.VolumeId, err)
return resp, volumeStatusError(fmt.Errorf("commit compact volume %d: %w", req.VolumeId, err))
}
glog.V(1).Infof("commit volume %d", req.VolumeId)
return resp, nil
return resp, err
}
@@ -126,10 +124,10 @@ func (vs *VolumeServer) VacuumVolumeCleanup(ctx context.Context, req *volume_ser
if err != nil {
glog.Errorf("failed cleanup volume %d: %v", req.VolumeId, err)
return resp, volumeStatusError(fmt.Errorf("cleanup volume %d: %w", req.VolumeId, err))
} else {
glog.V(1).Infof("cleanup volume %d", req.VolumeId)
}
glog.V(1).Infof("cleanup volume %d", req.VolumeId)
return resp, nil
return resp, err
}
+2 -2
View File
@@ -232,14 +232,14 @@ func startTailNeedleStream(grpcDialOption grpc.DialOption, volumeId needle.Volum
ch := make(chan *needle.Needle, 32)
stream := &tailNeedleStream{ch: ch}
go func() {
err := operation.TailVolumeFromSource(server, volumeId, 0, mergeIdleTimeoutSeconds, func(n *needle.Needle) error {
err := operation.TailVolumeFromSource(server, volumeId, 0, mergeIdleTimeoutSeconds, grpcDialOption, func(n *needle.Needle) error {
select {
case ch <- n:
case <-done:
return fmt.Errorf("merge cancelled")
}
return nil
}, grpcDialOption)
})
close(ch)
stream.setErr(err)
}()
+5 -31
View File
@@ -26,11 +26,6 @@ var (
destroyDelaySeconds int64 = 0
)
// ecjLoadChunkBytes bounds each positional read when seeding deletedNeedles
// from .ecj. A multiple of NeedleIdSize; 1 MiB is 131072 entries per syscall,
// which keeps a bloated journal from spending mount in per-entry reads.
const ecjLoadChunkBytes = 1 << 20
type EcVolume struct {
VolumeId needle.VolumeId
Collection string
@@ -212,20 +207,6 @@ func NewEcVolume(diskType types.DiskType, dir string, dirIdx string, collection
} else {
glog.Warningf("stat ec volume journal %s.ecj: %v", indexBaseFileName, statErr)
}
// Truncate a torn tail before the loader runs: appends land at the
// physical end, so a trailing partial record would misalign every later
// delete and lose it on the next mount.
if ragged := ev.ecjFileSize % int64(types.NeedleIdSize); ragged != 0 {
whole := ev.ecjFileSize - ragged
glog.Warningf("ec volume %d: truncating torn .ecj tail %d -> %d bytes", vid, ev.ecjFileSize, whole)
if truncErr := ev.ecjFile.Truncate(whole); truncErr != nil {
return nil, fmt.Errorf("ec volume %d: repair torn .ecj tail: %w", vid, truncErr)
}
if syncErr := ev.ecjFile.Sync(); syncErr != nil {
return nil, fmt.Errorf("ec volume %d: sync .ecj after tail repair: %w", vid, syncErr)
}
ev.ecjFileSize = whole
}
ev.deletedNeedles = make(map[types.NeedleId]struct{})
if loadErr := ev.loadDeletedNeedlesFromEcj(); loadErr != nil {
glog.Warningf("ec volume %d: load deleted needles from .ecj: %v", vid, loadErr)
@@ -655,20 +636,13 @@ func (ev *EcVolume) loadDeletedNeedlesFromEcj() error {
if ev.ecjFile == nil || ev.ecjFileSize < int64(types.NeedleIdSize) {
return nil
}
buf := make([]byte, ecjLoadChunkBytes)
for off := int64(0); off+int64(types.NeedleIdSize) <= ev.ecjFileSize; {
want := min(int64(ecjLoadChunkBytes), ev.ecjFileSize-off)
want -= want % int64(types.NeedleIdSize)
if want == 0 {
break
}
if _, err := ev.ecjFile.ReadAt(buf[:want], off); err != nil {
buf := make([]byte, types.NeedleIdSize)
for off := int64(0); off+int64(types.NeedleIdSize) <= ev.ecjFileSize; off += int64(types.NeedleIdSize) {
if _, err := ev.ecjFile.ReadAt(buf, off); err != nil {
return fmt.Errorf("read ecj at %d: %w", off, err)
}
for i := int64(0); i+int64(types.NeedleIdSize) <= want; i += int64(types.NeedleIdSize) {
ev.deletedNeedles[types.BytesToNeedleId(buf[i:i+types.NeedleIdSize])] = struct{}{}
}
off += want
id := types.BytesToNeedleId(buf)
ev.deletedNeedles[id] = struct{}{}
}
return nil
}
@@ -1,86 +0,0 @@
package erasure_coding_test
import (
"os"
"testing"
erasure_coding "github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
"github.com/seaweedfs/seaweedfs/weed/storage/types"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func ecjBytes(ids ...types.NeedleId) []byte {
b := make([]byte, 0, len(ids)*types.NeedleIdSize)
rec := make([]byte, types.NeedleIdSize)
for _, id := range ids {
types.NeedleIdToBytes(rec, id)
b = append(b, rec...)
}
return b
}
func mountEcVolume(t *testing.T, dir string, ecx, ecj []byte) (*erasure_coding.EcVolume, string) {
t.Helper()
base := erasure_coding.EcShardFileName("", dir, 7)
require.NoError(t, os.WriteFile(base+".ecx", ecx, 0644))
if ecj != nil {
require.NoError(t, os.WriteFile(base+".ecj", ecj, 0644))
}
require.NoError(t, os.WriteFile(base+".vif", []byte{}, 0644))
ev, err := erasure_coding.NewEcVolume("hdd", dir, dir, "", 7)
require.NoError(t, err)
return ev, base
}
// A crash mid-append leaves a partial record at the end of .ecj. Mount must
// drop it: deletes append at the physical end, so keeping the fragment would
// misalign every later record and lose those deletes on the next mount.
func TestEcjTornTailTruncatedOnMount(t *testing.T) {
dir := t.TempDir()
ecx := append(makeNeedleMapEntry(types.NeedleId(1), types.ToOffset(0), types.Size(100)),
makeNeedleMapEntry(types.NeedleId(4), types.ToOffset(8), types.Size(100))...)
ecj := append(ecjBytes(1, 2, 3), []byte{1, 2, 3, 4, 5}...)
ev, base := mountEcVolume(t, dir, ecx, ecj)
fi, err := os.Stat(base + ".ecj")
require.NoError(t, err)
assert.Equal(t, int64(3*types.NeedleIdSize), fi.Size())
for _, id := range []types.NeedleId{1, 2, 3} {
assert.True(t, ev.IsNeedleDeleted(id), "id %d", id)
}
assert.False(t, ev.IsNeedleDeleted(4))
require.NoError(t, ev.DeleteNeedleFromEcx(4))
ev.Close()
ev, _ = mountEcVolume(t, dir, ecx, nil)
defer ev.Close()
for _, id := range []types.NeedleId{1, 2, 3, 4} {
assert.True(t, ev.IsNeedleDeleted(id), "id %d", id)
}
}
// A journal larger than one load chunk must seed every record, including the
// ones straddling and following the chunk boundary.
func TestEcjLoadsAcrossChunkBoundary(t *testing.T) {
dir := t.TempDir()
const count = types.NeedleId(200_000) // 1.6 MiB > 1 MiB chunk
ecj := make([]byte, 0, int(count)*types.NeedleIdSize)
for id := types.NeedleId(1); id <= count; id++ {
rec := make([]byte, types.NeedleIdSize)
types.NeedleIdToBytes(rec, id)
ecj = append(ecj, rec...)
}
ev, _ := mountEcVolume(t, dir, nil, ecj)
defer ev.Close()
for _, id := range []types.NeedleId{1, 131072, 131073, count} {
assert.True(t, ev.IsNeedleDeleted(id), "id %d", id)
}
assert.False(t, ev.IsNeedleDeleted(count+1))
}
+2 -11
View File
@@ -39,9 +39,6 @@ func (n *Needle) DiskSize(version Version) int64 {
func ReadNeedleBlob(r backend.BackendStorageFile, offset int64, size Size, version Version) (dataSlice []byte, err error) {
if size.IsDeleted() {
return nil, fmt.Errorf("invalid needle size %d: %w", size, ErrorSizeInvalid)
}
dataSize := GetActualSize(size, version)
dataSlice = make([]byte, int(dataSize))
@@ -227,14 +224,8 @@ func (n *Needle) ReadNeedleBody(r backend.BackendStorageFile, version Version, o
func (n *Needle) ReadNeedleBodyBytes(needleBody []byte, version Version) (err error) {
// n.Size comes from the on-disk header, so a corrupted header can carry a
// negative size or one the body cannot hold along with its tail. Deriving
// the tail from the version's own layout keeps the bound exact for every
// on-disk format.
tailSize := NeedleBodyLength(n.Size, version) - int64(n.Size) - int64(PaddingLength(n.Size, version))
if n.Size < 0 || int64(n.Size)+tailSize > int64(len(needleBody)) {
stats.VolumeServerHandlerCounter.WithLabelValues(stats.ErrorIndexOutOfRange).Inc()
return fmt.Errorf("needle %v size %d out of range for body length %d: %w", n.Id, n.Size, len(needleBody), ErrorCorrupted)
if len(needleBody) <= 0 {
return nil
}
switch version {
case Version1:
-105
View File
@@ -1,105 +0,0 @@
package needle
import (
"bytes"
"errors"
"strings"
"testing"
. "github.com/seaweedfs/seaweedfs/weed/storage/types"
)
// readNeedleBodyBytes runs ReadNeedleBodyBytes and turns a panic into a test
// failure, so a regression reports which case broke instead of killing the run.
func readNeedleBodyBytes(t *testing.T, n *Needle, body []byte, version Version) (err error) {
t.Helper()
defer func() {
if r := recover(); r != nil {
t.Fatalf("ReadNeedleBodyBytes panicked for size %d, body length %d: %v", n.Size, len(body), r)
}
}()
return n.ReadNeedleBodyBytes(body, version)
}
// The size feeds the read buffer's length, so a negative one never reaches make().
func TestReadNeedleBlobRejectsNegativeSize(t *testing.T) {
for _, version := range []Version{Version1, Version2, Version3} {
for _, size := range []Size{TombstoneFileSize, -100} {
if _, err := ReadNeedleBlob(nil, 0, size, version); !errors.Is(err, ErrorSizeInvalid) {
t.Fatalf("version %d size %d: expected ErrorSizeInvalid, got %v", version, size, err)
}
}
}
}
// A corrupted .dat header can carry a size that does not fit the body read for
// it. Vacuum used to panic on it with "slice bounds out of range [:-1]" (#6763).
func TestReadNeedleBodyBytesRejectsCorruptSize(t *testing.T) {
for version := Version1; IsSupportedVersion(version); version++ {
t.Run(versionString(version), func(t *testing.T) {
// A size of -1 is the case from #6763: its body length is still
// positive, so the scan reads a body and hands it over.
bodyLength := NeedleBodyLength(-1, version)
if bodyLength <= 0 {
t.Fatalf("expected a positive body length for size -1, got %d", bodyLength)
}
cases := []struct {
name string
size Size
body int
}{
{"size -1", -1, int(bodyLength)},
{"size -12", -12, 32},
{"size larger than body", 64, 32},
{"no room for the tail", 32, 32},
{"empty body", 0, 0},
{"empty body with data size", 1, 0},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
n := &Needle{Size: c.size}
err := readNeedleBodyBytes(t, n, make([]byte, c.body), version)
if !errors.Is(err, ErrorCorrupted) {
t.Fatalf("expected an error wrapping ErrorCorrupted, got %v", err)
}
})
}
})
}
}
// The size guard must still accept every record the writer produces,
// including the size-0 record a delete appends.
func TestReadNeedleBodyBytesWrittenNeedles(t *testing.T) {
for version := Version1; IsSupportedVersion(version); version++ {
t.Run(versionString(version), func(t *testing.T) {
for _, data := range [][]byte{nil, []byte("hello seaweed")} {
written := &Needle{Id: 7, Cookie: 9, Data: data, Checksum: NewCRC(data), AppendAtNs: 42}
buf := new(bytes.Buffer)
if _, _, err := writeNeedleByVersion(version, written, 0, buf); err != nil {
// Some builds can read a version they cannot write; skip
// only that recognized case so a real writer regression
// still fails the test.
if strings.Contains(strings.ToLower(err.Error()), "unsupported version") {
t.Skipf("version %d is not writable in this build: %v", version, err)
}
t.Fatalf("write needle: %v", err)
}
n := new(Needle)
n.ParseNeedleHeader(buf.Bytes())
body := buf.Bytes()[NeedleHeaderSize:]
if int64(len(body)) != NeedleBodyLength(n.Size, version) {
t.Fatalf("body length %d, want %d", len(body), NeedleBodyLength(n.Size, version))
}
if err := readNeedleBodyBytes(t, n, body, version); err != nil {
t.Fatalf("read %d-byte needle: %v", len(data), err)
}
if !bytes.Equal(n.Data, data) {
t.Fatalf("data %q, want %q", n.Data, data)
}
}
})
}
}

Some files were not shown because too many files have changed in this diff Show More