Files
seaweedfs/.github/workflows/s3-mutation-regression-tests.yml
T
Chris LuandGitHub a5e8254ffd s3: give a versioned metadata-only copy its own chunks (#10594)
* s3: give a versioned metadata-only copy its own chunks

A self-copy that only rewrites metadata clones the source entry, chunk fids
and all, and writes the clone back. With no versioning that is exactly right:
the clone replaces the entry it came from, so one entry owns the needles the
whole time. Under versioning the clone lands in a new .versions/ file and the
source stays live, and nothing refcounts a plain shared chunk list -- deleting
either version (a NoncurrentVersionExpiration rule, say) frees needles the
other still points at, and the next vacuum makes that permanent. rclone hits
this on every upload, since it stamps mtime with exactly this copy.

Take the metadata-only path only where the write replaces the entry it read:
the bare key of a bucket without versioning. Versioned, suspended, and
versionId-pinned copies fall through to the regular copy path, which gives the
destination its own chunks.

* s3: reencrypt a versioned SSE-KMS key rotation instead of reusing the chunks

A same-object copy that changes the KMS key id hands the source chunks straight
back, on the assumption that the copy overwrites the entry they came from. A
versioned bucket writes a new version beside the source instead, so the two end
up sharing needles that nothing refcounts, and deleting either one frees the
other's data. Reuse the chunks only when the destination really is the source
entry; otherwise fall through to the reencrypt path, which also gives the new
version the key it asked for rather than leaving it on the old one.

* s3: make one predicate decide whether a copy replaces its source

The metadata-only branch and the key-rotation strategy both answer the same
question -- does this copy write back to the entry it read -- so let them share
one predicate instead of pairing a same-destination check with it separately at
each site.

* test(s3): fail the copy regression tests when the vacuum does not run

The helper swallowed a failed or non-200 request to the master, so a vacuum
that never ran turned both chunk-ownership assertions into no-ops: the
tombstoned needles were still readable and the surviving version looked fine
either way. Require the endpoint, the request, and a 200.

* ci(s3): run every versioning test in the regression gate

The gate named the tests it wanted, so a new regression test sat there
uncovered until someone remembered this file -- it fooled me into thinking two
tests added in this PR never ran anywhere, when the comprehensive job had them
all along. Invert it: run everything, and name a test only to keep it out. The
delete job beside this one already works that way, and the suite costs about
two minutes.

Only the pagination stress tests are excluded; they build 1500+ versions, skip
themselves without ENABLE_STRESS_TESTS, and have their own make target.

Go's regexp has no negation, so the pattern is still assembled from a listing,
the way the volume-server integration workflow does it. Note the trailing $$:
make eats a lone trailing $ and takes the anchor with it.
2026-08-05 20:29:28 -07:00

149 lines
4.5 KiB
YAML

name: "S3 Mutation Regression Tests"
on:
pull_request:
paths:
- 'weed/s3api/**'
- 'test/s3/delete/**'
- 'test/s3/distributed_lock/**'
- 'test/s3/versioning/**'
- 'test/volume_server/framework/**'
- 'docker/compose/s3.json'
- '.github/workflows/s3-mutation-regression-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-mutation-regression-tests
cancel-in-progress: true
permissions:
contents: read
jobs:
s3-versioning-regressions:
name: S3 Versioning Regression Tests
runs-on: ubuntu-22.04
timeout-minutes: 25
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
- name: Run S3 versioning regression tests
timeout-minutes: 20
working-directory: test/s3/versioning
run: |
set -x
# Run every versioning test, so a regression test lands covered instead
# of waiting for someone to remember this file. Name a test in EXCLUDE,
# with the reason, to keep it out.
#
# TestVersioningPagination*: opt-in stress tests that build 1500+
# versions. They self-skip without ENABLE_STRESS_TESTS and have their
# own make target, so this gate should not carry them.
EXCLUDE='TestVersioningPagination.*'
tests=$(go test . -list '.*' | grep '^Test' | sort -u)
# An empty list would make -run match nothing and pass this job vacuously.
[ -n "$tests" ] || { echo "listed no versioning tests"; exit 1; }
selected=$(echo "$tests" | grep -vE "^($EXCLUDE)$")
[ -n "$selected" ] || { echo "EXCLUDE matched every test"; exit 1; }
echo "running $(echo "$selected" | wc -l) of $(echo "$tests" | wc -l) versioning tests"
# make swallows a lone trailing $, taking the anchor with it, so escape it.
make test-with-server TEST_PATTERN="^($(echo "$selected" | paste -sd'|' -))"'$$'
- name: Show server logs on failure
if: failure()
working-directory: test/s3/versioning
run: |
echo "=== Server Logs ==="
if [ -f weed-test.log ]; then
tail -100 weed-test.log
fi
- name: Upload versioning logs on failure
if: failure()
uses: actions/upload-artifact@v7
with:
name: s3-versioning-regression-logs
path: test/s3/versioning/weed-test*.log
retention-days: 3
s3-delete-regressions:
name: S3 Delete Regression Tests
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
- name: Run S3 delete regression tests
timeout-minutes: 15
working-directory: test/s3/delete
run: |
set -x
make test-with-server
- name: Show server logs on failure
if: failure()
working-directory: test/s3/delete
run: |
echo "=== Server Logs ==="
if [ -f weed-test.log ]; then
tail -100 weed-test.log
fi
- name: Upload delete logs on failure
if: failure()
uses: actions/upload-artifact@v7
with:
name: s3-delete-regression-logs
path: test/s3/delete/weed-test*.log
retention-days: 3
s3-distributed-lock-regressions:
name: S3 Distributed Lock Regression Tests
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
- name: Build SeaweedFS
run: |
go build -o weed/weed -buildvcs=false ./weed
- name: Run distributed lock regressions
timeout-minutes: 25
env:
TMPDIR: ${{ github.workspace }}/test/s3/distributed_lock/tmp
S3_DISTRIBUTED_LOCK_KEEP_LOGS: "1"
WEED_BINARY: ${{ github.workspace }}/weed/weed
run: |
set -x
mkdir -p "$TMPDIR"
go test -v -count=1 -timeout=20m ./test/s3/distributed_lock
- name: Upload distributed lock logs on failure
if: failure()
uses: actions/upload-artifact@v7
with:
name: s3-distributed-lock-regression-logs
path: test/s3/distributed_lock/tmp/seaweedfs_s3_distributed_lock_*
retention-days: 3