mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-07-23 08:23:19 +00:00
Chris Lu
6206f60032
* fix(master): let the growth initiator wait for the growth it triggered The growth-in-flight shed also fired on the request that initiated the growth: it sets the pending flag right before the shed check, so a cold-start assign enqueued growth and immediately failed itself with "volume growth in progress". With no concurrent assigns around to pick up the freshly grown volume, a single writer against an empty cluster never completes a write despite ample free space. Claim the pending flag with a compare-and-swap so exactly one request becomes the initiator, triggering growth at most once, and let it wait for that growth to land. Everyone else still sheds retryably instead of pinning a goroutine: followers behind an in-flight growth, an initiator whose growth concluded without yielding a writable volume, and an initiator whose growth outlives the 10s wait budget, which previously surfaced a non-retryable error (gRPC Unknown, HTTP 406) even though a retry would have succeeded moments later. * fix(master): stop assign waits when the request is cancelled The assign retry loops slept through client cancellation, keeping a goroutine spinning for the rest of the 10s budget after the caller had gone; StreamAssign also ran assigns on a background context detached from the stream. Wait on the request context and pass the stream context through. * topology: drop the unconditional grow-request setter Growth is only claimed through AddGrowRequestIfAbsent's compare-and-swap now; keeping the raw Store(true) around invites the check-then-set race back. * test: cover cold-start first write with a real cluster Boot a fresh master plus three empty volume servers and require the very first assign - HTTP and gRPC, each on a cold volume layout, no client retries - to complete a write. The assign that triggers volume growth must wait for it rather than answering "volume growth in progress"; unit tests stub the topology, so only a real cluster exercises the assign-grow-wait path end to end.
80 lines
2.5 KiB
YAML
80 lines
2.5 KiB
YAML
name: "Master Cold Start Tests"
|
|
|
|
on:
|
|
push:
|
|
branches: [ master ]
|
|
paths:
|
|
- 'weed/server/master_*.go'
|
|
- 'weed/topology/**'
|
|
- 'weed/operation/**'
|
|
- 'test/master_cold_start/**'
|
|
- 'test/testutil/**'
|
|
- '.github/workflows/master-cold-start-tests.yml'
|
|
pull_request:
|
|
branches: [ master ]
|
|
paths:
|
|
- 'weed/server/master_*.go'
|
|
- 'weed/topology/**'
|
|
- 'weed/operation/**'
|
|
- 'test/master_cold_start/**'
|
|
- 'test/testutil/**'
|
|
- '.github/workflows/master-cold-start-tests.yml'
|
|
|
|
concurrency:
|
|
group: ${{ github.head_ref || github.ref }}/master-cold-start-tests
|
|
cancel-in-progress: true
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
master-cold-start-tests:
|
|
name: Master Cold Start Tests
|
|
runs-on: ubuntu-22.04
|
|
timeout-minutes: 10
|
|
steps:
|
|
- name: Check out code
|
|
uses: actions/checkout@v7
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Set up Go
|
|
uses: actions/setup-go@v6
|
|
with:
|
|
go-version-file: 'go.mod'
|
|
|
|
- name: Build weed binary
|
|
run: |
|
|
cd weed && go install -buildvcs=false
|
|
|
|
- name: Run master cold start tests
|
|
# test/master_cold_start boots a fresh master plus empty volume
|
|
# servers and requires the very first assign (HTTP and gRPC, no
|
|
# client retries) to complete a write: the assign that triggers
|
|
# volume growth must wait for it instead of failing with
|
|
# "volume growth in progress".
|
|
run: |
|
|
export WEED_BINARY=$(go env GOPATH)/bin/weed
|
|
go test -v -timeout=8m ./test/master_cold_start/...
|
|
|
|
- name: Collect server logs on failure
|
|
if: failure()
|
|
run: |
|
|
# test/master_cold_start/cluster.go keeps failing-test dirs created
|
|
# via os.MkdirTemp("", "seaweedfs_master_cold_start_it_") with each
|
|
# process log under <baseDir>/logs/.
|
|
mkdir -p /tmp/master-cold-start-logs
|
|
find /tmp -maxdepth 1 -type d -name "seaweedfs_master_cold_start_it_*" 2>/dev/null | while read dir; do
|
|
echo "Found test directory: $dir"
|
|
cp -r "$dir" /tmp/master-cold-start-logs/ 2>/dev/null || true
|
|
done
|
|
find /tmp/master-cold-start-logs -type f -name "*.log" -print -exec tail -n 100 {} \; 2>/dev/null || echo "No logs found"
|
|
|
|
- name: Archive logs
|
|
if: failure()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: master-cold-start-test-logs
|
|
path: /tmp/master-cold-start-logs/
|
|
retention-days: 7
|