Compare commits

..
Author SHA1 Message Date
Chris Lu 4e421400e4 filer: evaluate a write precondition in CreateEntry
Add an optional WriteCondition to CreateEntryRequest. When set, the filer
evaluates it against the current entry while holding the per-path lock, so the
check and the write are atomic on this filer, and returns PRECONDITION_FAILED
when it does not hold.

The condition is a single primitive (exists / not-exists / etag match / etag
mismatch / [un]modified-since). The caller reduces request semantics such as
RFC 7232 precedence to one primitive; the filer just compares. ETag comparison
mirrors the S3 gateway's precedence (stored Seaweed ETag attribute, then the
Md5/chunk fallback) so results match without coupling the filer to S3 handling.
2026-05-22 22:30:26 -07:00
Chris Lu 56c51e7f50 filer: serialize same-path mutations with a local lock
CreateEntry is a FindEntry-then-write with no lock, so concurrent creates to
the same path race: OExcl can admit two creators, and a conditional write has
no atomic check-then-act. Add a per-path exclusive lock (util.LockTable, which
evicts idle keys so it stays bounded) in the CreateEntry handler so the read
and the write are atomic on this filer. Once callers route a key's writes to
its owner filer, this local lock is the authoritative serialization point.

AppendToEntry moves from the distributed lock to the same per-path lock.
2026-05-22 22:24:16 -07:00
1500 changed files with 29350 additions and 168530 deletions
-22
View File
@@ -1,22 +0,0 @@
[codespell]
# Ref: https://github.com/codespell-project/codespell#using-a-config-file
skip = .git,.git-meta,.gitignore,.gitattributes,*.svg,go.sum,vendor,*.lock,*.css,*.min.*,.codespellrc,*_templ.go
check-hidden = true
# Ignore camelCase and PascalCase identifiers (very common in Go/Rust/JS
# source, e.g. allLocations, publishErr, ReadInside, FlushInterval).
ignore-regex = \b[a-z]+[A-Z]\w*\b|\b[A-Z][a-z]+[A-Z]\w*\b
# visibles: variable name for VisibleInterval collections in filer/mount code
# fo: `*FilerOptions` receiver name (e.g. `func (fo *FilerOptions) ...`)
# te: "truncate error" local variable (e.g. `if te := w.Truncate(end); te != nil`)
# ser: Rust serde serializer variable (`serde_json::ser`, `let mut ser = ...`)
# bject: intentional wildcard test data (e.g. `s3:Get?bject` matching `s3:GetObject`)
# unparseable: accepted alternate spelling used throughout the codebase
# keep-alives: correct plural of the technical term (SSH/HTTP keep-alive)
# tread: valid English word in the idiom "tread carefully" (help text)
# anc: variable abbreviation for "ancestor" in tree/path tests
# ue: appears inside JSON test fixtures with embedded escaped quotes (Bl\"ue)
# auther: local variable meaning "authenticator" (tls.go: `auther := Authenticator{}`)
# thirdparty: literal Maven groupId `org.apache.hadoop.thirdparty` (external, cannot rename)
# unknwon: GitHub username / Go module path (`github.com/unknwon/goconfig`)
# atleast: CLI mode literal string in test/benchmark/fuse_db/bin/sqlite_verify.py
ignore-words-list = visibles,fo,te,ser,bject,unparseable,keep-alives,tread,anc,ue,auther,thirdparty,unknwon,atleast
@@ -1,66 +0,0 @@
name: Fix fusermount3 setuid
description: >
Make sure the fusermount3 an unprivileged mount will find can actually mount.
Some runner images carry a second, source-built fusermount3 in /usr/local/bin
that shadows the distro one in PATH; it is neither setuid nor root-owned, so
every mount fails with "mount failed: Operation not permitted". Both the Go
mount and sw-fuse resolve the helper through PATH.
Run it after the step that apt-installs fuse3.
runs:
using: composite
steps:
- name: Point PATH at a fusermount3 that can mount
shell: bash
run: |
set -euo pipefail
# setuid only grants root when root owns the file, and exec follows
# symlinks, so judge the target.
can_mount() {
[ -u "$1" ] && [ "$(stat -Lc %u "$1")" = 0 ]
}
bin=$(command -v fusermount3 || true)
if [ -z "$bin" ]; then
echo "no fusermount3 in PATH" >&2
exit 1
fi
if can_mount "$bin"; then
ls -l "$bin"
exit 0
fi
echo "$bin cannot mount unprivileged:"
ls -l "$bin"
# The distro fusermount3 is setuid root and is the one meant to be used.
# Reach it through a symlink earlier in PATH: exec resolves the link, so
# the target keeps its setuid bit, and nothing on the image is modified.
for distro in /usr/bin/fusermount3 /bin/fusermount3; do
if [ "$distro" != "$bin" ] && can_mount "$distro"; then
mkdir -p "$RUNNER_TEMP/fuse-bin"
ln -sf "$distro" "$RUNNER_TEMP/fuse-bin/fusermount3"
echo "$RUNNER_TEMP/fuse-bin" >> "$GITHUB_PATH"
echo "using $distro instead"
ls -l "$distro"
exit 0
fi
done
# No usable distro binary, so setting the bit is the only way out - but
# the target came from PATH: do that only for a root-owned system binary,
# never for anything else that happens to sit there.
case "$bin" in
/bin/*|/sbin/*|/usr/bin/*|/usr/sbin/*|/usr/local/bin/*|/usr/local/sbin/*) ;;
*) echo "refusing to setuid $bin: outside the system bin paths" >&2; exit 1 ;;
esac
if [ -L "$bin" ] || [ ! -f "$bin" ] || [ ! -x "$bin" ]; then
echo "refusing to setuid $bin: not a regular executable file" >&2
exit 1
fi
if [ "$(stat -c %u "$bin")" != 0 ] || [ "$(stat -c %g "$bin")" != 0 ]; then
echo "refusing to setuid $bin: not owned by root:root" >&2
exit 1
fi
sudo chmod u+s "$bin"
ls -l "$bin"
+2 -7
View File
@@ -3,11 +3,6 @@ name: "go: build dev binaries"
on:
push:
branches: [ master ]
paths:
- 'weed/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/binaries_dev.yml'
concurrency:
group: binaries-dev-${{ github.ref }}
@@ -48,7 +43,7 @@ jobs:
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v2
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v2
- name: Set BUILD_TIME env
run: echo BUILD_TIME=$(date -u +%Y%m%d-%H%M) >> ${GITHUB_ENV}
@@ -97,7 +92,7 @@ jobs:
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v2
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v2
- name: Set BUILD_TIME env
run: echo BUILD_TIME=$(date -u +%Y%m%d-%H%M) >> ${GITHUB_ENV}
+1 -1
View File
@@ -28,7 +28,7 @@ jobs:
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v2
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v2
- name: Go Release Binaries Normal Volume Size
uses: wangyoucao577/go-release-action@279495102627de7960cbc33434ab01a12bae144b # v1.22
with:
+1 -1
View File
@@ -28,7 +28,7 @@ jobs:
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v2
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v2
- name: Go Release Binaries Normal Volume Size
uses: wangyoucao577/go-release-action@279495102627de7960cbc33434ab01a12bae144b # v1.22
with:
+1 -1
View File
@@ -28,7 +28,7 @@ jobs:
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v2
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v2
- name: Go Release Binaries Normal Volume Size
uses: wangyoucao577/go-release-action@279495102627de7960cbc33434ab01a12bae144b # v1.22
with:
+1 -1
View File
@@ -28,7 +28,7 @@ jobs:
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v2
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v2
- name: Go Release Binaries Normal Volume Size
uses: wangyoucao577/go-release-action@279495102627de7960cbc33434ab01a12bae144b # v1.22
with:
+1 -1
View File
@@ -28,7 +28,7 @@ jobs:
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v2
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v2
- name: Go Release Binaries Normal Volume Size
uses: wangyoucao577/go-release-action@279495102627de7960cbc33434ab01a12bae144b # v1.22
with:
+1 -1
View File
@@ -28,7 +28,7 @@ jobs:
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v2
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v2
- name: Go Release Binaries Normal Volume Size
uses: wangyoucao577/go-release-action@279495102627de7960cbc33434ab01a12bae144b # v1.22
with:
+4 -9
View File
@@ -2,11 +2,6 @@ name: "Code Scanning - Action"
on:
pull_request:
paths:
- '**/*.go'
- 'go.mod'
- 'go.sum'
- '.github/workflows/codeql.yml'
concurrency:
group: ${{ github.head_ref }}/codeql
@@ -23,11 +18,11 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4.37.4
uses: github/codeql-action/init@v4
# Override language selection by uncommenting this and choosing your languages
with:
languages: go
@@ -35,7 +30,7 @@ jobs:
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below).
- name: Autobuild
uses: github/codeql-action/autobuild@v4.37.4
uses: github/codeql-action/autobuild@v4
# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -49,4 +44,4 @@ jobs:
# make release
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4.37.4
uses: github/codeql-action/analyze@v4
-23
View File
@@ -1,23 +0,0 @@
# Codespell configuration is within .codespellrc
---
name: Codespell
on:
push:
branches: [master]
pull_request:
branches: [master]
permissions:
contents: read
jobs:
codespell:
name: Check for spelling errors
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v7
- name: Codespell
uses: codespell-project/actions-codespell@8f01853be192eb0f849a5c7d721450e7a467c579 # v2.2
+6 -13
View File
@@ -3,13 +3,6 @@ name: "docker: build dev containers"
on:
push:
branches: [ master ]
paths:
- 'weed/**'
- 'seaweed-volume/**'
- 'docker/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/container_dev.yml'
permissions:
contents: read
@@ -29,7 +22,7 @@ jobs:
cross: true
steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Install protobuf compiler
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
@@ -52,7 +45,7 @@ jobs:
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -82,7 +75,7 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Download pre-built Rust binaries
uses: actions/download-artifact@v8
@@ -118,7 +111,7 @@ jobs:
org.opencontainers.image.vendor=Chris Lu
- name: Set up QEMU
uses: docker/setup-qemu-action@v4.2.0
uses: docker/setup-qemu-action@v4
- name: Create BuildKit config
run: |
@@ -135,14 +128,14 @@ jobs:
- name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.1.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.1.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -33,7 +33,7 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -58,7 +58,7 @@ jobs:
sudo ldconfig
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: go.mod
@@ -126,31 +126,27 @@ jobs:
echo "seaweedfs_ref=$seaweed" >> "$GITHUB_OUTPUT"
- name: Set up QEMU
uses: docker/setup-qemu-action@v4.2.0
uses: docker/setup-qemu-action@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.1.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Determine branch to build
id: branch
env:
INPUT_REF: ${{ inputs.seaweedfs_ref }}
HEAD_REF: ${{ github.head_ref }}
REF_NAME: ${{ github.ref_name }}
run: |
if [ -n "$INPUT_REF" ]; then
echo "branch=$INPUT_REF" >> "$GITHUB_OUTPUT"
if [ -n "${{ inputs.seaweedfs_ref }}" ]; then
echo "branch=${{ inputs.seaweedfs_ref }}" >> "$GITHUB_OUTPUT"
elif [ "${{ github.event_name }}" = "pull_request" ]; then
echo "branch=$HEAD_REF" >> "$GITHUB_OUTPUT"
echo "branch=${{ github.head_ref }}" >> "$GITHUB_OUTPUT"
else
echo "branch=$REF_NAME" >> "$GITHUB_OUTPUT"
echo "branch=${{ github.ref_name }}" >> "$GITHUB_OUTPUT"
fi
- name: Build and push image
+12 -12
View File
@@ -72,7 +72,7 @@ jobs:
cross: true
steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v6
with:
ref: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.source_ref || github.ref }}
@@ -97,7 +97,7 @@ jobs:
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -141,7 +141,7 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v6
with:
ref: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.source_ref || github.ref }}
- name: Free Disk Space
@@ -207,7 +207,7 @@ jobs:
org.opencontainers.image.vendor=Chris Lu
- name: Set up QEMU
if: matrix.platform != 'amd64'
uses: docker/setup-qemu-action@v4.2.0
uses: docker/setup-qemu-action@v4
- name: Create BuildKit config
run: |
cat > /tmp/buildkitd.toml <<EOF
@@ -221,13 +221,13 @@ jobs:
buildkitd-config: /tmp/buildkitd.toml
- name: Login to Docker Hub
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.1.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.1.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -275,14 +275,14 @@ jobs:
fi
- name: Login to GHCR
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.1.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_TOKEN }}
- name: Checkout for local scan build
if: needs.setup.outputs.publish != 'true'
uses: actions/checkout@v7
uses: actions/checkout@v6
with:
ref: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.source_ref || github.ref }}
- name: Download pre-built Rust binaries for local scan
@@ -367,7 +367,7 @@ jobs:
output: trivy-results.sarif
exit-code: '0'
- name: Upload Trivy scan results to GitHub Security
uses: github/codeql-action/upload-sarif@v4.37.4
uses: github/codeql-action/upload-sarif@v4
if: always()
with:
sarif_file: trivy-results.sarif
@@ -408,7 +408,7 @@ jobs:
variant: ${{ fromJSON(needs.setup.outputs.variants) }}
steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v6
with:
ref: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.source_ref || github.ref }}
@@ -430,12 +430,12 @@ jobs:
ghcr.io/chrislusf/seaweedfs
tags: type=raw,value=${{ github.event_name == 'workflow_dispatch' && github.event.inputs.image_tag || 'latest' }},suffix=${{ steps.config.outputs.tag_suffix }}
- name: Login to Docker Hub
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.1.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.1.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -25,7 +25,7 @@ jobs:
steps:
-
name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v6
-
name: Docker meta
id: docker_meta
@@ -43,14 +43,14 @@ jobs:
org.opencontainers.image.vendor=Chris Lu
-
name: Set up QEMU
uses: docker/setup-qemu-action@v4.2.0
uses: docker/setup-qemu-action@v4
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
-
name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.1.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+121 -115
View File
@@ -33,7 +33,6 @@ permissions:
env:
RELEASE_TAG: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.release_tag || github.ref_name }}
IMAGE: ghcr.io/chrislusf/seaweedfs
# Limit concurrent builds to avoid rate limits
concurrency:
@@ -57,7 +56,7 @@ jobs:
cross: true
steps:
- name: Checkout
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Install protobuf compiler
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
@@ -80,7 +79,7 @@ jobs:
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -114,45 +113,59 @@ jobs:
weed-volume-large-disk-${{ matrix.arch }}
weed-volume-normal-${{ matrix.arch }}
# One job per (variant, platform) on a native runner, pushed by digest;
# the merge job stitches the digests into one multi-arch tag.
# ── Build Docker containers ─────────────────────────────────────────
build:
needs: [build-rust-binaries]
runs-on: ${{ matrix.runner }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
# Build sequentially to avoid rate limits
max-parallel: 2
matrix:
include:
# Normal volume - multi-arch
- { variant: normal, tag_suffix: "", dockerfile: ./docker/Dockerfile.go_build, build_args: "", rust_variant: normal, platform: linux/amd64, arch: amd64, runner: ubuntu-latest, qemu: false }
- { variant: normal, tag_suffix: "", dockerfile: ./docker/Dockerfile.go_build, build_args: "", rust_variant: normal, platform: linux/arm64, arch: arm64, runner: ubuntu-24.04-arm, qemu: false }
- { variant: normal, tag_suffix: "", dockerfile: ./docker/Dockerfile.go_build, build_args: "", rust_variant: normal, platform: linux/arm/v7, arch: armv7, runner: ubuntu-latest, qemu: true }
- { variant: normal, tag_suffix: "", dockerfile: ./docker/Dockerfile.go_build, build_args: "", rust_variant: normal, platform: linux/386, arch: i386, runner: ubuntu-latest, qemu: false }
- variant: normal
platforms: linux/amd64,linux/arm64,linux/arm/v7,linux/386
dockerfile: ./docker/Dockerfile.go_build
build_args: ""
tag_suffix: ""
rust_variant: normal
# Large disk - multi-arch
- { variant: large_disk, tag_suffix: _large_disk, dockerfile: ./docker/Dockerfile.go_build, build_args: TAGS=5BytesOffset, rust_variant: large-disk, platform: linux/amd64, arch: amd64, runner: ubuntu-latest, qemu: false }
- { variant: large_disk, tag_suffix: _large_disk, dockerfile: ./docker/Dockerfile.go_build, build_args: TAGS=5BytesOffset, rust_variant: large-disk, platform: linux/arm64, arch: arm64, runner: ubuntu-24.04-arm, qemu: false }
- { variant: large_disk, tag_suffix: _large_disk, dockerfile: ./docker/Dockerfile.go_build, build_args: TAGS=5BytesOffset, rust_variant: large-disk, platform: linux/arm/v7, arch: armv7, runner: ubuntu-latest, qemu: true }
- { variant: large_disk, tag_suffix: _large_disk, dockerfile: ./docker/Dockerfile.go_build, build_args: TAGS=5BytesOffset, rust_variant: large-disk, platform: linux/386, arch: i386, runner: ubuntu-latest, qemu: false }
- variant: large_disk
platforms: linux/amd64,linux/arm64,linux/arm/v7,linux/386
dockerfile: ./docker/Dockerfile.go_build
build_args: TAGS=5BytesOffset
tag_suffix: _large_disk
rust_variant: large-disk
# Full tags - multi-arch
- { variant: full, tag_suffix: _full, dockerfile: ./docker/Dockerfile.go_build, build_args: "TAGS=elastic,gocdk,rclone,sqlite,tarantool,tikv,ydb", rust_variant: normal, platform: linux/amd64, arch: amd64, runner: ubuntu-latest, qemu: false }
- { variant: full, tag_suffix: _full, dockerfile: ./docker/Dockerfile.go_build, build_args: "TAGS=elastic,gocdk,rclone,sqlite,tarantool,tikv,ydb", rust_variant: normal, platform: linux/arm64, arch: arm64, runner: ubuntu-24.04-arm, qemu: false }
- variant: full
platforms: linux/amd64,linux/arm64
dockerfile: ./docker/Dockerfile.go_build
build_args: TAGS=elastic,gocdk,rclone,sqlite,tarantool,tikv,ydb
tag_suffix: _full
rust_variant: normal
# Large disk + full tags - multi-arch
- { variant: large_disk_full, tag_suffix: _large_disk_full, dockerfile: ./docker/Dockerfile.go_build, build_args: "TAGS=5BytesOffset,elastic,gocdk,rclone,sqlite,tarantool,tikv,ydb", rust_variant: large-disk, platform: linux/amd64, arch: amd64, runner: ubuntu-latest, qemu: false }
- { variant: large_disk_full, tag_suffix: _large_disk_full, dockerfile: ./docker/Dockerfile.go_build, build_args: "TAGS=5BytesOffset,elastic,gocdk,rclone,sqlite,tarantool,tikv,ydb", rust_variant: large-disk, platform: linux/arm64, arch: arm64, runner: ubuntu-24.04-arm, qemu: false }
- variant: large_disk_full
platforms: linux/amd64,linux/arm64
dockerfile: ./docker/Dockerfile.go_build
build_args: TAGS=5BytesOffset,elastic,gocdk,rclone,sqlite,tarantool,tikv,ydb
tag_suffix: _large_disk_full
rust_variant: large-disk
# RocksDB large disk - amd64 only
- { variant: rocksdb, tag_suffix: _large_disk_rocksdb, dockerfile: ./docker/Dockerfile.rocksdb_large, build_args: "", rust_variant: large-disk, platform: linux/amd64, arch: amd64, runner: ubuntu-latest, qemu: false }
- variant: rocksdb
platforms: linux/amd64
dockerfile: ./docker/Dockerfile.rocksdb_large
build_args: ""
tag_suffix: _large_disk_rocksdb
rust_variant: large-disk
steps:
- name: Skip unselected variant
if: github.event_name == 'workflow_dispatch' && github.event.inputs.variant != 'all' && github.event.inputs.variant != matrix.variant
run: echo "Skipping ${{ matrix.variant }} (${{ matrix.platform }})" && exit 0
- name: Checkout
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Download pre-built Rust binaries
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
@@ -178,31 +191,36 @@ jobs:
- name: Free Disk Space
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
run: |
echo "Available disk space before cleanup:"
df -h
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
sudo apt-get clean
sudo rm -rf /var/lib/apt/lists/*
sudo docker system prune -af --volumes
[ -d ~/.cache/go-build ] && rm -rf ~/.cache/go-build || true
[ -d /go/pkg ] && rm -rf /go/pkg || true
echo "Available disk space after cleanup:"
df -h
- name: Docker meta
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
id: docker_meta
uses: docker/metadata-action@v6
with:
images: ${{ env.IMAGE }}
images: |
chrislusf/seaweedfs
ghcr.io/chrislusf/seaweedfs
tags: type=raw,value=${{ env.RELEASE_TAG }}${{ matrix.tag_suffix }}
flavor: latest=false
labels: |
org.opencontainers.image.title=seaweedfs
org.opencontainers.image.description=SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast!
org.opencontainers.image.vendor=Chris Lu
- name: Set up QEMU
if: (github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant) && matrix.qemu
uses: docker/setup-qemu-action@v4.2.0
if: (github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant) && contains(matrix.platforms, 'arm')
uses: docker/setup-qemu-action@v4
- name: Create BuildKit config
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
run: |
@@ -210,111 +228,91 @@ jobs:
[registry."docker.io"]
mirrors = ["https://mirror.gcr.io"]
EOF
- name: Set up Docker Buildx
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/setup-buildx-action@v4
with:
buildkitd-config: /tmp/buildkitd.toml
- name: Login to Docker Hub
if: (github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant) && github.event_name != 'pull_request'
uses: docker/login-action@v4.1.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.6.0
if: (github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant) && github.event_name != 'pull_request'
uses: docker/login-action@v4.1.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_TOKEN }}
- name: Build and push ${{ matrix.variant }} (${{ matrix.platform }})
- name: Build and push ${{ matrix.variant }}
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
id: build
uses: docker/build-push-action@v7
env:
DOCKER_BUILDKIT: 1
with:
context: ./docker
push: ${{ github.event_name != 'pull_request' }}
file: ${{ matrix.dockerfile }}
platforms: ${{ matrix.platform }}
platforms: ${{ matrix.platforms }}
# Push to GHCR to avoid Docker Hub rate limits on pulls
tags: |
ghcr.io/chrislusf/seaweedfs:${{ env.RELEASE_TAG }}${{ matrix.tag_suffix }}
labels: ${{ steps.docker_meta.outputs.labels }}
# Flat single-platform manifest so imagetools create assembles cleanly.
provenance: false
outputs: type=image,name=${{ env.IMAGE }},push-by-digest=true,name-canonical=true,push=true
cache-from: type=gha,scope=${{ matrix.variant }}-${{ matrix.arch }}
# max only for rocksdb: its RocksDB compile is sha-independent and worth
# keeping; go-build layers are sha-busted every release, so min elsewhere.
cache-to: type=gha,mode=${{ matrix.variant == 'rocksdb' && 'max' || 'min' }},scope=${{ matrix.variant }}-${{ matrix.arch }}
cache-from: type=gha,scope=${{ matrix.variant }}
cache-to: type=gha,mode=max,scope=${{ matrix.variant }}
build-args: |
${{ matrix.build_args }}
BUILDKIT_INLINE_CACHE=1
BRANCH=${{ github.sha }}
${{ matrix.variant == 'rocksdb' && format('ROCKSDB_VERSION={0}', github.event.inputs.rocksdb_version || 'v10.10.1') || '' }}
- name: Export digest
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
- name: Clean up build artifacts
if: always() && (github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant)
run: |
mkdir -p /tmp/digests
digest="${{ steps.build.outputs.digest }}"
touch "/tmp/digests/${digest#sha256:}"
sudo docker system prune -f
sudo rm -rf /tmp/go-build*
- name: Upload digest
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: actions/upload-artifact@v7
with:
name: digest-${{ matrix.variant }}-${{ matrix.arch }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1
# Assemble each variant's per-platform digests into one tag, then mirror to Docker Hub.
merge:
needs: [build]
copy-to-dockerhub:
runs-on: ubuntu-latest
needs: [build]
if: github.event_name != 'pull_request'
strategy:
fail-fast: false
matrix:
variant: [normal, large_disk, full, large_disk_full, rocksdb]
include:
- { variant: normal, tag_suffix: "" }
- { variant: large_disk, tag_suffix: _large_disk }
- { variant: full, tag_suffix: _full }
- { variant: large_disk_full, tag_suffix: _large_disk_full }
- { variant: rocksdb, tag_suffix: _large_disk_rocksdb }
- variant: normal
tag_suffix: ""
- variant: large_disk
tag_suffix: _large_disk
- variant: full
tag_suffix: _full
- variant: large_disk_full
tag_suffix: _large_disk_full
- variant: rocksdb
tag_suffix: _large_disk_rocksdb
steps:
- name: Download digests
- name: Login to Docker Hub
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: actions/download-artifact@v8
uses: docker/login-action@v4.1.0
with:
pattern: digest-${{ matrix.variant }}-*
merge-multiple: true
path: /tmp/digests
- name: Set up Docker Buildx
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/setup-buildx-action@v4
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.1.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_TOKEN }}
- name: Create multi-arch tag on GHCR
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
working-directory: /tmp/digests
run: |
docker buildx imagetools create \
-t ${{ env.IMAGE }}:${{ env.RELEASE_TAG }}${{ matrix.tag_suffix }} \
$(printf '${{ env.IMAGE }}@sha256:%s ' *)
docker buildx imagetools inspect ${{ env.IMAGE }}:${{ env.RELEASE_TAG }}${{ matrix.tag_suffix }}
- name: Login to Docker Hub
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.6.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Install crane
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
run: |
@@ -322,37 +320,45 @@ jobs:
curl -sL "https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz" | tar xz
sudo mv crane /usr/local/bin/
crane version
- name: Copy ${{ matrix.variant }} to Docker Hub
- name: Copy ${{ matrix.variant }} from GHCR to Docker Hub
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
run: |
# Function to retry with exponential backoff
retry_with_backoff() {
local max_attempts=5
local timeout=1
local attempt=1
local exit_code=0
while [ $attempt -le $max_attempts ]; do
if "$@"; then
return 0
else
exit_code=$?
fi
if [ $attempt -lt $max_attempts ]; then
echo "Attempt $attempt failed. Retrying in ${timeout}s..." >&2
sleep $timeout
timeout=$((timeout * 2))
fi
attempt=$((attempt + 1))
done
echo "Command failed after $max_attempts attempts" >&2
return $exit_code
}
# Copy multi-arch image from GHCR to Docker Hub with retry
# This is much more efficient than pulling/pushing individual arch images
echo "Copying ${{ matrix.variant }} from GHCR to Docker Hub..."
retry_with_backoff crane copy \
${{ env.IMAGE }}:${{ env.RELEASE_TAG }}${{ matrix.tag_suffix }} \
ghcr.io/chrislusf/seaweedfs:${{ env.RELEASE_TAG }}${{ matrix.tag_suffix }} \
chrislusf/seaweedfs:${{ env.RELEASE_TAG }}${{ matrix.tag_suffix }}
echo "Copied ${{ matrix.variant }} to Docker Hub"
echo "Successfully copied ${{ matrix.variant }} to Docker Hub"
# Report-only trivy scan: uploads fixable HIGH/CRITICAL findings to GitHub
# Security for visibility, but never blocks the release. Releases (including
@@ -361,7 +367,7 @@ jobs:
# CVE, base-image lag).
trivy-scan:
runs-on: ubuntu-latest
needs: [merge]
needs: [build]
if: github.event_name == 'push'
continue-on-error: true
strategy:
@@ -374,7 +380,7 @@ jobs:
variant: large_disk
steps:
- name: Login to GHCR
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.1.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -388,7 +394,7 @@ jobs:
# Scan the multi-arch tag on GHCR (already pushed by the build job).
# Trivy scans the runner's native platform; OS packages are identical
# across architectures since they all share the same alpine base.
image-ref: ${{ env.IMAGE }}:${{ env.RELEASE_TAG }}${{ matrix.source_suffix }}
image-ref: ghcr.io/chrislusf/seaweedfs:${{ env.RELEASE_TAG }}${{ matrix.source_suffix }}
scanners: vuln
vuln-type: os,library
severity: HIGH,CRITICAL
@@ -400,7 +406,7 @@ jobs:
- name: Upload Trivy scan results to GitHub Security
if: always()
uses: github/codeql-action/upload-sarif@v4.37.4
uses: github/codeql-action/upload-sarif@v4
with:
sarif_file: trivy-results.sarif
category: trivy-${{ matrix.variant }}
@@ -412,7 +418,7 @@ jobs:
# trivy-scan: vuln findings are reported but do not block `latest`.
tag-latest:
runs-on: ubuntu-latest
needs: [merge]
needs: [copy-to-dockerhub]
if: github.event_name == 'push'
strategy:
matrix:
@@ -423,13 +429,13 @@ jobs:
latest_tag: latest_large_disk
steps:
- name: Login to Docker Hub
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.1.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.1.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -469,24 +475,24 @@ jobs:
SRC_TAG="${{ env.RELEASE_TAG }}${{ matrix.source_suffix }}"
DST_TAG="${{ matrix.latest_tag }}"
echo "Tagging ${{ env.IMAGE }}:${SRC_TAG} as ${DST_TAG}"
retry_with_backoff crane tag "${{ env.IMAGE }}:${SRC_TAG}" "${DST_TAG}"
echo "Tagging ghcr.io/chrislusf/seaweedfs:${SRC_TAG} as ${DST_TAG}"
retry_with_backoff crane tag "ghcr.io/chrislusf/seaweedfs:${SRC_TAG}" "${DST_TAG}"
echo "Tagging chrislusf/seaweedfs:${SRC_TAG} as ${DST_TAG}"
retry_with_backoff crane tag "chrislusf/seaweedfs:${SRC_TAG}" "${DST_TAG}"
helm-release:
runs-on: ubuntu-latest
needs: [build]
needs: [copy-to-dockerhub]
if: github.event_name == 'push' || github.event_name == 'workflow_dispatch'
permissions:
contents: write
pages: write
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v6
- name: Publish Helm charts
uses: stefanprodan/helm-gh-pages@v1.7.0
with:
with:
token: ${{ secrets.GITHUB_TOKEN }}
charts_dir: k8s/charts
target_dir: helm
@@ -25,7 +25,7 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v2
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v2
- name: Prepare Docker tag
id: tag
@@ -82,13 +82,13 @@ jobs:
echo "seaweedfs_ref=$seaweed" >> "$GITHUB_OUTPUT"
- name: Set up QEMU
uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v1
uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v1
- name: Login to Docker Hub
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v1
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v1
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+4 -7
View File
@@ -21,22 +21,19 @@ jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'telemetry/server/go.mod'
go-version-file: 'go.mod'
- name: Build Telemetry Server
if: github.event_name == 'workflow_dispatch' && inputs.deploy
run: |
# telemetry/server is its own Go module; build from within it
cd telemetry/server
go mod tidy
echo "Building telemetry server..."
GOOS=linux GOARCH=amd64 go build -o ../../telemetry-server .
cd ../..
GOOS=linux GOARCH=amd64 go build -o telemetry-server ./telemetry/server/main.go
ls -la telemetry-server
echo "Build completed successfully"
+1 -1
View File
@@ -9,6 +9,6 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: 'Checkout Repository'
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
- name: 'Dependency Review'
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294
+3 -20
View File
@@ -3,20 +3,8 @@ name: "End to End"
on:
push:
branches: [ master ]
paths:
- 'weed/**'
- 'docker/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/e2e.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/**'
- 'docker/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/e2e.yml'
concurrency:
group: ${{ github.head_ref }}/e2e
@@ -36,23 +24,18 @@ jobs:
timeout-minutes: 30
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Cache Docker layers
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-e2e-${{ github.sha }}
+2 -14
View File
@@ -3,20 +3,8 @@ name: "EC Integration Tests"
on:
push:
branches: [ master ]
paths:
- 'weed/**'
- 'test/erasure_coding/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/ec-integration-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/**'
- 'test/erasure_coding/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/ec-integration-tests.yml'
permissions:
contents: read
@@ -28,13 +16,13 @@ jobs:
timeout-minutes: 30
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Build weed binary
run: |
+2 -7
View File
@@ -8,7 +8,6 @@ on:
- 'weed/cluster/**'
- 'test/fuse_dlm/**'
- '.github/workflows/fuse-dlm-integration.yml'
- '.github/actions/fix-fusermount-setuid/**'
push:
branches: [master]
paths:
@@ -16,7 +15,6 @@ on:
- 'weed/mount/**'
- 'weed/cluster/**'
- 'test/fuse_dlm/**'
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.head_ref || github.ref }}/fuse-dlm-integration
@@ -33,10 +31,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
@@ -47,9 +45,6 @@ jobs:
echo 'user_allow_other' | sudo tee -a /etc/fuse.conf
sudo chmod 644 /etc/fuse.conf
- name: Repair the fusermount3 setuid bit
uses: ./.github/actions/fix-fusermount-setuid
- name: Build SeaweedFS
run: go build -o weed/weed -buildvcs=false ./weed
+2 -7
View File
@@ -7,14 +7,12 @@ on:
- 'weed/**'
- 'test/fuse_integration/**'
- '.github/workflows/fuse-integration.yml'
- '.github/actions/fix-fusermount-setuid/**'
pull_request:
branches: [ master, main ]
paths:
- 'weed/**'
- 'test/fuse_integration/**'
- '.github/workflows/fuse-integration.yml'
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.head_ref }}/fuse-integration
@@ -31,10 +29,10 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
@@ -51,9 +49,6 @@ jobs:
fusermount3 --version || fusermount --version || true
ls -la /dev/fuse
- name: Repair the fusermount3 setuid bit
uses: ./.github/actions/fix-fusermount-setuid
- name: Build SeaweedFS
run: |
cd weed
+2 -7
View File
@@ -11,7 +11,6 @@ on:
- 'weed/pb/filer.proto'
- 'test/fuse_p2p/**'
- '.github/workflows/fuse-p2p-integration.yml'
- '.github/actions/fix-fusermount-setuid/**'
push:
branches: [master]
paths:
@@ -22,7 +21,6 @@ on:
- 'weed/pb/mount_peer.proto'
- 'weed/pb/filer.proto'
- 'test/fuse_p2p/**'
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.head_ref || github.ref }}/fuse-p2p-integration
@@ -39,10 +37,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
@@ -53,9 +51,6 @@ jobs:
echo 'user_allow_other' | sudo tee -a /etc/fuse.conf
sudo chmod 644 /etc/fuse.conf
- name: Repair the fusermount3 setuid bit
uses: ./.github/actions/fix-fusermount-setuid
- name: Build SeaweedFS
run: go build -o weed/weed -buildvcs=false ./weed
+6 -80
View File
@@ -3,18 +3,8 @@ name: "go: build binary"
on:
push:
branches: [ master ]
paths:
- '**/*.go'
- 'go.mod'
- 'go.sum'
- '.github/workflows/go.yml'
pull_request:
branches: [ master ]
paths:
- '**/*.go'
- 'go.mod'
- 'go.sum'
- '.github/workflows/go.yml'
concurrency:
group: ${{ github.head_ref }}/go
@@ -30,9 +20,9 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Get dependencies
@@ -47,92 +37,28 @@ jobs:
# Fail only if there are actual vet errors (not counting the filtered lock warnings)
if grep -q "vet:" vet-output.txt; then exit 1; fi
vet-32bit:
name: Go Vet 32-bit
runs-on: ubuntu-latest
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
- name: Go Vet linux/386 (type-checks code and tests for 32-bit int overflows)
run: |
GOOS=linux GOARCH=386 go vet ./... 2>&1 | grep -v "MessageState contains sync.Mutex" | grep -v "IdentityAccessManagement contains sync.RWMutex" | tee vet-32bit-output.txt
if grep -q "vet:" vet-32bit-output.txt; then exit 1; fi
build:
name: Build
runs-on: ubuntu-latest
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Build
run: cd weed; go build -tags "elastic gocdk sqlite ydb tarantool tikv rclone" -v .
build-cross:
name: Build cross-platform (${{ matrix.goos }}/${{ matrix.goarch }})
runs-on: ubuntu-latest
strategy:
# One target's breakage should not hide the other three.
fail-fast: false
matrix:
include:
- { goos: windows, goarch: amd64 }
- { goos: windows, goarch: arm64 }
- { goos: freebsd, goarch: amd64 }
- { goos: darwin, goarch: arm64 }
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
# Nothing else on a PR compiles these, so per-OS syscall constants creep
# back into shared files unnoticed and only a release build catches it.
- name: Cross-compile
env:
GOOS: ${{ matrix.goos }}
GOARCH: ${{ matrix.goarch }}
run: |
go build ./weed/...
# Tests too: they reach for per-OS syscall constants the build does
# not, and only compiling them catches an untagged one.
go vet ./weed/mount/... ./weed/command/... 2>&1 |
grep -v "MessageState contains sync.Mutex" | tee /tmp/vet.txt
if grep -q "vet:" /tmp/vet.txt; then exit 1; fi
test:
name: Test
runs-on: ubuntu-latest
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Test
run: cd weed; go test -tags "elastic gocdk sqlite ydb tarantool tikv rclone" -v ./...
test-32bit:
name: Test 32-bit
runs-on: ubuntu-latest
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
# 386 test binaries run natively on the amd64 runner. This catches what vet
# can't: unaligned 64-bit atomics and arithmetic that wraps at runtime.
# -short skips the e2e suites already covered on amd64.
- name: Test linux/386
run: cd weed; GOOS=linux GOARCH=386 go test -short ./...
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -15,7 +15,7 @@ jobs:
helm-release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v6
with:
fetch-depth: 0
+2 -2
View File
@@ -25,10 +25,10 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
-183
View File
@@ -1,183 +0,0 @@
name: "release: java clients"
# Publishes the SeaweedFS Java clients (seaweedfs-client and
# seaweedfs-hadoop3-client) to Maven Central via the Sonatype Central Portal.
#
# Required repository secrets:
# MAVEN_CENTRAL_USERNAME - Central Portal user token username (central.sonatype.com -> Account -> Generate User Token)
# MAVEN_CENTRAL_PASSWORD - Central Portal user token password
# MAVEN_GPG_PRIVATE_KEY - ASCII-armored signing key:
# gpg --homedir <keyring> --armor --export-secret-keys <KEYID> > key.asc
# MAVEN_GPG_PASSPHRASE - passphrase for that key
#
# A plain run publishes to Central. Check dry_run to exercise secrets, GPG
# signing, and the build without committing or publishing.
on:
workflow_dispatch:
inputs:
version:
description: "Version to publish, e.g. 4.39. Leave blank to use the current SeaweedFS version."
type: string
required: false
dry_run:
description: "Build and GPG-sign only; skip the version commit and the Central upload"
type: boolean
default: false
permissions:
contents: write
concurrency:
group: java-release
cancel-in-progress: false
env:
# Java 17 needs these opens for the Maven publishing plugins' reflective access.
MAVEN_OPTS: >-
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.text=ALL-UNNAMED
--add-opens=java.desktop/java.awt.font=ALL-UNNAMED
jobs:
publish:
name: Publish to Maven Central
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
# Full history so the version-bump commit can rebase onto a moved master.
fetch-depth: 0
# An explicit input wins; otherwise fall back to the current SeaweedFS
# version (MAJOR.MINOR) from constants.go, matching its %d.%02d formatting.
- name: Resolve version
id: resolve
env:
INPUT_VERSION: ${{ inputs.version }}
run: |
set -euo pipefail
VERSION="$INPUT_VERSION"
if [ -z "$VERSION" ]; then
CONST=weed/util/version/constants.go
MAJOR=$(grep -oP 'MAJOR_VERSION\s*=\s*int32\(\K[0-9]+' "$CONST")
MINOR=$(grep -oP 'MINOR_VERSION\s*=\s*int32\(\K[0-9]+' "$CONST")
VERSION=$(printf '%d.%02d' "$MAJOR" "$MINOR")
echo "No version given; using current SeaweedFS version ${VERSION}."
fi
if ! [[ "$VERSION" =~ ^[0-9]+\.[0-9]+$ ]]; then
echo "::error::version must be MAJOR.MINOR, e.g. 4.39 (got '$VERSION')"
exit 1
fi
echo "version=${VERSION}" >> "$GITHUB_OUTPUT"
- name: Set up JDK 17
uses: actions/setup-java@v5
with:
java-version: '17'
distribution: 'temurin'
cache: 'maven'
server-id: central
server-username: MAVEN_CENTRAL_USERNAME
server-password: MAVEN_CENTRAL_PASSWORD
gpg-private-key: ${{ secrets.MAVEN_GPG_PRIVATE_KEY }}
gpg-passphrase: MAVEN_GPG_PASSPHRASE
# client carries the version literally; hdfs3 derives its own version and
# its seaweedfs-client dependency from the seaweedfs.client.version property.
- name: Set versions in poms
env:
VERSION: ${{ steps.resolve.outputs.version }}
run: |
set -euo pipefail
V=org.codehaus.mojo:versions-maven-plugin:2.16.2
mvn -B -ntp -f other/java/client/pom.xml "${V}:set" \
-DnewVersion="$VERSION" -DgenerateBackupPoms=false
mvn -B -ntp -f other/java/hdfs3/pom.xml "${V}:set-property" \
-Dproperty=seaweedfs.client.version -DnewVersion="$VERSION" -DgenerateBackupPoms=false
- name: Commit version bump
if: ${{ !inputs.dry_run }}
env:
VERSION: ${{ steps.resolve.outputs.version }}
run: |
set -euo pipefail
if git diff --quiet; then
echo "Poms already at ${VERSION}; nothing to commit."
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add other/java/client/pom.xml other/java/hdfs3/pom.xml
git commit -m "java ${VERSION}"
# Rebase and retry so a concurrent push to master doesn't lose the bump.
for attempt in 1 2 3 4 5; do
if git push origin HEAD:master; then
exit 0
fi
echo "push rejected (attempt ${attempt}); rebasing onto latest master"
git pull --rebase origin master
done
echo "::error::could not push version bump after retries"
exit 1
# Tests are skipped here: the integration tests need a live filer, and the
# offline unit tests already run on every push in java_unit_tests.yml.
# dry_run stops at `install` (build + GPG sign, no upload). A real run deploys.
# client goes first either way so its install populates the local repo for hdfs3.
- name: Publish seaweedfs-client
working-directory: other/java/client
env:
MAVEN_CENTRAL_USERNAME: ${{ secrets.MAVEN_CENTRAL_USERNAME }}
MAVEN_CENTRAL_PASSWORD: ${{ secrets.MAVEN_CENTRAL_PASSWORD }}
MAVEN_GPG_PASSPHRASE: ${{ secrets.MAVEN_GPG_PASSPHRASE }}
DRY_RUN: ${{ inputs.dry_run }}
run: |
set -euo pipefail
[ "$DRY_RUN" = "true" ] && GOAL="clean install" || GOAL="clean deploy"
mvn -B -ntp $GOAL -DskipTests
- name: Publish seaweedfs-hadoop3-client
working-directory: other/java/hdfs3
env:
MAVEN_CENTRAL_USERNAME: ${{ secrets.MAVEN_CENTRAL_USERNAME }}
MAVEN_CENTRAL_PASSWORD: ${{ secrets.MAVEN_CENTRAL_PASSWORD }}
MAVEN_GPG_PASSPHRASE: ${{ secrets.MAVEN_GPG_PASSPHRASE }}
DRY_RUN: ${{ inputs.dry_run }}
run: |
set -euo pipefail
[ "$DRY_RUN" = "true" ] && GOAL="clean install" || GOAL="clean deploy"
mvn -B -ntp $GOAL -DskipTests
# Bump every version the wiki quotes for the Java clients. Runs only after
# a successful real publish. Uses GITHUB_TOKEN; set a WIKI_TOKEN secret if
# the token can't push to the wiki.
- name: Update wiki client versions
if: ${{ !inputs.dry_run }}
env:
VERSION: ${{ steps.resolve.outputs.version }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
WIKI_TOKEN: ${{ secrets.WIKI_TOKEN }}
run: |
set -euo pipefail
TOKEN="${WIKI_TOKEN:-$GH_TOKEN}"
git clone -q "https://x-access-token:${TOKEN}@github.com/${{ github.repository }}.wiki.git" wiki
cd wiki
for f in $(grep -rlE "seaweedfs-(client|hadoop3-client)" --include="*.md" .); do
NEWV="$VERSION" perl -0777 -i -pe '
my $v = $ENV{NEWV};
s{(<artifactId>seaweedfs-(?:client|hadoop3-client)</artifactId>\s*<version>)[0-9]+\.[0-9]+(</version>)}{$1$v$2}g;
s{(seaweedfs-(?:client|hadoop3-client):)[0-9]+\.[0-9]+}{$1$v}g;
s{(seaweedfs-(?:client|hadoop3-client)-)[0-9]+\.[0-9]+(\.jar)}{$1$v$2}g;
s{(seaweedfs-(?:client|hadoop3-client)/)[0-9]+\.[0-9]+(/)}{$1$v$2}g;
' "$f"
done
if git diff --quiet; then
echo "Wiki already references ${VERSION}."
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git commit -am "java clients ${VERSION}"
git push
+1 -1
View File
@@ -23,7 +23,7 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Java
uses: actions/setup-java@v5
+2 -23
View File
@@ -3,24 +3,8 @@ name: "Kafka Quick Test (Load Test with Schema Registry)"
on:
push:
branches: [ master ]
paths:
- 'weed/mq/**'
- 'weed/pb/mq_pb/**'
- 'weed/pb/schema_pb/**'
- 'test/kafka/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/kafka-quicktest.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/mq/**'
- 'weed/pb/mq_pb/**'
- 'weed/pb/schema_pb/**'
- 'test/kafka/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/kafka-quicktest.yml'
workflow_dispatch: # Allow manual trigger
concurrency:
@@ -37,10 +21,10 @@ jobs:
timeout-minutes: 15
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
cache: true
@@ -48,11 +32,6 @@ jobs:
**/go.sum
id: go
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
+14 -30
View File
@@ -3,24 +3,8 @@ name: "Kafka Gateway Tests"
on:
push:
branches: [ master ]
paths:
- 'weed/mq/**'
- 'weed/pb/mq_pb/**'
- 'weed/pb/schema_pb/**'
- 'test/kafka/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/kafka-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/mq/**'
- 'weed/pb/mq_pb/**'
- 'weed/pb/schema_pb/**'
- 'test/kafka/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/kafka-tests.yml'
concurrency:
group: ${{ github.head_ref }}/kafka-tests
@@ -51,13 +35,13 @@ jobs:
CONTAINER_ID: ${{ matrix.container-id }}
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
id: go
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Setup Container Environment
run: |
@@ -96,13 +80,13 @@ jobs:
CONTAINER_ID: ${{ matrix.container-id }}
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
id: go
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Setup Integration Container Environment
run: |
@@ -143,10 +127,10 @@ jobs:
CONTAINER_ID: ${{ matrix.container-id }}
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
cache: true
@@ -322,10 +306,10 @@ jobs:
CONTAINER_ID: ${{ matrix.container-id }}
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
cache: true
@@ -484,10 +468,10 @@ jobs:
CONTAINER_ID: ${{ matrix.container-id }}
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
cache: true
@@ -642,10 +626,10 @@ jobs:
CONTAINER_ID: ${{ matrix.container-id }}
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
cache: true
@@ -803,13 +787,13 @@ jobs:
CONTAINER_ID: ${{ matrix.container-id }}
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
id: go
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Setup Protocol Container Environment
run: |
+4 -4
View File
@@ -37,10 +37,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -82,10 +82,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -1,79 +0,0 @@
name: "Master Cold Start Tests"
on:
push:
branches: [ master ]
paths:
- 'weed/server/master_*.go'
- 'weed/topology/**'
- 'weed/operation/**'
- 'test/master_cold_start/**'
- 'test/testutil/**'
- '.github/workflows/master-cold-start-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/server/master_*.go'
- 'weed/topology/**'
- 'weed/operation/**'
- 'test/master_cold_start/**'
- 'test/testutil/**'
- '.github/workflows/master-cold-start-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/master-cold-start-tests
cancel-in-progress: true
permissions:
contents: read
jobs:
master-cold-start-tests:
name: Master Cold Start Tests
runs-on: ubuntu-22.04
timeout-minutes: 10
steps:
- name: Check out code
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
- name: Build weed binary
run: |
cd weed && go install -buildvcs=false
- name: Run master cold start tests
# test/master_cold_start boots a fresh master plus empty volume
# servers and requires the very first assign (HTTP and gRPC, no
# client retries) to complete a write: the assign that triggers
# volume growth must wait for it instead of failing with
# "volume growth in progress".
run: |
export WEED_BINARY=$(go env GOPATH)/bin/weed
go test -v -timeout=8m ./test/master_cold_start/...
- name: Collect server logs on failure
if: failure()
run: |
# test/master_cold_start/cluster.go keeps failing-test dirs created
# via os.MkdirTemp("", "seaweedfs_master_cold_start_it_") with each
# process log under <baseDir>/logs/.
mkdir -p /tmp/master-cold-start-logs
find /tmp -maxdepth 1 -type d -name "seaweedfs_master_cold_start_it_*" 2>/dev/null | while read dir; do
echo "Found test directory: $dir"
cp -r "$dir" /tmp/master-cold-start-logs/ 2>/dev/null || true
done
find /tmp/master-cold-start-logs -type f -name "*.log" -print -exec tail -n 100 {} \; 2>/dev/null || echo "No logs found"
- name: Archive logs
if: failure()
uses: actions/upload-artifact@v7
with:
name: master-cold-start-test-logs
path: /tmp/master-cold-start-logs/
retention-days: 7
@@ -8,7 +8,6 @@ on:
- 'weed/pb/filer_pb/**'
- 'weed/util/log_buffer/**'
- 'weed/server/filer_grpc_server_sub_meta.go'
- 'weed/server/master_grpc_server.go'
- 'weed/command/filer_backup.go'
- 'test/metadata_subscribe/**'
- '.github/workflows/metadata-subscribe-tests.yml'
@@ -19,7 +18,6 @@ on:
- 'weed/pb/filer_pb/**'
- 'weed/util/log_buffer/**'
- 'weed/server/filer_grpc_server_sub_meta.go'
- 'weed/server/master_grpc_server.go'
- 'weed/command/filer_backup.go'
- 'test/metadata_subscribe/**'
- '.github/workflows/metadata-subscribe-tests.yml'
@@ -42,10 +40,10 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
@@ -1,124 +0,0 @@
name: "mount: windows conformance"
on:
push:
branches: [ master ]
paths:
- 'weed/mount/**'
- 'weed/command/mount*.go'
- 'weed/storage/volume_vacuum*.go'
- 'weed/storage/volume_loading.go'
- 'weed/storage/disk_location.go'
- 'test/winfsp-conformance/**'
- '.github/workflows/mount-windows-conformance.yml'
# No base branch filter: this is the only thing that runs the Windows mount,
# so it should cover a pull request stacked on another one too.
pull_request:
paths:
- 'weed/mount/**'
- 'weed/command/mount*.go'
- 'weed/storage/volume_vacuum*.go'
- 'weed/storage/volume_loading.go'
- 'weed/storage/disk_location.go'
- 'test/winfsp-conformance/**'
- '.github/workflows/mount-windows-conformance.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions:
contents: read
jobs:
conformance:
name: WinFsp conformance
runs-on: windows-latest
timeout-minutes: 60
env:
# The runner ships MinGW, so cgo is on by default and cgofuse picks its
# cgo variant, which wants WinFsp's headers. The nocgo variant loads
# winfsp-x64.dll at run time instead, which is how weed.exe is released.
CGO_ENABLED: 0
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
# cgofuse loads winfsp-x64.dll at run time, so WinFsp is needed here but
# not to build.
- name: Install WinFsp
run: choco install winfsp -y --no-progress
- name: Build weed.exe
run: go build -o weed.exe ./weed
# The runner tears down a step's process tree when its shell exits, so a
# cluster started in one step is gone by the next. Everything that needs
# the cluster and the mount alive has to share a step.
- name: Mount and run winfsp-tests
shell: pwsh
run: |
$ErrorActionPreference = 'Stop'
function Test-Port($port) {
# A plain connect, because Test-NetConnection has reported success
# here for a port nothing was listening on.
$client = New-Object System.Net.Sockets.TcpClient
try { $client.Connect('127.0.0.1', $port); return $client.Connected }
catch { return $false }
finally { $client.Dispose() }
}
function Start-Mount($log) {
Start-Process -FilePath .\weed.exe `
-ArgumentList '-logtostderr','mount','-filer=127.0.0.1:8888','-dir=S:' `
-RedirectStandardOutput "C:\$log.log" -RedirectStandardError "C:\$log.err.log"
$deadline = (Get-Date).AddMinutes(2)
while ((Get-Date) -lt $deadline) {
if (Test-Path S:\) { Write-Host "S: is mounted"; return }
Start-Sleep -Seconds 2
}
Get-Content "C:\$log.log", "C:\$log.err.log" -ErrorAction SilentlyContinue
throw "S: never appeared"
}
New-Item -ItemType Directory -Force -Path C:\seaweed-data | Out-Null
# -ip pins the cluster to loopback; it otherwise advertises and binds
# the runner's LAN address, which 127.0.0.1 cannot reach.
Start-Process -FilePath .\weed.exe `
-ArgumentList '-logtostderr','mini','-dir=C:\seaweed-data','-ip=127.0.0.1' `
-RedirectStandardOutput C:\seaweed-mini.log -RedirectStandardError C:\seaweed-mini.err.log
$deadline = (Get-Date).AddMinutes(3)
while ((Get-Date) -lt $deadline) {
# The mount dials grpc, not http, so both ports have to answer.
if ((Test-Port 8888) -and (Test-Port 18888)) { break }
Start-Sleep -Seconds 3
}
if (-not ((Test-Port 8888) -and (Test-Port 18888))) {
Get-Content C:\seaweed-mini.log, C:\seaweed-mini.err.log -ErrorAction SilentlyContinue
throw "filer never came up"
}
Write-Host "filer is up on http 8888 and grpc 18888"
Start-Mount 'seaweed-mount'
Write-Host "::group::winfsp-tests"
& pwsh -File test/winfsp-conformance/run.ps1 -MountPoint S:\
$code = $LASTEXITCODE
Write-Host "::endgroup::"
if ($code -ne 0) { throw "winfsp-tests failed with exit $code" }
- name: Logs
if: always()
shell: pwsh
run: |
foreach ($f in 'C:\seaweed-mount.log','C:\seaweed-mount.err.log','C:\seaweed-mini.log','C:\seaweed-mini.err.log') {
if (Test-Path $f) { Write-Host "===== $f"; Get-Content $f -Tail 200 }
}
-194
View File
@@ -1,194 +0,0 @@
name: "mount: windows"
on:
push:
branches: [ master ]
paths:
- 'weed/mount/**'
- 'weed/command/mount*.go'
- 'weed/storage/volume_vacuum*.go'
- 'weed/storage/volume_loading.go'
- 'weed/storage/disk_location.go'
- 'test/winfsp/**'
- '.github/workflows/mount-windows.yml'
# No base branch filter: this is the only thing that runs the Windows mount,
# so it should cover a pull request stacked on another one too.
pull_request:
paths:
- 'weed/mount/**'
- 'weed/command/mount*.go'
- 'weed/storage/volume_vacuum*.go'
- 'weed/storage/volume_loading.go'
- 'weed/storage/disk_location.go'
- 'test/winfsp/**'
- '.github/workflows/mount-windows.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions:
contents: read
jobs:
mount-windows:
name: Mount on Windows
runs-on: windows-latest
timeout-minutes: 40
env:
# The runner ships MinGW, so cgo is on by default and cgofuse picks its
# cgo variant, which wants WinFsp's headers. The nocgo variant loads
# winfsp-x64.dll at run time instead, which is how weed.exe is released.
CGO_ENABLED: 0
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
# cgofuse loads winfsp-x64.dll at run time, so WinFsp is needed here but
# not to build.
- name: Install WinFsp
run: choco install winfsp -y --no-progress
- name: Build weed.exe
run: go build -o weed.exe ./weed
# The runner tears down a step's process tree when its shell exits, so a
# cluster started in one step is gone by the next. Everything that needs
# the cluster and the mount alive has to share a step.
- name: Mount and exercise
shell: pwsh
run: |
$ErrorActionPreference = 'Stop'
function Test-Port($port) {
# A plain connect, because Test-NetConnection has reported success
# here for a port nothing was listening on.
$client = New-Object System.Net.Sockets.TcpClient
try { $client.Connect('127.0.0.1', $port); return $client.Connected }
catch { return $false }
finally { $client.Dispose() }
}
function Start-Mount($log) {
Start-Process -FilePath .\weed.exe `
-ArgumentList '-logtostderr','mount','-filer=127.0.0.1:8888','-dir=S:' `
-RedirectStandardOutput "C:\$log.log" -RedirectStandardError "C:\$log.err.log"
$deadline = (Get-Date).AddMinutes(2)
while ((Get-Date) -lt $deadline) {
if (Test-Path S:\) { Write-Host "S: is mounted"; return }
Start-Sleep -Seconds 2
}
Get-Content "C:\$log.log", "C:\$log.err.log" -ErrorAction SilentlyContinue
throw "S: never appeared"
}
function Stop-Mount {
Get-CimInstance Win32_Process -Filter "Name = 'weed.exe'" |
Where-Object { $_.CommandLine -like '*mount*' } |
ForEach-Object { Stop-Process -Id $_.ProcessId -Force }
$deadline = (Get-Date).AddMinutes(1)
while ((Get-Date) -lt $deadline -and (Test-Path S:\)) { Start-Sleep -Seconds 2 }
if (Test-Path S:\) { throw "S: still present after stopping the mount" }
Write-Host "unmounted"
}
function Invoke-Tests($label, [string[]]$goArgs) {
Write-Host "::group::$label"
& go @goArgs
$code = $LASTEXITCODE
Write-Host "::endgroup::"
if ($code -ne 0) { throw "$label failed with exit $code" }
}
New-Item -ItemType Directory -Force -Path C:\seaweed-data | Out-Null
# -ip pins the cluster to loopback; it otherwise advertises and binds
# the runner's LAN address, which 127.0.0.1 cannot reach.
Start-Process -FilePath .\weed.exe `
-ArgumentList '-logtostderr','mini','-dir=C:\seaweed-data','-ip=127.0.0.1' `
-RedirectStandardOutput C:\seaweed-mini.log -RedirectStandardError C:\seaweed-mini.err.log
$deadline = (Get-Date).AddMinutes(3)
while ((Get-Date) -lt $deadline) {
# The mount dials grpc, not http, so both ports have to answer.
if ((Test-Port 8888) -and (Test-Port 18888)) { break }
Start-Sleep -Seconds 3
}
if (-not ((Test-Port 8888) -and (Test-Port 18888))) {
Get-Content C:\seaweed-mini.log, C:\seaweed-mini.err.log -ErrorAction SilentlyContinue
throw "filer never came up"
}
Write-Host "filer is up on http 8888 and grpc 18888"
Start-Mount 'seaweed-mount'
Invoke-Tests 'exercise' @('test','-v','-timeout','20m','./test/winfsp','-mountpoint=S:\')
Invoke-Tests 'persist-write' @('test','-v','-timeout','15m','./test/winfsp','-run','TestPersistence','-mountpoint=S:\','-phase=write','-filer=127.0.0.1:8888')
Stop-Mount
Start-Mount 'seaweed-remount'
Invoke-Tests 'persist-verify' @('test','-v','-timeout','15m','./test/winfsp','-run','TestPersistence','-mountpoint=S:\','-phase=verify')
# WinFsp creates the mount directory itself, so the path must not
# exist; only its parent has to.
Write-Host "::group::mount over a directory"
Stop-Mount
Remove-Item C:\seaweed-mnt -Recurse -Force -ErrorAction SilentlyContinue
Start-Process -FilePath .\weed.exe `
-ArgumentList '-logtostderr','mount','-filer=127.0.0.1:8888','-dir=C:\seaweed-mnt' `
-RedirectStandardOutput C:\seaweed-dirmount.log -RedirectStandardError C:\seaweed-dirmount.err.log
$deadline = (Get-Date).AddMinutes(2)
$ok = $false
while ((Get-Date) -lt $deadline) {
# Listing succeeds on the plain empty directory too, so wait for the
# reparse point WinFsp turns it into. Otherwise this step passes
# without a mount and writes to local disk.
$item = Get-Item C:\seaweed-mnt -Force -ErrorAction SilentlyContinue
if ($null -ne $item -and $item.Attributes.ToString() -like '*ReparsePoint*') { $ok = $true; break }
Start-Sleep -Seconds 2
}
if (-not $ok) {
Get-Content C:\seaweed-dirmount.log, C:\seaweed-dirmount.err.log -ErrorAction SilentlyContinue
throw "mounting over a directory failed"
}
Set-Content -Path C:\seaweed-mnt\dirmount.txt -Value 'via directory mount'
if ((Get-Content C:\seaweed-mnt\dirmount.txt) -ne 'via directory mount') { throw "readback through the directory mount differs" }
Remove-Item C:\seaweed-mnt\dirmount.txt -Force
Get-CimInstance Win32_Process -Filter "Name = 'weed.exe'" |
Where-Object { $_.CommandLine -like '*mount*' } |
ForEach-Object { Stop-Process -Id $_.ProcessId -Force }
Start-Sleep -Seconds 5
Write-Host "::endgroup::"
Start-Mount 'seaweed-remount2'
Write-Host "::group::explorer-style walk"
New-Item -ItemType Directory -Force -Path S:\walk | Out-Null
1..200 | ForEach-Object { Set-Content -Path "S:\walk\f$_.txt" -Value "line $_" }
$names = @(Get-ChildItem S:\walk | ForEach-Object { $_.Name })
if ($names.Count -ne 200) {
# Name the strays: a dot entry surfacing here is a different problem
# from a missing or duplicated file.
$unexpected = $names | Where-Object { $_ -notmatch '^f\d+\.txt$' }
throw "listed $($names.Count) entries, expected 200; unexpected: $($unexpected -join ', ')"
}
$body = Get-Content S:\walk\f42.txt
if ($body -ne 'line 42') { throw "unexpected content: $body" }
Copy-Item S:\walk\f42.txt S:\walk\copy.txt
Remove-Item S:\walk -Recurse -Force
if (Test-Path S:\walk) { throw "directory survived recursive delete" }
Write-Host "::endgroup::"
- name: Logs
if: always()
shell: pwsh
run: |
foreach ($f in 'C:\seaweed-mount.log','C:\seaweed-mount.err.log','C:\seaweed-remount.log','C:\seaweed-remount.err.log','C:\seaweed-remount2.log','C:\seaweed-remount2.err.log','C:\seaweed-dirmount.log','C:\seaweed-dirmount.err.log','C:\seaweed-mini.log','C:\seaweed-mini.err.log') {
if (Test-Path $f) { Write-Host "===== $f"; Get-Content $f -Tail 200 }
}
+2 -2
View File
@@ -34,10 +34,10 @@ jobs:
timeout-minutes: 10
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
+137
View File
@@ -0,0 +1,137 @@
name: "NFS Integration Tests"
on:
push:
branches: [ master, main ]
paths:
- 'weed/server/nfs/**'
- 'weed/command/nfs.go'
- 'weed/filer/filer_inode.go'
- 'weed/filer/filer_inode_index.go'
- 'weed/filer/filerstore_wrapper.go'
- 'weed/server/filer_grpc_server_rename.go'
- 'test/nfs/**'
- '.github/workflows/nfs-tests.yml'
pull_request:
branches: [ master, main ]
paths:
- 'weed/server/nfs/**'
- 'weed/command/nfs.go'
- 'weed/filer/filer_inode.go'
- 'weed/filer/filer_inode_index.go'
- 'weed/filer/filerstore_wrapper.go'
- 'weed/server/filer_grpc_server_rename.go'
- 'test/nfs/**'
- '.github/workflows/nfs-tests.yml'
concurrency:
group: ${{ github.head_ref }}/nfs-tests
cancel-in-progress: true
permissions:
contents: read
env:
TEST_TIMEOUT: '15m'
jobs:
nfs-integration:
name: NFS Integration Testing
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Build SeaweedFS
run: |
cd weed
go build -o weed .
chmod +x weed
./weed version
- name: Run NFS Integration Tests
run: |
cd test/nfs
echo "Running NFS integration tests..."
echo "============================================"
# Install test dependencies
go mod download
# Run the protocol-layer tests. The kernel-mount tests require root
# for mount(2) and are exercised in their own privileged step below;
# skip them here so a "skipped because not root" line doesn't show
# up as noise on every CI run.
go test -v -timeout=${{ env.TEST_TIMEOUT }} -skip '^TestKernelMount' ./...
echo "============================================"
echo "NFS integration tests completed"
- name: Install kernel NFS client
run: |
# nfs-common provides mount.nfs; netbase provides /etc/protocols
# which mount.nfs's protocol-name lookups (`tcp`, `udp`) need.
sudo apt-get update
sudo apt-get install -y nfs-common netbase
- name: Run kernel-mount E2E tests
run: |
cd test/nfs
echo "Running kernel-mount end-to-end tests..."
echo "These mount the running 'weed nfs' subprocess via the actual"
echo "Linux NFS client to catch protocol regressions invisible to"
echo "the go-nfs-client-based tests above."
echo "============================================"
# mount(2) is privileged. Preserve PATH so 'go' (and the weed
# binary that test/nfs/framework.go locates via $PATH) resolve
# correctly under sudo, and pass through the Go module/cache dirs
# so we don't redownload modules under root.
sudo env "PATH=$PATH" \
GOMODCACHE="$(go env GOMODCACHE)" \
GOCACHE="$(go env GOCACHE)" \
go test -v -timeout=${{ env.TEST_TIMEOUT }} -run '^TestKernelMount' ./...
echo "============================================"
echo "Kernel-mount E2E tests completed"
- name: Test Summary
if: always()
run: |
echo "## NFS Integration Test Summary" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "### Test Coverage" >> $GITHUB_STEP_SUMMARY
echo "- **Read/Write Round Trip**: Basic file create + read" >> $GITHUB_STEP_SUMMARY
echo "- **Directory Operations**: Mkdir, ReadDirPlus, RmDir" >> $GITHUB_STEP_SUMMARY
echo "- **Nested Directories**: Deep tree creation and leaf I/O" >> $GITHUB_STEP_SUMMARY
echo "- **Rename**: Content preserved across rename" >> $GITHUB_STEP_SUMMARY
echo "- **Overwrite + Truncate**: Setattr(size=0) + shorter write" >> $GITHUB_STEP_SUMMARY
echo "- **Large Files**: 3 MiB binary round trip" >> $GITHUB_STEP_SUMMARY
echo "- **Edge Payloads**: All 256 byte values + empty files" >> $GITHUB_STEP_SUMMARY
echo "- **Symlinks**: Symlink + Lookup" >> $GITHUB_STEP_SUMMARY
echo "- **Missing Path**: Remove on missing entry errors cleanly" >> $GITHUB_STEP_SUMMARY
echo "- **FSINFO**: Non-zero rtpref/wtpref advertised" >> $GITHUB_STEP_SUMMARY
echo "- **Sequential Append**: Two-part concatenation" >> $GITHUB_STEP_SUMMARY
echo "- **ReadDir After Remove**: Meta cache does not serve stale entries" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "### Kernel-Mount E2E Coverage" >> $GITHUB_STEP_SUMMARY
echo "- **V3 over TCP**: baseline NFSv3 mount + readdir" >> $GITHUB_STEP_SUMMARY
echo "- **V3 with mountproto=udp**: regression test for UDP MOUNT v3 responder" >> $GITHUB_STEP_SUMMARY
echo "- **V4 rejects cleanly**: regression test for the v4 PROG_MISMATCH path (#9262)" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "### Harness" >> $GITHUB_STEP_SUMMARY
echo "Most tests boot their own master + volume + filer + nfs subprocess" >> $GITHUB_STEP_SUMMARY
echo "stack on loopback and drive it via the NFSv3 RPC protocol using" >> $GITHUB_STEP_SUMMARY
echo "go-nfs-client. The kernel-mount E2E tests reuse the same harness" >> $GITHUB_STEP_SUMMARY
echo "but mount the export through the in-tree Linux NFS client to" >> $GITHUB_STEP_SUMMARY
echo "catch protocol regressions a Go-only client can't see; they run" >> $GITHUB_STEP_SUMMARY
echo "in a separate privileged step (mount(2) requires root)." >> $GITHUB_STEP_SUMMARY
-477
View File
@@ -1,477 +0,0 @@
name: "Performance"
on:
push:
branches: [ master ]
paths:
- '**/*.go'
- 'go.mod'
- 'go.sum'
- 'seaweed-volume/**'
- 'test/perf/**'
- '.github/workflows/performance.yml'
workflow_dispatch:
inputs:
profile_duration:
description: "CPU profiling duration in seconds"
required: false
default: "30"
type: string
benchmark_files:
description: "Number of files for the throughput benchmark"
required: false
default: "100000"
type: string
benchmark_concurrency:
description: "Concurrent read/write workers"
required: false
default: "16"
type: string
benchmark_size:
description: "Simulated file size in bytes"
required: false
default: "1024"
type: string
s3_objects:
description: "Number of objects for the S3 benchmark"
required: false
default: "20000"
type: string
s3_size:
description: "S3 object size in bytes"
required: false
default: "4096"
type: string
concurrency:
group: ${{ github.head_ref || github.ref }}/performance
cancel-in-progress: true
permissions:
contents: read
env:
VOL_SIZE_LIMIT: "1024"
jobs:
performance-profile:
name: CPU and Heap Profile
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
- name: Build weed
run: go build -o weed_bin ./weed
- name: Start server with profiling enabled
run: |
mkdir -p ./perfdata
./weed_bin -v=1 server -debug -debug.port=6060 -dir=./perfdata \
-s3 -filer -volume.max=0 -master.volumeSizeLimitMB=100 \
-s3.port=8000 -s3.config=./docker/compose/s3.json \
> weed.log 2>&1 &
echo "WEED_PID=$!" >> "$GITHUB_ENV"
for i in $(seq 1 60); do
if curl -sf http://localhost:9333/dir/status >/dev/null 2>&1; then
echo "master is ready"
break
fi
sleep 1
done
# give the volume server a moment to register with the master
sleep 3
- name: Start memory sampler
run: |
bash test/perf/mem_sample.sh mem-profile.csv "server=${WEED_PID}" &
echo "SAMPLER_PID=$!" >> "$GITHUB_ENV"
- name: Capture profiles under load
run: |
DURATION="${{ github.event.inputs.profile_duration || '30' }}"
# drive write load so the sampled profile reflects real work
./weed_bin benchmark -master=localhost:9333 -writeOnly \
-c=16 -n=5000000 -size=1024 > benchmark-load.log 2>&1 &
echo "Sampling CPU profile for ${DURATION}s..."
curl -s "http://localhost:6060/debug/pprof/profile?seconds=${DURATION}" -o cpu.pprof
curl -s "http://localhost:6060/debug/pprof/heap" -o heap.pprof
curl -s "http://localhost:6060/debug/pprof/goroutine?debug=1" -o goroutine.txt
go tool pprof -top -nodecount=50 cpu.pprof > cpu-top.txt 2>/dev/null || true
go tool pprof -top -nodecount=50 -sample_index=inuse_space heap.pprof > heap-top.txt 2>/dev/null || true
- name: Record memory usage
if: always()
run: |
kill -TERM "${SAMPLER_PID}" 2>/dev/null || true
sleep 2
{
echo "## Memory usage (peak RSS)"
echo '```'
if [ -f mem-profile.csv.peak ]; then
awk -F'\t' '{printf "%-10s %8d KB (%.1f MB)\n", $1, $2, $2/1024}' mem-profile.csv.peak
else
echo "no memory samples captured"
fi
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Profile summary
if: always()
run: |
{
echo "## CPU profile (top functions)"
echo '```'
head -45 cpu-top.txt 2>/dev/null || echo "no cpu profile captured"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Stop server
if: always()
run: kill "${WEED_PID}" 2>/dev/null || true
- name: Show server log on failure
if: failure()
run: tail -200 weed.log || true
- name: Upload profiles
if: always()
uses: actions/upload-artifact@v7
with:
name: performance-profile-${{ github.run_number }}
path: |
cpu.pprof
heap.pprof
cpu-top.txt
heap-top.txt
goroutine.txt
mem-profile.csv
mem-profile.csv.peak
weed.log
retention-days: 30
benchmark:
name: Throughput Benchmark (${{ matrix.impl }} volume)
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
impl: [go, rust]
env:
IMPL: ${{ matrix.impl }}
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
- name: Install protobuf compiler
if: matrix.impl == 'rust'
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
- name: Install Rust toolchain
if: matrix.impl == 'rust'
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: matrix.impl == 'rust'
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ hashFiles('seaweed-volume/Cargo.lock') }}
restore-keys: |
rust-
- name: Build weed
run: go build -o weed_bin ./weed
- name: Build Rust volume server
if: matrix.impl == 'rust'
run: cd seaweed-volume && cargo build --release
- name: Start master and volume server
run: |
mkdir -p ./perfdata/master ./perfdata/vol
./weed_bin -v=1 master -ip=127.0.0.1 -port=9333 \
-mdir=./perfdata/master -peers=none \
-volumeSizeLimitMB="${VOL_SIZE_LIMIT}" -defaultReplication=000 \
> master.log 2>&1 &
echo "MASTER_PID=$!" >> "$GITHUB_ENV"
for i in $(seq 1 60); do
if curl -sf http://localhost:9333/dir/status >/dev/null 2>&1; then
echo "master is ready"
break
fi
sleep 1
done
if [ "${IMPL}" = "rust" ]; then
./seaweed-volume/target/release/weed-volume \
--master 127.0.0.1:9333 --ip 127.0.0.1 --ip.bind 127.0.0.1 \
--port 8080 --dir ./perfdata/vol --max 100 --preStopSeconds 0 \
> volume.log 2>&1 &
else
./weed_bin -v=1 volume -master=127.0.0.1:9333 -ip=127.0.0.1 \
-port=8080 -dir=./perfdata/vol -max=100 \
> volume.log 2>&1 &
fi
echo "VOLUME_PID=$!" >> "$GITHUB_ENV"
for i in $(seq 1 60); do
if curl -sf http://localhost:8080/status >/dev/null 2>&1; then
echo "volume server is ready"
break
fi
sleep 1
done
# let the volume server register with the master via heartbeat
sleep 3
- name: Start memory sampler
run: |
bash test/perf/mem_sample.sh mem-benchmark.csv \
"master=${MASTER_PID}" "volume=${VOLUME_PID}" &
echo "SAMPLER_PID=$!" >> "$GITHUB_ENV"
- name: Run throughput benchmark
run: |
N="${{ github.event.inputs.benchmark_files || '100000' }}"
C="${{ github.event.inputs.benchmark_concurrency || '16' }}"
SIZE="${{ github.event.inputs.benchmark_size || '1024' }}"
./weed_bin benchmark -master=localhost:9333 \
-c="${C}" -n="${N}" -size="${SIZE}" 2>&1 | tee benchmark-results.txt
- name: Run Go micro-benchmarks
if: matrix.impl == 'go'
continue-on-error: true
run: |
go test -run='^$' -bench=. -benchmem -benchtime=10x \
./weed/topology/... ./weed/util/log_buffer/... ./weed/util/buffered_queue/... \
2>&1 | tee go-benchmarks.txt
- name: Record memory usage
if: always()
run: |
kill -TERM "${SAMPLER_PID}" 2>/dev/null || true
sleep 2
{
echo "## Memory usage (peak RSS, ${IMPL} volume)"
echo '```'
if [ -f mem-benchmark.csv.peak ]; then
awk -F'\t' '{printf "%-10s %8d KB (%.1f MB)\n", $1, $2, $2/1024}' mem-benchmark.csv.peak
else
echo "no memory samples captured"
fi
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Benchmark summary
if: always()
run: |
{
echo "## Throughput benchmark (${IMPL} volume)"
echo '```'
grep -E "Concurrency Level|Time taken|Completed requests|Failed requests|Requests per second|Transfer rate" \
benchmark-results.txt 2>/dev/null || echo "no benchmark results captured"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Stop processes
if: always()
run: |
kill "${VOLUME_PID}" "${MASTER_PID}" 2>/dev/null || true
- name: Show logs on failure
if: failure()
run: |
echo "=== master.log ==="; tail -100 master.log 2>/dev/null || true
echo "=== volume.log ==="; tail -200 volume.log 2>/dev/null || true
- name: Upload benchmark results
if: always()
uses: actions/upload-artifact@v7
with:
name: benchmark-results-${{ matrix.impl }}-${{ github.run_number }}
path: |
benchmark-results.txt
go-benchmarks.txt
mem-benchmark.csv
mem-benchmark.csv.peak
master.log
volume.log
retention-days: 7
s3-benchmark:
name: S3 Read/Write Benchmark (${{ matrix.impl }} volume)
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
impl: [go, rust]
env:
IMPL: ${{ matrix.impl }}
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
- name: Install protobuf compiler
if: matrix.impl == 'rust'
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
- name: Install Rust toolchain
if: matrix.impl == 'rust'
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: matrix.impl == 'rust'
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ hashFiles('seaweed-volume/Cargo.lock') }}
restore-keys: |
rust-
- name: Build weed and S3 load tool
run: |
go build -o weed_bin ./weed
go build -o s3bench ./test/s3/benchmark
- name: Build Rust volume server
if: matrix.impl == 'rust'
run: cd seaweed-volume && cargo build --release
- name: Start cluster with S3 gateway
run: |
mkdir -p ./perfdata/master ./perfdata/vol ./perfdata/filer
./weed_bin -v=1 master -ip=127.0.0.1 -port=9333 \
-mdir=./perfdata/master -peers=none \
-volumeSizeLimitMB="${VOL_SIZE_LIMIT}" -defaultReplication=000 \
> master.log 2>&1 &
echo "MASTER_PID=$!" >> "$GITHUB_ENV"
for i in $(seq 1 60); do
if curl -sf http://localhost:9333/dir/status >/dev/null 2>&1; then
echo "master is ready"
break
fi
sleep 1
done
if [ "${IMPL}" = "rust" ]; then
./seaweed-volume/target/release/weed-volume \
--master 127.0.0.1:9333 --ip 127.0.0.1 --ip.bind 127.0.0.1 \
--port 8080 --dir ./perfdata/vol --max 100 --preStopSeconds 0 \
> volume.log 2>&1 &
else
./weed_bin -v=1 volume -master=127.0.0.1:9333 -ip=127.0.0.1 \
-port=8080 -dir=./perfdata/vol -max=100 \
> volume.log 2>&1 &
fi
echo "VOLUME_PID=$!" >> "$GITHUB_ENV"
for i in $(seq 1 60); do
if curl -sf http://localhost:8080/status >/dev/null 2>&1; then
echo "volume server is ready"
break
fi
sleep 1
done
sleep 3
./weed_bin -v=1 filer -master=127.0.0.1:9333 -ip=127.0.0.1 -port=8888 \
-s3 -s3.port=8000 -s3.config=./docker/compose/s3.json \
> filer.log 2>&1 &
echo "FILER_PID=$!" >> "$GITHUB_ENV"
for i in $(seq 1 30); do
if nc -z localhost 8000 2>/dev/null; then
echo "s3 gateway is ready"
break
fi
sleep 1
done
sleep 2
- name: Start memory sampler
run: |
bash test/perf/mem_sample.sh mem-s3.csv \
"master=${MASTER_PID}" "volume=${VOLUME_PID}" "filer=${FILER_PID}" &
echo "SAMPLER_PID=$!" >> "$GITHUB_ENV"
- name: Run S3 read/write benchmark
run: |
OBJECTS="${{ github.event.inputs.s3_objects || '20000' }}"
C="${{ github.event.inputs.benchmark_concurrency || '16' }}"
SIZE="${{ github.event.inputs.s3_size || '4096' }}"
./s3bench -endpoint=http://localhost:8000 \
-access-key=some_access_key1 -secret-key=some_secret_key1 \
-objects="${OBJECTS}" -size="${SIZE}" -concurrency="${C}" -mode=both \
2>&1 | tee s3-benchmark-results.txt
- name: Record memory usage
if: always()
run: |
kill -TERM "${SAMPLER_PID}" 2>/dev/null || true
sleep 2
{
echo "## Memory usage (peak RSS, ${IMPL} volume)"
echo '```'
if [ -f mem-s3.csv.peak ]; then
awk -F'\t' '{printf "%-10s %8d KB (%.1f MB)\n", $1, $2, $2/1024}' mem-s3.csv.peak
else
echo "no memory samples captured"
fi
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: S3 benchmark summary
if: always()
run: |
{
echo "## S3 read/write benchmark (${IMPL} volume)"
echo '```'
grep -E "results:|Concurrency Level|Time taken|Completed requests|Failed requests|Requests per second|Transfer rate|Latency" \
s3-benchmark-results.txt 2>/dev/null || echo "no S3 benchmark results captured"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Stop processes
if: always()
run: |
kill "${FILER_PID}" "${VOLUME_PID}" "${MASTER_PID}" 2>/dev/null || true
- name: Show logs on failure
if: failure()
run: |
echo "=== master.log ==="; tail -100 master.log 2>/dev/null || true
echo "=== volume.log ==="; tail -200 volume.log 2>/dev/null || true
echo "=== filer.log ==="; tail -200 filer.log 2>/dev/null || true
- name: Upload S3 benchmark results
if: always()
uses: actions/upload-artifact@v7
with:
name: s3-benchmark-results-${{ matrix.impl }}-${{ github.run_number }}
path: |
s3-benchmark-results.txt
mem-s3.csv
mem-s3.csv.peak
master.log
volume.log
filer.log
retention-days: 7
+2 -2
View File
@@ -32,10 +32,10 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
+2 -14
View File
@@ -3,20 +3,8 @@ name: "Plugin Worker Integration Tests"
on:
push:
branches: [ master ]
paths:
- 'weed/**'
- 'test/plugin_workers/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/plugin-workers.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/**'
- 'test/plugin_workers/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/plugin-workers.yml'
permissions:
contents: read
@@ -39,13 +27,13 @@ jobs:
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.26
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Run plugin worker tests
run: go test -v ./${{ matrix.path }}
+3 -24
View File
@@ -3,24 +3,8 @@ name: "PostgreSQL Gateway Tests"
on:
push:
branches: [ master ]
paths:
- 'weed/server/postgres/**'
- 'weed/query/**'
- 'weed/mq/**'
- 'test/postgres/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/postgres-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/server/postgres/**'
- 'weed/query/**'
- 'weed/mq/**'
- 'test/postgres/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/postgres-tests.yml'
concurrency:
group: ${{ github.head_ref }}/postgres-tests
@@ -39,24 +23,19 @@ jobs:
working-directory: test/postgres
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
id: go
- name: Check out code
uses: actions/checkout@v7
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
uses: actions/checkout@v6
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Cache Docker layers
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-postgres-${{ github.sha }}
-115
View File
@@ -1,115 +0,0 @@
name: "release: bump version"
on:
workflow_dispatch:
inputs:
bump:
description: "Which part to increment (ignored when 'version' is set)"
type: choice
options:
- minor
- major
default: minor
version:
description: "Explicit MAJOR.MINOR to set, e.g. 4.36 (overrides 'bump')"
type: string
required: false
permissions:
contents: write
jobs:
bump-version:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- name: Compute new version
id: compute
env:
BUMP: ${{ inputs.bump }}
INPUT_VERSION: ${{ inputs.version }}
run: |
set -euo pipefail
CONST=weed/util/version/constants.go
MAJOR=$(grep -oP 'MAJOR_VERSION\s*=\s*int32\(\K[0-9]+' "$CONST")
MINOR=$(grep -oP 'MINOR_VERSION\s*=\s*int32\(\K[0-9]+' "$CONST")
echo "current: ${MAJOR}.${MINOR}"
if [ -n "$INPUT_VERSION" ]; then
if ! [[ "$INPUT_VERSION" =~ ^[0-9]+\.[0-9]+$ ]]; then
echo "::error::version must be MAJOR.MINOR, e.g. 5.01 (got '$INPUT_VERSION')"
exit 1
fi
# 10# forces base 10 so 08/09 are not parsed as octal.
MAJOR=$((10#${INPUT_VERSION%%.*}))
MINOR=$((10#${INPUT_VERSION##*.}))
if [ "$MINOR" -gt 99 ]; then
echo "::error::minor must be 0-99 (got $MINOR); it rolls into the next major at 99"
exit 1
fi
else
case "$BUMP" in
# Minor is a 2-digit field: 4.99 -> 5.00 -> 5.01.
major) MAJOR=$((MAJOR + 1)); MINOR=0 ;;
minor)
if [ "$MINOR" -ge 99 ]; then
MAJOR=$((MAJOR + 1)); MINOR=0
else
MINOR=$((MINOR + 1))
fi
;;
*) echo "::error::unknown bump '$BUMP'"; exit 1 ;;
esac
fi
# appVersion mirrors the Go VERSION_NUMBER (zero-padded minor);
# chart version is plain SemVer (no leading zeros).
APP_VERSION=$(printf '%d.%02d' "$MAJOR" "$MINOR")
CHART_VERSION="${MAJOR}.${MINOR}.0"
echo "new: app=${APP_VERSION} chart=${CHART_VERSION}"
{
echo "major=${MAJOR}"
echo "minor=${MINOR}"
echo "app_version=${APP_VERSION}"
echo "chart_version=${CHART_VERSION}"
} >> "$GITHUB_OUTPUT"
- name: Apply version to constants.go
env:
MAJOR: ${{ steps.compute.outputs.major }}
MINOR: ${{ steps.compute.outputs.minor }}
run: |
set -euo pipefail
CONST=weed/util/version/constants.go
sed -i -E "s/(MAJOR_VERSION[[:space:]]*=[[:space:]]*int32\()[0-9]+(\))/\1${MAJOR}\2/" "$CONST"
sed -i -E "s/(MINOR_VERSION[[:space:]]*=[[:space:]]*int32\()[0-9]+(\))/\1${MINOR}\2/" "$CONST"
grep -E 'MAJOR_VERSION|MINOR_VERSION' "$CONST"
- name: Apply version to Chart.yaml
env:
APP_VERSION: ${{ steps.compute.outputs.app_version }}
CHART_VERSION: ${{ steps.compute.outputs.chart_version }}
run: |
set -euo pipefail
CHART=k8s/charts/seaweedfs/Chart.yaml
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" "$CHART"
sed -i -E "s/^version:.*/version: ${CHART_VERSION}/" "$CHART"
cat "$CHART"
- name: Commit and push
env:
APP_VERSION: ${{ steps.compute.outputs.app_version }}
run: |
set -euo pipefail
if git diff --quiet; then
echo "No version change to commit."
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add weed/util/version/constants.go k8s/charts/seaweedfs/Chart.yaml
git commit -m "${APP_VERSION}"
git push
+50 -52
View File
@@ -34,7 +34,7 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Install protobuf compiler
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
@@ -42,22 +42,16 @@ jobs:
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
# cargo tracks its own inputs but not the runner's C toolchain, so a cached
# target/ can carry C objects built against a different glibc than we link against.
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock') }}
key: rust-${{ hashFiles('seaweed-volume/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
rust-
- name: Build Rust volume server
run: cd seaweed-volume && cargo build --release
@@ -72,10 +66,10 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
@@ -85,22 +79,16 @@ jobs:
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
# cargo tracks its own inputs but not the runner's C toolchain, so a cached
# target/ can carry C objects built against a different glibc than we link against.
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock') }}
key: rust-${{ hashFiles('seaweed-volume/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
rust-
- name: Build Go weed binary
run: |
@@ -146,9 +134,6 @@ jobs:
name: Go Tests with Rust Volume (${{ matrix.test-type }} - Shard ${{ matrix.shard }})
runs-on: ubuntu-22.04
timeout-minutes: 45
env:
# Keep in step with the length of matrix.shard below.
SHARD_COUNT: 3
strategy:
fail-fast: false
matrix:
@@ -157,10 +142,10 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
@@ -170,22 +155,16 @@ jobs:
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
# cargo tracks its own inputs but not the runner's C toolchain, so a cached
# target/ can carry C objects built against a different glibc than we link against.
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock') }}
key: rust-${{ hashFiles('seaweed-volume/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
rust-
- name: Build Go weed binary
run: |
@@ -197,28 +176,30 @@ jobs:
- name: Build Rust volume binary
run: cd seaweed-volume && cargo build --release
# Dealing the listed tests out one by one keeps the shards even. Bucketing
# them by first letter did not: names cluster, so ^Test[I-S] drew 50 of
# the 114 grpc tests and ran nearly twice as long as the other two shards.
- name: Select this shard's tests
env:
TEST_TYPE: ${{ matrix.test-type }}
SHARD: ${{ matrix.shard }}
run: |
tests=$(go test -tags 5BytesOffset ./test/volume_server/"$TEST_TYPE"/... -list '.*' | grep '^Test' | sort -u)
# An empty list would make -run match nothing and the shard pass vacuously.
[ -n "$tests" ] || { echo "listed no tests in test/volume_server/$TEST_TYPE"; exit 1; }
selected=$(echo "$tests" | awk -v n="$SHARD_COUNT" -v i="$SHARD" 'NR % n == i - 1')
echo "shard $SHARD of $SHARD_COUNT runs $(echo "$selected" | wc -l) of $(echo "$tests" | wc -l) tests"
echo "TEST_PATTERN=^($(echo "$selected" | paste -sd'|' -))\$" >> "$GITHUB_ENV"
- name: Run volume server integration tests with Rust volume
env:
WEED_BINARY: ${{ github.workspace }}/weed/weed
RUST_VOLUME_BINARY: ${{ github.workspace }}/seaweed-volume/target/release/weed-volume
VOLUME_SERVER_IMPL: rust
run: |
echo "Running Go volume server tests with Rust volume for ${{ matrix.test-type }} (Shard ${{ matrix.shard }} of ${SHARD_COUNT})..."
if [ "${{ matrix.test-type }}" == "grpc" ]; then
if [ "${{ matrix.shard }}" == "1" ]; then
TEST_PATTERN="^Test[A-H]"
elif [ "${{ matrix.shard }}" == "2" ]; then
TEST_PATTERN="^Test[I-S]"
else
TEST_PATTERN="^Test[T-Z]"
fi
else
if [ "${{ matrix.shard }}" == "1" ]; then
TEST_PATTERN="^Test[A-G]"
elif [ "${{ matrix.shard }}" == "2" ]; then
TEST_PATTERN="^Test[H-R]"
else
TEST_PATTERN="^Test[S-Z]"
fi
fi
echo "Running Go volume server tests with Rust volume for ${{ matrix.test-type }} (Shard ${{ matrix.shard }}, pattern: ${TEST_PATTERN})..."
go test -v -count=1 -tags 5BytesOffset -timeout=30m ./test/volume_server/${{ matrix.test-type }}/... -run "${TEST_PATTERN}"
- name: Collect logs on failure
@@ -239,6 +220,23 @@ jobs:
- name: Test summary
if: always()
run: |
if [ "${{ matrix.test-type }}" == "grpc" ]; then
if [ "${{ matrix.shard }}" == "1" ]; then
TEST_PATTERN="^Test[A-H]"
elif [ "${{ matrix.shard }}" == "2" ]; then
TEST_PATTERN="^Test[I-S]"
else
TEST_PATTERN="^Test[T-Z]"
fi
else
if [ "${{ matrix.shard }}" == "1" ]; then
TEST_PATTERN="^Test[A-G]"
elif [ "${{ matrix.shard }}" == "2" ]; then
TEST_PATTERN="^Test[H-R]"
else
TEST_PATTERN="^Test[S-Z]"
fi
fi
echo "## Rust Volume - Go Test Summary (${{ matrix.test-type }} - Shard ${{ matrix.shard }})" >> "$GITHUB_STEP_SUMMARY"
echo "- Suite: test/volume_server/${{ matrix.test-type }} (shard ${{ matrix.shard }} of ${SHARD_COUNT}, see 'Select this shard's tests' for the split)" >> "$GITHUB_STEP_SUMMARY"
echo "- Suite: test/volume_server/${{ matrix.test-type }} (Pattern: ${TEST_PATTERN})" >> "$GITHUB_STEP_SUMMARY"
echo "- Volume server: Rust (VOLUME_SERVER_IMPL=rust)" >> "$GITHUB_STEP_SUMMARY"
+4 -4
View File
@@ -39,7 +39,7 @@ jobs:
asset_suffix: linux-amd64
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v6
- name: Install protobuf compiler
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
@@ -48,7 +48,7 @@ jobs:
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -108,7 +108,7 @@ jobs:
asset_suffix: darwin-amd64
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v6
- name: Install protobuf compiler
run: brew install protobuf
@@ -119,7 +119,7 @@ jobs:
targets: ${{ matrix.target }}
- name: Cache cargo registry and target
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
+18 -49
View File
@@ -26,7 +26,7 @@ jobs:
cross: true
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v6
- name: Install protobuf compiler
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
@@ -51,7 +51,7 @@ jobs:
echo "OPENSSL_LIB_DIR=/usr/lib/aarch64-linux-gnu" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -66,42 +66,34 @@ jobs:
SEAWEEDFS_COMMIT: ${{ github.sha }}
run: |
cd seaweed-volume
cargo build --release --target ${{ matrix.target }} --target-dir target/large-disk
cargo build --release --target ${{ matrix.target }}
- name: Build Rust volume server (normal)
env:
SEAWEEDFS_COMMIT: ${{ github.sha }}
run: |
cd seaweed-volume
cargo build --release --target ${{ matrix.target }} --no-default-features --target-dir target/normal
cargo build --release --target ${{ matrix.target }} --no-default-features
- name: Package binaries
run: |
# Large disk (default, 5bytes feature)
cp seaweed-volume/target/large-disk/${{ matrix.target }}/release/weed-volume weed-volume-large-disk
cp seaweed-volume/target/${{ matrix.target }}/release/weed-volume weed-volume-large-disk
tar czf weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz weed-volume-large-disk
rm weed-volume-large-disk
# Normal volume size
cp seaweed-volume/target/normal/${{ matrix.target }}/release/weed-volume weed-volume-normal
cp seaweed-volume/target/${{ matrix.target }}/release/weed-volume weed-volume-normal
tar czf weed-volume_${{ matrix.asset_suffix }}.tar.gz weed-volume-normal
rm weed-volume-normal
- name: Generate md5 checksums
run: |
for f in weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz weed-volume_${{ matrix.asset_suffix }}.tar.gz; do
md5sum "$f" > "$f.md5"
done
- name: Upload release assets
if: startsWith(github.ref, 'refs/tags/')
uses: softprops/action-gh-release@v3
with:
files: |
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz.md5
weed-volume_${{ matrix.asset_suffix }}.tar.gz
weed-volume_${{ matrix.asset_suffix }}.tar.gz.md5
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -112,9 +104,7 @@ jobs:
name: rust-volume-${{ matrix.asset_suffix }}
path: |
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz.md5
weed-volume_${{ matrix.asset_suffix }}.tar.gz
weed-volume_${{ matrix.asset_suffix }}.tar.gz.md5
build-rust-volume-darwin:
permissions:
@@ -129,7 +119,7 @@ jobs:
asset_suffix: darwin_arm64
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v6
- name: Install protobuf compiler
run: brew install protobuf
@@ -140,7 +130,7 @@ jobs:
targets: ${{ matrix.target }}
- name: Cache cargo registry and target
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -155,40 +145,32 @@ jobs:
SEAWEEDFS_COMMIT: ${{ github.sha }}
run: |
cd seaweed-volume
cargo build --release --target ${{ matrix.target }} --target-dir target/large-disk
cargo build --release --target ${{ matrix.target }}
- name: Build Rust volume server (normal)
env:
SEAWEEDFS_COMMIT: ${{ github.sha }}
run: |
cd seaweed-volume
cargo build --release --target ${{ matrix.target }} --no-default-features --target-dir target/normal
cargo build --release --target ${{ matrix.target }} --no-default-features
- name: Package binaries
run: |
cp seaweed-volume/target/large-disk/${{ matrix.target }}/release/weed-volume weed-volume-large-disk
cp seaweed-volume/target/${{ matrix.target }}/release/weed-volume weed-volume-large-disk
tar czf weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz weed-volume-large-disk
rm weed-volume-large-disk
cp seaweed-volume/target/normal/${{ matrix.target }}/release/weed-volume weed-volume-normal
cp seaweed-volume/target/${{ matrix.target }}/release/weed-volume weed-volume-normal
tar czf weed-volume_${{ matrix.asset_suffix }}.tar.gz weed-volume-normal
rm weed-volume-normal
- name: Generate md5 checksums
run: |
for f in weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz weed-volume_${{ matrix.asset_suffix }}.tar.gz; do
md5 -r "$f" > "$f.md5"
done
- name: Upload release assets
if: startsWith(github.ref, 'refs/tags/')
uses: softprops/action-gh-release@v3
with:
files: |
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz.md5
weed-volume_${{ matrix.asset_suffix }}.tar.gz
weed-volume_${{ matrix.asset_suffix }}.tar.gz.md5
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -199,9 +181,7 @@ jobs:
name: rust-volume-${{ matrix.asset_suffix }}
path: |
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz.md5
weed-volume_${{ matrix.asset_suffix }}.tar.gz
weed-volume_${{ matrix.asset_suffix }}.tar.gz.md5
build-rust-volume-windows:
permissions:
@@ -209,7 +189,7 @@ jobs:
runs-on: windows-latest
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v6
- name: Install protobuf compiler
run: choco install protoc -y
@@ -218,7 +198,7 @@ jobs:
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -233,42 +213,33 @@ jobs:
SEAWEEDFS_COMMIT: ${{ github.sha }}
run: |
cd seaweed-volume
cargo build --release --target-dir target/large-disk
cargo build --release
- name: Build Rust volume server (normal)
env:
SEAWEEDFS_COMMIT: ${{ github.sha }}
run: |
cd seaweed-volume
cargo build --release --no-default-features --target-dir target/normal
cargo build --release --no-default-features
- name: Package binaries
shell: bash
run: |
cp seaweed-volume/target/large-disk/release/weed-volume.exe weed-volume-large-disk.exe
cp seaweed-volume/target/release/weed-volume.exe weed-volume-large-disk.exe
7z a weed-volume_large_disk_windows_amd64.zip weed-volume-large-disk.exe
rm weed-volume-large-disk.exe
cp seaweed-volume/target/normal/release/weed-volume.exe weed-volume-normal.exe
cp seaweed-volume/target/release/weed-volume.exe weed-volume-normal.exe
7z a weed-volume_windows_amd64.zip weed-volume-normal.exe
rm weed-volume-normal.exe
- name: Generate md5 checksums
shell: bash
run: |
for f in weed-volume_large_disk_windows_amd64.zip weed-volume_windows_amd64.zip; do
md5sum "$f" > "$f.md5"
done
- name: Upload release assets
if: startsWith(github.ref, 'refs/tags/')
uses: softprops/action-gh-release@v3
with:
files: |
weed-volume_large_disk_windows_amd64.zip
weed-volume_large_disk_windows_amd64.zip.md5
weed-volume_windows_amd64.zip
weed-volume_windows_amd64.zip.md5
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -279,6 +250,4 @@ jobs:
name: rust-volume-windows_amd64
path: |
weed-volume_large_disk_windows_amd64.zip
weed-volume_large_disk_windows_amd64.zip.md5
weed-volume_windows_amd64.zip
weed-volume_windows_amd64.zip.md5
+2 -2
View File
@@ -34,10 +34,10 @@ jobs:
timeout-minutes: 15
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
@@ -2,16 +2,7 @@ name: "S3 Authenticated Integration Tests"
on:
pull_request:
paths:
- 'weed/s3api/**'
- 'weed/iam/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'test/s3/normal/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/s3-example-integration-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-integration-tests
cancel-in-progress: true
@@ -27,10 +18,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
+3 -11
View File
@@ -2,15 +2,7 @@ name: "S3 Filer Group Tests"
on:
pull_request:
paths:
- 'weed/s3api/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'test/s3/filer_group/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/s3-filer-group-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-filer-group-tests
cancel-in-progress: true
@@ -30,10 +22,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
+23 -31
View File
@@ -2,15 +2,7 @@ name: "S3 Go Tests"
on:
pull_request:
paths:
- 'weed/s3api/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'test/s3/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/s3-go-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-go-tests
cancel-in-progress: true
@@ -33,10 +25,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -97,10 +89,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -145,10 +137,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -196,10 +188,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -286,10 +278,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -331,10 +323,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -389,10 +381,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -453,10 +445,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -502,10 +494,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -570,10 +562,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -625,10 +617,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
+8 -8
View File
@@ -39,10 +39,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -88,10 +88,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -202,10 +202,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -262,10 +262,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
+2 -2
View File
@@ -35,10 +35,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -26,10 +26,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
@@ -38,22 +38,7 @@ jobs:
working-directory: test/s3/versioning
run: |
set -x
# Run every versioning test, so a regression test lands covered instead
# of waiting for someone to remember this file. Name a test in EXCLUDE,
# with the reason, to keep it out.
#
# TestVersioningPagination*: opt-in stress tests that build 1500+
# versions. They self-skip without ENABLE_STRESS_TESTS and have their
# own make target, so this gate should not carry them.
EXCLUDE='TestVersioningPagination.*'
tests=$(go test . -list '.*' | grep '^Test' | sort -u)
# An empty list would make -run match nothing and pass this job vacuously.
[ -n "$tests" ] || { echo "listed no versioning tests"; exit 1; }
selected=$(echo "$tests" | grep -vE "^($EXCLUDE)$")
[ -n "$selected" ] || { echo "EXCLUDE matched every test"; exit 1; }
echo "running $(echo "$selected" | wc -l) of $(echo "$tests" | wc -l) versioning tests"
# make swallows a lone trailing $, taking the anchor with it, so escape it.
make test-with-server TEST_PATTERN="^($(echo "$selected" | paste -sd'|' -))"'$$'
make test-with-server TEST_PATTERN="TestVersioningCompleteMultipartUploadIsIdempotent|TestVersioningSelfCopyMetadataReplaceCreatesNewVersion|TestVersioningSelfCopyMetadataReplaceSuspendedKeepsNullVersion|TestSuspendedDeleteCreatesDeleteMarker"
- name: Show server logs on failure
if: failure()
@@ -79,10 +64,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
@@ -117,10 +102,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
+5 -5
View File
@@ -36,16 +36,16 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
cache: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v7
uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
@@ -143,10 +143,10 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
cache: true
+10 -10
View File
@@ -43,10 +43,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -86,10 +86,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -195,10 +195,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -313,10 +313,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -400,10 +400,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
+2 -21
View File
@@ -3,22 +3,8 @@ name: "S3 Proxy Signature Tests"
on:
push:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/server/**'
- 'test/s3/proxy_signature/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/s3-proxy-signature-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/server/**'
- 'test/s3/proxy_signature/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/s3-proxy-signature-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/s3-proxy-signature-tests
@@ -34,19 +20,14 @@ jobs:
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
@@ -28,10 +28,10 @@ jobs:
timeout-minutes: 10
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
+5 -9
View File
@@ -25,27 +25,23 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Set up Docker
uses: docker/setup-buildx-action@v4
- name: Install SeaweedFS
run: |
go install -buildvcs=false ./weed
- name: Pre-pull Spark image
run: |
pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
pull apache/spark:3.5.1
run: docker pull apache/spark:3.5.8
- name: Run S3 Spark integration tests
working-directory: test/s3/spark
+14 -14
View File
@@ -44,10 +44,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -112,10 +112,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -160,10 +160,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -209,10 +209,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -258,10 +258,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -309,10 +309,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -357,10 +357,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
+73 -202
View File
@@ -2,14 +2,6 @@ name: "S3 Tables Integration Tests"
on:
pull_request:
paths:
- 'weed/s3api/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'test/s3tables/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/s3-tables-tests.yml'
permissions:
contents: read
@@ -23,10 +15,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -83,19 +75,14 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Run go mod tidy
run: go mod tidy
@@ -144,23 +131,19 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Set up Docker
uses: docker/setup-buildx-action@v4
- name: Pre-pull Trino image
run: |
pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
pull trinodb/trino:479
run: docker pull trinodb/trino:479
- name: Run go mod tidy
run: go mod tidy
@@ -214,24 +197,22 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Set up Docker
uses: docker/setup-buildx-action@v4
- name: Pre-pull images
run: |
pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
pull dremio/dremio-oss:25.2.0
pull python:3.11-slim
- name: Pre-pull Dremio image
run: docker pull dremio/dremio-oss:25.2.0
- name: Pre-pull Python image for PyIceberg writer
run: docker pull python:3.11-slim
- name: Run go mod tidy
run: go mod tidy
@@ -287,24 +268,22 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Set up Docker
uses: docker/setup-buildx-action@v4
- name: Pre-pull images
run: |
pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
pull apache/doris:doris-all-in-one-2.1.0
pull python:3.11-slim
- name: Pre-pull Doris image
run: docker pull apache/doris:doris-all-in-one-2.1.0
- name: Pre-pull Python image for PyIceberg writer
run: docker pull python:3.11-slim
- name: Run go mod tidy
run: go mod tidy
@@ -353,79 +332,6 @@ jobs:
path: test/s3tables/catalog_doris/test-output.log
retention-days: 3
clickhouse-iceberg-catalog-tests:
name: ClickHouse Iceberg Catalog Integration Tests
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
id: go
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Pre-pull images
run: |
pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
pull clickhouse/clickhouse-server:25.8
pull python:3.11-slim
- name: Run go mod tidy
run: go mod tidy
- name: Install SeaweedFS
run: |
go install -buildvcs=false ./weed
- name: Run ClickHouse Iceberg Catalog Integration Tests
timeout-minutes: 25
working-directory: test/s3tables/catalog_clickhouse
run: |
set -x
set -o pipefail
echo "=== System Information ==="
uname -a
free -h
df -h
docker info
echo "=== Starting ClickHouse Iceberg Catalog Tests ==="
go test -v -timeout 20m . 2>&1 | tee test-output.log || {
echo "ClickHouse Iceberg catalog integration tests failed"
exit 1
}
- name: Show test output on failure
if: failure()
working-directory: test/s3tables/catalog_clickhouse
run: |
echo "=== Test Output ==="
if [ -f test-output.log ]; then
tail -200 test-output.log
fi
echo "=== Process information ==="
ps aux | grep -E "(weed|test|docker|clickhouse)" || true
echo "=== ClickHouse containers ==="
docker ps -a --filter "name=seaweed-clickhouse" || true
- name: Upload test logs on failure
if: failure()
uses: actions/upload-artifact@v7
with:
name: clickhouse-iceberg-catalog-test-logs
path: test/s3tables/catalog_clickhouse/test-output.log
retention-days: 3
polaris-integration-tests:
name: Polaris Integration Tests
runs-on: ubuntu-22.04
@@ -433,10 +339,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -448,15 +354,8 @@ jobs:
run: |
go install -buildvcs=false ./weed
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Pre-pull Polaris image
run: |
pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
pull apache/polaris:latest
run: docker pull apache/polaris:latest
- name: Run Polaris Integration Tests
timeout-minutes: 25
@@ -501,23 +400,19 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Set up Docker
uses: docker/setup-buildx-action@v4
- name: Pre-pull Spark image
run: |
pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
pull apache/spark:3.5.1
run: docker pull apache/spark:3.5.1
- name: Run go mod tidy
run: go mod tidy
@@ -571,24 +466,21 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Set up Docker
uses: docker/setup-buildx-action@v4
- name: Pre-pull RisingWave image
run: |
pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
pull risingwavelabs/risingwave:v2.5.0
pull postgres:16-alpine
docker pull risingwavelabs/risingwave:v2.5.0
docker pull postgres:16-alpine
- name: Run go mod tidy
run: go mod tidy
@@ -642,47 +534,19 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Set up Docker
uses: docker/setup-buildx-action@v4
- name: Week stamp for image cache key
id: week
run: echo "week=$(date -u +%G-%V)" >> "$GITHUB_OUTPUT"
- name: Restore python:3 image cache
id: python-image
uses: actions/cache@v6
with:
path: /tmp/python3-image.tar
key: python3-image-${{ steps.week.outputs.week }}
restore-keys: |
python3-image-
- name: Load or pull python:3
run: |
if [ "${{ steps.python-image.outputs.cache-hit }}" = "true" ]; then
docker load -i /tmp/python3-image.tar
exit 0
fi
pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
if pull python:3; then
docker save -o /tmp/python3-image.tar python:3
elif [ -f /tmp/python3-image.tar ]; then
# Docker Hub unreachable; fall back to last week's cached image
docker load -i /tmp/python3-image.tar
else
exit 1
fi
- name: Pre-pull Python image
run: docker pull python:3
- name: Run go mod tidy
run: go mod tidy
@@ -736,14 +600,23 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Set up Docker
uses: docker/setup-buildx-action@v4
- name: Pre-pull Python image
run: docker pull python:3
- name: Pre-pull LocalStack image (if needed)
run: docker pull localstack/localstack:latest || true
- name: Run go mod tidy
run: go mod tidy
@@ -796,24 +669,22 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Configure Docker Hub mirror
run: |
echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
- name: Set up Docker
uses: docker/setup-buildx-action@v4
- name: Pre-pull images
run: |
pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
pull unitycatalog/unitycatalog:v0.4.0
pull python:3.11-slim
- name: Pre-pull Unity Catalog image
run: docker pull unitycatalog/unitycatalog:v0.4.0
- name: Pre-pull Python image for delta-rs writer
run: docker pull python:3.11-slim
- name: Run go mod tidy
run: go mod tidy
@@ -869,10 +740,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -934,10 +805,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -985,10 +856,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
+18 -38
View File
@@ -3,26 +3,8 @@ name: "Ceph S3 tests"
on:
push:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'weed/iam/**'
- 'test/s3/compatibility/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/s3tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'weed/iam/**'
- 'test/s3/compatibility/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/s3tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3tests
@@ -38,16 +20,16 @@ jobs:
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Set up Python
uses: actions/setup-python@v7
uses: actions/setup-python@v6
with:
python-version: '3.9'
@@ -63,7 +45,7 @@ jobs:
- name: Fix S3 tests bucket creation conflicts
run: |
python3 test/s3/compatibility/fix_s3_tests_bucket_conflicts.py
python3 test/s3/fix_s3_tests_bucket_conflicts.py
env:
S3_TESTS_PATH: s3-tests
@@ -381,16 +363,16 @@ jobs:
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Set up Python
uses: actions/setup-python@v7
uses: actions/setup-python@v6
with:
python-version: '3.9'
@@ -406,7 +388,7 @@ jobs:
- name: Fix S3 tests bucket creation conflicts
run: |
python3 test/s3/compatibility/fix_s3_tests_bucket_conflicts.py
python3 test/s3/fix_s3_tests_bucket_conflicts.py
env:
S3_TESTS_PATH: s3-tests
@@ -551,16 +533,16 @@ jobs:
timeout-minutes: 10
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Set up Python
uses: actions/setup-python@v7
uses: actions/setup-python@v6
with:
python-version: '3.9'
@@ -676,10 +658,10 @@ jobs:
timeout-minutes: 10
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
@@ -694,12 +676,10 @@ jobs:
export WEED_DATA_DIR="/tmp/seaweedfs-copy-test-$(date +%s)"
mkdir -p "$WEED_DATA_DIR"
set -x
# Each test bucket is its own collection and grows 7 volumes; the suite runs
# faster than the heartbeat that returns slots from the deleted collections.
weed -v 0 server -filer -filer.maxMB=64 -s3 -ip.bind 0.0.0.0 \
-dir="$WEED_DATA_DIR" \
-master.raftHashicorp -master.electionTimeout 1s -master.volumeSizeLimitMB=100 \
-volume.max=300 -volume.preStopSeconds=1 \
-volume.max=100 -volume.preStopSeconds=1 \
-master.port=9336 -volume.port=8083 -filer.port=8891 -s3.port=8003 -metricsPort=9327 \
-s3.allowDeleteBucketNotEmpty=true -s3.config="$GITHUB_WORKSPACE/docker/compose/s3.json" -master.peers=none &
pid=$!
@@ -767,7 +747,7 @@ jobs:
sleep 2
done
MASTER_ENDPOINT="http://127.0.0.1:9336" go test -v
go test -v
kill -9 $pid || true
# Clean up data directory
rm -rf "$WEED_DATA_DIR" || true
@@ -778,16 +758,16 @@ jobs:
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Set up Python
uses: actions/setup-python@v7
uses: actions/setup-python@v6
with:
python-version: '3.9'
+2 -2
View File
@@ -34,10 +34,10 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
+2 -2
View File
@@ -34,10 +34,10 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
@@ -31,7 +31,7 @@ jobs:
# SETUP & BUILD
# ========================================
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up JDK 11
uses: actions/setup-java@v5
@@ -41,7 +41,7 @@ jobs:
cache: maven
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
@@ -150,7 +150,7 @@ jobs:
- name: Cache Apache Spark
if: false && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
id: cache-spark
uses: actions/cache@v6
uses: actions/cache@v5
with:
path: spark-3.5.0-bin-hadoop3
key: spark-3.5.0-hadoop3
+2 -2
View File
@@ -24,10 +24,10 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
-80
View File
@@ -1,80 +0,0 @@
name: "terraform: validate and test modules"
on:
push:
branches: [ master ]
paths: ['terraform/**', '.github/workflows/terraform_ci.yml']
pull_request:
branches: [ master ]
paths: ['terraform/**', '.github/workflows/terraform_ci.yml']
permissions:
contents: read
jobs:
validate:
name: fmt, validate, plan-level tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- name: Set up OpenTofu
uses: opentofu/setup-opentofu@v2
with:
tofu_version: 1.12.1
- name: fmt check
working-directory: terraform
run: tofu fmt -recursive -check -diff
- name: validate core
working-directory: terraform/modules/core
run: |
tofu init -backend=false -input=false
tofu validate
- name: validate security
working-directory: terraform/modules/security
run: |
tofu init -backend=false -input=false
tofu validate
- name: plan-level tests (core)
working-directory: terraform/modules/core
run: tofu test
- name: validate examples
run: |
set -e
for ex in terraform/examples/*/; do
echo "== validate $ex =="
tofu -chdir="$ex" init -backend=false -input=false
tofu -chdir="$ex" validate
done
smoke:
name: local cluster smoke test (real weed)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: go.mod
- name: Build weed
run: go build -o "$RUNNER_TEMP/weed" ./weed
- name: Set up OpenTofu
uses: opentofu/setup-opentofu@v2
with:
tofu_version: 1.12.1
- name: Run local cluster harness
working-directory: terraform/test/local
run: WEED="$RUNNER_TEMP/weed" ./run_local_cluster.sh
- name: Run local mTLS cluster harness
working-directory: terraform/test/local-secure
run: WEED="$RUNNER_TEMP/weed" ./run_local_secure.sh
@@ -3,20 +3,8 @@ name: "test s3 over https using aws-cli"
on:
push:
branches: [master, test-https-s3-awscli]
paths:
- 'weed/s3api/**'
- 'weed/server/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/test-s3-over-https-using-awscli.yml'
pull_request:
branches: [master, test-https-s3-awscli]
paths:
- 'weed/s3api/**'
- 'weed/server/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/test-s3-over-https-using-awscli.yml'
env:
AWS_ACCESS_KEY_ID: some_access_key1
@@ -32,9 +20,9 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v6
- uses: actions/setup-go@v7
- uses: actions/setup-go@v6
with:
go-version: ^1.25
+2 -14
View File
@@ -3,20 +3,8 @@ name: "TLS Rotation Integration Tests"
on:
push:
branches: [ master ]
paths:
- 'weed/**'
- 'test/tls_rotation/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/tls-rotation-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/**'
- 'test/tls_rotation/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/tls-rotation-tests.yml'
permissions:
contents: read
@@ -28,13 +16,13 @@ jobs:
timeout-minutes: 10
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Build weed binary
run: |
+2 -9
View File
@@ -2,13 +2,6 @@ name: "TUS Protocol Tests"
on:
pull_request:
paths:
- 'weed/server/**'
- 'weed/filer/**'
- 'test/tus/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/tus-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/tus-tests
@@ -29,10 +22,10 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
+2 -14
View File
@@ -3,20 +3,8 @@ name: "Vacuum Integration Tests"
on:
push:
branches: [ master ]
paths:
- 'weed/**'
- 'test/vacuum/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/vacuum-integration-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/**'
- 'test/vacuum/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/vacuum-integration-tests.yml'
permissions:
contents: read
@@ -28,13 +16,13 @@ jobs:
timeout-minutes: 15
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version: ^1.25
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Build weed binary
run: |
@@ -29,8 +29,6 @@ permissions:
env:
TEST_TIMEOUT: '30m'
# Keep in step with the length of matrix.shard below.
SHARD_COUNT: 3
jobs:
volume-server-integration-tests:
@@ -45,10 +43,10 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v7
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v7
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
@@ -59,26 +57,28 @@ jobs:
chmod +x weed
./weed version
# Dealing the listed tests out one by one keeps the shards even. Bucketing
# them by first letter did not: names cluster, so ^Test[I-S] drew 50 of
# the 114 grpc tests and ran nearly twice as long as the other two shards.
- name: Select this shard's tests
env:
TEST_TYPE: ${{ matrix.test-type }}
SHARD: ${{ matrix.shard }}
run: |
tests=$(go test ./test/volume_server/"$TEST_TYPE"/... -list '.*' | grep '^Test' | sort -u)
# An empty list would make -run match nothing and the shard pass vacuously.
[ -n "$tests" ] || { echo "listed no tests in test/volume_server/$TEST_TYPE"; exit 1; }
selected=$(echo "$tests" | awk -v n="$SHARD_COUNT" -v i="$SHARD" 'NR % n == i - 1')
echo "shard $SHARD of $SHARD_COUNT runs $(echo "$selected" | wc -l) of $(echo "$tests" | wc -l) tests"
echo "TEST_PATTERN=^($(echo "$selected" | paste -sd'|' -))\$" >> "$GITHUB_ENV"
- name: Run volume server integration tests
env:
WEED_BINARY: ${{ github.workspace }}/weed/weed
run: |
echo "Running volume server integration tests for ${{ matrix.test-type }} (Shard ${{ matrix.shard }} of ${SHARD_COUNT})..."
if [ "${{ matrix.test-type }}" == "grpc" ]; then
if [ "${{ matrix.shard }}" == "1" ]; then
TEST_PATTERN="^Test[A-H]"
elif [ "${{ matrix.shard }}" == "2" ]; then
TEST_PATTERN="^Test[I-S]"
else
TEST_PATTERN="^Test[T-Z]"
fi
else
if [ "${{ matrix.shard }}" == "1" ]; then
TEST_PATTERN="^Test[A-G]"
elif [ "${{ matrix.shard }}" == "2" ]; then
TEST_PATTERN="^Test[H-R]"
else
TEST_PATTERN="^Test[S-Z]"
fi
fi
echo "Running volume server integration tests for ${{ matrix.test-type }} (Shard ${{ matrix.shard }}, pattern: ${TEST_PATTERN})..."
go test -v -count=1 -timeout=${{ env.TEST_TIMEOUT }} ./test/volume_server/${{ matrix.test-type }}/... -run "${TEST_PATTERN}"
- name: Collect logs on failure
@@ -99,6 +99,23 @@ jobs:
- name: Test summary
if: always()
run: |
if [ "${{ matrix.test-type }}" == "grpc" ]; then
if [ "${{ matrix.shard }}" == "1" ]; then
TEST_PATTERN="^Test[A-H]"
elif [ "${{ matrix.shard }}" == "2" ]; then
TEST_PATTERN="^Test[I-S]"
else
TEST_PATTERN="^Test[T-Z]"
fi
else
if [ "${{ matrix.shard }}" == "1" ]; then
TEST_PATTERN="^Test[A-G]"
elif [ "${{ matrix.shard }}" == "2" ]; then
TEST_PATTERN="^Test[H-R]"
else
TEST_PATTERN="^Test[S-Z]"
fi
fi
echo "## Volume Server Integration Test Summary (${{ matrix.test-type }} - Shard ${{ matrix.shard }})" >> "$GITHUB_STEP_SUMMARY"
echo "- Suite: test/volume_server/${{ matrix.test-type }} (shard ${{ matrix.shard }} of ${SHARD_COUNT}, see 'Select this shard's tests' for the split)" >> "$GITHUB_STEP_SUMMARY"
echo "- Command: go test -v -count=1 -timeout=${{ env.TEST_TIMEOUT }} ./test/volume_server/${{ matrix.test-type }}/... -run \"\${TEST_PATTERN}\"" >> "$GITHUB_STEP_SUMMARY"
echo "- Suite: test/volume_server/${{ matrix.test-type }} (Pattern: ${TEST_PATTERN})" >> "$GITHUB_STEP_SUMMARY"
echo "- Command: go test -v -count=1 -timeout=${{ env.TEST_TIMEOUT }} ./test/volume_server/${{ matrix.test-type }}/... -run \"${TEST_PATTERN}\"" >> "$GITHUB_STEP_SUMMARY"
+3 -5
View File
@@ -35,7 +35,6 @@ Your support will be really appreciated by me and other supporters!
[![nodion](https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/note/sponsor_nodion.png)](https://www.nodion.com)
[![piknik](https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/note/piknik.png)](https://www.piknik.com)
[![keepsec](https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/note/keepsec.png)](https://www.keepsec.ca)
[![zyner](https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/note/sponsor_zyner.png)](https://zyner.org)
---
@@ -426,7 +425,7 @@ If the hot/warm data is split as 20/80, with 20 servers, you can achieve storage
## SeaweedFS Filer ##
Built on top of the blob store, SeaweedFS Filer adds directory structure to create a file system. The directory structure is an interface that is implemented in many key-value stores or databases.
Built on top of the blob store, SeaweedFS Filer adds directory structure to create a file system. The directory sturcture is an interface that is implemented in many key-value stores or databases.
The content of a file is mapped to one or many blobs, distributed to multiple volumes on multiple volume servers.
@@ -510,7 +509,7 @@ SeaweedFS Filer uses off-the-shelf stores, such as MySql, Postgres, Sqlite, Mong
### Compared to MinIO ###
Please note, as Apr 25, 2026 MinIO ceased development. It's strongly discouraged to use that unmaintained software with multiple security bugs.
Please note, as Apr 25, 2026 MinIO ceased developement. It's strongly discouraged to use that unmaintained software with multiple security bugs.
MinIO followed AWS S3 closely and was ideal for testing for S3 API. It had good UI, policies, versionings, etc. SeaweedFS is trying to catch up here.
@@ -655,8 +654,7 @@ Cluster Total: 3302.88 MiB/s, 550.51 obj/s over 43s.
## Enterprise ##
For enterprise users, please visit [seaweedfs.com](https://seaweedfs.com) for the SeaweedFS Enterprise Edition,
which has advanced features, including data recovery, self-healing storage,
customizable erasure coding, EC vacuum and repair, etc.
which has a self-healing storage format with better data protection.
[Back to TOC](#table-of-contents)
+15 -69
View File
@@ -1,78 +1,24 @@
# Security Policy
## Supported versions
## Reporting a Vulnerability
Security fixes land in the latest release. Please reproduce against a recent
release or `master` before reporting; issues that only reproduce on old,
unsupported versions are not eligible for a fix or an advisory.
If you find a security issue in SeaweedFS, please report it privately:
## Reporting a vulnerability
- Email: support@seaweedfs.com
- Do not open a public GitHub issue
Report privately through GitHub private vulnerability reporting (the "Report a
vulnerability" button under the repository's Security tab). This keeps the
report, the fix, and any CVE in one place. Do not open a public issue.
Please include:
- A clear description of the issue
- Steps to reproduce (if possible)
- Affected versions
### What a report must include
## Response
We can only act on reports that show real impact. Please include:
- We will respond as soon as possible (usually within 1 business day)
- We will investigate and work on a fix
- We may coordinate disclosure with you
- Affected version (a release tag or `master` commit you reproduced on)
- The exact deployment and configuration: which components are running
(master, volume, filer, S3, admin), which ports are reachable by the
attacker, and what authentication is enabled
- The attacker's starting position: unauthenticated, a valid S3 user, an admin,
or someone with access to the internal cluster network
- The trust boundary that is crossed (e.g. an unauthenticated client reading
another tenant's data, an S3 user escalating to admin)
- A minimal, working reproduction or proof of concept
- Expected vs. actual behavior
## Notes
A report without a working reproduction and a clear trust boundary is a
hardening suggestion, not a vulnerability. We are glad to receive those, but
they are handled on the normal issue tracker, not as security advisories.
### Automated and AI-assisted reports
Output from static analysis, dependency scanners, fuzzers, or LLMs is welcome
only when you have manually validated it and can supply a working reproduction
against a supported version, per the requirements above. Raw tool output,
speculative findings, or generated reports without a demonstrated exploit will
be closed as hardening suggestions.
## Trust model
SeaweedFS is built to run with its cluster components (master, volume servers,
and the raw filer API) on a trusted network. Those internal APIs are not an
authentication boundary unless you explicitly enable a control (for example
volume JWT or filer authentication) and that control is bypassed. Exposing an
internal port directly to untrusted clients is a deployment mistake, not a
vulnerability in SeaweedFS.
Reports are in scope when they cross a boundary SeaweedFS is meant to enforce,
for example:
- Unauthenticated access to data or operations that require authentication
- One S3 identity reading, writing, or deleting another identity's data
- Privilege escalation from a normal S3 user to administrative capability
- Bypass of Object Lock / retention where it is configured
- Remotely triggered data corruption or loss
Reports are generally out of scope when they require:
- Direct access to an internal cluster port that is meant to be private
- Full master, filer, or volume server access (already a full compromise)
- An insecure example configuration rather than a documented secure setup
- Local-only impact on a host the attacker already controls
## CVE assignment
When a report is confirmed, we publish an advisory and request the CVE through
GitHub. CVEs assigned by third parties without coordinating with us, or for
issues that do not cross a boundary described above, may be disputed.
## Response and disclosure
- We aim to acknowledge a valid report within a few business days.
- We will investigate, work on a fix, and coordinate a disclosure timeline
with you.
- Please allow time for a fix before any public disclosure.
- Please allow time for a fix before public disclosure
- If you’re unsure whether something is a security issue, feel free to reach out
-167
View File
@@ -1,167 +0,0 @@
# Design: Serializing Bucket Configuration Mutations
Issue #9651 — concurrent `PutBucketVersioning` + `PutBucketEncryption` (as Terraform
issues them in parallel) intermittently lose the encryption write.
## Root cause
The bucket's entire config lives in one filer entry, `/buckets/<name>`. Every
config API does a read-modify-write of that single entry, and the writes are not
serialized:
- `updateBucketConfig(bucket, fn)` (`s3api_bucket_config.go:468`) — sources from a
possibly-stale cached `BucketConfig`, mutates `Entry.Extended`, writes the
**whole** entry. Used by: versioning, object-lock config, lifecycle, ACL/owner.
- `UpdateBucketMetadata` → `setBucketMetadata` (`:1042`) — reads a fresh entry,
mutates `Entry.Content`, writes the **whole** entry. Used by: encryption, CORS,
tagging, ownership, policy, notification.
Two ingredients produce the lost update:
1. **No serialization** of the read→modify→write (the cache mutexes only guard the
in-memory map, not the RMW).
2. **Whole-entry rewrite from an independent snapshot** — `updateBucketConfig`
rebuilds from a stale cached `BucketConfig` whose `Content` predates the
concurrent encryption write, so writing the whole entry reverts `Content`.
Sequential calls always pass (each sees the previous write), so it only surfaces
under concurrency — and CI's slower IO widens the window (the "2 of ~12 runs").
## Goals
- No lost updates across concurrent bucket-config changes — for **all** config
fields, not just versioning/encryption.
- Correct for a single S3 gateway (the reported case) and for multiple gateways.
- Reuse the filer primitives just merged (per-path lock, `WriteCondition`,
`ObjectTransaction`); do not reintroduce a distributed lock.
- Minimal blast radius: the fix lands at the two chokepoint helpers.
## Non-goals
- Changing the one-entry-per-bucket storage model.
- Multi-filer-concurrent bucket writes (addressed only as an optional phase 3).
## The two ingredients map to two complementary fixes
### Fix A — serialize + read fresh (closes the window for whole-entry writers)
Both `updateBucketConfig` and `UpdateBucketMetadata` must run their RMW under one
per-bucket critical section, and **re-read the entry fresh from the filer inside
it** — not rebuild from the cached `BucketConfig`. The lock alone is insufficient:
without the fresh read, two serialized writers still each apply a stale snapshot.
### Fix B — field-level updates (removes the collision entirely)
The two writers touch disjoint fields (`Extended[versioning]` vs `Content`). If
each path updated only its own field instead of rewriting the whole entry, neither
could clobber the other regardless of ordering. This is the structural fix and
makes serialization a defense-in-depth concern rather than a correctness
requirement for cross-field cases.
## Where to serialize (layering)
The bucket entry is a single filer entry, so unlike object writes there is no
sharding — the question is purely the scope of the lock:
| Layer | Serializes across | Cost | Notes |
|---|---|---|---|
| 1. Gateway-local per-bucket lock | one gateway process | tiny | fixes the reported (single-gateway/CI) case |
| 2. Filer per-path lock via conditional write | all gateways on one filer | small | reuses #9640 `CreateEntry`+`WriteCondition` |
| 3. Route-by-key to bucket-key owner filer | all gateways and filers | medium | same mechanism as the object DLM-removal |
## Recommended plan (phased)
### Phase 1 — minimal fix for #9651 (gateway-local lock + fresh read)
Add a bounded per-bucket lock table to `S3ApiServer`, reusing the same
`util.LockTable` the filer uses for its per-path lock:
```go
// in S3ApiServer
bucketConfigLocks *util.LockTable[string] // serialize bucket-entry RMW
func (s3a *S3ApiServer) withBucketConfigLock(bucket string, fn func() s3err.ErrorCode) s3err.ErrorCode {
lk := s3a.bucketConfigLocks.AcquireLock("bucketConfig", bucket, util.ExclusiveLock)
defer s3a.bucketConfigLocks.ReleaseLock(bucket, lk)
return fn()
}
```
Wrap the RMW in **both** chokepoints, and inside the lock read the entry fresh:
- `updateBucketConfig`: acquire the lock; re-read `/buckets/<name>` from the filer
(not the cache); rebuild `BucketConfig` from that fresh entry; apply `fn`; write;
invalidate cache; release.
- `UpdateBucketMetadata`/`setBucketMetadata`: same lock key; it already reads fresh,
so it just needs to share the critical section.
Both must use the **same** lock keyed on `bucket`, so versioning and encryption
contend on one mutex. This closes the reported window. Limitation: only one
gateway; two gateways behind a load balancer still race.
Test: parallel `PutBucketVersioning` + `PutBucketEncryption`, assert both persist
(the exact Terraform scenario), plus an N-way parallel variant over distinct
fields.
### Phase 2 — robust across gateways (field-level + CAS via merged primitives)
Move the writers off whole-entry rewrites:
- **Extended-based config** (versioning, object-lock, ownership, tagging-in-Extended)
→ `ObjectTransaction` `PATCH_EXTENDED` on `/buckets/<name>`. The owner filer reads
the entry fresh under its per-path lock and merges only the named keys, so the
gateway never sends a whole-entry snapshot — this dissolves *both* ingredients for
these fields.
- **`Content`-based config** (encryption, CORS, tags blob) — **chosen and
implemented (b3): extend `PATCH_EXTENDED` with `set_content`.** Under the same
per-path lock the filer reads the entry fresh, merges extended attributes, and
replaces `Content`, preserving the rest. So a content write becomes a field-level
patch too — `setBucketMetadata` patches `Content`, `updateBucketConfig` patches
extended keys, and the two serialize on the lock instead of racing whole-entry
rewrites. This is cleaner than the alternatives below: no client-side retry, no
storage migration, and it reuses `ObjectTransaction`'s existing atomic lock.
- (b1, rejected) Conditional `CreateEntry` overwrite with `IF_ETAG_MATCH` + retry
(#9640): correct but needs client-side retry, and the bucket directory entry has
no reliable ETag to compare on.
- (b2, future) Migrate each per-feature config out of the single `Content` blob
into its own `Extended` key. Then even *intra-blob* writes (tags vs encryption)
stop racing. Larger migration; tracked separately.
Once all paths are field-level patches, the phase-1 gateway lock is unnecessary —
the filer enforces atomicity. (This is the path taken: phase 1 was skipped.)
### Phase 3 — multi-filer (only if needed)
If multiple filers can write `/buckets/<name>` concurrently, a filer-local per-path
lock no longer suffices. Route bucket-config writes to
`PrimaryForKey("/buckets/<name>")` (the lock-ring view) and serialize on that one
owner filer — the same route-by-key design used to take object writes off the DLM.
Overkill for rare config writes; include only if multi-filer bucket writes are real.
## Correctness summary
- Phase 1: all RMW for a bucket serialize within a gateway; the fresh read means the
second writer observes the first's change. Closes #9651 for single-gateway.
- Phase 2: `PATCH_EXTENDED` is atomic field-level merge at the filer (no snapshot);
CAS turns a concurrent `Content` write into a retry, enforced under the filer's
per-path lock — correct for any number of gateways sharing a filer.
- Phase 3: one owner filer serializes all writers — correct across filers too.
## Scope checklist (every path that RMWs the bucket entry)
All of these funnel through the two chokepoints, so fixing the chokepoints covers
them — but the fix must not leave any of them on an unserialized path:
- via `updateBucketConfig`: versioning, object-lock config, lifecycle, ACL/owner.
- via `UpdateBucketMetadata`/`setBucketMetadata`: encryption, CORS, tagging,
ownership controls, bucket policy, notification.
- bucket create/delete (`CreateEntry`/`DeleteEntry` of `/buckets/<name>`) already
go through the filer's per-path lock on `CreateEntry`; ensure they take the same
bucket lock if they also patch config.
## Cache rule (must document in code)
Under the lock, **read the entry from the filer, never rebuild from the cached
`BucketConfig`**. The cache is for reads; it must be invalidated on every write and
never be the source for an RMW. This is the single most important detail — the lock
without the fresh read does not fix the bug.
+2 -7
View File
@@ -1,6 +1,4 @@
# Pin the builder to the host arch and cross-compile the (CGO-free) Go binary,
# so arm64/arm/386 targets skip QEMU emulation of the whole compile.
FROM --platform=$BUILDPLATFORM golang:1.25-alpine AS builder
FROM golang:1.25-alpine AS builder
RUN apk add git g++ fuse
RUN mkdir -p /go/src/github.com/seaweedfs/
ARG BRANCH=${BRANCH:-master}
@@ -14,12 +12,9 @@ RUN cd /go/src/github.com/seaweedfs/seaweedfs && \
git checkout $BRANCH) || \
(echo "ERROR: Branch/commit $BRANCH not found in repository" && \
echo "Available branches:" && git branch -a && exit 1))
ARG TARGETOS TARGETARCH TARGETVARIANT
RUN cd /go/src/github.com/seaweedfs/seaweedfs/weed \
&& export LDFLAGS="-X github.com/seaweedfs/seaweedfs/weed/util/version.COMMIT=$(git rev-parse --short HEAD)" \
&& export GOOS=$TARGETOS GOARCH=$TARGETARCH \
&& case "$TARGETARCH" in arm) export GOARM="${TARGETVARIANT#v}";; esac \
&& CGO_ENABLED=0 go build -tags "$TAGS" -ldflags "-extldflags -static ${LDFLAGS}" -o /go/bin/weed .
&& CGO_ENABLED=0 go install -tags "$TAGS" -ldflags "-extldflags -static ${LDFLAGS}"
# Rust volume server: use pre-built binary from CI when available (placed in
# weed-volume-prebuilt/ by the build-rust-binaries job), otherwise compile
-6
View File
@@ -15,9 +15,3 @@ COPY tarantool /opt/tarantool/app
# build app
RUN tt build app
# the base image's default healthcheck assumes a single-instance layout and
# checks a control socket path that doesn't exist for this multi-instance
# app, so it never reports healthy; check instance status instead.
HEALTHCHECK --interval=5s --timeout=3s --start-period=15s --retries=6 \
CMD tt status app | awk 'NR>2{if ($2!="RUNNING") exit 1} END{if (NR<=2) exit 1}'
-3
View File
@@ -127,9 +127,6 @@ test_tarantool: tags = tarantool
test_tarantool: build_tarantool_dev_env build
docker compose -f compose/test-tarantool-filer.yml -p seaweedfs up
test_keycloak_s3: build
docker compose -f compose/test-keycloak-s3.yml -p seaweedfs up
clean:
rm ./weed
-46
View File
@@ -1,46 +0,0 @@
services:
keycloak:
image: quay.io/keycloak/keycloak:26.0.7
command: ["start-dev", "--import-realm"]
environment:
KC_BOOTSTRAP_ADMIN_USERNAME: admin
KC_BOOTSTRAP_ADMIN_PASSWORD: admin
KC_HTTP_ENABLED: "true"
KC_HOSTNAME: "http://keycloak:8080"
KC_HOSTNAME_STRICT: "false"
ports:
- "8080:8080"
volumes:
- ../../test/s3/iam/seaweedfs-test-realm.json:/opt/keycloak/data/import/seaweedfs-test-realm.json:ro
healthcheck:
test: ["CMD-SHELL", "exec 3<>/dev/tcp/localhost/8080"]
interval: 10s
timeout: 5s
retries: 5
start_period: 60s
s3:
image: chrislusf/seaweedfs:local
command:
- -v=9
- server
- -ip=s3
- -filer
- -master.volumeSizeLimitMB=16
- -volume
- -volume.max=0
- -volume.preStopSeconds=1
- -s3
- -s3.port=8333
- -s3.config=/etc/seaweedfs/s3_config.json
- -s3.iam.config=/etc/seaweedfs/iam_config.json
environment:
WEED_MASTER_VOLUME_GROWTH_COPY_OTHER: 1
volumes:
- ../../test/s3/iam/test_config.json:/etc/seaweedfs/s3_config.json:ro
- ../../test/s3/iam/iam_config_docker.json:/etc/seaweedfs/iam_config.json:ro
ports:
- "8333:8333"
depends_on:
keycloak:
condition: service_healthy
+2 -2
View File
@@ -46,7 +46,7 @@ local filer_metadata = {
delete_by_directory_idx = function(directory)
local space = box.space.filer_metadata
local index = space.index.directory_idx
-- for each found directory
-- for each finded directories
for _, tuple in index:pairs({ directory }, { iterator = 'EQ' }) do
space:delete({ tuple[1], tuple[3] })
end
@@ -64,7 +64,7 @@ local filer_metadata = {
end
-- init results
local results = {}
-- for each found directory
-- for each finded directories
for _, tuple in directory_idx:pairs({ dirPath }, { iterator = 'EQ' }) do
-- filter by name
if filter_filename_func(tuple[3]) then
+184 -188
View File
@@ -1,16 +1,17 @@
module github.com/seaweedfs/seaweedfs
go 1.25.8
go 1.25.0
require (
cloud.google.com/go v0.123.0 // indirect
cloud.google.com/go/pubsub v1.51.0
cloud.google.com/go/storage v1.64.0
cloud.google.com/go/pubsub v1.50.2
cloud.google.com/go/storage v1.62.1
github.com/Shopify/sarama v1.38.1
github.com/aws/aws-sdk-go v1.55.8
github.com/beorn7/perks v1.0.1 // indirect
github.com/bwmarrin/snowflake v0.3.0
github.com/cenkalti/backoff/v4 v4.3.0
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/coreos/go-semver v0.3.1 // indirect
github.com/coreos/go-systemd/v22 v22.6.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
@@ -24,32 +25,32 @@ require (
github.com/facebookgo/stats v0.0.0-20151006221625-1b76add642e4
github.com/facebookgo/subset v0.0.0-20200203212716-c811ad88dec4 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/go-redsync/redsync/v4 v4.17.0
github.com/go-sql-driver/mysql v1.10.0
github.com/go-redsync/redsync/v4 v4.16.0
github.com/go-sql-driver/mysql v1.9.3
github.com/go-zookeeper/zk v1.0.4 // indirect
github.com/golang/protobuf v1.5.4
github.com/golang/snappy v1.0.0
github.com/google/btree v1.1.3
github.com/google/uuid v1.6.0
github.com/google/wire v0.7.0 // indirect
github.com/googleapis/gax-go/v2 v2.23.0 // indirect
github.com/googleapis/gax-go/v2 v2.22.0 // indirect
github.com/gorilla/mux v1.8.1
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-uuid v1.0.3 // indirect
github.com/jackc/pgx/v5 v5.10.0
github.com/jackc/pgx/v5 v5.9.2
github.com/jcmturner/gofork v1.7.6 // indirect
github.com/jcmturner/gokrb5/v8 v8.4.4 // indirect
github.com/jinzhu/copier v0.4.0
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/json-iterator/go v1.1.12
github.com/karlseguin/ccache/v2 v2.0.8
github.com/klauspost/compress v1.19.1
github.com/klauspost/reedsolomon v1.14.1
github.com/klauspost/compress v1.18.6
github.com/klauspost/reedsolomon v1.14.0
github.com/kurin/blazer v0.5.3
github.com/linxGnu/grocksdb v1.10.8
github.com/mailru/easyjson v0.9.2 // indirect
github.com/mattn/go-isatty v0.0.23 // indirect
github.com/linxGnu/grocksdb v1.10.7
github.com/mailru/easyjson v0.9.1 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/olivere/elastic/v7 v7.0.32
@@ -58,14 +59,14 @@ require (
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/posener/complete v1.2.3
github.com/pquerna/cachecontrol v0.2.0
github.com/prometheus/client_golang v1.24.1
github.com/prometheus/client_golang v1.23.2
github.com/prometheus/client_model v0.6.2
github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.21.1
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.20.1
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/seaweedfs/goexif v1.0.3
github.com/seaweedfs/raft v1.2.0
github.com/seaweedfs/raft v1.1.8
github.com/sirupsen/logrus v1.9.4 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
@@ -84,56 +85,54 @@ require (
github.com/xdg-go/scram v1.2.0
github.com/xdg-go/stringprep v1.0.4 // indirect
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
go.etcd.io/etcd/client/v3 v3.6.12
go.etcd.io/etcd/client/v3 v3.6.10
go.mongodb.org/mongo-driver v1.17.9
go.opencensus.io v0.24.0 // indirect
gocloud.dev v0.46.0
gocloud.dev/pubsub/natspubsub v0.46.0
gocloud.dev/pubsub/rabbitpubsub v0.46.0
golang.org/x/crypto v0.54.0
golang.org/x/exp v0.0.0-20260709172345-9ea1abe57597
golang.org/x/image v0.44.0
golang.org/x/net v0.57.0
gocloud.dev v0.45.0
gocloud.dev/pubsub/natspubsub v0.45.0
gocloud.dev/pubsub/rabbitpubsub v0.45.0
golang.org/x/crypto v0.51.0
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f
golang.org/x/image v0.39.0
golang.org/x/net v0.54.0
golang.org/x/oauth2 v0.36.0
golang.org/x/sys v0.47.0
golang.org/x/text v0.40.0 // indirect
golang.org/x/tools v0.48.0 // indirect
golang.org/x/sys v0.44.0
golang.org/x/text v0.37.0 // indirect
golang.org/x/tools v0.44.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/api v0.289.0
google.golang.org/genproto v0.0.0-20260519071638-aa98bba5eb94 // indirect
google.golang.org/grpc v1.84.0-dev.0.20260723093437-b6eac429d7b6
google.golang.org/api v0.278.0
google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 // indirect
google.golang.org/grpc v1.80.0
google.golang.org/protobuf v1.36.11
gopkg.in/inf.v0 v0.9.1 // indirect
modernc.org/b v1.0.0 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect
modernc.org/sqlite v1.53.0
modernc.org/sqlite v1.49.1
)
require (
cloud.google.com/go/kms v1.31.0
github.com/Azure/azure-sdk-for-go/sdk/keyvault/azkeys v0.10.0
github.com/DATA-DOG/go-sqlmock v1.5.2
github.com/Jille/raft-grpc-transport v1.6.1
github.com/ThreeDotsLabs/watermill v1.5.2
github.com/a-h/templ v0.3.1020
github.com/apache/cassandra-gocql-driver/v2 v2.1.2
github.com/apache/iceberg-go v0.6.0
github.com/ThreeDotsLabs/watermill v1.5.1
github.com/a-h/templ v0.3.1001
github.com/apache/cassandra-gocql-driver/v2 v2.1.0
github.com/apache/iceberg-go v0.5.0
github.com/apple/foundationdb/bindings/go v0.0.0-20250911184653-27f7192f47c3
github.com/arangodb/go-driver v1.6.9
github.com/armon/go-metrics v0.4.1
github.com/aws/aws-sdk-go-v2 v1.43.2
github.com/aws/aws-sdk-go-v2/config v1.32.33
github.com/aws/aws-sdk-go-v2/credentials v1.19.32
github.com/aws/aws-sdk-go-v2/service/s3 v1.105.2
github.com/cespare/xxhash/v2 v2.3.0
github.com/cognusion/imaging v1.0.4
github.com/aws/aws-sdk-go-v2 v1.41.7
github.com/aws/aws-sdk-go-v2/config v1.32.14
github.com/aws/aws-sdk-go-v2/credentials v1.19.14
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0
github.com/cognusion/imaging v1.0.3
github.com/fluent/fluent-logger-golang v1.10.1
github.com/getsentry/sentry-go v0.44.1
github.com/go-git/go-billy/v5 v5.9.0
github.com/go-ldap/ldap/v3 v3.4.13
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/flatbuffers/go v0.0.0-20230108230133-3b8644d32c50
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/hashicorp/raft v1.7.3
github.com/hashicorp/raft-boltdb/v2 v2.3.1
github.com/hashicorp/vault/api v1.23.0
@@ -142,24 +141,25 @@ require (
github.com/minio/crc64nvme v1.1.1
github.com/orcaman/concurrent-map/v2 v2.0.1
github.com/parquet-go/parquet-go v0.30.1
github.com/pkg/sftp v1.13.11
github.com/pkg/sftp v1.13.10
github.com/rabbitmq/amqp091-go v1.11.0
github.com/rclone/rclone v1.75.0
github.com/rclone/rclone v1.74.1
github.com/rdleal/intervalst v1.5.0
github.com/redis/go-redis/v9 v9.21.0
github.com/schollz/progressbar/v3 v3.19.1
github.com/seaweedfs/go-fuse/v2 v2.9.4
github.com/shirou/gopsutil/v4 v4.26.6
github.com/tarantool/go-option v1.1.0
github.com/tarantool/go-tarantool/v3 v3.0.0
github.com/testcontainers/testcontainers-go v0.43.0
github.com/redis/go-redis/v9 v9.19.0
github.com/schollz/progressbar/v3 v3.19.0
github.com/seaweedfs/go-fuse/v2 v2.9.3
github.com/shirou/gopsutil/v4 v4.26.3
github.com/tarantool/go-tarantool/v2 v2.4.2
github.com/testcontainers/testcontainers-go v0.40.0
github.com/tikv/client-go/v2 v2.0.7
github.com/willscott/go-nfs v0.0.4
github.com/willscott/go-nfs-client v0.0.0-20251022144359-801f10d98886
github.com/xeipuuv/gojsonschema v1.2.0
github.com/ydb-platform/ydb-go-sdk-auth-environ v0.5.2
github.com/ydb-platform/ydb-go-sdk/v3 v3.146.3
go.etcd.io/etcd/client/pkg/v3 v3.6.12
github.com/ydb-platform/ydb-go-sdk-auth-environ v0.5.1
github.com/ydb-platform/ydb-go-sdk/v3 v3.134.2
go.etcd.io/etcd/client/pkg/v3 v3.6.10
go.uber.org/atomic v1.11.0
golang.org/x/sync v0.22.0
golang.org/x/sync v0.20.0
golang.org/x/tools/godoc v0.1.0-deprecated
google.golang.org/grpc/security/advancedtls v1.0.0
)
@@ -170,26 +170,25 @@ require (
atomicgo.dev/cursor v0.2.0 // indirect
atomicgo.dev/keyboard v0.2.9 // indirect
atomicgo.dev/schedule v0.1.0 // indirect
cloud.google.com/go/longrunning v1.2.0 // indirect
cloud.google.com/go/pubsub/v2 v2.6.0 // indirect
cloud.google.com/go/longrunning v0.9.0 // indirect
cloud.google.com/go/pubsub/v2 v2.4.0 // indirect
dario.cat/mergo v1.0.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/keyvault/internal v0.7.1 // indirect
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
github.com/FilenCloudDienste/filen-sdk-go v0.0.39 // indirect
github.com/ProtonMail/gopenpgp/v3 v3.4.1 // indirect
github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0 // indirect
github.com/adrg/xdg v0.5.3 // indirect
github.com/anchore/go-lzo v0.1.1 // indirect
github.com/anchore/go-lzo v0.1.0 // indirect
github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
github.com/apache/arrow-go/v18 v18.7.0 // indirect
github.com/apache/thrift v0.24.0 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.5.2 // indirect
github.com/apache/arrow-go/v18 v18.5.2-0.20260220015023-a886a5722b87 // indirect
github.com/apache/thrift v0.23.0 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.9 // indirect
github.com/bahlo/generic-list-go v0.2.0 // indirect
github.com/bazelbuild/rules_go v0.46.0 // indirect
github.com/biogo/store v0.0.0-20201120204734-aad293a2328f // indirect
github.com/blevesearch/snowballstem v0.9.0 // indirect
github.com/boombuler/barcode v1.1.0 // indirect
github.com/buger/jsonparser v1.2.0 // indirect
github.com/buger/jsonparser v1.1.2 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
github.com/cockroachdb/apd/v3 v3.2.1 // indirect
@@ -201,52 +200,53 @@ require (
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v1.0.0-rc.4 // indirect
github.com/containerd/platforms v1.0.0-rc.1 // indirect
github.com/cpuguy83/dockercfg v0.3.2 // indirect
github.com/dave/dst v0.27.2 // indirect
github.com/diskfs/go-diskfs v1.9.4 // indirect
github.com/diskfs/go-diskfs v1.7.0 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/docker v28.5.2+incompatible // indirect
github.com/docker/go-connections v0.6.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dromara/dongle v1.0.1 // indirect
github.com/gin-gonic/gin v1.11.0 // indirect
github.com/go-asn1-ber/asn1-ber v1.5.8-0.20250403174932-29230038a667 // indirect
github.com/go-git/go-billy/v5 v5.9.0 // indirect
github.com/goccy/go-yaml v1.18.0 // indirect
github.com/golang/geo v0.0.0-20210211234256-740aa86cb551 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/gookit/color v1.6.0 // indirect
github.com/gookit/color v1.5.4 // indirect
github.com/gopherjs/gopherjs v1.17.2 // indirect
github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect
github.com/hamba/avro/v2 v2.31.0 // indirect
github.com/hashicorp/go-rootcerts v1.0.2 // indirect
github.com/hashicorp/go-secure-stdlib/parseutil v0.2.0 // indirect
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect
github.com/hashicorp/go-sockaddr v1.0.7 // indirect
github.com/hashicorp/hcl v1.0.1-vault-7 // indirect
github.com/internxt/rclone-adapter v0.0.0-20260708165336-dd6561bacfa2 // indirect
github.com/internxt/rclone-adapter v0.0.0-20260331173834-036f908d0160 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jaegertracing/jaeger v1.47.0 // indirect
github.com/jhump/protoreflect/v2 v2.0.0-beta.1 // indirect
github.com/jtolds/gls v4.20.0+incompatible // indirect
github.com/klauspost/asmfmt v1.3.2 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/lib/pq v1.12.0 // indirect
github.com/lib/pq v1.11.1 // indirect
github.com/lithammer/fuzzysearch v1.1.8 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/lpar/calendar v0.2.0 // indirect
github.com/magiconair/properties v1.8.10 // indirect
github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 // indirect
github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/go-archive v0.2.0 // indirect
github.com/moby/moby/api v1.54.2 // indirect
github.com/moby/moby/client v0.4.0 // indirect
github.com/moby/patternmatcher v0.6.1 // indirect
github.com/moby/go-archive v0.1.0 // indirect
github.com/moby/patternmatcher v0.6.0 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/moby/term v0.5.2 // indirect
github.com/oklog/ulid/v2 v2.1.1 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/openzipkin/zipkin-go v0.4.3 // indirect
@@ -255,20 +255,19 @@ require (
github.com/petermattis/goid v0.0.0-20260113132338-7c7de50cc741 // indirect
github.com/pierrre/geohash v1.0.0 // indirect
github.com/pquerna/otp v1.5.0 // indirect
github.com/pterm/pterm v0.12.83 // indirect
github.com/pterm/pterm v0.12.82 // indirect
github.com/quic-go/qpack v0.6.0 // indirect
github.com/rclone/Proton-API-Bridge v1.0.4 // indirect
github.com/rclone/go-proton-api v1.0.3 // indirect
github.com/rogpeppe/go-internal v1.15.0 // indirect
github.com/rasky/go-xdr v0.0.0-20170124162913-1a41d1a06c93 // indirect
github.com/rclone/Proton-API-Bridge v1.0.3 // indirect
github.com/rclone/go-proton-api v1.0.2 // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/ryanuber/go-glob v1.0.0 // indirect
github.com/sasha-s/go-deadlock v0.3.1 // indirect
github.com/smarty/assertions v1.15.0 // indirect
github.com/sony/gobreaker/v2 v2.4.0 // indirect
github.com/stretchr/objx v0.5.3 // indirect
github.com/substrait-io/substrait v0.87.0 // indirect
github.com/substrait-io/substrait-go/v8 v8.1.1 // indirect
github.com/substrait-io/substrait-protobuf/go v0.85.0 // indirect
github.com/twmb/avro v1.7.2 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/substrait-io/substrait v0.81.0 // indirect
github.com/substrait-io/substrait-go/v7 v7.4.0 // indirect
github.com/substrait-io/substrait-protobuf/go v0.81.0 // indirect
github.com/twpayne/go-geom v1.6.1 // indirect
github.com/twpayne/go-kml/v3 v3.2.1 // indirect
github.com/tyler-smith/go-bip39 v1.1.0 // indirect
@@ -278,117 +277,122 @@ require (
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
github.com/zeebo/xxh3 v1.1.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.43.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.42.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.38.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.38.0 // indirect
go.opentelemetry.io/otel/exporters/zipkin v1.36.0 // indirect
go.opentelemetry.io/proto/otlp v1.10.0 // indirect
go.opentelemetry.io/proto/otlp v1.9.0 // indirect
go.uber.org/mock v0.5.2 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/mod v0.38.0 // indirect
golang.org/x/mod v0.35.0 // indirect
golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa // indirect
gonum.org/v1/gonum v0.17.0 // indirect
)
require (
cel.dev/expr v0.25.2 // indirect
cel.dev/expr v0.25.1 // indirect
cloud.google.com/go/auth v0.20.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
cloud.google.com/go/compute/metadata v0.9.0 // indirect
cloud.google.com/go/iam v1.11.0 // indirect
cloud.google.com/go/monitoring v1.29.0 // indirect
filippo.io/edwards25519 v1.2.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.22.0
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.0
cloud.google.com/go/iam v1.7.0 // indirect
cloud.google.com/go/monitoring v1.24.3 // indirect
filippo.io/edwards25519 v1.1.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.1
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.7.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.7.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.4 // indirect
github.com/Azure/go-ntlmssp v0.1.1 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.7.2 // indirect
github.com/Files-com/files-sdk-go/v3 v3.3.194 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.34.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.57.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.57.0 // indirect
github.com/IBM/go-sdk-core/v5 v5.23.1 // indirect
github.com/Files-com/files-sdk-go/v3 v3.3.82 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.31.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.55.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.55.0 // indirect
github.com/IBM/go-sdk-core/v5 v5.21.2 // indirect
github.com/Max-Sum/base32768 v0.0.0-20230304063302-18e6ce5945fd // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/ProtonMail/bcrypt v0.0.0-20211005172633-e235017c1baf // indirect
github.com/ProtonMail/gluon v0.17.1-0.20230724134000-308be39be96e // indirect
github.com/ProtonMail/go-crypto v1.4.1 // indirect
github.com/ProtonMail/go-mime v0.0.0-20230322103455-7d82a3887f2f // indirect
github.com/ProtonMail/go-srp v0.0.7 // indirect
github.com/PuerkitoBio/goquery v1.12.0 // indirect
github.com/ProtonMail/gopenpgp/v2 v2.9.0 // indirect
github.com/PuerkitoBio/goquery v1.11.0 // indirect
github.com/abbot/go-http-auth v0.4.0 // indirect
github.com/andybalholm/brotli v1.2.2 // indirect
github.com/andybalholm/cascadia v1.3.4 // indirect
github.com/andybalholm/brotli v1.2.0 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc // indirect
github.com/arangodb/go-velocypack v0.0.0-20200318135517-5af53c29c67e // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.33 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.34 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.33 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.33 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.34 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.14 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.33 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 // indirect
github.com/aws/aws-sdk-go-v2/service/sns v1.39.14 // indirect
github.com/aws/aws-sdk-go-v2/service/sqs v1.42.24 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.33.2 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.2 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.45.2
github.com/aws/smithy-go v1.27.5
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.21 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.13 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.6 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23 // indirect
github.com/aws/aws-sdk-go-v2/service/sns v1.39.7 // indirect
github.com/aws/aws-sdk-go-v2/service/sqs v1.42.17 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.15 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.19 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.41.10
github.com/aws/smithy-go v1.25.1
github.com/boltdb/bolt v1.3.1 // indirect
github.com/bradenaw/juniper v0.15.3 // indirect
github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 // indirect
github.com/buengese/sgzip v0.1.1 // indirect
github.com/calebcase/tmpfile v1.0.3 // indirect
github.com/chilts/sid v0.0.0-20190607042430-660e94789ec9 // indirect
github.com/cloudflare/circl v1.6.4 // indirect
github.com/cloudinary/cloudinary-go/v2 v2.16.0 // indirect
github.com/cloudsoda/go-smb2 v0.0.0-20260701064823-d8c5600d73b8 // indirect
github.com/cloudflare/circl v1.6.3 // indirect
github.com/cloudinary/cloudinary-go/v2 v2.15.0 // indirect
github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc // indirect
github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc // indirect
github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 // indirect
github.com/cncf/xds/go v0.0.0-20251210132809-ee656c7534f5 // indirect
github.com/colinmarc/hdfs/v2 v2.4.0 // indirect
github.com/creasty/defaults v1.8.0 // indirect
github.com/cronokirby/saferith v0.33.1-0.20250226174546-1f11f94ce488 // indirect
github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 // indirect
github.com/d4l3k/messagediff v1.2.1 // indirect
github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 // indirect
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.4.0 // indirect
github.com/ebitengine/purego v0.10.1 // indirect
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5 // indirect
github.com/ebitengine/purego v0.10.0 // indirect
github.com/elastic/gosigar v0.14.3 // indirect
github.com/emersion/go-message v0.18.2 // indirect
github.com/emersion/go-vcard v0.0.0-20260618161152-d854b7e0e2d3 // indirect
github.com/envoyproxy/go-control-plane/envoy v1.37.0 // indirect
github.com/envoyproxy/protoc-gen-validate v1.3.3 // indirect
github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff // indirect
github.com/envoyproxy/go-control-plane/envoy v1.36.0 // indirect
github.com/envoyproxy/protoc-gen-validate v1.3.0 // indirect
github.com/fatih/color v1.18.0 // indirect
github.com/felixge/httpsnoop v1.1.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/flynn/noise v1.1.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.13 // indirect
github.com/geoffgarside/ber v1.2.0 // indirect
github.com/go-chi/chi/v5 v5.3.1 // indirect
github.com/go-chi/chi/v5 v5.2.5 // indirect
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 // indirect
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-openapi/errors v0.22.8 // indirect
github.com/go-openapi/strfmt v0.27.0 // indirect
github.com/go-openapi/errors v0.22.6 // indirect
github.com/go-openapi/strfmt v0.25.0 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.30.3 // indirect
github.com/go-playground/validator/v10 v10.30.1 // indirect
github.com/go-resty/resty/v2 v2.17.2 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/goccy/go-json v0.10.6 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/gofrs/flock v0.13.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.18 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.15 // indirect
github.com/gorilla/schema v1.4.1 // indirect
github.com/gorilla/securecookie v1.1.2 // indirect
github.com/gorilla/sessions v1.4.0
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.3 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
@@ -406,23 +410,24 @@ require (
github.com/jtolio/noiseconn v0.0.0-20231127013910-f6d9ecbf1de7 // indirect
github.com/jzelinskie/whirlpool v0.0.0-20201016144138-0675e54bb004 // indirect
github.com/k0kubun/pp v3.0.1+incompatible
github.com/klauspost/cpuid/v2 v2.4.0 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/koofr/go-httpclient v0.0.0-20240520111329-e20f8f203988 // indirect
github.com/koofr/go-koofrclient v0.0.0-20221207135200-cbd7fc9ad6a6 // indirect
github.com/kr/fs v0.1.0 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/lanrat/extsort v1.4.2 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/lufia/plan9stats v0.0.0-20260627054121-477a66015f15 // indirect
github.com/mattn/go-colorable v0.1.15 // indirect
github.com/mattn/go-runewidth v0.0.24 // indirect
github.com/lpar/date v1.0.0 // indirect
github.com/lufia/plan9stats v0.0.0-20260216142805-b3301c5f2a88 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-runewidth v0.0.22 // indirect
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4
github.com/montanaflynn/stats v0.7.1 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nats-io/nats.go v1.49.0 // indirect
github.com/nats-io/nkeys v0.4.15 // indirect
github.com/nats-io/nats.go v1.48.0 // indirect
github.com/nats-io/nkeys v0.4.12 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/ncw/swift/v2 v2.0.5 // indirect
@@ -430,20 +435,20 @@ require (
github.com/oklog/ulid v1.3.1 // indirect
github.com/onsi/ginkgo/v2 v2.23.3 // indirect
github.com/opentracing/opentracing-go v1.2.0 // indirect
github.com/oracle/oci-go-sdk/v65 v65.121.0 // indirect
github.com/panjf2000/ants/v2 v2.12.1 // indirect
github.com/oracle/oci-go-sdk/v65 v65.111.0 // indirect
github.com/panjf2000/ants/v2 v2.11.5 // indirect
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/pengsrc/go-shared v0.2.1-0.20190131101655-1999055a4a14 // indirect
github.com/philhofer/fwd v1.2.0 // indirect
github.com/pierrec/lz4/v4 v4.1.27
github.com/pierrec/lz4/v4 v4.1.26
github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c // indirect
github.com/pingcap/failpoint v0.0.0-20220801062533-2eaa32854a6c // indirect
github.com/pingcap/kvproto v0.0.0-20230403051650-e166ae588106 // indirect
github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/pkg/xattr v0.4.12 // indirect
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8 // indirect
github.com/relvacode/iso8601 v1.7.0 // indirect
@@ -451,79 +456,70 @@ require (
github.com/rivo/uniseg v0.4.7 // indirect
github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect
github.com/sagikazarmark/locafero v0.11.0 // indirect
github.com/samber/lo v1.53.0 // indirect
github.com/samber/lo v1.52.0 // indirect
github.com/seaweedfs/cockroachdb-parser v0.0.0-20260225204133-2f342c5ea564
github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966 // indirect
github.com/sony/gobreaker v1.0.0 // indirect
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
github.com/spacemonkeygo/monkit/v3 v3.0.25-0.20251022131615-eb24eb109368 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/spiffe/go-spiffe/v2 v2.8.1 // indirect
github.com/spiffe/go-spiffe/v2 v2.6.0 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/t3rm1n4l/go-mega v0.0.0-20260717075258-c6acd6a5bd04 // indirect
github.com/t3rm1n4l/go-mega v0.0.0-20251120131202-6845944c051c // indirect
github.com/tarantool/go-iproto v1.1.0 // indirect
github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a // indirect
github.com/tikv/pd/client v0.0.0-20230329114254-1948c247c2b1 // indirect
github.com/tinylib/msgp v1.6.4 // indirect
github.com/tklauser/go-sysconf v0.4.0 // indirect
github.com/tklauser/numcpus v0.12.0 // indirect
github.com/tinylib/msgp v1.6.3 // indirect
github.com/tklauser/go-sysconf v0.3.16 // indirect
github.com/tklauser/numcpus v0.11.0 // indirect
github.com/twmb/murmur3 v1.1.8 // indirect
github.com/unknwon/goconfig v1.0.0 // indirect
github.com/vmihailenco/msgpack/v5 v5.4.1 // indirect
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
github.com/winfsp/cgofuse v1.6.1-0.20260126094232-f2c4fccdb286
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/yandex-cloud/go-genproto v0.0.0-20211115083454-9ca41db5ed9e // indirect
github.com/ydb-platform/ydb-go-genproto v0.0.0-20260428144813-1c07baab7f7b // indirect
github.com/ydb-platform/ydb-go-genproto v0.0.0-20260311095541-ebbf792c1180 // indirect
github.com/ydb-platform/ydb-go-yc v0.12.1 // indirect
github.com/ydb-platform/ydb-go-yc-metadata v0.6.1 // indirect
github.com/yunify/qingstor-sdk-go/v3 v3.2.0 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
github.com/zeebo/blake3 v0.2.4 // indirect
github.com/zeebo/errs v1.4.0 // indirect
go.etcd.io/bbolt v1.5.0 // indirect
go.etcd.io/etcd/api/v3 v3.6.12 // indirect
go.etcd.io/bbolt v1.4.3 // indirect
go.etcd.io/etcd/api/v3 v3.6.10 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.44.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.68.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0 // indirect
go.opentelemetry.io/otel v1.44.0 // indirect
go.opentelemetry.io/otel/metric v1.44.0 // indirect
go.opentelemetry.io/otel/sdk v1.44.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.44.0 // indirect
go.opentelemetry.io/otel/trace v1.44.0 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.39.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.67.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0 // indirect
go.opentelemetry.io/otel v1.43.0 // indirect
go.opentelemetry.io/otel/metric v1.43.0 // indirect
go.opentelemetry.io/otel/sdk v1.43.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.43.0 // indirect
go.opentelemetry.io/otel/trace v1.43.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.1 // indirect
golang.org/x/term v0.45.0
golang.org/x/term v0.43.0
golang.org/x/time v0.15.0
google.golang.org/genproto/googleapis/api v0.0.0-20260706201446-f0a921348800 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260715232425-e75dac1f907d // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
gopkg.in/validator.v2 v2.0.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
modernc.org/libc v1.73.4 // indirect
modernc.org/libc v1.72.0 // indirect
moul.io/http2curl/v2 v2.3.0 // indirect
sigs.k8s.io/yaml v1.6.0 // indirect
storj.io/common v0.0.0-20260629224719-ba1bff0a7846 // indirect
storj.io/drpc v1.0.0 // indirect
storj.io/eventkit v0.0.0-20260716074419-6861a92e2aa5 // indirect
storj.io/common v0.0.0-20260225132117-99155641c30a // indirect
storj.io/drpc v0.0.35-0.20250513201419-f7819ea69b55 // indirect
storj.io/eventkit v0.0.0-20250410172343-61f26d3de156 // indirect
storj.io/infectious v0.0.2 // indirect
storj.io/picobuf v0.0.4 // indirect
storj.io/uplink v1.14.3 // indirect
storj.io/uplink v1.14.0 // indirect
)
// replace github.com/seaweedfs/raft => /Users/chrislu/go/src/github.com/seaweedfs/raft
// apache/thrift v0.23.0 fixes CVE-2026-41602 but compares int against the
// untyped math.MaxUint32 in lib/go/thrift/framed_transport.go, which overflows
// int on 32-bit GOARCHes (e.g. openbsd/arm, linux/arm) and fails to compile.
// Upstream fixed the range check post-release; pin to that commit until the
// next tagged release carries both the CVE fix and the 32-bit fix.
replace github.com/apache/thrift => github.com/apache/thrift v0.23.1-0.20260429145742-d2acd3c49e58
// tyler-smith/go-bip39 was deleted from GitHub, so `go mod download` fails for
// anyone resolving it directly (GOPROXY=direct). It only reaches us transitively
// through rclone's internxt backend, which calls IsMnemonicValid and NewSeed.
// cosmos/go-bip39 is a maintained, API-compatible fork.
replace github.com/tyler-smith/go-bip39 => github.com/cosmos/go-bip39 v1.0.0
// apache/thrift v0.23.0 uses math.MaxUint32 as an untyped int constant in
// lib/go/thrift/framed_transport.go, which overflows int on 32-bit GOARCHes
// (e.g. openbsd/arm, linux/arm). Pin to v0.22.0 until upstream fixes it.
replace github.com/apache/thrift => github.com/apache/thrift v0.22.0
+527 -457
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -6,7 +6,7 @@
# Usage:
# curl -fsSL https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/install.sh | bash
# curl -fsSL ... | bash -s -- --component volume-rust --large-disk
# curl -fsSL ... | bash -s -- --version 4.34 --dir /usr/local/bin
# curl -fsSL ... | bash -s -- --version v3.93 --dir /usr/local/bin
#
# Options:
# --component COMP Which binary to install: weed, volume-rust, all (default: weed)
+2 -2
View File
@@ -1,6 +1,6 @@
apiVersion: v1
description: SeaweedFS
name: seaweedfs
appVersion: "4.41"
appVersion: "4.28"
# Dev note: Trigger a helm chart release by `git tag -a helm-<version>`
version: 4.41.0
version: 4.28.0
+1 -191
View File
@@ -107,120 +107,6 @@ https://github.com/rancher/local-path-provisioner
you can use ANY storage class you like, just update the correct storage-class
for your deployment.
### Master data: hostPath vs a claim
The master's `-mdir` holds its Raft log and snapshots, and with them the
cluster's identity (its topology UUID). `master.data.type` defaults to
`hostPath`, which does not follow a pod to another node: a master that is
rescheduled comes back with an empty data directory and a brand new cluster
UUID. With the chart's default of a single master replica there is no peer to
recover the identity from either.
Putting the master's data on a claim avoids that:
```yaml
master:
data:
type: "persistentVolumeClaim"
size: "1Gi"
storageClass: "" # empty uses the cluster's default StorageClass
```
Raft state is small, so a modest claim is enough — sizing matters far more for
volume and filer.
The default is left at `hostPath` for backward compatibility:
`volumeClaimTemplates` is immutable on a StatefulSet, so flipping the type on a
release that already exists fails, whether the chart changes the default or you
change it yourself:
```text
StatefulSet.apps "<release>-seaweedfs-master" is invalid: spec: Forbidden:
updates to statefulset spec for fields other than 'replicas', ... are forbidden
```
New installs can set the claim from the start. To move an **existing** release
onto a claim without losing the cluster UUID, use the migration below. The
claim has to be seeded while the master is stopped: a running master rewrites
its Raft state, so copying into a live pod is silently undone by the next
restart.
The steps below are for the chart's default of a single master
(`master.replicas: 1`). With several master replicas, repeat steps 1, 3 and 4
for every ordinal, or migrate one at a time and let the remaining quorum
re-replicate.
Take the names from the cluster rather than assembling them — the release
name, `nameOverride` and `fullnameOverride` all feed the chart's fullname
helper, so `<release>-seaweedfs` is not always right:
```bash
NS=<namespace>; REL=<release>
# scope by instance as well as component: several releases can share a namespace
STS=$(kubectl -n $NS get sts \
-l app.kubernetes.io/instance=$REL,app.kubernetes.io/component=master \
-o jsonpath='{.items[0].metadata.name}')
POD=$STS-0
# a StatefulSet names its claims <template>-<statefulset>-<ordinal>, and this
# chart's template is data-<namespace>
PVC=data-$NS-$STS-0
# 1. back up the master data directory
kubectl -n $NS cp $POD:/data ./master-backup
# 2. stop the master, leaving the rest of the release running
kubectl -n $NS delete sts $STS --cascade=orphan
kubectl -n $NS delete pod $POD
# 3. create the claim the new StatefulSet will adopt, and seed it through a
# pod that actually mounts it
kubectl -n $NS apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: $PVC
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
name: seed
spec:
containers:
- name: seed
image: alpine:3.20
command: ["sleep", "600"]
volumeMounts:
- name: d
mountPath: /data
volumes:
- name: d
persistentVolumeClaim:
claimName: $PVC
EOF
kubectl -n $NS wait --for=condition=Ready pod/seed --timeout=120s
kubectl -n $NS cp ./master-backup/m9333 seed:/data/
kubectl -n $NS exec seed -- ls /data/m9333 # conf, log, snapshot, state
kubectl -n $NS delete pod seed
# 4. upgrade; the StatefulSet adopts the claim you created
helm upgrade $REL seaweedfs/seaweedfs -n $NS -f values.yaml
```
Confirm the UUID survived — it must match what the cluster reported before the
migration:
```bash
kubectl -n $NS exec $POD -- curl -s localhost:9333/license/status
```
**Or accept a new cluster UUID** and, if you run the enterprise edition, have
the license re-issued against it.
## current instances config (AIO):
1 instance for each type (master/filer+s3/volume)
@@ -479,47 +365,6 @@ helm install seaweedfs-worker-vacuum seaweedfs/seaweedfs -f values-worker-vacuum
helm install seaweedfs-worker-balance seaweedfs/seaweedfs -f values-worker-balance.yaml
```
## Network Policies
In a namespace with a default-deny policy the install hangs: the components cannot resolve each other, and the post-install bucket hook waits on the master and filer until it gives up. `networkPolicy.enabled` renders one `NetworkPolicy` per component, selecting its pods by the standard `app.kubernetes.io/{name,instance,component}` labels and admitting traffic from the other pods of the release on the ports that component listens on.
```bash
helm install seaweedfs seaweedfs/seaweedfs --set networkPolicy.enabled=true
```
That alone leaves outbound traffic untouched, which is enough when the namespace's default-deny only restricts ingress. If it lists `Egress` in its `policyTypes` too - the usual baseline - the components still cannot resolve DNS, and you need the second opt-in below as well.
Egress is separate because the chart knows where its own components live but not where your filer store, notification sink or remote tier does, and because in a namespace with no default-deny at all, adding egress rules would narrow the components from "may reach anything" to "may reach these peers":
```yaml
networkPolicy:
enabled: true
egress:
enabled: true
kubeApiServer:
# the endpoint behind the kubernetes service, not its ClusterIP
cidrs: ["172.18.0.2/32"]
extraEgress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: postgresql
ports:
- protocol: TCP
port: 5432
```
`kubeApiServer.cidrs` is only demanded when something in the release actually needs the API server, which is the COSI sidecar and, on an upgrade that grows a volume PVC, the resize hook. No seaweedfs component itself speaks to it.
Anything reaching the release from outside - an ingress controller, a Prometheus in another namespace - goes into `networkPolicy.extraIngress`, or into `networkPolicy.components.<component>.extraIngress` for a single component. See the `networkPolicy` block in `values.yaml` for the full set.
Two things worth knowing before you turn this on:
- **Monitoring stops.** The metrics ports are admitted from release pods like every other port, so with `global.seaweedfs.monitoring.enabled` the ServiceMonitors keep scraping targets a Prometheus in another namespace can no longer reach. Nothing reports it; add the scraper's namespace to `extraIngress`.
- **The resize hook's policy is a Helm hook.** Its Job runs before the release manifest is applied, so the policy has to be a `pre-install` hook too. Helm does not garbage-collect hook resources, so on an upgrade that grows a volume PVC the policy is created and then left behind on uninstall - delete `<release>-seaweedfs-volume-resize-hook` by hand if it bothers you.
The DNS selectors default to CoreDNS as kubeadm, kind and the managed offerings from AWS, Google and Azure install it. On OpenShift, override `egress.dnsNamespaceSelector` and `egress.dnsPodSelector` to match `openshift-dns`; see the comment in `values.yaml`.
## OpenShift Support
SeaweedFS can be deployed on OpenShift or any cluster enforcing the Kubernetes "restricted" Pod Security Standard. By default, OpenShift blocks containers that run as root or use `hostPath` volumes.
@@ -539,39 +384,4 @@ helm install seaweedfs seaweedfs/seaweedfs \
## Enterprise
For enterprise users, please visit [seaweedfs.com](https://seaweedfs.com) for the SeaweedFS Enterprise Edition,
which has advanced features, including data recovery, self-healing storage, customizable erasure coding, EC vacuum and repair, etc.
To run it, set the image and point the chart at a Secret holding the license
file:
```bash
kubectl create secret generic seaweedfs-license -n <namespace> \
--from-file=seaweed-license.json=/path/to/seaweed-license.json
```
```yaml
global:
seaweedfs:
image:
name: chrislusf/seaweedfs-enterprise
license:
existingSecret: seaweedfs-license
# secretKey: seaweed-license.json # key within the Secret
# mountPath: /etc/seaweedfs/license # directory it is mounted at
```
Set the image globally rather than per component: a per-component
`imageOverride` wins, and a cluster that mixes editions comes up looking
healthy with enterprise features quietly off.
Only the master reads the license, so the Secret is mounted read-only there
and on all-in-one (which runs `weed server -master`). It is mounted as a
directory, not a `subPath`, so a renewed Secret reaches the running master —
which re-reads the file periodically — without a restart.
The license is tied to the cluster UUID kept in the master's Raft state, so put
`master.data` on a claim — a master that restarts onto an empty data directory
generates a new UUID and the license stops matching. See
[Master data](#master-data-hostpath-vs-a-claim). Check the binding with
`kubectl exec <master-pod> -- curl -s localhost:9333/license/status`
(`cluster_uuid` must equal `license_uuid`).
which has a self-healing storage format with better data protection.
@@ -1 +0,0 @@
# Default configuration, kept so chart-testing still installs the chart as-is.
-5
View File
@@ -1,5 +0,0 @@
# SFTP install: the pod only becomes ready if the server loads a host key
# from the generated sftp-ssh-secret, so this exercises the whole path.
sftp:
enabled: true
enableAuth: true
@@ -3666,291 +3666,6 @@
],
"title": "S3 Bucket Object Count",
"type": "timeseries"
},
{
"collapsed": false,
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 130
},
"id": 92,
"panels": [],
"title": "Filer Object Size Distribution",
"type": "row"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "Rate of new objects created, split by size range.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "objects/s",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 25,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "normal"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "short",
"unitScale": true
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 131
},
"id": 93,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "8.1.2",
"targets": [
{
"exemplar": true,
"expr": "sum(rate(SeaweedFS_filer_object_size_bytes_bucket{le=\"1024\",namespace=\"$NAMESPACE\"}[$__rate_interval]))",
"format": "time_series",
"hide": false,
"instant": false,
"interval": "",
"intervalFactor": 2,
"legendFormat": "< 1KB",
"refId": "A",
"step": 60
},
{
"exemplar": true,
"expr": "clamp_min(sum(rate(SeaweedFS_filer_object_size_bytes_bucket{le=\"102400\",namespace=\"$NAMESPACE\"}[$__rate_interval])) - sum(rate(SeaweedFS_filer_object_size_bytes_bucket{le=\"1024\",namespace=\"$NAMESPACE\"}[$__rate_interval])), 0)",
"format": "time_series",
"hide": false,
"instant": false,
"interval": "",
"intervalFactor": 2,
"legendFormat": "1KB - 100KB",
"refId": "B",
"step": 60
},
{
"exemplar": true,
"expr": "clamp_min(sum(rate(SeaweedFS_filer_object_size_bytes_bucket{le=\"1.048576e+06\",namespace=\"$NAMESPACE\"}[$__rate_interval])) - sum(rate(SeaweedFS_filer_object_size_bytes_bucket{le=\"102400\",namespace=\"$NAMESPACE\"}[$__rate_interval])), 0)",
"format": "time_series",
"hide": false,
"instant": false,
"interval": "",
"intervalFactor": 2,
"legendFormat": "100KB - 1MB",
"refId": "C",
"step": 60
},
{
"exemplar": true,
"expr": "clamp_min(sum(rate(SeaweedFS_filer_object_size_bytes_bucket{le=\"1.048576e+08\",namespace=\"$NAMESPACE\"}[$__rate_interval])) - sum(rate(SeaweedFS_filer_object_size_bytes_bucket{le=\"1.048576e+06\",namespace=\"$NAMESPACE\"}[$__rate_interval])), 0)",
"format": "time_series",
"hide": false,
"instant": false,
"interval": "",
"intervalFactor": 2,
"legendFormat": "1MB - 100MB",
"refId": "D",
"step": 60
},
{
"exemplar": true,
"expr": "clamp_min(sum(rate(SeaweedFS_filer_object_size_bytes_bucket{le=\"1.073741824e+09\",namespace=\"$NAMESPACE\"}[$__rate_interval])) - sum(rate(SeaweedFS_filer_object_size_bytes_bucket{le=\"1.048576e+08\",namespace=\"$NAMESPACE\"}[$__rate_interval])), 0)",
"format": "time_series",
"hide": false,
"instant": false,
"interval": "",
"intervalFactor": 2,
"legendFormat": "100MB - 1GB",
"refId": "E",
"step": 60
},
{
"exemplar": true,
"expr": "clamp_min(sum(rate(SeaweedFS_filer_object_size_bytes_count{namespace=\"$NAMESPACE\"}[$__rate_interval])) - sum(rate(SeaweedFS_filer_object_size_bytes_bucket{le=\"1.073741824e+09\",namespace=\"$NAMESPACE\"}[$__rate_interval])), 0)",
"format": "time_series",
"hide": false,
"instant": false,
"interval": "",
"intervalFactor": 2,
"legendFormat": "> 1GB",
"refId": "F",
"step": 60
}
],
"title": "Filer Object Write Rate by Size Range",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "Objects created per size range over the selected time window.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 131
},
"id": 94,
"options": {
"displayMode": "gradient",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showUnfilled": true,
"valueMode": "color"
},
"pluginVersion": "8.1.2",
"targets": [
{
"exemplar": true,
"expr": "sum(increase(SeaweedFS_filer_object_size_bytes_bucket{le=\"1024\",namespace=\"$NAMESPACE\"}[$__range]))",
"format": "time_series",
"instant": true,
"interval": "",
"intervalFactor": 2,
"legendFormat": "< 1KB",
"refId": "A"
},
{
"exemplar": true,
"expr": "clamp_min(sum(increase(SeaweedFS_filer_object_size_bytes_bucket{le=\"102400\",namespace=\"$NAMESPACE\"}[$__range])) - sum(increase(SeaweedFS_filer_object_size_bytes_bucket{le=\"1024\",namespace=\"$NAMESPACE\"}[$__range])), 0)",
"format": "time_series",
"instant": true,
"interval": "",
"intervalFactor": 2,
"legendFormat": "1KB - 100KB",
"refId": "B"
},
{
"exemplar": true,
"expr": "clamp_min(sum(increase(SeaweedFS_filer_object_size_bytes_bucket{le=\"1.048576e+06\",namespace=\"$NAMESPACE\"}[$__range])) - sum(increase(SeaweedFS_filer_object_size_bytes_bucket{le=\"102400\",namespace=\"$NAMESPACE\"}[$__range])), 0)",
"format": "time_series",
"instant": true,
"interval": "",
"intervalFactor": 2,
"legendFormat": "100KB - 1MB",
"refId": "C"
},
{
"exemplar": true,
"expr": "clamp_min(sum(increase(SeaweedFS_filer_object_size_bytes_bucket{le=\"1.048576e+08\",namespace=\"$NAMESPACE\"}[$__range])) - sum(increase(SeaweedFS_filer_object_size_bytes_bucket{le=\"1.048576e+06\",namespace=\"$NAMESPACE\"}[$__range])), 0)",
"format": "time_series",
"instant": true,
"interval": "",
"intervalFactor": 2,
"legendFormat": "1MB - 100MB",
"refId": "D"
},
{
"exemplar": true,
"expr": "clamp_min(sum(increase(SeaweedFS_filer_object_size_bytes_bucket{le=\"1.073741824e+09\",namespace=\"$NAMESPACE\"}[$__range])) - sum(increase(SeaweedFS_filer_object_size_bytes_bucket{le=\"1.048576e+08\",namespace=\"$NAMESPACE\"}[$__range])), 0)",
"format": "time_series",
"instant": true,
"interval": "",
"intervalFactor": 2,
"legendFormat": "100MB - 1GB",
"refId": "E"
},
{
"exemplar": true,
"expr": "clamp_min(sum(increase(SeaweedFS_filer_object_size_bytes_count{namespace=\"$NAMESPACE\"}[$__range])) - sum(increase(SeaweedFS_filer_object_size_bytes_bucket{le=\"1.073741824e+09\",namespace=\"$NAMESPACE\"}[$__range])), 0)",
"format": "time_series",
"instant": true,
"interval": "",
"intervalFactor": 2,
"legendFormat": "> 1GB",
"refId": "F"
}
],
"title": "Filer Object Size Distribution",
"type": "bargauge"
}
],
"refresh": "",
@@ -15,26 +15,15 @@ metadata:
{{- toYaml .Values.admin.service.annotations | nindent 4 }}
{{- end }}
spec:
{{- $serviceType := .Values.admin.service.type }}
{{- $nodePorts := dict }}
{{- if or (eq $serviceType "NodePort") (eq $serviceType "LoadBalancer") }}
{{- $nodePorts = .Values.admin.service.nodePorts | default dict }}
{{- end }}
type: {{ $serviceType }}
type: {{ .Values.admin.service.type }}
ports:
- name: "http"
port: {{ .Values.admin.port }}
targetPort: {{ .Values.admin.port }}
{{- if $nodePorts.http }}
nodePort: {{ $nodePorts.http }}
{{- end }}
protocol: TCP
- name: "grpc"
port: {{ .Values.admin.grpcPort }}
targetPort: {{ .Values.admin.grpcPort }}
{{- if $nodePorts.grpc }}
nodePort: {{ $nodePorts.grpc }}
{{- end }}
protocol: TCP
selector:
app.kubernetes.io/name: {{ template "seaweedfs.name" . }}
@@ -80,11 +80,9 @@ spec:
image: {{ template "seaweedfs.master.image" . }}
imagePullPolicy: {{ default "IfNotPresent" .Values.global.seaweedfs.imagePullPolicy }}
env:
{{- include "seaweedfs.licenseEnv" . | nindent 12 }}
{{- /* Determine default cluster alias and the corresponding env var keys to avoid conflicts */}}
{{- $mergedExtraEnvironmentVars := dict }}
{{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global.seaweedfs "component" .Values.allInOne "target" $mergedExtraEnvironmentVars) }}
{{- $clusterDefault := default "sw" (index $mergedExtraEnvironmentVars "WEED_CLUSTER_DEFAULT") }}
{{- $envMerged := merge (.Values.global.seaweedfs.extraEnvironmentVars | default dict) (.Values.allInOne.extraEnvironmentVars | default dict) }}
{{- $clusterDefault := default "sw" (index $envMerged "WEED_CLUSTER_DEFAULT") }}
{{- $clusterUpper := upper $clusterDefault }}
{{- $clusterMasterKey := printf "WEED_CLUSTER_%s_MASTER" $clusterUpper }}
{{- $clusterFilerKey := printf "WEED_CLUSTER_%s_FILER" $clusterUpper }}
@@ -102,8 +100,8 @@ spec:
fieldPath: metadata.namespace
- name: SEAWEEDFS_FULLNAME
value: "{{ include "seaweedfs.fullname" . }}"
{{- range $key := keys $mergedExtraEnvironmentVars | sortAlpha }}
{{- $value := index $mergedExtraEnvironmentVars $key }}
{{- if .Values.allInOne.extraEnvironmentVars }}
{{- range $key, $value := .Values.allInOne.extraEnvironmentVars }}
{{- if and (ne $key $clusterMasterKey) (ne $key $clusterFilerKey) }}
- name: {{ $key }}
{{- if kindIs "string" $value }}
@@ -114,6 +112,20 @@ spec:
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- if .Values.global.seaweedfs.extraEnvironmentVars }}
{{- range $key, $value := .Values.global.seaweedfs.extraEnvironmentVars }}
{{- if and (ne $key $clusterMasterKey) (ne $key $clusterFilerKey) }}
- name: {{ $key }}
{{- if kindIs "string" $value }}
value: {{ tpl $value $ | quote }}
{{- else }}
valueFrom:
{{ toYaml $value | nindent 16 }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
# Inject computed cluster endpoints for the default cluster
- name: {{ $clusterMasterKey }}
value: {{ include "seaweedfs.cluster.masterAddress" . | quote }}
@@ -121,13 +133,11 @@ spec:
value: {{ include "seaweedfs.cluster.filerAddress" . | quote }}
{{- if .Values.allInOne.secretExtraEnvironmentVars }}
{{- range $key, $value := .Values.allInOne.secretExtraEnvironmentVars }}
{{- if not (and $key (eq $key "SEAWEED_LICENSE") (include "seaweedfs.licenseEnabled" $)) }}
- name: {{ $key }}
valueFrom:
{{ toYaml $value | nindent 16 }}
{{- end }}
{{- end }}
{{- end }}
command:
- "/bin/sh"
- "-ec"
@@ -262,9 +272,6 @@ spec:
{{- $authMethods := .Values.allInOne.sftp.authMethods | default .Values.sftp.authMethods }}
{{- if $authMethods }}
-sftp.authMethods={{ $authMethods }} \
{{- if contains "certificate" $authMethods }}
-sftp.trustedUserCAKeysFile=/etc/sw/sftp_ca/ca_user.pub \
{{- end }}
{{- end }}
{{- $maxAuthTries := .Values.allInOne.sftp.maxAuthTries | default .Values.sftp.maxAuthTries }}
{{- if $maxAuthTries }}
@@ -305,19 +312,13 @@ spec:
{{- end }}
{{- if .Values.allInOne.sftp.enabled }}
- name: config-ssh
mountPath: {{ .Values.allInOne.sftp.hostKeysFolder | default .Values.sftp.hostKeysFolder | default "/etc/sw/ssh" }}
mountPath: /etc/sw/ssh
readOnly: true
{{- if or .Values.allInOne.sftp.enableAuth .Values.sftp.enableAuth }}
- mountPath: /etc/sw/sftp
name: config-users
readOnly: true
{{- end }}
{{- $aioAuthMethods := .Values.allInOne.sftp.authMethods | default .Values.sftp.authMethods }}
{{- if and $aioAuthMethods (contains "certificate" $aioAuthMethods) }}
- mountPath: /etc/sw/sftp_ca
name: config-sftp-ca
readOnly: true
{{- end }}
{{- end }}
{{- if .Values.filer.notificationConfig }}
- name: notification-config
@@ -355,7 +356,6 @@ spec:
{{- include "seaweedfs.s3.tlsVolumeMount" . | nindent 12 }}
{{- end }}
{{- end }}
{{- include "seaweedfs.licenseVolumeMount" . | nindent 12 }}
{{ tpl .Values.allInOne.extraVolumeMounts . | nindent 12 }}
ports:
- containerPort: {{ .Values.master.port }}
@@ -423,7 +423,6 @@ spec:
{{- include "seaweedfs.tplvalues.render" (dict "value" .Values.allInOne.sidecars "context" $) | nindent 8 }}
{{- end }}
volumes:
{{- include "seaweedfs.licenseVolume" . | nindent 8 }}
- name: data
{{- if eq .Values.allInOne.data.type "hostPath" }}
hostPath:
@@ -455,18 +454,6 @@ spec:
defaultMode: 420
secretName: {{ default (printf "%s-sftp-secret" (include "seaweedfs.fullname" .)) (or .Values.allInOne.sftp.existingConfigSecret .Values.sftp.existingConfigSecret) }}
{{- end }}
{{- $aioAuthMethodsVol := .Values.allInOne.sftp.authMethods | default .Values.sftp.authMethods }}
{{- if and $aioAuthMethodsVol (contains "certificate" $aioAuthMethodsVol) }}
{{- $aioCASecret := or .Values.allInOne.sftp.existingCAKeysSecret .Values.sftp.existingCAKeysSecret }}
{{- $aioCAKeys := or .Values.allInOne.sftp.trustedUserCAKeys .Values.sftp.trustedUserCAKeys }}
{{- if and (not $aioCASecret) (not $aioCAKeys) }}
{{- fail "allInOne.sftp.authMethods includes \"certificate\" but neither trustedUserCAKeys nor existingCAKeysSecret is set" }}
{{- end }}
- name: config-sftp-ca
secret:
defaultMode: 420
secretName: {{ default (printf "%s-sftp-ca-secret" (include "seaweedfs.fullname" .)) $aioCASecret }}
{{- end }}
{{- end }}
{{- if .Values.filer.notificationConfig }}
- name: notification-config
@@ -15,12 +15,7 @@ metadata:
{{- toYaml .Values.allInOne.service.annotations | nindent 4 }}
{{- end }}
spec:
{{- $serviceType := .Values.allInOne.service.type | default "ClusterIP" }}
{{- $nodePorts := dict }}
{{- if or (eq $serviceType "NodePort") (eq $serviceType "LoadBalancer") }}
{{- $nodePorts = .Values.allInOne.service.nodePorts | default dict }}
{{- end }}
type: {{ $serviceType }}
type: {{ .Values.allInOne.service.type | default "ClusterIP" }}
internalTrafficPolicy: {{ .Values.allInOne.service.internalTrafficPolicy | default "Cluster" }}
{{- if and (semverCompare ">=1.31-0" .Capabilities.KubeVersion.GitVersion) .Values.allInOne.s3.trafficDistribution }}
trafficDistribution: {{ include "seaweedfs.trafficDistribution" (dict "value" .Values.allInOne.s3.trafficDistribution "Capabilities" .Capabilities) }}
@@ -30,48 +25,30 @@ spec:
- name: "swfs-master"
port: {{ .Values.master.port }}
targetPort: {{ .Values.master.port }}
{{- if $nodePorts.master }}
nodePort: {{ $nodePorts.master }}
{{- end }}
protocol: TCP
- name: "swfs-master-grpc"
port: {{ .Values.master.grpcPort }}
targetPort: {{ .Values.master.grpcPort }}
{{- if $nodePorts.masterGrpc }}
nodePort: {{ $nodePorts.masterGrpc }}
{{- end }}
protocol: TCP
# Volume ports
- name: "swfs-volume"
port: {{ .Values.volume.port }}
targetPort: {{ .Values.volume.port }}
{{- if $nodePorts.volume }}
nodePort: {{ $nodePorts.volume }}
{{- end }}
protocol: TCP
- name: "swfs-volume-grpc"
port: {{ .Values.volume.grpcPort }}
targetPort: {{ .Values.volume.grpcPort }}
{{- if $nodePorts.volumeGrpc }}
nodePort: {{ $nodePorts.volumeGrpc }}
{{- end }}
protocol: TCP
# Filer ports
- name: "swfs-filer"
port: {{ .Values.filer.port }}
targetPort: {{ .Values.filer.port }}
{{- if $nodePorts.filer }}
nodePort: {{ $nodePorts.filer }}
{{- end }}
protocol: TCP
- name: "swfs-filer-grpc"
port: {{ .Values.filer.grpcPort }}
targetPort: {{ .Values.filer.grpcPort }}
{{- if $nodePorts.filerGrpc }}
nodePort: {{ $nodePorts.filerGrpc }}
{{- end }}
protocol: TCP
# S3 ports (if enabled)
@@ -79,30 +56,21 @@ spec:
- name: "swfs-s3"
port: {{ .Values.allInOne.s3.port | default .Values.s3.port }}
targetPort: {{ .Values.allInOne.s3.port | default .Values.s3.port }}
{{- if $nodePorts.s3 }}
nodePort: {{ $nodePorts.s3 }}
{{- end }}
protocol: TCP
{{- $httpsPort := .Values.allInOne.s3.httpsPort | default .Values.s3.httpsPort }}
{{- if $httpsPort }}
- name: "swfs-s3-tls"
port: {{ $httpsPort }}
targetPort: {{ $httpsPort }}
{{- if $nodePorts.s3Https }}
nodePort: {{ $nodePorts.s3Https }}
{{- end }}
protocol: TCP
{{- end }}
{{- end }}
# SFTP ports (if enabled)
{{- if .Values.allInOne.sftp.enabled }}
- name: "swfs-sftp"
port: {{ .Values.allInOne.sftp.port | default .Values.sftp.port }}
targetPort: {{ .Values.allInOne.sftp.port | default .Values.sftp.port }}
{{- if $nodePorts.sftp }}
nodePort: {{ $nodePorts.sftp }}
{{- end }}
protocol: TCP
{{- end }}
@@ -111,9 +79,6 @@ spec:
- name: "server-metrics"
port: {{ .Values.allInOne.metricsPort }}
targetPort: {{ .Values.allInOne.metricsPort }}
{{- if $nodePorts.metrics }}
nodePort: {{ $nodePorts.metrics }}
{{- end }}
protocol: TCP
{{- end }}
@@ -29,11 +29,6 @@ spec:
- '*.{{ include "seaweedfs.fullname" . }}-admin.{{ .Release.Namespace }}'
- '*.{{ include "seaweedfs.fullname" . }}-admin.{{ .Release.Namespace }}.svc'
- '*.{{ include "seaweedfs.fullname" . }}-admin.{{ .Release.Namespace }}.svc.cluster.local'
{{- if .Values.certificates.dnsNames }}
{{- range .Values.certificates.dnsNames }}
- {{ . | quote }}
{{- end }}
{{- end }}
{{- if .Values.certificates.ipAddresses }}
ipAddresses:
{{- range .Values.certificates.ipAddresses }}
@@ -23,11 +23,6 @@ spec:
- '*.{{ .Release.Namespace }}'
- '*.{{ .Release.Namespace }}.svc'
- '*.{{ .Release.Namespace }}.svc.cluster.local'
{{- if .Values.certificates.dnsNames }}
{{- range .Values.certificates.dnsNames }}
- {{ . | quote }}
{{- end }}
{{- end }}
- '*.{{ include "seaweedfs.fullname" . }}-master'
- '*.{{ include "seaweedfs.fullname" . }}-master.{{ .Release.Namespace }}'
- '*.{{ include "seaweedfs.fullname" . }}-master.{{ .Release.Namespace }}.svc'
@@ -28,11 +28,6 @@ spec:
- '*.{{ .Release.Namespace }}'
- '*.{{ .Release.Namespace }}.svc'
- '*.{{ .Release.Namespace }}.svc.cluster.local'
{{- if .Values.certificates.dnsNames }}
{{- range .Values.certificates.dnsNames }}
- {{ . | quote }}
{{- end }}
{{- end }}
- '*.{{ include "seaweedfs.fullname" . }}-master'
- '*.{{ include "seaweedfs.fullname" . }}-master.{{ .Release.Namespace }}'
- '*.{{ include "seaweedfs.fullname" . }}-master.{{ .Release.Namespace }}.svc'
@@ -28,11 +28,6 @@ spec:
- '*.{{ .Release.Namespace }}'
- '*.{{ .Release.Namespace }}.svc'
- '*.{{ .Release.Namespace }}.svc.cluster.local'
{{- if .Values.certificates.dnsNames }}
{{- range .Values.certificates.dnsNames }}
- {{ . | quote }}
{{- end }}
{{- end }}
- '*.{{ include "seaweedfs.fullname" . }}-master'
- '*.{{ include "seaweedfs.fullname" . }}-master.{{ .Release.Namespace }}'
- '*.{{ include "seaweedfs.fullname" . }}-master.{{ .Release.Namespace }}.svc'
@@ -28,11 +28,6 @@ spec:
- '*.{{ .Release.Namespace }}'
- '*.{{ .Release.Namespace }}.svc'
- '*.{{ .Release.Namespace }}.svc.cluster.local'
{{- if .Values.certificates.dnsNames }}
{{- range .Values.certificates.dnsNames }}
- {{ . | quote }}
{{- end }}
{{- end }}
- '*.{{ include "seaweedfs.fullname" . }}-master'
- '*.{{ include "seaweedfs.fullname" . }}-master.{{ .Release.Namespace }}'
- '*.{{ include "seaweedfs.fullname" . }}-master.{{ .Release.Namespace }}.svc'
@@ -29,11 +29,6 @@ spec:
- '*.{{ include "seaweedfs.fullname" . }}-worker.{{ .Release.Namespace }}'
- '*.{{ include "seaweedfs.fullname" . }}-worker.{{ .Release.Namespace }}.svc'
- '*.{{ include "seaweedfs.fullname" . }}-worker.{{ .Release.Namespace }}.svc.cluster.local'
{{- if .Values.certificates.dnsNames }}
{{- range .Values.certificates.dnsNames }}
- {{ . | quote }}
{{- end }}
{{- end }}
{{- if .Values.certificates.ipAddresses }}
ipAddresses:
{{- range .Values.certificates.ipAddresses }}
@@ -1,47 +0,0 @@
{{- /* Traefik IngressRouteTCP for the filer gRPC port. Rendered instead of the
standard Ingress when the gRPC ingress className starts with "traefik",
since a Kubernetes Ingress can't do raw TCP/gRPC passthrough.
global.seaweedfs.enableSecurity controls the mode:
false: plain TCP, Traefik forwards h2c to the pod. A non-TLS TCP
router can only match HostSNI(`*`) (SNI needs TLS).
true: TLS passthrough, matched by SNI so it can route on the host. */}}
{{- $filerEnabled := or .Values.filer.enabled .Values.allInOne.enabled }}
{{- $isTraefik := hasPrefix "traefik" (default "" .Values.filer.ingresses.grpc.className) }}
{{- $securityEnabled := .Values.global.seaweedfs.enableSecurity }}
{{- if and $filerEnabled .Values.filer.ingresses.grpc.enabled $isTraefik }}
{{- $serviceName := ternary (include "seaweedfs.componentName" (list . "all-in-one")) (include "seaweedfs.componentName" (list . "filer")) .Values.allInOne.enabled }}
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
name: {{ include "seaweedfs.fullname" . }}-filer-grpc
namespace: {{ .Release.Namespace }}
labels:
app.kubernetes.io/name: {{ template "seaweedfs.name" . }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: filer
spec:
{{- if $securityEnabled }}
tls:
passthrough: true
{{- end }}
entryPoints:
{{- with .Values.filer.ingresses.grpc.entryPoint }}
- {{ . | quote }}
{{- else }}
- filer-grpc
{{- end }}
routes:
{{- if $securityEnabled }}
- match: HostSNI(`{{ .Values.filer.ingresses.grpc.host }}`)
{{- else }}
- match: HostSNI(`*`)
{{- end }}
services:
- name: {{ $serviceName }}
port: {{ .Values.filer.grpcPort }}
{{- end }}
@@ -1,8 +1,6 @@
{{- /* Filer ingresses work for both normal mode (filer.enabled) and all-in-one mode (allInOne.enabled) */}}
{{- /* Filer ingress works for both normal mode (filer.enabled) and all-in-one mode (allInOne.enabled) */}}
{{- $filerEnabled := or .Values.filer.enabled .Values.allInOne.enabled }}
{{- /* HTTP Ingress */}}
{{- if and $filerEnabled .Values.filer.ingresses.http.enabled }}
{{- if and $filerEnabled .Values.filer.ingress.enabled }}
{{- /* Determine service name based on deployment mode */}}
{{- $serviceName := ternary (include "seaweedfs.componentName" (list . "all-in-one")) (include "seaweedfs.componentName" (list . "filer")) .Values.allInOne.enabled }}
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion }}
@@ -14,13 +12,10 @@ apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
name: ingress-{{ include "seaweedfs.fullname" . }}-filer-http
name: ingress-{{ include "seaweedfs.fullname" . }}-filer
namespace: {{ .Release.Namespace }}
{{- with .Values.filer.ingress.annotations }}
annotations:
{{- if and (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) .Values.filer.ingresses.http.className }}
kubernetes.io/ingress.class: {{ .Values.filer.ingresses.http.className }}
{{- end }}
{{- with .Values.filer.ingresses.http.annotations }}
{{- toYaml . | nindent 4 }}
{{- end }}
labels:
@@ -30,21 +25,14 @@ metadata:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: filer
spec:
{{- if and (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion) .Values.filer.ingresses.http.className }}
ingressClassName: {{ .Values.filer.ingresses.http.className | quote }}
{{- end }}
ingressClassName: {{ .Values.filer.ingress.className | quote }}
tls:
{{ .Values.filer.ingresses.http.tls | default list | toYaml | nindent 6}}
{{ .Values.filer.ingress.tls | default list | toYaml | nindent 6}}
rules:
- {{- if .Values.filer.ingresses.http.host }}
host: {{ .Values.filer.ingresses.http.host | quote }}
{{- end }}
http:
- http:
paths:
- path: {{ .Values.filer.ingresses.http.path | quote }}
{{- if semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion }}
pathType: {{ .Values.filer.ingresses.http.pathType | quote }}
{{- end }}
- path: {{ .Values.filer.ingress.path | quote }}
pathType: {{ .Values.filer.ingress.pathType | quote }}
backend:
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion }}
service:
@@ -55,62 +43,7 @@ spec:
serviceName: {{ $serviceName }}
servicePort: {{ .Values.filer.port }}
{{- end }}
{{- end }}
---
{{- /* gRPC Ingress (standard Kubernetes Ingress, NOT for Traefik) */}}
{{- if and $filerEnabled .Values.filer.ingresses.grpc.enabled (not (hasPrefix "traefik" (default "" .Values.filer.ingresses.grpc.className))) }}
{{- /* Determine service name based on deployment mode */}}
{{- $serviceName := ternary (include "seaweedfs.componentName" (list . "all-in-one")) (include "seaweedfs.componentName" (list . "filer")) .Values.allInOne.enabled }}
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion }}
apiVersion: networking.k8s.io/v1
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion }}
apiVersion: networking.k8s.io/v1beta1
{{- else }}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
name: ingress-{{ include "seaweedfs.fullname" . }}-filer-grpc
namespace: {{ .Release.Namespace }}
annotations:
{{- if and (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) .Values.filer.ingresses.grpc.className }}
kubernetes.io/ingress.class: {{ .Values.filer.ingresses.grpc.className }}
{{- end }}
{{- with .Values.filer.ingresses.grpc.annotations }}
{{- toYaml . | nindent 4 }}
{{- end }}
labels:
app.kubernetes.io/name: {{ template "seaweedfs.name" . }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: filer
spec:
{{- if and (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion) .Values.filer.ingresses.grpc.className }}
ingressClassName: {{ .Values.filer.ingresses.grpc.className | quote }}
{{- end }}
tls:
{{ .Values.filer.ingresses.grpc.tls | default list | toYaml | nindent 6}}
rules:
- {{- if .Values.filer.ingresses.grpc.host }}
host: {{ .Values.filer.ingresses.grpc.host | quote }}
{{- end }}
http:
paths:
- path: {{ .Values.filer.ingresses.grpc.path | quote }}
{{- if semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion }}
pathType: {{ .Values.filer.ingresses.grpc.pathType | quote }}
{{- end }}
backend:
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion }}
service:
name: {{ $serviceName }}
port:
number: {{ .Values.filer.grpcPort }}
{{- else }}
serviceName: {{ $serviceName }}
servicePort: {{ .Values.filer.grpcPort }}
{{- if .Values.filer.ingress.host }}
host: {{ .Values.filer.ingress.host }}
{{- end }}
{{- end }}
@@ -2,6 +2,8 @@
apiVersion: v1
kind: Service
metadata:
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
name: {{ include "seaweedfs.componentName" (list . "filer") }}
namespace: {{ .Release.Namespace }}
labels:
@@ -10,9 +12,8 @@ metadata:
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: filer
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
{{- if .Values.filer.annotations }}
annotations:
{{- toYaml .Values.filer.annotations | nindent 4 }}
{{- end }}
spec:

Some files were not shown because too many files have changed in this diff Show More