mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-17 04:36:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

ci: bring previously-uncovered integration tests into CI (#9281 follow-up) (#9283)

Browse Source 
* ci: bring previously-uncovered integration tests into CI (#9281 follow-up)

Six integration test packages had _test.go files but no GitHub workflow
running them. The s3-sse-tests CI gap that let #8908's UploadPartCopy bug
(and the four cross-SSE copy bugs in #9281) ship undetected was an
instance of this same pattern. This change wires three of them into CI
and removes a fourth that was deadcode:

  test/multi_master/                    NEW workflow: multi-master-tests.yml
    - 3-node master raft cluster failover/recovery (5 tests, ~65s)
  test/testutil/                         (run alongside multi_master)
    - port-allocator regression test
  test/s3/etag/                          NEW workflow: s3-etag-acl-tests.yml
    - PutObject ETag format regression for #7768 (must be pure MD5 hex,
      not "<md5>-N" composite, for AWS Java SDK v2 compatibility)
  test/s3/acl/                           (same workflow as etag)
    - object-ACL behavior on versioned buckets

  test/s3/catalog_trino/                 DELETED (deadcode)
    - Single-file copy of test/s3tables/catalog_trino/trino_catalog_test.go
      from a 2024 commit that was never iterated, while the
      test/s3tables/ counterpart has been actively maintained (and IS
      in CI via s3-tables-tests.yml's trino-iceberg-catalog-tests job).

Both workflows trigger only on changes to relevant code paths and use the
existing simple "build weed → run go test" pattern (no per-test-dir
Makefile boilerplate). The S3 workflow starts a single `weed mini`
shared by etag and acl, which keeps the job under 2 minutes on a fresh
runner.

Two tests remain knowingly uncovered:

  test/s3/basic/  — order-dependent state across tests (TestListObjectV2
    expects a bucket created by an earlier test, etc.) and uses the
    deprecated aws-sdk-go v1. Treated as sample programs, not a
    regression suite. Fixing them is out of scope for this PR.

  test/s3/catalog_trino/  — see "DELETED" above.

Verified locally:
  - go test -v -timeout=8m ./test/multi_master/... ./test/testutil/...
    PASS  (5 multi_master + 1 testutil tests, 64s)
  - weed mini + go test ./test/s3/etag/... + go test ./test/s3/acl/...
    PASS  (8 etag + 5 acl tests, ~6s after server startup)

* ci: fix log-collector glob for multi-master tests (review feedback on #9283)

test/multi_master/cluster.go creates per-test temp dirs via
os.MkdirTemp("", "seaweedfs_multi_master_it_"), so the glob has to
match that prefix. The previous version looked for MasterCluster* /
TestLeader* / TestTwoMasters* / TestAllMasters* which never matches —
the failure-artifact upload would have been empty on a real failure.

Switch the find to /tmp/seaweedfs_multi_master_it_* (maxdepth 1) so it
actually picks up the per-node master*.log files under
<baseDir>/logs/.

Found by coderabbitai review on PR #9283.
This commit is contained in:
Chris Lu
2026-04-29 10:06:59 -07:00
committed by GitHub
parent 1f515f9d02
commit 1da091f798
3 changed files with 206 additions and 344 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/multi-master-tests.yml
+77
View File
@@ -0,0 +1,77 @@
name: "Multi-Master Tests"
on:
push:
branches: [ master ]
paths:
- 'weed/server/master_*.go'
- 'weed/server/raft_*.go'
- 'weed/topology/**'
- 'test/multi_master/**'
- 'test/testutil/**'
- '.github/workflows/multi-master-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/server/master_*.go'
- 'weed/server/raft_*.go'
- 'weed/topology/**'
- 'test/multi_master/**'
- 'test/testutil/**'
- '.github/workflows/multi-master-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/multi-master-tests
cancel-in-progress: true
permissions:
contents: read
jobs:
multi-master-failover-tests:
name: Multi-Master Failover Tests
runs-on: ubuntu-22.04
timeout-minutes: 10
steps:
- name: Check out code
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Install SeaweedFS
run: |
cd weed && go install -buildvcs=false
- name: Run multi-master failover tests
# The tests in test/multi_master spin up their own 3-node master raft
# cluster (using the freshly-installed `weed` binary) and exercise
# leader-election, failover and recovery scenarios. The shared
# test/testutil port-allocator regression test runs alongside since it
# is a prerequisite for the cluster fixtures.
run: |
go test -v -timeout=8m ./test/multi_master/... ./test/testutil/...
- name: Collect server logs on failure
if: failure()
run: |
# test/multi_master/cluster.go creates per-test dirs via
# os.MkdirTemp("", "seaweedfs_multi_master_it_") and writes each
# node's log into <baseDir>/logs/master*.log.
echo "Collecting per-node master logs from temp directories..."
mkdir -p /tmp/multi-master-logs
find /tmp -maxdepth 1 -type d -name "seaweedfs_multi_master_it_*" 2>/dev/null | while read dir; do
echo "Found test directory: $dir"
cp -r "$dir" /tmp/multi-master-logs/ 2>/dev/null || true
done
find /tmp/multi-master-logs -type f -name "*.log" -print -exec tail -n 100 {} \; 2>/dev/null || echo "No logs found"
- name: Archive logs
if: failure()
uses: actions/upload-artifact@v7
with:
name: multi-master-test-logs
path: /tmp/multi-master-logs/
retention-days: 7
.github/workflows/s3-etag-acl-tests.yml
+129
View File
@@ -0,0 +1,129 @@
name: "S3 ETag and ACL Tests"
on:
push:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/filer/etag*.go'
- 'weed/server/filer_server_handlers_*.go'
- 'test/s3/etag/**'
- 'test/s3/acl/**'
- '.github/workflows/s3-etag-acl-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/filer/etag*.go'
- 'weed/server/filer_server_handlers_*.go'
- 'test/s3/etag/**'
- 'test/s3/acl/**'
- '.github/workflows/s3-etag-acl-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/s3-etag-acl-tests
cancel-in-progress: true
permissions:
contents: read
jobs:
s3-etag-acl-tests:
name: S3 ETag + ACL Integration Tests
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
- name: Check out code
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Install SeaweedFS
run: |
cd weed && go install -buildvcs=false
- name: Start weed mini (S3 on :8333)
run: |
mkdir -p /tmp/seaweedfs-etag-acl
# Minimal identity config so SSE-aware tests under acl/ can authenticate.
cat > /tmp/seaweedfs-etag-acl-s3.json <<'JSON'
{
"identities": [
{
"name": "admin",
"credentials": [
{"accessKey": "some_access_key1", "secretKey": "some_secret_key1"}
],
"actions": ["Admin", "Read", "Write"]
}
]
}
JSON
AWS_ACCESS_KEY_ID=some_access_key1 \
AWS_SECRET_ACCESS_KEY=some_secret_key1 \
weed mini \
-dir=/tmp/seaweedfs-etag-acl \
-s3.port=8333 \
-s3.config=/tmp/seaweedfs-etag-acl-s3.json \
-ip=127.0.0.1 \
> /tmp/weed-mini.log 2>&1 &
echo $! > /tmp/weed-mini.pid
# Wait for the S3 endpoint to come up (returns 403 unauth before any
# request is signed; that's fine — it means the server is listening).
for i in $(seq 1 30); do
if curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8333/ | grep -qE "^(200|403)$"; then
echo "weed mini is ready"
exit 0
fi
sleep 1
done
echo "weed mini failed to start within 30s"
tail -50 /tmp/weed-mini.log
exit 1
- name: Run ETag tests
# Pins the regression for #7768: PutObject of an auto-chunked file (>8MB)
# must return a pure MD5 hex ETag, not a `<md5>-N` composite — the AWS
# SDK for Java v2 rejects the latter on the PutObject path.
env:
S3_ENDPOINT: http://127.0.0.1:8333
AWS_ACCESS_KEY_ID: some_access_key1
AWS_SECRET_ACCESS_KEY: some_secret_key1
AWS_REGION: us-east-1
run: go test -v -timeout=5m ./test/s3/etag/...
- name: Run ACL versioning tests
# Pins object-ACL behavior on a versioned bucket: GetObjectAcl /
# PutObjectAcl with and without versionId, modifying ACLs on different
# versions independently.
env:
S3_ENDPOINT: http://127.0.0.1:8333
AWS_ACCESS_KEY_ID: some_access_key1
AWS_SECRET_ACCESS_KEY: some_secret_key1
AWS_REGION: us-east-1
run: go test -v -timeout=5m ./test/s3/acl/...
- name: Stop weed mini
if: always()
run: |
if [ -f /tmp/weed-mini.pid ]; then
kill "$(cat /tmp/weed-mini.pid)" 2>/dev/null || true
fi
- name: Show server log on failure
if: failure()
run: |
echo "=== weed mini log (last 200 lines) ==="
tail -n 200 /tmp/weed-mini.log 2>/dev/null || echo "no log available"
- name: Archive log
if: failure()
uses: actions/upload-artifact@v7
with:
name: s3-etag-acl-server-log
path: /tmp/weed-mini.log
retention-days: 3
test/s3/catalog_trino/trino_catalog_test.go
-344
View File
@@ -1,344 +0,0 @@
package catalog_trino
import (
"context"
"crypto/rand"
"fmt"
"io"
"net/http"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/test/testutil"
)
type TestEnvironment struct {
seaweedDir string
weedBinary string
dataDir string
bindIP string
s3Port int
s3GrpcPort int
icebergPort int
masterPort int
masterGrpcPort int
filerPort int
filerGrpcPort int
volumePort int
volumeGrpcPort int
weedProcess *exec.Cmd
weedCancel context.CancelFunc
trinoContainer string
dockerAvailable bool
}
func TestTrinoIcebergCatalog(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
if !env.dockerAvailable {
t.Skip("Docker not available, skipping Trino integration test")
}
env.StartSeaweedFS(t)
catalogBucket := "default"
createTableBucket(t, env, catalogBucket)
createObjectBucket(t, env, catalogBucket)
configDir := env.writeTrinoConfig(t, catalogBucket)
env.startTrinoContainer(t, configDir)
waitForTrino(t, env.trinoContainer, 60*time.Second)
schemaName := "trino_" + randomString(6)
runTrinoSQL(t, env.trinoContainer, fmt.Sprintf("CREATE SCHEMA IF NOT EXISTS iceberg.%s", schemaName))
output := runTrinoSQL(t, env.trinoContainer, "SHOW SCHEMAS FROM iceberg")
if !strings.Contains(output, schemaName) {
t.Fatalf("Expected schema %s in output:\n%s", schemaName, output)
}
runTrinoSQL(t, env.trinoContainer, fmt.Sprintf("SHOW TABLES FROM iceberg.%s", schemaName))
}
func NewTestEnvironment(t *testing.T) *TestEnvironment {
t.Helper()
wd, err := os.Getwd()
if err != nil {
t.Fatalf("Failed to get working directory: %v", err)
}
seaweedDir := wd
for i := 0; i < 6; i++ {
if _, err := os.Stat(filepath.Join(seaweedDir, "go.mod")); err == nil {
break
}
seaweedDir = filepath.Dir(seaweedDir)
}
weedBinary := filepath.Join(seaweedDir, "weed", "weed")
if _, err := os.Stat(weedBinary); os.IsNotExist(err) {
weedBinary = "weed"
if _, err := exec.LookPath(weedBinary); err != nil {
t.Skip("weed binary not found, skipping integration test")
}
}
dataDir, err := os.MkdirTemp("", "seaweed-trino-test-*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
bindIP := testutil.FindBindIP()
// 9 ports: master(2), volume(2), filer(2), s3(2), iceberg(1)
ports := testutil.MustAllocatePorts(t, 9)
return &TestEnvironment{
seaweedDir: seaweedDir,
weedBinary: weedBinary,
dataDir: dataDir,
bindIP: bindIP,
masterPort: ports[0],
masterGrpcPort: ports[1],
volumePort: ports[2],
volumeGrpcPort: ports[3],
filerPort: ports[4],
filerGrpcPort: ports[5],
s3Port: ports[6],
s3GrpcPort: ports[7],
icebergPort: ports[8],
dockerAvailable: testutil.HasDocker(),
}
}
func (env *TestEnvironment) StartSeaweedFS(t *testing.T) {
t.Helper()
securityToml := filepath.Join(env.dataDir, "security.toml")
if err := os.WriteFile(securityToml, []byte("# Empty security config for testing\n"), 0644); err != nil {
t.Fatalf("Failed to create security.toml: %v", err)
}
ctx, cancel := context.WithCancel(context.Background())
env.weedCancel = cancel
cmd := exec.CommandContext(ctx, env.weedBinary, "mini",
"-master.port", fmt.Sprintf("%d", env.masterPort),
"-master.port.grpc", fmt.Sprintf("%d", env.masterGrpcPort),
"-volume.port", fmt.Sprintf("%d", env.volumePort),
"-volume.port.grpc", fmt.Sprintf("%d", env.volumeGrpcPort),
"-filer.port", fmt.Sprintf("%d", env.filerPort),
"-filer.port.grpc", fmt.Sprintf("%d", env.filerGrpcPort),
"-s3.port", fmt.Sprintf("%d", env.s3Port),
"-s3.port.grpc", fmt.Sprintf("%d", env.s3GrpcPort),
"-s3.port.iceberg", fmt.Sprintf("%d", env.icebergPort),
"-s3.iam.readOnly=false",
"-ip", env.bindIP,
"-ip.bind", env.bindIP,
"-dir", env.dataDir,
)
cmd.Dir = env.dataDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
t.Fatalf("Failed to start SeaweedFS: %v", err)
}
env.weedProcess = cmd
if !env.waitForService(fmt.Sprintf("http://%s:%d/v1/config", env.bindIP, env.icebergPort), 30*time.Second) {
t.Fatalf("Iceberg REST API did not become ready")
}
}
func (env *TestEnvironment) Cleanup(t *testing.T) {
t.Helper()
if env.trinoContainer != "" {
_ = exec.Command("docker", "rm", "-f", env.trinoContainer).Run()
}
if env.weedCancel != nil {
env.weedCancel()
}
if env.weedProcess != nil {
time.Sleep(2 * time.Second)
_ = env.weedProcess.Wait()
}
if env.dataDir != "" {
_ = os.RemoveAll(env.dataDir)
}
}
func (env *TestEnvironment) waitForService(url string, timeout time.Duration) bool {
client := &http.Client{Timeout: 2 * time.Second}
deadline := time.Now().Add(timeout)
for time.Now().Before(deadline) {
resp, err := client.Get(url)
if err == nil {
resp.Body.Close()
if resp.StatusCode == http.StatusOK {
return true
}
}
time.Sleep(500 * time.Millisecond)
}
return false
}
func (env *TestEnvironment) writeTrinoConfig(t *testing.T, warehouseBucket string) string {
t.Helper()
configDir := filepath.Join(env.dataDir, "trino")
if err := os.MkdirAll(configDir, 0755); err != nil {
t.Fatalf("Failed to create Trino config dir: %v", err)
}
config := fmt.Sprintf(`connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://%s:%d
iceberg.rest-catalog.warehouse=s3://%s/
iceberg.file-format=PARQUET
fs.native-s3.enabled=true
s3.endpoint=http://%s:%d
s3.path-style-access=true
s3.aws-access-key=test
s3.aws-secret-key=test
s3.region=us-west-2
`, env.bindIP, env.icebergPort, warehouseBucket, env.bindIP, env.s3Port)
if err := os.WriteFile(filepath.Join(configDir, "iceberg.properties"), []byte(config), 0644); err != nil {
t.Fatalf("Failed to write Trino config: %v", err)
}
return configDir
}
func (env *TestEnvironment) startTrinoContainer(t *testing.T, configDir string) {
t.Helper()
containerName := "seaweed-trino-" + randomString(8)
env.trinoContainer = containerName
cmd := exec.Command("docker", "run", "-d",
"--name", containerName,
"--add-host", "host.docker.internal:host-gateway",
"-v", fmt.Sprintf("%s:/etc/trino/catalog", configDir),
"-v", fmt.Sprintf("%s:/test", env.dataDir),
"-e", "AWS_ACCESS_KEY_ID=test",
"-e", "AWS_SECRET_ACCESS_KEY=test",
"-e", "AWS_REGION=us-west-2",
"trinodb/trino",
)
if output, err := cmd.CombinedOutput(); err != nil {
t.Fatalf("Failed to start Trino container: %v\n%s", err, string(output))
}
}
func waitForTrino(t *testing.T, containerName string, timeout time.Duration) {
t.Helper()
deadline := time.Now().Add(timeout)
var lastOutput []byte
for time.Now().Before(deadline) {
cmd := exec.Command("docker", "exec", containerName,
"trino", "--catalog", "system", "--schema", "runtime",
"--execute", "SELECT 1",
)
if output, err := cmd.CombinedOutput(); err == nil {
return
} else {
lastOutput = output
outputStr := string(output)
if strings.Contains(outputStr, "No such container") ||
strings.Contains(outputStr, "is not running") {
break
}
}
time.Sleep(1 * time.Second)
}
logs, _ := exec.Command("docker", "logs", containerName).CombinedOutput()
t.Fatalf("Timed out waiting for Trino to be ready\nLast output:\n%s\nTrino logs:\n%s", string(lastOutput), string(logs))
}
func runTrinoSQL(t *testing.T, containerName, sql string) string {
t.Helper()
cmd := exec.Command("docker", "exec", containerName,
"trino", "--catalog", "system", "--schema", "runtime",
"--output-format", "CSV",
"--execute", sql,
)
output, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("Trino command failed: %v\nSQL: %s\nOutput:\n%s", err, sql, string(output))
}
return string(output)
}
func createTableBucket(t *testing.T, env *TestEnvironment, bucketName string) {
t.Helper()
endpoint := fmt.Sprintf("http://%s:%d/buckets", env.bindIP, env.s3Port)
reqBody := fmt.Sprintf(`{"name":"%s"}`, bucketName)
req, err := http.NewRequest(http.MethodPut, endpoint, strings.NewReader(reqBody))
if err != nil {
t.Fatalf("Failed to create request: %v", err)
}
req.Header.Set("Content-Type", "application/x-amz-json-1.1")
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("Failed to create table bucket %s: %v", bucketName, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusConflict {
body, _ := io.ReadAll(resp.Body)
t.Fatalf("Failed to create table bucket %s, status %d: %s", bucketName, resp.StatusCode, body)
}
}
func createObjectBucket(t *testing.T, env *TestEnvironment, bucketName string) {
t.Helper()
endpoint := fmt.Sprintf("http://%s:%d/%s", env.bindIP, env.s3Port, bucketName)
req, err := http.NewRequest(http.MethodPut, endpoint, nil)
if err != nil {
t.Fatalf("Failed to create S3 bucket request: %v", err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("Failed to create S3 bucket %s: %v", bucketName, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusConflict {
body, _ := io.ReadAll(resp.Body)
t.Fatalf("Failed to create S3 bucket %s, status %d: %s", bucketName, resp.StatusCode, body)
}
}
func randomString(length int) string {
const charset = "abcdefghijklmnopqrstuvwxyz0123456789"
b := make([]byte, length)
if _, err := rand.Read(b); err != nil {
panic("failed to generate random string: " + err.Error())
}
for i := range b {
b[i] = charset[int(b[i])%len(charset)]
}
return string(b)
}