ci: bring previously-uncovered integration tests into CI (#9281 follow-up) (#9283)

* ci: bring previously-uncovered integration tests into CI (#9281 follow-up)

Six integration test packages had _test.go files but no GitHub workflow
running them. The s3-sse-tests CI gap that let #8908's UploadPartCopy bug
(and the four cross-SSE copy bugs in #9281) ship undetected was an
instance of this same pattern. This change wires three of them into CI
and removes a fourth that was deadcode:

  test/multi_master/                    NEW workflow: multi-master-tests.yml
    - 3-node master raft cluster failover/recovery (5 tests, ~65s)
  test/testutil/                         (run alongside multi_master)
    - port-allocator regression test
  test/s3/etag/                          NEW workflow: s3-etag-acl-tests.yml
    - PutObject ETag format regression for #7768 (must be pure MD5 hex,
      not "<md5>-N" composite, for AWS Java SDK v2 compatibility)
  test/s3/acl/                           (same workflow as etag)
    - object-ACL behavior on versioned buckets

  test/s3/catalog_trino/                 DELETED (deadcode)
    - Single-file copy of test/s3tables/catalog_trino/trino_catalog_test.go
      from a 2024 commit that was never iterated, while the
      test/s3tables/ counterpart has been actively maintained (and IS
      in CI via s3-tables-tests.yml's trino-iceberg-catalog-tests job).

Both workflows trigger only on changes to relevant code paths and use the
existing simple "build weed → run go test" pattern (no per-test-dir
Makefile boilerplate). The S3 workflow starts a single `weed mini`
shared by etag and acl, which keeps the job under 2 minutes on a fresh
runner.

Two tests remain knowingly uncovered:

  test/s3/basic/  — order-dependent state across tests (TestListObjectV2
    expects a bucket created by an earlier test, etc.) and uses the
    deprecated aws-sdk-go v1. Treated as sample programs, not a
    regression suite. Fixing them is out of scope for this PR.

  test/s3/catalog_trino/  — see "DELETED" above.

Verified locally:
  - go test -v -timeout=8m ./test/multi_master/... ./test/testutil/...
    PASS  (5 multi_master + 1 testutil tests, 64s)
  - weed mini + go test ./test/s3/etag/... + go test ./test/s3/acl/...
    PASS  (8 etag + 5 acl tests, ~6s after server startup)

* ci: fix log-collector glob for multi-master tests (review feedback on #9283)

test/multi_master/cluster.go creates per-test temp dirs via
os.MkdirTemp("", "seaweedfs_multi_master_it_"), so the glob has to
match that prefix. The previous version looked for MasterCluster* /
TestLeader* / TestTwoMasters* / TestAllMasters* which never matches —
the failure-artifact upload would have been empty on a real failure.

Switch the find to /tmp/seaweedfs_multi_master_it_* (maxdepth 1) so it
actually picks up the per-node master*.log files under
<baseDir>/logs/.

Found by coderabbitai review on PR #9283.

.github/workflows/multi-master-tests.yml

@@ -0,0 +1,77 @@
+name: "Multi-Master Tests"
+
+on:
+  push:
+    branches: [ master ]
+    paths:
+      - 'weed/server/master_*.go'
+      - 'weed/server/raft_*.go'
+      - 'weed/topology/**'
+      - 'test/multi_master/**'
+      - 'test/testutil/**'
+      - '.github/workflows/multi-master-tests.yml'
+  pull_request:
+    branches: [ master ]
+    paths:
+      - 'weed/server/master_*.go'
+      - 'weed/server/raft_*.go'
+      - 'weed/topology/**'
+      - 'test/multi_master/**'
+      - 'test/testutil/**'
+      - '.github/workflows/multi-master-tests.yml'
+
+concurrency:
+  group: ${{ github.head_ref || github.ref }}/multi-master-tests
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  multi-master-failover-tests:
+    name: Multi-Master Failover Tests
+    runs-on: ubuntu-22.04
+    timeout-minutes: 10
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v6
+
+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: 'go.mod'
+
+      - name: Install SeaweedFS
+        run: |
+          cd weed && go install -buildvcs=false
+
+      - name: Run multi-master failover tests
+        # The tests in test/multi_master spin up their own 3-node master raft
+        # cluster (using the freshly-installed `weed` binary) and exercise
+        # leader-election, failover and recovery scenarios. The shared
+        # test/testutil port-allocator regression test runs alongside since it
+        # is a prerequisite for the cluster fixtures.
+        run: |
+          go test -v -timeout=8m ./test/multi_master/... ./test/testutil/...
+
+      - name: Collect server logs on failure
+        if: failure()
+        run: |
+          # test/multi_master/cluster.go creates per-test dirs via
+          # os.MkdirTemp("", "seaweedfs_multi_master_it_") and writes each
+          # node's log into <baseDir>/logs/master*.log.
+          echo "Collecting per-node master logs from temp directories..."
+          mkdir -p /tmp/multi-master-logs
+          find /tmp -maxdepth 1 -type d -name "seaweedfs_multi_master_it_*" 2>/dev/null | while read dir; do
+            echo "Found test directory: $dir"
+            cp -r "$dir" /tmp/multi-master-logs/ 2>/dev/null || true
+          done
+          find /tmp/multi-master-logs -type f -name "*.log" -print -exec tail -n 100 {} \; 2>/dev/null || echo "No logs found"
+
+      - name: Archive logs
+        if: failure()
+        uses: actions/upload-artifact@v7
+        with:
+          name: multi-master-test-logs
+          path: /tmp/multi-master-logs/
+          retention-days: 7

.github/workflows/s3-etag-acl-tests.yml

@@ -0,0 +1,129 @@
+name: "S3 ETag and ACL Tests"
+
+on:
+  push:
+    branches: [ master ]
+    paths:
+      - 'weed/s3api/**'
+      - 'weed/filer/etag*.go'
+      - 'weed/server/filer_server_handlers_*.go'
+      - 'test/s3/etag/**'
+      - 'test/s3/acl/**'
+      - '.github/workflows/s3-etag-acl-tests.yml'
+  pull_request:
+    branches: [ master ]
+    paths:
+      - 'weed/s3api/**'
+      - 'weed/filer/etag*.go'
+      - 'weed/server/filer_server_handlers_*.go'
+      - 'test/s3/etag/**'
+      - 'test/s3/acl/**'
+      - '.github/workflows/s3-etag-acl-tests.yml'
+
+concurrency:
+  group: ${{ github.head_ref || github.ref }}/s3-etag-acl-tests
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  s3-etag-acl-tests:
+    name: S3 ETag + ACL Integration Tests
+    runs-on: ubuntu-22.04
+    timeout-minutes: 15
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v6
+
+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: 'go.mod'
+
+      - name: Install SeaweedFS
+        run: |
+          cd weed && go install -buildvcs=false
+
+      - name: Start weed mini (S3 on :8333)
+        run: |
+          mkdir -p /tmp/seaweedfs-etag-acl
+          # Minimal identity config so SSE-aware tests under acl/ can authenticate.
+          cat > /tmp/seaweedfs-etag-acl-s3.json <<'JSON'
+          {
+            "identities": [
+              {
+                "name": "admin",
+                "credentials": [
+                  {"accessKey": "some_access_key1", "secretKey": "some_secret_key1"}
+                ],
+                "actions": ["Admin", "Read", "Write"]
+              }
+            ]
+          }
+          JSON
+          AWS_ACCESS_KEY_ID=some_access_key1 \
+          AWS_SECRET_ACCESS_KEY=some_secret_key1 \
+            weed mini \
+              -dir=/tmp/seaweedfs-etag-acl \
+              -s3.port=8333 \
+              -s3.config=/tmp/seaweedfs-etag-acl-s3.json \
+              -ip=127.0.0.1 \
+              > /tmp/weed-mini.log 2>&1 &
+          echo $! > /tmp/weed-mini.pid
+
+          # Wait for the S3 endpoint to come up (returns 403 unauth before any
+          # request is signed; that's fine — it means the server is listening).
+          for i in $(seq 1 30); do
+            if curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8333/ | grep -qE "^(200|403)$"; then
+              echo "weed mini is ready"
+              exit 0
+            fi
+            sleep 1
+          done
+          echo "weed mini failed to start within 30s"
+          tail -50 /tmp/weed-mini.log
+          exit 1
+
+      - name: Run ETag tests
+        # Pins the regression for #7768: PutObject of an auto-chunked file (>8MB)
+        # must return a pure MD5 hex ETag, not a `<md5>-N` composite — the AWS
+        # SDK for Java v2 rejects the latter on the PutObject path.
+        env:
+          S3_ENDPOINT: http://127.0.0.1:8333
+          AWS_ACCESS_KEY_ID: some_access_key1
+          AWS_SECRET_ACCESS_KEY: some_secret_key1
+          AWS_REGION: us-east-1
+        run: go test -v -timeout=5m ./test/s3/etag/...
+
+      - name: Run ACL versioning tests
+        # Pins object-ACL behavior on a versioned bucket: GetObjectAcl /
+        # PutObjectAcl with and without versionId, modifying ACLs on different
+        # versions independently.
+        env:
+          S3_ENDPOINT: http://127.0.0.1:8333
+          AWS_ACCESS_KEY_ID: some_access_key1
+          AWS_SECRET_ACCESS_KEY: some_secret_key1
+          AWS_REGION: us-east-1
+        run: go test -v -timeout=5m ./test/s3/acl/...
+
+      - name: Stop weed mini
+        if: always()
+        run: |
+          if [ -f /tmp/weed-mini.pid ]; then
+            kill "$(cat /tmp/weed-mini.pid)" 2>/dev/null || true
+          fi
+
+      - name: Show server log on failure
+        if: failure()
+        run: |
+          echo "=== weed mini log (last 200 lines) ==="
+          tail -n 200 /tmp/weed-mini.log 2>/dev/null || echo "no log available"
+
+      - name: Archive log
+        if: failure()
+        uses: actions/upload-artifact@v7
+        with:
+          name: s3-etag-acl-server-log
+          path: /tmp/weed-mini.log
+          retention-days: 3