Compare commits

..
Author SHA1 Message Date
Chris Lu 4a5243886a 4.17 2026-03-11 02:29:24 -07:00
Chris LuandGitHub e1e4c9437a fix(s3api): ListObjects with trailing-slash prefix matches sibling directories (#8599)
fix(s3api): ListObjects with trailing-slash prefix returns wrong results

When ListObjectsV2 is called with a prefix ending in "/" (e.g., "foo/"),
normalizePrefixMarker strips the trailing slash and splits into
dir="parent" and prefix="foo". The filer then lists entries matching
prefix "foo", which returns both directory "foo" and "foo1000".

The prefixEndsOnDelimiter guard correctly identifies directory "foo" as
the target and recurses into it, but then resets the guard to false.
The loop continues and incorrectly recurses into "foo1000" as well,
causing the listing to return objects from unrelated directories.

Fix: after recursing into the exact directory targeted by the
trailing-slash prefix, return immediately from the listing loop.
There is no reason to process sibling entries since the original
prefix specifically targeted one directory.
2026-03-11 02:28:34 -07:00
Chris LuandGitHub f950a941e3 Fix trust policy validation for specific AWS user principals (#8597)
* Add tests for AWS user principal in AssumeRole trust policies

Add test cases that verify trust policy validation when using specific
AWS user principals (e.g., "arn:aws:iam::000000000000:user/backend")
in the Principal field of trust policies for AssumeRole.

Covers single user, multiple users (array), wildcard, and plain string
principal formats. These tests demonstrate the bug reported in #8588
where specific user principals always fail validation.

* Populate RequestContext in ValidateTrustPolicyForPrincipal

ValidateTrustPolicyForPrincipal was creating an EvaluationContext with
a nil RequestContext. The policy engine's principal matching logic looks
up "aws:PrincipalArn" in RequestContext for non-wildcard principals,
so specific user ARNs like "arn:aws:iam::000000000000:user/backend"
always failed to match, while wildcard "*" worked because it
short-circuits before the lookup.

Populate RequestContext with both "principal" and "aws:PrincipalArn"
keys, consistent with how IsActionAllowed already does it.

Fixes #8588

* Remove GitHub discussion URL from source code comments

* Add specific error message assertions in trust policy tests
2026-03-10 21:19:40 -07:00
Chris LuandGitHub ac579c1746 Fix plugin configuration tab layout overflow (#8596)
Fix plugin configuration tab layout overflow (#8587)

Remove h-100 from Job Scheduling Settings card, which caused it to
stretch to 100% of the row height and push the Next Run card below
the row boundary, overflowing into the Detection Results section.
2026-03-10 19:19:42 -07:00
Chris LuandGitHub 0a5c5ed4ce Persist S3 bucket counter metrics across idle periods (#8595)
* Stop deleting counter metrics during bucket TTL cleanup

Counter metrics (traffic bytes, request counts, object counts) are
monotonically increasing by design. Deleting them after 10 minutes of
bucket inactivity causes them to vanish from /metrics output and reset
to zero when traffic resumes, breaking Prometheus rate()/increase()
queries and making historical traffic reporting impossible.

Only delete gauges and histograms in the TTL cleanup loop, as these
represent current state and are safely re-populated on next activity.

Fixes https://github.com/seaweedfs/seaweedfs/issues/8521

* Clean up all bucket metrics on bucket deletion

Add DeleteBucketMetrics() to delete all metrics (including counters)
for a bucket when it is explicitly deleted. This prevents unbounded
label cardinality from accumulating for buckets that no longer exist.

Called from DeleteBucketHandler after successful bucket deletion.

* Reduce mutex scope in bucket metrics TTL sweep

Collect expired bucket names under the lock, then release before
calling DeletePartialMatch on Prometheus metrics. This prevents
RecordBucketActiveTime from blocking during the expensive cleanup.
2026-03-10 19:00:40 -07:00
Chris Lu 0a2dac1e56 Reduce mutex scope in bucket metrics TTL sweep
Collect expired bucket names under the lock, then release before
calling DeletePartialMatch on Prometheus metrics. This prevents
RecordBucketActiveTime from blocking during the expensive cleanup.
2026-03-10 18:43:35 -07:00
Chris Lu 737116e83c fix port probing 2026-03-10 18:30:19 -07:00
Chris Lu b20eae697e Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2026-03-10 18:03:39 -07:00
Chris Lu 07f3f5eec5 remove worker links 2026-03-10 18:03:38 -07:00
Chris LuandGitHub 47cad59c70 Remove misleading Workers sub-menu items from admin sidebar (#8594)
* Remove misleading Workers sub-menu items from admin sidebar

The sidebar sub-items (Job Detection, Job Queue, Job Execution,
Configuration) always navigated to the first job type's tabs
(typically EC Encoding) rather than showing cross-job-type views.
This was confusing as noted in #8590. Since the in-page tabs already
provide this navigation, remove the redundant sidebar sub-items and
keep only the top-level Workers link.

Fixes #8590

* Update layout_templ.go
2026-03-10 15:55:14 -07:00
Chris LuandGitHub b17e2b411a Add dynamic timeouts to plugin worker vacuum gRPC calls (#8593)
* add dynamic timeouts to plugin worker vacuum gRPC calls

All vacuum gRPC calls used context.Background() with no deadline,
so the plugin scheduler's execution timeout could kill a job while
a large volume compact was still in progress. Use volume-size-scaled
timeouts matching the topology vacuum approach: 3 min/GB for compact,
1 min/GB for check, commit, and cleanup.

Fixes #8591

* scale scheduler execution timeout by volume size

The scheduler's per-job execution timeout (default 240s) would kill
vacuum jobs on large volumes before they finish. Three changes:

1. Vacuum detection now includes estimated_runtime_seconds in job
   proposals, computed as 5 min/GB of volume size.

2. The scheduler checks for estimated_runtime_seconds in job
   parameters and uses it as the execution timeout when larger than
   the default — a generic mechanism any handler can use.

3. Vacuum task gRPC calls now use the passed-in ctx as parent
   instead of context.Background(), so scheduler cancellation
   propagates to in-flight RPCs.

* extend job type runtime when proposals need more time

The JobTypeMaxRuntime (default 30 min) wraps both detection and
execution. Its context is the parent of all per-job execution
contexts, so even with per-job estimated_runtime_seconds, jobCtx
would cancel everything when it expires.

After detection, scan proposals for the maximum
estimated_runtime_seconds. If any proposal needs more time than
the remaining JobTypeMaxRuntime, create a new execution context
with enough headroom. This lets large vacuum jobs complete without
being killed by the job type deadline while still respecting the
configured limit for normal-sized jobs.

* log missing volume size metric, remove dead minimum runtime guard

Add a debug log in vacuumTimeout when t.volumeSize is 0 so
operators can investigate why metrics are missing for a volume.

Remove the unreachable estimatedRuntimeSeconds < 180 check in
buildVacuumProposal — volumeSizeGB always >= 1 (due to +1 floor),
so estimatedRuntimeSeconds is always >= 300.

* cap estimated runtime and fix status check context

- Cap maxEstimatedRuntime and per-job timeout overrides to 8 hours
  to prevent unbounded timeouts from bad metrics.
- Check execCtx.Err() instead of jobCtx.Err() for status reporting,
  since dispatch runs under execCtx which may have a longer deadline.
  A successful dispatch under execCtx was misreported as "timeout"
  when jobCtx had expired.
2026-03-10 13:48:42 -07:00
Chris LuandGitHub 4c88fbfd5e Fix nil pointer crash during concurrent vacuum compaction (#8592)
* check for nil needle map before compaction sync

When CommitCompact runs concurrently, it sets v.nm = nil under
dataFileAccessLock. CompactByIndex does not hold that lock, so
v.nm.Sync() can hit a nil pointer. Add an early nil check to
return an error instead of crashing.

Fixes #8591

* guard copyDataBasedOnIndexFile size check against nil needle map

The post-compaction size validation at line 538 accesses
v.nm.ContentSize() and v.nm.DeletedSize(). If CommitCompact has
concurrently set v.nm to nil, this causes a SIGSEGV. Skip the
validation when v.nm is nil since the actual data copy uses local
needle maps (oldNm/newNm) and is unaffected.

Fixes #8591

* use atomic.Bool for compaction flags to prevent concurrent vacuum races

The isCompacting and isCommitCompacting flags were plain bools
read and written from multiple goroutines without synchronization.
This allowed concurrent vacuums on the same volume to pass the
guard checks and run simultaneously, leading to the nil pointer
crash. Using atomic.Bool with CompareAndSwap ensures only one
compaction or commit can run per volume at a time.

Fixes #8591

* use go-version-file in CI workflows instead of hardcoded versions

Use go-version-file: 'go.mod' so CI automatically picks up the Go
version from go.mod, avoiding future version drift. Reordered
checkout before setup-go in go.yml and e2e.yml so go.mod is
available. Removed the now-unused GO_VERSION env vars.

* capture v.nm locally in CompactByIndex to close TOCTOU race

A bare nil check on v.nm followed by v.nm.Sync() has a race window
where CommitCompact can set v.nm = nil between the two. Snapshot
the pointer into a local variable so the nil check and Sync operate
on the same reference.

* add dynamic timeouts to plugin worker vacuum gRPC calls

All vacuum gRPC calls used context.Background() with no deadline,
so the plugin scheduler's execution timeout could kill a job while
a large volume compact was still in progress. Use volume-size-scaled
timeouts matching the topology vacuum approach: 3 min/GB for compact,
1 min/GB for check, commit, and cleanup.

Fixes #8591

* Revert "add dynamic timeouts to plugin worker vacuum gRPC calls"

This reverts commit 80951934c3.

* unify compaction lifecycle into single atomic flag

Replace separate isCompacting and isCommitCompacting flags with a
single isCompactionInProgress atomic.Bool. This ensures CompactBy*,
CommitCompact, Close, and Destroy are mutually exclusive — only one
can run at a time per volume.

Key changes:
- All entry points use CompareAndSwap(false, true) to claim exclusive
  access. CompactByVolumeData and CompactByIndex now also guard v.nm
  and v.DataBackend with local captures.
- Close() waits for the flag outside dataFileAccessLock to avoid
  deadlocking with CommitCompact (which holds the flag while waiting
  for the lock). It claims the flag before acquiring the lock so no
  new compaction can start.
- Destroy() uses CAS instead of a racy Load check, preventing
  concurrent compaction from racing with volume teardown.
- unmountVolumeByCollection no longer deletes from the map;
  DeleteCollectionFromDiskLocation removes entries only after
  successful Destroy, preventing orphaned volumes on failure.

Fixes #8591
2026-03-10 13:31:45 -07:00
Chris Lu d4d2e511ed for mini, default to bind all 2026-03-10 00:56:40 -07:00
Chris Lu 3d9f7f6f81 go 1.25 2026-03-09 23:10:27 -07:00
Chris Lu d89a78d9e3 reduce logs 2026-03-09 22:42:03 -07:00
Chris Lu 00000ec006 Update s3_buckets_templ.go 2026-03-09 22:41:07 -07:00
Chris Lu 1bd7a98a4a simplify plugin scheduler: remove configurable IdleSleepSeconds, use constant 61s
The SchedulerConfig struct and its persistence/API were unnecessary
indirection. Replace with a simple constant (reduced from 613s to 61s)
so the scheduler re-checks for detectable job types promptly after
going idle, improving the clean-install experience.
2026-03-09 22:41:03 -07:00
Chris Lu 8ad58e7002 4.16 2026-03-09 21:52:43 -07:00
Chris Lu f220328ae4 test: assert ReadFileStatusCount in batch execution test
Verify that pre-delete verification called ReadVolumeFileStatus on
both source and target for each volume move.
2026-03-09 19:33:06 -07:00
Chris Lu cf3693651c fix: add IdxFileSize check to pre-delete volume verification
The verification step checked DatFileSize and FileCount but not
IdxFileSize, leaving a gap in the copy validation before source
deletion.
2026-03-09 19:33:02 -07:00
Chris LuandGitHub 5f85bf5e8a Batch volume balance: run multiple moves per job (#8561)
* proto: add BalanceMoveSpec and batch fields to BalanceTaskParams

Add BalanceMoveSpec message for encoding individual volume moves,
and max_concurrent_moves + repeated moves fields to BalanceTaskParams
to support batching multiple volume moves in a single job.

* balance handler: add batch execution with concurrent volume moves

Refactor Execute() into executeSingleMove() (backward compatible) and
executeBatchMoves() which runs multiple volume moves concurrently using
a semaphore-bounded goroutine pool. When BalanceTaskParams.Moves is
populated, the batch path is taken; otherwise the single-move path.

Includes aggregate progress reporting across concurrent moves,
per-move error collection, and partial failure support.

* balance handler: add batch config fields to Descriptor and worker config

Add max_concurrent_moves and batch_size fields to the worker config
form and deriveBalanceWorkerConfig(). These control how many volume
moves run concurrently within a batch job and the maximum batch size.

* balance handler: group detection proposals into batch jobs

When batch_size > 1, the Detect method groups detection results into
batch proposals where each proposal encodes multiple BalanceMoveSpec
entries in BalanceTaskParams.Moves. Single-result batches fall back
to the existing single-move proposal format for backward compatibility.

* admin UI: add volume balance execution plan and batch badge

Add renderBalanceExecutionPlan() for rich rendering of volume balance
jobs in the job detail modal. Single-move jobs show source/target/volume
info; batch jobs show a moves table with all volume moves.

Add batch badge (e.g., "5 moves") next to job type in the execution
jobs table when the job has batch=true label.

* Update plugin_templ.go

* fix: detection algorithm uses greedy target instead of divergent topology scores

The detection loop tracked effective volume counts via an adjustments map,
but createBalanceTask independently called planBalanceDestination which used
the topology's LoadCount — a separate, unadjusted source of truth. This
divergence caused multiple moves to pile onto the same server.

Changes:
- Add resolveBalanceDestination to resolve the detection loop's greedy
  target (minServer) rather than independently picking a destination
- Add oscillation guard: stop when max-min <= 1 since no single move
  can improve the balance beyond that point
- Track unseeded destinations: if a target server wasn't in the initial
  serverVolumeCounts, add it so subsequent iterations include it
- Add TestDetection_UnseededDestinationDoesNotOverload

* fix: handler force_move propagation, partial failure, deterministic dedupe

- Propagate ForceMove from outer BalanceTaskParams to individual move
  TaskParams so batch moves respect the force_move flag
- Fix partial failure: mark job successful if at least one move
  succeeded (succeeded > 0 || failed == 0) to avoid re-running
  already-completed moves on retry
- Use SHA-256 hash for deterministic dedupe key fallback instead of
  time.Now().UnixNano() which is non-deterministic
- Remove unused successDetails variable
- Extract maxProposalStringLength constant to replace magic number 200

* admin UI: use template literals in balance execution plan rendering

* fix: integration test handles batch proposals from batched detection

With batch_size=20, all moves are grouped into a single proposal
containing BalanceParams.Moves instead of top-level Sources/Targets.
Update assertions to handle both batch and single-move proposal formats.

* fix: verify volume size on target before deleting source during balance

Add a pre-delete safety check that reads the volume file status on both
source and target, then compares .dat file size and file count. If they
don't match, the move is aborted — leaving the source intact rather than
risking irreversible data loss.

Also removes the redundant mountVolume call since VolumeCopy already
mounts the volume on the target server.

* fix: clamp maxConcurrent, serialize progress sends, validate config as int64

- Clamp maxConcurrentMoves to defaultMaxConcurrentMoves before creating
  the semaphore so a stale or malicious job cannot request unbounded
  concurrent volume moves
- Extend progressMu to cover sender.SendProgress calls since the
  underlying gRPC stream is not safe for concurrent writes
- Perform bounds checks on max_concurrent_moves and batch_size in int64
  space before casting to int, avoiding potential overflow on 32-bit

* fix: check disk capacity in resolveBalanceDestination

Skip disks where VolumeCount >= MaxVolumeCount so the detection loop
does not propose moves to a full disk that would fail at execution time.

* test: rename unseeded destination test to match actual behavior

The test exercises a server with 0 volumes that IS seeded from topology
(matching disk type), not an unseeded destination. Rename to
TestDetection_ZeroVolumeServerIncludedInBalance and fix comments.

* test: tighten integration test to assert exactly one batch proposal

With default batch_size=20, all moves should be grouped into a single
batch proposal. Assert len(proposals)==1 and require BalanceParams with
Moves, removing the legacy single-move else branch.

* fix: propagate ctx to RPCs and restore source writability on abort

- All helper methods (markVolumeReadonly, copyVolume, tailVolume,
  readVolumeFileStatus, deleteVolume) now accept a context parameter
  instead of using context.Background(), so Execute's ctx propagates
  cancellation and timeouts into every volume server RPC
- Add deferred cleanup that restores the source volume to writable if
  any step after markVolumeReadonly fails, preventing the source from
  being left permanently readonly on abort
- Add markVolumeWritable helper using VolumeMarkWritableRequest

* fix: deep-copy protobuf messages in test recording sender

Use proto.Clone in recordingExecutionSender to store immutable snapshots
of JobProgressUpdate and JobCompleted, preventing assertions from
observing mutations if the handler reuses message pointers.

* fix: add VolumeMarkWritable and ReadVolumeFileStatus to fake volume server

The balance task now calls ReadVolumeFileStatus for pre-delete
verification and VolumeMarkWritable to restore writability on abort.
Add both RPCs to the test fake, and drop the mountCalls assertion since
BalanceTask no longer calls VolumeMount directly (VolumeCopy handles it).

* fix: use maxConcurrentMovesLimit (50) for clamp, not defaultMaxConcurrentMoves

defaultMaxConcurrentMoves (5) is the fallback when the field is unset,
not an upper bound. Clamping to it silently overrides valid config
values like 10/20/50. Introduce maxConcurrentMovesLimit (50) matching
the descriptor's MaxValue and clamp to that instead.

* fix: cancel batch moves on progress stream failure

Derive a cancellable batchCtx from the caller's ctx. If
sender.SendProgress returns an error (client disconnect, context
cancelled), capture it, skip further sends, and cancel batchCtx so
in-flight moves abort via their propagated context rather than running
blind to completion.

* fix: bound cleanup timeout and validate batch move fields

- Use a 30-second timeout for the deferred markVolumeWritable cleanup
  instead of context.Background() which can block indefinitely if the
  volume server is unreachable
- Validate required fields (VolumeID, SourceNode, TargetNode) before
  appending moves to a batch proposal, skipping invalid entries
- Fall back to a single-move proposal when filtering leaves only one
  valid move in a batch

* fix: cancel task execution on SendProgress stream failure

All handler progress callbacks previously ignored SendProgress errors,
allowing tasks to continue executing after the client disconnected.
Now each handler creates a derived cancellable context and cancels it
on the first SendProgress error, stopping the in-flight task promptly.

Handlers fixed: erasure_coding, vacuum, volume_balance (single-move),
and admin_script (breaks command loop on send failure).

* fix: validate batch moves before scheduling in executeBatchMoves

Reject empty batches, enforce a hard upper bound (100 moves), and
filter out nil or incomplete move specs (missing source/target/volume)
before allocating progress tracking and launching goroutines.

* test: add batch balance execution integration test

Tests the batch move path with 3 volumes, max concurrency 2, using
fake volume servers. Verifies all moves complete with correct readonly,
copy, tail, and delete RPC counts.

* test: add MarkWritableCount and ReadFileStatusCount accessors

Expose the markWritableCalls and readFileStatusCalls counters on the
fake volume server, following the existing MarkReadonlyCount pattern.

* fix: oscillation guard uses global effective counts for heterogeneous capacity

The oscillation guard (max-min <= 1) previously used maxServer/minServer
which are determined by utilization ratio. With heterogeneous capacity,
maxServer by utilization can have fewer raw volumes than minServer,
producing a negative diff and incorrectly triggering the guard.

Now scans all servers' effective counts to find the true global max/min
volume counts, so the guard works correctly regardless of whether
utilization-based or raw-count balancing is used.

* fix: admin script handler breaks outer loop on SendProgress failure

The break on SendProgress error inside the shell.Commands scan only
exited the inner loop, letting the outer command loop continue
executing commands on a broken stream. Use a sendBroken flag to
propagate the break to the outer execCommands loop.
2026-03-09 19:30:08 -07:00
Chris LuandGitHub b991acf634 fix: paginate bucket listing in Admin UI to show all buckets (#8585)
* fix: paginate bucket listing in Admin UI to show all buckets

The Admin UI's GetS3Buckets() had a hardcoded Limit of 1000 in the
ListEntries request, causing the Total Buckets count to cap at 1000
even when more buckets exist. This adds pagination to iterate through
all buckets by continuing from the last entry name when a full page
is returned.

Fixes seaweedfs/seaweedfs#8564

* feat: add server-side pagination and sorting to S3 buckets page

Add pagination controls, page size selector, and sortable column
headers to the Admin UI's Object Store buckets page, following the
same pattern used by the Cluster Volumes page. This ensures the UI
remains responsive with thousands of buckets.

- Add CurrentPage, TotalPages, PageSize, SortBy, SortOrder to S3BucketsData
- Accept page/pageSize/sortBy/sortOrder query params in ShowS3Buckets handler
- Sort buckets by name, owner, created, objects, logical/physical size
- Paginate results server-side (default 100 per page)
- Add pagination nav, page size dropdown, and sort indicators to template

* Update s3_buckets_templ.go

* Update object_store_users_templ.go

* fix: use errors.Is(err, io.EOF) instead of string comparison

Replace brittle err.Error() == "EOF" string comparison with idiomatic
errors.Is(err, io.EOF) for checking stream end in bucket listing.

* fix: address PR review findings for bucket pagination

- Clamp page to totalPages when page exceeds total, preventing empty
  results with misleading pagination state
- Fix sort comparator to use explicit ascending/descending comparisons
  with a name tie-breaker, satisfying strict weak ordering for sort.Slice
- Capture SnapshotTsNs from first ListEntries response and pass it to
  subsequent requests for consistent pagination across pages
- Replace non-focusable <th onclick> sort headers with <a> tags and
  reuse getSortIcon, matching the cluster_volumes accessibility pattern
- Change exportBucketList() to fetch all buckets from /api/s3/buckets
  instead of scraping DOM rows (which now only contain the current page)
2026-03-09 18:55:47 -07:00
Chris Lu 02d3e3195c Update object_store_users_templ.go 2026-03-09 18:34:58 -07:00
Chris LuandGitHub 470075dd90 admin/balance: fix Max Volumes display and balancer source selection (#8583)
* admin: fix Max Volumes column always showing 0

GetClusterVolumeServers() computed DiskCapacity from
diskInfo.MaxVolumeCount but never populated the MaxVolumes field
on the VolumeServer struct, causing the column to always display 0.

* balance: use utilization ratio for source server selection

The balancer selected the source server (to move volumes FROM) by raw
volume count. In clusters with heterogeneous MaxVolumeCount settings,
the server with the highest capacity naturally holds the most volumes
and was always picked as the source, even when it had the lowest
utilization ratio.

Change source selection and imbalance calculation to use utilization
ratio (effectiveCount / maxVolumeCount) so servers are compared by how
full they are relative to their capacity, not by absolute volume count.

This matches how destination scoring already works via
calculateBalanceScore().
2026-03-09 18:34:11 -07:00
f8b7357350 weed/server: fix dropped error (#8584)
* weed/server: fix dropped error

* Removed the redundant check.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-03-09 18:04:12 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Chris Lu
e1c4faba38 build(deps): bump org.apache.zookeeper:zookeeper from 3.9.4 to 3.9.5 in /test/java/spark (#8580)
* build(deps): bump org.apache.zookeeper:zookeeper in /test/java/spark

Bumps org.apache.zookeeper:zookeeper from 3.9.4 to 3.9.5.

---
updated-dependencies:
- dependency-name: org.apache.zookeeper:zookeeper
  dependency-version: 3.9.5
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix: use go-version-file instead of hardcoded Go version in CI workflows

The hardcoded go-version '1.24' is too old for go.mod which requires
go >= 1.25.0, causing build failures in Spark integration tests.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-03-09 17:29:53 -07:00
Chris LuandGitHub 6c7fe87a72 helm: add s3.tlsSecret for custom S3 HTTPS certificate (#8582)
* helm: add s3.tlsSecret to allow custom TLS certificate for S3 HTTPS endpoint

Allow users to specify an external Kubernetes TLS secret for the S3
HTTPS endpoint instead of using the internal self-signed client
certificate. This enables using publicly trusted certificates (e.g.
from Let's Encrypt) so S3 clients don't need to trust the internal CA.

The new s3.tlsSecret value is supported in the standalone S3 gateway,
filer with embedded S3, and all-in-one deployment templates.

Closes #8581

* refactor: extract S3 TLS helpers to reduce duplication

Move repeated S3 TLS cert/key logic into shared helper templates
(seaweedfs.s3.tlsArgs, seaweedfs.s3.tlsVolumeMount, seaweedfs.s3.tlsVolume)
in _helpers.tpl, and use them across all three deployment templates.

* helm: add allInOne.s3.trafficDistribution support

Add the missing allInOne.s3.trafficDistribution branch to the
seaweedfs.trafficDistribution helper and wire it into the all-in-one
service template, mirroring the existing s3-service.yaml behavior.
PreferClose is auto-converted to PreferSameZone on k8s >=1.35.

* fix: scope S3 TLS mounts to S3-enabled pods and simplify trafficDistribution helper

- Wrap S3 TLS volume/volumeMount includes in allInOne.s3.enabled and
  filer.s3.enabled guards so the custom TLS secret is only mounted
  when S3 is actually enabled in that deployment mode.
- Refactor seaweedfs.trafficDistribution helper to accept an explicit
  value+Capabilities dict instead of walking multiple .Values paths,
  making each call site responsible for passing its own setting.
2026-03-09 14:24:42 -07:00
Chris Lu b3d32fe73b fix go version 2026-03-09 14:18:46 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Chris Lu
f439c84d01 build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.41.1 to 1.41.3 (#8576)
Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.41.1 to 1.41.3.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.41.1...v1.41.3)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-version: 1.41.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-03-09 14:14:29 -07:00
Chris Lu 89f1096c0e Update ec-integration.yml 2026-03-09 14:05:39 -07:00
Chris LuandGitHub 6dab90472b admin: fix access key creation UX (#8579)
* admin: remove misleading "secret key only shown once" warning

The access key details modal already allows viewing both the access key
and secret key at any time, so the warning about the secret key only
being displayed once is incorrect and misleading.

* admin: allow specifying custom access key and secret key

Add optional access_key and secret_key fields to the create access key
API. When provided, the specified keys are used instead of generating
random ones. The UI now shows a form with optional fields when creating
a new key, with a note that leaving them blank auto-generates keys.

* admin: check access key uniqueness before creating

Access keys must be globally unique across all users since S3 auth
looks them up in a single global map. Add an explicit check using
GetUserByAccessKey before creating, so the user gets a clear error
("access key is already in use") rather than a generic store error.

* Update object_store_users_templ.go

* admin: address review feedback for access key creation

Handler:
- Use decodeJSONBody/newJSONMaxReader instead of raw json.Decode to
  enforce request size limits and handle malformed JSON properly
- Return 409 Conflict for duplicate access keys, 400 Bad Request for
  validation errors, instead of generic 500

Backend:
- Validate access key length (4-128 chars) and secret key length
  (8-128 chars) when user-provided

Frontend:
- Extract resetCreateKeyForm() helper to avoid duplicated cleanup logic
- Wire resetCreateKeyForm to accessKeysModal hidden.bs.modal event so
  form state is always cleared when modal is dismissed
- Change secret key input to type="password" with a visibility toggle

* admin: guard against nil request and handle GetUserByAccessKey errors

- Add nil check for the CreateAccessKeyRequest pointer before
  dereferencing, defaulting to an empty request (auto-generate both
  keys).
- Handle non-"not found" errors from GetUserByAccessKey explicitly
  instead of silently proceeding, so store errors (e.g. db connection
  failures) surface rather than being swallowed.

* Update object_store_users_templ.go

* admin: fix access key uniqueness check with gRPC store

GetUserByAccessKey returns a gRPC NotFound status error (not the
sentinel credential.ErrAccessKeyNotFound) when using the gRPC store,
causing the uniqueness check to fail with a spurious error.

Treat the lookup as best-effort: only reject when a user is found
(err == nil). Any error (not-found via any store, connectivity issues)
falls through to the store's own CreateAccessKey which enforces
uniqueness definitively.

* admin: fix error handling and input validation for access key creation

Backend:
- Remove access key value from the duplicate-key error message to avoid
  logging the caller-supplied identifier.

Handler:
- Handle empty POST body (io.EOF) as a valid request that auto-generates
  both keys, instead of rejecting it as malformed JSON.
- Return 404 for "not found" errors (e.g. non-existent user) instead of
  collapsing them into a 500.

Frontend:
- Add minlength/maxlength attributes matching backend constraints
  (access key 4-128, secret key 8-128).
- Call reportValidity() before submitting so invalid lengths are caught
  client-side without a round trip.

* admin: use sentinel errors and fix GetUserByAccessKey error handling

Backend (user_management.go):
- Define sentinel errors (ErrAccessKeyInUse, ErrUserNotFound,
  ErrInvalidInput) and wrap them in returned errors so callers can use
  errors.Is.
- Handle GetUserByAccessKey errors properly: check the sentinel
  credential.ErrAccessKeyNotFound first, then fall back to string
  matching for stores (gRPC) that return non-sentinel not-found errors.
  Surface unexpected errors instead of silently proceeding.

Handler (user_handlers.go):
- Replace fragile strings.Contains error matching with errors.Is
  against the new dash sentinels.

Frontend (object_store_users.templ):
- Add double-submit guard (isCreatingKey flag + button disabling) to
  prevent duplicate access key creation requests.
2026-03-09 14:03:41 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Chris Lu
a00d38d8d4 build(deps): bump go.mongodb.org/mongo-driver from 1.17.6 to 1.17.9 (#8575)
Bumps [go.mongodb.org/mongo-driver](https://github.com/mongodb/mongo-go-driver) from 1.17.6 to 1.17.9.
- [Release notes](https://github.com/mongodb/mongo-go-driver/releases)
- [Commits](https://github.com/mongodb/mongo-go-driver/compare/v1.17.6...v1.17.9)

---
updated-dependencies:
- dependency-name: go.mongodb.org/mongo-driver
  dependency-version: 1.17.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-03-09 13:11:58 -07:00
Chris LuandGitHub f8d783f80e fix: ListObjectVersions interleave Version and DeleteMarker in sort order (#8567)
* fix: ListObjectVersions interleave Version and DeleteMarker in sort order

Go's default xml.Marshal serializes struct fields in definition order,
causing all <Version> elements to appear before all <DeleteMarker>
elements. The S3 API contract requires these elements to be interleaved
in the correct global sort order (by key ascending, then newest version
first within each key).

This broke clients that validate version list ordering within a single
key — an older Version would appear before a newer DeleteMarker for the
same object.

Fix: Replace the separate Versions/DeleteMarkers/CommonPrefixes arrays
with a single Entries []VersionListEntry slice. Each VersionListEntry
uses a per-element MarshalXML that outputs the correct XML tag name
(<Version>, <DeleteMarker>, or <CommonPrefixes>) based on which field
is populated. Since the entries are already in their correct sorted
order from buildSortedCombinedList, the XML output is automatically
interleaved correctly.

Also removes the unused ListObjectVersionsResult struct.

Note: The reporter also mentioned a cross-key timestamp ordering issue
when paginating with max-keys=1, but that is correct S3 behavior —
ListObjectVersions sorts by key name (ascending), not by timestamp.
Different keys having non-monotonic timestamps is expected.

* test: add CommonPrefixes XML marshaling coverage for ListObjectVersions

* fix: validate VersionListEntry has exactly one field set in MarshalXML

Return an error instead of silently emitting an empty <Version> element
when no field (or multiple fields) are populated. Also clean up the
misleading xml:"Version" struct tag on the Entries field.
2026-03-09 12:37:59 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Chris Lu
120d38176f build(deps): bump golang.org/x/sys from 0.41.0 to 0.42.0 (#8573)
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.41.0 to 0.42.0.
- [Commits](https://github.com/golang/sys/compare/v0.41.0...v0.42.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-version: 0.42.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-03-09 12:24:28 -07:00
Chris Lu 55bce53953 reduce logs 2026-03-09 12:14:25 -07:00
Chris LuandGitHub 992db11d2b iam: add IAM group management (#8560)
* iam: add Group message to protobuf schema

Add Group message (name, members, policy_names, disabled) and
add groups field to S3ApiConfiguration for IAM group management
support (issue #7742).

* iam: add group CRUD to CredentialStore interface and all backends

Add group management methods (CreateGroup, GetGroup, DeleteGroup,
ListGroups, UpdateGroup) to the CredentialStore interface with
implementations for memory, filer_etc, postgres, and grpc stores.
Wire group loading/saving into filer_etc LoadConfiguration and
SaveConfiguration.

* iam: add group IAM response types

Add XML response types for group management IAM actions:
CreateGroup, DeleteGroup, GetGroup, ListGroups, AddUserToGroup,
RemoveUserFromGroup, AttachGroupPolicy, DetachGroupPolicy,
ListAttachedGroupPolicies, ListGroupsForUser.

* iam: add group management handlers to embedded IAM API

Add CreateGroup, DeleteGroup, GetGroup, ListGroups, AddUserToGroup,
RemoveUserFromGroup, AttachGroupPolicy, DetachGroupPolicy,
ListAttachedGroupPolicies, and ListGroupsForUser handlers with
dispatch in ExecuteAction.

* iam: add group management handlers to standalone IAM API

Add group handlers (CreateGroup, DeleteGroup, GetGroup, ListGroups,
AddUserToGroup, RemoveUserFromGroup, AttachGroupPolicy, DetachGroupPolicy,
ListAttachedGroupPolicies, ListGroupsForUser) and wire into DoActions
dispatch. Also add helper functions for user/policy side effects.

* iam: integrate group policies into authorization

Add groups and userGroups reverse index to IdentityAccessManagement.
Populate both maps during ReplaceS3ApiConfiguration and
MergeS3ApiConfiguration. Modify evaluateIAMPolicies to evaluate
policies from user's enabled groups in addition to user policies.
Update VerifyActionPermission to consider group policies when
checking hasAttachedPolicies.

* iam: add group side effects on user deletion and rename

When a user is deleted, remove them from all groups they belong to.
When a user is renamed, update group membership references. Applied
to both embedded and standalone IAM handlers.

* iam: watch /etc/iam/groups directory for config changes

Add groups directory to the filer subscription watcher so group
file changes trigger IAM configuration reloads.

* admin: add group management page to admin UI

Add groups page with CRUD operations, member management, policy
attachment, and enable/disable toggle. Register routes in admin
handlers and add Groups entry to sidebar navigation.

* test: add IAM group management integration tests

Add comprehensive integration tests for group CRUD, membership,
policy attachment, policy enforcement, disabled group behavior,
user deletion side effects, and multi-group membership. Add
"group" test type to CI matrix in s3-iam-tests workflow.

* iam: address PR review comments for group management

- Fix XSS vulnerability in groups.templ: replace innerHTML string
  concatenation with DOM APIs (createElement/textContent) for rendering
  member and policy lists
- Use userGroups reverse index in embedded IAM ListGroupsForUser for
  O(1) lookup instead of iterating all groups
- Add buildUserGroupsIndex helper in standalone IAM handlers; use it
  in ListGroupsForUser and removeUserFromAllGroups for efficient lookup
- Add note about gRPC store load-modify-save race condition limitation

* iam: add defensive copies, validation, and XSS fixes for group management

- Memory store: clone groups on store/retrieve to prevent mutation
- Admin dash: deep copy groups before mutation, validate user/policy exists
- HTTP handlers: translate credential errors to proper HTTP status codes,
  use *bool for Enabled field to distinguish missing vs false
- Groups templ: use data attributes + event delegation instead of inline
  onclick for XSS safety, prevent stale async responses

* iam: add explicit group methods to PropagatingCredentialStore

Add CreateGroup, GetGroup, DeleteGroup, ListGroups, and UpdateGroup
methods instead of relying on embedded interface fallthrough. Group
changes propagate via filer subscription so no RPC propagation needed.

* iam: detect postgres unique constraint violation and add groups index

Return ErrGroupAlreadyExists when INSERT hits SQLState 23505 instead of
a generic error. Add index on groups(disabled) for filtered queries.

* iam: add Marker field to group list response types

Add Marker string field to GetGroupResult, ListGroupsResult,
ListAttachedGroupPoliciesResult, and ListGroupsForUserResult to
match AWS IAM pagination response format.

* iam: check group attachment before policy deletion

Reject DeletePolicy if the policy is attached to any group, matching
AWS IAM behavior. Add PolicyArn to ListAttachedGroupPolicies response.

* iam: include group policies in IAM authorization

Merge policy names from user's enabled groups into the IAMIdentity
used for authorization, so group-attached policies are evaluated
alongside user-attached policies.

* iam: check for name collision before renaming user in UpdateUser

Scan identities and inline policies for newUserName before mutating,
returning EntityAlreadyExists if a collision is found. Reuse the
already-loaded policies instead of loading them again inside the loop.

* test: use t.Cleanup for bucket cleanup in group policy test

* iam: wrap ErrUserNotInGroup sentinel in RemoveGroupMember error

Wrap credential.ErrUserNotInGroup so errors.Is works in
groupErrorToHTTPStatus, returning proper 400 instead of 500.

* admin: regenerate groups_templ.go with XSS-safe data attributes

Regenerated from groups.templ which uses data-group-name attributes
instead of inline onclick with string interpolation.

* iam: add input validation and persist groups during migration

- Validate nil/empty group name in CreateGroup and UpdateGroup
- Save groups in migrateToMultiFile so they survive legacy migration

* admin: use groupErrorToHTTPStatus in GetGroupMembers and GetGroupPolicies

* iam: short-circuit UpdateUser when newUserName equals current name

* iam: require empty PolicyNames before group deletion

Reject DeleteGroup when group has attached policies, matching the
existing members check. Also fix GetGroup error handling in
DeletePolicy to only skip ErrGroupNotFound, not all errors.

* ci: add weed/pb/** to S3 IAM test trigger paths

* test: replace time.Sleep with require.Eventually for propagation waits

Use polling with timeout instead of fixed sleeps to reduce flakiness
in integration tests waiting for IAM policy propagation.

* fix: use credentialManager.GetPolicy for AttachGroupPolicy validation

Policies created via CreatePolicy through credentialManager are stored
in the credential store, not in s3cfg.Policies (which only has static
config policies). Change AttachGroupPolicy to use credentialManager.GetPolicy()
for policy existence validation.

* feat: add UpdateGroup handler to embedded IAM API

Add UpdateGroup action to enable/disable groups and rename groups
via the IAM API. This is a SeaweedFS extension (not in AWS SDK) used
by tests to toggle group disabled status.

* fix: authenticate raw IAM API calls in group tests

The embedded IAM endpoint rejects anonymous requests. Replace
callIAMAPI with callIAMAPIAuthenticated that uses JWT bearer token
authentication via the test framework.

* feat: add UpdateGroup handler to standalone IAM API

Mirror the embedded IAM UpdateGroup handler in the standalone IAM API
for parity.

* fix: add omitempty to Marker XML tags in group responses

Non-truncated responses should not emit an empty <Marker/> element.

* fix: distinguish backend errors from missing policies in AttachGroupPolicy

Return ServiceFailure for credential manager errors instead of masking
them as NoSuchEntity. Also switch ListGroupsForUser to use s3cfg.Groups
instead of in-memory reverse index to avoid stale data. Add duplicate
name check to UpdateGroup rename.

* fix: standalone IAM AttachGroupPolicy uses persisted policy store

Check managed policies from GetPolicies() instead of s3cfg.Policies
so dynamically created policies are found. Also add duplicate name
check to UpdateGroup rename.

* fix: rollback inline policies on UpdateUser PutPolicies failure

If PutPolicies fails after moving inline policies to the new username,
restore both the identity name and the inline policies map to their
original state to avoid a partial-write window.

* fix: correct test cleanup ordering for group tests

Replace scattered defers with single ordered t.Cleanup in each test
to ensure resources are torn down in reverse-creation order:
remove membership, detach policies, delete access keys, delete users,
delete groups, delete policies. Move bucket cleanup to parent test
scope and delete objects before bucket.

* fix: move identity nil check before map lookup and refine hasAttachedPolicies

Move the nil check on identity before accessing identity.Name to
prevent panic. Also refine hasAttachedPolicies to only consider groups
that are enabled and have actual policies attached, so membership in
a no-policy group doesn't incorrectly trigger IAM authorization.

* fix: fail group reload on unreadable or corrupt group files

Return errors instead of logging and continuing when group files
cannot be read or unmarshaled. This prevents silently applying a
partial IAM config with missing group memberships or policies.

* fix: use errors.Is for sql.ErrNoRows comparison in postgres group store

* docs: explain why group methods skip propagateChange

Group changes propagate to S3 servers via filer subscription
(watching /etc/iam/groups/) rather than gRPC RPCs, since there
are no group-specific RPCs in the S3 cache protocol.

* fix: remove unused policyNameFromArn and strings import

* fix: update service account ParentUser on user rename

When renaming a user via UpdateUser, also update ParentUser references
in service accounts to prevent them from becoming orphaned after the
next configuration reload.

* fix: wrap DetachGroupPolicy error with ErrPolicyNotAttached sentinel

Use credential.ErrPolicyNotAttached so groupErrorToHTTPStatus maps
it to 400 instead of falling back to 500.

* fix: use admin S3 client for bucket cleanup in enforcement test

The user S3 client may lack permissions by cleanup time since the
user is removed from the group in an earlier subtest. Use the admin
S3 client to ensure bucket and object cleanup always succeeds.

* fix: add nil guard for group param in propagating store log calls

Prevent potential nil dereference when logging group.Name in
CreateGroup and UpdateGroup of PropagatingCredentialStore.

* fix: validate Disabled field in UpdateGroup handlers

Reject values other than "true" or "false" with InvalidInputException
instead of silently treating them as false.

* fix: seed mergedGroups from existing groups in MergeS3ApiConfiguration

Previously the merge started with empty group maps, dropping any
static-file groups. Now seeds from existing iam.groups before
overlaying dynamic config, and builds the reverse index after
merging to avoid stale entries from overridden groups.

* fix: use errors.Is for filer_pb.ErrNotFound comparison in group loading

Replace direct equality (==) with errors.Is() to correctly match
wrapped errors, consistent with the rest of the codebase.

* fix: add ErrUserNotFound and ErrPolicyNotFound to groupErrorToHTTPStatus

Map these sentinel errors to 404 so AddGroupMember and
AttachGroupPolicy return proper HTTP status codes.

* fix: log cleanup errors in group integration tests

Replace fire-and-forget cleanup calls with error-checked versions
that log failures via t.Logf for debugging visibility.

* fix: prevent duplicate group test runs in CI matrix

The basic lane's -run "TestIAM" regex also matched TestIAMGroup*
tests, causing them to run in both the basic and group lanes.
Replace with explicit test function names.

* fix: add GIN index on groups.members JSONB for membership lookups

Without this index, ListGroupsForUser and membership queries
require full table scans on the groups table.

* fix: handle cross-directory moves in IAM config subscription

When a file is moved out of an IAM directory (e.g., /etc/iam/groups),
the dir variable was overwritten with NewParentPath, causing the
source directory change to be missed. Now also notifies handlers
about the source directory for cross-directory moves.

* fix: validate members/policies before deleting group in admin handler

AdminServer.DeleteGroup now checks for attached members and policies
before delegating to credentialManager, matching the IAM handler guards.

* fix: merge groups by name instead of blind append during filer load

Match the identity loader's merge behavior: find existing group
by name and replace, only append when no match exists. Prevents
duplicates when legacy and multi-file configs overlap.

* fix: check DeleteEntry response error when cleaning obsolete group files

Capture and log resp.Error from filer DeleteEntry calls during
group file cleanup, matching the pattern used in deleteGroupFile.

* fix: verify source user exists before no-op check in UpdateUser

Reorder UpdateUser to find the source identity first and return
NoSuchEntityException if not found, before checking if the rename
is a no-op. Previously a non-existent user renamed to itself
would incorrectly return success.

* fix: update service account parent refs on user rename in embedded IAM

The embedded IAM UpdateUser handler updated group membership but
not service account ParentUser fields, unlike the standalone handler.

* fix: replay source-side events for all handlers on cross-dir moves

Pass nil newEntry to bucket, IAM, and circuit-breaker handlers for
the source directory during cross-directory moves, so all watchers
can clear caches for the moved-away resource.

* fix: don't seed mergedGroups from existing iam.groups in merge

Groups are always dynamic (from filer), never static (from s3.config).
Seeding from iam.groups caused stale deleted groups to persist.
Now only uses config.Groups from the dynamic filer config.

* fix: add deferred user cleanup in TestIAMGroupUserDeletionSideEffect

Register t.Cleanup for the created user so it gets cleaned up
even if the test fails before the inline DeleteUser call.

* fix: assert UpdateGroup HTTP status in disabled group tests

Add require.Equal checks for 200 status after UpdateGroup calls
so the test fails immediately on API errors rather than relying
on the subsequent Eventually timeout.

* fix: trim whitespace from group name in filer store operations

Trim leading/trailing whitespace from group.Name before validation
in CreateGroup and UpdateGroup to prevent whitespace-only filenames.
Also merge groups by name during multi-file load to prevent duplicates.

* fix: add nil/empty group validation in gRPC store

Guard CreateGroup and UpdateGroup against nil group or empty name
to prevent panics and invalid persistence.

* fix: add nil/empty group validation in postgres store

Guard CreateGroup and UpdateGroup against nil group or empty name
to prevent panics from nil member access and empty-name row inserts.

* fix: add name collision check in embedded IAM UpdateUser

The embedded IAM handler renamed users without checking if the
target name already existed, unlike the standalone handler.

* fix: add ErrGroupNotEmpty sentinel and map to HTTP 409

AdminServer.DeleteGroup now wraps conflict errors with
ErrGroupNotEmpty, and groupErrorToHTTPStatus maps it to
409 Conflict instead of 500.

* fix: use appropriate error message in GetGroupDetails based on status

Return "Group not found" only for 404, use "Failed to retrieve group"
for other error statuses instead of always saying "Group not found".

* fix: use backend-normalized group.Name in CreateGroup response

After credentialManager.CreateGroup may normalize the name (e.g.,
trim whitespace), use group.Name instead of the raw input for
the returned GroupData to ensure consistency.

* fix: add nil/empty group validation in memory store

Guard CreateGroup and UpdateGroup against nil group or empty name
to prevent panics from nil pointer dereference on map access.

* fix: reorder embedded IAM UpdateUser to verify source first

Find the source identity before checking for collisions, matching
the standalone handler's logic. Previously a non-existent user
renamed to an existing name would get EntityAlreadyExists instead
of NoSuchEntity.

* fix: handle same-directory renames in metadata subscription

Replay a delete event for the old entry name during same-directory
renames so handlers like onBucketMetadataChange can clean up stale
state for the old name.

* fix: abort GetGroups on non-ErrGroupNotFound errors

Only skip groups that return ErrGroupNotFound. Other errors (e.g.,
transient backend failures) now abort the handler and return the
error to the caller instead of silently producing partial results.

* fix: add aria-label and title to icon-only group action buttons

Add accessible labels to View and Delete buttons so screen readers
and tooltips provide meaningful context.

* fix: validate group name in saveGroup to prevent invalid filenames

Trim whitespace and reject empty names before writing group JSON
files, preventing creation of files like ".json".

* fix: add /etc/iam/groups to filer subscription watched directories

The groups directory was missing from the watched directories list,
so S3 servers in a cluster would not detect group changes made by
other servers via filer. The onIamConfigChange handler already had
code to handle group directory changes but it was never triggered.

* add direct gRPC propagation for group changes to S3 servers

Groups now have the same dual propagation as identities and policies:
direct gRPC push via propagateChange + async filer subscription.

- Add PutGroup/RemoveGroup proto messages and RPCs
- Add PutGroup/RemoveGroup in-memory cache methods on IAM
- Add PutGroup/RemoveGroup gRPC server handlers
- Update PropagatingCredentialStore to call propagateChange on group mutations

* reduce log verbosity for config load summary

Change ReplaceS3ApiConfiguration log from Infof to V(1).Infof
to avoid noisy output on every config reload.

* admin: show user groups in view and edit user modals

- Add Groups field to UserDetails and populate from credential manager
- Show groups as badges in user details view modal
- Add group management to edit user modal: display current groups,
  add to group via dropdown, remove from group via badge x button

* fix: remove duplicate showAlert that broke modal-alerts.js

admin.js defined showAlert(type, message) which overwrote the
modal-alerts.js version showAlert(message, type), causing broken
unstyled alert boxes. Remove the duplicate and swap all callers
in admin.js to use the correct (message, type) argument order.

* fix: unwrap groups API response in edit user modal

The /api/groups endpoint returns {"groups": [...]}, not a bare array.

* Update object_store_users_templ.go

* test: assert AccessDenied error code in group denial tests

Replace plain assert.Error checks with awserr.Error type assertion
and AccessDenied code verification, matching the pattern used in
other IAM integration tests.

* fix: propagate GetGroups errors in ShowGroups handler

getGroupsPageData was swallowing errors and returning an empty page
with 200 status. Now returns the error so ShowGroups can respond
with a proper error status.

* fix: reject AttachGroupPolicy when credential manager is nil

Previously skipped policy existence validation when credentialManager
was nil, allowing attachment of nonexistent policies. Now returns
a ServiceFailureException error.

* fix: preserve groups during partial MergeS3ApiConfiguration updates

UpsertIdentity calls MergeS3ApiConfiguration with a partial config
containing only the updated identity (nil Groups). This was wiping
all in-memory group state. Now only replaces groups when
config.Groups is non-nil (full config reload).

* fix: propagate errors from group lookup in GetObjectStoreUserDetails

ListGroups and GetGroup errors were silently ignored, potentially
showing incomplete group data in the UI.

* fix: use DOM APIs for group badge remove button to prevent XSS

Replace innerHTML with onclick string interpolation with DOM
createElement + addEventListener pattern. Also add aria-label
and title to the add-to-group button.

* fix: snapshot group policies under RLock to prevent concurrent map access

evaluateIAMPolicies was copying the map reference via groupMap :=
iam.groups under RLock then iterating after RUnlock, while PutGroup
mutates the map in-place. Now copies the needed policy names into
a slice while holding the lock.

* fix: add nil IAM check to PutGroup and RemoveGroup gRPC handlers

Match the nil guard pattern used by PutPolicy/DeletePolicy to
prevent nil pointer dereference when IAM is not initialized.
2026-03-09 11:54:32 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
115dcb5ada build(deps): bump github.com/prometheus/procfs from 0.19.2 to 0.20.1 (#8578)
Bumps [github.com/prometheus/procfs](https://github.com/prometheus/procfs) from 0.19.2 to 0.20.1.
- [Release notes](https://github.com/prometheus/procfs/releases)
- [Commits](https://github.com/prometheus/procfs/compare/v0.19.2...v0.20.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/procfs
  dependency-version: 0.20.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-09 11:37:32 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
7be2d1ecfb build(deps): bump github.com/getsentry/sentry-go from 0.42.0 to 0.43.0 (#8577)
Bumps [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go) from 0.42.0 to 0.43.0.
- [Release notes](https://github.com/getsentry/sentry-go/releases)
- [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md)
- [Commits](https://github.com/getsentry/sentry-go/compare/v0.42.0...v0.43.0)

---
updated-dependencies:
- dependency-name: github.com/getsentry/sentry-go
  dependency-version: 0.43.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-09 11:37:24 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1272612bbd build(deps): bump docker/setup-qemu-action from 3 to 4 (#8574)
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 3 to 4.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-09 11:33:38 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
e568d85a5c build(deps): bump docker/build-push-action from 6 to 7 (#8572)
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6 to 7.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v6...v7)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-09 11:33:17 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
f79ba1eb37 build(deps): bump docker/login-action from 3 to 4 (#8569)
Bumps [docker/login-action](https://github.com/docker/login-action) from 3 to 4.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-09 11:13:06 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
b132232895 build(deps): bump docker/setup-buildx-action from 3 to 4 (#8570)
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3 to 4.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-09 11:12:56 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
d765ff50e6 build(deps): bump actions/dependency-review-action from 4.8.3 to 4.9.0 (#8571)
Bumps [actions/dependency-review-action](https://github.com/actions/dependency-review-action) from 4.8.3 to 4.9.0.
- [Release notes](https://github.com/actions/dependency-review-action/releases)
- [Commits](https://github.com/actions/dependency-review-action/compare/05fe4576374b728f0c523d6a13d64c25081e0803...2031cfc080254a8a887f58cffee85186f0e49e48)

---
updated-dependencies:
- dependency-name: actions/dependency-review-action
  dependency-version: 4.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-09 11:12:40 -07:00
Chris Lu bff084ff6a update go version 2026-03-09 11:12:05 -07:00
Chris LuandGitHub 78a3441b30 fix: volume balance detection returns multiple tasks per run (#8559)
* fix: volume balance detection now returns multiple tasks per run (#8551)

Previously, detectForDiskType() returned at most 1 balance task per disk
type, making the MaxJobsPerDetection setting ineffective. The detection
loop now iterates within each disk type, planning multiple moves until
the imbalance drops below threshold or maxResults is reached. Effective
volume counts are adjusted after each planned move so the algorithm
correctly re-evaluates which server is overloaded.

* fix: factor pending tasks into destination scoring and use UnixNano for task IDs

- Use UnixNano instead of Unix for task IDs to avoid collisions when
  multiple tasks are created within the same second
- Adjust calculateBalanceScore to include LoadCount (pending + assigned
  tasks) in the utilization estimate, so the destination picker avoids
  stacking multiple planned moves onto the same target disk

* test: add comprehensive balance detection tests for complex scenarios

Cover multi-server convergence, max-server shifting, destination
spreading, pre-existing pending task skipping, no-duplicate-volume
invariant, and parameterized convergence verification across different
cluster shapes and thresholds.

* fix: address PR review findings in balance detection

- hasMore flag: compute from len(results) >= maxResults so the scheduler
  knows more pages may exist, matching vacuum/EC handler pattern
- Exhausted server fallthrough: when no eligible volumes remain on the
  current maxServer (all have pending tasks) or destination planning
  fails, mark the server as exhausted and continue to the next
  overloaded server instead of stopping the entire detection loop
- Return canonical destination server ID directly from createBalanceTask
  instead of resolving via findServerIDByAddress, eliminating the
  fragile address→ID lookup for adjustment tracking
- Fix bestScore sentinel: use math.Inf(-1) instead of -1.0 so disks
  with negative scores (high pending load, same rack/DC) are still
  selected as the best available destination
- Add TestDetection_ExhaustedServerFallsThrough covering the scenario
  where the top server's volumes are all blocked by pre-existing tasks

* test: fix computeEffectiveCounts and add len guard in no-duplicate test

- computeEffectiveCounts now takes a servers slice to seed counts for all
  known servers (including empty ones) and uses an address→ID map from
  the topology spec instead of scanning metrics, so destination servers
  with zero initial volumes are tracked correctly
- TestDetection_NoDuplicateVolumesAcrossIterations now asserts len > 1
  before checking duplicates, so the test actually fails if Detection
  regresses to returning a single task

* fix: remove redundant HasAnyTask check in createBalanceTask

The HasAnyTask check in createBalanceTask duplicated the same check
already performed in detectForDiskType's volume selection loop.
Since detection runs single-threaded (MaxDetectionConcurrency: 1),
no race can occur between the two points.

* fix: consistent hasMore pattern and remove double-counted LoadCount in scoring

- Adopt vacuum_handler's hasMore pattern: over-fetch by 1, check
  len > maxResults, and truncate — consistent truncation semantics
- Remove direct LoadCount penalty in calculateBalanceScore since
  LoadCount is already factored into effectiveVolumeCount for
  utilization scoring; bump utilization weight from 40 to 50 to
  compensate for the removed 10-point load penalty

* fix: handle zero maxResults as no-cap, emit trace after trim, seed empty servers

- When MaxResults is 0 (omitted), treat as no explicit cap instead of
  defaulting to 1; only apply the +1 over-fetch probe when caller
  supplies a positive limit
- Move decision trace emission after hasMore/trim so the trace
  accurately reflects the returned proposals
- Seed serverVolumeCounts from ActiveTopology so servers that have a
  matching disk type but zero volumes are included in the imbalance
  calculation and MinServerCount check

* fix: nil-guard clusterInfo, uncap legacy DetectionFunc, deterministic disk type order

- Add early nil guard for clusterInfo in Detection to prevent panics
  in downstream helpers (detectForDiskType, createBalanceTask)
- Change register.go DetectionFunc wrapper from maxResults=1 to 0
  (no cap) so the legacy code path returns all detected tasks
- Sort disk type keys before iteration so results are deterministic
  when maxResults spans multiple disk types (HDD/SSD)

* fix: don't over-fetch in stateful detection to avoid orphaned pending tasks

Detection registers planned moves in ActiveTopology via AddPendingTask,
so requesting maxResults+1 would create an extra pending task that gets
discarded during trim. Use len(results) >= maxResults as the hasMore
signal instead, which is correct since Detection already caps internally.

* fix: return explicit truncated flag from Detection instead of approximating

Detection now returns (results, truncated, error) where truncated is true
only when the loop stopped because it hit maxResults, not when it ran out
of work naturally. This eliminates false hasMore signals when detection
happens to produce exactly maxResults results by resolving the imbalance.

* cleanup: simplify detection logic and remove redundancies

- Remove redundant clusterInfo nil check in detectForDiskType since
  Detection already guards against nil clusterInfo
- Remove adjustments loop for destination servers not in
  serverVolumeCounts — topology seeding ensures all servers with
  matching disk type are already present
- Merge two-loop min/max calculation into a single loop: min across
  all servers, max only among non-exhausted servers
- Replace magic number 100 with len(metrics) for minC initialization
  in convergence test

* fix: accurate truncation flag, deterministic server order, indexed volume lookup

- Track balanced flag to distinguish "hit maxResults cap" from "cluster
  balanced at exactly maxResults" — truncated is only true when there's
  genuinely more work to do
- Sort servers for deterministic iteration and tie-breaking when
  multiple servers have equal volume counts
- Pre-index volumes by server with per-server cursors to avoid
  O(maxResults * volumes) rescanning on each iteration
- Add truncation flag assertions to RespectsMaxResults test: true when
  capped, false when detection finishes naturally

* fix: seed trace server counts from ActiveTopology to match detection logic

The decision trace was building serverVolumeCounts only from metrics,
missing zero-volume servers seeded from ActiveTopology by Detection.
This could cause the trace to report wrong server counts, incorrect
imbalance ratios, or spurious "too few servers" messages. Pass
activeTopology into the trace function and seed server counts the
same way Detection does.

* fix: don't exhaust server on per-volume planning failure, sort volumes by ID

- When createBalanceTask returns nil, continue to the next volume on
  the same server instead of marking the entire server as exhausted.
  The failure may be volume-specific (not found in topology, pending
  task registration failed) and other volumes on the server may still
  be viable candidates.
- Sort each server's volume slice by VolumeID after pre-indexing so
  volume selection is fully deterministic regardless of input order.

* fix: use require instead of assert to prevent nil dereference panic in CORS test

The test used assert.NoError (non-fatal) for GetBucketCors, then
immediately accessed getResp.CORSRules. When the API returns an error,
getResp is nil causing a panic. Switch to require.NoError/NotNil/Len
so the test stops before dereferencing a nil response.

* fix: deterministic disk tie-breaking and stronger pre-existing task test

- Sort available disks by NodeID then DiskID before scoring so
  destination selection is deterministic when two disks score equally
- Add task count bounds assertion to SkipsPreExistingPendingTasks test:
  with 15 of 20 volumes already having pending tasks, at most 5 new
  tasks should be created and at least 1 (imbalance still exists)

* fix: seed adjustments from existing pending/assigned tasks to prevent over-scheduling

Detection now calls ActiveTopology.GetTaskServerAdjustments() to
initialize the adjustments map with source/destination deltas from
existing pending and assigned balance tasks. This ensures
effectiveCounts reflects in-flight moves, preventing the algorithm
from planning additional moves in the same direction when prior
moves already address the imbalance.

Added GetTaskServerAdjustments(taskType) to ActiveTopology which
iterates pending and assigned tasks, decrementing source servers
and incrementing destination servers for the given task type.
2026-03-08 21:34:03 -07:00
Chris LuandGitHub 2ec0a67ee3 master: return 503/Unavailable during topology warmup after leader change (#8529)
* master: return 503/Unavailable during topology warmup after leader change

After a master restart or leader change, the topology is empty until
volume servers reconnect and send heartbeats. During this warmup window
(3 heartbeat intervals = 15 seconds), volume lookups that fail now
return 503 Service Unavailable (HTTP) or gRPC Unavailable instead of
404 Not Found, signaling clients to retry with other masters.

* master: skip warmup 503 on fresh start and single-master setups

- Check MaxVolumeId > 0 to distinguish restart from fresh start
  (MaxVolumeId is Raft-persisted, so 0 means no prior data)
- Check peer count > 1 so single-master deployments aren't affected
  (no point suggesting "retry with other masters" if there are none)

* master: address review feedback and block assigns during warmup

- Protect LastLeaderChangeTime with dedicated mutex (fix data race)
- Extract warmup multiplier as WarmupPulseMultiplier constant
- Derive Retry-After header from pulse config instead of hardcoding
- Only trigger warmup 503 for "not found" errors, not parse errors
- Return nil response (not partial) on gRPC Unavailable
- Add doc comments to IsWarmingUp, getter/setter, WarmupDuration
- Block volume assign requests (HTTP and gRPC) during warmup,
  since the topology is incomplete and assignments would be unreliable
- Skip warmup behavior for single-master setups (no peers to retry)

* master: apply warmup to all setups, skip only on fresh start

Single-master restarts still have an empty topology until heartbeats
arrive, so warmup protection should apply there too. The only case
to skip is a fresh cluster start (MaxVolumeId == 0), which already
has no volumes to look up.

- Remove GetMasterCount() > 1 guard from all warmup checks
- Remove now-unused GetMasterCount helper
- Update error messages to "topology is still loading" (not
  "retry with other masters" which doesn't apply to single-master)

* master: add client-side retry on Unavailable for lookup and assign

The server-side 503/Unavailable during warmup needs client cooperation.
Previously, LookupVolumeIds and Assign would immediately propagate the
error without retry.

Now both paths retry with exponential backoff (1s -> 1.5s -> ... up to
6s) when receiving Unavailable, respecting context cancellation. This
covers the warmup window where the master's topology is still loading
after a restart or leader change.

* master: seed warmup timestamp in legacy raft path at setup

The legacy raft path only set lastLeaderChangeTime inside the event
listener callback, which could fire after IsLeader() was already
observed as true in SetRaftServer. Seed the timestamp at setup time
(matching the hashicorp path) so IsWarmingUp() is active immediately.

* master: fix assign retry loop to cover full warmup window

The retry loop used waitTime <= maxWaitTime as a stop condition,
causing it to give up after ~13s while warmup lasts 15s. Now cap
each individual sleep at maxWaitTime but keep retrying until the
context is cancelled.

* master: preserve gRPC status in lookup retry and fix retry window

Return the raw gRPC error instead of wrapping with fmt.Errorf so
status.FromError() can extract the status code. Use proper gRPC
status check (codes.Unavailable) instead of string matching. Also
cap individual sleep at maxWaitTime while retrying until ctx is done.

* master: use gRPC status code instead of string matching in assign retry

Use status.FromError/codes.Unavailable instead of brittle
strings.Contains for detecting retriable gRPC errors in the
assign retry loop.

* master: use remaining warmup duration for Retry-After header

Set Retry-After to the remaining warmup time instead of the full
warmup duration, so clients don't wait longer than necessary.

* master: reset ret.Replicas before populating from assign response

Clear Replicas slice before appending to prevent duplicate entries
when the assign response is retried or when alternative requests
are attempted.

* master: add unit tests for warmup retry behavior

Test that Assign() and LookupVolumeIds() retry on codes.Unavailable
and stop promptly when the context is cancelled.

* master: record leader change time before initialization work

Move SetLastLeaderChangeTime() to fire immediately when the leader
change event is received, before DoBarrier(), EnsureTopologyId(),
and updatePeers(), so the warmup clock starts at the true moment
of leadership transition.

* master: use topology warmup duration in volume growth wait loop

Replace hardcoded constants.VolumePulsePeriod * 2 with
topo.IsWarmingUp() and topo.WarmupDuration() so the growth wait
stays in sync with the configured warmup window. Remove unused
constants import.

* master: resolve master before creating RPC timeout context

Move GetMaster() call before context.WithTimeout() so master
resolution blocking doesn't consume the gRPC call timeout.

* master: use NotFound flag instead of string matching for volume lookup

Add a NotFound field to LookupResult and set it in findVolumeLocation
when a volume is genuinely missing. Update HTTP and gRPC warmup
checks to use this flag instead of strings.Contains on the error
message.

* master: bound assign retry loop to 30s for deadline-free contexts

Without a context deadline, the Unavailable retry loop could spin
forever. Add a maxRetryDuration of 30s so the loop gives up even
when no context deadline is set.

* master: strengthen assign retry cancellation test

Verify the retry loop actually retried (callCount > 1) and that
the returned error is context.DeadlineExceeded, not just any error.

* master: extract shared retry-with-backoff utility

Add util.RetryWithBackoff for context-aware, bounded retry with
exponential backoff. Refactor both Assign() and LookupVolumeIds()
to use it instead of duplicating the retry/sleep/backoff logic.

* master: cap waitTime in RetryWithBackoff to prevent unbounded growth

Cap the backoff waitTime at maxWaitTime so it doesn't grow
indefinitely in long-running retry scenarios.

* master: only return Unavailable during warmup when all lookups failed

For batched LookupVolume requests, return partial results when some
volumes are found. Only return codes.Unavailable when no volumes
were successfully resolved, so clients benefit from partial results
instead of retrying unnecessarily.

* master: set retriable error message in 503 response body

When returning 503 during warmup, replace the "not found" error
in the JSON body with "service warming up, please retry" so
clients don't treat it as a permanent error.

* master: guard empty master address in LookupVolumeIds

If GetMaster() returns empty (no master found or ctx cancelled),
return an appropriate error instead of dialing an empty address.
Returns ctx.Err() if context is done, otherwise codes.Unavailable
to trigger retry.

* master: add comprehensive tests for RetryWithBackoff

Test success after retries, non-retryable error handling, context
cancellation, and maxDuration cap with context.Background().

* master: enforce hard maxDuration bound in RetryWithBackoff

Use a deadline instead of elapsed-time check so the last sleep is
capped to remaining time. This prevents the total retry duration
from overshooting maxDuration by up to one full backoff interval.

* master: respect fresh-start bypass in RemainingWarmupDuration

Check IsWarmingUp() first (which returns false when MaxVolumeId==0)
so RemainingWarmupDuration returns 0 on fresh clusters.

* master: round up Retry-After seconds to avoid underestimating

Use math.Ceil so fractional remaining seconds (e.g. 1.9s) round
up to the next integer (2) instead of flooring down (1).

* master: tighten batch lookup warmup to all-NotFound only

Only return codes.Unavailable when every requested volume ID was
a transient not-found. Mixed cases with non-NotFound errors now
return the response with per-volume error details preserved.

* master: reduce retry log noise and fix timer leak

Lower per-attempt retry log from V(0) to V(1) to reduce noise
during warmup. Replace time.After with time.NewTimer to avoid
lingering timers when context is cancelled.

* master: add per-attempt timeout for assign RPC

Use a 10s per-attempt timeout so a single slow RPC can't consume
the entire 30s retry budget when ctx has no deadline.

* master: share single 30s retry deadline across assign request entries

The Assign() function iterates over primary and fallback requests,
previously giving each its own 30s RetryWithBackoff budget. With a
primary + fallback, the total could reach 60s. Compute one deadline
up front and pass the remaining budget to each RetryWithBackoff call
so the entire Assign() call stays within a single 30s cap.

* master: strengthen context-cancel test with DeadlineExceeded and retry assertions

Assert errors.Is(err, context.DeadlineExceeded) to verify the error
is specifically from the context deadline, and check callCount > 1
to prove retries actually occurred before cancellation. Mirrors the
pattern used in TestAssignStopsOnContextCancel.

* master: bound GetMaster with per-attempt timeout in LookupVolumeIds

GetMaster() calls WaitUntilConnected() which can block indefinitely
if no master is available. Previously it used the outer ctx, so a
slow master resolution could consume the entire RetryWithBackoff
budget in a single attempt. Move the per-attempt timeoutCtx creation
before the GetMaster call so both master resolution and the gRPC
LookupVolume RPC share one grpcTimeout-bounded attempt.

* master: use deadline-aware context for assign retry budget

The shared 30s deadline only limited RetryWithBackoff's internal
wall-clock tracking, but per-attempt contexts were still derived
from the original ctx and could run for up to 10s even when the
budget was nearly exhausted. Create a deadlineCtx from the computed
deadline and derive both RetryWithBackoff and per-attempt timeouts
from it so all operations honor the shared 30s cap.

* master: skip warmup gate for empty lookup requests

When VolumeOrFileIds is empty, notFoundCount == len(req.VolumeOrFileIds)
is 0 == 0 which is true, causing empty lookup batches during warmup to
return codes.Unavailable and be retried endlessly. Add a
len(req.VolumeOrFileIds) > 0 guard so empty requests pass through.

* master: validate request fields before warmup gate in Assign

Move Replication and Ttl parsing before the IsWarmingUp() check so
invalid inputs get a proper validation error instead of being masked
by codes.Unavailable during warmup. Pure syntactic validation does
not depend on topology state and should run first.

* master: check deadline and context before starting retry attempt

RetryWithBackoff only checked the deadline and context after an
attempt completed or during the sleep select. If the deadline
expired or context was canceled during sleep, the next iteration
would still call operation() before detecting it. Add pre-operation
checks so no new attempt starts after the budget is exhausted.

* master: always return ctx.Err() on context cancellation in RetryWithBackoff

When ctx.Err() is non-nil, the pre-operation check was returning
lastErr instead of ctx.Err(). This broke callers checking
errors.Is(err, context.DeadlineExceeded) and contradicted the
documented contract. Always return ctx.Err() so the cancellation
reason is properly surfaced.

* master: handle warmup errors in StreamAssign without killing the stream

StreamAssign was returning codes.Unavailable errors from Assign
directly, which terminates the gRPC stream and breaks pooled
connections. Instead, return transient errors as in-band error
responses so the stream survives warmup periods.

Also reset assignClient in doAssign on Send/Recv failures so a
broken stream doesn't leave the proxy permanently dead.

* master: wait for warmup before slot search in findAndGrow

findEmptySlotsForOneVolume was called before the warmup wait loop,
selecting slots from an incomplete topology. Move the warmup wait
before slot search so volume placement uses the fully warmed-up
topology with all servers registered.

* master: add Retry-After header to /dir/assign warmup response

The /dir/lookup handler already sets Retry-After during warmup but
/dir/assign did not, leaving HTTP clients without guidance on when
to retry. Add the same header using RemainingWarmupDuration().

* master: only seed warmup timestamp on leader at startup

SetLastLeaderChangeTime was called unconditionally for both leader
and follower nodes. Followers don't need warmup state, and the
leader change event listener handles real elections. Move the seed
into the IsLeader() block so only the startup leader gets warmup
initialized.

* master: preserve codes.Unavailable for StreamAssign warmup errors in doAssign

StreamAssign returns transient warmup errors as in-band
AssignResponse.Error messages. doAssign was converting these to plain
fmt.Errorf, losing the codes.Unavailable classification needed for
the caller's retry logic. Detect warmup error messages and wrap them
as status.Error(codes.Unavailable) so RetryWithBackoff can retry.
2026-03-08 16:05:45 -07:00
0647f66bb5 filer.sync: add exponential backoff on unexpected EOF during replication (#8557)
* filer.sync: add exponential backoff on unexpected EOF during replication

When the source volume server drops connections under high traffic,
filer.sync retries aggressively (every 1-6s), hammering the already
overloaded source. This adds a longer exponential backoff (10s to 2min)
specifically for "unexpected EOF" errors, reducing pressure on the
source while still retrying indefinitely until success.

Also adds more logging throughout the replication path:
- Log source URL and error at V(0) when ReadPart or io.ReadAll fails
- Log content-length and byte counts at V(4) on success
- Log backoff duration in retry messages

Fixes #8542

* filer.sync: extract backoff helper and fix 2-minute cap

- Extract nextEofBackoff() and isEofError() helpers to deduplicate
  the backoff logic between fetchAndWrite and uploadManifestChunk
- Fix the cap: previously 80s would double to 160s and pass the
  < 2min check uncapped. Now doubles first, then clamps to 2min.

* filer.sync: log source URL instead of empty upload URL on read errors

UploadUrl is not populated until after the reader is consumed, so the
V(0) and V(4) logs were printing an empty string. Add SourceUrl field
to UploadOption and populate it from the HTTP response in fetchAndWrite.

* filer.sync: guard isEofError against nil error

* filer.sync: use errors.Is for EOF detection, fix log wording

- Replace broad substring matching ("read input", "unexpected EOF")
  with errors.Is(err, io.ErrUnexpectedEOF) and errors.Is(err, io.EOF)
  so only actual EOF errors trigger the longer backoff
- Fix awkward log phrasing: "interrupted replicate" → "interrupted
  while replicating"

* filer.sync: remove EOF backoff from uploadManifestChunk

uploadManifestChunk reads from an in-memory bytes.Reader, so any EOF
errors there are from the destination side, not a broken source stream.
The long source-oriented backoff is inappropriate; let RetryUntil
handle destination retries at its normal cadence.

---------

Co-authored-by: Copilot <copilot@github.com>
2026-03-08 14:33:37 -07:00
Chris Lu ba66411337 Update plugin_templ.go 2026-03-08 14:29:06 -07:00
Chris LuandGitHub 7808b301ef admin: remove Scheduler Settings cards from plugin UI (#8558)
* admin: remove Scheduler Settings cards, make Next Run full-width

Remove the two "Scheduler Settings" placeholder cards from the plugin
UI (overview page and scheduler tab). They only contained a text note
saying detection intervals are configured per job type, which is
self-evident from the per-job-type settings form.

Make the "Next Run" card full-width on the overview page since it no
longer shares a row with the removed card.

* plugin UI: promote Next Run to top summary card row

Move "Next Run" from a standalone card into the top row alongside
Workers, Active Jobs, and Activities as a compact stat card.
2026-03-08 14:27:57 -07:00
160 changed files with 9913 additions and 1966 deletions
+5 -5
View File
@@ -32,7 +32,7 @@ jobs:
org.opencontainers.image.vendor=Chris Lu
-
name: Set up QEMU
uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v1
uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v1
-
name: Create BuildKit config
run: |
@@ -42,28 +42,28 @@ jobs:
EOF
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v1
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v1
with:
buildkitd-flags: "--debug"
buildkitd-config: /tmp/buildkitd.toml
-
name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v1
uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # v1
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
-
name: Login to GHCR
if: github.event_name != 'pull_request'
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v1
uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # v1
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_TOKEN }}
-
name: Build
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v2
uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # v2
with:
context: ./docker
push: ${{ github.event_name != 'pull_request' }}
@@ -126,14 +126,14 @@ jobs:
echo "seaweedfs_ref=$seaweed" >> "$GITHUB_OUTPUT"
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
uses: docker/setup-qemu-action@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
@@ -150,7 +150,7 @@ jobs:
fi
- name: Build and push image
uses: docker/build-push-action@v6
uses: docker/build-push-action@v7
with:
context: ./docker
push: ${{ github.event_name != 'pull_request' }}
+7 -7
View File
@@ -98,7 +98,7 @@ jobs:
org.opencontainers.image.vendor=Chris Lu
- name: Set up QEMU
if: matrix.platform != 'amd64'
uses: docker/setup-qemu-action@v3
uses: docker/setup-qemu-action@v4
- name: Create BuildKit config
run: |
cat > /tmp/buildkitd.toml <<EOF
@@ -106,25 +106,25 @@ jobs:
mirrors = ["https://mirror.gcr.io"]
EOF
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
with:
buildkitd-flags: "--debug"
buildkitd-config: /tmp/buildkitd.toml
- name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: github.event_name != 'pull_request'
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_TOKEN }}
- name: Build ${{ matrix.platform }} ${{ matrix.variant }}
uses: docker/build-push-action@v6
uses: docker/build-push-action@v7
env:
DOCKER_BUILDKIT: 1
with:
@@ -180,12 +180,12 @@ jobs:
ghcr.io/chrislusf/seaweedfs
tags: type=raw,value=${{ github.event_name == 'workflow_dispatch' && github.event.inputs.image_tag || 'latest' }},suffix=${{ steps.config.outputs.tag_suffix }}
- name: Login to Docker Hub
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -35,14 +35,14 @@ jobs:
org.opencontainers.image.vendor=Chris Lu
-
name: Set up QEMU
uses: docker/setup-qemu-action@v3
uses: docker/setup-qemu-action@v4
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
-
name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
@@ -57,7 +57,7 @@ jobs:
fi
-
name: Build
uses: docker/build-push-action@v6
uses: docker/build-push-action@v7
with:
context: ./docker
push: ${{ github.event_name != 'pull_request' }}
@@ -117,7 +117,7 @@ jobs:
- name: Set up QEMU
if: (github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant) && contains(matrix.platforms, 'arm')
uses: docker/setup-qemu-action@v3
uses: docker/setup-qemu-action@v4
- name: Create BuildKit config
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
@@ -129,20 +129,20 @@ jobs:
- name: Set up Docker Buildx
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
with:
buildkitd-config: /tmp/buildkitd.toml
- name: Login to Docker Hub
if: (github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant) && github.event_name != 'pull_request'
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: (github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant) && github.event_name != 'pull_request'
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -150,7 +150,7 @@ jobs:
- name: Build and push ${{ matrix.variant }}
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/build-push-action@v6
uses: docker/build-push-action@v7
env:
DOCKER_BUILDKIT: 1
with:
@@ -198,14 +198,14 @@ jobs:
steps:
- name: Login to Docker Hub
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -82,19 +82,19 @@ jobs:
echo "seaweedfs_ref=$seaweed" >> "$GITHUB_OUTPUT"
- name: Set up QEMU
uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v1
uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v1
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v1
- name: Login to Docker Hub
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v1
uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # v1
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Build and push image
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v2
uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # v2
with:
context: ./docker
push: true
+1 -1
View File
@@ -26,7 +26,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: '1.24'
go-version-file: 'go.mod'
- name: Build Telemetry Server
if: github.event_name == 'workflow_dispatch' && inputs.deploy
+1 -1
View File
@@ -11,4 +11,4 @@ jobs:
- name: 'Checkout Repository'
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8
- name: 'Dependency Review'
uses: actions/dependency-review-action@05fe4576374b728f0c523d6a13d64c25081e0803
uses: actions/dependency-review-action@2031cfc080254a8a887f58cffee85186f0e49e48
+7 -8
View File
@@ -23,17 +23,16 @@ jobs:
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Set up Go 1.x
uses: actions/setup-go@a5f9b05d2d216f63e13859e0d847461041025775 # v2
with:
go-version: ^1.13
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v2
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Cache Docker layers
uses: actions/cache@v5
+1 -1
View File
@@ -18,7 +18,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v6
with:
go-version: ^1.24
go-version: ^1.25
id: go
- name: Check out code into the Go module directory
+1 -1
View File
@@ -28,7 +28,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: '1.24'
go-version-file: 'go.mod'
- name: Build weed binary
run: |
+2 -3
View File
@@ -22,7 +22,6 @@ permissions:
contents: read
env:
GO_VERSION: '1.24'
TEST_TIMEOUT: '45m'
jobs:
@@ -35,10 +34,10 @@ jobs:
- name: Checkout code
uses: actions/checkout@v6
- name: Set up Go ${{ env.GO_VERSION }}
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: ${{ env.GO_VERSION }}
go-version-file: 'go.mod'
- name: Install FUSE and dependencies
run: |
+15 -18
View File
@@ -19,13 +19,12 @@ jobs:
name: Go Vet
runs-on: ubuntu-latest
steps:
- name: Set up Go 1.x
uses: actions/setup-go@a5f9b05d2d216f63e13859e0d847461041025775 # v2
with:
go-version: ^1.13
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v2
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Get dependencies
run: |
cd weed; go get -v -t -d ./...
@@ -42,13 +41,12 @@ jobs:
name: Build
runs-on: ubuntu-latest
steps:
- name: Set up Go 1.x
uses: actions/setup-go@a5f9b05d2d216f63e13859e0d847461041025775 # v2
with:
go-version: ^1.13
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v2
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Build
run: cd weed; go build -tags "elastic gocdk sqlite ydb tarantool tikv rclone" -v .
@@ -56,12 +54,11 @@ jobs:
name: Test
runs-on: ubuntu-latest
steps:
- name: Set up Go 1.x
uses: actions/setup-go@a5f9b05d2d216f63e13859e0d847461041025775 # v2
with:
go-version: ^1.13
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v2
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Test
run: cd weed; go test -tags "elastic gocdk sqlite ydb tarantool tikv rclone" -v ./...
+2 -2
View File
@@ -26,14 +26,14 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v6
with:
go-version: ^1.24
go-version: ^1.25
cache: true
cache-dependency-path: |
**/go.sum
id: go
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Install dependencies
run: |
+7 -7
View File
@@ -37,7 +37,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v6
with:
go-version: ^1.24
go-version: ^1.25
id: go
- name: Check out code
@@ -82,7 +82,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v6
with:
go-version: ^1.24
go-version: ^1.25
id: go
- name: Check out code
@@ -132,7 +132,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v6
with:
go-version: ^1.24
go-version: ^1.25
cache: true
cache-dependency-path: |
**/go.sum
@@ -311,7 +311,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v6
with:
go-version: ^1.24
go-version: ^1.25
cache: true
cache-dependency-path: |
**/go.sum
@@ -473,7 +473,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v6
with:
go-version: ^1.24
go-version: ^1.25
cache: true
cache-dependency-path: |
**/go.sum
@@ -631,7 +631,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v6
with:
go-version: ^1.24
go-version: ^1.25
cache: true
cache-dependency-path: |
**/go.sum
@@ -789,7 +789,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v6
with:
go-version: ^1.24
go-version: ^1.25
id: go
- name: Check out code
@@ -30,7 +30,6 @@ permissions:
contents: read
env:
GO_VERSION: '1.24'
TEST_TIMEOUT: '10m'
jobs:
@@ -43,10 +42,10 @@ jobs:
- name: Checkout code
uses: actions/checkout@v6
- name: Set up Go ${{ env.GO_VERSION }}
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: ${{ env.GO_VERSION }}
go-version-file: 'go.mod'
- name: Build SeaweedFS
run: |
+2 -2
View File
@@ -25,14 +25,14 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v6
with:
go-version: ^1.24
go-version: ^1.25
id: go
- name: Check out code
uses: actions/checkout@v6
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Cache Docker layers
uses: actions/cache@v5
+11 -2
View File
@@ -5,6 +5,8 @@ on:
paths:
- 'weed/iam/**'
- 'weed/s3api/**'
- 'weed/credential/**'
- 'weed/pb/**'
- 'test/s3/iam/**'
- '.github/workflows/s3-iam-tests.yml'
push:
@@ -12,6 +14,8 @@ on:
paths:
- 'weed/iam/**'
- 'weed/s3api/**'
- 'weed/credential/**'
- 'weed/pb/**'
- 'test/s3/iam/**'
- '.github/workflows/s3-iam-tests.yml'
@@ -80,7 +84,7 @@ jobs:
timeout-minutes: 25
strategy:
matrix:
test-type: ["basic", "advanced", "policy-enforcement"]
test-type: ["basic", "advanced", "policy-enforcement", "group"]
steps:
- name: Check out code
@@ -117,7 +121,7 @@ jobs:
"basic")
echo "Running basic IAM functionality tests..."
make clean setup start-services wait-for-services
go test -v -timeout 15m -run "TestS3IAMAuthentication|TestS3IAMBasicWorkflow|TestS3IAMTokenValidation|TestIAM" ./...
go test -v -timeout 15m -run "TestS3IAMAuthentication|TestS3IAMBasicWorkflow|TestS3IAMTokenValidation|TestIAMUserManagement|TestIAMAccessKeyManagement|TestIAMPolicyManagement" ./...
;;
"advanced")
echo "Running advanced IAM feature tests..."
@@ -129,6 +133,11 @@ jobs:
make clean setup start-services wait-for-services
go test -v -timeout 15m -run "TestS3IAMPolicyEnforcement|TestS3IAMBucketPolicy|TestS3IAMContextual" ./...
;;
"group")
echo "Running IAM group management tests..."
make clean setup start-services wait-for-services
go test -v -timeout 15m -run "TestIAMGroup" ./...
;;
*)
echo "Unknown test type: ${{ matrix.test-type }}"
exit 1
+2 -2
View File
@@ -41,7 +41,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: ^1.24
go-version: ^1.25
cache: true
- name: Set up Python ${{ matrix.python-version }}
@@ -148,7 +148,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: ^1.24
go-version: ^1.25
cache: true
- name: Run Go unit tests
@@ -29,7 +29,7 @@ jobs:
id: go
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Build SeaweedFS binary for Linux
run: |
+1 -1
View File
@@ -33,7 +33,7 @@ jobs:
id: go
- name: Set up Docker
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Install SeaweedFS
run: |
+5 -5
View File
@@ -144,7 +144,7 @@ jobs:
id: go
- name: Set up Docker
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Pre-pull Trino image
run: docker pull trinodb/trino:479
@@ -271,7 +271,7 @@ jobs:
id: go
- name: Set up Docker
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Pre-pull Spark image
run: docker pull apache/spark:3.5.1
@@ -337,7 +337,7 @@ jobs:
id: go
- name: Set up Docker
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Pre-pull RisingWave image
run: |
@@ -405,7 +405,7 @@ jobs:
id: go
- name: Set up Docker
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Pre-pull Python image
run: docker pull python:3
@@ -471,7 +471,7 @@ jobs:
id: go
- name: Set up Docker
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Pre-pull Python image
run: docker pull python:3
+2 -3
View File
@@ -24,7 +24,6 @@ permissions:
contents: read
env:
GO_VERSION: '1.24'
TEST_TIMEOUT: '15m'
jobs:
@@ -37,10 +36,10 @@ jobs:
- name: Checkout code
uses: actions/checkout@v6
- name: Set up Go ${{ env.GO_VERSION }}
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: ${{ env.GO_VERSION }}
go-version-file: 'go.mod'
- name: Install dependencies
run: |
@@ -43,7 +43,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: '1.24'
go-version-file: 'go.mod'
- name: Build SeaweedFS binary
run: |
@@ -24,7 +24,7 @@ jobs:
- uses: actions/setup-go@v6
with:
go-version: ^1.24
go-version: ^1.25
- name: Build SeaweedFS
run: |
@@ -28,7 +28,6 @@ permissions:
contents: read
env:
GO_VERSION: '1.24'
TEST_TIMEOUT: '30m'
jobs:
@@ -46,10 +45,10 @@ jobs:
- name: Checkout code
uses: actions/checkout@v6
- name: Set up Go ${{ env.GO_VERSION }}
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: ${{ env.GO_VERSION }}
go-version-file: 'go.mod'
- name: Build SeaweedFS binary
run: |
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.24 AS builder
FROM golang:1.25 AS builder
RUN apt-get update && \
apt-get install -y build-essential wget ca-certificates && \
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.24-alpine as builder
FROM golang:1.25-alpine AS builder
RUN apk add git g++ fuse
RUN mkdir -p /go/src/github.com/seaweedfs/
ARG BRANCH=${BRANCH:-master}
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.24 AS builder
FROM golang:1.25 AS builder
RUN apt-get update
RUN apt-get install -y build-essential libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev liblz4-dev libzstd-dev
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.24 AS builder
FROM golang:1.25 AS builder
RUN apt-get update
RUN apt-get install -y build-essential libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev liblz4-dev libzstd-dev
+7 -7
View File
@@ -1,6 +1,6 @@
module github.com/seaweedfs/seaweedfs
go 1.24.9
go 1.25.0
require (
cloud.google.com/go v0.123.0 // indirect
@@ -63,7 +63,7 @@ require (
github.com/prometheus/client_golang v1.23.2
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.2 // indirect
github.com/prometheus/procfs v0.19.2
github.com/prometheus/procfs v0.20.1
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/seaweedfs/goexif v1.0.3
@@ -87,7 +87,7 @@ require (
github.com/xdg-go/stringprep v1.0.4 // indirect
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
go.etcd.io/etcd/client/v3 v3.6.7
go.mongodb.org/mongo-driver v1.17.6
go.mongodb.org/mongo-driver v1.17.9
go.opencensus.io v0.24.0 // indirect
gocloud.dev v0.45.0
gocloud.dev/pubsub/natspubsub v0.45.0
@@ -97,7 +97,7 @@ require (
golang.org/x/image v0.36.0
golang.org/x/net v0.49.0
golang.org/x/oauth2 v0.35.0
golang.org/x/sys v0.41.0
golang.org/x/sys v0.42.0
golang.org/x/text v0.34.0 // indirect
golang.org/x/tools v0.41.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
@@ -124,13 +124,13 @@ require (
github.com/apple/foundationdb/bindings/go v0.0.0-20250911184653-27f7192f47c3
github.com/arangodb/go-driver v1.6.9
github.com/armon/go-metrics v0.4.1
github.com/aws/aws-sdk-go-v2 v1.41.1
github.com/aws/aws-sdk-go-v2 v1.41.3
github.com/aws/aws-sdk-go-v2/config v1.32.7
github.com/aws/aws-sdk-go-v2/credentials v1.19.7
github.com/aws/aws-sdk-go-v2/service/s3 v1.95.0
github.com/cognusion/imaging v1.0.2
github.com/fluent/fluent-logger-golang v1.10.1
github.com/getsentry/sentry-go v0.42.0
github.com/getsentry/sentry-go v0.43.0
github.com/go-ldap/ldap/v3 v3.4.12
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/flatbuffers/go v0.0.0-20230108230133-3b8644d32c50
@@ -341,7 +341,7 @@ require (
github.com/aws/aws-sdk-go-v2/service/sso v1.30.9 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.13 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.41.6
github.com/aws/smithy-go v1.24.0
github.com/aws/smithy-go v1.24.2
github.com/boltdb/bolt v1.3.1 // indirect
github.com/bradenaw/juniper v0.15.3 // indirect
github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 // indirect
+12 -372
View File
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -1,6 +1,6 @@
apiVersion: v1
description: SeaweedFS
name: seaweedfs
appVersion: "4.15"
appVersion: "4.17"
# Dev note: Trigger a helm chart release by `git tag -a helm-<version>`
version: 4.15.0
version: 4.17.0
@@ -243,8 +243,7 @@ spec:
{{- if $httpsPort }}
-s3.port.https={{ $httpsPort }} \
{{- end }}
-s3.cert.file=/usr/local/share/ca-certificates/client/tls.crt \
-s3.key.file=/usr/local/share/ca-certificates/client/tls.key \
{{ include "seaweedfs.s3.tlsArgs" (dict "root" . "prefix" "s3.") | nindent 14 }}
{{- end }}
{{- if or .Values.allInOne.s3.enableAuth .Values.s3.enableAuth .Values.filer.s3.enableAuth }}
-s3.config=/etc/sw/s3/seaweedfs_s3_config \
@@ -346,6 +345,9 @@ spec:
- name: client-cert
mountPath: /usr/local/share/ca-certificates/client/
readOnly: true
{{- if .Values.allInOne.s3.enabled }}
{{- include "seaweedfs.s3.tlsVolumeMount" . | nindent 12 }}
{{- end }}
{{- end }}
{{ tpl .Values.allInOne.extraVolumeMounts . | nindent 12 }}
ports:
@@ -473,6 +475,9 @@ spec:
- name: client-cert
secret:
secretName: {{ include "seaweedfs.fullname" . }}-client-cert
{{- if .Values.allInOne.s3.enabled }}
{{- include "seaweedfs.s3.tlsVolume" . | nindent 8 }}
{{- end }}
{{- end }}
{{ tpl .Values.allInOne.extraVolumes . | nindent 8 }}
{{- if .Values.allInOne.nodeSelector }}
@@ -17,6 +17,9 @@ metadata:
spec:
type: {{ .Values.allInOne.service.type | default "ClusterIP" }}
internalTrafficPolicy: {{ .Values.allInOne.service.internalTrafficPolicy | default "Cluster" }}
{{- if and (semverCompare ">=1.31-0" .Capabilities.KubeVersion.GitVersion) .Values.allInOne.s3.trafficDistribution }}
trafficDistribution: {{ include "seaweedfs.trafficDistribution" (dict "value" .Values.allInOne.s3.trafficDistribution "Capabilities" .Capabilities) }}
{{- end }}
ports:
# Master ports
- name: "swfs-master"
@@ -200,8 +200,7 @@ spec:
{{- if .Values.filer.s3.httpsPort }}
-s3.port.https={{ .Values.filer.s3.httpsPort }} \
{{- end }}
-s3.cert.file=/usr/local/share/ca-certificates/client/tls.crt \
-s3.key.file=/usr/local/share/ca-certificates/client/tls.key \
{{ include "seaweedfs.s3.tlsArgs" (dict "root" . "prefix" "s3.") | nindent 14 }}
{{- end }}
{{- if .Values.filer.s3.enableAuth }}
-s3.config=/etc/sw/seaweedfs_s3_config \
@@ -254,6 +253,9 @@ spec:
- name: client-cert
readOnly: true
mountPath: /usr/local/share/ca-certificates/client
{{- if .Values.filer.s3.enabled }}
{{- include "seaweedfs.s3.tlsVolumeMount" . | nindent 12 }}
{{- end }}
{{- end }}
{{ tpl .Values.filer.extraVolumeMounts . | nindent 12 | trim }}
ports:
@@ -384,6 +386,9 @@ spec:
- name: client-cert
secret:
secretName: {{ include "seaweedfs.fullname" . }}-client-cert
{{- if .Values.filer.s3.enabled }}
{{- include "seaweedfs.s3.tlsVolume" . | nindent 8 }}
{{- end }}
{{- end }}
{{ tpl .Values.filer.extraVolumes . | indent 8 | trim }}
{{- if .Values.filer.nodeSelector }}
@@ -127,8 +127,7 @@ spec:
{{- if .Values.s3.httpsPort }}
-port.https={{ .Values.s3.httpsPort }} \
{{- end }}
-cert.file=/usr/local/share/ca-certificates/client/tls.crt \
-key.file=/usr/local/share/ca-certificates/client/tls.key \
{{ include "seaweedfs.s3.tlsArgs" (dict "root" . "prefix" "") | nindent 14 }}
{{- end }}
{{- if .Values.s3.domainName }}
-domainName={{ .Values.s3.domainName }} \
@@ -176,6 +175,7 @@ spec:
- name: client-cert
readOnly: true
mountPath: /usr/local/share/ca-certificates/client/
{{- include "seaweedfs.s3.tlsVolumeMount" . | nindent 12 }}
{{- end }}
{{ tpl .Values.s3.extraVolumeMounts . | nindent 12 | trim }}
ports:
@@ -267,6 +267,7 @@ spec:
- name: client-cert
secret:
secretName: {{ include "seaweedfs.fullname" . }}-client-cert
{{- include "seaweedfs.s3.tlsVolume" . | nindent 8 }}
{{- end }}
{{ tpl .Values.s3.extraVolumes . | indent 8 | trim }}
{{- if .Values.s3.nodeSelector }}
@@ -16,8 +16,9 @@ metadata:
{{- end }}
spec:
internalTrafficPolicy: {{ .Values.s3.internalTrafficPolicy | default "Cluster" }}
{{- if and (semverCompare ">=1.31-0" .Capabilities.KubeVersion.GitVersion) (or .Values.s3.trafficDistribution .Values.filer.s3.trafficDistribution) }}
trafficDistribution: {{ include "seaweedfs.trafficDistribution" . }}
{{- $td := .Values.s3.trafficDistribution | default .Values.filer.s3.trafficDistribution }}
{{- if and (semverCompare ">=1.31-0" .Capabilities.KubeVersion.GitVersion) $td }}
trafficDistribution: {{ include "seaweedfs.trafficDistribution" (dict "value" $td "Capabilities" .Capabilities) }}
{{- end }}
ports:
- name: "swfs-s3"
@@ -338,11 +338,41 @@ Create the name of the service account to use
{{- .Values.global.serviceAccountName | default "seaweedfs" -}}
{{- end -}}
{{/* Generate a compatible trafficDistribution value due to "PreferClose" fast deprecation in k8s v1.35 */}}
{{/* S3 TLS cert/key arguments, using custom secret if s3.tlsSecret is set */}}
{{- define "seaweedfs.s3.tlsArgs" -}}
{{- $prefix := .prefix -}}
{{- $root := .root -}}
{{- if $root.Values.s3.tlsSecret -}}
-{{ $prefix }}cert.file=/usr/local/share/ca-certificates/s3/tls.crt \
-{{ $prefix }}key.file=/usr/local/share/ca-certificates/s3/tls.key \
{{- else -}}
-{{ $prefix }}cert.file=/usr/local/share/ca-certificates/client/tls.crt \
-{{ $prefix }}key.file=/usr/local/share/ca-certificates/client/tls.key \
{{- end -}}
{{- end -}}
{{/* S3 custom TLS volume mount */}}
{{- define "seaweedfs.s3.tlsVolumeMount" -}}
{{- if .Values.s3.tlsSecret }}
- name: s3-tls-cert
readOnly: true
mountPath: /usr/local/share/ca-certificates/s3/
{{- end }}
{{- end -}}
{{/* S3 custom TLS volume */}}
{{- define "seaweedfs.s3.tlsVolume" -}}
{{- if .Values.s3.tlsSecret }}
- name: s3-tls-cert
secret:
secretName: {{ .Values.s3.tlsSecret }}
{{- end }}
{{- end -}}
{{/* Generate a compatible trafficDistribution value due to "PreferClose" fast deprecation in k8s v1.35.
Accepts a dict with "value" (the trafficDistribution string) and "Capabilities". */}}
{{- define "seaweedfs.trafficDistribution" -}}
{{- if .Values.s3.trafficDistribution -}}
{{- and (eq .Values.s3.trafficDistribution "PreferClose") (semverCompare ">=1.35-0" .Capabilities.KubeVersion.GitVersion) | ternary "PreferSameZone" .Values.s3.trafficDistribution -}}
{{- else if .Values.filer.s3.trafficDistribution -}}
{{- and (eq .Values.filer.s3.trafficDistribution "PreferClose") (semverCompare ">=1.35-0" .Capabilities.KubeVersion.GitVersion) | ternary "PreferSameZone" .Values.filer.s3.trafficDistribution -}}
{{- if .value -}}
{{- and (eq .value "PreferClose") (semverCompare ">=1.35-0" .Capabilities.KubeVersion.GitVersion) | ternary "PreferSameZone" .value -}}
{{- end -}}
{{- end -}}
+8
View File
@@ -919,6 +919,13 @@ s3:
port: 8333
# add additional https port
httpsPort: 0
# Use a custom TLS certificate secret for the S3 HTTPS endpoint.
# When set, this Kubernetes Secret (must contain tls.crt and tls.key) is used
# instead of the internal self-signed client certificate generated by cert-manager.
# This allows using a publicly trusted certificate (e.g., from Let's Encrypt)
# so that S3 clients don't need to trust the internal CA.
# Requires global.enableSecurity to be true.
tlsSecret: null
metricsPort: 9327
# Iceberg catalog REST port (Apache Iceberg REST Catalog API)
# Set to a port number to enable, or 0/null to disable
@@ -1453,6 +1460,7 @@ allInOne:
# The s3-secret.yaml template only reads from .Values.s3.credentials.
# See: s3.credentials.admin.accessKey, s3.credentials.read.accessKey
auditLogConfig: null # S3 audit log configuration (null inherits from s3.auditLogConfig)
trafficDistribution: null # Service traffic distribution (e.g., "PreferClose"); auto-converts to "PreferSameZone" on k8s >=1.35
# You may specify buckets to be created during the install process.
# Buckets may be exposed publicly by setting `anonymousRead` to `true`
# ttl format: [1-255][m|h|d|w|M|y] (e.g., 7d)
+1 -1
View File
@@ -1,5 +1,5 @@
# Multi-stage build for Go Sidecar
FROM golang:1.24-alpine AS builder
FROM golang:1.25-alpine AS builder
# Install build dependencies
RUN apk add --no-cache git ca-certificates tzdata
@@ -1,5 +1,5 @@
# Multi-stage build for Test Client
FROM golang:1.23-alpine AS builder
FROM golang:1.25-alpine AS builder
# Install build dependencies
RUN apk add --no-cache git ca-certificates tzdata
@@ -194,28 +194,6 @@ func TestEcEndToEnd(t *testing.T) {
// 1. Configure plugin job types for fast EC detection/execution.
t.Log("Configuring plugin job types via API...")
schedulerConfig := map[string]interface{}{
"idle_sleep_seconds": 1,
}
jsonBody, err := json.Marshal(schedulerConfig)
if err != nil {
t.Fatalf("Failed to marshal scheduler config: %v", err)
}
req, err := http.NewRequest("PUT", AdminUrl+"/api/plugin/scheduler-config", bytes.NewBuffer(jsonBody))
if err != nil {
t.Fatalf("Failed to create scheduler config request: %v", err)
}
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
t.Fatalf("Failed to update scheduler config: %v", err)
}
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
t.Fatalf("Failed to update scheduler config (status %d): %s", resp.StatusCode, string(body))
}
resp.Body.Close()
// Disable volume balance to reduce interference for this EC-focused test.
balanceConfig := map[string]interface{}{
"job_type": "volume_balance",
@@ -223,16 +201,16 @@ func TestEcEndToEnd(t *testing.T) {
"enabled": false,
},
}
jsonBody, err = json.Marshal(balanceConfig)
jsonBody, err := json.Marshal(balanceConfig)
if err != nil {
t.Fatalf("Failed to marshal volume_balance config: %v", err)
}
req, err = http.NewRequest("PUT", AdminUrl+"/api/plugin/job-types/volume_balance/config", bytes.NewBuffer(jsonBody))
req, err := http.NewRequest("PUT", AdminUrl+"/api/plugin/job-types/volume_balance/config", bytes.NewBuffer(jsonBody))
if err != nil {
t.Fatalf("Failed to create volume_balance config request: %v", err)
}
req.Header.Set("Content-Type", "application/json")
resp, err = client.Do(req)
resp, err := client.Do(req)
if err != nil {
t.Fatalf("Failed to update volume_balance config: %v", err)
}
+1 -1
View File
@@ -1,6 +1,6 @@
# Simplified single-stage build for SeaweedFS with FoundationDB support
# Force x86_64 platform to use AMD64 FoundationDB packages
FROM --platform=linux/amd64 golang:1.24-bookworm
FROM --platform=linux/amd64 golang:1.25-bookworm
ARG FOUNDATIONDB_VERSION=7.4.5
ENV FOUNDATIONDB_VERSION=${FOUNDATIONDB_VERSION}
+1 -1
View File
@@ -1,5 +1,5 @@
# Multi-stage Dockerfile to build SeaweedFS with FoundationDB support for ARM64
FROM --platform=linux/arm64 golang:1.24-bookworm AS builder
FROM --platform=linux/arm64 golang:1.25-bookworm AS builder
ARG FOUNDATIONDB_VERSION=7.4.5
ENV FOUNDATIONDB_VERSION=${FOUNDATIONDB_VERSION}
+1 -1
View File
@@ -1,5 +1,5 @@
# Test environment with Go and FoundationDB support
FROM golang:1.24-bookworm
FROM golang:1.25-bookworm
# Install system dependencies
RUN apt-get update && apt-get install -y \
+1 -1
View File
@@ -123,7 +123,7 @@
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.9.4</version>
<version>3.9.5</version>
</dependency>
<!-- Apache Commons - Fix CVEs -->
+1 -1
View File
@@ -1,5 +1,5 @@
# Dockerfile for Kafka Gateway Integration Testing
FROM golang:1.24-alpine AS builder
FROM golang:1.25-alpine AS builder
# Install build dependencies
RUN apk add --no-cache git make gcc musl-dev sqlite-dev
+1 -1
View File
@@ -1,5 +1,5 @@
# Dockerfile for building SeaweedFS components from the current workspace
FROM golang:1.24-alpine AS builder
FROM golang:1.25-alpine AS builder
RUN apk add --no-cache git make gcc musl-dev sqlite-dev
+1 -1
View File
@@ -1,5 +1,5 @@
# Dockerfile for Kafka Integration Test Setup
FROM golang:1.24-alpine AS builder
FROM golang:1.25-alpine AS builder
# Install build dependencies
RUN apk add --no-cache git make gcc musl-dev
@@ -2,7 +2,7 @@
# Multi-stage build for cross-platform support
# Stage 1: Builder
FROM golang:1.24-alpine AS builder
FROM golang:1.25-alpine AS builder
WORKDIR /app
+48 -13
View File
@@ -27,19 +27,21 @@ type VolumeServer struct {
address string
baseDir string
mu sync.Mutex
receivedFiles map[string]uint64
mountRequests []*volume_server_pb.VolumeEcShardsMountRequest
deleteRequests []*volume_server_pb.VolumeDeleteRequest
markReadonlyCalls int
vacuumGarbageRatio float64
vacuumCheckCalls int
vacuumCompactCalls int
vacuumCommitCalls int
vacuumCleanupCalls int
volumeCopyCalls int
volumeMountCalls int
tailReceiverCalls int
mu sync.Mutex
receivedFiles map[string]uint64
mountRequests []*volume_server_pb.VolumeEcShardsMountRequest
deleteRequests []*volume_server_pb.VolumeDeleteRequest
markReadonlyCalls int
markWritableCalls int
readFileStatusCalls int
vacuumGarbageRatio float64
vacuumCheckCalls int
vacuumCompactCalls int
vacuumCommitCalls int
vacuumCleanupCalls int
volumeCopyCalls int
volumeMountCalls int
tailReceiverCalls int
}
// NewVolumeServer starts a test volume server using the provided base directory.
@@ -151,6 +153,20 @@ func (v *VolumeServer) MarkReadonlyCount() int {
return v.markReadonlyCalls
}
// MarkWritableCount returns the number of writable calls.
func (v *VolumeServer) MarkWritableCount() int {
v.mu.Lock()
defer v.mu.Unlock()
return v.markWritableCalls
}
// ReadFileStatusCount returns the number of ReadVolumeFileStatus calls.
func (v *VolumeServer) ReadFileStatusCount() int {
v.mu.Lock()
defer v.mu.Unlock()
return v.readFileStatusCalls
}
// Shutdown stops the volume server.
func (v *VolumeServer) Shutdown() {
if v.server != nil {
@@ -280,6 +296,25 @@ func (v *VolumeServer) VolumeMarkReadonly(ctx context.Context, req *volume_serve
return &volume_server_pb.VolumeMarkReadonlyResponse{}, nil
}
func (v *VolumeServer) VolumeMarkWritable(ctx context.Context, req *volume_server_pb.VolumeMarkWritableRequest) (*volume_server_pb.VolumeMarkWritableResponse, error) {
v.mu.Lock()
v.markWritableCalls++
v.mu.Unlock()
return &volume_server_pb.VolumeMarkWritableResponse{}, nil
}
func (v *VolumeServer) ReadVolumeFileStatus(ctx context.Context, req *volume_server_pb.ReadVolumeFileStatusRequest) (*volume_server_pb.ReadVolumeFileStatusResponse, error) {
v.mu.Lock()
v.readFileStatusCalls++
v.mu.Unlock()
return &volume_server_pb.ReadVolumeFileStatusResponse{
VolumeId: req.VolumeId,
DatFileSize: 1024,
IdxFileSize: 16,
FileCount: 1,
}, nil
}
func (v *VolumeServer) VacuumVolumeCheck(ctx context.Context, req *volume_server_pb.VacuumVolumeCheckRequest) (*volume_server_pb.VacuumVolumeCheckResponse, error) {
v.mu.Lock()
v.vacuumCheckCalls++
@@ -37,7 +37,9 @@ func TestVolumeBalanceDetectionIntegration(t *testing.T) {
MasterGrpcAddresses: []string{master.Address()},
}, 10)
require.NoError(t, err)
require.Len(t, proposals, 1)
// With default batch_size=20 and 10 overloaded volumes vs 1 underloaded,
// all moves are grouped into a single batch proposal.
require.Len(t, proposals, 1, "expected exactly one batch proposal")
proposal := proposals[0]
require.Equal(t, "volume_balance", proposal.JobType)
@@ -46,8 +48,15 @@ func TestVolumeBalanceDetectionIntegration(t *testing.T) {
params := &worker_pb.TaskParams{}
require.NoError(t, proto.Unmarshal(paramsValue.GetBytesValue(), params))
require.NotEmpty(t, params.Sources)
require.NotEmpty(t, params.Targets)
bp := params.GetBalanceParams()
require.NotNil(t, bp, "expected BalanceParams in batch proposal")
require.Greater(t, len(bp.Moves), 1, "batch proposal should contain multiple moves")
for _, move := range bp.Moves {
require.NotZero(t, move.VolumeId)
require.NotEmpty(t, move.SourceNode)
require.NotEmpty(t, move.TargetNode)
}
}
func buildBalanceVolumeListResponse(t *testing.T) *master_pb.VolumeListResponse {
@@ -8,10 +8,12 @@ import (
pluginworkers "github.com/seaweedfs/seaweedfs/test/plugin_workers"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/worker_pb"
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
"github.com/stretchr/testify/require"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/protobuf/proto"
)
func TestVolumeBalanceExecutionIntegration(t *testing.T) {
@@ -60,8 +62,92 @@ func TestVolumeBalanceExecutionIntegration(t *testing.T) {
require.GreaterOrEqual(t, source.MarkReadonlyCount(), 1)
require.GreaterOrEqual(t, len(source.DeleteRequests()), 1)
copyCalls, mountCalls, tailCalls := target.BalanceStats()
copyCalls, _, tailCalls := target.BalanceStats()
require.GreaterOrEqual(t, copyCalls, 1)
require.GreaterOrEqual(t, mountCalls, 1)
require.GreaterOrEqual(t, tailCalls, 1)
}
func TestVolumeBalanceBatchExecutionIntegration(t *testing.T) {
dialOption := grpc.WithTransportCredentials(insecure.NewCredentials())
handler := pluginworker.NewVolumeBalanceHandler(dialOption)
harness := pluginworkers.NewHarness(t, pluginworkers.HarnessConfig{
WorkerOptions: pluginworker.WorkerOptions{
GrpcDialOption: dialOption,
},
Handlers: []pluginworker.JobHandler{handler},
})
harness.WaitForJobType("volume_balance")
// Create one source and one target fake volume server.
source := pluginworkers.NewVolumeServer(t, "")
target := pluginworkers.NewVolumeServer(t, "")
// Build a batch job with 3 volume moves from source → target.
volumeIDs := []uint32{401, 402, 403}
moves := make([]*worker_pb.BalanceMoveSpec, len(volumeIDs))
for i, vid := range volumeIDs {
moves[i] = &worker_pb.BalanceMoveSpec{
VolumeId: vid,
SourceNode: source.Address(),
TargetNode: target.Address(),
Collection: "batch-test",
}
}
params := &worker_pb.TaskParams{
TaskId: "batch-balance-test",
TaskParams: &worker_pb.TaskParams_BalanceParams{
BalanceParams: &worker_pb.BalanceTaskParams{
MaxConcurrentMoves: 2,
Moves: moves,
},
},
}
paramBytes, err := proto.Marshal(params)
require.NoError(t, err)
job := &plugin_pb.JobSpec{
JobId: "batch-balance-test",
JobType: "volume_balance",
Parameters: map[string]*plugin_pb.ConfigValue{
"task_params_pb": {
Kind: &plugin_pb.ConfigValue_BytesValue{BytesValue: paramBytes},
},
},
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
result, err := harness.Plugin().ExecuteJob(ctx, job, nil, 1)
require.NoError(t, err)
require.NotNil(t, result)
require.True(t, result.Success, "batch balance job should succeed; result: %+v", result)
// Each of the 3 moves should have marked the source readonly and deleted.
require.Equal(t, len(volumeIDs), source.MarkReadonlyCount(),
"each move should mark source volume readonly")
require.Equal(t, len(volumeIDs), len(source.DeleteRequests()),
"each move should delete the source volume")
// Verify delete requests reference the expected volume IDs.
deletedVols := make(map[uint32]bool)
for _, req := range source.DeleteRequests() {
deletedVols[req.VolumeId] = true
}
for _, vid := range volumeIDs {
require.True(t, deletedVols[vid], "volume %d should have been deleted from source", vid)
}
// Pre-delete verification should have called ReadVolumeFileStatus on both
// source and target for each volume.
require.Equal(t, len(volumeIDs), source.ReadFileStatusCount(),
"each move should read source volume status before delete")
require.Equal(t, len(volumeIDs), target.ReadFileStatusCount(),
"each move should read target volume status before delete")
// Target should have received copy and tail calls for all 3 volumes.
copyCalls, _, tailCalls := target.BalanceStats()
require.Equal(t, len(volumeIDs), copyCalls, "target should receive one copy per volume")
require.Equal(t, len(volumeIDs), tailCalls, "target should receive one tail per volume")
}
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.24-alpine AS builder
FROM golang:1.25-alpine AS builder
# Set working directory
WORKDIR /app
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.24-alpine AS builder
FROM golang:1.25-alpine AS builder
# Set working directory
WORKDIR /app
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.24-alpine AS builder
FROM golang:1.25-alpine AS builder
# Install git and other build dependencies
RUN apk add --no-cache git make
+4 -4
View File
@@ -140,7 +140,7 @@ func TestCORSConfigurationManagement(t *testing.T) {
Bucket: aws.String(bucketName),
CORSConfiguration: corsConfig,
})
assert.NoError(t, err, "Should be able to put CORS configuration")
require.NoError(t, err, "Should be able to put CORS configuration")
// Wait for metadata subscription to update cache
time.Sleep(50 * time.Millisecond)
@@ -149,9 +149,9 @@ func TestCORSConfigurationManagement(t *testing.T) {
getResp, err := client.GetBucketCors(context.TODO(), &s3.GetBucketCorsInput{
Bucket: aws.String(bucketName),
})
assert.NoError(t, err, "Should be able to get CORS configuration")
assert.NotNil(t, getResp.CORSRules, "CORS configuration should not be nil")
assert.Len(t, getResp.CORSRules, 1, "Should have one CORS rule")
require.NoError(t, err, "Should be able to get CORS configuration")
require.NotNil(t, getResp.CORSRules, "CORS configuration should not be nil")
require.Len(t, getResp.CORSRules, 1, "Should have one CORS rule")
rule := getResp.CORSRules[0]
assert.Equal(t, []string{"*"}, rule.AllowedHeaders, "Allowed headers should match")
+1 -1
View File
@@ -1,5 +1,5 @@
# Multi-stage build for SeaweedFS S3 with IAM
FROM golang:1.23-alpine AS builder
FROM golang:1.25-alpine AS builder
# Install build dependencies
RUN apk add --no-cache git make curl wget
+4 -1
View File
@@ -185,6 +185,9 @@ test-context: ## Test only contextual policy enforcement
test-presigned: ## Test only presigned URL integration
go test -v -run TestS3IAMPresignedURLIntegration ./...
test-group: ## Run IAM group management tests
go test -v -run "TestIAMGroup" ./...
test-sts: ## Run all STS tests
go test -v -run "TestSTS" ./...
@@ -263,7 +266,7 @@ docker-build: ## Build custom SeaweedFS image for Docker tests
# All PHONY targets
.PHONY: test test-quick run-tests setup start-services stop-services wait-for-services clean logs status debug
.PHONY: test-auth test-policy test-expiration test-multipart test-bucket-policy test-context test-presigned test-sts test-sts-assume-role test-sts-ldap
.PHONY: test-auth test-policy test-expiration test-multipart test-bucket-policy test-context test-presigned test-group test-sts test-sts-assume-role test-sts-ldap
.PHONY: benchmark ci watch install-deps docker-test docker-up docker-down docker-logs docker-build
.PHONY: test-distributed test-performance test-stress test-versioning-stress test-keycloak-full test-all-previously-skipped setup-all-tests help-advanced
+792
View File
@@ -0,0 +1,792 @@
package iam
import (
"encoding/xml"
"io"
"net/http"
"net/url"
"strings"
"testing"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/iam"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestIAMGroupLifecycle tests the full lifecycle of group management:
// CreateGroup, GetGroup, ListGroups, DeleteGroup
func TestIAMGroupLifecycle(t *testing.T) {
framework := NewS3IAMTestFramework(t)
defer framework.Cleanup()
iamClient, err := framework.CreateIAMClientWithJWT("admin-user", "TestAdminRole")
require.NoError(t, err)
groupName := "test-group-lifecycle"
t.Run("create_group", func(t *testing.T) {
resp, err := iamClient.CreateGroup(&iam.CreateGroupInput{
GroupName: aws.String(groupName),
})
require.NoError(t, err)
assert.Equal(t, groupName, *resp.Group.GroupName)
})
t.Run("get_group", func(t *testing.T) {
resp, err := iamClient.GetGroup(&iam.GetGroupInput{
GroupName: aws.String(groupName),
})
require.NoError(t, err)
assert.Equal(t, groupName, *resp.Group.GroupName)
})
t.Run("list_groups_contains_created", func(t *testing.T) {
resp, err := iamClient.ListGroups(&iam.ListGroupsInput{})
require.NoError(t, err)
found := false
for _, g := range resp.Groups {
if *g.GroupName == groupName {
found = true
break
}
}
assert.True(t, found, "Created group should appear in ListGroups")
})
t.Run("create_duplicate_group_fails", func(t *testing.T) {
_, err := iamClient.CreateGroup(&iam.CreateGroupInput{
GroupName: aws.String(groupName),
})
assert.Error(t, err, "Creating a duplicate group should fail")
})
t.Run("delete_group", func(t *testing.T) {
_, err := iamClient.DeleteGroup(&iam.DeleteGroupInput{
GroupName: aws.String(groupName),
})
require.NoError(t, err)
// Verify it's gone
resp, err := iamClient.ListGroups(&iam.ListGroupsInput{})
require.NoError(t, err)
for _, g := range resp.Groups {
assert.NotEqual(t, groupName, *g.GroupName,
"Deleted group should not appear in ListGroups")
}
})
t.Run("delete_nonexistent_group_fails", func(t *testing.T) {
_, err := iamClient.DeleteGroup(&iam.DeleteGroupInput{
GroupName: aws.String("nonexistent-group-xyz"),
})
assert.Error(t, err)
})
}
// TestIAMGroupMembership tests adding and removing users from groups
func TestIAMGroupMembership(t *testing.T) {
framework := NewS3IAMTestFramework(t)
defer framework.Cleanup()
iamClient, err := framework.CreateIAMClientWithJWT("admin-user", "TestAdminRole")
require.NoError(t, err)
groupName := "test-group-members"
userName := "test-user-for-group"
// Setup: create group and user
_, err = iamClient.CreateGroup(&iam.CreateGroupInput{
GroupName: aws.String(groupName),
})
require.NoError(t, err)
defer iamClient.DeleteGroup(&iam.DeleteGroupInput{GroupName: aws.String(groupName)})
_, err = iamClient.CreateUser(&iam.CreateUserInput{
UserName: aws.String(userName),
})
require.NoError(t, err)
defer iamClient.DeleteUser(&iam.DeleteUserInput{UserName: aws.String(userName)})
t.Run("add_user_to_group", func(t *testing.T) {
_, err := iamClient.AddUserToGroup(&iam.AddUserToGroupInput{
GroupName: aws.String(groupName),
UserName: aws.String(userName),
})
require.NoError(t, err)
})
t.Run("get_group_shows_member", func(t *testing.T) {
resp, err := iamClient.GetGroup(&iam.GetGroupInput{
GroupName: aws.String(groupName),
})
require.NoError(t, err)
found := false
for _, u := range resp.Users {
if *u.UserName == userName {
found = true
break
}
}
assert.True(t, found, "Added user should appear in GetGroup members")
})
t.Run("list_groups_for_user", func(t *testing.T) {
resp, err := iamClient.ListGroupsForUser(&iam.ListGroupsForUserInput{
UserName: aws.String(userName),
})
require.NoError(t, err)
found := false
for _, g := range resp.Groups {
if *g.GroupName == groupName {
found = true
break
}
}
assert.True(t, found, "Group should appear in ListGroupsForUser")
})
t.Run("add_duplicate_member_is_idempotent", func(t *testing.T) {
_, err := iamClient.AddUserToGroup(&iam.AddUserToGroupInput{
GroupName: aws.String(groupName),
UserName: aws.String(userName),
})
// Should succeed (idempotent) or return a benign error
// AWS IAM allows duplicate add without error
assert.NoError(t, err)
})
t.Run("remove_user_from_group", func(t *testing.T) {
_, err := iamClient.RemoveUserFromGroup(&iam.RemoveUserFromGroupInput{
GroupName: aws.String(groupName),
UserName: aws.String(userName),
})
require.NoError(t, err)
// Verify removal
resp, err := iamClient.GetGroup(&iam.GetGroupInput{
GroupName: aws.String(groupName),
})
require.NoError(t, err)
for _, u := range resp.Users {
assert.NotEqual(t, userName, *u.UserName,
"Removed user should not appear in group members")
}
})
}
// TestIAMGroupPolicyAttachment tests attaching and detaching policies from groups
func TestIAMGroupPolicyAttachment(t *testing.T) {
framework := NewS3IAMTestFramework(t)
defer framework.Cleanup()
iamClient, err := framework.CreateIAMClientWithJWT("admin-user", "TestAdminRole")
require.NoError(t, err)
groupName := "test-group-policies"
policyName := "test-group-attach-policy"
policyDoc := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"s3:ListBucket","Resource":"*"}]}`
// Setup: create group and policy
_, err = iamClient.CreateGroup(&iam.CreateGroupInput{
GroupName: aws.String(groupName),
})
require.NoError(t, err)
createPolicyResp, err := iamClient.CreatePolicy(&iam.CreatePolicyInput{
PolicyName: aws.String(policyName),
PolicyDocument: aws.String(policyDoc),
})
require.NoError(t, err)
policyArn := createPolicyResp.Policy.Arn
// Cleanup in correct order: detach policy, delete group, delete policy
t.Cleanup(func() {
if _, err := iamClient.DetachGroupPolicy(&iam.DetachGroupPolicyInput{
GroupName: aws.String(groupName),
PolicyArn: policyArn,
}); err != nil {
t.Logf("cleanup: failed to detach group policy: %v", err)
}
if _, err := iamClient.DeleteGroup(&iam.DeleteGroupInput{GroupName: aws.String(groupName)}); err != nil {
t.Logf("cleanup: failed to delete group: %v", err)
}
if _, err := iamClient.DeletePolicy(&iam.DeletePolicyInput{PolicyArn: policyArn}); err != nil {
t.Logf("cleanup: failed to delete policy: %v", err)
}
})
t.Run("attach_group_policy", func(t *testing.T) {
_, err := iamClient.AttachGroupPolicy(&iam.AttachGroupPolicyInput{
GroupName: aws.String(groupName),
PolicyArn: policyArn,
})
require.NoError(t, err)
})
t.Run("list_attached_group_policies", func(t *testing.T) {
resp, err := iamClient.ListAttachedGroupPolicies(&iam.ListAttachedGroupPoliciesInput{
GroupName: aws.String(groupName),
})
require.NoError(t, err)
found := false
for _, p := range resp.AttachedPolicies {
if *p.PolicyName == policyName {
found = true
break
}
}
assert.True(t, found, "Attached policy should appear in ListAttachedGroupPolicies")
})
t.Run("detach_group_policy", func(t *testing.T) {
_, err := iamClient.DetachGroupPolicy(&iam.DetachGroupPolicyInput{
GroupName: aws.String(groupName),
PolicyArn: policyArn,
})
require.NoError(t, err)
// Verify detachment
resp, err := iamClient.ListAttachedGroupPolicies(&iam.ListAttachedGroupPoliciesInput{
GroupName: aws.String(groupName),
})
require.NoError(t, err)
for _, p := range resp.AttachedPolicies {
assert.NotEqual(t, policyName, *p.PolicyName,
"Detached policy should not appear in ListAttachedGroupPolicies")
}
})
}
// TestIAMGroupPolicyEnforcement tests that group policies are enforced during S3 operations.
// Creates a user with no direct policies, adds them to a group with S3 access,
// and verifies they can access S3 through the group policy.
func TestIAMGroupPolicyEnforcement(t *testing.T) {
framework := NewS3IAMTestFramework(t)
defer framework.Cleanup()
iamClient, err := framework.CreateIAMClientWithJWT("admin-user", "TestAdminRole")
require.NoError(t, err)
groupName := "test-enforcement-group"
userName := "test-enforcement-user"
policyName := "test-enforcement-policy"
bucketName := "test-group-enforce-bucket"
policyDoc := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:*"],"Resource":["arn:aws:s3:::` + bucketName + `","arn:aws:s3:::` + bucketName + `/*"]}]}`
// Create user
_, err = iamClient.CreateUser(&iam.CreateUserInput{
UserName: aws.String(userName),
})
require.NoError(t, err)
// Create access key for the user
keyResp, err := iamClient.CreateAccessKey(&iam.CreateAccessKeyInput{
UserName: aws.String(userName),
})
require.NoError(t, err)
accessKeyId := *keyResp.AccessKey.AccessKeyId
secretKey := *keyResp.AccessKey.SecretAccessKey
// Create an S3 client with the user's credentials
userS3Client := createS3Client(t, accessKeyId, secretKey)
// Create group
_, err = iamClient.CreateGroup(&iam.CreateGroupInput{
GroupName: aws.String(groupName),
})
require.NoError(t, err)
// Create policy
createPolicyResp, err := iamClient.CreatePolicy(&iam.CreatePolicyInput{
PolicyName: aws.String(policyName),
PolicyDocument: aws.String(policyDoc),
})
require.NoError(t, err)
policyArn := createPolicyResp.Policy.Arn
// Cleanup in correct order: remove user from group, detach policy,
// delete access key, delete user, delete group, delete policy
t.Cleanup(func() {
if _, err := iamClient.RemoveUserFromGroup(&iam.RemoveUserFromGroupInput{
GroupName: aws.String(groupName),
UserName: aws.String(userName),
}); err != nil {
t.Logf("cleanup: failed to remove user from group: %v", err)
}
if _, err := iamClient.DetachGroupPolicy(&iam.DetachGroupPolicyInput{
GroupName: aws.String(groupName),
PolicyArn: policyArn,
}); err != nil {
t.Logf("cleanup: failed to detach group policy: %v", err)
}
if _, err := iamClient.DeleteAccessKey(&iam.DeleteAccessKeyInput{
UserName: aws.String(userName),
AccessKeyId: keyResp.AccessKey.AccessKeyId,
}); err != nil {
t.Logf("cleanup: failed to delete access key: %v", err)
}
if _, err := iamClient.DeleteUser(&iam.DeleteUserInput{UserName: aws.String(userName)}); err != nil {
t.Logf("cleanup: failed to delete user: %v", err)
}
if _, err := iamClient.DeleteGroup(&iam.DeleteGroupInput{GroupName: aws.String(groupName)}); err != nil {
t.Logf("cleanup: failed to delete group: %v", err)
}
if _, err := iamClient.DeletePolicy(&iam.DeletePolicyInput{PolicyArn: policyArn}); err != nil {
t.Logf("cleanup: failed to delete policy: %v", err)
}
})
// Register bucket cleanup on parent test with admin credentials
// (userS3Client may lack permissions by cleanup time)
adminS3, err := framework.CreateS3ClientWithJWT("admin-user", "TestAdminRole")
require.NoError(t, err)
t.Cleanup(func() {
if _, err := adminS3.DeleteObject(&s3.DeleteObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String("test-key"),
}); err != nil {
t.Logf("cleanup: failed to delete object: %v", err)
}
if _, err := adminS3.DeleteBucket(&s3.DeleteBucketInput{Bucket: aws.String(bucketName)}); err != nil {
t.Logf("cleanup: failed to delete bucket: %v", err)
}
})
t.Run("user_without_group_denied", func(t *testing.T) {
// User has no policies and is not in any group — should be denied
_, err := userS3Client.CreateBucket(&s3.CreateBucketInput{
Bucket: aws.String(bucketName),
})
require.Error(t, err, "User without any policies should be denied")
awsErr, ok := err.(awserr.Error)
require.True(t, ok, "Expected awserr.Error")
assert.Equal(t, "AccessDenied", awsErr.Code())
})
t.Run("user_with_group_policy_allowed", func(t *testing.T) {
// Attach policy to group
_, err := iamClient.AttachGroupPolicy(&iam.AttachGroupPolicyInput{
GroupName: aws.String(groupName),
PolicyArn: policyArn,
})
require.NoError(t, err)
// Add user to group
_, err = iamClient.AddUserToGroup(&iam.AddUserToGroupInput{
GroupName: aws.String(groupName),
UserName: aws.String(userName),
})
require.NoError(t, err)
// Wait for policy propagation, then create bucket
require.Eventually(t, func() bool {
_, err = userS3Client.CreateBucket(&s3.CreateBucketInput{
Bucket: aws.String(bucketName),
})
return err == nil
}, 10*time.Second, 500*time.Millisecond, "User with group policy should be allowed")
// Should also be able to put/get objects
_, err = userS3Client.PutObject(&s3.PutObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String("test-key"),
Body: aws.ReadSeekCloser(strings.NewReader("test-data")),
})
require.NoError(t, err, "User should be able to put objects through group policy")
})
t.Run("user_removed_from_group_denied", func(t *testing.T) {
// Remove user from group
_, err := iamClient.RemoveUserFromGroup(&iam.RemoveUserFromGroupInput{
GroupName: aws.String(groupName),
UserName: aws.String(userName),
})
require.NoError(t, err)
// Wait for policy propagation — user should now be denied
var lastErr error
require.Eventually(t, func() bool {
_, lastErr = userS3Client.ListObjects(&s3.ListObjectsInput{
Bucket: aws.String(bucketName),
})
return lastErr != nil
}, 10*time.Second, 500*time.Millisecond, "User removed from group should be denied")
awsErr, ok := lastErr.(awserr.Error)
require.True(t, ok, "Expected awserr.Error")
assert.Equal(t, "AccessDenied", awsErr.Code())
})
}
// TestIAMGroupDisabledPolicyEnforcement tests that disabled groups do not contribute policies.
// Uses the raw IAM API (callIAMAPI) since the AWS SDK doesn't support custom group status.
func TestIAMGroupDisabledPolicyEnforcement(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
if !isSeaweedFSRunning(t) {
t.Skip("SeaweedFS is not running at", TestIAMEndpoint)
}
framework := NewS3IAMTestFramework(t)
defer framework.Cleanup()
iamClient, err := framework.CreateIAMClientWithJWT("admin-user", "TestAdminRole")
require.NoError(t, err)
groupName := "test-disabled-group"
userName := "test-disabled-grp-user"
policyName := "test-disabled-grp-policy"
bucketName := "test-disabled-grp-bucket"
policyDoc := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:*"],"Resource":["arn:aws:s3:::` + bucketName + `","arn:aws:s3:::` + bucketName + `/*"]}]}`
// Create user, group, policy
_, err = iamClient.CreateUser(&iam.CreateUserInput{UserName: aws.String(userName)})
require.NoError(t, err)
keyResp, err := iamClient.CreateAccessKey(&iam.CreateAccessKeyInput{UserName: aws.String(userName)})
require.NoError(t, err)
_, err = iamClient.CreateGroup(&iam.CreateGroupInput{GroupName: aws.String(groupName)})
require.NoError(t, err)
createPolicyResp, err := iamClient.CreatePolicy(&iam.CreatePolicyInput{
PolicyName: aws.String(policyName), PolicyDocument: aws.String(policyDoc),
})
require.NoError(t, err)
// Cleanup in correct order: remove user from group, detach policy,
// delete access key, delete user, delete group, delete policy
t.Cleanup(func() {
if _, err := iamClient.RemoveUserFromGroup(&iam.RemoveUserFromGroupInput{
GroupName: aws.String(groupName), UserName: aws.String(userName),
}); err != nil {
t.Logf("cleanup: failed to remove user from group: %v", err)
}
if _, err := iamClient.DetachGroupPolicy(&iam.DetachGroupPolicyInput{
GroupName: aws.String(groupName),
PolicyArn: aws.String("arn:aws:iam:::policy/" + policyName),
}); err != nil {
t.Logf("cleanup: failed to detach group policy: %v", err)
}
if _, err := iamClient.DeleteAccessKey(&iam.DeleteAccessKeyInput{
UserName: aws.String(userName), AccessKeyId: keyResp.AccessKey.AccessKeyId,
}); err != nil {
t.Logf("cleanup: failed to delete access key: %v", err)
}
if _, err := iamClient.DeleteUser(&iam.DeleteUserInput{UserName: aws.String(userName)}); err != nil {
t.Logf("cleanup: failed to delete user: %v", err)
}
if _, err := iamClient.DeleteGroup(&iam.DeleteGroupInput{GroupName: aws.String(groupName)}); err != nil {
t.Logf("cleanup: failed to delete group: %v", err)
}
if _, err := iamClient.DeletePolicy(&iam.DeletePolicyInput{PolicyArn: createPolicyResp.Policy.Arn}); err != nil {
t.Logf("cleanup: failed to delete policy: %v", err)
}
})
// Setup: attach policy, add user, create bucket with admin
_, err = iamClient.AttachGroupPolicy(&iam.AttachGroupPolicyInput{
GroupName: aws.String(groupName), PolicyArn: createPolicyResp.Policy.Arn,
})
require.NoError(t, err)
_, err = iamClient.AddUserToGroup(&iam.AddUserToGroupInput{
GroupName: aws.String(groupName), UserName: aws.String(userName),
})
require.NoError(t, err)
userS3Client := createS3Client(t, *keyResp.AccessKey.AccessKeyId, *keyResp.AccessKey.SecretAccessKey)
// Create bucket using admin first so we can test listing
adminS3, err := framework.CreateS3ClientWithJWT("admin-user", "TestAdminRole")
require.NoError(t, err)
_, err = adminS3.CreateBucket(&s3.CreateBucketInput{Bucket: aws.String(bucketName)})
require.NoError(t, err)
defer adminS3.DeleteBucket(&s3.DeleteBucketInput{Bucket: aws.String(bucketName)})
t.Run("enabled_group_allows_access", func(t *testing.T) {
require.Eventually(t, func() bool {
_, err := userS3Client.ListObjects(&s3.ListObjectsInput{
Bucket: aws.String(bucketName),
})
return err == nil
}, 10*time.Second, 500*time.Millisecond, "User in enabled group should have access")
})
t.Run("disabled_group_denies_access", func(t *testing.T) {
// Disable group via raw IAM API (no SDK support for this extension)
resp, err := callIAMAPIAuthenticated(t, framework, "UpdateGroup", url.Values{
"GroupName": {groupName},
"Disabled": {"true"},
})
require.NoError(t, err)
defer resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode, "UpdateGroup (disable) should return 200")
// Wait for propagation — user should be denied
var lastErr error
require.Eventually(t, func() bool {
_, lastErr = userS3Client.ListObjects(&s3.ListObjectsInput{
Bucket: aws.String(bucketName),
})
return lastErr != nil
}, 10*time.Second, 500*time.Millisecond, "User in disabled group should be denied access")
awsErr, ok := lastErr.(awserr.Error)
require.True(t, ok, "Expected awserr.Error")
assert.Equal(t, "AccessDenied", awsErr.Code())
})
t.Run("re_enabled_group_restores_access", func(t *testing.T) {
// Re-enable the group
resp, err := callIAMAPIAuthenticated(t, framework, "UpdateGroup", url.Values{
"GroupName": {groupName},
"Disabled": {"false"},
})
require.NoError(t, err)
defer resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode, "UpdateGroup (re-enable) should return 200")
// Wait for propagation — user should have access again
require.Eventually(t, func() bool {
_, err = userS3Client.ListObjects(&s3.ListObjectsInput{
Bucket: aws.String(bucketName),
})
return err == nil
}, 10*time.Second, 500*time.Millisecond, "User in re-enabled group should have access again")
})
}
// TestIAMGroupUserDeletionSideEffect tests that deleting a user removes them from all groups.
func TestIAMGroupUserDeletionSideEffect(t *testing.T) {
framework := NewS3IAMTestFramework(t)
defer framework.Cleanup()
iamClient, err := framework.CreateIAMClientWithJWT("admin-user", "TestAdminRole")
require.NoError(t, err)
groupName := "test-deletion-group"
userName := "test-deletion-user"
// Create group and user
_, err = iamClient.CreateGroup(&iam.CreateGroupInput{GroupName: aws.String(groupName)})
require.NoError(t, err)
defer iamClient.DeleteGroup(&iam.DeleteGroupInput{GroupName: aws.String(groupName)})
_, err = iamClient.CreateUser(&iam.CreateUserInput{UserName: aws.String(userName)})
require.NoError(t, err)
t.Cleanup(func() {
// Best-effort: user may already be deleted by the test
iamClient.DeleteUser(&iam.DeleteUserInput{UserName: aws.String(userName)})
})
// Add user to group
_, err = iamClient.AddUserToGroup(&iam.AddUserToGroupInput{
GroupName: aws.String(groupName),
UserName: aws.String(userName),
})
require.NoError(t, err)
// Verify user is in group
getResp, err := iamClient.GetGroup(&iam.GetGroupInput{GroupName: aws.String(groupName)})
require.NoError(t, err)
assert.Len(t, getResp.Users, 1, "Group should have 1 member before deletion")
// Delete the user
_, err = iamClient.DeleteUser(&iam.DeleteUserInput{UserName: aws.String(userName)})
require.NoError(t, err)
// Verify user was removed from the group
getResp, err = iamClient.GetGroup(&iam.GetGroupInput{GroupName: aws.String(groupName)})
require.NoError(t, err)
assert.Empty(t, getResp.Users, "Group should have no members after user deletion")
}
// TestIAMGroupMultipleGroups tests that a user can belong to multiple groups
// and inherits policies from all of them.
func TestIAMGroupMultipleGroups(t *testing.T) {
framework := NewS3IAMTestFramework(t)
defer framework.Cleanup()
iamClient, err := framework.CreateIAMClientWithJWT("admin-user", "TestAdminRole")
require.NoError(t, err)
group1 := "test-multi-group-1"
group2 := "test-multi-group-2"
userName := "test-multi-group-user"
// Create two groups
_, err = iamClient.CreateGroup(&iam.CreateGroupInput{GroupName: aws.String(group1)})
require.NoError(t, err)
defer iamClient.DeleteGroup(&iam.DeleteGroupInput{GroupName: aws.String(group1)})
_, err = iamClient.CreateGroup(&iam.CreateGroupInput{GroupName: aws.String(group2)})
require.NoError(t, err)
defer iamClient.DeleteGroup(&iam.DeleteGroupInput{GroupName: aws.String(group2)})
// Create user
_, err = iamClient.CreateUser(&iam.CreateUserInput{UserName: aws.String(userName)})
require.NoError(t, err)
defer func() {
iamClient.RemoveUserFromGroup(&iam.RemoveUserFromGroupInput{
GroupName: aws.String(group1), UserName: aws.String(userName),
})
iamClient.RemoveUserFromGroup(&iam.RemoveUserFromGroupInput{
GroupName: aws.String(group2), UserName: aws.String(userName),
})
iamClient.DeleteUser(&iam.DeleteUserInput{UserName: aws.String(userName)})
}()
// Add user to both groups
_, err = iamClient.AddUserToGroup(&iam.AddUserToGroupInput{
GroupName: aws.String(group1), UserName: aws.String(userName),
})
require.NoError(t, err)
_, err = iamClient.AddUserToGroup(&iam.AddUserToGroupInput{
GroupName: aws.String(group2), UserName: aws.String(userName),
})
require.NoError(t, err)
// Verify user appears in both groups
resp, err := iamClient.ListGroupsForUser(&iam.ListGroupsForUserInput{
UserName: aws.String(userName),
})
require.NoError(t, err)
groupNames := make(map[string]bool)
for _, g := range resp.Groups {
groupNames[*g.GroupName] = true
}
assert.True(t, groupNames[group1], "User should be in group 1")
assert.True(t, groupNames[group2], "User should be in group 2")
}
// --- Response types for raw IAM API calls ---
type CreateGroupResponse struct {
XMLName xml.Name `xml:"CreateGroupResponse"`
CreateGroupResult struct {
Group struct {
GroupName string `xml:"GroupName"`
} `xml:"Group"`
} `xml:"CreateGroupResult"`
}
type ListGroupsResponse struct {
XMLName xml.Name `xml:"ListGroupsResponse"`
ListGroupsResult struct {
Groups []struct {
GroupName string `xml:"GroupName"`
} `xml:"Groups>member"`
} `xml:"ListGroupsResult"`
}
// callIAMAPIAuthenticated sends an authenticated raw IAM API request using the
// framework's JWT token. This is needed for custom extensions not in the AWS SDK
// (like UpdateGroup with Disabled parameter).
func callIAMAPIAuthenticated(_ *testing.T, framework *S3IAMTestFramework, action string, params url.Values) (*http.Response, error) {
params.Set("Action", action)
req, err := http.NewRequest(http.MethodPost, TestIAMEndpoint+"/",
strings.NewReader(params.Encode()))
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
token, err := framework.generateSTSSessionToken("admin-user", "TestAdminRole", time.Hour, "", nil)
if err != nil {
return nil, err
}
client := &http.Client{
Timeout: 30 * time.Second,
Transport: &BearerTokenTransport{Token: token},
}
return client.Do(req)
}
// TestIAMGroupRawAPI tests group operations using raw HTTP IAM API calls,
// verifying XML response format for group operations.
func TestIAMGroupRawAPI(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
if !isSeaweedFSRunning(t) {
t.Skip("SeaweedFS is not running at", TestIAMEndpoint)
}
framework := NewS3IAMTestFramework(t)
defer framework.Cleanup()
groupName := "test-raw-api-group"
t.Run("create_group_raw", func(t *testing.T) {
resp, err := callIAMAPIAuthenticated(t, framework, "CreateGroup", url.Values{
"GroupName": {groupName},
})
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
var createResp CreateGroupResponse
err = xml.Unmarshal(body, &createResp)
require.NoError(t, err)
assert.Equal(t, groupName, createResp.CreateGroupResult.Group.GroupName)
})
t.Run("list_groups_raw", func(t *testing.T) {
resp, err := callIAMAPIAuthenticated(t, framework, "ListGroups", url.Values{})
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
var listResp ListGroupsResponse
err = xml.Unmarshal(body, &listResp)
require.NoError(t, err)
found := false
for _, g := range listResp.ListGroupsResult.Groups {
if g.GroupName == groupName {
found = true
break
}
}
assert.True(t, found, "Created group should appear in raw ListGroups")
})
t.Run("delete_group_raw", func(t *testing.T) {
resp, err := callIAMAPIAuthenticated(t, framework, "DeleteGroup", url.Values{
"GroupName": {groupName},
})
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
})
}
// createS3Client creates an S3 client with static credentials
func createS3Client(t *testing.T, accessKey, secretKey string) *s3.S3 {
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1"),
Endpoint: aws.String(TestS3Endpoint),
Credentials: credentials.NewStaticCredentials(accessKey, secretKey, ""),
DisableSSL: aws.Bool(true),
S3ForcePathStyle: aws.Bool(true),
})
require.NoError(t, err)
return s3.New(sess)
}
+6
View File
@@ -80,6 +80,11 @@ type AccessKeyInfo struct {
CreatedAt time.Time `json:"created_at"`
}
type CreateAccessKeyRequest struct {
AccessKey string `json:"access_key"`
SecretKey string `json:"secret_key"`
}
type UpdateAccessKeyStatusRequest struct {
Status string `json:"status" binding:"required"`
}
@@ -90,6 +95,7 @@ type UserDetails struct {
Actions []string `json:"actions"`
PolicyNames []string `json:"policy_names"`
AccessKeys []AccessKeyInfo `json:"access_keys"`
Groups []string `json:"groups"`
}
type FilerNode struct {
+144 -20
View File
@@ -2,7 +2,9 @@ package dash
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"sort"
"strings"
@@ -371,8 +373,21 @@ func (s *AdminServer) GetCredentialManager() *credential.CredentialManager {
// InvalidateCache method moved to cluster_topology.go
// GetS3BucketsData retrieves all Object Store buckets and aggregates total storage metrics
func (s *AdminServer) GetS3BucketsData() (S3BucketsData, error) {
// GetS3BucketsData retrieves Object Store buckets with pagination and sorting
func (s *AdminServer) GetS3BucketsData(page, pageSize int, sortBy, sortOrder string) (S3BucketsData, error) {
if page < 1 {
page = 1
}
if pageSize < 1 || pageSize > 1000 {
pageSize = 100
}
if sortBy == "" {
sortBy = "name"
}
if sortOrder == "" {
sortOrder = "asc"
}
buckets, err := s.GetS3Buckets()
if err != nil {
return S3BucketsData{}, err
@@ -383,14 +398,97 @@ func (s *AdminServer) GetS3BucketsData() (S3BucketsData, error) {
totalSize += bucket.PhysicalSize
}
totalBuckets := len(buckets)
// Sort buckets
s.sortBuckets(buckets, sortBy, sortOrder)
// Calculate pagination
totalPages := (totalBuckets + pageSize - 1) / pageSize
if totalPages == 0 {
totalPages = 1
}
if page > totalPages {
page = totalPages
}
startIndex := (page - 1) * pageSize
endIndex := startIndex + pageSize
if startIndex >= totalBuckets {
buckets = []S3Bucket{}
} else {
if endIndex > totalBuckets {
endIndex = totalBuckets
}
buckets = buckets[startIndex:endIndex]
}
return S3BucketsData{
Buckets: buckets,
TotalBuckets: len(buckets),
TotalBuckets: totalBuckets,
TotalSize: totalSize,
LastUpdated: time.Now(),
CurrentPage: page,
TotalPages: totalPages,
PageSize: pageSize,
SortBy: sortBy,
SortOrder: sortOrder,
}, nil
}
// sortBuckets sorts the bucket slice in place by the given field and order
func (s *AdminServer) sortBuckets(buckets []S3Bucket, sortBy, sortOrder string) {
desc := sortOrder == "desc"
sort.Slice(buckets, func(i, j int) bool {
a, b := buckets[i], buckets[j]
switch sortBy {
case "owner":
if a.Owner != b.Owner {
if desc {
return a.Owner > b.Owner
}
return a.Owner < b.Owner
}
case "created":
if !a.CreatedAt.Equal(b.CreatedAt) {
if desc {
return a.CreatedAt.After(b.CreatedAt)
}
return a.CreatedAt.Before(b.CreatedAt)
}
case "objects":
if a.ObjectCount != b.ObjectCount {
if desc {
return a.ObjectCount > b.ObjectCount
}
return a.ObjectCount < b.ObjectCount
}
case "logical_size":
if a.LogicalSize != b.LogicalSize {
if desc {
return a.LogicalSize > b.LogicalSize
}
return a.LogicalSize < b.LogicalSize
}
case "physical_size":
if a.PhysicalSize != b.PhysicalSize {
if desc {
return a.PhysicalSize > b.PhysicalSize
}
return a.PhysicalSize < b.PhysicalSize
}
}
// Tie-breaker: sort by name (also the default/primary for sortBy=="name")
if a.Name != b.Name {
if desc {
return a.Name > b.Name
}
return a.Name < b.Name
}
return false
})
}
// GetS3Buckets retrieves all Object Store buckets from the filer and collects size/object data from collections
func (s *AdminServer) GetS3Buckets() ([]S3Bucket, error) {
var buckets []S3Bucket
@@ -406,28 +504,48 @@ func (s *AdminServer) GetS3Buckets() ([]S3Bucket, error) {
// Now list buckets from the filer and match with collection data
err = s.WithFilerClient(func(client filer_pb.SeaweedFilerClient) error {
// List buckets by looking at the buckets directory
stream, err := client.ListEntries(context.Background(), &filer_pb.ListEntriesRequest{
Directory: filerConfig.BucketsPath,
Prefix: "",
StartFromFileName: "",
InclusiveStartFrom: false,
Limit: 1000,
})
if err != nil {
return err
}
// Paginate through all buckets in the buckets directory
const listPageSize = 1000
startFrom := ""
var snapshotTsNs int64
for {
resp, err := stream.Recv()
stream, err := client.ListEntries(context.Background(), &filer_pb.ListEntriesRequest{
Directory: filerConfig.BucketsPath,
Prefix: "",
StartFromFileName: startFrom,
InclusiveStartFrom: false,
Limit: listPageSize,
SnapshotTsNs: snapshotTsNs,
})
if err != nil {
if err.Error() == "EOF" {
break
}
return err
}
if resp.Entry != nil && resp.Entry.IsDirectory {
pageCount := 0
lastName := ""
for {
resp, err := stream.Recv()
if err != nil {
if errors.Is(err, io.EOF) {
break
}
return err
}
if snapshotTsNs == 0 && resp.SnapshotTsNs != 0 {
snapshotTsNs = resp.SnapshotTsNs
}
if resp.Entry == nil {
continue
}
lastName = resp.Entry.Name
pageCount++
if !resp.Entry.IsDirectory {
continue
}
bucketName := resp.Entry.Name
if strings.HasPrefix(bucketName, ".") {
// Skip internal/system directories from Object Store bucket listing.
@@ -502,6 +620,12 @@ func (s *AdminServer) GetS3Buckets() ([]S3Bucket, error) {
}
buckets = append(buckets, bucket)
}
// If we received fewer entries than the page size, we've listed everything
if pageCount < listPageSize {
break
}
startFrom = lastName
}
return nil
+10 -1
View File
@@ -26,6 +26,15 @@ type S3BucketsData struct {
TotalBuckets int `json:"total_buckets"`
TotalSize int64 `json:"total_size"`
LastUpdated time.Time `json:"last_updated"`
// Pagination
CurrentPage int `json:"current_page"`
TotalPages int `json:"total_pages"`
PageSize int `json:"page_size"`
// Sorting
SortBy string `json:"sort_by"`
SortOrder string `json:"sort_order"`
}
type CreateBucketRequest struct {
@@ -48,7 +57,7 @@ type CreateBucketRequest struct {
func (s *AdminServer) ShowS3Buckets(w http.ResponseWriter, r *http.Request) {
username := UsernameFromContext(r.Context())
data, err := s.GetS3BucketsData()
data, err := s.GetS3BucketsData(1, 100, "name", "asc")
if err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to get Object Store buckets: "+err.Error())
return
+250
View File
@@ -0,0 +1,250 @@
package dash
import (
"context"
"errors"
"fmt"
"github.com/seaweedfs/seaweedfs/weed/credential"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/iam_pb"
)
// cloneGroup creates a deep copy of an iam_pb.Group to avoid mutating stored state.
func cloneGroup(g *iam_pb.Group) *iam_pb.Group {
clone := &iam_pb.Group{
Name: g.Name,
Disabled: g.Disabled,
}
if g.Members != nil {
clone.Members = make([]string, len(g.Members))
copy(clone.Members, g.Members)
}
if g.PolicyNames != nil {
clone.PolicyNames = make([]string, len(g.PolicyNames))
copy(clone.PolicyNames, g.PolicyNames)
}
return clone
}
func (s *AdminServer) GetGroups(ctx context.Context) ([]GroupData, error) {
if s.credentialManager == nil {
return nil, fmt.Errorf("credential manager not available")
}
groupNames, err := s.credentialManager.ListGroups(ctx)
if err != nil {
return nil, fmt.Errorf("failed to list groups: %w", err)
}
var groups []GroupData
for _, name := range groupNames {
g, err := s.credentialManager.GetGroup(ctx, name)
if err != nil {
if errors.Is(err, credential.ErrGroupNotFound) {
glog.V(1).Infof("Group %s listed but not found, skipping", name)
continue
}
return nil, fmt.Errorf("failed to get group %s: %w", name, err)
}
status := "enabled"
if g.Disabled {
status = "disabled"
}
groups = append(groups, GroupData{
Name: g.Name,
MemberCount: len(g.Members),
PolicyCount: len(g.PolicyNames),
Status: status,
Members: g.Members,
PolicyNames: g.PolicyNames,
})
}
return groups, nil
}
func (s *AdminServer) GetGroupDetails(ctx context.Context, name string) (*GroupData, error) {
if s.credentialManager == nil {
return nil, fmt.Errorf("credential manager not available")
}
g, err := s.credentialManager.GetGroup(ctx, name)
if err != nil {
return nil, fmt.Errorf("failed to get group: %w", err)
}
status := "enabled"
if g.Disabled {
status = "disabled"
}
return &GroupData{
Name: g.Name,
MemberCount: len(g.Members),
PolicyCount: len(g.PolicyNames),
Status: status,
Members: g.Members,
PolicyNames: g.PolicyNames,
}, nil
}
func (s *AdminServer) CreateGroup(ctx context.Context, name string) (*GroupData, error) {
if s.credentialManager == nil {
return nil, fmt.Errorf("credential manager not available")
}
group := &iam_pb.Group{Name: name}
if err := s.credentialManager.CreateGroup(ctx, group); err != nil {
return nil, fmt.Errorf("failed to create group: %w", err)
}
glog.V(1).Infof("Created group %s", group.Name)
return &GroupData{
Name: group.Name,
Status: "enabled",
}, nil
}
func (s *AdminServer) DeleteGroup(ctx context.Context, name string) error {
if s.credentialManager == nil {
return fmt.Errorf("credential manager not available")
}
// Check for members and attached policies before deleting (same guards as IAM handlers)
g, err := s.credentialManager.GetGroup(ctx, name)
if err != nil {
return fmt.Errorf("failed to get group: %w", err)
}
if len(g.Members) > 0 {
return fmt.Errorf("cannot delete group %s: group has %d member(s): %w", name, len(g.Members), credential.ErrGroupNotEmpty)
}
if len(g.PolicyNames) > 0 {
return fmt.Errorf("cannot delete group %s: group has %d attached policy(ies): %w", name, len(g.PolicyNames), credential.ErrGroupNotEmpty)
}
if err := s.credentialManager.DeleteGroup(ctx, name); err != nil {
return fmt.Errorf("failed to delete group: %w", err)
}
glog.V(1).Infof("Deleted group %s", name)
return nil
}
func (s *AdminServer) AddGroupMember(ctx context.Context, groupName, username string) error {
if s.credentialManager == nil {
return fmt.Errorf("credential manager not available")
}
g, err := s.credentialManager.GetGroup(ctx, groupName)
if err != nil {
return fmt.Errorf("failed to get group: %w", err)
}
g = cloneGroup(g)
if _, err := s.credentialManager.GetUser(ctx, username); err != nil {
return fmt.Errorf("user %s not found: %w", username, err)
}
for _, m := range g.Members {
if m == username {
return nil // already a member
}
}
g.Members = append(g.Members, username)
if err := s.credentialManager.UpdateGroup(ctx, g); err != nil {
return fmt.Errorf("failed to update group: %w", err)
}
glog.V(1).Infof("Added user %s to group %s", username, groupName)
return nil
}
func (s *AdminServer) RemoveGroupMember(ctx context.Context, groupName, username string) error {
if s.credentialManager == nil {
return fmt.Errorf("credential manager not available")
}
g, err := s.credentialManager.GetGroup(ctx, groupName)
if err != nil {
return fmt.Errorf("failed to get group: %w", err)
}
g = cloneGroup(g)
found := false
var newMembers []string
for _, m := range g.Members {
if m == username {
found = true
} else {
newMembers = append(newMembers, m)
}
}
if !found {
return fmt.Errorf("user %s is not a member of group %s: %w", username, groupName, credential.ErrUserNotInGroup)
}
g.Members = newMembers
if err := s.credentialManager.UpdateGroup(ctx, g); err != nil {
return fmt.Errorf("failed to update group: %w", err)
}
glog.V(1).Infof("Removed user %s from group %s", username, groupName)
return nil
}
func (s *AdminServer) AttachGroupPolicy(ctx context.Context, groupName, policyName string) error {
if s.credentialManager == nil {
return fmt.Errorf("credential manager not available")
}
g, err := s.credentialManager.GetGroup(ctx, groupName)
if err != nil {
return fmt.Errorf("failed to get group: %w", err)
}
g = cloneGroup(g)
if _, err := s.credentialManager.GetPolicy(ctx, policyName); err != nil {
return fmt.Errorf("policy %s not found: %w", policyName, err)
}
for _, p := range g.PolicyNames {
if p == policyName {
return nil // already attached
}
}
g.PolicyNames = append(g.PolicyNames, policyName)
if err := s.credentialManager.UpdateGroup(ctx, g); err != nil {
return fmt.Errorf("failed to update group: %w", err)
}
glog.V(1).Infof("Attached policy %s to group %s", policyName, groupName)
return nil
}
func (s *AdminServer) DetachGroupPolicy(ctx context.Context, groupName, policyName string) error {
if s.credentialManager == nil {
return fmt.Errorf("credential manager not available")
}
g, err := s.credentialManager.GetGroup(ctx, groupName)
if err != nil {
return fmt.Errorf("failed to get group: %w", err)
}
g = cloneGroup(g)
found := false
var newPolicies []string
for _, p := range g.PolicyNames {
if p == policyName {
found = true
} else {
newPolicies = append(newPolicies, p)
}
}
if !found {
return fmt.Errorf("policy %s is not attached to group %s: %w", policyName, groupName, credential.ErrPolicyNotAttached)
}
g.PolicyNames = newPolicies
if err := s.credentialManager.UpdateGroup(ctx, g); err != nil {
return fmt.Errorf("failed to update group: %w", err)
}
glog.V(1).Infof("Detached policy %s from group %s", policyName, groupName)
return nil
}
func (s *AdminServer) SetGroupStatus(ctx context.Context, groupName string, enabled bool) error {
if s.credentialManager == nil {
return fmt.Errorf("credential manager not available")
}
g, err := s.credentialManager.GetGroup(ctx, groupName)
if err != nil {
return fmt.Errorf("failed to get group: %w", err)
}
g = cloneGroup(g)
g.Disabled = !enabled
if err := s.credentialManager.UpdateGroup(ctx, g); err != nil {
return fmt.Errorf("failed to update group: %w", err)
}
glog.V(1).Infof("Set group %s status to enabled=%v", groupName, enabled)
return nil
}
-47
View File
@@ -235,53 +235,6 @@ func (s *AdminServer) GetPluginSchedulerStatusAPI(w http.ResponseWriter, r *http
writeJSON(w, http.StatusOK, response)
}
// GetPluginSchedulerConfigAPI returns scheduler configuration.
func (s *AdminServer) GetPluginSchedulerConfigAPI(w http.ResponseWriter, r *http.Request) {
pluginSvc := s.GetPlugin()
if pluginSvc == nil {
writeJSONError(w, http.StatusNotFound, "plugin is not enabled")
return
}
writeJSON(w, http.StatusOK, pluginSvc.GetSchedulerConfig())
}
// UpdatePluginSchedulerConfigAPI updates scheduler configuration.
func (s *AdminServer) UpdatePluginSchedulerConfigAPI(w http.ResponseWriter, r *http.Request) {
pluginSvc := s.GetPlugin()
if pluginSvc == nil {
writeJSONError(w, http.StatusNotFound, "plugin is not enabled")
return
}
var req struct {
IdleSleepSeconds *int32 `json:"idle_sleep_seconds"`
}
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
if errors.Is(err, io.EOF) {
writeJSONError(w, http.StatusBadRequest, "request body is required")
return
}
writeJSONError(w, http.StatusBadRequest, "invalid request body: "+err.Error())
return
}
if req.IdleSleepSeconds == nil {
writeJSONError(w, http.StatusBadRequest, "idle_sleep_seconds is required")
return
}
updated, err := pluginSvc.UpdateSchedulerConfig(plugin.SchedulerConfig{
IdleSleepSeconds: *req.IdleSleepSeconds,
})
if err != nil {
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
writeJSON(w, http.StatusOK, updated)
}
// RequestPluginJobTypeSchemaAPI asks a worker for one job type schema.
func (s *AdminServer) RequestPluginJobTypeSchemaAPI(w http.ResponseWriter, r *http.Request) {
jobType := strings.TrimSpace(mux.Vars(r)["jobType"])
+24
View File
@@ -589,6 +589,30 @@ type UpdateServiceAccountRequest struct {
Expiration string `json:"expiration,omitempty"`
}
// Group management structures
type GroupData struct {
Name string `json:"name"`
MemberCount int `json:"member_count"`
PolicyCount int `json:"policy_count"`
Status string `json:"status"` // "enabled" or "disabled"
Members []string `json:"members"`
PolicyNames []string `json:"policy_names"`
}
type GroupsPageData struct {
Username string `json:"username"`
Groups []GroupData `json:"groups"`
TotalGroups int `json:"total_groups"`
ActiveGroups int `json:"active_groups"`
AvailableUsers []string `json:"available_users"`
AvailablePolicies []string `json:"available_policies"`
LastUpdated time.Time `json:"last_updated"`
}
type CreateGroupRequest struct {
Name string `json:"name"`
}
// STS Configuration display types
type STSConfigData struct {
Enabled bool `json:"enabled"`
+64 -5
View File
@@ -4,13 +4,21 @@ import (
"context"
"crypto/rand"
"encoding/base64"
"errors"
"fmt"
"strings"
"time"
"github.com/seaweedfs/seaweedfs/weed/credential"
"github.com/seaweedfs/seaweedfs/weed/pb/iam_pb"
)
var (
ErrAccessKeyInUse = errors.New("access key already in use")
ErrUserNotFound = errors.New("user not found")
ErrInvalidInput = errors.New("invalid input")
)
// CreateObjectStoreUser creates a new user using the credential manager
func (s *AdminServer) CreateObjectStoreUser(req CreateUserRequest) (*ObjectStoreUser, error) {
if s.credentialManager == nil {
@@ -187,6 +195,24 @@ func (s *AdminServer) GetObjectStoreUserDetails(username string) (*UserDetails,
details.Email = identity.Account.EmailAddress
}
// Look up groups the user belongs to
groupNames, err := s.credentialManager.ListGroups(ctx)
if err != nil {
return nil, fmt.Errorf("failed to list groups: %w", err)
}
for _, gName := range groupNames {
g, err := s.credentialManager.GetGroup(ctx, gName)
if err != nil {
return nil, fmt.Errorf("failed to get group %s: %w", gName, err)
}
for _, member := range g.Members {
if member == username {
details.Groups = append(details.Groups, gName)
break
}
}
}
// Convert credentials to access key info
for _, cred := range identity.Credentials {
details.AccessKeys = append(details.AccessKeys, AccessKeyInfo{
@@ -201,7 +227,7 @@ func (s *AdminServer) GetObjectStoreUserDetails(username string) (*UserDetails,
}
// CreateAccessKey creates a new access key for a user
func (s *AdminServer) CreateAccessKey(username string) (*AccessKeyInfo, error) {
func (s *AdminServer) CreateAccessKey(username string, req *CreateAccessKeyRequest) (*AccessKeyInfo, error) {
if s.credentialManager == nil {
return nil, fmt.Errorf("credential manager not available")
}
@@ -212,14 +238,41 @@ func (s *AdminServer) CreateAccessKey(username string) (*AccessKeyInfo, error) {
_, err := s.credentialManager.GetUser(ctx, username)
if err != nil {
if err == credential.ErrUserNotFound {
return nil, fmt.Errorf("user %s not found", username)
return nil, fmt.Errorf("user %s: %w", username, ErrUserNotFound)
}
return nil, fmt.Errorf("failed to get user: %w", err)
}
// Generate new access key
accessKey := generateAccessKey()
secretKey := generateSecretKey()
if req == nil {
req = &CreateAccessKeyRequest{}
}
// Validate provided keys
if req.AccessKey != "" && (len(req.AccessKey) < 4 || len(req.AccessKey) > 128) {
return nil, fmt.Errorf("access key must be between 4 and 128 characters: %w", ErrInvalidInput)
}
if req.SecretKey != "" && (len(req.SecretKey) < 8 || len(req.SecretKey) > 128) {
return nil, fmt.Errorf("secret key must be between 8 and 128 characters: %w", ErrInvalidInput)
}
// Use provided keys or generate new ones
accessKey := req.AccessKey
if accessKey == "" {
accessKey = generateAccessKey()
}
secretKey := req.SecretKey
if secretKey == "" {
secretKey = generateSecretKey()
}
// Verify access key is globally unique
existingUser, err := s.credentialManager.GetUserByAccessKey(ctx, accessKey)
if existingUser != nil {
return nil, ErrAccessKeyInUse
}
if err != nil && !errors.Is(err, credential.ErrAccessKeyNotFound) && !isNotFoundError(err) {
return nil, fmt.Errorf("failed to check access key uniqueness: %w", err)
}
credential := &iam_pb.Credential{
AccessKey: accessKey,
@@ -364,6 +417,12 @@ func (s *AdminServer) UpdateUserPolicies(username string, actions []string) erro
return nil
}
// isNotFoundError checks for "not found" in the error message as a fallback
// for stores (e.g. gRPC) that don't return the credential.ErrAccessKeyNotFound sentinel.
func isNotFoundError(err error) bool {
return err != nil && strings.Contains(strings.ToLower(err.Error()), "not found")
}
// Helper functions for generating keys and IDs
func generateAccessKey() string {
// Generate 20-character access key (AWS standard)
+1
View File
@@ -457,6 +457,7 @@ func (s *AdminServer) GetClusterVolumeServers() (*ClusterVolumeServersData, erro
// Process disk information
for _, diskInfo := range node.DiskInfos {
vs.MaxVolumes += int(diskInfo.MaxVolumeCount)
vs.DiskCapacity += int64(diskInfo.MaxVolumeCount) * int64(volumeSizeLimitMB) * 1024 * 1024 // Use actual volume size limit
// Count regular volumes and calculate disk usage
+46 -7
View File
@@ -3,6 +3,7 @@ package handlers
import (
"net/http"
"net/url"
"strconv"
"time"
"github.com/gorilla/mux"
@@ -28,6 +29,7 @@ type AdminHandlers struct {
pluginHandlers *PluginHandlers
mqHandlers *MessageQueueHandlers
serviceAccountHandlers *ServiceAccountHandlers
groupHandlers *GroupHandlers
}
// NewAdminHandlers creates a new instance of AdminHandlers
@@ -40,6 +42,7 @@ func NewAdminHandlers(adminServer *dash.AdminServer, store sessions.Store) *Admi
pluginHandlers := NewPluginHandlers(adminServer)
mqHandlers := NewMessageQueueHandlers(adminServer)
serviceAccountHandlers := NewServiceAccountHandlers(adminServer)
groupHandlers := NewGroupHandlers(adminServer)
return &AdminHandlers{
adminServer: adminServer,
sessionStore: store,
@@ -51,6 +54,7 @@ func NewAdminHandlers(adminServer *dash.AdminServer, store sessions.Store) *Admi
pluginHandlers: pluginHandlers,
mqHandlers: mqHandlers,
serviceAccountHandlers: serviceAccountHandlers,
groupHandlers: groupHandlers,
}
}
@@ -104,6 +108,7 @@ func (h *AdminHandlers) registerUIRoutes(r *mux.Router) {
r.HandleFunc("/object-store/buckets/{bucket}", h.ShowBucketDetails).Methods(http.MethodGet)
r.HandleFunc("/object-store/users", h.userHandlers.ShowObjectStoreUsers).Methods(http.MethodGet)
r.HandleFunc("/object-store/policies", h.policyHandlers.ShowPolicies).Methods(http.MethodGet)
r.HandleFunc("/object-store/groups", h.groupHandlers.ShowGroups).Methods(http.MethodGet)
r.HandleFunc("/object-store/service-accounts", h.serviceAccountHandlers.ShowServiceAccounts).Methods(http.MethodGet)
r.HandleFunc("/object-store/s3tables/buckets", h.ShowS3TablesBuckets).Methods(http.MethodGet)
r.HandleFunc("/object-store/s3tables/buckets/{bucket}/namespaces", h.ShowS3TablesNamespaces).Methods(http.MethodGet)
@@ -185,6 +190,19 @@ func (h *AdminHandlers) registerAPIRoutes(api *mux.Router, enforceWrite bool) {
saApi.Handle("/{id}", wrapWrite(h.serviceAccountHandlers.UpdateServiceAccount)).Methods(http.MethodPut)
saApi.Handle("/{id}", wrapWrite(h.serviceAccountHandlers.DeleteServiceAccount)).Methods(http.MethodDelete)
groupsApi := api.PathPrefix("/groups").Subrouter()
groupsApi.HandleFunc("", h.groupHandlers.GetGroups).Methods(http.MethodGet)
groupsApi.Handle("", wrapWrite(h.groupHandlers.CreateGroup)).Methods(http.MethodPost)
groupsApi.HandleFunc("/{name}", h.groupHandlers.GetGroupDetails).Methods(http.MethodGet)
groupsApi.Handle("/{name}", wrapWrite(h.groupHandlers.DeleteGroup)).Methods(http.MethodDelete)
groupsApi.Handle("/{name}/status", wrapWrite(h.groupHandlers.SetGroupStatus)).Methods(http.MethodPut)
groupsApi.HandleFunc("/{name}/members", h.groupHandlers.GetGroupMembers).Methods(http.MethodGet)
groupsApi.Handle("/{name}/members", wrapWrite(h.groupHandlers.AddGroupMember)).Methods(http.MethodPost)
groupsApi.Handle("/{name}/members/{username}", wrapWrite(h.groupHandlers.RemoveGroupMember)).Methods(http.MethodDelete)
groupsApi.HandleFunc("/{name}/policies", h.groupHandlers.GetGroupPolicies).Methods(http.MethodGet)
groupsApi.Handle("/{name}/policies", wrapWrite(h.groupHandlers.AttachGroupPolicy)).Methods(http.MethodPost)
groupsApi.Handle("/{name}/policies/{policyName}", wrapWrite(h.groupHandlers.DetachGroupPolicy)).Methods(http.MethodDelete)
policyApi := api.PathPrefix("/object-store/policies").Subrouter()
policyApi.HandleFunc("", h.policyHandlers.GetPolicies).Methods(http.MethodGet)
policyApi.Handle("", wrapWrite(h.policyHandlers.CreatePolicy)).Methods(http.MethodPost)
@@ -229,8 +247,6 @@ func (h *AdminHandlers) registerAPIRoutes(api *mux.Router, enforceWrite bool) {
pluginApi.HandleFunc("/status", h.adminServer.GetPluginStatusAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/workers", h.adminServer.GetPluginWorkersAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/job-types", h.adminServer.GetPluginJobTypesAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/scheduler-config", h.adminServer.GetPluginSchedulerConfigAPI).Methods(http.MethodGet)
pluginApi.Handle("/scheduler-config", wrapWrite(h.adminServer.UpdatePluginSchedulerConfigAPI)).Methods(http.MethodPut)
pluginApi.HandleFunc("/jobs", h.adminServer.GetPluginJobsAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/jobs/{jobId}", h.adminServer.GetPluginJobAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/jobs/{jobId}/detail", h.adminServer.GetPluginJobDetailAPI).Methods(http.MethodGet)
@@ -278,8 +294,26 @@ func (h *AdminHandlers) ShowDashboard(w http.ResponseWriter, r *http.Request) {
// ShowS3Buckets renders the Object Store buckets management page
func (h *AdminHandlers) ShowS3Buckets(w http.ResponseWriter, r *http.Request) {
// Get Object Store buckets data from the server
s3Data := h.getS3BucketsData(r)
// Get pagination and sorting parameters from query string
page := 1
if p := r.URL.Query().Get("page"); p != "" {
if parsed, err := strconv.Atoi(p); err == nil && parsed > 0 {
page = parsed
}
}
pageSize := 100
if ps := r.URL.Query().Get("pageSize"); ps != "" {
if parsed, err := strconv.Atoi(ps); err == nil && parsed > 0 && parsed <= 1000 {
pageSize = parsed
}
}
sortBy := defaultQuery(r.URL.Query().Get("sortBy"), "name")
sortOrder := defaultQuery(r.URL.Query().Get("sortOrder"), "asc")
// Get Object Store buckets data with pagination
s3Data := h.getS3BucketsData(r, page, pageSize, sortBy, sortOrder)
username := h.getUsername(r)
// Render HTML template
@@ -446,15 +480,15 @@ func (h *AdminHandlers) ShowBucketDetails(w http.ResponseWriter, r *http.Request
writeJSON(w, http.StatusOK, details)
}
// getS3BucketsData retrieves Object Store buckets data from the server
func (h *AdminHandlers) getS3BucketsData(r *http.Request) dash.S3BucketsData {
// getS3BucketsData retrieves Object Store buckets data from the server with pagination
func (h *AdminHandlers) getS3BucketsData(r *http.Request, page, pageSize int, sortBy, sortOrder string) dash.S3BucketsData {
username := dash.UsernameFromContext(r.Context())
if username == "" {
username = "admin"
}
// Get Object Store buckets data
data, err := h.adminServer.GetS3BucketsData()
data, err := h.adminServer.GetS3BucketsData(page, pageSize, sortBy, sortOrder)
if err != nil {
// Return empty data on error
return dash.S3BucketsData{
@@ -463,6 +497,11 @@ func (h *AdminHandlers) getS3BucketsData(r *http.Request) dash.S3BucketsData {
TotalBuckets: 0,
TotalSize: 0,
LastUpdated: time.Now(),
CurrentPage: 1,
TotalPages: 1,
PageSize: pageSize,
SortBy: sortBy,
SortOrder: sortOrder,
}
}
+271
View File
@@ -0,0 +1,271 @@
package handlers
import (
"bytes"
"errors"
"net/http"
"time"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
"github.com/seaweedfs/seaweedfs/weed/admin/view/app"
"github.com/seaweedfs/seaweedfs/weed/admin/view/layout"
"github.com/seaweedfs/seaweedfs/weed/credential"
"github.com/seaweedfs/seaweedfs/weed/glog"
)
func groupErrorToHTTPStatus(err error) int {
if errors.Is(err, credential.ErrGroupNotFound) {
return http.StatusNotFound
}
if errors.Is(err, credential.ErrGroupAlreadyExists) {
return http.StatusConflict
}
if errors.Is(err, credential.ErrUserNotInGroup) {
return http.StatusBadRequest
}
if errors.Is(err, credential.ErrPolicyNotAttached) {
return http.StatusBadRequest
}
if errors.Is(err, credential.ErrUserNotFound) {
return http.StatusNotFound
}
if errors.Is(err, credential.ErrPolicyNotFound) {
return http.StatusNotFound
}
if errors.Is(err, credential.ErrGroupNotEmpty) {
return http.StatusConflict
}
return http.StatusInternalServerError
}
type GroupHandlers struct {
adminServer *dash.AdminServer
}
func NewGroupHandlers(adminServer *dash.AdminServer) *GroupHandlers {
return &GroupHandlers{adminServer: adminServer}
}
func (h *GroupHandlers) ShowGroups(w http.ResponseWriter, r *http.Request) {
data, err := h.getGroupsPageData(r)
if err != nil {
glog.Errorf("Failed to get groups data: %v", err)
writeJSONError(w, http.StatusInternalServerError, "Failed to load groups: "+err.Error())
return
}
var buf bytes.Buffer
component := app.Groups(data)
viewCtx := layout.NewViewContext(r, dash.UsernameFromContext(r.Context()), dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, component)
if err := layoutComponent.Render(r.Context(), &buf); err != nil {
glog.Errorf("Failed to render groups template: %v", err)
w.WriteHeader(http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "text/html")
_, _ = w.Write(buf.Bytes())
}
func (h *GroupHandlers) GetGroups(w http.ResponseWriter, r *http.Request) {
groups, err := h.adminServer.GetGroups(r.Context())
if err != nil {
glog.Errorf("Failed to get groups: %v", err)
writeJSONError(w, http.StatusInternalServerError, "Failed to get groups")
return
}
writeJSON(w, http.StatusOK, map[string]interface{}{"groups": groups})
}
func (h *GroupHandlers) CreateGroup(w http.ResponseWriter, r *http.Request) {
var req dash.CreateGroupRequest
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if req.Name == "" {
writeJSONError(w, http.StatusBadRequest, "Group name is required")
return
}
group, err := h.adminServer.CreateGroup(r.Context(), req.Name)
if err != nil {
glog.Errorf("Failed to create group: %v", err)
writeJSONError(w, groupErrorToHTTPStatus(err), "Failed to create group: "+err.Error())
return
}
writeJSON(w, http.StatusOK, group)
}
func (h *GroupHandlers) GetGroupDetails(w http.ResponseWriter, r *http.Request) {
name := mux.Vars(r)["name"]
group, err := h.adminServer.GetGroupDetails(r.Context(), name)
if err != nil {
glog.Errorf("Failed to get group details: %v", err)
status := groupErrorToHTTPStatus(err)
msg := "Failed to retrieve group"
if status == http.StatusNotFound {
msg = "Group not found"
}
writeJSONError(w, status, msg)
return
}
writeJSON(w, http.StatusOK, group)
}
func (h *GroupHandlers) DeleteGroup(w http.ResponseWriter, r *http.Request) {
name := mux.Vars(r)["name"]
if err := h.adminServer.DeleteGroup(r.Context(), name); err != nil {
glog.Errorf("Failed to delete group: %v", err)
writeJSONError(w, groupErrorToHTTPStatus(err), "Failed to delete group: "+err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"message": "Group deleted successfully"})
}
func (h *GroupHandlers) GetGroupMembers(w http.ResponseWriter, r *http.Request) {
name := mux.Vars(r)["name"]
group, err := h.adminServer.GetGroupDetails(r.Context(), name)
if err != nil {
writeJSONError(w, groupErrorToHTTPStatus(err), "Failed to get group: "+err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]interface{}{"members": group.Members})
}
func (h *GroupHandlers) AddGroupMember(w http.ResponseWriter, r *http.Request) {
name := mux.Vars(r)["name"]
var req struct {
Username string `json:"username"`
}
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if req.Username == "" {
writeJSONError(w, http.StatusBadRequest, "Username is required")
return
}
if err := h.adminServer.AddGroupMember(r.Context(), name, req.Username); err != nil {
writeJSONError(w, groupErrorToHTTPStatus(err), "Failed to add member: "+err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"message": "Member added successfully"})
}
func (h *GroupHandlers) RemoveGroupMember(w http.ResponseWriter, r *http.Request) {
name := mux.Vars(r)["name"]
username := mux.Vars(r)["username"]
if err := h.adminServer.RemoveGroupMember(r.Context(), name, username); err != nil {
writeJSONError(w, groupErrorToHTTPStatus(err), "Failed to remove member: "+err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"message": "Member removed successfully"})
}
func (h *GroupHandlers) GetGroupPolicies(w http.ResponseWriter, r *http.Request) {
name := mux.Vars(r)["name"]
group, err := h.adminServer.GetGroupDetails(r.Context(), name)
if err != nil {
writeJSONError(w, groupErrorToHTTPStatus(err), "Failed to get group: "+err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]interface{}{"policies": group.PolicyNames})
}
func (h *GroupHandlers) AttachGroupPolicy(w http.ResponseWriter, r *http.Request) {
name := mux.Vars(r)["name"]
var req struct {
PolicyName string `json:"policy_name"`
}
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if req.PolicyName == "" {
writeJSONError(w, http.StatusBadRequest, "Policy name is required")
return
}
if err := h.adminServer.AttachGroupPolicy(r.Context(), name, req.PolicyName); err != nil {
writeJSONError(w, groupErrorToHTTPStatus(err), "Failed to attach policy: "+err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"message": "Policy attached successfully"})
}
func (h *GroupHandlers) DetachGroupPolicy(w http.ResponseWriter, r *http.Request) {
name := mux.Vars(r)["name"]
policyName := mux.Vars(r)["policyName"]
if err := h.adminServer.DetachGroupPolicy(r.Context(), name, policyName); err != nil {
writeJSONError(w, groupErrorToHTTPStatus(err), "Failed to detach policy: "+err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"message": "Policy detached successfully"})
}
func (h *GroupHandlers) SetGroupStatus(w http.ResponseWriter, r *http.Request) {
name := mux.Vars(r)["name"]
var req struct {
Enabled *bool `json:"enabled"`
}
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if req.Enabled == nil {
writeJSONError(w, http.StatusBadRequest, "enabled field is required")
return
}
if err := h.adminServer.SetGroupStatus(r.Context(), name, *req.Enabled); err != nil {
writeJSONError(w, groupErrorToHTTPStatus(err), "Failed to update group status: "+err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"message": "Group status updated"})
}
func (h *GroupHandlers) getGroupsPageData(r *http.Request) (dash.GroupsPageData, error) {
username := dash.UsernameFromContext(r.Context())
if username == "" {
username = "admin"
}
groups, err := h.adminServer.GetGroups(r.Context())
if err != nil {
return dash.GroupsPageData{}, err
}
activeCount := 0
for _, g := range groups {
if g.Status == "enabled" {
activeCount++
}
}
// Get available users for dropdown
var availableUsers []string
users, err := h.adminServer.GetObjectStoreUsers(r.Context())
if err == nil {
for _, user := range users {
availableUsers = append(availableUsers, user.Username)
}
}
// Get available policies for dropdown
var availablePolicies []string
policies, err := h.adminServer.GetPolicies()
if err == nil {
for _, p := range policies {
availablePolicies = append(availablePolicies, p.Name)
}
}
return dash.GroupsPageData{
Username: username,
Groups: groups,
TotalGroups: len(groups),
ActiveGroups: activeCount,
AvailableUsers: availableUsers,
AvailablePolicies: availablePolicies,
LastUpdated: time.Now(),
}, nil
}
+24 -2
View File
@@ -1,7 +1,9 @@
package handlers
import (
"errors"
"fmt"
"io"
"net/http"
"time"
@@ -155,10 +157,30 @@ func (h *UserHandlers) CreateAccessKey(w http.ResponseWriter, r *http.Request) {
return
}
accessKey, err := h.adminServer.CreateAccessKey(username)
var req *dash.CreateAccessKeyRequest
var body dash.CreateAccessKeyRequest
if err := decodeJSONBody(newJSONMaxReader(w, r), &body); err != nil {
if !errors.Is(err, io.EOF) {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
// Empty body: auto-generate both keys
} else {
req = &body
}
accessKey, err := h.adminServer.CreateAccessKey(username, req)
if err != nil {
glog.Errorf("Failed to create access key for user %s: %v", username, err)
writeJSONError(w, http.StatusInternalServerError, "Failed to create access key: "+err.Error())
if errors.Is(err, dash.ErrAccessKeyInUse) {
writeJSONError(w, http.StatusConflict, err.Error())
} else if errors.Is(err, dash.ErrUserNotFound) {
writeJSONError(w, http.StatusNotFound, err.Error())
} else if errors.Is(err, dash.ErrInvalidInput) {
writeJSONError(w, http.StatusBadRequest, err.Error())
} else {
writeJSONError(w, http.StatusInternalServerError, "Failed to create access key: "+err.Error())
}
return
}
@@ -297,7 +297,7 @@ func (mm *MaintenanceManager) logTopologyStatus() {
errorCount := mm.errorCount
mm.mutex.RUnlock()
glog.V(0).Infof("Topology status: %d nodes, %d disks, %d workers, %d pending tasks, %d running tasks, errors: %d",
glog.V(1).Infof("Topology status: %d nodes, %d disks, %d workers, %d pending tasks, %d running tasks, errors: %d",
nodeCount, diskCount, workerCount,
stats.TasksByStatus[TaskStatusPending],
stats.TasksByStatus[TaskStatusInProgress]+stats.TasksByStatus[TaskStatusAssigned],
-56
View File
@@ -30,7 +30,6 @@ const (
runsJSONFileName = "runs.json"
trackedJobsJSONFileName = "tracked_jobs.json"
activitiesJSONFileName = "activities.json"
schedulerJSONFileName = "scheduler.json"
defaultDirPerm = 0o755
defaultFilePerm = 0o644
)
@@ -54,7 +53,6 @@ type ConfigStore struct {
memTrackedJobs []TrackedJob
memActivities []JobActivity
memJobDetails map[string]TrackedJob
memScheduler *SchedulerConfig
}
func NewConfigStore(adminDataDir string) (*ConfigStore, error) {
@@ -95,60 +93,6 @@ func (s *ConfigStore) BaseDir() string {
return s.baseDir
}
func (s *ConfigStore) LoadSchedulerConfig() (*SchedulerConfig, error) {
s.mu.RLock()
if !s.configured {
cfg := s.memScheduler
s.mu.RUnlock()
if cfg == nil {
return nil, nil
}
clone := *cfg
return &clone, nil
}
s.mu.RUnlock()
path := filepath.Join(s.baseDir, schedulerJSONFileName)
data, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, fmt.Errorf("read scheduler config: %w", err)
}
var cfg SchedulerConfig
if err := json.Unmarshal(data, &cfg); err != nil {
return nil, fmt.Errorf("unmarshal scheduler config: %w", err)
}
return &cfg, nil
}
func (s *ConfigStore) SaveSchedulerConfig(config *SchedulerConfig) error {
if config == nil {
return fmt.Errorf("scheduler config is nil")
}
normalized := normalizeSchedulerConfig(*config)
s.mu.Lock()
if !s.configured {
s.memScheduler = &normalized
s.mu.Unlock()
return nil
}
s.mu.Unlock()
payload, err := json.MarshalIndent(normalized, "", " ")
if err != nil {
return fmt.Errorf("marshal scheduler config: %w", err)
}
path := filepath.Join(s.baseDir, schedulerJSONFileName)
if err := os.WriteFile(path, payload, defaultFilePerm); err != nil {
return fmt.Errorf("save scheduler config: %w", err)
}
return nil
}
func (s *ConfigStore) SaveDescriptor(jobType string, descriptor *plugin_pb.JobTypeDescriptor) error {
if descriptor == nil {
return fmt.Errorf("descriptor is nil")
+1 -43
View File
@@ -78,9 +78,7 @@ type Plugin struct {
schedulerRun map[string]*schedulerRunInfo
schedulerLoopMu sync.Mutex
schedulerLoopState schedulerLoopState
schedulerConfigMu sync.RWMutex
schedulerConfig SchedulerConfig
schedulerWakeCh chan struct{}
schedulerWakeCh chan struct{}
dedupeMu sync.Mutex
recentDedupeByType map[string]map[string]time.Time
@@ -188,21 +186,6 @@ func New(options Options) (*Plugin, error) {
}
plugin.ctx, plugin.ctxCancel = context.WithCancel(context.Background())
if cfg, err := plugin.store.LoadSchedulerConfig(); err != nil {
glog.Warningf("Plugin failed to load scheduler config: %v", err)
plugin.schedulerConfig = DefaultSchedulerConfig()
} else if cfg == nil {
defaults := DefaultSchedulerConfig()
plugin.schedulerConfig = defaults
if plugin.store.IsConfigured() {
if err := plugin.store.SaveSchedulerConfig(&defaults); err != nil {
glog.Warningf("Plugin failed to persist scheduler defaults: %v", err)
}
}
} else {
plugin.schedulerConfig = normalizeSchedulerConfig(*cfg)
}
if err := plugin.loadPersistedMonitorState(); err != nil {
glog.Warningf("Plugin failed to load persisted monitoring state: %v", err)
}
@@ -426,31 +409,6 @@ func (r *Plugin) BaseDir() string {
return r.store.BaseDir()
}
func (r *Plugin) GetSchedulerConfig() SchedulerConfig {
if r == nil {
return DefaultSchedulerConfig()
}
r.schedulerConfigMu.RLock()
cfg := r.schedulerConfig
r.schedulerConfigMu.RUnlock()
return normalizeSchedulerConfig(cfg)
}
func (r *Plugin) UpdateSchedulerConfig(cfg SchedulerConfig) (SchedulerConfig, error) {
if r == nil {
return DefaultSchedulerConfig(), fmt.Errorf("plugin is not initialized")
}
normalized := normalizeSchedulerConfig(cfg)
if err := r.store.SaveSchedulerConfig(&normalized); err != nil {
return SchedulerConfig{}, err
}
r.schedulerConfigMu.Lock()
r.schedulerConfig = normalized
r.schedulerConfigMu.Unlock()
r.wakeScheduler()
return normalized, nil
}
func (r *Plugin) acquireAdminLock(reason string) (func(), error) {
if r == nil || r.lockManager == nil {
return func() {}, nil
+53 -20
View File
@@ -32,6 +32,7 @@ const (
defaultClusterContextTimeout = 10 * time.Second
defaultWaitingBacklogFloor = 8
defaultWaitingBacklogMultiplier = 4
maxEstimatedRuntimeCap = 8 * time.Hour
)
type schedulerPolicy struct {
@@ -64,7 +65,7 @@ func (r *Plugin) schedulerLoop() {
}
r.setSchedulerLoopState("", "sleeping")
idleSleep := r.GetSchedulerConfig().IdleSleepDuration()
idleSleep := defaultSchedulerIdleSleep
if nextRun := r.earliestNextDetectionAt(); !nextRun.IsZero() {
if until := time.Until(nextRun); until <= 0 {
idleSleep = 0
@@ -293,6 +294,26 @@ func (r *Plugin) runJobTypeIteration(jobType string, policy schedulerPolicy) boo
r.setSchedulerLoopState(jobType, "executing")
// Scan proposals for the maximum estimated_runtime_seconds so the
// execution phase gets enough time for large jobs (e.g. vacuum on
// big volumes). If any proposal needs more time than the remaining
// JobTypeMaxRuntime, extend the execution context accordingly.
var maxEstimatedRuntime time.Duration
for _, p := range filtered {
if p.Parameters != nil {
if est, ok := p.Parameters["estimated_runtime_seconds"]; ok {
if v := est.GetInt64Value(); v > 0 {
if d := time.Duration(v) * time.Second; d > maxEstimatedRuntime {
maxEstimatedRuntime = d
}
}
}
}
}
if maxEstimatedRuntime > maxEstimatedRuntimeCap {
maxEstimatedRuntime = maxEstimatedRuntimeCap
}
remaining = time.Until(start.Add(maxRuntime))
if remaining <= 0 {
r.appendActivity(JobActivity{
@@ -306,6 +327,17 @@ func (r *Plugin) runJobTypeIteration(jobType string, policy schedulerPolicy) boo
return detected
}
// If the longest estimated job exceeds the remaining JobTypeMaxRuntime,
// create a new execution context with enough headroom instead of using
// jobCtx which would cancel too early.
execCtx := jobCtx
execCancel := context.CancelFunc(func() {})
if maxEstimatedRuntime > 0 && maxEstimatedRuntime > remaining {
execCtx, execCancel = context.WithTimeout(context.Background(), maxEstimatedRuntime)
remaining = maxEstimatedRuntime
}
defer execCancel()
execPolicy := policy
if execPolicy.ExecutionTimeout <= 0 {
execPolicy.ExecutionTimeout = defaultScheduledExecutionTimeout
@@ -314,10 +346,10 @@ func (r *Plugin) runJobTypeIteration(jobType string, policy schedulerPolicy) boo
execPolicy.ExecutionTimeout = remaining
}
successCount, errorCount, canceledCount := r.dispatchScheduledProposals(jobCtx, jobType, filtered, clusterContext, execPolicy)
successCount, errorCount, canceledCount := r.dispatchScheduledProposals(execCtx, jobType, filtered, clusterContext, execPolicy)
status := "success"
if jobCtx.Err() != nil {
if execCtx.Err() != nil {
status = "timeout"
} else if errorCount > 0 || canceledCount > 0 {
status = "error"
@@ -937,7 +969,24 @@ func (r *Plugin) executeScheduledJobWithExecutor(
if parent == nil {
parent = context.Background()
}
execCtx, cancel := context.WithTimeout(parent, policy.ExecutionTimeout)
// Use the job's estimated runtime if provided and larger than the
// default execution timeout. This lets handlers like vacuum scale
// the timeout based on volume size so large volumes are not killed.
timeout := policy.ExecutionTimeout
if job.Parameters != nil {
if est, ok := job.Parameters["estimated_runtime_seconds"]; ok {
if v := est.GetInt64Value(); v > 0 {
estimated := time.Duration(v) * time.Second
if estimated > maxEstimatedRuntimeCap {
estimated = maxEstimatedRuntimeCap
}
if estimated > timeout {
timeout = estimated
}
}
}
}
execCtx, cancel := context.WithTimeout(parent, timeout)
_, err := r.executeJobWithExecutor(execCtx, executor, job, clusterContext, int32(attempt))
cancel()
if err == nil {
@@ -1134,22 +1183,6 @@ func secondsFromDuration(duration time.Duration) int32 {
return int32(duration / time.Second)
}
func waitForShutdownOrTimer(shutdown <-chan struct{}, duration time.Duration) bool {
if duration <= 0 {
return true
}
timer := time.NewTimer(duration)
defer timer.Stop()
select {
case <-shutdown:
return false
case <-timer.C:
return true
}
}
func waitForShutdownOrTimerWithContext(shutdown <-chan struct{}, ctx context.Context, duration time.Duration) bool {
if duration <= 0 {
return true
+1 -27
View File
@@ -2,30 +2,4 @@ package plugin
import "time"
const (
defaultSchedulerIdleSleep = 613 * time.Second
)
type SchedulerConfig struct {
IdleSleepSeconds int32 `json:"idle_sleep_seconds"`
}
func DefaultSchedulerConfig() SchedulerConfig {
return SchedulerConfig{
IdleSleepSeconds: int32(defaultSchedulerIdleSleep / time.Second),
}
}
func normalizeSchedulerConfig(cfg SchedulerConfig) SchedulerConfig {
if cfg.IdleSleepSeconds <= 0 {
return DefaultSchedulerConfig()
}
return cfg
}
func (c SchedulerConfig) IdleSleepDuration() time.Duration {
if c.IdleSleepSeconds <= 0 {
return defaultSchedulerIdleSleep
}
return time.Duration(c.IdleSleepSeconds) * time.Second
}
const defaultSchedulerIdleSleep = 61 * time.Second
+2 -6
View File
@@ -216,22 +216,18 @@ func (r *Plugin) snapshotSchedulerLoopState() schedulerLoopState {
func (r *Plugin) GetSchedulerStatus() SchedulerStatus {
now := time.Now().UTC()
loopState := r.snapshotSchedulerLoopState()
schedulerConfig := r.GetSchedulerConfig()
status := SchedulerStatus{
Now: now,
SchedulerTickSeconds: int(secondsFromDuration(r.schedulerTick)),
InProcessJobs: r.listInProcessJobs(now),
IdleSleepSeconds: int(schedulerConfig.IdleSleepSeconds),
IdleSleepSeconds: int(defaultSchedulerIdleSleep / time.Second),
CurrentJobType: loopState.currentJobType,
CurrentPhase: loopState.currentPhase,
LastIterationHadJobs: loopState.lastIterationHadJobs,
}
nextDetectionAt := r.earliestNextDetectionAt()
if nextDetectionAt.IsZero() && loopState.currentPhase == "sleeping" && !loopState.lastIterationCompleted.IsZero() {
idleSleep := schedulerConfig.IdleSleepDuration()
if idleSleep > 0 {
nextDetectionAt = loopState.lastIterationCompleted.Add(idleSleep)
}
nextDetectionAt = loopState.lastIterationCompleted.Add(defaultSchedulerIdleSleep)
}
if !nextDetectionAt.IsZero() {
at := nextDetectionAt
+38 -73
View File
@@ -478,7 +478,7 @@ async function handleCreateBucket(event) {
if (response.ok) {
// Success
showAlert('success', `Bucket "${bucketData.name}" created successfully!`);
showAlert(`Bucket "${bucketData.name}" created successfully!`, 'success');
// Close modal
const modal = bootstrap.Modal.getInstance(document.getElementById('createBucketModal'));
@@ -493,11 +493,11 @@ async function handleCreateBucket(event) {
}, 1500);
} else {
// Error
showAlert('danger', result.error || 'Failed to create bucket');
showAlert(result.error || 'Failed to create bucket', 'danger');
}
} catch (error) {
console.error('Error creating bucket:', error);
showAlert('danger', 'Network error occurred while creating bucket');
showAlert('Network error occurred while creating bucket', 'danger');
}
}
@@ -538,7 +538,7 @@ async function deleteBucket() {
if (response.ok) {
// Success
showAlert('success', `Bucket "${bucketToDelete}" deleted successfully!`);
showAlert(`Bucket "${bucketToDelete}" deleted successfully!`, 'success');
// Close modal
const modal = bootstrap.Modal.getInstance(document.getElementById('deleteBucketModal'));
@@ -550,11 +550,11 @@ async function deleteBucket() {
}, 1500);
} else {
// Error
showAlert('danger', result.error || 'Failed to delete bucket');
showAlert(result.error || 'Failed to delete bucket', 'danger');
}
} catch (error) {
console.error('Error deleting bucket:', error);
showAlert('danger', 'Network error occurred while deleting bucket');
showAlert('Network error occurred while deleting bucket', 'danger');
}
bucketToDelete = '';
@@ -609,38 +609,7 @@ function exportBucketList() {
window.URL.revokeObjectURL(url);
}
// Show alert message
function showAlert(type, message) {
// Remove existing alerts
const existingAlerts = document.querySelectorAll('.alert-floating');
existingAlerts.forEach(alert => alert.remove());
// Create new alert
const alert = document.createElement('div');
alert.className = `alert alert-${type} alert-dismissible fade show alert-floating`;
alert.style.cssText = `
position: fixed;
top: 20px;
right: 20px;
z-index: 9999;
min-width: 300px;
box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
`;
alert.innerHTML = `
${message}
<button type="button" class="btn-close" data-bs-dismiss="alert" aria-label="Close"></button>
`;
document.body.appendChild(alert);
// Auto-remove after 5 seconds
setTimeout(() => {
if (alert.parentNode) {
alert.remove();
}
}, 5000);
}
// showAlert is provided by modal-alerts.js with signature: showAlert(message, type)
// Format date for display
function formatDate(date) {
@@ -651,7 +620,7 @@ function formatDate(date) {
function adminCopyToClipboard(text) {
if (navigator.clipboard && navigator.clipboard.writeText) {
navigator.clipboard.writeText(text).then(() => {
showAlert('success', 'Copied to clipboard!');
showAlert('Copied to clipboard!', 'success');
}).catch(err => {
console.error('Failed to copy text: ', err);
fallbackCopyText(text);
@@ -677,13 +646,13 @@ function fallbackCopyText(text) {
try {
const successful = document.execCommand('copy');
if (successful) {
showAlert('success', 'Copied to clipboard!');
showAlert('Copied to clipboard!', 'success');
} else {
showAlert('danger', 'Failed to copy to clipboard');
showAlert('Failed to copy to clipboard', 'danger');
}
} catch (err) {
console.error('Fallback copy failed: ', err);
showAlert('danger', 'Failed to copy to clipboard');
showAlert('Failed to copy to clipboard', 'danger');
}
document.body.removeChild(textArea);
@@ -764,7 +733,7 @@ function exportVolumes() {
function exportCollections() {
const table = document.getElementById('collectionsTable');
if (!table) {
showAlert('error', 'Collections table not found');
showAlert('Collections table not found', 'error');
return;
}
@@ -800,7 +769,7 @@ function exportCollections() {
function exportMasters() {
const table = document.getElementById('mastersTable');
if (!table) {
showAlert('error', 'Masters table not found');
showAlert('Masters table not found', 'error');
return;
}
@@ -834,7 +803,7 @@ function exportMasters() {
function exportFilers() {
const table = document.getElementById('filersTable');
if (!table) {
showAlert('error', 'Filers table not found');
showAlert('Filers table not found', 'error');
return;
}
@@ -870,7 +839,7 @@ function exportFilers() {
function exportUsers() {
const table = document.getElementById('usersTable');
if (!table) {
showAlert('error', 'Users table not found');
showAlert('Users table not found', 'error');
return;
}
@@ -1020,7 +989,7 @@ function confirmDeleteSelected() {
const selectedPaths = getSelectedFilePaths();
if (selectedPaths.length === 0) {
showAlert('warning', 'No files selected');
showAlert('No files selected', 'warning');
return;
}
@@ -1041,7 +1010,7 @@ function confirmDeleteSelected() {
// Delete multiple selected files
async function deleteSelectedFiles(filePaths) {
if (!filePaths || filePaths.length === 0) {
showAlert('warning', 'No files selected');
showAlert('No files selected', 'warning');
return;
}
@@ -1065,9 +1034,9 @@ async function deleteSelectedFiles(filePaths) {
if (result.deleted > 0) {
if (result.failed === 0) {
showAlert('success', `Successfully deleted ${result.deleted} item(s)`);
showAlert(`Successfully deleted ${result.deleted} item(s)`, 'success');
} else {
showAlert('warning', `Deleted ${result.deleted} item(s), failed to delete ${result.failed} item(s)`);
showAlert(`Deleted ${result.deleted} item(s), failed to delete ${result.failed} item(s)`, 'warning');
if (result.errors && result.errors.length > 0) {
console.warn('Deletion errors:', result.errors);
}
@@ -1082,15 +1051,15 @@ async function deleteSelectedFiles(filePaths) {
if (result.errors && result.errors.length > 0) {
errorMessage += ': ' + result.errors.join(', ');
}
showAlert('error', errorMessage);
showAlert(errorMessage, 'error');
}
} else {
const error = await response.json();
showAlert('error', `Failed to delete files: ${error.error || 'Unknown error'}`);
showAlert(`Failed to delete files: ${error.error || 'Unknown error'}`, 'error');
}
} catch (error) {
console.error('Delete error:', error);
showAlert('error', 'Failed to delete files');
showAlert('Failed to delete files', 'error');
} finally {
// Re-enable the button
deleteBtn.disabled = false;
@@ -1311,7 +1280,7 @@ async function submitUploadFile() {
function exportFileList() {
const table = document.getElementById('fileTable');
if (!table) {
showAlert('error', 'File table not found');
showAlert('File table not found', 'error');
return;
}
@@ -1357,7 +1326,7 @@ async function viewFile(filePath) {
if (!response.ok) {
const error = await response.json();
showAlert('error', `Failed to view file: ${error.error || 'Unknown error'}`);
showAlert(`Failed to view file: ${error.error || 'Unknown error'}`, 'error');
return;
}
@@ -1366,7 +1335,7 @@ async function viewFile(filePath) {
} catch (error) {
console.error('View file error:', error);
showAlert('error', 'Failed to view file');
showAlert('Failed to view file', 'error');
}
}
@@ -1377,7 +1346,7 @@ async function showProperties(filePath) {
if (!response.ok) {
const error = await response.json();
showAlert('error', `Failed to get file properties: ${error.error || 'Unknown error'}`);
showAlert(`Failed to get file properties: ${error.error || 'Unknown error'}`, 'error');
return;
}
@@ -1386,7 +1355,7 @@ async function showProperties(filePath) {
} catch (error) {
console.error('Properties error:', error);
showAlert('error', 'Failed to get file properties');
showAlert('Failed to get file properties', 'error');
}
}
@@ -1413,16 +1382,16 @@ async function deleteFile(filePath) {
});
if (response.ok) {
showAlert('success', `Successfully deleted "${filePath}"`);
showAlert(`Successfully deleted "${filePath}"`, 'success');
// Reload the page to update the file list
window.location.reload();
} else {
const error = await response.json();
showAlert('error', `Failed to delete file: ${error.error || 'Unknown error'}`);
showAlert(`Failed to delete file: ${error.error || 'Unknown error'}`, 'error');
}
} catch (error) {
console.error('Delete error:', error);
showAlert('error', 'Failed to delete file');
showAlert('Failed to delete file', 'error');
}
}
@@ -1737,7 +1706,7 @@ async function handleUpdateQuota(event) {
if (response.ok) {
// Success
showAlert('success', `Quota for bucket "${bucketName}" updated successfully!`);
showAlert(`Quota for bucket "${bucketName}" updated successfully!`, 'success');
// Close modal
const modal = bootstrap.Modal.getInstance(document.getElementById('manageQuotaModal'));
@@ -1749,11 +1718,11 @@ async function handleUpdateQuota(event) {
}, 1500);
} else {
// Error
showAlert('danger', result.error || 'Failed to update bucket quota');
showAlert(result.error || 'Failed to update bucket quota', 'danger');
}
} catch (error) {
console.error('Error updating bucket quota:', error);
showAlert('danger', 'Network error occurred while updating bucket quota');
showAlert('Network error occurred while updating bucket quota', 'danger');
}
}
@@ -2227,10 +2196,6 @@ function showNewAccessKeyModal(accessKeyData) {
<i class="fas fa-check-circle me-2"></i>
<strong>Success!</strong> Your new access key has been created.
</div>
<div class="alert alert-warning">
<i class="fas fa-exclamation-triangle me-2"></i>
<strong>Important:</strong> This is the only time the secret key will be displayed. Please save it securely.
</div>
<div class="mb-3">
<label class="form-label"><strong>Access Key:</strong></label>
<div class="input-group">
@@ -2274,21 +2239,21 @@ function copyFromInput(inputId) {
try {
const successful = document.execCommand('copy');
if (successful) {
showAlert('success', 'Copied to clipboard!');
showAlert('Copied to clipboard!', 'success');
} else {
// Try modern clipboard API as fallback
navigator.clipboard.writeText(input.value).then(() => {
showAlert('success', 'Copied to clipboard!');
showAlert('Copied to clipboard!', 'success');
}).catch(() => {
showAlert('danger', 'Failed to copy');
showAlert('Failed to copy', 'danger');
});
}
} catch (err) {
// Try modern clipboard API as fallback
navigator.clipboard.writeText(input.value).then(() => {
showAlert('success', 'Copied to clipboard!');
showAlert('Copied to clipboard!', 'success');
}).catch(() => {
showAlert('danger', 'Failed to copy');
showAlert('Failed to copy', 'danger');
});
}
}
+23
View File
@@ -25,6 +25,29 @@ async function deleteUser(username) {
}, 'Are you sure you want to delete this user? This action cannot be undone.');
}
// Delete group function
async function deleteGroup(name) {
showDeleteConfirm(name, async function () {
try {
const encodedName = encodeURIComponent(name);
const response = await fetch(`/api/groups/${encodedName}`, {
method: 'DELETE'
});
if (response.ok) {
showAlert('Group deleted successfully', 'success');
setTimeout(() => window.location.reload(), 1000);
} else {
const error = await response.json().catch(() => ({}));
showAlert('Failed to delete group: ' + (error.error || 'Unknown error'), 'error');
}
} catch (error) {
console.error('Error deleting group:', error);
showAlert('Failed to delete group: ' + error.message, 'error');
}
}, 'Are you sure you want to delete this group? This action cannot be undone.');
}
// Delete access key function
async function deleteAccessKey(username, accessKey) {
showDeleteConfirm(accessKey, async function () {
+34
View File
@@ -272,6 +272,40 @@ func (at *ActiveTopology) HasAnyTask(volumeID uint32) bool {
return at.HasTask(volumeID, TaskTypeNone)
}
// GetTaskServerAdjustments returns per-server volume count adjustments for
// pending and assigned tasks of the given type. For each task, source servers
// are decremented and destination servers are incremented, reflecting the
// projected volume distribution once in-flight tasks complete.
func (at *ActiveTopology) GetTaskServerAdjustments(taskType TaskType) map[string]int {
at.mutex.RLock()
defer at.mutex.RUnlock()
adjustments := make(map[string]int)
for _, task := range at.pendingTasks {
if task.TaskType != taskType {
continue
}
for _, src := range task.Sources {
adjustments[src.SourceServer]--
}
for _, dst := range task.Destinations {
adjustments[dst.TargetServer]++
}
}
for _, task := range at.assignedTasks {
if task.TaskType != taskType {
continue
}
for _, src := range task.Sources {
adjustments[src.SourceServer]--
}
for _, dst := range task.Destinations {
adjustments[dst.TargetServer]++
}
}
return adjustments
}
// calculateSourceStorageImpact calculates storage impact for sources based on task type and cleanup type
func (at *ActiveTopology) calculateSourceStorageImpact(taskType TaskType, cleanupType SourceCleanupType, volumeSize int64) StorageSlotChange {
switch taskType {
+443
View File
@@ -0,0 +1,443 @@
package app
import (
"fmt"
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
)
templ Groups(data dash.GroupsPageData) {
<div class="container-fluid">
<!-- Page Header -->
<div class="d-sm-flex align-items-center justify-content-between mb-4">
<div>
<h1 class="h3 mb-0 text-gray-800">
<i class="fas fa-users-cog me-2"></i>Groups
</h1>
<p class="mb-0 text-muted">Manage IAM groups for organizing users and policies</p>
</div>
<div class="d-flex gap-2">
<button type="button" class="btn btn-primary"
data-bs-toggle="modal"
data-bs-target="#createGroupModal">
<i class="fas fa-plus me-1"></i>Create Group
</button>
</div>
</div>
<!-- Summary Cards -->
<div class="row mb-4">
<div class="col-xl-3 col-md-6 mb-4">
<div class="card border-left-primary shadow h-100 py-2">
<div class="card-body">
<div class="row no-gutters align-items-center">
<div class="col mr-2">
<div class="text-xs font-weight-bold text-primary text-uppercase mb-1">
Total Groups
</div>
<div class="h5 mb-0 font-weight-bold text-gray-800">
{fmt.Sprintf("%d", data.TotalGroups)}
</div>
</div>
<div class="col-auto">
<i class="fas fa-users-cog fa-2x text-gray-300"></i>
</div>
</div>
</div>
</div>
</div>
<div class="col-xl-3 col-md-6 mb-4">
<div class="card border-left-success shadow h-100 py-2">
<div class="card-body">
<div class="row no-gutters align-items-center">
<div class="col mr-2">
<div class="text-xs font-weight-bold text-success text-uppercase mb-1">
Active Groups
</div>
<div class="h5 mb-0 font-weight-bold text-gray-800">
{fmt.Sprintf("%d", data.ActiveGroups)}
</div>
</div>
<div class="col-auto">
<i class="fas fa-check-circle fa-2x text-gray-300"></i>
</div>
</div>
</div>
</div>
</div>
</div>
<!-- Groups Table -->
<div class="card shadow mb-4">
<div class="card-header py-3">
<h6 class="m-0 font-weight-bold text-primary">Groups</h6>
</div>
<div class="card-body">
if len(data.Groups) == 0 {
<div class="text-center py-5 text-muted">
<i class="fas fa-users-cog fa-3x mb-3"></i>
<p>No groups found. Create a group to get started.</p>
</div>
} else {
<div class="table-responsive">
<table class="table table-bordered table-hover" id="groupsTable" width="100%" cellspacing="0">
<thead>
<tr>
<th>Name</th>
<th>Members</th>
<th>Policies</th>
<th>Status</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
for _, group := range data.Groups {
<tr>
<td>
<strong>{group.Name}</strong>
</td>
<td>
<span class="badge bg-info">{fmt.Sprintf("%d", group.MemberCount)}</span>
</td>
<td>
<span class="badge bg-secondary">{fmt.Sprintf("%d", group.PolicyCount)}</span>
</td>
<td>
if group.Status == "enabled" {
<span class="badge bg-success">Enabled</span>
} else {
<span class="badge bg-danger">Disabled</span>
}
</td>
<td>
<button class="btn btn-sm btn-outline-primary me-1"
data-group-name={group.Name}
data-action="view"
aria-label={"View group " + group.Name}
title={"View " + group.Name}>
<i class="fas fa-eye"></i>
</button>
<button class="btn btn-sm btn-outline-danger"
data-group-name={group.Name}
data-action="delete"
aria-label={"Delete group " + group.Name}
title={"Delete " + group.Name}>
<i class="fas fa-trash"></i>
</button>
</td>
</tr>
}
</tbody>
</table>
</div>
}
</div>
</div>
<!-- Create Group Modal -->
<div class="modal fade" id="createGroupModal" tabindex="-1">
<div class="modal-dialog">
<div class="modal-content">
<div class="modal-header">
<h5 class="modal-title">Create Group</h5>
<button type="button" class="btn-close" data-bs-dismiss="modal"></button>
</div>
<div class="modal-body">
<form id="createGroupForm">
<div class="mb-3">
<label for="groupName" class="form-label">Group Name</label>
<input type="text" class="form-control" id="groupName" name="name" required
placeholder="Enter group name"/>
</div>
</form>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-secondary" data-bs-dismiss="modal">Cancel</button>
<button type="button" class="btn btn-primary" onclick="createGroup()">Create</button>
</div>
</div>
</div>
</div>
<!-- View Group Modal -->
<div class="modal fade" id="viewGroupModal" tabindex="-1">
<div class="modal-dialog modal-lg">
<div class="modal-content">
<div class="modal-header">
<h5 class="modal-title" id="viewGroupTitle">Group Details</h5>
<button type="button" class="btn-close" data-bs-dismiss="modal"></button>
</div>
<div class="modal-body">
<ul class="nav nav-tabs" id="groupTabs" role="tablist">
<li class="nav-item">
<a class="nav-link active" id="members-tab" data-bs-toggle="tab" href="#membersPane" role="tab">Members</a>
</li>
<li class="nav-item">
<a class="nav-link" id="policies-tab" data-bs-toggle="tab" href="#policiesPane" role="tab">Policies</a>
</li>
<li class="nav-item">
<a class="nav-link" id="settings-tab" data-bs-toggle="tab" href="#settingsPane" role="tab">Settings</a>
</li>
</ul>
<div class="tab-content mt-3" id="groupTabContent">
<!-- Members Tab -->
<div class="tab-pane fade show active" id="membersPane" role="tabpanel">
<div class="mb-3">
<div class="input-group">
<select class="form-select" id="addMemberSelect">
<option value="">Select user to add...</option>
for _, user := range data.AvailableUsers {
<option value={user}>{user}</option>
}
</select>
<button class="btn btn-outline-primary" type="button" onclick="addMemberToGroup()">
<i class="fas fa-plus"></i> Add
</button>
</div>
</div>
<div id="membersList"></div>
</div>
<!-- Policies Tab -->
<div class="tab-pane fade" id="policiesPane" role="tabpanel">
<div class="mb-3">
<div class="input-group">
<select class="form-select" id="attachPolicySelect">
<option value="">Select policy to attach...</option>
for _, policy := range data.AvailablePolicies {
<option value={policy}>{policy}</option>
}
</select>
<button class="btn btn-outline-primary" type="button" onclick="attachPolicyToGroup()">
<i class="fas fa-plus"></i> Attach
</button>
</div>
</div>
<div id="policiesList"></div>
</div>
<!-- Settings Tab -->
<div class="tab-pane fade" id="settingsPane" role="tabpanel">
<div class="form-check form-switch mb-3">
<input class="form-check-input" type="checkbox" id="groupEnabledSwitch" checked
onchange="toggleGroupStatus()"/>
<label class="form-check-label" for="groupEnabledSwitch">Group Enabled</label>
</div>
</div>
</div>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-secondary" data-bs-dismiss="modal">Close</button>
</div>
</div>
</div>
</div>
</div>
<script src="/static/js/iam-utils.js"></script>
<script>
// Groups page JavaScript
let currentGroupName = '';
async function createGroup() {
const name = document.getElementById('groupName').value.trim();
if (!name) {
showAlert('Group name is required', 'error');
return;
}
try {
const response = await fetch('/api/groups', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ name: name })
});
if (response.ok) {
showAlert('Group created successfully', 'success');
setTimeout(() => window.location.reload(), 1000);
} else {
const error = await response.json().catch(() => ({}));
showAlert('Failed to create group: ' + (error.error || 'Unknown error'), 'error');
}
} catch (error) {
showAlert('Failed to create group: ' + error.message, 'error');
}
}
async function viewGroup(name) {
currentGroupName = name;
document.getElementById('viewGroupTitle').textContent = 'Group: ' + name;
await refreshGroupDetails(name);
new bootstrap.Modal(document.getElementById('viewGroupModal')).show();
}
async function refreshGroupDetails(requestedName) {
try {
const response = await fetch('/api/groups/' + encodeURIComponent(requestedName));
if (!response.ok) throw new Error('Failed to fetch group');
if (requestedName !== currentGroupName) return; // stale response
const group = await response.json();
// Render members using DOM APIs to prevent XSS
const membersList = document.getElementById('membersList');
membersList.innerHTML = '';
const membersTable = document.createElement('table');
membersTable.className = 'table table-sm';
const membersTbody = document.createElement('tbody');
if (group.members && group.members.length > 0) {
for (const member of group.members) {
const tr = membersTbody.insertRow();
const td1 = tr.insertCell();
td1.textContent = member;
const td2 = tr.insertCell();
const btn = document.createElement('button');
btn.className = 'btn btn-sm btn-outline-danger';
btn.onclick = () => removeMember(member);
btn.innerHTML = '<i class="fas fa-times"></i>';
td2.appendChild(btn);
}
} else {
const tr = membersTbody.insertRow();
const td = tr.insertCell();
td.className = 'text-muted';
td.textContent = 'No members';
}
membersTable.appendChild(membersTbody);
membersList.appendChild(membersTable);
// Render policies using DOM APIs to prevent XSS
const policiesList = document.getElementById('policiesList');
policiesList.innerHTML = '';
const policiesTable = document.createElement('table');
policiesTable.className = 'table table-sm';
const policiesTbody = document.createElement('tbody');
if (group.policy_names && group.policy_names.length > 0) {
for (const policy of group.policy_names) {
const tr = policiesTbody.insertRow();
const td1 = tr.insertCell();
td1.textContent = policy;
const td2 = tr.insertCell();
const btn = document.createElement('button');
btn.className = 'btn btn-sm btn-outline-danger';
btn.onclick = () => detachPolicy(policy);
btn.innerHTML = '<i class="fas fa-times"></i>';
td2.appendChild(btn);
}
} else {
const tr = policiesTbody.insertRow();
const td = tr.insertCell();
td.className = 'text-muted';
td.textContent = 'No policies attached';
}
policiesTable.appendChild(policiesTbody);
policiesList.appendChild(policiesTable);
// Update status toggle
document.getElementById('groupEnabledSwitch').checked = (group.status === 'enabled');
} catch (error) {
console.error('Error fetching group details:', error);
}
}
async function addMemberToGroup() {
const username = document.getElementById('addMemberSelect').value;
if (!username) return;
try {
const response = await fetch('/api/groups/' + encodeURIComponent(currentGroupName) + '/members', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ username: username })
});
if (response.ok) {
await refreshGroupDetails(currentGroupName);
showAlert('Member added', 'success');
} else {
const error = await response.json().catch(() => ({}));
showAlert('Failed to add member: ' + (error.error || 'Unknown error'), 'error');
}
} catch (error) {
showAlert('Failed to add member: ' + error.message, 'error');
}
}
async function removeMember(username) {
try {
const response = await fetch('/api/groups/' + encodeURIComponent(currentGroupName) + '/members/' + encodeURIComponent(username), {
method: 'DELETE'
});
if (response.ok) {
await refreshGroupDetails(currentGroupName);
showAlert('Member removed', 'success');
} else {
const error = await response.json().catch(() => ({}));
showAlert('Failed to remove member: ' + (error.error || 'Unknown error'), 'error');
}
} catch (error) {
showAlert('Failed to remove member: ' + error.message, 'error');
}
}
async function attachPolicyToGroup() {
const policyName = document.getElementById('attachPolicySelect').value;
if (!policyName) return;
try {
const response = await fetch('/api/groups/' + encodeURIComponent(currentGroupName) + '/policies', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ policy_name: policyName })
});
if (response.ok) {
await refreshGroupDetails(currentGroupName);
showAlert('Policy attached', 'success');
} else {
const error = await response.json().catch(() => ({}));
showAlert('Failed to attach policy: ' + (error.error || 'Unknown error'), 'error');
}
} catch (error) {
showAlert('Failed to attach policy: ' + error.message, 'error');
}
}
async function detachPolicy(policyName) {
try {
const response = await fetch('/api/groups/' + encodeURIComponent(currentGroupName) + '/policies/' + encodeURIComponent(policyName), {
method: 'DELETE'
});
if (response.ok) {
await refreshGroupDetails(currentGroupName);
showAlert('Policy detached', 'success');
} else {
const error = await response.json().catch(() => ({}));
showAlert('Failed to detach policy: ' + (error.error || 'Unknown error'), 'error');
}
} catch (error) {
showAlert('Failed to detach policy: ' + error.message, 'error');
}
}
async function toggleGroupStatus() {
const enabled = document.getElementById('groupEnabledSwitch').checked;
try {
const response = await fetch('/api/groups/' + encodeURIComponent(currentGroupName) + '/status', {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ enabled: enabled })
});
if (response.ok) {
showAlert('Group status updated', 'success');
} else {
const error = await response.json().catch(() => ({}));
showAlert('Failed to update status: ' + (error.error || 'Unknown error'), 'error');
}
} catch (error) {
showAlert('Failed to update status: ' + error.message, 'error');
}
}
// Event delegation for group action buttons
document.addEventListener('click', function(e) {
const btn = e.target.closest('[data-action]');
if (!btn) return;
const name = btn.dataset.groupName;
if (!name) return;
if (btn.dataset.action === 'view') viewGroup(name);
else if (btn.dataset.action === 'delete') deleteGroup(name);
});
</script>
}
File diff suppressed because one or more lines are too long
+216 -8
View File
@@ -384,6 +384,21 @@ templ ObjectStoreUsers(data dash.ObjectStoreUsersData) {
<!-- Options loaded dynamically -->
</select>
</div>
<div class="mb-3">
<label class="form-label">Groups</label>
<div id="editUserGroups">
<!-- Groups loaded dynamically -->
</div>
<div class="input-group mt-2">
<select class="form-select" id="editGroupSelect">
<option value="">Add to group...</option>
</select>
<button class="btn btn-outline-primary" type="button" onclick="addUserToGroupFromEdit()"
aria-label="Add user to group" title="Add user to group">
<i class="fas fa-plus"></i>
</button>
</div>
</div>
</form>
</div>
<div class="modal-footer">
@@ -427,10 +442,32 @@ templ ObjectStoreUsers(data dash.ObjectStoreUsersData) {
<div class="modal-body">
<div class="d-flex justify-content-between align-items-center mb-3">
<h6>Access Keys for <span id="accessKeysUsername"></span></h6>
<button type="button" class="btn btn-primary btn-sm" onclick="createAccessKey()">
<button type="button" class="btn btn-primary btn-sm" onclick="toggleCreateKeyForm()">
<i class="fas fa-plus me-1"></i>Create New Key
</button>
</div>
<div id="createKeyForm" class="card mb-3" style="display: none;">
<div class="card-body">
<p class="text-muted small mb-2">Leave blank to auto-generate.</p>
<div class="mb-2">
<label for="newAccessKeyInput" class="form-label form-label-sm">Access Key</label>
<input type="text" class="form-control form-control-sm" id="newAccessKeyInput" placeholder="Auto-generated if empty" minlength="4" maxlength="128">
</div>
<div class="mb-2">
<label for="newSecretKeyInput" class="form-label form-label-sm">Secret Key</label>
<div class="input-group input-group-sm">
<input type="password" class="form-control form-control-sm" id="newSecretKeyInput" placeholder="Auto-generated if empty" minlength="8" maxlength="128">
<button class="btn btn-outline-secondary" type="button" onclick="toggleSecretKeyVisibility()" aria-label="Toggle secret key visibility">
<i class="fas fa-eye" id="secretKeyToggleIcon"></i>
</button>
</div>
</div>
<div class="d-flex gap-2">
<button type="button" class="btn btn-primary btn-sm" onclick="createAccessKey()">Create</button>
<button type="button" class="btn btn-secondary btn-sm" onclick="toggleCreateKeyForm()">Cancel</button>
</div>
</div>
</div>
<div id="accessKeysContent">
<!-- Content will be loaded dynamically -->
</div>
@@ -912,6 +949,9 @@ templ ObjectStoreUsers(data dash.ObjectStoreUsersData) {
}
}
// Populate groups
await populateEditUserGroups(username);
// Show modal
const modal = new bootstrap.Modal(document.getElementById('editUserModal'));
modal.show();
@@ -1027,6 +1067,103 @@ templ ObjectStoreUsers(data dash.ObjectStoreUsersData) {
}
// Populate groups in the edit user modal
async function populateEditUserGroups(username) {
const container = document.getElementById('editUserGroups');
const groupSelect = document.getElementById('editGroupSelect');
container.innerHTML = '';
groupSelect.innerHTML = '<option value="">Add to group...</option>';
try {
// Fetch all groups
const groupsResp = await fetch('/api/groups');
if (!groupsResp.ok) return;
const groupsData = await groupsResp.json();
const allGroups = groupsData.groups || [];
// Fetch user details to get current groups
const userResp = await fetch(`/api/users/${encodeURIComponent(username)}`);
if (!userResp.ok) return;
const user = await userResp.json();
const userGroups = user.groups || [];
// Show current group badges with remove button
if (userGroups.length > 0) {
userGroups.forEach(function(group) {
const badge = document.createElement('span');
badge.className = 'badge bg-primary me-1 mb-1';
badge.textContent = group + ' ';
const removeIcon = document.createElement('i');
removeIcon.className = 'fas fa-times ms-1';
removeIcon.style.cursor = 'pointer';
removeIcon.setAttribute('aria-label', 'Remove from group ' + group);
removeIcon.setAttribute('title', 'Remove from group');
removeIcon.addEventListener('click', function() {
removeUserFromGroupInEdit(group);
});
badge.appendChild(removeIcon);
container.appendChild(badge);
});
} else {
container.innerHTML = '<span class="text-muted">No groups</span>';
}
// Populate dropdown with groups the user is NOT in
allGroups.forEach(function(g) {
if (!userGroups.includes(g.name)) {
const opt = document.createElement('option');
opt.value = g.name;
opt.textContent = g.name;
groupSelect.appendChild(opt);
}
});
} catch (error) {
console.error('Error loading groups:', error);
}
}
// Add user to group from edit modal
async function addUserToGroupFromEdit() {
const username = document.getElementById('editUsername').value;
const groupSelect = document.getElementById('editGroupSelect');
const groupName = groupSelect.value;
if (!groupName) return;
try {
const response = await fetch(`/api/groups/${encodeURIComponent(groupName)}/members`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ username: username })
});
if (response.ok) {
await populateEditUserGroups(username);
} else {
const error = await response.json();
showAlert('Failed to add to group: ' + (error.error || 'Unknown error'), 'error');
}
} catch (error) {
showAlert('Failed to add to group: ' + error.message, 'error');
}
}
// Remove user from group in edit modal
async function removeUserFromGroupInEdit(groupName) {
const username = document.getElementById('editUsername').value;
try {
const response = await fetch(`/api/groups/${encodeURIComponent(groupName)}/members/${encodeURIComponent(username)}`, {
method: 'DELETE'
});
if (response.ok) {
await populateEditUserGroups(username);
} else {
const error = await response.json();
showAlert('Failed to remove from group: ' + (error.error || 'Unknown error'), 'error');
}
} catch (error) {
showAlert('Failed to remove from group: ' + error.message, 'error');
}
}
// Handle update user form submission
async function handleUpdateUser() {
const username = document.getElementById('editUsername').value;
@@ -1115,6 +1252,16 @@ templ ObjectStoreUsers(data dash.ObjectStoreUsersData) {
detailsHtml += '<span class="text-muted">No policies attached</span>';
}
detailsHtml += '</div>';
detailsHtml += '<h6 class="text-muted">Groups</h6>';
detailsHtml += '<div class="mb-3">';
if (user.groups && user.groups.length > 0) {
detailsHtml += user.groups.map(function(group) {
return '<span class="badge bg-primary me-1">' + escapeHtml(group) + '</span>';
}).join('');
} else {
detailsHtml += '<span class="text-muted">No groups</span>';
}
detailsHtml += '</div>';
detailsHtml += '<h6 class="text-muted">Access Keys</h6>';
if (user.access_keys && user.access_keys.length > 0) {
detailsHtml += '<div class="mb-2">';
@@ -1222,10 +1369,66 @@ templ ObjectStoreUsers(data dash.ObjectStoreUsersData) {
}
}
// Reset and hide the create key form
function resetCreateKeyForm() {
document.getElementById('createKeyForm').style.display = 'none';
document.getElementById('newAccessKeyInput').value = '';
document.getElementById('newSecretKeyInput').value = '';
document.getElementById('newSecretKeyInput').type = 'password';
document.getElementById('secretKeyToggleIcon').className = 'fas fa-eye';
}
// Toggle create key form visibility
function toggleCreateKeyForm() {
const form = document.getElementById('createKeyForm');
if (form.style.display === 'none') {
form.style.display = 'block';
} else {
resetCreateKeyForm();
}
}
// Toggle secret key input visibility
function toggleSecretKeyVisibility() {
const input = document.getElementById('newSecretKeyInput');
const icon = document.getElementById('secretKeyToggleIcon');
if (input.type === 'password') {
input.type = 'text';
icon.className = 'fas fa-eye-slash';
} else {
input.type = 'password';
icon.className = 'fas fa-eye';
}
}
// Reset form when modal is dismissed
document.getElementById('accessKeysModal').addEventListener('hidden.bs.modal', resetCreateKeyForm);
// Create new access key
var isCreatingKey = false;
async function createAccessKey() {
if (isCreatingKey) return;
const username = document.getElementById('accessKeysUsername').textContent;
const accessKeyInput = document.getElementById('newAccessKeyInput');
const secretKeyInput = document.getElementById('newSecretKeyInput');
if ((accessKeyInput.value.trim() && !accessKeyInput.reportValidity()) ||
(secretKeyInput.value.trim() && !secretKeyInput.reportValidity())) {
return;
}
const accessKey = accessKeyInput.value.trim();
const secretKey = secretKeyInput.value.trim();
const body = {};
if (accessKey) body.access_key = accessKey;
if (secretKey) body.secret_key = secretKey;
isCreatingKey = true;
const createBtn = document.querySelector('#createKeyForm .btn-primary');
const cancelBtn = document.querySelector('#createKeyForm .btn-secondary');
if (createBtn) createBtn.disabled = true;
if (cancelBtn) cancelBtn.disabled = true;
try {
const encodedUsername = encodeURIComponent(username);
const response = await fetch(`/api/users/${encodedUsername}/access-keys`, {
@@ -1233,19 +1436,20 @@ templ ObjectStoreUsers(data dash.ObjectStoreUsersData) {
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({})
body: JSON.stringify(body)
});
if (response.ok) {
const result = await response.json();
// Show the new access key details (IMPORTANT: secret key is only shown once!)
// Show the new access key details
if (result.access_key) {
showNewAccessKeyModal(result.access_key);
}
showSuccessMessage('Access key created successfully');
resetCreateKeyForm();
// Refresh access keys display
refreshAccessKeysList(username);
} else {
@@ -1255,6 +1459,10 @@ templ ObjectStoreUsers(data dash.ObjectStoreUsersData) {
} catch (error) {
console.error('Error creating access key:', error);
showAlert('Failed to create access key: ' + error.message, 'error');
} finally {
isCreatingKey = false;
if (createBtn) createBtn.disabled = false;
if (cancelBtn) cancelBtn.disabled = false;
}
}
File diff suppressed because one or more lines are too long
+49 -2
View File
@@ -239,7 +239,7 @@ templ Plugin(page string) {
</div>
<div class="col-lg-4 mb-3">
<div class="card shadow-sm h-100">
<div class="card shadow-sm">
<div class="card-header">
<h5 class="mb-0"><i class="fas fa-cogs me-2"></i>Job Scheduling Settings</h5>
</div>
@@ -1026,6 +1026,9 @@ templ Plugin(page string) {
}
var jobType = String(plan.job_type || '').trim().toLowerCase();
if (jobType === 'volume_balance') {
return renderBalanceExecutionPlan(plan);
}
if (jobType !== 'erasure_coding') {
var fallbackText = toPrettyJson(plan);
if (!fallbackText) {
@@ -1107,6 +1110,44 @@ templ Plugin(page string) {
return html;
}
function renderBalanceExecutionPlan(plan) {
var html = '<div class="mb-3"><h6>Execution Plan</h6>';
var moves = Array.isArray(plan.moves) ? plan.moves : [];
if (moves.length === 0) {
// Single-move balance job
var src = textOrDash(plan.source_node || plan.source_server);
var dst = textOrDash(plan.target_node || plan.target_server);
var vid = textOrDash(plan.volume_id);
var col = textOrDash(plan.collection);
html += `<div class="row g-2 mb-2">
<div class="col-md-3"><small><strong>Volume:</strong> ${escapeHtml(vid)}</small></div>
<div class="col-md-3"><small><strong>Collection:</strong> ${escapeHtml(col)}</small></div>
<div class="col-md-3"><small><strong>Source:</strong> <code>${escapeHtml(src)}</code></small></div>
<div class="col-md-3"><small><strong>Target:</strong> <code>${escapeHtml(dst)}</code></small></div>
</div>`;
} else {
// Batch balance job
html += '<div class="mb-2"><span class="badge bg-info">' + escapeHtml(String(moves.length)) + ' moves</span></div>';
html += '<div class="table-responsive"><table class="table table-sm table-striped mb-0">' +
'<thead><tr><th>#</th><th>Volume</th><th>Source</th><th>Target</th><th>Collection</th></tr></thead><tbody>';
for (var i = 0; i < moves.length; i++) {
var move = moves[i] || {};
html += `<tr>
<td>${escapeHtml(String(i + 1))}</td>
<td>${escapeHtml(textOrDash(move.volume_id))}</td>
<td><code>${escapeHtml(textOrDash(move.source_node))}</code></td>
<td><code>${escapeHtml(textOrDash(move.target_node))}</code></td>
<td>${escapeHtml(textOrDash(move.collection))}</td>
</tr>`;
}
html += '</tbody></table></div>';
}
html += '</div>';
return html;
}
function isActiveJobState(candidateState) {
var jobState = candidateState;
if (candidateState && typeof candidateState === 'object' && candidateState.state !== undefined) {
@@ -1676,9 +1717,15 @@ templ Plugin(page string) {
barClass = 'bg-warning';
}
var jobTypeCell = escapeHtml(textOrDash(executionJob.job_type));
var execLabels = executionJob.labels || {};
if (execLabels.batch === 'true' && execLabels.batch_size) {
jobTypeCell += ' <span class="badge bg-info">' + escapeHtml(execLabels.batch_size) + ' moves</span>';
}
rows += '<tr>' +
'<td>' + renderJobLink(executionJob.job_id) + '</td>' +
'<td>' + escapeHtml(textOrDash(executionJob.job_type)) + '</td>' +
'<td>' + jobTypeCell + '</td>' +
'<td><span class="badge bg-light text-dark">' + escapeHtml(textOrDash(executionJob.state)) + '</span></td>' +
'<td class="plugin-job-progress"><div class="progress" style="height: 14px;"><div class="progress-bar ' + barClass + '" role="progressbar" style="width:' + progress + '%">' + Math.round(progress) + '%</div></div></td>' +
'<td><small>' + escapeHtml(textOrDash(executionJob.worker_id)) + '</small></td>' +
File diff suppressed because one or more lines are too long
+179 -47
View File
@@ -12,12 +12,17 @@ templ S3Buckets(data dash.S3BucketsData) {
</h1>
<div class="btn-toolbar mb-2 mb-md-0">
<div class="btn-group me-2">
<button type="button" class="btn btn-sm btn-primary"
data-bs-toggle="modal"
<select class="form-select form-select-sm me-2" id="pageSizeSelect" onchange="changePageSize()" style="width: auto;">
<option value="50" if data.PageSize == 50 { selected="selected" }>50 per page</option>
<option value="100" if data.PageSize == 100 { selected="selected" }>100 per page</option>
<option value="200" if data.PageSize == 200 { selected="selected" }>200 per page</option>
<option value="500" if data.PageSize == 500 { selected="selected" }>500 per page</option>
</select>
<button type="button" class="btn btn-sm btn-primary"
data-bs-toggle="modal"
data-bs-target="#createBucketModal">
<i class="fas fa-plus me-1"></i>Create Bucket
</button>
</div>
</div>
</div>
@@ -112,12 +117,42 @@ templ S3Buckets(data dash.S3BucketsData) {
<table class="table table-hover" width="100%" cellspacing="0" id="bucketsTable">
<thead>
<tr>
<th>Name</th>
<th>Owner</th>
<th>Created</th>
<th>Objects</th>
<th>Logical Size</th>
<th>Physical Size</th>
<th>
<a href="#" onclick="sortTable('name')" class="text-decoration-none text-dark">
Name
@getSortIcon("name", data.SortBy, data.SortOrder)
</a>
</th>
<th>
<a href="#" onclick="sortTable('owner')" class="text-decoration-none text-dark">
Owner
@getSortIcon("owner", data.SortBy, data.SortOrder)
</a>
</th>
<th>
<a href="#" onclick="sortTable('created')" class="text-decoration-none text-dark">
Created
@getSortIcon("created", data.SortBy, data.SortOrder)
</a>
</th>
<th>
<a href="#" onclick="sortTable('objects')" class="text-decoration-none text-dark">
Objects
@getSortIcon("objects", data.SortBy, data.SortOrder)
</a>
</th>
<th>
<a href="#" onclick="sortTable('logical_size')" class="text-decoration-none text-dark">
Logical Size
@getSortIcon("logical_size", data.SortBy, data.SortOrder)
</a>
</th>
<th>
<a href="#" onclick="sortTable('physical_size')" class="text-decoration-none text-dark">
Physical Size
@getSortIcon("physical_size", data.SortBy, data.SortOrder)
</a>
</th>
<th>Quota</th>
<th>Versioning</th>
<th>Object Lock</th>
@@ -256,6 +291,61 @@ templ S3Buckets(data dash.S3BucketsData) {
</tbody>
</table>
</div>
<!-- Pagination Controls -->
if data.TotalPages > 1 {
<div class="d-flex justify-content-between align-items-center mt-3">
<small class="text-muted">
Showing { fmt.Sprintf("%d", (data.CurrentPage-1)*data.PageSize+1) } to { fmt.Sprintf("%d", minInt(data.CurrentPage*data.PageSize, data.TotalBuckets)) } of { fmt.Sprintf("%d", data.TotalBuckets) } buckets
</small>
<nav aria-label="Buckets pagination">
<ul class="pagination pagination-sm mb-0">
<!-- Previous Button -->
if data.CurrentPage > 1 {
<li class="page-item">
<a class="page-link pagination-link" href="#" data-page={fmt.Sprintf("%d", data.CurrentPage-1)}>
<i class="fas fa-chevron-left"></i>
</a>
</li>
} else {
<li class="page-item disabled">
<span class="page-link">
<i class="fas fa-chevron-left"></i>
</span>
</li>
}
<!-- Page Numbers -->
for i := maxInt(1, data.CurrentPage-2); i <= minInt(data.TotalPages, data.CurrentPage+2); i++ {
if i == data.CurrentPage {
<li class="page-item active">
<span class="page-link">{fmt.Sprintf("%d", i)}</span>
</li>
} else {
<li class="page-item">
<a class="page-link pagination-link" href="#" data-page={fmt.Sprintf("%d", i)}>{fmt.Sprintf("%d", i)}</a>
</li>
}
}
<!-- Next Button -->
if data.CurrentPage < data.TotalPages {
<li class="page-item">
<a class="page-link pagination-link" href="#" data-page={fmt.Sprintf("%d", data.CurrentPage+1)}>
<i class="fas fa-chevron-right"></i>
</a>
</li>
} else {
<li class="page-item disabled">
<span class="page-link">
<i class="fas fa-chevron-right"></i>
</span>
</li>
}
</ul>
</nav>
</div>
}
</div>
</div>
</div>
@@ -574,6 +664,15 @@ templ S3Buckets(data dash.S3BucketsData) {
let cachedUsers = null;
document.addEventListener('DOMContentLoaded', function() {
// Add click handlers to pagination links
document.querySelectorAll('.pagination-link').forEach(link => {
link.addEventListener('click', function(e) {
e.preventDefault();
const page = this.getAttribute('data-page');
goToPage(page);
});
});
// Initialize modal instances once (reuse with show/hide)
deleteModalInstance = new bootstrap.Modal(document.getElementById('deleteBucketModal'));
quotaModalInstance = new bootstrap.Modal(document.getElementById('manageQuotaModal'));
@@ -1032,57 +1131,90 @@ function displayBucketDetails(data) {
document.getElementById('bucketDetailsContent').innerHTML = rows.join('');
}
function goToPage(page) {
const url = new URL(window.location);
url.searchParams.set('page', page);
window.location.href = url.toString();
}
function changePageSize() {
const pageSize = document.getElementById('pageSizeSelect').value;
const url = new URL(window.location);
url.searchParams.set('pageSize', pageSize);
url.searchParams.set('page', '1');
window.location.href = url.toString();
}
function sortTable(column) {
const url = new URL(window.location);
const currentSort = url.searchParams.get('sortBy');
const currentOrder = url.searchParams.get('sortOrder') || 'asc';
let newOrder = 'asc';
if (currentSort === column && currentOrder === 'asc') {
newOrder = 'desc';
}
url.searchParams.set('sortBy', column);
url.searchParams.set('sortOrder', newOrder);
url.searchParams.set('page', '1');
window.location.href = url.toString();
}
function exportBucketList() {
// RFC 4180 compliant CSV escaping: escape double quotes by doubling them
function escapeCsvField(value) {
const str = String(value ?? '');
// If the field contains comma, double quote, or newline, wrap in quotes and escape internal quotes
if (str.includes(',') || str.includes('"') || str.includes('\n') || str.includes('\r')) {
return '"' + str.replace(/"/g, '""') + '"';
}
return '"' + str + '"';
}
const buckets = Array.from(document.querySelectorAll('#bucketsTable tbody tr')).map(row => {
const cells = row.querySelectorAll('td');
if (cells.length > 1) {
return {
name: cells[0].textContent.trim(),
owner: cells[1].textContent.trim(),
created: cells[2].textContent.trim(),
objects: cells[3].textContent.trim(),
function formatBytes(bytes) {
if (bytes === 0) return '0 Bytes';
const k = 1024;
const sizes = ['Bytes', 'KB', 'MB', 'GB', 'TB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i];
}
logicalSize: cells[4].textContent.trim(),
physicalSize: cells[5].textContent.trim(),
quota: cells[6].textContent.trim(),
versioning: cells[7].textContent.trim(),
objectLock: cells[8].textContent.trim()
};
}
return null;
}).filter(bucket => bucket !== null);
// Fetch all buckets from the API (not just the current page)
fetch('/api/s3/buckets')
.then(response => response.json())
.then(data => {
if (data.error) {
alert('Error exporting buckets: ' + data.error);
return;
}
const csvContent = "data:text/csv;charset=utf-8," +
"Name,Owner,Logical Size,Physical Size,Object Count,Created,Quota,Versioning,Object Lock\n" +
buckets.map(b => [
escapeCsvField(b.name),
escapeCsvField(b.owner),
escapeCsvField(b.logicalSize),
escapeCsvField(b.physicalSize),
escapeCsvField(b.objects),
escapeCsvField(b.created),
escapeCsvField(b.quota),
escapeCsvField(b.versioning),
escapeCsvField(b.objectLock)
].join(',')).join("\n");
const buckets = data.buckets || [];
const csvContent = "data:text/csv;charset=utf-8," +
"Name,Owner,Logical Size,Physical Size,Object Count,Created,Quota,Versioning,Object Lock\n" +
buckets.map(b => [
escapeCsvField(b.name),
escapeCsvField(b.owner),
escapeCsvField(formatBytes(b.logical_size)),
escapeCsvField(formatBytes(b.physical_size)),
escapeCsvField(b.object_count),
escapeCsvField(b.created_at),
escapeCsvField(b.quota_enabled ? formatBytes(b.quota) : 'No quota'),
escapeCsvField(b.versioning_status || 'Not configured'),
escapeCsvField(b.object_lock_enabled ? 'Enabled' : 'Not configured')
].join(',')).join("\n");
const encodedUri = encodeURI(csvContent);
const link = document.createElement("a");
link.setAttribute("href", encodedUri);
link.setAttribute("download", "buckets.csv");
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
const encodedUri = encodeURI(csvContent);
const link = document.createElement("a");
link.setAttribute("href", encodedUri);
link.setAttribute("download", "buckets.csv");
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
})
.catch(error => {
console.error('Error:', error);
alert('Error exporting buckets: ' + error.message);
});
}
</script>
}
File diff suppressed because one or more lines are too long
+5 -44
View File
@@ -168,6 +168,11 @@ templ Layout(view ViewContext, content templ.Component) {
<i class="fas fa-users me-2"></i>Users
</a>
</li>
<li class="nav-item">
<a class="nav-link" href="/object-store/groups">
<i class="fas fa-users-cog me-2"></i>Groups
</a>
</li>
<li class="nav-item">
<a class="nav-link" href="/object-store/service-accounts">
<i class="fas fa-robot me-2"></i>Service Accounts
@@ -264,50 +269,6 @@ templ Layout(view ViewContext, content templ.Component) {
</a>
}
</li>
<li class="nav-item">
if currentPath == "/plugin/detection" {
<a class="nav-link active" href="/plugin/detection">
<i class="fas fa-search me-2"></i>Job Detection
</a>
} else {
<a class="nav-link" href="/plugin/detection">
<i class="fas fa-search me-2"></i>Job Detection
</a>
}
</li>
<li class="nav-item">
if currentPath == "/plugin/queue" {
<a class="nav-link active" href="/plugin/queue">
<i class="fas fa-list me-2"></i>Job Queue
</a>
} else {
<a class="nav-link" href="/plugin/queue">
<i class="fas fa-list me-2"></i>Job Queue
</a>
}
</li>
<li class="nav-item">
if currentPath == "/plugin/execution" {
<a class="nav-link active" href="/plugin/execution">
<i class="fas fa-tasks me-2"></i>Job Execution
</a>
} else {
<a class="nav-link" href="/plugin/execution">
<i class="fas fa-tasks me-2"></i>Job Execution
</a>
}
</li>
<li class="nav-item">
if currentPath == "/plugin/configuration" {
<a class="nav-link active" href="/plugin/configuration">
<i class="fas fa-sliders-h me-2"></i>Configuration
</a>
} else {
<a class="nav-link" href="/plugin/configuration">
<i class="fas fa-sliders-h me-2"></i>Configuration
</a>
}
</li>
</ul>
</div>
</div>
+20 -80
View File
@@ -198,7 +198,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 13, "\" id=\"storageSubmenu\"><ul class=\"nav flex-column ms-3\"><li class=\"nav-item\"><a class=\"nav-link py-2\" href=\"/storage/volumes\"><i class=\"fas fa-database me-2\"></i>Volumes</a></li><li class=\"nav-item\"><a class=\"nav-link py-2\" href=\"/storage/ec-shards\"><i class=\"fas fa-th-large me-2\"></i>EC Volumes</a></li><li class=\"nav-item\"><a class=\"nav-link py-2\" href=\"/storage/collections\"><i class=\"fas fa-layer-group me-2\"></i>Collections</a></li></ul></div></li></ul><h6 class=\"sidebar-heading px-3 mt-4 mb-1 text-muted\"><span>OBJECT STORE</span></h6><ul class=\"nav flex-column\"><li class=\"nav-item\"><a class=\"nav-link\" href=\"/object-store/buckets\"><i class=\"fas fa-cube me-2\"></i>Buckets</a></li><li class=\"nav-item\"><a class=\"nav-link\" href=\"/object-store/s3tables/buckets\"><i class=\"fas fa-table me-2\"></i>Table Buckets</a></li><li class=\"nav-item\"><a class=\"nav-link\" href=\"/object-store/users\"><i class=\"fas fa-users me-2\"></i>Users</a></li><li class=\"nav-item\"><a class=\"nav-link\" href=\"/object-store/service-accounts\"><i class=\"fas fa-robot me-2\"></i>Service Accounts</a></li><li class=\"nav-item\"><a class=\"nav-link\" href=\"/object-store/policies\"><i class=\"fas fa-shield-alt me-2\"></i>Policies</a></li></ul><h6 class=\"sidebar-heading px-3 mt-4 mb-1 text-muted\"><span>MANAGEMENT</span></h6><ul class=\"nav flex-column\"><li class=\"nav-item\"><a class=\"nav-link\" href=\"/files\"><i class=\"fas fa-folder me-2\"></i>File Browser</a></li><li class=\"nav-item\">")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 13, "\" id=\"storageSubmenu\"><ul class=\"nav flex-column ms-3\"><li class=\"nav-item\"><a class=\"nav-link py-2\" href=\"/storage/volumes\"><i class=\"fas fa-database me-2\"></i>Volumes</a></li><li class=\"nav-item\"><a class=\"nav-link py-2\" href=\"/storage/ec-shards\"><i class=\"fas fa-th-large me-2\"></i>EC Volumes</a></li><li class=\"nav-item\"><a class=\"nav-link py-2\" href=\"/storage/collections\"><i class=\"fas fa-layer-group me-2\"></i>Collections</a></li></ul></div></li></ul><h6 class=\"sidebar-heading px-3 mt-4 mb-1 text-muted\"><span>OBJECT STORE</span></h6><ul class=\"nav flex-column\"><li class=\"nav-item\"><a class=\"nav-link\" href=\"/object-store/buckets\"><i class=\"fas fa-cube me-2\"></i>Buckets</a></li><li class=\"nav-item\"><a class=\"nav-link\" href=\"/object-store/s3tables/buckets\"><i class=\"fas fa-table me-2\"></i>Table Buckets</a></li><li class=\"nav-item\"><a class=\"nav-link\" href=\"/object-store/users\"><i class=\"fas fa-users me-2\"></i>Users</a></li><li class=\"nav-item\"><a class=\"nav-link\" href=\"/object-store/groups\"><i class=\"fas fa-users-cog me-2\"></i>Groups</a></li><li class=\"nav-item\"><a class=\"nav-link\" href=\"/object-store/service-accounts\"><i class=\"fas fa-robot me-2\"></i>Service Accounts</a></li><li class=\"nav-item\"><a class=\"nav-link\" href=\"/object-store/policies\"><i class=\"fas fa-shield-alt me-2\"></i>Policies</a></li></ul><h6 class=\"sidebar-heading px-3 mt-4 mb-1 text-muted\"><span>MANAGEMENT</span></h6><ul class=\"nav flex-column\"><li class=\"nav-item\"><a class=\"nav-link\" href=\"/files\"><i class=\"fas fa-folder me-2\"></i>File Browser</a></li><li class=\"nav-item\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -269,67 +269,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 27, "</li><li class=\"nav-item\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if currentPath == "/plugin/detection" {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 28, "<a class=\"nav-link active\" href=\"/plugin/detection\"><i class=\"fas fa-search me-2\"></i>Job Detection</a>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
} else {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 29, "<a class=\"nav-link\" href=\"/plugin/detection\"><i class=\"fas fa-search me-2\"></i>Job Detection</a>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 30, "</li><li class=\"nav-item\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if currentPath == "/plugin/queue" {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 31, "<a class=\"nav-link active\" href=\"/plugin/queue\"><i class=\"fas fa-list me-2\"></i>Job Queue</a>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
} else {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 32, "<a class=\"nav-link\" href=\"/plugin/queue\"><i class=\"fas fa-list me-2\"></i>Job Queue</a>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 33, "</li><li class=\"nav-item\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if currentPath == "/plugin/execution" {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 34, "<a class=\"nav-link active\" href=\"/plugin/execution\"><i class=\"fas fa-tasks me-2\"></i>Job Execution</a>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
} else {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, "<a class=\"nav-link\" href=\"/plugin/execution\"><i class=\"fas fa-tasks me-2\"></i>Job Execution</a>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, "</li><li class=\"nav-item\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if currentPath == "/plugin/configuration" {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "<a class=\"nav-link active\" href=\"/plugin/configuration\"><i class=\"fas fa-sliders-h me-2\"></i>Configuration</a>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
} else {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, "<a class=\"nav-link\" href=\"/plugin/configuration\"><i class=\"fas fa-sliders-h me-2\"></i>Configuration</a>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 39, "</li></ul></div></div><!-- Sidebar backdrop for mobile --><div class=\"sidebar-backdrop\" id=\"sidebarBackdrop\"></div><!-- Main content --><main class=\"col-md-9 ms-sm-auto col-lg-10 px-3 px-md-4\"><div class=\"pt-3\">")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 27, "</li></ul></div></div><!-- Sidebar backdrop for mobile --><div class=\"sidebar-backdrop\" id=\"sidebarBackdrop\"></div><!-- Main content --><main class=\"col-md-9 ms-sm-auto col-lg-10 px-3 px-md-4\"><div class=\"pt-3\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -337,43 +277,43 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 40, "</div></main></div></div><!-- Footer --><footer class=\"footer mt-auto py-3 bg-light\"><div class=\"container-fluid text-center\"><small class=\"text-muted\">&copy; ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 28, "</div></main></div></div><!-- Footer --><footer class=\"footer mt-auto py-3 bg-light\"><div class=\"container-fluid text-center\"><small class=\"text-muted\">&copy; ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var14 string
templ_7745c5c3_Var14, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", time.Now().Year()))
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 331, Col: 60}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 292, Col: 60}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var14))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 41, " SeaweedFS Admin v")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 29, " SeaweedFS Admin v")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var15 string
templ_7745c5c3_Var15, templ_7745c5c3_Err = templ.JoinStringErrs(version.VERSION_NUMBER)
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 331, Col: 102}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 292, Col: 102}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var15))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 42, " ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 30, " ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if !strings.Contains(version.VERSION, "enterprise") {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 43, "<span class=\"mx-2\">•</span> <a href=\"https://seaweedfs.com\" target=\"_blank\" class=\"text-decoration-none\"><i class=\"fas fa-star me-1\"></i>Enterprise Version Available</a>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 31, "<span class=\"mx-2\">•</span> <a href=\"https://seaweedfs.com\" target=\"_blank\" class=\"text-decoration-none\"><i class=\"fas fa-star me-1\"></i>Enterprise Version Available</a>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 44, "</small></div></footer><!-- Bootstrap JS --><script src=\"/static/js/bootstrap.bundle.min.js\"></script><!-- Modal Alerts JS (replaces native alert/confirm) --><script src=\"/static/js/modal-alerts.js\"></script><!-- Custom JS --><script src=\"/static/js/admin.js\"></script><script src=\"/static/js/iam-utils.js\"></script><script src=\"/static/js/s3tables.js\"></script></body></html>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 32, "</small></div></footer><!-- Bootstrap JS --><script src=\"/static/js/bootstrap.bundle.min.js\"></script><!-- Modal Alerts JS (replaces native alert/confirm) --><script src=\"/static/js/modal-alerts.js\"></script><!-- Custom JS --><script src=\"/static/js/admin.js\"></script><script src=\"/static/js/iam-utils.js\"></script><script src=\"/static/js/s3tables.js\"></script></body></html>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -402,69 +342,69 @@ func LoginForm(title string, errorMessage string, csrfToken string) templ.Compon
templ_7745c5c3_Var16 = templ.NopComponent
}
ctx = templ.ClearChildren(ctx)
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 45, "<!doctype html><html lang=\"en\"><head><meta charset=\"UTF-8\"><title>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 33, "<!doctype html><html lang=\"en\"><head><meta charset=\"UTF-8\"><title>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var17 string
templ_7745c5c3_Var17, templ_7745c5c3_Err = templ.JoinStringErrs(title)
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 359, Col: 17}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 320, Col: 17}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var17))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 46, " - Login</title><link rel=\"icon\" href=\"/static/favicon.ico\" type=\"image/x-icon\"><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><link href=\"/static/css/bootstrap.min.css\" rel=\"stylesheet\"><link href=\"/static/css/fontawesome.min.css\" rel=\"stylesheet\"></head><body class=\"bg-light\"><div class=\"container\"><div class=\"row justify-content-center min-vh-100 align-items-center\"><div class=\"col-md-6 col-lg-4\"><div class=\"card shadow\"><div class=\"card-body p-5\"><div class=\"text-center mb-4\"><i class=\"fas fa-server fa-3x text-primary mb-3\"></i><h4 class=\"card-title\">")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 34, " - Login</title><link rel=\"icon\" href=\"/static/favicon.ico\" type=\"image/x-icon\"><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><link href=\"/static/css/bootstrap.min.css\" rel=\"stylesheet\"><link href=\"/static/css/fontawesome.min.css\" rel=\"stylesheet\"></head><body class=\"bg-light\"><div class=\"container\"><div class=\"row justify-content-center min-vh-100 align-items-center\"><div class=\"col-md-6 col-lg-4\"><div class=\"card shadow\"><div class=\"card-body p-5\"><div class=\"text-center mb-4\"><i class=\"fas fa-server fa-3x text-primary mb-3\"></i><h4 class=\"card-title\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var18 string
templ_7745c5c3_Var18, templ_7745c5c3_Err = templ.JoinStringErrs(title)
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 373, Col: 57}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 334, Col: 57}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var18))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 47, "</h4><p class=\"text-muted\">Please sign in to continue</p></div>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, "</h4><p class=\"text-muted\">Please sign in to continue</p></div>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if errorMessage != "" {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 48, "<div class=\"alert alert-danger\" role=\"alert\"><i class=\"fas fa-exclamation-triangle me-2\"></i> ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, "<div class=\"alert alert-danger\" role=\"alert\"><i class=\"fas fa-exclamation-triangle me-2\"></i> ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var19 string
templ_7745c5c3_Var19, templ_7745c5c3_Err = templ.JoinStringErrs(errorMessage)
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 380, Col: 45}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 341, Col: 45}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var19))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 49, "</div>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "</div>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, "<form method=\"POST\" action=\"/login\"><input type=\"hidden\" name=\"csrf_token\" value=\"")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, "<form method=\"POST\" action=\"/login\"><input type=\"hidden\" name=\"csrf_token\" value=\"")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var20 string
templ_7745c5c3_Var20, templ_7745c5c3_Err = templ.JoinStringErrs(csrfToken)
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 385, Col: 84}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 346, Col: 84}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var20))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 51, "\"><div class=\"mb-3\"><label for=\"username\" class=\"form-label\">Username</label><div class=\"input-group\"><span class=\"input-group-text\"><i class=\"fas fa-user\"></i></span> <input type=\"text\" class=\"form-control\" id=\"username\" name=\"username\" required></div></div><div class=\"mb-4\"><label for=\"password\" class=\"form-label\">Password</label><div class=\"input-group\"><span class=\"input-group-text\"><i class=\"fas fa-lock\"></i></span> <input type=\"password\" class=\"form-control\" id=\"password\" name=\"password\" required></div></div><button type=\"submit\" class=\"btn btn-primary w-100\"><i class=\"fas fa-sign-in-alt me-2\"></i>Sign In</button></form></div></div></div></div></div><script src=\"/static/js/bootstrap.bundle.min.js\"></script></body></html>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 39, "\"><div class=\"mb-3\"><label for=\"username\" class=\"form-label\">Username</label><div class=\"input-group\"><span class=\"input-group-text\"><i class=\"fas fa-user\"></i></span> <input type=\"text\" class=\"form-control\" id=\"username\" name=\"username\" required></div></div><div class=\"mb-4\"><label for=\"password\" class=\"form-label\">Password</label><div class=\"input-group\"><span class=\"input-group-text\"><i class=\"fas fa-lock\"></i></span> <input type=\"password\" class=\"form-control\" id=\"password\" name=\"password\" required></div></div><button type=\"submit\" class=\"btn btn-primary w-100\"><i class=\"fas fa-sign-in-alt me-2\"></i>Sign In</button></form></div></div></div></div></div><script src=\"/static/js/bootstrap.bundle.min.js\"></script></body></html>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
+21 -7
View File
@@ -114,7 +114,7 @@ Admin UI (http://localhost:23646) to manage users and policies.
var (
miniIp = cmdMini.Flag.String("ip", util.DetectedHostAddress(), "ip or server name, also used as identifier")
miniBindIp = cmdMini.Flag.String("ip.bind", "", "ip address to bind to. If empty, default to same as -ip option.")
miniBindIp = cmdMini.Flag.String("ip.bind", "0.0.0.0", "ip address to bind to. If empty, default to same as -ip option.")
miniTimeout = cmdMini.Flag.Int("idleTimeout", 30, "connection idle seconds")
miniDataCenter = cmdMini.Flag.String("dataCenter", "", "current volume server's data center name")
miniRack = cmdMini.Flag.String("rack", "", "current volume server's rack name")
@@ -555,9 +555,23 @@ func ensureAllPortsAvailableOnIP(bindIp string) error {
// If a gRPC port is 0, it will be set to httpPort + GrpcPortOffset
// This must be called after HTTP ports are finalized and before services start
func initializeGrpcPortsOnIP(bindIp string) {
// Track gRPC ports allocated during this function to prevent collisions between services
// when multiple services need fallback port allocation
allocatedGrpcPorts := make(map[int]bool)
// Track all ports allocated (both HTTP and gRPC) to prevent collisions.
// We must reserve HTTP ports so that gRPC fallback allocation never picks
// a port already assigned to an HTTP service (which hasn't bound yet).
allocatedPorts := make(map[int]bool)
// Reserve all HTTP ports first
allocatedPorts[*miniMasterOptions.port] = true
allocatedPorts[*miniFilerOptions.port] = true
allocatedPorts[*miniOptions.v.port] = true
allocatedPorts[*miniWebDavOptions.port] = true
allocatedPorts[*miniAdminOptions.port] = true
if *miniEnableS3 {
allocatedPorts[*miniS3Options.port] = true
if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 {
allocatedPorts[*miniS3Options.portIceberg] = true
}
}
grpcConfigs := []struct {
httpPort *int
@@ -593,10 +607,10 @@ func initializeGrpcPortsOnIP(bindIp string) {
// Verify the gRPC port is available (whether calculated or explicitly set)
// Check on both specific IP and all interfaces, and check against already allocated ports
if !isPortOpenOnIP(bindIp, *config.grpcPort) || !isPortAvailable(*config.grpcPort) || allocatedGrpcPorts[*config.grpcPort] {
if !isPortOpenOnIP(bindIp, *config.grpcPort) || !isPortAvailable(*config.grpcPort) || allocatedPorts[*config.grpcPort] {
glog.Warningf("gRPC port %d for %s is not available, finding alternative...", *config.grpcPort, config.name)
originalPort := *config.grpcPort
newPort := findAvailablePortOnIP(bindIp, originalPort+1, 100, allocatedGrpcPorts)
newPort := findAvailablePortOnIP(bindIp, originalPort+1, 100, allocatedPorts)
if newPort == 0 {
glog.Errorf("Could not find available gRPC port for %s starting from %d, will use %d and fail on binding", config.name, originalPort+1, originalPort)
} else {
@@ -604,7 +618,7 @@ func initializeGrpcPortsOnIP(bindIp string) {
*config.grpcPort = newPort
}
}
allocatedGrpcPorts[*config.grpcPort] = true
allocatedPorts[*config.grpcPort] = true
glog.V(1).Infof("%s gRPC port set to %d", config.name, *config.grpcPort)
}
}
+22
View File
@@ -236,3 +236,25 @@ func (cm *CredentialManager) DetachUserPolicy(ctx context.Context, username stri
func (cm *CredentialManager) ListAttachedUserPolicies(ctx context.Context, username string) ([]string, error) {
return cm.Store.ListAttachedUserPolicies(ctx, username)
}
// Group Management
func (cm *CredentialManager) CreateGroup(ctx context.Context, group *iam_pb.Group) error {
return cm.Store.CreateGroup(ctx, group)
}
func (cm *CredentialManager) GetGroup(ctx context.Context, groupName string) (*iam_pb.Group, error) {
return cm.Store.GetGroup(ctx, groupName)
}
func (cm *CredentialManager) DeleteGroup(ctx context.Context, groupName string) error {
return cm.Store.DeleteGroup(ctx, groupName)
}
func (cm *CredentialManager) ListGroups(ctx context.Context) ([]string, error) {
return cm.Store.ListGroups(ctx)
}
func (cm *CredentialManager) UpdateGroup(ctx context.Context, group *iam_pb.Group) error {
return cm.Store.UpdateGroup(ctx, group)
}
+11
View File
@@ -18,6 +18,10 @@ var (
ErrPolicyNotFound = errors.New("policy not found")
ErrPolicyAlreadyAttached = errors.New("policy already attached")
ErrPolicyNotAttached = errors.New("policy not attached to user")
ErrGroupNotFound = errors.New("group not found")
ErrGroupAlreadyExists = errors.New("group already exists")
ErrGroupNotEmpty = errors.New("group is not empty")
ErrUserNotInGroup = errors.New("user is not a member of the group")
)
// CredentialStoreTypeName represents the type name of a credential store
@@ -94,6 +98,13 @@ type CredentialStore interface {
// ListAttachedUserPolicies returns the list of policy names attached to a user
ListAttachedUserPolicies(ctx context.Context, username string) ([]string, error)
// Group Management
CreateGroup(ctx context.Context, group *iam_pb.Group) error
GetGroup(ctx context.Context, groupName string) (*iam_pb.Group, error)
DeleteGroup(ctx context.Context, groupName string) error
ListGroups(ctx context.Context) ([]string, error)
UpdateGroup(ctx context.Context, group *iam_pb.Group) error
// Shutdown performs cleanup when the store is being shut down
Shutdown()
}
@@ -0,0 +1,182 @@
package filer_etc
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"github.com/seaweedfs/seaweedfs/weed/credential"
"github.com/seaweedfs/seaweedfs/weed/filer"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/iam_pb"
)
const IamGroupsDirectory = "groups"
func (store *FilerEtcStore) loadGroupsFromMultiFile(ctx context.Context, s3cfg *iam_pb.S3ApiConfiguration) error {
return store.withFilerClient(func(client filer_pb.SeaweedFilerClient) error {
dir := filer.IamConfigDirectory + "/" + IamGroupsDirectory
entries, err := listEntries(ctx, client, dir)
if err != nil {
if errors.Is(err, filer_pb.ErrNotFound) {
return nil
}
return err
}
for _, entry := range entries {
if entry.IsDirectory {
continue
}
var content []byte
if len(entry.Content) > 0 {
content = entry.Content
} else {
c, err := filer.ReadInsideFiler(ctx, client, dir, entry.Name)
if err != nil {
return fmt.Errorf("failed to read group file %s: %w", entry.Name, err)
}
content = c
}
if len(content) > 0 {
g := &iam_pb.Group{}
if err := json.Unmarshal(content, g); err != nil {
return fmt.Errorf("failed to unmarshal group %s: %w", entry.Name, err)
}
// Merge: overwrite existing group with same name or append
found := false
for i, existing := range s3cfg.Groups {
if existing.Name == g.Name {
s3cfg.Groups[i] = g
found = true
break
}
}
if !found {
s3cfg.Groups = append(s3cfg.Groups, g)
}
}
}
return nil
})
}
func (store *FilerEtcStore) saveGroup(ctx context.Context, group *iam_pb.Group) error {
if group == nil {
return fmt.Errorf("group is nil")
}
group.Name = strings.TrimSpace(group.Name)
if group.Name == "" {
return fmt.Errorf("group name is required")
}
return store.withFilerClient(func(client filer_pb.SeaweedFilerClient) error {
data, err := json.MarshalIndent(group, "", " ")
if err != nil {
return err
}
return filer.SaveInsideFiler(client, filer.IamConfigDirectory+"/"+IamGroupsDirectory, group.Name+".json", data)
})
}
func (store *FilerEtcStore) deleteGroupFile(ctx context.Context, groupName string) error {
return store.withFilerClient(func(client filer_pb.SeaweedFilerClient) error {
resp, err := client.DeleteEntry(ctx, &filer_pb.DeleteEntryRequest{
Directory: filer.IamConfigDirectory + "/" + IamGroupsDirectory,
Name: groupName + ".json",
})
if err != nil {
if strings.Contains(err.Error(), filer_pb.ErrNotFound.Error()) {
return credential.ErrGroupNotFound
}
return err
}
if resp != nil && resp.Error != "" {
if strings.Contains(resp.Error, filer_pb.ErrNotFound.Error()) {
return credential.ErrGroupNotFound
}
return fmt.Errorf("delete group %s: %s", groupName, resp.Error)
}
return nil
})
}
func (store *FilerEtcStore) CreateGroup(ctx context.Context, group *iam_pb.Group) error {
if group != nil {
group.Name = strings.TrimSpace(group.Name)
}
if group == nil || group.Name == "" {
return fmt.Errorf("group name is required")
}
existing, err := store.GetGroup(ctx, group.Name)
if err != nil {
if !errors.Is(err, credential.ErrGroupNotFound) {
return err
}
} else if existing != nil {
return credential.ErrGroupAlreadyExists
}
return store.saveGroup(ctx, group)
}
func (store *FilerEtcStore) GetGroup(ctx context.Context, groupName string) (*iam_pb.Group, error) {
var group *iam_pb.Group
err := store.withFilerClient(func(client filer_pb.SeaweedFilerClient) error {
data, err := filer.ReadInsideFiler(ctx, client, filer.IamConfigDirectory+"/"+IamGroupsDirectory, groupName+".json")
if err != nil {
if errors.Is(err, filer_pb.ErrNotFound) {
return credential.ErrGroupNotFound
}
return err
}
if len(data) == 0 {
return credential.ErrGroupNotFound
}
group = &iam_pb.Group{}
return json.Unmarshal(data, group)
})
return group, err
}
func (store *FilerEtcStore) DeleteGroup(ctx context.Context, groupName string) error {
if _, err := store.GetGroup(ctx, groupName); err != nil {
return err
}
return store.deleteGroupFile(ctx, groupName)
}
func (store *FilerEtcStore) ListGroups(ctx context.Context) ([]string, error) {
var names []string
err := store.withFilerClient(func(client filer_pb.SeaweedFilerClient) error {
entries, err := listEntries(ctx, client, filer.IamConfigDirectory+"/"+IamGroupsDirectory)
if err != nil {
if errors.Is(err, filer_pb.ErrNotFound) {
return nil
}
return err
}
for _, entry := range entries {
if !entry.IsDirectory && strings.HasSuffix(entry.Name, ".json") {
names = append(names, strings.TrimSuffix(entry.Name, ".json"))
}
}
return nil
})
return names, err
}
func (store *FilerEtcStore) UpdateGroup(ctx context.Context, group *iam_pb.Group) error {
if group != nil {
group.Name = strings.TrimSpace(group.Name)
}
if group == nil || group.Name == "" {
return fmt.Errorf("group name is required")
}
if _, err := store.GetGroup(ctx, group.Name); err != nil {
return err
}
return store.saveGroup(ctx, group)
}
@@ -45,6 +45,11 @@ func (store *FilerEtcStore) LoadConfiguration(ctx context.Context) (*iam_pb.S3Ap
return s3cfg, fmt.Errorf("failed to load service accounts: %w", err)
}
// 3b. Load groups
if err := store.loadGroupsFromMultiFile(ctx, s3cfg); err != nil {
return s3cfg, fmt.Errorf("failed to load groups: %w", err)
}
// 4. Perform migration if we loaded legacy config
// This ensures that all identities (including legacy ones) are written to individual files
// and the legacy file is renamed.
@@ -144,7 +149,14 @@ func (store *FilerEtcStore) migrateToMultiFile(ctx context.Context, s3cfg *iam_p
}
}
// 3. Rename legacy file
// 3. Save all groups
for _, g := range s3cfg.Groups {
if err := store.saveGroup(ctx, g); err != nil {
return err
}
}
// 4. Rename legacy file
return store.withFilerClient(func(client filer_pb.SeaweedFilerClient) error {
_, err := client.AtomicRenameEntry(ctx, &filer_pb.AtomicRenameEntryRequest{
OldDirectory: filer.IamConfigDirectory,
@@ -171,6 +183,13 @@ func (store *FilerEtcStore) SaveConfiguration(ctx context.Context, config *iam_p
}
}
// 2b. Save all groups
for _, g := range config.Groups {
if err := store.saveGroup(ctx, g); err != nil {
return err
}
}
// 3. Cleanup removed identities (Full Sync)
if err := store.withFilerClient(func(client filer_pb.SeaweedFilerClient) error {
dir := filer.IamConfigDirectory + "/" + IamIdentitiesDirectory
@@ -234,6 +253,40 @@ func (store *FilerEtcStore) SaveConfiguration(ctx context.Context, config *iam_p
return err
}
// 5. Cleanup removed groups (Full Sync)
if err := store.withFilerClient(func(client filer_pb.SeaweedFilerClient) error {
dir := filer.IamConfigDirectory + "/" + IamGroupsDirectory
entries, err := listEntries(ctx, client, dir)
if err != nil {
if err == filer_pb.ErrNotFound {
return nil
}
return err
}
validNames := make(map[string]bool)
for _, g := range config.Groups {
validNames[g.Name+".json"] = true
}
for _, entry := range entries {
if !entry.IsDirectory && !validNames[entry.Name] {
resp, err := client.DeleteEntry(ctx, &filer_pb.DeleteEntryRequest{
Directory: dir,
Name: entry.Name,
})
if err != nil {
glog.Warningf("Failed to delete obsolete group file %s: %v", entry.Name, err)
} else if resp != nil && resp.Error != "" {
glog.Warningf("Failed to delete obsolete group file %s: %s", entry.Name, resp.Error)
}
}
}
return nil
}); err != nil {
return err
}
return nil
}
+87
View File
@@ -0,0 +1,87 @@
package grpc
import (
"context"
"fmt"
"github.com/seaweedfs/seaweedfs/weed/credential"
"github.com/seaweedfs/seaweedfs/weed/pb/iam_pb"
)
// NOTE: The gRPC store uses a load-modify-save pattern for all operations,
// which is inherently subject to race conditions under concurrent access.
// This matches the existing pattern used for identities and policies.
// A future improvement would add dedicated gRPC RPCs for atomic group operations.
func (store *IamGrpcStore) CreateGroup(ctx context.Context, group *iam_pb.Group) error {
if group == nil || group.Name == "" {
return fmt.Errorf("group name is required")
}
config, err := store.LoadConfiguration(ctx)
if err != nil {
return err
}
for _, g := range config.Groups {
if g.Name == group.Name {
return credential.ErrGroupAlreadyExists
}
}
config.Groups = append(config.Groups, group)
return store.SaveConfiguration(ctx, config)
}
func (store *IamGrpcStore) GetGroup(ctx context.Context, groupName string) (*iam_pb.Group, error) {
config, err := store.LoadConfiguration(ctx)
if err != nil {
return nil, err
}
for _, g := range config.Groups {
if g.Name == groupName {
return g, nil
}
}
return nil, credential.ErrGroupNotFound
}
func (store *IamGrpcStore) DeleteGroup(ctx context.Context, groupName string) error {
config, err := store.LoadConfiguration(ctx)
if err != nil {
return err
}
for i, g := range config.Groups {
if g.Name == groupName {
config.Groups = append(config.Groups[:i], config.Groups[i+1:]...)
return store.SaveConfiguration(ctx, config)
}
}
return credential.ErrGroupNotFound
}
func (store *IamGrpcStore) ListGroups(ctx context.Context) ([]string, error) {
config, err := store.LoadConfiguration(ctx)
if err != nil {
return nil, err
}
var names []string
for _, g := range config.Groups {
names = append(names, g.Name)
}
return names, nil
}
func (store *IamGrpcStore) UpdateGroup(ctx context.Context, group *iam_pb.Group) error {
if group == nil || group.Name == "" {
return fmt.Errorf("group name is required")
}
config, err := store.LoadConfiguration(ctx)
if err != nil {
return err
}
for i, g := range config.Groups {
if g.Name == group.Name {
config.Groups[i] = group
return store.SaveConfiguration(ctx, config)
}
}
return credential.ErrGroupNotFound
}
+89
View File
@@ -0,0 +1,89 @@
package memory
import (
"context"
"fmt"
"github.com/seaweedfs/seaweedfs/weed/credential"
"github.com/seaweedfs/seaweedfs/weed/pb/iam_pb"
)
// cloneGroup creates a deep copy of an iam_pb.Group.
func cloneGroup(g *iam_pb.Group) *iam_pb.Group {
if g == nil {
return nil
}
clone := &iam_pb.Group{
Name: g.Name,
Disabled: g.Disabled,
}
if g.Members != nil {
clone.Members = make([]string, len(g.Members))
copy(clone.Members, g.Members)
}
if g.PolicyNames != nil {
clone.PolicyNames = make([]string, len(g.PolicyNames))
copy(clone.PolicyNames, g.PolicyNames)
}
return clone
}
func (store *MemoryStore) CreateGroup(ctx context.Context, group *iam_pb.Group) error {
if group == nil || group.Name == "" {
return fmt.Errorf("group name is required")
}
store.mu.Lock()
defer store.mu.Unlock()
if _, exists := store.groups[group.Name]; exists {
return credential.ErrGroupAlreadyExists
}
store.groups[group.Name] = cloneGroup(group)
return nil
}
func (store *MemoryStore) GetGroup(ctx context.Context, groupName string) (*iam_pb.Group, error) {
store.mu.RLock()
defer store.mu.RUnlock()
if g, exists := store.groups[groupName]; exists {
return cloneGroup(g), nil
}
return nil, credential.ErrGroupNotFound
}
func (store *MemoryStore) DeleteGroup(ctx context.Context, groupName string) error {
store.mu.Lock()
defer store.mu.Unlock()
if _, exists := store.groups[groupName]; !exists {
return credential.ErrGroupNotFound
}
delete(store.groups, groupName)
return nil
}
func (store *MemoryStore) ListGroups(ctx context.Context) ([]string, error) {
store.mu.RLock()
defer store.mu.RUnlock()
var names []string
for name := range store.groups {
names = append(names, name)
}
return names, nil
}
func (store *MemoryStore) UpdateGroup(ctx context.Context, group *iam_pb.Group) error {
if group == nil || group.Name == "" {
return fmt.Errorf("group name is required")
}
store.mu.Lock()
defer store.mu.Unlock()
if _, exists := store.groups[group.Name]; !exists {
return credential.ErrGroupNotFound
}
store.groups[group.Name] = cloneGroup(group)
return nil
}
+4
View File
@@ -23,6 +23,7 @@ type MemoryStore struct {
serviceAccounts map[string]*iam_pb.ServiceAccount // id -> service_account
serviceAccountAccessKeys map[string]string // access_key -> id
policies map[string]policy_engine.PolicyDocument // policy_name -> policy_document
groups map[string]*iam_pb.Group // group_name -> group
initialized bool
}
@@ -43,6 +44,7 @@ func (store *MemoryStore) Initialize(configuration util.Configuration, prefix st
store.serviceAccounts = make(map[string]*iam_pb.ServiceAccount)
store.serviceAccountAccessKeys = make(map[string]string)
store.policies = make(map[string]policy_engine.PolicyDocument)
store.groups = make(map[string]*iam_pb.Group)
store.initialized = true
return nil
@@ -57,6 +59,7 @@ func (store *MemoryStore) Shutdown() {
store.serviceAccounts = nil
store.serviceAccountAccessKeys = nil
store.policies = nil
store.groups = nil
store.initialized = false
}
@@ -71,6 +74,7 @@ func (store *MemoryStore) Reset() {
store.serviceAccounts = make(map[string]*iam_pb.ServiceAccount)
store.serviceAccountAccessKeys = make(map[string]string)
store.policies = make(map[string]policy_engine.PolicyDocument)
store.groups = make(map[string]*iam_pb.Group)
}
}
+127
View File
@@ -0,0 +1,127 @@
package postgres
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"github.com/jackc/pgx/v5/pgconn"
"github.com/seaweedfs/seaweedfs/weed/credential"
"github.com/seaweedfs/seaweedfs/weed/pb/iam_pb"
)
func (store *PostgresStore) CreateGroup(ctx context.Context, group *iam_pb.Group) error {
if group == nil || group.Name == "" {
return fmt.Errorf("group name is required")
}
membersJSON, err := json.Marshal(group.Members)
if err != nil {
return fmt.Errorf("failed to marshal members: %w", err)
}
policyNamesJSON, err := json.Marshal(group.PolicyNames)
if err != nil {
return fmt.Errorf("failed to marshal policy_names: %w", err)
}
_, err = store.db.ExecContext(ctx,
`INSERT INTO groups (name, members, policy_names, disabled) VALUES ($1, $2, $3, $4)`,
group.Name, membersJSON, policyNamesJSON, group.Disabled)
if err != nil {
var pgErr *pgconn.PgError
if errors.As(err, &pgErr) && pgErr.Code == "23505" {
return credential.ErrGroupAlreadyExists
}
return fmt.Errorf("failed to create group: %w", err)
}
return nil
}
func (store *PostgresStore) GetGroup(ctx context.Context, groupName string) (*iam_pb.Group, error) {
var membersJSON, policyNamesJSON []byte
var disabled bool
err := store.db.QueryRowContext(ctx,
`SELECT members, policy_names, disabled FROM groups WHERE name = $1`, groupName).
Scan(&membersJSON, &policyNamesJSON, &disabled)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return nil, credential.ErrGroupNotFound
}
return nil, fmt.Errorf("failed to get group: %w", err)
}
group := &iam_pb.Group{
Name: groupName,
Disabled: disabled,
}
if err := json.Unmarshal(membersJSON, &group.Members); err != nil {
return nil, fmt.Errorf("failed to unmarshal members: %w", err)
}
if err := json.Unmarshal(policyNamesJSON, &group.PolicyNames); err != nil {
return nil, fmt.Errorf("failed to unmarshal policy_names: %w", err)
}
return group, nil
}
func (store *PostgresStore) DeleteGroup(ctx context.Context, groupName string) error {
result, err := store.db.ExecContext(ctx, `DELETE FROM groups WHERE name = $1`, groupName)
if err != nil {
return fmt.Errorf("failed to delete group: %w", err)
}
rows, err := result.RowsAffected()
if err != nil {
return fmt.Errorf("failed to get rows affected: %w", err)
}
if rows == 0 {
return credential.ErrGroupNotFound
}
return nil
}
func (store *PostgresStore) ListGroups(ctx context.Context) ([]string, error) {
rows, err := store.db.QueryContext(ctx, `SELECT name FROM groups ORDER BY name`)
if err != nil {
return nil, fmt.Errorf("failed to list groups: %w", err)
}
defer rows.Close()
var names []string
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
return nil, fmt.Errorf("failed to scan group name: %w", err)
}
names = append(names, name)
}
return names, rows.Err()
}
func (store *PostgresStore) UpdateGroup(ctx context.Context, group *iam_pb.Group) error {
if group == nil || group.Name == "" {
return fmt.Errorf("group name is required")
}
membersJSON, err := json.Marshal(group.Members)
if err != nil {
return fmt.Errorf("failed to marshal members: %w", err)
}
policyNamesJSON, err := json.Marshal(group.PolicyNames)
if err != nil {
return fmt.Errorf("failed to marshal policy_names: %w", err)
}
result, err := store.db.ExecContext(ctx,
`UPDATE groups SET members = $1, policy_names = $2, disabled = $3, updated_at = CURRENT_TIMESTAMP WHERE name = $4`,
membersJSON, policyNamesJSON, group.Disabled, group.Name)
if err != nil {
return fmt.Errorf("failed to update group: %w", err)
}
rows, err := result.RowsAffected()
if err != nil {
return fmt.Errorf("failed to get rows affected: %w", err)
}
if rows == 0 {
return credential.ErrGroupNotFound
}
return nil
}

Some files were not shown because too many files have changed in this diff Show More