Compare commits

..
Author SHA1 Message Date
Chris Lu 064d4e06f5 guard source deletion on ec success 2026-02-27 17:48:35 -08:00
Chris Lu 8f0ec1cce7 make WithFields compatible and wrap errors 2026-02-27 16:58:32 -08:00
Chris Lu d14668aa3f fix mount error aggregation scope 2026-02-27 16:51:04 -08:00
Chris Lu b8d13afb3f fix shard assignment merge and mount errors 2026-02-27 16:42:08 -08:00
Chris Lu cb8b604087 refactor ec shard distribution 2026-02-27 16:32:56 -08:00
Chris LuandGitHub e8946e59ca fix(s3api): correctly extract host header port in extractHostHeader (#8464)
* Prevent concurrent maintenance tasks per volume

* fix panic

* fix(s3api): correctly extract host header port when X-Forwarded-Port is present

* test(s3api): add test cases for misreported X-Forwarded-Port
2026-02-27 13:41:45 -08:00
Chris LuandGitHub b9e560dcf1 Prevent overlapping maintenance tasks per volume (#8463)
* Prevent concurrent maintenance tasks per volume

* fix panic
2026-02-27 13:14:52 -08:00
Chris LuandGitHub 4f647e1036 Worker set its working directory (#8461)
* set working directory

* consolidate to worker directory

* working directory

* correct directory name

* refactoring to use wildcard matcher

* simplify

* cleaning ec working directory

* fix reference

* clean

* adjust test
2026-02-27 12:22:21 -08:00
Chris Lu cf3b7b3ad7 adjust weight 2026-02-26 19:46:38 -08:00
Chris Lu 09a1ace53a adjust display name 2026-02-26 19:33:52 -08:00
Chris LuandGitHub c73e65ad5e Add customizable plugin display names and weights (#8459)
* feat: add customizable plugin display names and weights

- Add weight field to JobTypeCapability proto message
- Modify ListKnownJobTypes() to return JobTypeInfo with display names and weights
- Modify ListPluginJobTypes() to return JobTypeInfo instead of string
- Sort plugins by weight (descending) then alphabetically
- Update admin API to return enriched job type metadata
- Update plugin UI template to display names instead of IDs
- Consolidate API by reusing existing function names instead of suffixed variants

* perf: optimize plugin job type capability lookup and add null-safe parsing

- Pre-calculate job type capabilities in a map to reduce O(n*m) nested loops
  to O(n+m) lookup time in ListKnownJobTypes()
- Add parseJobTypeItem() helper function for null-safe job type item parsing
- Refactor plugin.templ to use parseJobTypeItem() in all job type access points
  (hasJobType, applyInitialNavigation, ensureActiveNavigation, renderTopTabs)
- Deterministic capability resolution by using first worker's capability

* templ

* refactor: use parseJobTypeItem helper consistently in plugin.templ

Replace duplicated job type extraction logic at line 1296-1298 with
parseJobTypeItem() helper function for consistency and maintainability.

* improve: prefer richer capability metadata and add null-safety checks

- Improve capability selection in ListKnownJobTypes() to prefer capabilities
  with non-empty DisplayName and higher Weight across all workers instead of
  first-wins approach. Handles mixed-version clusters better.
- Add defensive null checks in renderJobTypeSummary() to safely access
  parseJobTypeItem() result before property access
- Ensures malformed or missing entries won't break the rendering pipeline

* fix: preserve existing DisplayName when merging capabilities

Fix capability merge logic to respect existing DisplayName values:
- If existing has DisplayName but candidate doesn't, preserve existing
- If existing doesn't have DisplayName but candidate does, use candidate
- Only use Weight comparison if DisplayName status is equal
- Prevents higher-weight capabilities with empty DisplayName from
  overriding capabilities with non-empty DisplayName
2026-02-26 19:20:48 -08:00
Chris LuandGitHub 8eba7ba5b2 feat: drop table location mapping support (#8458)
* feat: drop table location mapping support

Disable external metadata locations for S3 Tables and remove the table location
mapping index entirely. Table metadata must live under the table bucket paths,
so lookups no longer use mapping directories.

Changes:
- Remove mapping lookup and cache from bucket path resolution
- Reject metadataLocation in CreateTable and UpdateTable
- Remove mapping helpers and tests

* compile

* refactor

* fix: accept metadataLocation in S3 Tables API requests

We removed the external table location mapping feature, but still need to
accept and store metadataLocation values from clients like Trino. The mapping
feature was an internal implementation detail that mapped external buckets to
internal table paths. The metadataLocation field itself is part of the S3 Tables
API and should be preserved.

* fmt

* fix: handle MetadataLocation in UpdateTable requests

Mirror handleCreateTable behavior by updating metadata.MetadataLocation
when req.MetadataLocation is provided in UpdateTable requests. This ensures
table metadata location can be updated, not just set during creation.
2026-02-26 16:36:24 -08:00
Chris LuandGitHub 641351da78 fix: table location mappings to /etc/s3tables (#8457)
* fix: move table location mappings to /etc/s3tables to avoid bucket name validation

Fixes #8362 - table location mappings were stored under /buckets/.table-location-mappings
which fails bucket name validation because it starts with a dot. Moving them to
/etc/s3tables resolves the migration error for upgrades.

Changes:
- Table location mappings now stored under /etc/s3tables
- Ensure parent /etc directory exists before creating /etc/s3tables
- Normal writes go to new location only (no legacy compatibility)
- Removed bucket name validation exception for old location

* refactor: simplify lookupTableLocationMapping by removing redundant mappingPath parameter

The mappingPath function parameter was redundant as the path can be derived
from mappingDir and bucket using path.Join. This simplifies the code and
reduces the risk of path mismatches between parameters.
2026-02-26 15:35:13 -08:00
3d81d5bef7 Fix S3 signature verification behind reverse proxies (#8444)
* Fix S3 signature verification behind reverse proxies

When SeaweedFS is deployed behind a reverse proxy (e.g. nginx, Kong,
Traefik), AWS S3 Signature V4 verification fails because the Host header
the client signed with (e.g. "localhost:9000") differs from the Host
header SeaweedFS receives on the backend (e.g. "seaweedfs:8333").

This commit adds a new -s3.externalUrl parameter (and S3_EXTERNAL_URL
environment variable) that tells SeaweedFS what public-facing URL clients
use to connect. When set, SeaweedFS uses this host value for signature
verification instead of the Host header from the incoming request.

New parameter:
  -s3.externalUrl  (flag) or S3_EXTERNAL_URL (environment variable)
  Example: -s3.externalUrl=http://localhost:9000
  Example: S3_EXTERNAL_URL=https://s3.example.com

The environment variable is particularly useful in Docker/Kubernetes
deployments where the external URL is injected via container config.
The flag takes precedence over the environment variable when both are set.

At startup, the URL is parsed and default ports are stripped to match
AWS SDK behavior (port 80 for HTTP, port 443 for HTTPS), so
"http://s3.example.com:80" and "http://s3.example.com" are equivalent.

Bugs fixed:
- Default port stripping was removed by a prior PR, causing signature
  mismatches when clients connect on standard ports (80/443)
- X-Forwarded-Port was ignored when X-Forwarded-Host was not present
- Scheme detection now uses proper precedence: X-Forwarded-Proto >
  TLS connection > URL scheme > "http"
- Test expectations for standard port stripping were incorrect
- expectedHost field in TestSignatureV4WithForwardedPort was declared
  but never actually checked (self-referential test)

* Add Docker integration test for S3 proxy signature verification

Docker Compose setup with nginx reverse proxy to validate that the
-s3.externalUrl parameter (or S3_EXTERNAL_URL env var) correctly
resolves S3 signature verification when SeaweedFS runs behind a proxy.

The test uses nginx proxying port 9000 to SeaweedFS on port 8333,
with X-Forwarded-Host/Port/Proto headers set. SeaweedFS is configured
with -s3.externalUrl=http://localhost:9000 so it uses "localhost:9000"
for signature verification, matching what the AWS CLI signs with.

The test can be run with aws CLI on the host or without it by using
the amazon/aws-cli Docker image with --network host.

Test covers: create-bucket, list-buckets, put-object, head-object,
list-objects-v2, get-object, content round-trip integrity,
delete-object, and delete-bucket — all through the reverse proxy.

* Create s3-proxy-signature-tests.yml

* fix CLI

* fix CI

* Update s3-proxy-signature-tests.yml

* address comments

* Update Dockerfile

* add user

* no need for fuse

* Update s3-proxy-signature-tests.yml

* debug

* weed mini

* fix health check

* health check

* fix health checking

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-02-26 14:20:42 -08:00
ae02d47433 helm: add optional parameters to COSI BucketClass (#8453)
Add cosi.bucketClassParameters to allow passing arbitrary parameters
to the default BucketClass resource. This enables use cases like
tiered storage where a diskType parameter needs to be set on the
BucketClass to route objects to specific volume servers.

When bucketClassParameters is empty (default), the BucketClass is
rendered without a parameters block, preserving backward compatibility.

Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Co-authored-by: Claude <noreply@anthropic.com>
2026-02-26 12:19:07 -08:00
Chris LuGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
9b6fc49946 Chart createBuckets config #8368: Add TTL, Object Lock, and Versioning support (#8375)
* Chart createBuckets config #8368: Add TTL, Object Lock, and Versioning support

* Update weed/shell/command_s3_bucket_versioning.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* address comments

* address comments

* go fmt

* fix failures are still treated like “bucket not found”

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-02-26 11:56:10 -08:00
0fac6e39ea weed/s3api/s3tables: fix dropped errors (#8456)
* weed/s3api/s3tables: fix dropped errors

* enhance errors

* fail fast when listing tables

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-02-26 11:12:10 -08:00
Chris LuandGitHub 453310b057 Add plugin worker integration tests for erasure coding (#8450)
* test: add plugin worker integration harness

* test: add erasure coding detection integration tests

* test: add erasure coding execution integration tests

* ci: add plugin worker integration workflow

* test: extend fake volume server for vacuum and balance

* test: expand erasure coding detection topologies

* test: add large erasure coding detection topology

* test: add vacuum plugin worker integration tests

* test: add volume balance plugin worker integration tests

* ci: run plugin worker tests per worker

* fixes

* erasure coding: stop after placement failures

* erasure coding: record hasMore when early stopping

* erasure coding: relax large topology expectations
2026-02-25 22:11:41 -08:00
9b6bbf7d45 Remove volumePreallocate option from docker containers (#8451)
Some filesystems, such as XFS, may over-allocate disk spaces when using
volume preallocation. Remove this option from the default docker entrypoint
scripts to allow volumes to use only the necessary disk space.

Fixes: https://github.com/seaweedfs/seaweedfs/issues/6465#issuecomment-3964174718

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-02-25 22:11:15 -08:00
Chris LuandGitHub d2b92938ee Make EC detection context aware (#8449)
* Make EC detection context aware

* Update register.go

* Speed up EC detection planning

* Add tests for EC detection planner

* optimizations

detection.go: extracted ParseCollectionFilter (exported) and feed it into the detection loop so both detection and tracing share the same parsing/whitelisting logic; the detection loop now iterates on a sorted list of volume IDs, checks the context at every iteration, and only sets hasMore when there are still unprocessed groups after hitting maxResults, keeping runtime bounded while still scheduling planned tasks before returning the results.
erasure_coding_handler.go: dropped the duplicated inline filter parsing in emitErasureCodingDetectionDecisionTrace and now reuse erasurecodingtask.ParseCollectionFilter, and the summary suffix logic now only accounts for the hasMore case that can actually happen.
detection_test.go: updated the helper topology builder to use master_pb.VolumeInformationMessage (matching the current protobuf types) and tightened the cancellation/max-results tests so they reliably exercise the detection logic (cancel before calling Detection, and provide enough disks so one result is produced before the limit).

* use working directory

* fix compilation

* fix compilation

* rename

* go vet

* fix getenv

* address comments, fix error
2026-02-25 18:02:35 -08:00
Chris LuGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
7f6e58b791 Fix SFTP file upload failures with JWT filer tokens (#8448)
* Fix SFTP file upload failures with JWT filer tokens (issue #8425)

When JWT authentication is enabled for filer operations via jwt.filer_signing.*
configuration, SFTP server file upload requests were rejected because they lacked
JWT authorization headers.

Changes:
- Added JWT signing key and expiration fields to SftpServer struct
- Modified putFile() to generate and include JWT tokens in upload requests
- Enhanced SFTPServiceOptions with JWT configuration fields
- Updated SFTP command startup to load and pass JWT config to service

This allows SFTP uploads to authenticate with JWT-enabled filers, consistent
with how other SeaweedFS components (S3 API, file browser) handle filer auth.

Fixes #8425

* Apply suggestion from @gemini-code-assist[bot]

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-02-25 14:30:21 -08:00
Chris Lu a92e9baddf Add integration test for multipart operations inheriting s3:PutObject permissions
TestS3MultipartOperationsInheritPutObjectPermissions verifies that multipart
upload operations (CreateMultipartUpload, UploadPart, ListParts,
CompleteMultipartUpload, AbortMultipartUpload, ListMultipartUploads) work
correctly when a user has only s3:PutObject permission granted.

This test validates the behavior where multipart operations are implicitly
granted when s3:PutObject is authorized, as multipart upload is an
implementation detail of putting objects in S3.
2026-02-25 13:38:14 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Chris Lu
52be2edcb1 build(deps): bump github.com/parquet-go/parquet-go from 0.26.4 to 0.27.0 (#8410)
* build(deps): bump github.com/parquet-go/parquet-go from 0.26.4 to 0.27.0

Bumps [github.com/parquet-go/parquet-go](https://github.com/parquet-go/parquet-go) from 0.26.4 to 0.27.0.
- [Release notes](https://github.com/parquet-go/parquet-go/releases)
- [Changelog](https://github.com/parquet-go/parquet-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/parquet-go/parquet-go/compare/v0.26.4...v0.27.0)

---
updated-dependencies:
- dependency-name: github.com/parquet-go/parquet-go
  dependency-version: 0.27.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix parser

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-02-25 13:15:36 -08:00
Chris LuGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
b9fa05153a Allow multipart upload operations when s3:PutObject is authorized (#8445)
* Allow multipart upload operations when s3:PutObject is authorized

Multipart upload is an implementation detail of putting objects, not a separate
permission. When a policy grants s3:PutObject, it should implicitly allow:
- s3:CreateMultipartUpload
- s3:UploadPart
- s3:CompleteMultipartUpload
- s3:AbortMultipartUpload
- s3:ListParts

This fixes a compatibility issue where clients like PyArrow that use multipart
uploads by default would fail even though the role had s3:PutObject permission.
The session policy intersection still applies - both the identity-based policy
AND session policy must allow s3:PutObject for multipart operations to work.

Implementation:
- Added constants for S3 multipart action strings
- Added multipartActionSet to efficiently check if action is multipart-related
- Updated MatchesAction method to implicitly grant multipart when PutObject allowed

* Update weed/s3api/policy_engine/types.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Add s3:ListMultipartUploads to multipart action set

Include s3:ListMultipartUploads in the multipartActionSet so that listing
multipart uploads is implicitly granted when s3:PutObject is authorized.
ListMultipartUploads is a critical part of the multipart upload workflow,
allowing clients to query in-progress uploads before completing them.

Changes:
- Added s3ListMultipartUploads constant definition
- Included s3ListMultipartUploads in multipartActionSet initialization
- Existing references to multipartActionSet automatically now cover ListMultipartUploads

All policy engine tests pass (0.351s execution time)

* Refactor: reuse multipart action constants from s3_constants package

Remove duplicate constant definitions from policy_engine/types.go and import
the canonical definitions from s3api/s3_constants/s3_actions.go instead.
This eliminates duplication and ensures a single source of truth for
multipart action strings:

- ACTION_CREATE_MULTIPART_UPLOAD
- ACTION_UPLOAD_PART
- ACTION_COMPLETE_MULTIPART
- ACTION_ABORT_MULTIPART
- ACTION_LIST_PARTS
- ACTION_LIST_MULTIPART_UPLOADS

All policy engine tests pass (0.350s execution time)

* Fix S3_ACTION_LIST_MULTIPART_UPLOADS constant value

Move S3_ACTION_LIST_MULTIPART_UPLOADS from bucket operations to multipart
operations section and change value from 's3:ListBucketMultipartUploads' to
's3:ListMultipartUploads' to match the action strings used in policy_engine
and s3_actions.go.

This ensures consistent action naming across all S3 constant definitions.

* refactor names

* Fix S3 action constant mismatches and MatchesAction early return bug

Fix two critical issues in policy engine:

1. S3Actions map had incorrect multipart action mappings:
   - 'ListMultipartUploads' was 's3:ListMultipartUploads' (should be 's3:ListBucketMultipartUploads')
   - 'ListParts' was 's3:ListParts' (should be 's3:ListMultipartUploadParts')
   These mismatches caused authorization checks to fail for list operations

2. CompiledStatement.MatchesAction() had early return bug:
   - Previously returned true immediately upon first direct action match
   - This prevented scanning remaining matchers for s3:PutObject permission
   - Now scans ALL matchers before returning, tracking both direct match and PutObject grant
   - Ensures multipart operations inherit s3:PutObject authorization even when
     explicitly requested action doesn't match (e.g., s3:ListMultipartUploadParts)

Changes:
- Track matchedAction flag to defer
Fix two critical issues in policy engine:

1. S3Actions map had incorrect multipart action mappings:
   - 'ListMultipartUploads' was 's3:ListMultipartUplPer
1. S3Actions map had incorrect multiparAll   - 'ListMultipartUploads(0.334s execution time)

* Refactor S3Actions map to use s3_constants

Replace hardcoded action strings in the S3Actions map with references to
canonical S3_ACTION_* constants from s3_constants/s3_action_strings.go.

Benefits:
- Single source of truth for S3 action values
- Eliminates string duplication across codebase
- Ensures consistency between policy engine and middleware
- Reduces maintenance burden when action strings need updates

All policy engine tests pass (0.334s execution time)

* Remove unused S3Actions map

The S3Actions map in types.go was never referenced anywhere in the codebase.
All action mappings are handled by GetActionMappings() in integration.go instead.
This removes 42 lines of dead code.

* Fix test: reload configuration function must also reload IAM state

TestEmbeddedIamAttachUserPolicyRefreshesIAM was failing because the test's
reloadConfigurationFunc only updated mockConfig but didn't reload the actual IAM
state. When AttachUserPolicy calls refreshIAMConfiguration(), it would use the
test's incomplete reload function instead of the real LoadS3ApiConfigurationFromCredentialManager().

Fixed by making the test's reloadConfigurationFunc also call
e.iam.LoadS3ApiConfigurationFromCredentialManager() so lookupByIdentityName()
sees the updated policy attachments.

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-02-25 12:31:04 -08:00
Chris Lu d5e71eb0d8 Revert "s3api: preserve Host header port in signature verification (#8434)"
This reverts commit 98d89ffad7.
2026-02-25 12:28:44 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
f82ee08972 build(deps): bump github.com/cloudflare/circl from 1.6.1 to 1.6.3 (#8446)
Bumps [github.com/cloudflare/circl](https://github.com/cloudflare/circl) from 1.6.1 to 1.6.3.
- [Release notes](https://github.com/cloudflare/circl/releases)
- [Commits](https://github.com/cloudflare/circl/compare/v1.6.1...v1.6.3)

---
updated-dependencies:
- dependency-name: github.com/cloudflare/circl
  dependency-version: 1.6.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-25 12:23:15 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
96078bc87e build(deps): bump github.com/cloudflare/circl from 1.6.1 to 1.6.3 in /test/kafka (#8447)
build(deps): bump github.com/cloudflare/circl in /test/kafka

Bumps [github.com/cloudflare/circl](https://github.com/cloudflare/circl) from 1.6.1 to 1.6.3.
- [Release notes](https://github.com/cloudflare/circl/releases)
- [Commits](https://github.com/cloudflare/circl/compare/v1.6.1...v1.6.3)

---
updated-dependencies:
- dependency-name: github.com/cloudflare/circl
  dependency-version: 1.6.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-25 12:23:02 -08:00
Chris LuandGitHub 8c0c7248b3 Refresh IAM config after policy attachments (#8439)
* Refresh IAM cache after policy attachments

* error handling
2026-02-25 10:30:05 -08:00
Chris Lu e3decd2e3b go fmt 2026-02-25 10:25:44 -08:00
Chris Lu 7296f51f48 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2026-02-25 10:25:31 -08:00
Chris Lu a3cb7fa8cc go fmt 2026-02-25 10:25:23 -08:00
0910252e31 feat: add statfile remote storage (#8443)
* feat: add statfile; add error for remote storage misses

* feat: statfile implementations for storage providers

* test: add unit tests for StatFile method across providers

Add comprehensive unit tests for the StatFile implementation covering:
- S3: interface compliance and error constant accessibility
- Azure: interface compliance, error constants, and field population
- GCS: interface compliance, error constants, error detection, and field population

Also fix variable shadowing issue in S3 and Azure StatFile implementations where
named return parameters were being shadowed by local variable declarations.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: address StatFile review feedback

- Use errors.New for ErrRemoteObjectNotFound sentinel
- Fix S3 HeadObject 404 detection to use awserr.Error code check
- Remove hollow field-population tests that tested nothing
- Remove redundant stdlib error detection tests
- Trim verbose doc comment on ErrRemoteObjectNotFound

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: address second round of StatFile review feedback

- Rename interface assertion tests to TestXxxRemoteStorageClientImplementsInterface
- Delegate readFileRemoteEntry to StatFile in all three providers
- Revert S3 404 detection to RequestFailure.StatusCode() check
- Fix double-slash in GCS error message format string
- Add storage type prefix to S3 error message for consistency

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: comments

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-25 10:24:06 -08:00
Chris LuandGitHub b565a0cc86 Adds volume.merge command with deduplication and disk-based backend (#8441)
* Enhance volume.merge command with deduplication and disk-based backend

* Fix copyVolume function call with correct argument order and missing bool parameter

* Revert "Fix copyVolume function call with correct argument order and missing bool parameter"

This reverts commit 7b4a190643.

* Fix critical issues: per-replica writable tracking, tail goroutine cancellation via done channel, and debug logging for allocation failures

* Optimize memory usage with watermark approach for duplicate detection

* Fix critical issues: swap copyVolume arguments, increase idle timeout, remove file double-close, use glog for logging

* Replace temporary file with in-memory buffer for needle blob serialization

* test(volume.merge): Add comprehensive unit and integration tests

Add 7 unit tests covering:
- Ordering by timestamp
- Cross-stream duplicate deduplication
- Empty stream handling
- Complex multi-stream deduplication
- Single stream passthrough
- Large needle ID support
- LastModified fallback when timestamp unavailable

Add 2 integration validation tests:
- TestMergeWorkflowValidation: Documents 9-stage merge workflow
- TestMergeEdgeCaseHandling: Validates 10 edge case handling

All tests passing (9/9)

* fix(volume.merge): Use time window for deduplication to handle clock skew

The same needle ID can have different timestamps on different servers due to
clock skew and replication lag. Needles with the same ID within a 5-second
time window are now treated as duplicates (same write with timestamp variance).

Key changes:
- Add mergeDeduplicationWindowNs constant (5 seconds)
- Replace exact timestamp matching with time window comparison
- Use windowInitialized flag to properly detect window transitions
- Add TestMergeNeedleStreamsTimeWindowDeduplication test

This ensures that replicated writes with slight timestamp differences are
properly deduplicated during merge, while separate updates to the same file
ID (outside the window) are preserved.

All tests passing (10/10)

* test: Add volume.merge integration tests with 5 comprehensive test cases

* test: integration tests for volume.merge command

* Fix integration tests: use TripleVolumeCluster for volume.merge testing

- Created new TripleVolumeCluster framework (cluster_triple.go) with 3 volume servers
- Rebuilt weed binary with volume.merge command compiled in
- Updated all 5 integration tests to use TripleVolumeCluster instead of DualVolumeCluster
- Tests now properly allocate volumes on 2 servers and let merge allocate on 3rd
- All 5 integration tests now pass:
  - TestVolumeMergeBasic
  - TestVolumeMergeReadonly
  - TestVolumeMergeRestore
  - TestVolumeMergeTailNeedles
  - TestVolumeMergeDivergentReplicas

* Refactor test framework: use parameterized server count instead of hardcoded

- Renamed TripleVolumeCluster to MultiVolumeCluster with serverCount parameter
- Replaced hardcoded volumePort0/1/2 with slices for flexible server count
- Updated StartTripleVolumeCluster as backward-compatible wrapper calling StartMultiVolumeCluster(t, profile, 3)
- Made directory creation, port allocation, and server startup loop-based
- Updated accessor methods (VolumeAdminAddress, VolumeGRPCAddress, etc.) to support any server count
- All 5 integration tests continue to pass with new parameterized cluster framework
- Enables future testing with 2, 4, 5+ volume servers by calling StartMultiVolumeCluster directly

* Consolidate cluster frameworks: StartDualVolumeCluster now uses MultiVolumeCluster

- Made DualVolumeCluster a type alias for MultiVolumeCluster
- Updated StartDualVolumeCluster to call StartMultiVolumeCluster(t, profile, 2)
- Removed duplicate code from cluster_dual.go (now just 17 lines)
- All existing tests using StartDualVolumeCluster continue to work without changes
- Backward compatible: existing code continues to use the old function signatures
- Added wrapper functions in cluster_multi.go for StartTripleVolumeCluster
- Enables unified cluster management across all test suites

* Address PR review comments: improve error handling and clean up code

- Replace parse error swallow with proper error return
- Log cleanup and restoration errors instead of silently discarding them
- Remove unused offset field from memoryBackendFile struct
- Fix WriteAt buffer truncation bug to preserve trailing bytes
- All unit tests passing (10/10)
- Code compiles successfully

* Fix PR review findings: test improvements and code quality

- Add timeout to runWeedShell to prevent hanging
- Add server 1 readonly status verification in tests
- Assert merge fails when replicas writable (not just log output)
- Replace sleep with polling for writable restoration check
- Fix WriteAt stale data snapshot bug in memoryBackendFile
- Fix startVolume error logging to show current server log
- Fix volumePubPorts double assignment in port allocation
- Rename test to reflect behavior: DoesNotDeduplicateAcrossWindows
- Fix misleading dedup window comment

Unit tests: 10/10 passing
Binary: Compiles successfully

* Fix test assumption: merge command marks volumes readonly automatically

TestVolumeMergeReadonly was expecting merge to fail on writable volumes, but the
merge command is designed to mark volumes readonly as part of its operation. Fixed
test to verify merge succeeds on writable volumes and properly restores writable
state afterward. Removed redundant Test 2 code that duplicated the new behavior.

* fmt

* Fix deduplication logic to correctly handle same-stream vs cross-stream duplicates

The dedup map previously used only NeedleId as key, causing same-stream
overwrites to be incorrectly skipped as duplicates. Changed to track which
stream first processed each needle ID in the current window:

- Cross-stream duplicates (same ID from different streams, within window) are skipped
- Same-stream duplicates (overwrites from same stream) are kept
- Map now stores: needleId -> streamIndex of first occurrence in window

Added TestMergeNeedleStreamsSameStreamDuplicates to verify same-stream
overwrites are preserved while cross-stream duplicates are skipped.

All unit tests passing (11/11)
Binary compiles successfully
2026-02-25 10:12:09 -08:00
Chris LuandGitHub da4edb5fe6 Fix live volume move tail timestamp (#8440)
* Improve move tail timestamp

* Add move tail timestamp integration test

* Simulate traffic during move
2026-02-24 20:07:26 -08:00
Chris LuandGitHub 27e763222a Fix inline user policy retrieval (#8437)
* Fix IAM inline user policy retrieval

* fmt

* Persist inline user policies to avoid loss on server restart

- Use s3ApiConfig.PutPolicies/GetPolicies for persistent storage instead of non-persistent global map
- Remove unused global policyDocuments map
- Update PutUserPolicy to store policies in persistent storage
- Update GetUserPolicy to read from persistent storage
- Update DeleteUserPolicy to clean up persistent storage
- Add mock IamS3ApiConfig for testing
- Improve test to verify policy statements are not merged or lost

* Fix inline policy key collision and action aggregation

* Improve error handling and optimize inline policy management

- GetUserPolicy: Propagate GetPolicies errors instead of silently falling through
- DeleteUserPolicy: Return error immediately on GetPolicies failure
- computeAggregatedActionsForUser: Add optional Policies parameter for I/O optimization
- PutUserPolicy: Reuse fetched policies to avoid redundant GetPolicies call
- Improve logging with clearer messages about best-effort aggregation
- Update test to use exact action string matching instead of substring checks

All 15 tests pass with no regressions.

* Add per-user policy index for O(1) lookup performance

- Extend Policies struct with InlinePolicies map[userName]map[policyName]
- Add getOrCreateUserPolicies() helper for safe user map management
- Update computeAggregatedActionsForUser to use direct user map access
- Update PutUserPolicy, GetUserPolicy, DeleteUserPolicy for new structure
- Performance: O(1) user lookups instead of O(all_policies) iteration
- Eliminates string prefix matching loop
- All tests pass; backward compatible with managed policies

* Fix DeleteUserPolicy to validate user existence before storage modification

Refactor DeleteUserPolicy handler to check user existence early:
- First iterate s3cfg.Identities to verify user exists
- Return NoSuchEntity error immediately if user not found
- Only then proceed with GetPolicies and policy deletion
- Capture reference to found identity for direct update

This ensures consistency: if user doesn't exist, storage is not modified.
Previously the code would delete from storage first and check identity
afterwards, potentially leaving orphaned policies.

Benefits:
- Fail-fast validation before storage operations
- No orphaned policies in storage if validation fails
- Atomic from logical perspective
- Direct identity reference eliminates redundant loop
- All error paths preserved and tested

All 15 tests pass; no functional changes to behavior.

* Fix GetUserPolicy to return NoSuchEntity when inline policy not found

When InlinePolicies[userName] exists but does not contain policyName,
the handler now immediately returns NoSuchEntity error instead of
falling through to the reconstruction logic.

Changes:
- Add else clause after userPolicies[policyName] lookup
- Return IamError(NoSuchEntityException, "policy not found") immediately
- Prevents incorrect fallback to reconstructing ident.Actions
- Ensures explicit error when policy explicitly doesn't exist

This improves error semantics:
- Policy exists in stored inline policies → return error (not reconstruct)
- Policy doesn't exist in stored inline policies → try reconstruction (backward compat)
- Storage error → return service failure error

All 15 tests pass; no behavioral changes to existing error or success paths.
2026-02-24 18:01:17 -08:00
427c975ff3 fix(plugin/worker): make VacuumHandler report MaxExecutionConcurrency from worker startup flag (#8435)
* fix(plugin/worker): make VacuumHandler report MaxExecutionConcurrency from worker startup flag

Previously, MaxExecutionConcurrency was hardcoded to 2 in VacuumHandler.Capability().
The scheduler's schedulerWorkerExecutionLimit() takes the minimum of the UI-configured
PerWorkerExecutionConcurrency and the worker-reported capability limit, so the hardcoded
value silently capped each worker to 2 concurrent vacuum executions regardless of the
--max-execute flag passed at worker startup.

Pass maxExecutionConcurrency into NewVacuumHandler() and wire it through
buildPluginWorkerHandler/buildPluginWorkerHandlers so the capability reflects the actual
worker configuration. The default falls back to 2 when the value is unset or zero.

* Update weed/command/worker_runtime.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Anton Ustyugov <anton@devops>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-24 15:13:00 -08:00
Chris Lu ce4940b441 fix filer link on dashboard 2026-02-24 14:21:27 -08:00
b4c7d42a06 fix(admin): release mutex before disk I/O in maintenance queue; remove per-request LoadAllTaskStates (#8433)
* fix(admin): release mutex before disk I/O in maintenance queue

saveTaskState performs synchronous BoltDB writes. Calling it while
holding mq.mutex.Lock() in AddTask, GetNextTask, and CompleteTask
blocks all readers (GetTasks via RLock) for the full disk write
duration on every task state change.

During a maintenance scan AddTasksFromResults calls AddTask for every
volume — potentially hundreds of times — meaning the write lock is
held almost continuously. The HTTP handler for /maintenance calls
GetTasks which blocks on RLock, exceeding the 30s timeout and
returning 408 to the browser.

Fix: update in-memory state (mq.tasks, mq.pendingTasks) under the
lock as before, then unlock before calling saveTaskState. In-memory
state is the authoritative source; persistence is crash-recovery only
and does not require lock protection during the write.

* fix(admin): add mutex to ConfigPersistence to synchronize tasks/ filesystem ops

saveTaskState is now called outside mq.mutex, meaning SaveTaskState,
LoadAllTaskStates, DeleteTaskState, and CleanupCompletedTasks can be
invoked concurrently from multiple goroutines. ConfigPersistence had no
internal synchronization, creating races on the tasks/ directory:

- concurrent os.WriteFile + os.ReadFile on the same .pb file could
  yield a partial read and unmarshal error
- LoadAllTaskStates (ReadDir + per-file ReadFile) could see a
  directory entry for a file being written or deleted concurrently
- CleanupCompletedTasks (LoadAllTaskStates + DeleteTaskState) could
  race with SaveTaskState on the same file

Fix: add tasksMu sync.Mutex to ConfigPersistence, acquired at the top
of SaveTaskState, LoadTaskState, LoadAllTaskStates, DeleteTaskState,
and CleanupCompletedTasks. Extract private Locked helpers so that
CleanupCompletedTasks (which holds tasksMu) can call them internally
without deadlocking.

---------

Co-authored-by: Anton Ustyugov <anton@devops>
2026-02-24 13:41:41 -08:00
Chris Lu cba69f4593 Update layout_templ.go 2026-02-24 13:22:12 -08:00
Chris Lu 91f59e73e5 close ports 2026-02-24 13:20:21 -08:00
Chris LuandGitHub 98d89ffad7 s3api: preserve Host header port in signature verification (#8434)
Avoid stripping default ports (80/443) from the Host header in extractHostHeader.
This fixes SignatureDoesNotMatch errors when SeaweedFS is accessed via a proxy
(like Kong Ingress) that explicitly includes the port in the Host header or
X-Forwarded-Host, which S3 clients sign.

Also cleaned up unused variables and logic after refactoring.
2026-02-24 13:09:40 -08:00
Peter DoddandGitHub f4af1cc0ba feat(helm): annotations for service account (#8429) 2026-02-24 07:35:13 -08:00
Xiao WeiandGitHub 9fa95dd2c6 fix: unload leveldb not take effect (#8431) 2026-02-24 07:32:13 -08:00
Plamen NikolovandGitHub ff84ef880d fix(s3api): make ListObjectsV1 namespaced and prevent marker-echo pagination loops (#8409)
* fix(s3api): make ListObjectsV1 namespaced and stop marker-echo pagination loops

* test(s3api): harden marker-echo coverage and align V1 encoding tag

* test(s3api): cover encoded marker matching and trim redundant setup

* refactor(s3api): tighten V1 list helper visibility and test mock docs
2026-02-23 23:45:08 -08:00
Chris LuandGitHub 2d65d7f499 Embed role policies in AssumeRole STS tokens (#8421)
* Embed role policies in AssumeRole STS tokens

* Log STS policy lookup failures

* Use IAMManager provider

* Guard policy embedding role lookup
2026-02-23 22:59:53 -08:00
Chris LuandGitHub 3f58e3bf8f Use master shard sizes for EC volumes (#8423)
* Use master shard sizes for EC volumes

* Remove EC volume shard size fallback

* Remove unused EC dash imports
2026-02-23 21:59:09 -08:00
Justin CichraandGitHub 016391530b fix: plural clientType on ListExistingPeerUpdates (#8422) 2026-02-23 20:19:12 -08:00
Chris LuandGitHub 8d59ef41d5 Admin UI: replace gin with mux (#8420)
* Replace admin gin router with mux

* Update layout_templ.go

* Harden admin handlers

* Add login CSRF handling

* Fix filer copy naming conflict

* address comments

* address comments
2026-02-23 19:11:17 -08:00
Chris LuandGitHub e596542295 Move SQL engine and PostgreSQL server to their own binaries (#8417)
* Drop SQL engine and PostgreSQL server

* Split SQL tooling into weed-db and weed-sql

* move

* fix building
2026-02-23 16:27:08 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
61db4d0966 build(deps): bump github.com/rclone/rclone from 1.72.1 to 1.73.1 (#8416)
Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.72.1 to 1.73.1.
- [Release notes](https://github.com/rclone/rclone/releases)
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md)
- [Commits](https://github.com/rclone/rclone/compare/v1.72.1...v1.73.1)

---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
  dependency-version: 1.73.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-23 13:42:12 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
74de694447 build(deps): bump modernc.org/sqlite from 1.44.3 to 1.46.1 (#8415)
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.44.3 to 1.46.1.
- [Changelog](https://gitlab.com/cznic/sqlite/blob/master/CHANGELOG.md)
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.44.3...v1.46.1)

---
updated-dependencies:
- dependency-name: modernc.org/sqlite
  dependency-version: 1.46.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-23 13:42:05 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
c5e8e4f049 build(deps): bump actions/dependency-review-action from 4.8.2 to 4.8.3 (#8414)
Bumps [actions/dependency-review-action](https://github.com/actions/dependency-review-action) from 4.8.2 to 4.8.3.
- [Release notes](https://github.com/actions/dependency-review-action/releases)
- [Commits](https://github.com/actions/dependency-review-action/compare/3c4e3dcb1aa7874d2c16be7d79418e9b7efd6261...05fe4576374b728f0c523d6a13d64c25081e0803)

---
updated-dependencies:
- dependency-name: actions/dependency-review-action
  dependency-version: 4.8.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-23 13:41:57 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
c96b0913ed build(deps): bump golang.org/x/image from 0.35.0 to 0.36.0 (#8413)
Bumps [golang.org/x/image](https://github.com/golang/image) from 0.35.0 to 0.36.0.
- [Commits](https://github.com/golang/image/compare/v0.35.0...v0.36.0)

---
updated-dependencies:
- dependency-name: golang.org/x/image
  dependency-version: 0.36.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-23 13:41:49 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
b033823611 build(deps): bump helm/kind-action from 1.13.0 to 1.14.0 (#8412)
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.13.0 to 1.14.0.
- [Release notes](https://github.com/helm/kind-action/releases)
- [Commits](https://github.com/helm/kind-action/compare/v1.13.0...v1.14.0)

---
updated-dependencies:
- dependency-name: helm/kind-action
  dependency-version: 1.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-23 13:41:40 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
3044b51e7c build(deps): bump github.com/pierrec/lz4/v4 from 4.1.22 to 4.1.25 (#8411)
Bumps [github.com/pierrec/lz4/v4](https://github.com/pierrec/lz4) from 4.1.22 to 4.1.25.
- [Release notes](https://github.com/pierrec/lz4/releases)
- [Commits](https://github.com/pierrec/lz4/compare/v4.1.22...v4.1.25)

---
updated-dependencies:
- dependency-name: github.com/pierrec/lz4/v4
  dependency-version: 4.1.25
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-23 13:41:25 -08:00
Chris Lu 8e8edd7706 not empty only if there are actual files in the bucket 2026-02-23 00:12:04 -08:00
Chris LuandGitHub 57ab99d13e fix: generate topology uuid uniformly in single-master mode (#8405)
* fix: ensure topology uuid is generated in single master setups

* ensureTopologyId adds a Hashicorp-aware implementation

* simplify
2026-02-22 23:45:48 -08:00
Chris LuandGitHub 998c8d2702 Worker maintenance tasks now use non-default grpcPort if configured (#8407)
Fixes #8401

When creating balance/vacuum tasks, the worker maintenance scheduler was
accidentally discarding the custom grpcPort defined on the DataNodeInfo
by using just its HTTP Address string, which defaults to +10000
during grpc dialing.

By using pb.NewServerAddressFromDataNode, the grpcPort suffix is correctly
encoded in the server address string, preventing connection refused errors
for users running volume servers with custom gRPC ports.
2026-02-22 22:40:14 -08:00
Chris LuandGitHub cd6832249b Fix volume.fsck crashing on EC volumes and add multi-volume vacuum support (#8406)
* helm: refine openshift-values.yaml to remove hardcoded UIDs

Remove hardcoded runAsUser, runAsGroup, and fsGroup from the
openshift-values.yaml example. This allows OpenShift's admission
controller to automatically assign a valid UID from the namespace's
allocated range, avoiding "forbidden" errors when UID 1000 is
outside the permissible range.

Updates #8381, #8390.

* helm: fix volume.logs and add consistent security context comments

* Update README.md

* fix volume.fsck crashing on EC volumes and add multi-volume vacuum support

* address comments
2026-02-22 22:07:15 -08:00
Chris Lu b5f3094619 fix format of internal node URLs in master UI templates 2026-02-22 13:47:29 -08:00
Chris Lu 07f284c391 fix links 2026-02-22 13:40:50 -08:00
Chris Lu 7b08cf74ed consistent template generation 2026-02-22 13:34:06 -08:00
Sheya BernsteinandGitHub d8b8f0dffd fix(helm): add missing app.kubernetes.io/instance label to volume service (#8403) 2026-02-22 07:20:38 -08:00
Chris LuandGitHub 8e25c55bfb S3: Truncate timestamps to milliseconds for CopyObjectResult and CopyPartResult (#8398)
* S3: Truncate timestamps to milliseconds for CopyObjectResult and CopyPartResult

Fixes #8394

* S3: Address nitpick comments in copy handlers

- Synchronize Mtime and LastModified by capturing time once\n- Optimize copyChunksForRange loop\n- Use built-in min/max\n- Remove dead previewLen code
2026-02-20 21:01:31 -08:00
Chris Lu e4b70c2521 go fix 2026-02-20 18:42:00 -08:00
Chris Lu f7c27cc81f go fmt 2026-02-20 18:40:47 -08:00
Chris Lu 66680c58b7 consistent time 2026-02-20 18:40:27 -08:00
Chris LuandGitHub 2a1ae896e4 helm: refine openshift-values.yaml for assigned UID ranges (#8396)
* helm: refine openshift-values.yaml to remove hardcoded UIDs

Remove hardcoded runAsUser, runAsGroup, and fsGroup from the
openshift-values.yaml example. This allows OpenShift's admission
controller to automatically assign a valid UID from the namespace's
allocated range, avoiding "forbidden" errors when UID 1000 is
outside the permissible range.

Updates #8381, #8390.

* helm: fix volume.logs and add consistent security context comments

* Update README.md
2026-02-20 12:05:57 -08:00
Chris LuandGitHub bd0b1fe9d5 S3 IAM: Added ListPolicyVersions and GetPolicyVersion support (#8395)
* test(s3/iam): add managed policy CRUD lifecycle integration coverage

* s3/iam: add ListPolicyVersions and GetPolicyVersion support

* test(s3/iam): cover ListPolicyVersions and GetPolicyVersion
2026-02-20 11:04:18 -08:00
Richard Chen ZhengandGitHub 964a8f5fde Allow user to define access and secret key via values (#8389)
* Allow user to define admin access and secret key via values

* Add comments to values.yaml

* Add support for read for consistency

* Simplify templating

* Add checksum to s3 config

* Update comments

* Revert "Add checksum to s3 config"

This reverts commit d21a7038a8.
2026-02-20 00:37:54 -08:00
Chris LuandGitHub 40cc0e04a6 docker: fix entrypoint chown guard; helm: add openshift-values.yaml (#8390)
* Enforce IAM for s3tables bucket creation

* Prefer IAM path when policies exist

* Ensure IAM enforcement honors default allow

* address comments

* Reused the precomputed principal when setting tableBucketMetadata.OwnerAccountID, avoiding the redundant getAccountID call.

* get identity

* fix

* dedup

* fix

* comments

* fix tests

* update iam config

* go fmt

* fix ports

* fix flags

* mini clean shutdown

* Revert "update iam config"

This reverts commit ca48fdbb0a.

Revert "mini clean shutdown"

This reverts commit 9e17f6baff.

Revert "fix flags"

This reverts commit e9e7b29d2f.

Revert "go fmt"

This reverts commit bd3241960b.

* test/s3tables: share single weed mini per test package via TestMain

Previously each top-level test function in the catalog and s3tables
package started and stopped its own weed mini instance. This caused
failures when a prior instance wasn't cleanly stopped before the next
one started (port conflicts, leaked global state).

Changes:
- catalog/iceberg_catalog_test.go: introduce TestMain that starts one
  shared TestEnvironment (external weed binary) before all tests and
  tears it down after. All individual test functions now use sharedEnv.
  Added randomSuffix() for unique resource names across tests.
- catalog/pyiceberg_test.go: updated to use sharedEnv instead of
  per-test environments.
- catalog/pyiceberg_test_helpers.go -> pyiceberg_test_helpers_test.go:
  renamed to a _test.go file so it can access TestEnvironment which is
  defined in a test file.
- table-buckets/setup.go: add package-level sharedCluster variable.
- table-buckets/s3tables_integration_test.go: introduce TestMain that
  starts one shared TestCluster before all tests. TestS3TablesIntegration
  now uses sharedCluster. Extract startMiniClusterInDir (no *testing.T)
  for TestMain use. TestS3TablesCreateBucketIAMPolicy keeps its own
  cluster (different IAM config). Remove miniClusterMutex (no longer
  needed). Fix Stop() to not panic when t is nil."

* delete

* parse

* default allow should work with anonymous

* fix port

* iceberg route

The failures are from Iceberg REST using the default bucket warehouse when no prefix is provided. Your tests create random buckets, so /v1/namespaces was looking in warehouse and failing. I updated the tests to use the prefixed Iceberg routes (/v1/{bucket}/...) via a small helper.

* test(s3tables): fix port conflicts and IAM ARN matching in integration tests

- Pass -master.dir explicitly to prevent filer store directory collision
  between shared cluster and per-test clusters running in the same process
- Pass -volume.port.public and -volume.publicUrl to prevent the global
  publicPort flag (mutated from 0 → concrete port by first cluster) from
  being reused by a second cluster, causing 'address already in use'
- Remove the flag-reset loop in Stop() that reset global flag values while
  other goroutines were reading them (race → panic)
- Fix IAM policy Resource ARN in TestS3TablesCreateBucketIAMPolicy to use
  wildcards (arn:aws:s3tables:*:*:bucket/<name>) because the handler
  generates ARNs with its own DefaultRegion (us-east-1) and principal name
  ('admin'), not the test constants testRegion/testAccountID

* docker: fix entrypoint chown guard; helm: add openshift-values.yaml

Fix a regression in entrypoint.sh where the DATA_UID/DATA_GID
ownership comparison was dropped, causing chown -R /data to run
unconditionally on every container start even when ownership was
already correct. Restore the guard so the recursive chown is
skipped when the seaweed user already owns /data — making startup
faster on subsequent runs and a no-op on OpenShift/PVC deployments
where fsGroup has already set correct ownership.

Add k8s/charts/seaweedfs/openshift-values.yaml: an example Helm
overrides file for deploying SeaweedFS on OpenShift (or any cluster
enforcing the Kubernetes restricted Pod Security Standard). Replaces
hostPath volumes with PVCs, sets runAsUser/fsGroup to 1000
(the seaweed user baked into the image), drops all capabilities,
disables privilege escalation, and enables RuntimeDefault seccomp —
satisfying OpenShift's default restricted SCC without needing a
custom SCC or root access.

Fixes #8381"
2026-02-20 00:35:42 -08:00
Michał SzynkiewiczandGitHub 2f837c4780 Fix error on deleting non-empty bucket (#8376)
* Move check for non-empty bucket deletion out of `WithFilerClient` call

* Added proper checking if a bucket has "user" objects
2026-02-19 22:56:50 -08:00
Chris LuandGitHub 36c469e34e Enforce IAM for S3 Tables bucket creation (#8388)
* Enforce IAM for s3tables bucket creation

* Prefer IAM path when policies exist

* Ensure IAM enforcement honors default allow

* address comments

* Reused the precomputed principal when setting tableBucketMetadata.OwnerAccountID, avoiding the redundant getAccountID call.

* get identity

* fix

* dedup

* fix

* comments

* fix tests

* update iam config

* go fmt

* fix ports

* fix flags

* mini clean shutdown

* Revert "update iam config"

This reverts commit ca48fdbb0a.

Revert "mini clean shutdown"

This reverts commit 9e17f6baff.

Revert "fix flags"

This reverts commit e9e7b29d2f.

Revert "go fmt"

This reverts commit bd3241960b.

* test/s3tables: share single weed mini per test package via TestMain

Previously each top-level test function in the catalog and s3tables
package started and stopped its own weed mini instance. This caused
failures when a prior instance wasn't cleanly stopped before the next
one started (port conflicts, leaked global state).

Changes:
- catalog/iceberg_catalog_test.go: introduce TestMain that starts one
  shared TestEnvironment (external weed binary) before all tests and
  tears it down after. All individual test functions now use sharedEnv.
  Added randomSuffix() for unique resource names across tests.
- catalog/pyiceberg_test.go: updated to use sharedEnv instead of
  per-test environments.
- catalog/pyiceberg_test_helpers.go -> pyiceberg_test_helpers_test.go:
  renamed to a _test.go file so it can access TestEnvironment which is
  defined in a test file.
- table-buckets/setup.go: add package-level sharedCluster variable.
- table-buckets/s3tables_integration_test.go: introduce TestMain that
  starts one shared TestCluster before all tests. TestS3TablesIntegration
  now uses sharedCluster. Extract startMiniClusterInDir (no *testing.T)
  for TestMain use. TestS3TablesCreateBucketIAMPolicy keeps its own
  cluster (different IAM config). Remove miniClusterMutex (no longer
  needed). Fix Stop() to not panic when t is nil."

* delete

* parse

* default allow should work with anonymous

* fix port

* iceberg route

The failures are from Iceberg REST using the default bucket warehouse when no prefix is provided. Your tests create random buckets, so /v1/namespaces was looking in warehouse and failing. I updated the tests to use the prefixed Iceberg routes (/v1/{bucket}/...) via a small helper.

* test(s3tables): fix port conflicts and IAM ARN matching in integration tests

- Pass -master.dir explicitly to prevent filer store directory collision
  between shared cluster and per-test clusters running in the same process
- Pass -volume.port.public and -volume.publicUrl to prevent the global
  publicPort flag (mutated from 0 → concrete port by first cluster) from
  being reused by a second cluster, causing 'address already in use'
- Remove the flag-reset loop in Stop() that reset global flag values while
  other goroutines were reading them (race → panic)
- Fix IAM policy Resource ARN in TestS3TablesCreateBucketIAMPolicy to use
  wildcards (arn:aws:s3tables:*:*:bucket/<name>) because the handler
  generates ARNs with its own DefaultRegion (us-east-1) and principal name
  ('admin'), not the test constants testRegion/testAccountID
2026-02-19 22:52:05 -08:00
Chris LuandGitHub a2005cb2a6 fix: resolve gRPC DNS resolution issues in Kubernetes #8384 (#8387)
* fix: resolve gRPC DNS resolution issues in Kubernetes #8384

- Replace direct `grpc.NewClient` calls with `pb.GrpcDial` for consistent connection establishment
- Fix async DNS resolution behavior in K8s with `ndots:5`
- Ensure high-level components use established helper for reliable networking

* refactor: refine gRPC DNS fix and add documentation

- Use instance's grpcDialOption in BrokerClient.ConfigureTopic
- Add detailed comments to GrpcDial explaining Kubernetes DNS resolution rationale

* fix: ensure proper context propagation in broker_client gRPC calls

- Pass the provided `ctx` to `pb.GrpcDial` in `ConfigureTopic` and `GetUnflushedMessages`
- Ensures that timeouts and cancellations are correctly honored during connection establishment

* docs: refine gRPC resolver documentation and cleanup dead code

- Enhanced documentation for `GrpcDial` with explicit warnings about global state mutation when using `resolver.SetDefaultScheme("passthrough")`.
- Recommended `passthrough:///` prefix as the primary migration path for `grpc.NewClient`.
- Removed dead commented-out code for `grpc.WithBlock()` and `grpc.WithTimeout()`.
2026-02-19 15:46:02 -08:00
Chris LuandGitHub e9c45144cf Implement managed policy storage (#8385)
* Persist managed IAM policies

* Add IAM list/get policy integration test

* Faster marker lookup and cleanup

* Handle delete conflict and improve listing

* Add delete-in-use policy integration test

* Stabilize policy ID and guard path prefix

* Tighten CreatePolicy guard and reload

* Add ListPolicyNames to credential store
2026-02-19 14:21:19 -08:00
Chris LuandGitHub 5ecee9e64d s3: fix signature mismatch with non-standard ports and capitalized host (#8386)
* s3: fix signature mismatch with non-standard ports and capitalized host

- ensure host header extraction is case-insensitive in SignedHeaders
- prioritize non-standard ports in X-Forwarded-Host over default ports in X-Forwarded-Port
- add regression tests for both scenarios

fixes https://github.com/seaweedfs/seaweedfs/issues/8382

* simplify
2026-02-19 14:17:31 -08:00
Konstantin LebedevandGitHub 01b3125815 [shell]: volume balance capacity by min volume density (#8026)
volume balance by min volume density and active volumes
2026-02-19 13:30:59 -08:00
Chris LuandGitHub 7b8df39cf7 s3api: add AttachUserPolicy/DetachUserPolicy/ListAttachedUserPolicies (#8379)
* iam: add XML responses for managed user policy APIs

* s3api: implement attach/detach/list attached user policies

* s3api: add embedded IAM tests for managed user policies

* iam: update CredentialStore interface and Manager for managed policies

Updated the `CredentialStore` interface to include `AttachUserPolicy`,
`DetachUserPolicy`, and `ListAttachedUserPolicies` methods.
The `CredentialManager` was updated to delegate these calls to the store.
Added common error variables for policy management.

* iam: implement managed policy methods in MemoryStore

Implemented `AttachUserPolicy`, `DetachUserPolicy`, and
`ListAttachedUserPolicies` in the MemoryStore.
Also ensured deep copying of identities includes PolicyNames.

* iam: implement managed policy methods in PostgresStore

Modified Postgres schema to include `policy_names` JSONB column in `users`.
Implemented `AttachUserPolicy`, `DetachUserPolicy`, and `ListAttachedUserPolicies`.
Updated user CRUD operations to handle policy names persistence.

* iam: implement managed policy methods in remaining stores

Implemented user policy management in:
- `FilerEtcStore` (partial implementation)
- `IamGrpcStore` (delegated via GetUser/UpdateUser)
- `PropagatingCredentialStore` (to broadcast updates)
Ensures cluster-wide consistency for policy attachments.

* s3api: refactor EmbeddedIamApi to use managed policy APIs

- Refactored `AttachUserPolicy`, `DetachUserPolicy`, and `ListAttachedUserPolicies`
  to use `e.credentialManager` directly.
- Fixed a critical error suppression bug in `ExecuteAction` that always
  returned success even on failure.
- Implemented robust error matching using string comparison fallbacks.
- Improved consistency by reloading configuration after policy changes.

* s3api: update and refine IAM integration tests

- Updated tests to use a real `MemoryStore`-backed `CredentialManager`.
- Refined test configuration synchronization using `sync.Once` and
  manual deep-copying to prevent state corruption.
- Improved `extractEmbeddedIamErrorCodeAndMessage` to handle more XML
  formats robustly.
- Adjusted test expectations to match current AWS IAM behavior.

* fix compilation

* visibility

* ensure 10 policies

* reload

* add integration tests

* Guard raft command registration

* Allow IAM actions in policy tests

* Validate gRPC policy attachments

* Revert Validate gRPC policy attachments

* Tighten gRPC policy attach/detach

* Improve IAM managed policy handling

* Improve managed policy filters
2026-02-19 12:26:27 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
6787dccace build(deps): bump filippo.io/edwards25519 from 1.1.0 to 1.1.1 (#8383)
Bumps [filippo.io/edwards25519](https://github.com/FiloSottile/edwards25519) from 1.1.0 to 1.1.1.
- [Commits](https://github.com/FiloSottile/edwards25519/compare/v1.1.0...v1.1.1)

---
updated-dependencies:
- dependency-name: filippo.io/edwards25519
  dependency-version: 1.1.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-19 10:51:27 -08:00
Chris LuandGitHub d1fecdface Fix IAM defaults and S3Tables IAM regression (#8374)
* Fix IAM defaults and s3tables identities

* Refine S3Tables identity tests

* Clarify identity tests
2026-02-18 18:20:03 -08:00
Chris LuandGitHub 38e14a867b fix: cancel volume server requests on client disconnect during S3 downloads (#8373)
* fix: cancel volume server requests on client disconnect during S3 downloads

- Use http.NewRequestWithContext in ReadUrlAsStream so in-flight volume
  server requests are properly aborted when the client disconnects and
  the request context is canceled
- Distinguish context-canceled errors (client disconnect, expected) from
  real server errors in streamFromVolumeServers; log at V(3) instead of
  ERROR to reduce noise from client-side disconnects (e.g. Nginx upstream
  timeout, browser cancel, curl --max-time)

Fixes: streamFromVolumeServers: streamFn failed...context canceled"

* fixup: separate Canceled/DeadlineExceeded log severity in streamFromVolumeServers

- context.Canceled → V(3) Infof "client disconnected" (expected, no noise)
- context.DeadlineExceeded → Warningf "server-side deadline exceeded" (unexpected, needs attention)
- all other errors → Errorf (unchanged)"
2026-02-18 17:14:54 -08:00
Chris Lu eda4a000cc Revert "Fix IAM defaults and s3tables identities"
This reverts commit bf71fe0039.
2026-02-18 16:23:13 -08:00
Chris Lu bf71fe0039 Fix IAM defaults and s3tables identities 2026-02-18 16:21:48 -08:00
53048ffffb Add md5 checksum validation support on PutObject and UploadPart (#8367)
* Add md5 checksum validation support on PutObject and UploadPart

Per the S3 specification, when a client sends a Content-MD5 header, the server must compare it against the MD5 of the received body and return BadDigest (HTTP 400) if they don't match.

SeaweedFS was silently accepting objects with incorrect Content-MD5 headers, which breaks data integrity verification for clients that rely on this feature (e.g. boto3). The error infrastructure (ErrBadDigest, ErrMsgBadDigest) already existed from PR #7306 but was never wired to an actual check.

This commit adds MD5 verification in putToFiler after the body is streamed and the MD5 is computed, and adds Content-MD5 header validation to PutObjectPartHandler (matching PutObjectHandler). Orphaned chunks are cleaned up on mismatch.

Refs: https://github.com/seaweedfs/seaweedfs/discussions/3908

* handle SSE, add uploadpart test

* s3 integration test: fix typo and add multipart upload checksum test

* s3api: move validateContentMd5 after GetBucketAndObject in PutObjectPartHandler

* s3api: move validateContentMd5 after GetBucketAndObject in PutObjectHandler

* s3api: fix MD5 validation for SSE uploads and logging in putToFiler

* add SSE test with checksum validation - mostly ai-generated

* Update s3_integration_test.go

* Address S3 integration test feedback: fix typos, rename variables, add verification steps, and clean up comments.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-02-18 15:40:08 -08:00
6a3a97333f Add support for TLS in gRPC communication between worker and volume server (#8370)
* Add support for TLS in gRPC communication between worker and volume server

* address comments

* worker: capture shared grpc.DialOption in BalanceTask registration closure

* worker: capture shared grpc.DialOption in ErasureCodingTask registration closure

* worker: capture shared grpc.DialOption in VacuumTask registration closure

* worker: use grpc.worker security configuration section for tasks

* plugin/worker: fix compilation errors by passing grpc.DialOption to task constructors

* plugin/worker: prevent double-counting in EC skip counters

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-02-18 15:39:53 -08:00
Chris LuGitHubCopilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
8ec9ff4a12 Refactor plugin system and migrate worker runtime (#8369)
* admin: add plugin runtime UI page and route wiring

* pb: add plugin gRPC contract and generated bindings

* admin/plugin: implement worker registry, runtime, monitoring, and config store

* admin/dash: wire plugin runtime and expose plugin workflow APIs

* command: add flags to enable plugin runtime

* admin: rename remaining plugin v2 wording to plugin

* admin/plugin: add detectable job type registry helper

* admin/plugin: add scheduled detection and dispatch orchestration

* admin/plugin: prefetch job type descriptors when workers connect

* admin/plugin: add known job type discovery API and UI

* admin/plugin: refresh design doc to match current implementation

* admin/plugin: enforce per-worker scheduler concurrency limits

* admin/plugin: use descriptor runtime defaults for scheduler policy

* admin/ui: auto-load first known plugin job type on page open

* admin/plugin: bootstrap persisted config from descriptor defaults

* admin/plugin: dedupe scheduled proposals by dedupe key

* admin/ui: add job type and state filters for plugin monitoring

* admin/ui: add per-job-type plugin activity summary

* admin/plugin: split descriptor read API from schema refresh

* admin/ui: keep plugin summary metrics global while tables are filtered

* admin/plugin: retry executor reservation before timing out

* admin/plugin: expose scheduler states for monitoring

* admin/ui: show per-job-type scheduler states in plugin monitor

* pb/plugin: rename protobuf package to plugin

* admin/plugin: rename pluginRuntime wiring to plugin

* admin/plugin: remove runtime naming from plugin APIs and UI

* admin/plugin: rename runtime files to plugin naming

* admin/plugin: persist jobs and activities for monitor recovery

* admin/plugin: lease one detector worker per job type

* admin/ui: show worker load from plugin heartbeats

* admin/plugin: skip stale workers for detector and executor picks

* plugin/worker: add plugin worker command and stream runtime scaffold

* plugin/worker: implement vacuum detect and execute handlers

* admin/plugin: document external vacuum plugin worker starter

* command: update plugin.worker help to reflect implemented flow

* command/admin: drop legacy Plugin V2 label

* plugin/worker: validate vacuum job type and respect min interval

* plugin/worker: test no-op detect when min interval not elapsed

* command/admin: document plugin.worker external process

* plugin/worker: advertise configured concurrency in hello

* command/plugin.worker: add jobType handler selection

* command/plugin.worker: test handler selection by job type

* command/plugin.worker: persist worker id in workingDir

* admin/plugin: document plugin.worker jobType and workingDir flags

* plugin/worker: support cancel request for in-flight work

* plugin/worker: test cancel request acknowledgements

* command/plugin.worker: document workingDir and jobType behavior

* plugin/worker: emit executor activity events for monitor

* plugin/worker: test executor activity builder

* admin/plugin: send last successful run in detection request

* admin/plugin: send cancel request when detect or execute context ends

* admin/plugin: document worker cancel request responsibility

* admin/handlers: expose plugin scheduler states API in no-auth mode

* admin/handlers: test plugin scheduler states route registration

* admin/plugin: keep worker id on worker-generated activity records

* admin/plugin: test worker id propagation in monitor activities

* admin/dash: always initialize plugin service

* command/admin: remove plugin enable flags and default to enabled

* admin/dash: drop pluginEnabled constructor parameter

* admin/plugin UI: stop checking plugin enabled state

* admin/plugin: remove docs for plugin enable flags

* admin/dash: remove unused plugin enabled check method

* admin/dash: fallback to in-memory plugin init when dataDir fails

* admin/plugin API: expose worker gRPC port in status

* command/plugin.worker: resolve admin gRPC port via plugin status

* split plugin UI into overview/configuration/monitoring pages

* Update layout_templ.go

* add volume_balance plugin worker handler

* wire plugin.worker CLI for volume_balance job type

* add erasure_coding plugin worker handler

* wire plugin.worker CLI for erasure_coding job type

* support multi-job handlers in plugin worker runtime

* allow plugin.worker jobType as comma-separated list

* admin/plugin UI: rename to Workers and simplify config view

* plugin worker: queue detection requests instead of capacity reject

* Update plugin_worker.go

* plugin volume_balance: remove force_move/timeout from worker config UI

* plugin erasure_coding: enforce local working dir and cleanup

* admin/plugin UI: rename admin settings to job scheduling

* admin/plugin UI: persist and robustly render detection results

* admin/plugin: record and return detection trace metadata

* admin/plugin UI: show detection process and decision trace

* plugin: surface detector decision trace as activities

* mini: start a plugin worker by default

* admin/plugin UI: split monitoring into detection and execution tabs

* plugin worker: emit detection decision trace for EC and balance

* admin workers UI: split monitoring into detection and execution pages

* plugin scheduler: skip proposals for active assigned/running jobs

* admin workers UI: add job queue tab

* plugin worker: add dummy stress detector and executor job type

* admin workers UI: reorder tabs to detection queue execution

* admin workers UI: regenerate plugin template

* plugin defaults: include dummy stress and add stress tests

* plugin dummy stress: rotate detection selections across runs

* plugin scheduler: remove cross-run proposal dedupe

* plugin queue: track pending scheduled jobs

* plugin scheduler: wait for executor capacity before dispatch

* plugin scheduler: skip detection when waiting backlog is high

* plugin: add disk-backed job detail API and persistence

* admin ui: show plugin job detail modal from job id links

* plugin: generate unique job ids instead of reusing proposal ids

* plugin worker: emit heartbeats on work state changes

* plugin registry: round-robin tied executor and detector picks

* add temporary EC overnight stress runner

* plugin job details: persist and render EC execution plans

* ec volume details: color data and parity shard badges

* shard labels: keep parity ids numeric and color-only distinction

* admin: remove legacy maintenance UI routes and templates

* admin: remove dead maintenance endpoint helpers

* Update layout_templ.go

* remove dummy_stress worker and command support

* refactor plugin UI to job-type top tabs and sub-tabs

* migrate weed worker command to plugin runtime

* remove plugin.worker command and keep worker runtime with metrics

* update helm worker args for jobType and execution flags

* set plugin scheduling defaults to global 16 and per-worker 4

* stress: fix RPC context reuse and remove redundant variables in ec_stress_runner

* admin/plugin: fix lifecycle races, safe channel operations, and terminal state constants

* admin/dash: randomize job IDs and fix priority zero-value overwrite in plugin API

* admin/handlers: implement buffered rendering to prevent response corruption

* admin/plugin: implement debounced persistence flusher and optimize BuildJobDetail memory lookups

* admin/plugin: fix priority overwrite and implement bounded wait in scheduler reserve

* admin/plugin: implement atomic file writes and fix run record side effects

* admin/plugin: use P prefix for parity shard labels in execution plans

* admin/plugin: enable parallel execution for cancellation tests

* admin: refactor time.Time fields to pointers for better JSON omitempty support

* admin/plugin: implement pointer-safe time assignments and comparisons in plugin core

* admin/plugin: fix time assignment and sorting logic in plugin monitor after pointer refactor

* admin/plugin: update scheduler activity tracking to use time pointers

* admin/plugin: fix time-based run history trimming after pointer refactor

* admin/dash: fix JobSpec struct literal in plugin API after pointer refactor

* admin/view: add D/P prefixes to EC shard badges for UI consistency

* admin/plugin: use lifecycle-aware context for schema prefetching

* Update ec_volume_details_templ.go

* admin/stress: fix proposal sorting and log volume cleanup errors

* stress: refine ec stress runner with math/rand and collection name

- Added Collection field to VolumeEcShardsDeleteRequest for correct filename construction.
- Replaced crypto/rand with seeded math/rand PRNG for bulk payloads.
- Added documentation for EcMinAge zero-value behavior.
- Added logging for ignored errors in volume/shard deletion.

* admin: return internal server error for plugin store failures

Changed error status code from 400 Bad Request to 500 Internal Server Error for failures in GetPluginJobDetail to correctly reflect server-side errors.

* admin: implement safe channel sends and graceful shutdown sync

- Added sync.WaitGroup to Plugin struct to manage background goroutines.
- Implemented safeSendCh helper using recover() to prevent panics on closed channels.
- Ensured Shutdown() waits for all background operations to complete.

* admin: robustify plugin monitor with nil-safe time and record init

- Standardized nil-safe assignment for *time.Time pointers (CreatedAt, UpdatedAt, CompletedAt).
- Ensured persistJobDetailSnapshot initializes new records correctly if they don't exist on disk.
- Fixed debounced persistence to trigger immediate write on job completion.

* admin: improve scheduler shutdown behavior and logic guards

- Replaced brittle error string matching with explicit r.shutdownCh selection for shutdown detection.
- Removed redundant nil guard in buildScheduledJobSpec.
- Standardized WaitGroup usage for schedulerLoop.

* admin: implement deep copy for job parameters and atomic write fixes

- Implemented deepCopyGenericValue and used it in cloneTrackedJob to prevent shared state.
- Ensured atomicWriteFile creates parent directories before writing.

* admin: remove unreachable branch in shard classification

Removed an unreachable 'totalShards <= 0' check in classifyShardID as dataShards and parityShards are already guarded.

* admin: secure UI links and use canonical shard constants

- Added rel="noopener noreferrer" to external links for security.
- Replaced magic number 14 with erasure_coding.TotalShardsCount.
- Used renderEcShardBadge for missing shard list consistency.

* admin: stabilize plugin tests and fix regressions

- Composed a robust plugin_monitor_test.go to handle asynchronous persistence.
- Updated all time.Time literals to use timeToPtr helper.
- Added explicit Shutdown() calls in tests to synchronize with debounced writes.
- Fixed syntax errors and orphaned struct literals in tests.

* Potential fix for code scanning alert no. 278: Slice memory allocation with excessive size value

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Potential fix for code scanning alert no. 283: Uncontrolled data used in path expression

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* admin: finalize refinements for error handling, scheduler, and race fixes

- Standardized HTTP 500 status codes for store failures in plugin_api.go.
- Tracked scheduled detection goroutines with sync.WaitGroup for safe shutdown.
- Fixed race condition in safeSendDetectionComplete by extracting channel under lock.
- Implemented deep copy for JobActivity details.
- Used defaultDirPerm constant in atomicWriteFile.

* test(ec): migrate admin dockertest to plugin APIs

* admin/plugin_api: fix RunPluginJobTypeAPI to return 500 for server-side detection/filter errors

* admin/plugin_api: fix ExecutePluginJobAPI to return 500 for job execution failures

* admin/plugin_api: limit parseProtoJSONBody request body to 1MB to prevent unbounded memory usage

* admin/plugin: consolidate regex to package-level validJobTypePattern; add char validation to sanitizeJobID

* admin/plugin: fix racy Shutdown channel close with sync.Once

* admin/plugin: track sendLoop and recv goroutines in WorkerStream with r.wg

* admin/plugin: document writeProtoFiles atomicity — .pb is source of truth, .json is human-readable only

* admin/plugin: extract activityLess helper to deduplicate nil-safe OccurredAt sort comparators

* test/ec: check http.NewRequest errors to prevent nil req panics

* test/ec: replace deprecated ioutil/math/rand, fix stale step comment 5.1→3.1

* plugin(ec): raise default detection and scheduling throughput limits

* topology: include empty disks in volume list and EC capacity fallback

* topology: remove hard 10-task cap for detection planning

* Update ec_volume_details_templ.go

* adjust default

* fix tests

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-02-18 13:42:41 -08:00
github-pawoandGitHub 5463038760 Remove trailing spaces (line 53) in seaweedfs-dev-compose.yml (#8365) 2026-02-18 07:32:07 -08:00
github-pawoandGitHub 828cbabb55 Add Admin UI to Docker Compose files (#8364) 2026-02-18 01:05:54 -08:00
Chris LuandGitHub 5919f519fd fix: allow overriding Enterprise image name using Helm #8361 (#8363)
* fix: allow overriding Enterprise image name using Helm #8361

* refactor: flatten image name construction logic for better readability
2026-02-17 13:49:16 -08:00
Chris Lu 63f641a6c9 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2026-02-16 17:01:26 -08:00
Chris Lu 3c3a78d08e 4.13 2026-02-16 17:01:19 -08:00
Chris LuandGitHub 3300874cb5 filer: add default log purging to master maintenance scripts (#8359)
* filer: add default log purging to master maintenance scripts

* filer: fix default maintenance scripts to include full set of tasks

* filer: refactor maintenance scripts to avoid duplication
2026-02-16 16:58:15 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Chris Lu
bddd7960c1 build(deps): bump org.apache.avro:avro from 1.11.4 to 1.11.5 in /test/java/spark (#8358)
build(deps): bump org.apache.avro:avro in /test/java/spark

Bumps org.apache.avro:avro from 1.11.4 to 1.11.5.

---
updated-dependencies:
- dependency-name: org.apache.avro:avro
  dependency-version: 1.11.5
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-02-16 15:28:55 -08:00
Lisandro PinandGitHub a9d12a0792 Implement full scrubbing for EC volumes (#8318)
Implement full scrubbing for EC volumes.
2026-02-16 15:09:01 -08:00
Chris Lu 564fc56698 Update docker-compose.yml 2026-02-16 15:01:09 -08:00
Lisandro PinandGitHub 11fdb68281 Fix superblock write error checks on volume compaction. (#8352) 2026-02-16 14:44:37 -08:00
Chris Lu 35ad7d08a5 remove debug 2026-02-16 14:03:02 -08:00
Chris LuandGitHub 0d8588e3ae S3: Implement IAM defaults and STS signing key fallback (#8348)
* S3: Implement IAM defaults and STS signing key fallback logic

* S3: Refactor startup order to init SSE-S3 key manager before IAM

* S3: Derive STS signing key from KEK using HKDF for security isolation

* S3: Document STS signing key fallback in security.toml

* fix(s3api): refine anonymous access logic and secure-by-default behavior

- Initialize anonymous identity by default in `NewIdentityAccessManagement` to prevent nil pointer exceptions.
- Ensure `ReplaceS3ApiConfiguration` preserves the anonymous identity if not present in the new configuration.
- Update `NewIdentityAccessManagement` signature to accept `filerClient`.
- In legacy mode (no policy engine), anonymous defaults to Deny (no actions), preserving secure-by-default behavior.
- Use specific `LookupAnonymous` method instead of generic map lookup.
- Update tests to accommodate signature changes and verify improved anonymous handling.

* feat(s3api): make IAM configuration optional

- Start S3 API server without a configuration file if `EnableIam` option is set.
- Default to `Allow` effect for policy engine when no configuration is provided (Zero-Config mode).
- Handle empty configuration path gracefully in `loadIAMManagerFromConfig`.
- Add integration test `iam_optional_test.go` to verify empty config behavior.

* fix(iamapi): fix signature mismatch in NewIdentityAccessManagementWithStore

* fix(iamapi): properly initialize FilerClient instead of passing nil

* fix(iamapi): properly initialize filer client for IAM management

- Instead of passing `nil`, construct a `wdclient.FilerClient` using the provided `Filers` addresses.
- Ensure `NewIdentityAccessManagementWithStore` receives a valid `filerClient` to avoid potential nil pointer dereferences or limited functionality.

* clean: remove dead code in s3api_server.go

* refactor(s3api): improve IAM initialization, safety and anonymous access security

* fix(s3api): ensure IAM config loads from filer after client init

* fix(s3): resolve test failures in integration, CORS, and tagging tests

- Fix CORS tests by providing explicit anonymous permissions config
- Fix S3 integration tests by setting admin credentials in init
- Align tagging test credentials in CI with IAM defaults
- Added goroutine to retry IAM config load in iamapi server

* fix(s3): allow anonymous access to health targets and S3 Tables when identities are present

* fix(ci): use /healthz for Caddy health check in awscli tests

* iam, s3api: expose DefaultAllow from IAM and Policy Engine

This allows checking the global "Open by Default" configuration from
other components like S3 Tables.

* s3api/s3tables: support DefaultAllow in permission logic and handler

Updated CheckPermissionWithContext to respect the DefaultAllow flag
in PolicyContext. This enables "Open by Default" behavior for
unauthenticated access in zero-config environments. Added a targeted
unit test to verify the logic.

* s3api/s3tables: propagate DefaultAllow through handlers

Propagated the DefaultAllow flag to individual handlers for
namespaces, buckets, tables, policies, and tagging. This ensures
consistent "Open by Default" behavior across all S3 Tables API
endpoints.

* s3api: wire up DefaultAllow for S3 Tables API initialization

Updated registerS3TablesRoutes to query the global IAM configuration
and set the DefaultAllow flag on the S3 Tables API server. This
completes the end-to-end propagation required for anonymous access in
zero-config environments. Added a SetDefaultAllow method to
S3TablesApiServer to facilitate this.

* s3api: fix tests by adding DefaultAllow to mock IAM integrations

The IAMIntegration interface was updated to include DefaultAllow(),
breaking several mock implementations in tests. This commit fixes
the build errors by adding the missing method to the mocks.

* env

* ensure ports

* env

* env

* fix default allow

* add one more test using non-anonymous user

* debug

* add more debug

* less logs
2026-02-16 13:59:13 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
cc58272219 build(deps): bump github.com/klauspost/compress from 1.18.3 to 1.18.4 (#8353)
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.18.3 to 1.18.4.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Commits](https://github.com/klauspost/compress/compare/v1.18.3...v1.18.4)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-version: 1.18.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-16 10:58:13 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
5be4ee9892 build(deps): bump github.com/redis/go-redis/v9 from 9.17.2 to 9.18.0 (#8356)
Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) from 9.17.2 to 9.18.0.
- [Release notes](https://github.com/redis/go-redis/releases)
- [Changelog](https://github.com/redis/go-redis/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/redis/go-redis/compare/v9.17.2...v9.18.0)

---
updated-dependencies:
- dependency-name: github.com/redis/go-redis/v9
  dependency-version: 9.18.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-16 10:57:54 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
22e74221cb build(deps): bump github.com/getsentry/sentry-go from 0.40.0 to 0.42.0 (#8357)
Bumps [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go) from 0.40.0 to 0.42.0.
- [Release notes](https://github.com/getsentry/sentry-go/releases)
- [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md)
- [Commits](https://github.com/getsentry/sentry-go/compare/v0.40.0...v0.42.0)

---
updated-dependencies:
- dependency-name: github.com/getsentry/sentry-go
  dependency-version: 0.42.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-16 10:57:43 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
cc80641be1 build(deps): bump github.com/mattn/go-sqlite3 from 1.14.33 to 1.14.34 (#8355)
Bumps [github.com/mattn/go-sqlite3](https://github.com/mattn/go-sqlite3) from 1.14.33 to 1.14.34.
- [Release notes](https://github.com/mattn/go-sqlite3/releases)
- [Commits](https://github.com/mattn/go-sqlite3/compare/v1.14.33...v1.14.34)

---
updated-dependencies:
- dependency-name: github.com/mattn/go-sqlite3
  dependency-version: 1.14.34
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-16 10:57:31 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
927c906379 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.6.3 to 1.6.4 (#8354)
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob

Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.6.3 to 1.6.4.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/storage/azblob/v1.6.3...sdk/storage/azblob/v1.6.4)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-version: 1.6.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-16 10:57:17 -08:00
Lisandro PinandGitHub 0721e3c1e9 Rework volume compaction (a.k.a vacuuming) logic to cleanly support new parameters. (#8337)
We'll leverage on this to support a "ignore broken needles" option, necessary
to properly recover damaged volumes, as described in
https://github.com/seaweedfs/seaweedfs/issues/7442#issuecomment-3897784283 .
2026-02-16 02:15:14 -08:00
449 changed files with 39885 additions and 19582 deletions
+1 -1
View File
@@ -11,4 +11,4 @@ jobs:
- name: 'Checkout Repository'
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8
- name: 'Dependency Review'
uses: actions/dependency-review-action@3c4e3dcb1aa7874d2c16be7d79418e9b7efd6261
uses: actions/dependency-review-action@05fe4576374b728f0c523d6a13d64c25081e0803
+1 -1
View File
@@ -119,7 +119,7 @@ jobs:
echo "✅ All template rendering tests passed!"
- name: Create kind cluster
uses: helm/kind-action@v1.13.0
uses: helm/kind-action@v1.14.0
- name: Run chart-testing (install)
run: ct install --target-branch ${{ github.event.repository.default_branch }} --all --chart-dirs k8s/charts
+1 -1
View File
@@ -111,7 +111,7 @@ jobs:
# Wait for S3 API
for i in {1..30}; do
if curl -s http://localhost:8333/ > /dev/null 2>&1; then
if curl -s http://localhost:8333/healthz > /dev/null 2>&1; then
echo "✓ S3 API is ready"
break
fi
+39
View File
@@ -0,0 +1,39 @@
name: "Plugin Worker Integration Tests"
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
permissions:
contents: read
jobs:
plugin-worker:
name: "Plugin Worker: ${{ matrix.worker }}"
runs-on: ubuntu-22.04
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
include:
- worker: erasure_coding
path: test/plugin_workers/erasure_coding
- worker: vacuum
path: test/plugin_workers/vacuum
- worker: volume_balance
path: test/plugin_workers/volume_balance
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v6
with:
go-version: ^1.26
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@v6
- name: Run plugin worker tests
run: go test -v ./${{ matrix.path }}
@@ -39,6 +39,14 @@ jobs:
echo "=== Running S3 Integration Tests ==="
go test -v -timeout=60s -run TestS3Integration ./...
- name: Run S3 DeleteBucketNotEmpty Tests
timeout-minutes: 15
working-directory: test/s3/normal
run: |
set -x
echo "=== Running S3 DeleteBucketNotEmpty Tests ==="
go test -v -timeout=60s -run TestS3DeleteBucketNotEmpty ./...
- name: Run IAM Integration Tests
timeout-minutes: 15
working-directory: test/s3/normal
+2
View File
@@ -461,6 +461,8 @@ jobs:
export S3_ENDPOINT="http://localhost:8006"
export S3_ACCESS_KEY="0555b35654ad1656d804"
export S3_SECRET_KEY="h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=="
export AWS_ACCESS_KEY_ID="$S3_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="$S3_SECRET_KEY"
# Run the specific test that is equivalent to AWS S3 tagging behavior
make test-with-server || {
+1 -1
View File
@@ -97,7 +97,7 @@ jobs:
# Verify service accessibility
echo "=== Verifying Service Accessibility ==="
curl -f http://localhost:8080/realms/master
curl -s http://localhost:8333
curl -s http://localhost:8333/healthz
echo "✅ SeaweedFS S3 API is responding (IAM-protected endpoint)"
# Run Keycloak-specific tests
@@ -0,0 +1,122 @@
name: "S3 Proxy Signature Tests"
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
concurrency:
group: ${{ github.head_ref || github.ref }}/s3-proxy-signature-tests
cancel-in-progress: true
permissions:
contents: read
jobs:
proxy-signature-tests:
name: S3 Proxy Signature Verification Tests
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@v6
- name: Set up Go 1.x
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
id: go
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build SeaweedFS binary for Linux
run: |
set -x
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -buildvcs=false -v -o test/s3/proxy_signature/weed ./weed
- name: Run S3 Proxy Signature Tests
timeout-minutes: 10
working-directory: test/s3/proxy_signature
run: |
set -x
echo "Starting Docker Compose services..."
docker compose up -d --build
# Check if containers are running
echo "Checking container status..."
docker compose ps
# Wait for services to be ready
echo "Waiting for nginx proxy to be ready..."
PROXY_READY=0
for i in $(seq 1 30); do
if curl -s http://localhost:9000/ > /dev/null 2>&1; then
echo "Proxy is ready"
PROXY_READY=1
break
fi
echo "Waiting for proxy... ($i/30)"
sleep 1
done
if [ $PROXY_READY -eq 0 ]; then
echo "ERROR: Proxy failed to become ready after 30 seconds"
echo "Docker compose logs:"
docker compose logs --no-color || true
exit 1
fi
# Wait for SeaweedFS to be ready
echo "Waiting for SeaweedFS S3 gateway to be ready via proxy..."
S3_READY=0
for i in $(seq 1 30); do
# Check logs first for startup message (weed mini says "S3 service is ready")
if docker compose logs seaweedfs 2>&1 | grep -qE "S3 (gateway|service).*(started|ready)"; then
echo "SeaweedFS S3 gateway is ready"
S3_READY=1
break
fi
# Fallback: check headers via proxy (which is already ready)
if curl -s -I http://localhost:9000/ | grep -qi "SeaweedFS"; then
echo "SeaweedFS S3 gateway is responding via proxy"
S3_READY=1
break
fi
echo "Waiting for S3 gateway... ($i/30)"
sleep 1
done
if [ $S3_READY -eq 0 ]; then
echo "ERROR: SeaweedFS S3 gateway failed to become ready after 30 seconds"
echo "Latest seaweedfs logs:"
docker compose logs --no-color --tail 20 seaweedfs || true
exit 1
fi
# Run the test script inside AWS CLI container
echo "Running test script..."
docker run --rm --network host \
--entrypoint bash \
amazon/aws-cli:latest \
-c "$(cat test.sh)"
TEST_RESULT=$?
# Cleanup
docker compose down
exit $TEST_RESULT
- name: Cleanup on failure
if: failure()
working-directory: test/s3/proxy_signature
run: |
echo "Cleaning up Docker containers..."
ls -al weed || true
ldd weed || true
echo "Docker compose logs:"
docker compose logs --no-color || true
echo "Container status before cleanup:"
docker ps -a
echo "Stopping services..."
docker compose down || true
@@ -35,7 +35,7 @@ jobs:
set -e
mkdir -p /tmp/data
./weed -v=3 server -s3 -dir=/tmp/data -s3.config=../docker/compose/s3.json -master.peers=none > weed.log 2>&1 &
until curl -s http://localhost:8333/ > /dev/null; do sleep 1; done
until curl -s http://localhost:8333/healthz > /dev/null; do sleep 1; done
- name: Setup Caddy
run: |
@@ -54,7 +54,7 @@ jobs:
- name: Start Caddy
run: |
./caddy start
until curl -fsS --insecure https://localhost:8443 > /dev/null; do sleep 1; done
until curl -fsS --insecure https://localhost:8443/healthz > /dev/null; do sleep 1; done
- name: Create Bucket
run: |
@@ -9,7 +9,6 @@ on:
- 'weed/storage/**'
- 'weed/pb/volume_server.proto'
- 'weed/pb/volume_server_pb/**'
- 'rust/volume_server/**'
- '.github/workflows/volume-server-integration-tests.yml'
push:
branches: [ master, main ]
@@ -19,7 +18,6 @@ on:
- 'weed/storage/**'
- 'weed/pb/volume_server.proto'
- 'weed/pb/volume_server_pb/**'
- 'rust/volume_server/**'
- '.github/workflows/volume-server-integration-tests.yml'
concurrency:
@@ -122,55 +120,3 @@ jobs:
echo "## Volume Server Integration Test Summary (${{ matrix.test-type }} - Shard ${{ matrix.shard }})" >> "$GITHUB_STEP_SUMMARY"
echo "- Suite: test/volume_server/${{ matrix.test-type }} (Pattern: ${TEST_PATTERN})" >> "$GITHUB_STEP_SUMMARY"
echo "- Command: go test -v -count=1 -timeout=${{ env.TEST_TIMEOUT }} ./test/volume_server/${{ matrix.test-type }}/... -run \"${TEST_PATTERN}\"" >> "$GITHUB_STEP_SUMMARY"
volume-server-rust-smoke:
name: Volume Server Rust Smoke (${{ matrix.mode }})
runs-on: ubuntu-22.04
timeout-minutes: 40
strategy:
fail-fast: false
matrix:
mode: [exec, proxy, native]
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Set up Go ${{ env.GO_VERSION }}
uses: actions/setup-go@v6
with:
go-version: ${{ env.GO_VERSION }}
- name: Set up Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Build SeaweedFS binary
run: |
cd weed
go build -o weed .
chmod +x weed
./weed version
- name: Run Rust-mode HTTP smoke tests
env:
WEED_BINARY: ${{ github.workspace }}/weed/weed
VOLUME_SERVER_IMPL: rust
VOLUME_SERVER_RUST_MODE: ${{ matrix.mode }}
run: |
go test -v -count=1 -timeout=25m ./test/volume_server/http/... -run "TestAdminStatusAndHealthz|TestUploadReadRangeHeadDeleteRoundTrip"
- name: Run Rust-mode gRPC smoke tests
env:
WEED_BINARY: ${{ github.workspace }}/weed/weed
VOLUME_SERVER_IMPL: rust
VOLUME_SERVER_RUST_MODE: ${{ matrix.mode }}
run: |
go test -v -count=1 -timeout=25m ./test/volume_server/grpc/... -run "TestStateAndStatusRPCs|TestVolumeSyncStatusAndReadVolumeFileStatus"
- name: Rust smoke summary
if: always()
run: |
echo "## Volume Server Rust Smoke Summary (${{ matrix.mode }})" >> "$GITHUB_STEP_SUMMARY"
echo "- Mode: Go master + Rust volume server launcher (VOLUME_SERVER_IMPL=rust, VOLUME_SERVER_RUST_MODE=${{ matrix.mode }})" >> "$GITHUB_STEP_SUMMARY"
echo "- HTTP Command: go test -v -count=1 -timeout=25m ./test/volume_server/http/... -run \"TestAdminStatusAndHealthz|TestUploadReadRangeHeadDeleteRoundTrip\"" >> "$GITHUB_STEP_SUMMARY"
echo "- gRPC Command: go test -v -count=1 -timeout=25m ./test/volume_server/grpc/... -run \"TestStateAndStatusRPCs|TestVolumeSyncStatusAndReadVolumeFileStatus\"" >> "$GITHUB_STEP_SUMMARY"
+5 -2
View File
@@ -1,4 +1,4 @@
.PHONY: test admin-generate admin-build admin-clean admin-dev admin-run admin-test admin-fmt admin-help
.PHONY: test admin-generate admin-build admin-clean admin-dev admin-run admin-test admin-fmt admin-help weed-commands
BINARY = weed
ADMIN_DIR = weed/admin
@@ -11,6 +11,9 @@ all: install
install: admin-generate
cd weed; go install
weed-commands:
cd weed && $(MAKE) weed-db weed-sql
warp_install:
go install github.com/minio/warp@v0.7.6
@@ -40,7 +43,7 @@ test: admin-generate
# Admin component targets
admin-generate:
@echo "Generating admin component templates..."
@templ generate
@cd $(ADMIN_DIR) && templ generate ./view
admin-build: admin-generate
@echo "Building admin component..."
+116 -101
View File
@@ -1,10 +1,11 @@
package command
package main
import (
"context"
"crypto/tls"
"encoding/json"
"fmt"
"io"
"os"
"os/signal"
"strings"
@@ -13,43 +14,12 @@ import (
"github.com/seaweedfs/seaweedfs/weed/server/postgres"
"github.com/seaweedfs/seaweedfs/weed/util"
flag "github.com/seaweedfs/seaweedfs/weed/util/fla9"
)
var (
dbOptions DBOptions
)
const usageLine = "weed-db -port=5432 -master=<master_server>"
type DBOptions struct {
host *string
port *int
masterAddr *string
authMethod *string
users *string
database *string
maxConns *int
idleTimeout *string
tlsCert *string
tlsKey *string
}
func init() {
cmdDB.Run = runDB // break init cycle
dbOptions.host = cmdDB.Flag.String("host", "localhost", "Database server host")
dbOptions.port = cmdDB.Flag.Int("port", 5432, "Database server port")
dbOptions.masterAddr = cmdDB.Flag.String("master", "localhost:9333", "SeaweedFS master server address")
dbOptions.authMethod = cmdDB.Flag.String("auth", "trust", "Authentication method: trust, password, md5")
dbOptions.users = cmdDB.Flag.String("users", "", "User credentials for auth (JSON format '{\"user1\":\"pass1\",\"user2\":\"pass2\"}' or file '@/path/to/users.json')")
dbOptions.database = cmdDB.Flag.String("database", "default", "Default database name")
dbOptions.maxConns = cmdDB.Flag.Int("max-connections", 100, "Maximum concurrent connections per server")
dbOptions.idleTimeout = cmdDB.Flag.String("idle-timeout", "1h", "Connection idle timeout")
dbOptions.tlsCert = cmdDB.Flag.String("tls-cert", "", "TLS certificate file path")
dbOptions.tlsKey = cmdDB.Flag.String("tls-key", "", "TLS private key file path")
}
var cmdDB = &Command{
UsageLine: "db -port=5432 -master=<master_server>",
Short: "start a PostgreSQL-compatible database server for SQL queries",
Long: `Start a PostgreSQL wire protocol compatible database server that provides SQL query access to SeaweedFS.
const longHelp = `Start a PostgreSQL wire protocol compatible database server that provides SQL query access to SeaweedFS.
This database server enables any PostgreSQL client, tool, or application to connect to SeaweedFS
and execute SQL queries against MQ topics. It implements the PostgreSQL wire protocol for maximum
@@ -58,25 +28,25 @@ compatibility with the existing PostgreSQL ecosystem.
Examples:
# Start database server on default port 5432
weed db
weed-db
# Start with MD5 authentication using JSON format (recommended)
weed db -auth=md5 -users='{"admin":"secret","readonly":"view123"}'
weed-db -auth=md5 -users='{"admin":"secret","readonly":"view123"}'
# Start with complex passwords using JSON format
weed db -auth=md5 -users='{"admin":"pass;with;semicolons","user":"password:with:colons"}'
weed-db -auth=md5 -users='{"admin":"pass;with;semicolons","user":"password:with:colons"}'
# Start with credentials from JSON file (most secure)
weed db -auth=md5 -users="@/etc/seaweedfs/users.json"
weed-db -auth=md5 -users="@/etc/seaweedfs/users.json"
# Start with custom port and master
weed db -port=5433 -master=master1:9333
weed-db -port=5433 -master=master1:9333
# Allow connections from any host
weed db -host=0.0.0.0 -port=5432
weed-db -host=0.0.0.0 -port=5432
# Start with TLS encryption
weed db -tls-cert=server.crt -tls-key=server.key
weed-db -tls-cert=server.crt -tls-key=server.key
Client Connection Examples:
@@ -95,7 +65,7 @@ Programming Language Examples:
# Python (psycopg2)
import psycopg2
conn = psycopg2.connect(
host="localhost", port=5432,
host="localhost", port=5432,
user="seaweedfs", database="default"
)
@@ -116,7 +86,7 @@ Supported SQL Operations:
- SELECT queries on MQ topics
- DESCRIBE/DESC table_name commands
- EXPLAIN query execution plans
- SHOW DATABASES/TABLES commands
- SHOW DATABASES/TABLES commands
- Aggregation functions (COUNT, SUM, AVG, MIN, MAX)
- WHERE clauses with filtering
- System columns (_timestamp_ns, _key, _source)
@@ -149,50 +119,95 @@ Performance Features:
- PostgreSQL wire protocol
- Query result streaming
`,
`
type Options struct {
Host string
Port int
MasterAddr string
AuthMethod string
Users string
Database string
MaxConns int
IdleTimeout string
TLSCert string
TLSKey string
}
func runDB(cmd *Command, args []string) bool {
// Run executes the weed-db CLI.
func Run(args []string) int {
fs := flag.NewFlagSet("weed-db", flag.ContinueOnError)
usageWriter := io.Writer(os.Stderr)
fs.SetOutput(usageWriter)
var opts Options
fs.StringVar(&opts.Host, "host", "localhost", "Database server host")
fs.IntVar(&opts.Port, "port", 5432, "Database server port")
fs.StringVar(&opts.MasterAddr, "master", "localhost:9333", "SeaweedFS master server address")
fs.StringVar(&opts.AuthMethod, "auth", "trust", "Authentication method: trust, password, md5")
fs.StringVar(&opts.Users, "users", "", "User credentials for auth (JSON format '{\"user1\":\"pass1\",\"user2\":\"pass2\"}' or file '@/path/to/users.json')")
fs.StringVar(&opts.Database, "database", "default", "Default database name")
fs.IntVar(&opts.MaxConns, "max-connections", 100, "Maximum concurrent connections per server")
fs.StringVar(&opts.IdleTimeout, "idle-timeout", "1h", "Connection idle timeout")
fs.StringVar(&opts.TLSCert, "tls-cert", "", "TLS certificate file path")
fs.StringVar(&opts.TLSKey, "tls-key", "", "TLS private key file path")
fs.Usage = func() {
fmt.Fprintf(usageWriter, "Usage: %s\n\n%s\n", usageLine, longHelp)
fmt.Fprintln(usageWriter, "Default Parameters:")
fs.PrintDefaults()
}
if err := fs.Parse(args); err != nil {
return 2
}
if !runWithOptions(&opts) {
return 1
}
return 0
}
func runWithOptions(opts *Options) bool {
util.LoadConfiguration("security", false)
// Validate options
if *dbOptions.masterAddr == "" {
// Validate options.
if opts.MasterAddr == "" {
fmt.Fprintf(os.Stderr, "Error: master address is required\n")
return false
}
// Parse authentication method
authMethod, err := parseAuthMethod(*dbOptions.authMethod)
// Parse authentication method.
authMethod, err := parseAuthMethod(opts.AuthMethod)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
return false
}
// Parse user credentials
users, err := parseUsers(*dbOptions.users, authMethod)
// Parse user credentials.
users, err := parseUsers(opts.Users, authMethod)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
return false
}
// Parse idle timeout
idleTimeout, err := time.ParseDuration(*dbOptions.idleTimeout)
// Parse idle timeout.
idleTimeout, err := time.ParseDuration(opts.IdleTimeout)
if err != nil {
fmt.Fprintf(os.Stderr, "Error parsing idle timeout: %v\n", err)
return false
}
// Validate port number
if err := validatePortNumber(*dbOptions.port); err != nil {
// Validate port number.
if err := validatePortNumber(opts.Port); err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
return false
}
// Setup TLS if requested
// Setup TLS if requested.
var tlsConfig *tls.Config
if *dbOptions.tlsCert != "" && *dbOptions.tlsKey != "" {
cert, err := tls.LoadX509KeyPair(*dbOptions.tlsCert, *dbOptions.tlsKey)
if opts.TLSCert != "" && opts.TLSKey != "" {
cert, err := tls.LoadX509KeyPair(opts.TLSCert, opts.TLSKey)
if err != nil {
fmt.Fprintf(os.Stderr, "Error loading TLS certificates: %v\n", err)
return false
@@ -202,34 +217,34 @@ func runDB(cmd *Command, args []string) bool {
}
}
// Create server configuration
// Create server configuration.
config := &postgres.PostgreSQLServerConfig{
Host: *dbOptions.host,
Port: *dbOptions.port,
Host: opts.Host,
Port: opts.Port,
AuthMethod: authMethod,
Users: users,
Database: *dbOptions.database,
MaxConns: *dbOptions.maxConns,
Database: opts.Database,
MaxConns: opts.MaxConns,
IdleTimeout: idleTimeout,
TLSConfig: tlsConfig,
}
// Create database server
dbServer, err := postgres.NewPostgreSQLServer(config, *dbOptions.masterAddr)
// Create database server.
dbServer, err := postgres.NewPostgreSQLServer(config, opts.MasterAddr)
if err != nil {
fmt.Fprintf(os.Stderr, "Error creating database server: %v\n", err)
return false
}
// Print startup information
// Print startup information.
fmt.Printf("Starting SeaweedFS Database Server...\n")
fmt.Printf("Host: %s\n", *dbOptions.host)
fmt.Printf("Port: %d\n", *dbOptions.port)
fmt.Printf("Master: %s\n", *dbOptions.masterAddr)
fmt.Printf("Database: %s\n", *dbOptions.database)
fmt.Printf("Auth Method: %s\n", *dbOptions.authMethod)
fmt.Printf("Max Connections: %d\n", *dbOptions.maxConns)
fmt.Printf("Idle Timeout: %s\n", *dbOptions.idleTimeout)
fmt.Printf("Host: %s\n", opts.Host)
fmt.Printf("Port: %d\n", opts.Port)
fmt.Printf("Master: %s\n", opts.MasterAddr)
fmt.Printf("Database: %s\n", opts.Database)
fmt.Printf("Auth Method: %s\n", opts.AuthMethod)
fmt.Printf("Max Connections: %d\n", opts.MaxConns)
fmt.Printf("Idle Timeout: %s\n", opts.IdleTimeout)
if tlsConfig != nil {
fmt.Printf("TLS: Enabled\n")
} else {
@@ -240,15 +255,15 @@ func runDB(cmd *Command, args []string) bool {
}
fmt.Printf("\nDatabase Connection Examples:\n")
fmt.Printf(" psql -h %s -p %d -U seaweedfs -d %s\n", *dbOptions.host, *dbOptions.port, *dbOptions.database)
fmt.Printf(" psql -h %s -p %d -U seaweedfs -d %s\n", opts.Host, opts.Port, opts.Database)
if len(users) > 0 {
// Show first user as example
// Show first user as example.
for username := range users {
fmt.Printf(" psql -h %s -p %d -U %s -d %s\n", *dbOptions.host, *dbOptions.port, username, *dbOptions.database)
fmt.Printf(" psql -h %s -p %d -U %s -d %s\n", opts.Host, opts.Port, username, opts.Database)
break
}
}
fmt.Printf(" postgresql://%s:%d/%s\n", *dbOptions.host, *dbOptions.port, *dbOptions.database)
fmt.Printf(" postgresql://%s:%d/%s\n", opts.Host, opts.Port, opts.Database)
fmt.Printf("\nSupported Operations:\n")
fmt.Printf(" - SELECT queries on MQ topics\n")
@@ -261,26 +276,26 @@ func runDB(cmd *Command, args []string) bool {
fmt.Printf("\nReady for database connections!\n\n")
// Start the server
// Start the server.
err = dbServer.Start()
if err != nil {
fmt.Fprintf(os.Stderr, "Error starting database server: %v\n", err)
return false
}
// Set up signal handling for graceful shutdown
// Set up signal handling for graceful shutdown.
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
// Wait for shutdown signal
// Wait for shutdown signal.
<-sigChan
fmt.Printf("\nReceived shutdown signal, stopping database server...\n")
// Create context with timeout for graceful shutdown
// Create context with timeout for graceful shutdown.
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
// Stop the server with timeout
// Stop the server with timeout.
done := make(chan error, 1)
go func() {
done <- dbServer.Stop()
@@ -301,7 +316,7 @@ func runDB(cmd *Command, args []string) bool {
return true
}
// parseAuthMethod parses the authentication method string
// parseAuthMethod parses the authentication method string.
func parseAuthMethod(method string) (postgres.AuthMethod, error) {
switch strings.ToLower(method) {
case "trust":
@@ -315,7 +330,7 @@ func parseAuthMethod(method string) (postgres.AuthMethod, error) {
}
}
// parseUsers parses the user credentials string with support for secure formats only
// parseUsers parses the user credentials string with support for secure formats only.
// Supported formats:
// 1. JSON format: {"username":"password","username2":"password2"}
// 2. File format: /path/to/users.json or @/path/to/users.json
@@ -323,41 +338,41 @@ func parseUsers(usersStr string, authMethod postgres.AuthMethod) (map[string]str
users := make(map[string]string)
if usersStr == "" {
// No users specified
// No users specified.
if authMethod != postgres.AuthTrust {
return nil, fmt.Errorf("users must be specified when auth method is not 'trust'")
}
return users, nil
}
// Trim whitespace
// Trim whitespace.
usersStr = strings.TrimSpace(usersStr)
// Determine format and parse accordingly
// Determine format and parse accordingly.
if strings.HasPrefix(usersStr, "{") && strings.HasSuffix(usersStr, "}") {
// JSON format
// JSON format.
return parseUsersJSON(usersStr, authMethod)
}
// Check if it's a file path (with or without @ prefix) before declaring invalid format
// Check if it's a file path (with or without @ prefix) before declaring invalid format.
filePath := strings.TrimPrefix(usersStr, "@")
if _, err := os.Stat(filePath); err == nil {
// File format
return parseUsersFile(usersStr, authMethod) // Pass original string to preserve @ handling
// File format.
return parseUsersFile(usersStr, authMethod) // Pass original string to preserve @ handling.
}
// Invalid format
// Invalid format.
return nil, fmt.Errorf("invalid user credentials format. Use JSON format '{\"user\":\"pass\"}' or file format '@/path/to/users.json' or 'path/to/users.json'. Legacy semicolon-separated format is no longer supported")
}
// parseUsersJSON parses user credentials from JSON format
// parseUsersJSON parses user credentials from JSON format.
func parseUsersJSON(jsonStr string, authMethod postgres.AuthMethod) (map[string]string, error) {
var users map[string]string
if err := json.Unmarshal([]byte(jsonStr), &users); err != nil {
return nil, fmt.Errorf("invalid JSON format for users: %v", err)
}
// Validate users
// Validate users.
for username, password := range users {
if username == "" {
return nil, fmt.Errorf("empty username in JSON user specification")
@@ -370,12 +385,12 @@ func parseUsersJSON(jsonStr string, authMethod postgres.AuthMethod) (map[string]
return users, nil
}
// parseUsersFile parses user credentials from a JSON file
// parseUsersFile parses user credentials from a JSON file.
func parseUsersFile(filePath string, authMethod postgres.AuthMethod) (map[string]string, error) {
// Remove @ prefix if present
// Remove @ prefix if present.
filePath = strings.TrimPrefix(filePath, "@")
// Read file content
// Read file content.
content, err := os.ReadFile(filePath)
if err != nil {
return nil, fmt.Errorf("failed to read users file '%s': %v", filePath, err)
@@ -383,16 +398,16 @@ func parseUsersFile(filePath string, authMethod postgres.AuthMethod) (map[string
contentStr := strings.TrimSpace(string(content))
// File must contain JSON format
// File must contain JSON format.
if !strings.HasPrefix(contentStr, "{") || !strings.HasSuffix(contentStr, "}") {
return nil, fmt.Errorf("users file '%s' must contain JSON format: {\"user\":\"pass\"}. Legacy formats are no longer supported", filePath)
}
// Parse as JSON
// Parse as JSON.
return parseUsersJSON(contentStr, authMethod)
}
// validatePortNumber validates that the port number is reasonable
// validatePortNumber validates that the port number is reasonable.
func validatePortNumber(port int) error {
if port < 1 || port > 65535 {
return fmt.Errorf("port number must be between 1 and 65535, got %d", port)
+7
View File
@@ -0,0 +1,7 @@
package main
import "os"
func main() {
os.Exit(Run(os.Args[1:]))
}
+7
View File
@@ -0,0 +1,7 @@
package main
import "os"
func main() {
os.Exit(Run(os.Args[1:]))
}
+134 -107
View File
@@ -1,4 +1,4 @@
package command
package main
import (
"context"
@@ -13,28 +13,24 @@ import (
"github.com/peterh/liner"
"github.com/seaweedfs/seaweedfs/weed/query/engine"
flag "github.com/seaweedfs/seaweedfs/weed/util/fla9"
"github.com/seaweedfs/seaweedfs/weed/util/grace"
"github.com/seaweedfs/seaweedfs/weed/util/sqlutil"
)
func init() {
cmdSql.Run = runSql
}
const usageLine = "weed-sql [-master=localhost:9333] [-interactive] [-file=query.sql] [-output=table|json|csv] [-database=dbname] [-query=\"SQL\"]"
var cmdSql = &Command{
UsageLine: "sql [-master=localhost:9333] [-interactive] [-file=query.sql] [-output=table|json|csv] [-database=dbname] [-query=\"SQL\"]",
Short: "advanced SQL query interface for SeaweedFS MQ topics with multiple execution modes",
Long: `Enhanced SQL interface for SeaweedFS Message Queue topics with multiple execution modes.
const longHelp = `Enhanced SQL interface for SeaweedFS Message Queue topics with multiple execution modes.
Execution Modes:
- Interactive shell (default): weed sql -interactive
- Single query: weed sql -query "SELECT * FROM user_events"
- Batch from file: weed sql -file queries.sql
- Context switching: weed sql -database analytics -interactive
- Interactive shell (default): weed-sql -interactive
- Single query: weed-sql -query "SELECT * FROM user_events"
- Batch from file: weed-sql -file queries.sql
- Context switching: weed-sql -database analytics -interactive
Output Formats:
- table: ASCII table format (default for interactive)
- json: JSON format (default for non-interactive)
- json: JSON format (default for non-interactive)
- csv: Comma-separated values
Features:
@@ -45,24 +41,23 @@ Features:
- Database context switching
Examples:
weed sql -interactive
weed sql -query "SHOW DATABASES" -output json
weed sql -file batch_queries.sql -output csv
weed sql -database analytics -query "SELECT COUNT(*) FROM metrics"
weed sql -master broker1:9333 -interactive
`,
weed-sql -interactive
weed-sql -query "SHOW DATABASES" -output json
weed-sql -file batch_queries.sql -output csv
weed-sql -database analytics -query "SELECT COUNT(*) FROM metrics"
weed-sql -master broker1:9333 -interactive
`
type Options struct {
Master string
Interactive bool
File string
Output string
Database string
Query string
}
var (
sqlMaster = cmdSql.Flag.String("master", "localhost:9333", "SeaweedFS master server HTTP address")
sqlInteractive = cmdSql.Flag.Bool("interactive", false, "start interactive shell mode")
sqlFile = cmdSql.Flag.String("file", "", "execute SQL queries from file")
sqlOutput = cmdSql.Flag.String("output", "", "output format: table, json, csv (auto-detected if not specified)")
sqlDatabase = cmdSql.Flag.String("database", "", "default database context")
sqlQuery = cmdSql.Flag.String("query", "", "execute single SQL query")
)
// OutputFormat represents different output formatting options
// OutputFormat represents different output formatting options.
type OutputFormat string
const (
@@ -71,50 +66,82 @@ const (
OutputCSV OutputFormat = "csv"
)
// SQLContext holds the execution context for SQL operations
// SQLContext holds the execution context for SQL operations.
type SQLContext struct {
engine *engine.SQLEngine
currentDatabase string
outputFormat OutputFormat
interactive bool
master string
}
func runSql(command *Command, args []string) bool {
// Initialize SQL engine with master address for service discovery
sqlEngine := engine.NewSQLEngine(*sqlMaster)
// Run executes the weed-sql CLI.
func Run(args []string) int {
fs := flag.NewFlagSet("weed-sql", flag.ContinueOnError)
usageWriter := io.Writer(os.Stderr)
fs.SetOutput(usageWriter)
// Determine execution mode and output format
interactive := *sqlInteractive || (*sqlQuery == "" && *sqlFile == "")
outputFormat := determineOutputFormat(*sqlOutput, interactive)
var opts Options
fs.StringVar(&opts.Master, "master", "localhost:9333", "SeaweedFS master server HTTP address")
fs.BoolVar(&opts.Interactive, "interactive", false, "start interactive shell mode")
fs.StringVar(&opts.File, "file", "", "execute SQL queries from file")
fs.StringVar(&opts.Output, "output", "", "output format: table, json, csv (auto-detected if not specified)")
fs.StringVar(&opts.Database, "database", "", "default database context")
fs.StringVar(&opts.Query, "query", "", "execute single SQL query")
// Create SQL context
fs.Usage = func() {
fmt.Fprintf(usageWriter, "Usage: %s\n\n%s\n", usageLine, longHelp)
fmt.Fprintln(usageWriter, "Default Parameters:")
fs.PrintDefaults()
}
if err := fs.Parse(args); err != nil {
return 2
}
if !runWithOptions(&opts) {
return 1
}
return 0
}
func runWithOptions(opts *Options) bool {
// Initialize SQL engine with master address for service discovery.
sqlEngine := engine.NewSQLEngine(opts.Master)
// Determine execution mode and output format.
interactive := opts.Interactive || (opts.Query == "" && opts.File == "")
outputFormat := determineOutputFormat(opts.Output, interactive)
// Create SQL context.
ctx := &SQLContext{
engine: sqlEngine,
currentDatabase: *sqlDatabase,
currentDatabase: opts.Database,
outputFormat: outputFormat,
interactive: interactive,
master: opts.Master,
}
// Set current database in SQL engine if specified via command line
if *sqlDatabase != "" {
ctx.engine.GetCatalog().SetCurrentDatabase(*sqlDatabase)
// Set current database in SQL engine if specified via command line.
if opts.Database != "" {
ctx.engine.GetCatalog().SetCurrentDatabase(opts.Database)
}
// Execute based on mode
// Execute based on mode.
switch {
case *sqlQuery != "":
// Single query mode
return executeSingleQuery(ctx, *sqlQuery)
case *sqlFile != "":
// Batch file mode
return executeFileQueries(ctx, *sqlFile)
case opts.Query != "":
// Single query mode.
return executeSingleQuery(ctx, opts.Query)
case opts.File != "":
// Batch file mode.
return executeFileQueries(ctx, opts.File)
default:
// Interactive mode
// Interactive mode.
return runInteractiveShell(ctx)
}
}
// determineOutputFormat selects the appropriate output format
// determineOutputFormat selects the appropriate output format.
func determineOutputFormat(specified string, interactive bool) OutputFormat {
switch strings.ToLower(specified) {
case "table":
@@ -124,7 +151,7 @@ func determineOutputFormat(specified string, interactive bool) OutputFormat {
case "csv":
return OutputCSV
default:
// Auto-detect based on mode
// Auto-detect based on mode.
if interactive {
return OutputTable
}
@@ -132,18 +159,18 @@ func determineOutputFormat(specified string, interactive bool) OutputFormat {
}
}
// executeSingleQuery executes a single query and outputs the result
// executeSingleQuery executes a single query and outputs the result.
func executeSingleQuery(ctx *SQLContext, query string) bool {
if ctx.outputFormat != OutputTable {
// Suppress banner for non-interactive output
// Suppress banner for non-interactive output.
return executeAndDisplay(ctx, query, false)
}
fmt.Printf("Executing query against %s...\n", *sqlMaster)
fmt.Printf("Executing query against %s...\n", ctx.master)
return executeAndDisplay(ctx, query, true)
}
// executeFileQueries processes SQL queries from a file
// executeFileQueries processes SQL queries from a file.
func executeFileQueries(ctx *SQLContext, filename string) bool {
content, err := os.ReadFile(filename)
if err != nil {
@@ -152,10 +179,10 @@ func executeFileQueries(ctx *SQLContext, filename string) bool {
}
if ctx.outputFormat == OutputTable && ctx.interactive {
fmt.Printf("Executing queries from %s against %s...\n", filename, *sqlMaster)
fmt.Printf("Executing queries from %s against %s...\n", filename, ctx.master)
}
// Split file content into individual queries (robust approach)
// Split file content into individual queries (robust approach).
queries := sqlutil.SplitStatements(string(content))
for i, query := range queries {
@@ -176,11 +203,11 @@ func executeFileQueries(ctx *SQLContext, filename string) bool {
return true
}
// runInteractiveShell starts the enhanced interactive shell with readline support
// runInteractiveShell starts the enhanced interactive shell with readline support.
func runInteractiveShell(ctx *SQLContext) bool {
fmt.Println("SeaweedFS Enhanced SQL Interface")
fmt.Println("Type 'help;' for help, 'exit;' to quit")
fmt.Printf("Connected to master: %s\n", *sqlMaster)
fmt.Printf("Connected to master: %s\n", ctx.master)
if ctx.currentDatabase != "" {
fmt.Printf("Current database: %s\n", ctx.currentDatabase)
}
@@ -188,24 +215,24 @@ func runInteractiveShell(ctx *SQLContext) bool {
fmt.Println("Use up/down arrows for command history")
fmt.Println()
// Initialize liner for readline functionality
// Initialize liner for readline functionality.
line := liner.NewLiner()
defer line.Close()
// Handle Ctrl+C gracefully
// Handle Ctrl+C gracefully.
line.SetCtrlCAborts(true)
grace.OnInterrupt(func() {
line.Close()
})
// Load command history
// Load command history.
historyPath := path.Join(os.TempDir(), "weed-sql-history")
if f, err := os.Open(historyPath); err == nil {
line.ReadHistory(f)
f.Close()
}
// Save history on exit
// Save history on exit.
defer func() {
if f, err := os.Create(historyPath); err == nil {
line.WriteHistory(f)
@@ -216,7 +243,7 @@ func runInteractiveShell(ctx *SQLContext) bool {
var queryBuffer strings.Builder
for {
// Show prompt with current database context
// Show prompt with current database context.
var prompt string
if queryBuffer.Len() == 0 {
if ctx.currentDatabase != "" {
@@ -225,10 +252,10 @@ func runInteractiveShell(ctx *SQLContext) bool {
prompt = "seaweedfs> "
}
} else {
prompt = " -> " // Continuation prompt
prompt = " -> " // Continuation prompt.
}
// Read line with readline support
// Read line with readline support.
input, err := line.Prompt(prompt)
if err != nil {
if err == liner.ErrPromptAborted {
@@ -244,30 +271,30 @@ func runInteractiveShell(ctx *SQLContext) bool {
lineStr := strings.TrimSpace(input)
// Handle empty lines
// Handle empty lines.
if lineStr == "" {
continue
}
// Accumulate lines in query buffer
// Accumulate lines in query buffer.
if queryBuffer.Len() > 0 {
queryBuffer.WriteString(" ")
}
queryBuffer.WriteString(lineStr)
// Check if we have a complete statement (ends with semicolon or special command)
// Check if we have a complete statement (ends with semicolon or special command).
fullQuery := strings.TrimSpace(queryBuffer.String())
isComplete := strings.HasSuffix(lineStr, ";") ||
isSpecialCommand(fullQuery)
if !isComplete {
continue // Continue reading more lines
continue // Continue reading more lines.
}
// Add completed command to history
// Add completed command to history.
line.AppendHistory(fullQuery)
// Handle special commands (with or without semicolon)
// Handle special commands (with or without semicolon).
cleanQuery := strings.TrimSuffix(fullQuery, ";")
cleanQuery = strings.TrimSpace(cleanQuery)
@@ -282,19 +309,19 @@ func runInteractiveShell(ctx *SQLContext) bool {
continue
}
// Handle database switching - use proper SQL parser instead of manual parsing
// Handle database switching - use proper SQL parser instead of manual parsing.
if strings.HasPrefix(strings.ToUpper(cleanQuery), "USE ") {
// Execute USE statement through the SQL engine for proper parsing
// Execute USE statement through the SQL engine for proper parsing.
result, err := ctx.engine.ExecuteSQL(context.Background(), cleanQuery)
if err != nil {
fmt.Printf("Error: %v\n\n", err)
} else if result.Error != nil {
fmt.Printf("Error: %v\n\n", result.Error)
} else {
// Extract the database name from the result message for CLI context
// Extract the database name from the result message for CLI context.
if len(result.Rows) > 0 && len(result.Rows[0]) > 0 {
message := result.Rows[0][0].ToString()
// Extract database name from "Database changed to: dbname"
// Extract database name from "Database changed to: dbname".
if strings.HasPrefix(message, "Database changed to: ") {
ctx.currentDatabase = strings.TrimPrefix(message, "Database changed to: ")
}
@@ -305,7 +332,7 @@ func runInteractiveShell(ctx *SQLContext) bool {
continue
}
// Handle output format switching
// Handle output format switching.
if strings.HasPrefix(strings.ToUpper(cleanQuery), "\\FORMAT ") {
format := strings.TrimSpace(strings.TrimPrefix(strings.ToUpper(cleanQuery), "\\FORMAT "))
switch format {
@@ -325,22 +352,22 @@ func runInteractiveShell(ctx *SQLContext) bool {
continue
}
// Execute SQL query (without semicolon)
// Execute SQL query (without semicolon).
executeAndDisplay(ctx, cleanQuery, true)
// Reset buffer for next query
// Reset buffer for next query.
queryBuffer.Reset()
}
return true
}
// isSpecialCommand checks if a command is a special command that doesn't require semicolon
// isSpecialCommand checks if a command is a special command that doesn't require semicolon.
func isSpecialCommand(query string) bool {
cleanQuery := strings.TrimSuffix(strings.TrimSpace(query), ";")
cleanQuery = strings.ToLower(cleanQuery)
// Special commands that work with or without semicolon
// Special commands that work with or without semicolon.
specialCommands := []string{
"exit", "quit", "\\q", "help",
}
@@ -351,7 +378,7 @@ func isSpecialCommand(query string) bool {
}
}
// Commands that are exactly specific commands (not just prefixes)
// Commands that are exactly specific commands (not just prefixes).
parts := strings.Fields(strings.ToUpper(cleanQuery))
if len(parts) == 0 {
return false
@@ -360,11 +387,11 @@ func isSpecialCommand(query string) bool {
strings.HasPrefix(strings.ToUpper(cleanQuery), "\\FORMAT ")
}
// executeAndDisplay executes a query and displays the result in the specified format
// executeAndDisplay executes a query and displays the result in the specified format.
func executeAndDisplay(ctx *SQLContext, query string, showTiming bool) bool {
startTime := time.Now()
// Execute the query
// Execute the query.
execCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
@@ -397,7 +424,7 @@ func executeAndDisplay(ctx *SQLContext, query string, showTiming bool) bool {
return false
}
// Display results in the specified format
// Display results in the specified format.
switch ctx.outputFormat {
case OutputTable:
displayTableResult(result)
@@ -407,8 +434,8 @@ func executeAndDisplay(ctx *SQLContext, query string, showTiming bool) bool {
displayCSVResult(result)
}
// Show execution time for interactive/table mode
// Only show timing if there are columns or if result is truly empty
// Show execution time for interactive/table mode.
// Only show timing if there are columns or if result is truly empty.
if showTiming && ctx.outputFormat == OutputTable && (len(result.Columns) > 0 || len(result.Rows) == 0) {
elapsed := time.Since(startTime)
fmt.Printf("\n(%d rows in set, %.3f sec)\n\n", len(result.Rows), elapsed.Seconds())
@@ -417,20 +444,20 @@ func executeAndDisplay(ctx *SQLContext, query string, showTiming bool) bool {
return true
}
// displayTableResult formats and displays query results in ASCII table format
// displayTableResult formats and displays query results in ASCII table format.
func displayTableResult(result *engine.QueryResult) {
if len(result.Columns) == 0 {
fmt.Println("Empty result set")
return
}
// Calculate column widths for formatting
// Calculate column widths for formatting.
colWidths := make([]int, len(result.Columns))
for i, col := range result.Columns {
colWidths[i] = len(col)
}
// Check data for wider columns
// Check data for wider columns.
for _, row := range result.Rows {
for i, val := range row {
if i < len(colWidths) {
@@ -442,28 +469,28 @@ func displayTableResult(result *engine.QueryResult) {
}
}
// Print header separator
// Print header separator.
fmt.Print("+")
for _, width := range colWidths {
fmt.Print(strings.Repeat("-", width+2) + "+")
}
fmt.Println()
// Print column headers
// Print column headers.
fmt.Print("|")
for i, col := range result.Columns {
fmt.Printf(" %-*s |", colWidths[i], col)
}
fmt.Println()
// Print separator
// Print separator.
fmt.Print("+")
for _, width := range colWidths {
fmt.Print(strings.Repeat("-", width+2) + "+")
}
fmt.Println()
// Print data rows
// Print data rows.
for _, row := range result.Rows {
fmt.Print("|")
for i, val := range row {
@@ -474,7 +501,7 @@ func displayTableResult(result *engine.QueryResult) {
fmt.Println()
}
// Print bottom separator
// Print bottom separator.
fmt.Print("+")
for _, width := range colWidths {
fmt.Print(strings.Repeat("-", width+2) + "+")
@@ -482,16 +509,16 @@ func displayTableResult(result *engine.QueryResult) {
fmt.Println()
}
// displayJSONResult outputs query results in JSON format
// displayJSONResult outputs query results in JSON format.
func displayJSONResult(result *engine.QueryResult) {
// Convert result to JSON-friendly format
// Convert result to JSON-friendly format.
jsonResult := map[string]interface{}{
"columns": result.Columns,
"rows": make([]map[string]interface{}, len(result.Rows)),
"count": len(result.Rows),
}
// Convert rows to JSON objects
// Convert rows to JSON objects.
for i, row := range result.Rows {
rowObj := make(map[string]interface{})
for j, val := range row {
@@ -502,7 +529,7 @@ func displayJSONResult(result *engine.QueryResult) {
jsonResult["rows"].([]map[string]interface{})[i] = rowObj
}
// Marshal and print JSON
// Marshal and print JSON.
jsonBytes, err := json.MarshalIndent(jsonResult, "", " ")
if err != nil {
fmt.Printf("Error formatting JSON: %v\n", err)
@@ -512,11 +539,11 @@ func displayJSONResult(result *engine.QueryResult) {
fmt.Println(string(jsonBytes))
}
// displayCSVResult outputs query results in CSV format
// displayCSVResult outputs query results in CSV format.
func displayCSVResult(result *engine.QueryResult) {
// Handle execution plan results specially to avoid CSV quoting issues
// Handle execution plan results specially to avoid CSV quoting issues.
if len(result.Columns) == 1 && result.Columns[0] == "Query Execution Plan" {
// For execution plans, output directly without CSV encoding to avoid quotes
// For execution plans, output directly without CSV encoding to avoid quotes.
for _, row := range result.Rows {
if len(row) > 0 {
fmt.Println(row[0].ToString())
@@ -525,17 +552,17 @@ func displayCSVResult(result *engine.QueryResult) {
return
}
// Standard CSV output for regular query results
// Standard CSV output for regular query results.
writer := csv.NewWriter(os.Stdout)
defer writer.Flush()
// Write headers
// Write headers.
if err := writer.Write(result.Columns); err != nil {
fmt.Printf("Error writing CSV headers: %v\n", err)
return
}
// Write data rows
// Write data rows.
for _, row := range result.Rows {
csvRow := make([]string, len(row))
for i, val := range row {
@@ -553,7 +580,7 @@ func showEnhancedHelp() {
METADATA OPERATIONS:
SHOW DATABASES; - List all MQ namespaces
SHOW TABLES; - List all topics in current namespace
SHOW TABLES; - List all topics in current namespace
SHOW TABLES FROM database; - List topics in specific namespace
DESCRIBE table_name; - Show table schema
@@ -581,7 +608,7 @@ SPECIAL COMMANDS:
EXTENDED WHERE OPERATORS:
=, <, >, <=, >= - Comparison operators
!=, <> - Not equal operators
!=, <> - Not equal operators
LIKE 'pattern%' - Pattern matching (% = any chars, _ = single char)
IN (value1, value2, ...) - Multi-value matching
AND, OR - Logical operators
+9 -5
View File
@@ -20,13 +20,17 @@ if [ "$(id -u)" = "0" ]; then
DATA_UID=$(stat -c '%u' /data 2>/dev/null)
DATA_GID=$(stat -c '%g' /data 2>/dev/null)
# Only run chown -R if ownership doesn't match (much faster for subsequent starts)
# Only run chown -R if ownership doesn't already match (avoids expensive
# recursive chown on subsequent starts, and is a no-op on OpenShift when
# fsGroup has already set correct ownership on the PVC).
if [ "$DATA_UID" != "$SEAWEED_UID" ] || [ "$DATA_GID" != "$SEAWEED_GID" ]; then
echo "Fixing /data ownership for seaweed user (uid=$SEAWEED_UID, gid=$SEAWEED_GID)"
if ! chown -R seaweed:seaweed /data; then
echo "Warning: Failed to change ownership of /data. This may cause permission errors." >&2
echo "If /data is read-only or has mount issues, the application may fail to start." >&2
fi
fi
# Use su-exec to drop privileges and run as seaweed user
exec su-exec seaweed "$0" "$@"
@@ -54,7 +58,7 @@ isArgPassed() {
case "$1" in
'master')
ARGS="-mdir=/data -volumePreallocate -volumeSizeLimitMB=1024"
ARGS="-mdir=/data -volumeSizeLimitMB=1024"
shift
exec /usr/bin/weed -logtostderr=true master $ARGS $@
;;
@@ -69,9 +73,9 @@ case "$1" in
;;
'server')
ARGS="-dir=/data -volume.max=0 -master.volumePreallocate -master.volumeSizeLimitMB=1024"
ARGS="-dir=/data -volume.max=0 -master.volumeSizeLimitMB=1024"
if isArgPassed "-volume.max" "$@"; then
ARGS="-dir=/data -master.volumePreallocate -master.volumeSizeLimitMB=1024"
ARGS="-dir=/data -master.volumeSizeLimitMB=1024"
fi
shift
exec /usr/bin/weed -logtostderr=true server $ARGS $@
+3 -3
View File
@@ -33,7 +33,7 @@ isArgPassed() {
case "$1" in
'master')
ARGS="-mdir=/data -volumePreallocate -volumeSizeLimitMB=1024"
ARGS="-mdir=/data -volumeSizeLimitMB=1024"
shift
exec /usr/bin/weed -logtostderr=true master $ARGS "$@"
;;
@@ -48,9 +48,9 @@ case "$1" in
;;
'server')
ARGS="-dir=/data -volume.max=0 -master.volumePreallocate -master.volumeSizeLimitMB=1024"
ARGS="-dir=/data -volume.max=0 -master.volumeSizeLimitMB=1024"
if isArgPassed "-volume.max" "$@"; then
ARGS="-dir=/data -master.volumePreallocate -master.volumeSizeLimitMB=1024"
ARGS="-dir=/data -master.volumeSizeLimitMB=1024"
fi
shift
exec /usr/bin/weed -logtostderr=true server $ARGS "$@"
+9
View File
@@ -42,3 +42,12 @@ services:
- master
- volume
- filer
admin:
image: chrislusf/seaweedfs:dev # use a remote dev image
ports:
- 23646:23646
command: 'admin -master=master:9333'
depends_on:
- master
- volume
- filer
+25 -30
View File
@@ -46,7 +46,7 @@ require (
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/json-iterator/go v1.1.12
github.com/karlseguin/ccache/v2 v2.0.8
github.com/klauspost/compress v1.18.3
github.com/klauspost/compress v1.18.4
github.com/klauspost/reedsolomon v1.13.0
github.com/kurin/blazer v0.5.3
github.com/linxGnu/grocksdb v1.10.7
@@ -94,7 +94,7 @@ require (
gocloud.dev/pubsub/rabbitpubsub v0.44.0
golang.org/x/crypto v0.48.0
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546
golang.org/x/image v0.35.0
golang.org/x/image v0.36.0
golang.org/x/net v0.49.0
golang.org/x/oauth2 v0.34.0
golang.org/x/sys v0.41.0
@@ -109,7 +109,7 @@ require (
modernc.org/b v1.0.0 // indirect
modernc.org/mathutil v1.7.1
modernc.org/memory v1.11.0 // indirect
modernc.org/sqlite v1.44.3
modernc.org/sqlite v1.46.1
modernc.org/strutil v1.2.1
)
@@ -130,9 +130,7 @@ require (
github.com/aws/aws-sdk-go-v2/service/s3 v1.95.0
github.com/cognusion/imaging v1.0.2
github.com/fluent/fluent-logger-golang v1.10.1
github.com/getsentry/sentry-go v0.40.0
github.com/gin-contrib/sessions v1.0.4
github.com/gin-gonic/gin v1.11.0
github.com/getsentry/sentry-go v0.42.0
github.com/go-ldap/ldap/v3 v3.4.12
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/flatbuffers/go v0.0.0-20230108230133-3b8644d32c50
@@ -142,15 +140,15 @@ require (
github.com/jhump/protoreflect v1.18.0
github.com/lib/pq v1.11.1
github.com/linkedin/goavro/v2 v2.14.1
github.com/mattn/go-sqlite3 v1.14.33
github.com/mattn/go-sqlite3 v1.14.34
github.com/minio/crc64nvme v1.1.1
github.com/orcaman/concurrent-map/v2 v2.0.1
github.com/parquet-go/parquet-go v0.26.4
github.com/parquet-go/parquet-go v0.28.0
github.com/pkg/sftp v1.13.10
github.com/rabbitmq/amqp091-go v1.10.0
github.com/rclone/rclone v1.72.1
github.com/rclone/rclone v1.73.1
github.com/rdleal/intervalst v1.5.0
github.com/redis/go-redis/v9 v9.17.2
github.com/redis/go-redis/v9 v9.18.0
github.com/schollz/progressbar/v3 v3.19.0
github.com/seaweedfs/go-fuse/v2 v2.9.1
github.com/shirou/gopsutil/v4 v4.26.1
@@ -180,6 +178,7 @@ require (
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest/to v0.4.1 // indirect
github.com/FilenCloudDienste/filen-sdk-go v0.0.37 // indirect
github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0 // indirect
github.com/anchore/go-lzo v0.1.0 // indirect
github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
@@ -212,6 +211,8 @@ require (
github.com/docker/docker v28.5.0+incompatible // indirect
github.com/docker/go-connections v0.6.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dromara/dongle v1.0.1 // indirect
github.com/gin-gonic/gin v1.11.0 // indirect
github.com/go-asn1-ber/asn1-ber v1.5.8-0.20250403174932-29230038a667 // indirect
github.com/go-git/go-billy/v5 v5.6.2 // indirect
github.com/goccy/go-yaml v1.18.0 // indirect
@@ -226,6 +227,7 @@ require (
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect
github.com/hashicorp/go-sockaddr v1.0.7 // indirect
github.com/hashicorp/hcl v1.0.1-vault-7 // indirect
github.com/internxt/rclone-adapter v0.0.0-20260213125353-6f59c89fcb7c // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
@@ -257,8 +259,9 @@ require (
github.com/pierrre/geohash v1.0.0 // indirect
github.com/pquerna/otp v1.5.0 // indirect
github.com/pterm/pterm v0.12.81 // indirect
github.com/quic-go/qpack v0.6.0 // indirect
github.com/quic-go/quic-go v0.57.0 // indirect
github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11 // indirect
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18 // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/ryanuber/go-glob v1.0.0 // indirect
github.com/sasha-s/go-deadlock v0.3.1 // indirect
@@ -267,8 +270,9 @@ require (
github.com/substrait-io/substrait v0.69.0 // indirect
github.com/substrait-io/substrait-go/v4 v4.4.0 // indirect
github.com/substrait-io/substrait-protobuf/go v0.71.0 // indirect
github.com/twpayne/go-geom v1.4.1 // indirect
github.com/twpayne/go-kml v1.5.2 // indirect
github.com/twpayne/go-geom v1.6.1 // indirect
github.com/twpayne/go-kml/v3 v3.2.1 // indirect
github.com/tyler-smith/go-bip39 v1.1.0 // indirect
github.com/ulikunitz/xz v0.5.15 // indirect
github.com/wk8/go-ordered-map/v2 v2.1.8 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
@@ -279,6 +283,7 @@ require (
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.37.0 // indirect
go.opentelemetry.io/otel/exporters/zipkin v1.36.0 // indirect
go.opentelemetry.io/proto/otlp v1.7.0 // indirect
go.uber.org/mock v0.5.2 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/mod v0.32.0 // indirect
@@ -293,11 +298,11 @@ require (
cloud.google.com/go/compute/metadata v0.9.0 // indirect
cloud.google.com/go/iam v1.5.3 // indirect
cloud.google.com/go/monitoring v1.24.2 // indirect
filippo.io/edwards25519 v1.1.0 // indirect
filippo.io/edwards25519 v1.1.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.3 // indirect
github.com/Azure/go-ntlmssp v0.1.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 // indirect
@@ -341,15 +346,12 @@ require (
github.com/bradenaw/juniper v0.15.3 // indirect
github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 // indirect
github.com/buengese/sgzip v0.1.1 // indirect
github.com/bytedance/sonic v1.14.0 // indirect
github.com/bytedance/sonic/loader v0.3.0 // indirect
github.com/calebcase/tmpfile v1.0.3 // indirect
github.com/chilts/sid v0.0.0-20190607042430-660e94789ec9 // indirect
github.com/cloudflare/circl v1.6.1 // indirect
github.com/cloudflare/circl v1.6.3 // indirect
github.com/cloudinary/cloudinary-go/v2 v2.13.0 // indirect
github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc // indirect
github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc // indirect
github.com/cloudwego/base64x v0.1.6 // indirect
github.com/cncf/xds/go v0.0.0-20251022180443-0feb69152e9f // indirect
github.com/colinmarc/hdfs/v2 v2.4.0 // indirect
github.com/creasty/defaults v1.8.0 // indirect
@@ -369,8 +371,7 @@ require (
github.com/flynn/noise v1.1.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.11 // indirect
github.com/geoffgarside/ber v1.2.0 // indirect
github.com/gin-contrib/sse v1.1.0 // indirect
github.com/go-chi/chi/v5 v5.2.3 // indirect
github.com/go-chi/chi/v5 v5.2.5 // indirect
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 // indirect
github.com/go-jose/go-jose/v4 v4.1.3 // indirect
github.com/go-logr/logr v1.4.3 // indirect
@@ -389,10 +390,9 @@ require (
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.7 // indirect
github.com/gorilla/context v1.1.2 // indirect
github.com/gorilla/schema v1.4.1 // indirect
github.com/gorilla/securecookie v1.1.2 // indirect
github.com/gorilla/sessions v1.4.0 // indirect
github.com/gorilla/sessions v1.4.0
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.3 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
@@ -402,8 +402,6 @@ require (
github.com/hashicorp/go-msgpack/v2 v2.1.2 // indirect
github.com/hashicorp/go-retryablehttp v0.7.8 // indirect
github.com/hashicorp/golang-lru v0.6.0 // indirect
github.com/henrybear327/Proton-API-Bridge v1.0.0 // indirect
github.com/henrybear327/go-proton-api v1.0.0 // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
github.com/jcmturner/goidentity/v6 v6.0.1 // indirect
@@ -445,7 +443,7 @@ require (
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/pengsrc/go-shared v0.2.1-0.20190131101655-1999055a4a14 // indirect
github.com/philhofer/fwd v1.2.0 // indirect
github.com/pierrec/lz4/v4 v4.1.22
github.com/pierrec/lz4/v4 v4.1.25
github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c // indirect
github.com/pingcap/failpoint v0.0.0-20220801062533-2eaa32854a6c // indirect
github.com/pingcap/kvproto v0.0.0-20230403051650-e166ae588106 // indirect
@@ -461,7 +459,7 @@ require (
github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect
github.com/sagikazarmark/locafero v0.11.0 // indirect
github.com/samber/lo v1.52.0 // indirect
github.com/seaweedfs/cockroachdb-parser v0.0.0-20251021184156-909763b17138
github.com/seaweedfs/cockroachdb-parser v0.0.0-20260225204133-2f342c5ea564
github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966 // indirect
github.com/sony/gobreaker v1.0.0 // indirect
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
@@ -476,9 +474,7 @@ require (
github.com/tinylib/msgp v1.5.0 // indirect
github.com/tklauser/go-sysconf v0.3.16 // indirect
github.com/tklauser/numcpus v0.11.0 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/twmb/murmur3 v1.1.8 // indirect
github.com/ugorji/go/codec v1.3.0 // indirect
github.com/unknwon/goconfig v1.0.0 // indirect
github.com/vmihailenco/msgpack/v5 v5.4.1 // indirect
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
@@ -504,7 +500,6 @@ require (
go.opentelemetry.io/otel/trace v1.38.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.1 // indirect
golang.org/x/arch v0.20.0 // indirect
golang.org/x/term v0.40.0 // indirect
golang.org/x/time v0.14.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20251124214823-79d6a2a48846 // indirect
+52 -75
View File
@@ -547,8 +547,8 @@ cloud.google.com/go/workflows v1.10.0/go.mod h1:fZ8LmRmZQWacon9UCX1r/g/DfAXx5VcP
dario.cat/mergo v1.0.2 h1:85+piFYR1tMbRrLcDwR18y4UKJ3aH1Tbzi24VRW1TK8=
dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA=
filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
filippo.io/edwards25519 v1.1.1 h1:YpjwWWlNmGIDyXOn8zLzqiD+9TyIlPhGFG96P39uBpw=
filippo.io/edwards25519 v1.1.1/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
gioui.org v0.0.0-20210308172011-57750fc8a0a6/go.mod h1:RSH6KIUZ0p2xy5zHDxgAM4zumjgTw83q2ge/PI+yyw8=
git.sr.ht/~sbinet/gg v0.3.1/go.mod h1:KGYtlADtqsqANL9ueOFkWymvzUvLMQllU5Ixo+8v3pc=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6 h1:He8afgbRMd7mFxO99hRNu+6tazq8nFF9lIwo9JFroBk=
@@ -569,11 +569,10 @@ github.com/Azure/azure-sdk-for-go/sdk/keyvault/internal v0.7.1 h1:FbH3BbSb4bvGlu
github.com/Azure/azure-sdk-for-go/sdk/keyvault/internal v0.7.1/go.mod h1:9V2j0jn9jDEkCkv8w/bKTNppX/d0FVA1ud77xCIP4KA=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1 h1:/Zt+cDPnpC3OVDm/JKLOs7M2DKmLRIIp3XIx9pHHiig=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1/go.mod h1:Ng3urmn6dYe8gnbCMoHHVl5APYz2txho3koEkV2o2HA=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3 h1:ZJJNFaQ86GVKQ9ehwqyAFE6pIfyicpuJ8IkVaPBc6/4=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3/go.mod h1:URuDvhmATVKqHBH9/0nOiNKk0+YcwfQ3WkK5PqHKxc8=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4 h1:jWQK1GI+LeGGUKBADtcH2rRqPxYB1Ljwms5gFA2LqrM=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4/go.mod h1:8mwH4klAm9DUgR2EEHyEEAQlRDvLPyg5fQry3y+cDew=
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.3 h1:sxgSqOB9CDToiaVFpxuvb5wGgGqWa3lCShcm5o0n3bE=
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.3/go.mod h1:XdED8i399lEVblYHTZM8eXaP07gv4Z58IL6ueMlVlrg=
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
github.com/Azure/go-autorest v14.2.0+incompatible h1:V5VMDjClD3GiElqLWO7mz2MxNAK/vTfRHdAubSIPRgs=
@@ -590,12 +589,14 @@ github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/Codefor/geohash v0.0.0-20140723084247-1b41c28e3a9d h1:iG9B49Q218F/XxXNRM7k/vWf7MKmLIS8AcJV9cGN4nA=
github.com/Codefor/geohash v0.0.0-20140723084247-1b41c28e3a9d/go.mod h1:RVnhzAX71far8Kc3TQeA0k/dcaEKUnTDSOyet/JCmGI=
github.com/DATA-DOG/go-sqlmock v1.3.2 h1:2L2f5t3kKnCLxnClDD/PrDfExFFa1wjESgxHG/B1ibo=
github.com/DATA-DOG/go-sqlmock v1.3.2/go.mod h1:f/Ixk793poVmq4qj/V1dPUg2JEAKC73Q5eFN3EC/SaM=
github.com/DATA-DOG/go-sqlmock v1.5.2 h1:OcvFkGmslmlZibjAjaHm3L//6LiuBgolP7OputlJIzU=
github.com/DATA-DOG/go-sqlmock v1.5.2/go.mod h1:88MAG/4G7SMwSE3CeA0ZKzrT5CiOU3OJ+JlNzwDqpNU=
github.com/DataDog/datadog-go v3.2.0+incompatible/go.mod h1:LButxg5PwREeZtORoXG3tL4fMGNddJ+vMq1mwgfaqoQ=
github.com/DataDog/zstd v1.5.2/go.mod h1:g4AWEaM3yOg3HYfnJ3YIawPnVdXJh9QME85blwSAmyw=
github.com/DefangLabs/secret-detector v0.0.0-20250403165618-22662109213e h1:rd4bOvKmDIx0WeTv9Qz+hghsgyjikFiPrseXHlKepO0=
github.com/DefangLabs/secret-detector v0.0.0-20250403165618-22662109213e/go.mod h1:blbwPQh4DTlCZEfk1BLU4oMIhLda2U+A840Uag9DsZw=
github.com/FilenCloudDienste/filen-sdk-go v0.0.37 h1:W8S9TrAyZ4//3PXsU6+Bi+fe/6uIL986GyS7PVzIDL4=
github.com/FilenCloudDienste/filen-sdk-go v0.0.37/go.mod h1:0cBhKXQg49XbKZZfk5TCDa3sVLP+xMxZTWL+7KY0XR0=
github.com/Files-com/files-sdk-go/v3 v3.2.264 h1:lMHTplAYI9FtmCo/QOcpRxmPA5REVAct1r2riQmDQKw=
github.com/Files-com/files-sdk-go/v3 v3.2.264/go.mod h1:wGqkOzRu/ClJibvDgcfuJNAqI2nLhe8g91tPlDKRCdE=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.30.0 h1:sBEjpZlNHzK1voKq9695PJSX2o5NEXl7/OL3coiIY0c=
@@ -620,19 +621,13 @@ github.com/MarvinJWendt/testza v0.3.0/go.mod h1:eFcL4I0idjtIx8P9C6KkAuLgATNKpX4/
github.com/MarvinJWendt/testza v0.4.2/go.mod h1:mSdhXiKH8sg/gQehJ63bINcCKp7RtYewEjXsvsVUPbE=
github.com/MarvinJWendt/testza v0.5.2 h1:53KDo64C1z/h/d/stCYCPY69bt/OSwjq5KpFNwi+zB4=
github.com/MarvinJWendt/testza v0.5.2/go.mod h1:xu53QFE5sCdjtMCKk8YMQ2MnymimEctc4n3EjyIYvEY=
github.com/Masterminds/goutils v1.1.0/go.mod h1:8cTjp+g8YejhMuvIA5y2vz3BpJxksy863GQaJW2MFNU=
github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww=
github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y=
github.com/Masterminds/semver/v3 v3.2.1 h1:RN9w6+7QoMeJVGyfmbcgs28Br8cvmnucEXnY0rYXWg0=
github.com/Masterminds/semver/v3 v3.2.1/go.mod h1:qvl/7zhW3nngYb5+80sSMF+FG2BjYrf8m9wsX0PNOMQ=
github.com/Masterminds/sprig v2.22.0+incompatible/go.mod h1:y6hNFY5UBTIWBxnzTeuNhlNS5hqE0NB0E6fgfo2Br3o=
github.com/Max-Sum/base32768 v0.0.0-20230304063302-18e6ce5945fd h1:nzE1YQBdx1bq9IlZinHa+HVffy+NmVRoKr+wHN8fpLE=
github.com/Max-Sum/base32768 v0.0.0-20230304063302-18e6ce5945fd/go.mod h1:C8yoIfvESpM3GD07OCHU7fqI7lhwyZ2Td1rbNbTAhnc=
github.com/Microsoft/go-winio v0.4.14/go.mod h1:qXqCSQ3Xa7+6tgxaGTIe4Kpcdsi+P8jBhyzoq1bpyYA=
github.com/Microsoft/go-winio v0.5.2/go.mod h1:WpS1mjBmmwHBEWmogvA2mj8546UReBk4v8QkMxJ6pZY=
github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5/go.mod h1:lmUJ/7eu/Q8D7ML55dXQrVaamCz2vxCfdQBasLZfHKk=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/ProtonMail/bcrypt v0.0.0-20210511135022-227b4adcab57/go.mod h1:HecWFHognK8GfRDGnFQbW/LiV7A3MX3gZVs45vk5h8I=
github.com/ProtonMail/bcrypt v0.0.0-20211005172633-e235017c1baf h1:yc9daCCYUefEs69zUkSzubzjBbL+cmOXgnmt9Fyd9ug=
@@ -673,6 +668,10 @@ github.com/ajstarks/deck v0.0.0-20200831202436-30c9fc6549a9/go.mod h1:JynElWSGnm
github.com/ajstarks/deck/generate v0.0.0-20210309230005-c3f852c02e19/go.mod h1:T13YZdzov6OU0A1+RfKZiZN9ca6VeKdBdyDV+BY97Tk=
github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af/go.mod h1:K08gAheRH3/J6wwsYMMT4xOr94bZjxIelGM0+d/wbFw=
github.com/ajstarks/svgo v0.0.0-20211024235047-1546f124cd8b/go.mod h1:1KcenG0jGWcpt8ov532z81sp/kMMUG485J2InIOyADM=
github.com/alecthomas/assert/v2 v2.10.0 h1:jjRCHsj6hBJhkmhznrCzoNpbA3zqy0fYiUcYZP/GkPY=
github.com/alecthomas/assert/v2 v2.10.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k=
github.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc=
github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
@@ -806,7 +805,6 @@ github.com/bytedance/sonic/loader v0.3.0 h1:dskwH8edlzNMctoruo8FPTJDF3vLtDT0sXZw
github.com/bytedance/sonic/loader v0.3.0/go.mod h1:N8A3vUdtUebEY2/VQC0MyhYeKUFosQU6FxH2JmUe6VI=
github.com/calebcase/tmpfile v1.0.3 h1:BZrOWZ79gJqQ3XbAQlihYZf/YCV0H4KPIdM5K5oMpJo=
github.com/calebcase/tmpfile v1.0.3/go.mod h1:UAUc01aHeC+pudPagY/lWvt2qS9ZO5Zzof6/tIUzqeI=
github.com/cenkalti/backoff/v3 v3.0.0/go.mod h1:cIeZDE3IrqwwJl6VUwCN6trj1oXrTS4rc0ij+ULvLYs=
github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
@@ -834,8 +832,8 @@ github.com/clipperhouse/stringish v0.1.1/go.mod h1:v/WhFtE1q0ovMta2+m+UbpZ+2/HEX
github.com/clipperhouse/uax29/v2 v2.3.0 h1:SNdx9DVUqMoBuBoW3iLOj4FQv3dN5mDtuqwuhIGpJy4=
github.com/clipperhouse/uax29/v2 v2.3.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g=
github.com/cloudflare/circl v1.1.0/go.mod h1:prBCrKB9DV4poKZY1l9zBXg2QJY7mvgRvtMxxK7fi4I=
github.com/cloudflare/circl v1.6.1 h1:zqIqSPIndyBh1bjLVVDHMPpVKqp8Su/V+6MeDzzQBQ0=
github.com/cloudflare/circl v1.6.1/go.mod h1:uddAzsPgqdMAYatqJ0lsjX1oECcQLIlRpzZh3pJrofs=
github.com/cloudflare/circl v1.6.3 h1:9GPOhQGF9MCYUeXyMYlqTR6a5gTrgR/fBLXvUgtVcg8=
github.com/cloudflare/circl v1.6.3/go.mod h1:2eXP6Qfat4O/Yhh8BznvKnJ+uzEoTQ6jVKJRn81BiS4=
github.com/cloudinary/cloudinary-go/v2 v2.13.0 h1:ugiQwb7DwpWQnete2AZkTh94MonZKmxD7hDGy1qTzDs=
github.com/cloudinary/cloudinary-go/v2 v2.13.0/go.mod h1:ireC4gqVetsjVhYlwjUJwKTbZuWjEIynbR9zQTlqsvo=
github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc h1:t8YjNUCt1DimB4HCIXBztwWMhgxr5yG5/YaRl9Afdfg=
@@ -882,7 +880,6 @@ github.com/containerd/containerd/api v1.8.0 h1:hVTNJKR8fMc/2Tiw60ZRijntNMd1U+JVM
github.com/containerd/containerd/api v1.8.0/go.mod h1:dFv4lt6S20wTu/hMcP4350RL87qPWLVa/OHOwmmdnYc=
github.com/containerd/containerd/v2 v2.0.5 h1:2vg/TjUXnaohAxiHnthQg8K06L9I4gdYEMcOLiMc8BQ=
github.com/containerd/containerd/v2 v2.0.5/go.mod h1:Qqo0UN43i2fX1FLkrSTCg6zcHNfjN7gEnx3NPRZI+N0=
github.com/containerd/continuity v0.0.0-20190827140505-75bee3e2ccb6/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
github.com/containerd/continuity v0.4.5 h1:ZRoN1sXq9u7V6QoHMcVWGhOwDFqZ4B9i5H6un1Wh0x4=
github.com/containerd/continuity v0.4.5/go.mod h1:/lNJvtJKUQStBzpVQ1+rasXO1LAWtUQssk28EZvJ3nE=
github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
@@ -953,15 +950,15 @@ github.com/docker/docker-credential-helpers v0.8.2 h1:bX3YxiGzFP5sOXWc3bTPEXdEaZ
github.com/docker/docker-credential-helpers v0.8.2/go.mod h1:P3ci7E3lwkZg6XiHdRKft1KckHiO9a2rNtyFbZ/ry9M=
github.com/docker/go v1.5.1-1.0.20160303222718-d30aec9fd63c h1:lzqkGL9b3znc+ZUgi7FlLnqjQhcXxkNM/quxIjBVMD0=
github.com/docker/go v1.5.1-1.0.20160303222718-d30aec9fd63c/go.mod h1:CADgU4DSXK5QUlFslkQu2yW2TKzFZcXq/leZfM0UH5Q=
github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec=
github.com/docker/go-connections v0.6.0 h1:LlMG9azAe1TqfR7sO+NJttz1gy6KO7VJBh+pMmjSD94=
github.com/docker/go-connections v0.6.0/go.mod h1:AahvXYshr6JgfUJGdDCs2b5EZG/vmaMAntpSFH5BFKE=
github.com/docker/go-metrics v0.0.1 h1:AgB/0SvBxihN0X8OR4SjsblXkbMvalQ8cjmtKQ2rQV8=
github.com/docker/go-metrics v0.0.1/go.mod h1:cG1hvH2utMXtqgqqYE9plW6lDxS3/5ayHzueweSI3Vw=
github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
github.com/dromara/dongle v1.0.1 h1:si/7UP/EXxnFVZok1cNos70GiMGxInAYMilHQFP5dJs=
github.com/dromara/dongle v1.0.1/go.mod h1:ebFhTaDgxaDIKppycENTWlBsxz8mWCPWOLnsEgDpMv4=
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5 h1:FT+t0UEDykcor4y3dMVKXIiWJETBpRgERYTGlmMd7HU=
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5/go.mod h1:rSS3kM9XMzSQ6pw91Qgd6yB5jdt70N4OdtrAf74As5M=
github.com/dsnet/try v0.0.3 h1:ptR59SsrcFUYbT/FhAbKTV6iLkeD6O18qfIWRml2fqI=
@@ -989,6 +986,7 @@ github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff h1:4N8wnS3f1hNHS
github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff/go.mod h1:HMJKR5wlh/ziNp+sHEDV2ltblO4JD2+IdDOWtGcQBTM=
github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g=
github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
github.com/emmansun/gmsm v0.15.5/go.mod h1:2m4jygryohSWkaSduFErgCwQKab5BNjURoFrn2DNwyU=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
@@ -1053,19 +1051,17 @@ github.com/gabriel-vasile/mimetype v1.4.11 h1:AQvxbp830wPhHTqc1u7nzoLT+ZFxGY7emj
github.com/gabriel-vasile/mimetype v1.4.11/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
github.com/geoffgarside/ber v1.2.0 h1:/loowoRcs/MWLYmGX9QtIAbA+V/FrnVLsMMPhwiRm64=
github.com/geoffgarside/ber v1.2.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNecGT85ZCc=
github.com/getsentry/sentry-go v0.40.0 h1:VTJMN9zbTvqDqPwheRVLcp0qcUcM+8eFivvGocAaSbo=
github.com/getsentry/sentry-go v0.40.0/go.mod h1:eRXCoh3uvmjQLY6qu63BjUZnaBu5L5WhMV1RwYO8W5s=
github.com/getsentry/sentry-go v0.42.0 h1:eeFMACuZTbUQf90RE8dE4tXeSe4CZyfvR1MBL7RLEt8=
github.com/getsentry/sentry-go v0.42.0/go.mod h1:eRXCoh3uvmjQLY6qu63BjUZnaBu5L5WhMV1RwYO8W5s=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gin-contrib/sessions v1.0.4 h1:ha6CNdpYiTOK/hTp05miJLbpTSNfOnFg5Jm2kbcqy8U=
github.com/gin-contrib/sessions v1.0.4/go.mod h1:ccmkrb2z6iU2osiAHZG3x3J4suJK+OU27oqzlWOqQgs=
github.com/gin-contrib/sse v1.1.0 h1:n0w2GMuUpWDVp7qSpvze6fAu9iRxJY4Hmj6AmBOU05w=
github.com/gin-contrib/sse v1.1.0/go.mod h1:hxRZ5gVpWMT7Z0B0gSNYqqsSCNIJMjzvm6fqCz9vjwM=
github.com/gin-gonic/gin v1.11.0 h1:OW/6PLjyusp2PPXtyxKHU0RbX6I/l28FTdDlae5ueWk=
github.com/gin-gonic/gin v1.11.0/go.mod h1:+iq/FyxlGzII0KHiBGjuNn4UNENUlKbGlNmc+W50Dls=
github.com/go-asn1-ber/asn1-ber v1.5.8-0.20250403174932-29230038a667 h1:BP4M0CvQ4S3TGls2FvczZtj5Re/2ZzkV9VwqPHH/3Bo=
github.com/go-asn1-ber/asn1-ber v1.5.8-0.20250403174932-29230038a667/go.mod h1:hEBeB/ic+5LoWskz+yKT7vGhhPYkProFKoKdwZRWMe0=
github.com/go-chi/chi/v5 v5.2.3 h1:WQIt9uxdsAbgIYgid+BpYc+liqQZGMHRaUwp0JUcvdE=
github.com/go-chi/chi/v5 v5.2.3/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 h1:JnrjqG5iR07/8k7NqrLNilRsl3s1EPRQEGvbPyOce68=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348/go.mod h1:Czxo/d1g948LtrALAZdL04TL/HnkopquAjxYUuI02bo=
github.com/go-errors/errors v1.5.1 h1:ZwEMSLRCapFLflTpT7NKaAc7ukJ8ZPEjzlxt8rPN8bk=
@@ -1274,7 +1270,6 @@ github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0=
github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM=
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4=
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ=
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
@@ -1309,8 +1304,6 @@ github.com/gookit/color v1.5.4 h1:FZmqs7XOyGgCAxmWyPslpiok1k05wmY3SJTytgvYFs0=
github.com/gookit/color v1.5.4/go.mod h1:pZJOeOS8DM43rXbp4AZo1n9zCU2qjpcRko0b6/QJi9w=
github.com/gopherjs/gopherjs v1.17.2 h1:fQnZVsXk8uxXIStYb0N4bGk7jeyTalG/wsZjQ25dO0g=
github.com/gopherjs/gopherjs v1.17.2/go.mod h1:pRRIvn/QzFLrKfvEz3qUuEhtE/zLCWfreZ6J5gM2i+k=
github.com/gorilla/context v1.1.2 h1:WRkNAv2uoa03QNIc1A6u4O7DAGMUVoopZhkiXWA2V1o=
github.com/gorilla/context v1.1.2/go.mod h1:KDPwT9i/MeWHiLl90fuTgrt4/wPcv75vFAZLaOOcbxM=
github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
github.com/gorilla/schema v1.4.1 h1:jUg5hUjCSDZpNGLuXQOgIWGdlgrIdYvgQ0wZtdK1M3E=
@@ -1390,18 +1383,12 @@ github.com/hashicorp/raft-boltdb/v2 v2.3.1 h1:ackhdCNPKblmOhjEU9+4lHSJYFkJd6Jqyv
github.com/hashicorp/raft-boltdb/v2 v2.3.1/go.mod h1:n4S+g43dXF1tqDT+yzcXHhXM6y7MrlUd3TTwGRcUvQE=
github.com/hashicorp/vault/api v1.22.0 h1:+HYFquE35/B74fHoIeXlZIP2YADVboaPjaSicHEZiH0=
github.com/hashicorp/vault/api v1.22.0/go.mod h1:IUZA2cDvr4Ok3+NtK2Oq/r+lJeXkeCrHRmqdyWfpmGM=
github.com/henrybear327/Proton-API-Bridge v1.0.0 h1:gjKAaWfKu++77WsZTHg6FUyPC5W0LTKWQciUm8PMZb0=
github.com/henrybear327/Proton-API-Bridge v1.0.0/go.mod h1:gunH16hf6U74W2b9CGDaWRadiLICsoJ6KRkSt53zLts=
github.com/henrybear327/go-proton-api v1.0.0 h1:zYi/IbjLwFAW7ltCeqXneUGJey0TN//Xo851a/BgLXw=
github.com/henrybear327/go-proton-api v1.0.0/go.mod h1:w63MZuzufKcIZ93pwRgiOtxMXYafI8H74D77AxytOBc=
github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/huandu/xstrings v1.3.0/go.mod h1:y5/lhBue+AyNmUVz9RLU9xbLR0o4KIIExikq4ovT0aE=
github.com/iancoleman/strcase v0.2.0/go.mod h1:iwCmte+B7n89clKwxIoIXy/HfoL7AsD47ZCWhYzw7ho=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/imdario/mergo v0.3.9/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
github.com/imdario/mergo v0.3.16 h1:wwQJbIsHYGMUyLSPrEq1CT16AhnhNJQ51+4fdHUnCl4=
github.com/imdario/mergo v0.3.16/go.mod h1:WBLT9ZmE3lPoWsEzCh9LPo3TiwVN+ZKEjmz+hD27ysY=
github.com/in-toto/in-toto-golang v0.5.0 h1:hb8bgwr0M2hGdDsLjkJ3ZqJ8JFLL/tgYdAxF/XEFBbY=
@@ -1410,6 +1397,8 @@ github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/inhies/go-bytesize v0.0.0-20220417184213-4913239db9cf h1:FtEj8sfIcaaBfAKrE1Cwb61YDtYq9JxChK1c7AKce7s=
github.com/inhies/go-bytesize v0.0.0-20220417184213-4913239db9cf/go.mod h1:yrqSXGoD/4EKfF26AOGzscPOgTTJcyAwM2rpixWT+t4=
github.com/internxt/rclone-adapter v0.0.0-20260213125353-6f59c89fcb7c h1:r+KtxPyrhsYeNbsfeqTfEM8xRdwgV6LuNhLZxpXecb4=
github.com/internxt/rclone-adapter v0.0.0-20260213125353-6f59c89fcb7c/go.mod h1:vdPya4AIcDjvng4ViaAzqjegJf0VHYpYHQguFx5xBp0=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
@@ -1487,8 +1476,8 @@ github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+o
github.com/klauspost/asmfmt v1.3.2 h1:4Ri7ox3EwapiOjCki+hw14RyKk201CN4rzyCJRFLpK4=
github.com/klauspost/asmfmt v1.3.2/go.mod h1:AG8TuvYojzulgDAMCnYn50l/5QV3Bs/tp6j0HLHbNSE=
github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
github.com/klauspost/compress v1.18.3 h1:9PJRvfbmTabkOX8moIpXPbMMbYN60bWImDDU7L+/6zw=
github.com/klauspost/compress v1.18.3/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/klauspost/compress v1.18.4 h1:RPhnKRAQ4Fh8zU2FY/6ZFDwTVTxgJ/EMydqSTzE9a2c=
github.com/klauspost/compress v1.18.4/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.0.10/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
github.com/klauspost/cpuid/v2 v2.0.12/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
@@ -1497,7 +1486,6 @@ github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu
github.com/klauspost/reedsolomon v1.13.0 h1:E0Cmgf2kMuhZTj6eefnvpKC4/Q4jhCi9YIjcZjK4arc=
github.com/klauspost/reedsolomon v1.13.0/go.mod h1:ggJT9lc71Vu+cSOPBlxGvBN6TfAS77qB4fp8vJ05NSA=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/koofr/go-httpclient v0.0.0-20240520111329-e20f8f203988 h1:CjEMN21Xkr9+zwPmZPaJJw+apzVbjGL5uK/6g9Q2jGU=
github.com/koofr/go-httpclient v0.0.0-20240520111329-e20f8f203988/go.mod h1:/agobYum3uo/8V6yPVnq+R82pyVGCeuWW5arT4Txn8A=
@@ -1524,8 +1512,6 @@ github.com/lanrat/extsort v1.4.2 h1:akbLIdo4PhNZtvjpaWnbXtGMmLtnGzXplkzfgl+XTTY=
github.com/lanrat/extsort v1.4.2/go.mod h1:hceP6kxKPKebjN1RVrDBXMXXECbaI41Y94tt6MDazc4=
github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
github.com/lib/pq v0.0.0-20180327071824-d34b9ff171c2/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.8.0/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/lib/pq v1.11.1 h1:wuChtj2hfsGmmx3nf1m7xC2XpK6OtelS2shMY+bGMtI=
github.com/lib/pq v1.11.1/go.mod h1:/p+8NSbOcwzAEI7wiMXFlgydTwcgTr3OSKMsD2BitpA=
github.com/linkedin/goavro/v2 v2.14.1 h1:/8VjDpd38PRsy02JS0jflAu7JZPfJcGTwqWgMkFS2iI=
@@ -1564,8 +1550,8 @@ github.com/mattn/go-runewidth v0.0.19/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhg
github.com/mattn/go-shellwords v1.0.12 h1:M2zGm7EW6UQJvDeQxo4T51eKPurbeFbe8WtebGE2xrk=
github.com/mattn/go-shellwords v1.0.12/go.mod h1:EZzvwXDESEeg03EKmM+RmDnNOPKG4lLtQsUlTZDWQ8Y=
github.com/mattn/go-sqlite3 v1.14.14/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU=
github.com/mattn/go-sqlite3 v1.14.33 h1:A5blZ5ulQo2AtayQ9/limgHEkFreKj1Dv226a1K73s0=
github.com/mattn/go-sqlite3 v1.14.33/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
github.com/mattn/go-sqlite3 v1.14.34 h1:3NtcvcUnFBPsuRcno8pUtupspG/GM+9nZ88zgJcp6Zk=
github.com/mattn/go-sqlite3 v1.14.34/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b h1:j7+1HpAFS1zy5+Q4qx1fWh90gTKwiN4QCGoY9TWyyO4=
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE=
@@ -1581,14 +1567,12 @@ github.com/minio/highwayhash v1.0.2 h1:Aak5U0nElisjDCfPSG79Tgzkn2gl66NxOMspRrKnA
github.com/minio/highwayhash v1.0.2/go.mod h1:BQskDq+xkJ12lmlUUi7U0M5Swg3EWR+dLTk+kldvVxY=
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db h1:62I3jR2EmQ4l5rM/4FEfDWcRD+abF5XlKShorW5LRoQ=
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db/go.mod h1:l0dey0ia/Uv7NcFFVbCLtqEBQbrT4OCwCSKTEv6enCw=
github.com/mitchellh/copystructure v1.0.0/go.mod h1:SNtv71yrdKgLRyLFxmLdkAbkKEFWgYaq1OVrnRcwhnw=
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
github.com/mitchellh/hashstructure/v2 v2.0.2 h1:vGKWl0YJqUNxE8d+h8f6NJLcCJrgbhC4NcD46KavDd4=
github.com/mitchellh/hashstructure/v2 v2.0.2/go.mod h1:MG3aRVU/N29oo/V/IhBX8GR/zz4kQkprJgF2EVszyDE=
github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4 h1:BpfhmLKZf+SjVanKKhCgf3bg+511DmU9eDQTen7LLbY=
github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/mitchellh/reflectwalk v1.0.0/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
github.com/mmcloughlin/geohash v0.9.0 h1:FihR004p/aE1Sju6gcVq5OLDqGcMnpBY+8moBqIsVOs=
github.com/mmcloughlin/geohash v0.9.0/go.mod h1:oNZxQo5yWJh0eMQEP/8hwQuVx9Z9tjwFUqcTB1SmG0c=
github.com/moby/buildkit v0.20.1 h1:sT0ZXhhNo5rVbMcYfgttma3TdUHfO5JjFA0UAL8p9fY=
@@ -1667,13 +1651,10 @@ github.com/onsi/ginkgo/v2 v2.23.3/go.mod h1:zXTP6xIp3U8aVuXN8ENK9IXRaTjFnpVB9mGm
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/onsi/gomega v1.37.0 h1:CdEG8g0S133B4OswTDC/5XPSzE1OeP29QOioj2PID2Y=
github.com/onsi/gomega v1.37.0/go.mod h1:8D9+Txp43QWKhM24yyOBEdpkzN8FvJyAwecBgsU4KU0=
github.com/opencontainers/go-digest v1.0.0-rc1/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQYUd2OVphdqWwCJHrFt2s=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.0.1/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
github.com/opencontainers/runc v1.0.0-rc9/go.mod h1:qT5XzbpPznkRYVz/mWwUaVBUv2rmF59PVA73FjuZG0U=
github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
github.com/opentracing/opentracing-go v1.2.0 h1:uEJPy/1a5RIPAJ0Ov+OIO8OxWu77jEv+1B0VhjKrZUs=
github.com/opentracing/opentracing-go v1.2.0/go.mod h1:GxEUsuufX4nBwe+T+Wl9TAgYrxe9dPLANfrWvHYVTgc=
@@ -1683,15 +1664,14 @@ github.com/oracle/oci-go-sdk/v65 v65.104.0 h1:l9awEvzWvxmYhy/97A0hZ87pa7BncYXmcO
github.com/oracle/oci-go-sdk/v65 v65.104.0/go.mod h1:oB8jFGVc/7/zJ+DbleE8MzGHjhs2ioCz5stRTdZdIcY=
github.com/orcaman/concurrent-map/v2 v2.0.1 h1:jOJ5Pg2w1oeB6PeDurIYf6k9PQ+aTITr/6lP/L/zp6c=
github.com/orcaman/concurrent-map/v2 v2.0.1/go.mod h1:9Eq3TG2oBe5FirmYWQfYO5iH1q0Jv47PLaNK++uCdOM=
github.com/ory/dockertest/v3 v3.6.0/go.mod h1:4ZOpj8qBUmh8fcBSVzkH2bws2s91JdGvHUqan4GHEuQ=
github.com/panjf2000/ants/v2 v2.11.3 h1:AfI0ngBoXJmYOpDh9m516vjqoUu2sLrIVgppI9TZVpg=
github.com/panjf2000/ants/v2 v2.11.3/go.mod h1:8u92CYMUc6gyvTIw8Ru7Mt7+/ESnJahz5EVtqfrilek=
github.com/parquet-go/bitpack v1.0.0 h1:AUqzlKzPPXf2bCdjfj4sTeacrUwsT7NlcYDMUQxPcQA=
github.com/parquet-go/bitpack v1.0.0/go.mod h1:XnVk9TH+O40eOOmvpAVZ7K2ocQFrQwysLMnc6M/8lgs=
github.com/parquet-go/jsonlite v1.0.0 h1:87QNdi56wOfsE5bdgas0vRzHPxfJgzrXGml1zZdd7VU=
github.com/parquet-go/jsonlite v1.0.0/go.mod h1:nDjpkpL4EOtqs6NQugUsi0Rleq9sW/OtC1NnZEnxzF0=
github.com/parquet-go/parquet-go v0.26.4 h1:zJ3l8ef5WJZE2m63pKwyEJ2BhyDlgS0PfOEhuCQQU2A=
github.com/parquet-go/parquet-go v0.26.4/go.mod h1:h9GcSt41Knf5qXI1tp1TfR8bDBUtvdUMzSKe26aZcHk=
github.com/parquet-go/parquet-go v0.28.0 h1:ECyksyv8T2pOrlLsN7aWJIoQakyk/HtxQ2lchgS4els=
github.com/parquet-go/parquet-go v0.28.0/go.mod h1:navtkAYr2LGoJVp141oXPlO/sxLvaOe3la2JEoD8+rg=
github.com/pascaldekloe/goe v0.1.0 h1:cBOtyMzM9HTpWjXfbbunk26uA6nG3a8n06Wieeh0MwY=
github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
@@ -1714,8 +1694,8 @@ github.com/phpdave11/gofpdf v1.4.2/go.mod h1:zpO6xFn9yxo3YLyMvW8HcKWVdbNqgIfOOp2
github.com/phpdave11/gofpdi v1.0.12/go.mod h1:vBmVV0Do6hSBHC8uKUQ71JGW+ZGQq74llk/7bXwjDoI=
github.com/phpdave11/gofpdi v1.0.13/go.mod h1:vBmVV0Do6hSBHC8uKUQ71JGW+ZGQq74llk/7bXwjDoI=
github.com/pierrec/lz4/v4 v4.1.15/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pierrec/lz4/v4 v4.1.22 h1:cKFw6uJDK+/gfw5BcDL0JL5aBsAFdsIT18eRtLj7VIU=
github.com/pierrec/lz4/v4 v4.1.22/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pierrec/lz4/v4 v4.1.25 h1:kocOqRffaIbU5djlIBr7Wh+cx82C0vtFb0fOurZHqD0=
github.com/pierrec/lz4/v4 v4.1.25/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
github.com/pierrre/compare v1.0.2 h1:k4IUsHgh+dbcAOIWCfxVa/7G6STjADH2qmhomv+1quc=
github.com/pierrre/compare v1.0.2/go.mod h1:8UvyRHH+9HS8Pczdd2z5x/wvv67krDwVxoOndaIIDVU=
github.com/pierrre/geohash v1.0.0 h1:f/zfjdV4rVofTCz1FhP07T+EMQAvcMM2ioGZVt+zqjI=
@@ -1808,14 +1788,18 @@ github.com/r3labs/sse v0.0.0-20210224172625-26fe804710bc h1:zAsgcP8MhzAbhMnB1QQ2
github.com/r3labs/sse v0.0.0-20210224172625-26fe804710bc/go.mod h1:S8xSOnV3CgpNrWd0GQ/OoQfMtlg2uPRSuTzcSGrzwK8=
github.com/rabbitmq/amqp091-go v1.10.0 h1:STpn5XsHlHGcecLmMFCtg7mqq0RnD+zFr4uzukfVhBw=
github.com/rabbitmq/amqp091-go v1.10.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
github.com/rclone/rclone v1.72.1 h1:Cc/NshKd3/TP3CC0cx9Jg9nTLG8YQ8yLYMTm6Z/LdHk=
github.com/rclone/rclone v1.72.1/go.mod h1:QjmSgz98CjQZZJhROIeYHYjpN5kN7rTA+jtChj/+3Do=
github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11 h1:4MI2alxM/Ye2gIRBlYf28JGWTipZ4Zz7yAziPKrttjs=
github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11/go.mod h1:3HLX7dwZgvB7nt+Yl/xdzVPcargQ1yBmJEUg3n+jMKM=
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18 h1:Lc+d3ISfQaMJKWZOE7z4ZSY4RVmdzbn1B0IM8xN18qM=
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18/go.mod h1:LB2kCEaZMzNn3ocdz+qYfxXmuLxxN0ka62KJd2x53Bc=
github.com/rclone/rclone v1.73.1 h1:FCGZUd7sXO8+nIpbuXGPkd+f23o/ZACefrHtRGDAiDM=
github.com/rclone/rclone v1.73.1/go.mod h1:T9HeSXv/SI9qJFkYLqsjY/5ljIFWJTVmTrG7H/4JrPY=
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 h1:N/ElC8H3+5XpJzTSTfLsJV/mx9Q9g7kxmchpfZyxgzM=
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/rdleal/intervalst v1.5.0 h1:SEB9bCFz5IqD1yhfH1Wv8IBnY/JQxDplwkxHjT6hamU=
github.com/rdleal/intervalst v1.5.0/go.mod h1:xO89Z6BC+LQDH+IPQQw/OESt5UADgFD41tYMUINGpxQ=
github.com/redis/go-redis/v9 v9.17.2 h1:P2EGsA4qVIM3Pp+aPocCJ7DguDHhqrXNhVcEp4ViluI=
github.com/redis/go-redis/v9 v9.17.2/go.mod h1:u410H11HMLoB+TP67dz8rL9s6QW2j76l0//kSOd3370=
github.com/redis/go-redis/v9 v9.18.0 h1:pMkxYPkEbMPwRdenAzUNyFNrDgHx9U+DrBabWNfSRQs=
github.com/redis/go-redis/v9 v9.18.0/go.mod h1:k3ufPphLU5YXwNTUcCRXGxUoF1fqxnhFQmscfkCoDA0=
github.com/redis/rueidis v1.0.69 h1:WlUefRhuDekji5LsD387ys3UCJtSFeBVf0e5yI0B8b4=
github.com/redis/rueidis v1.0.69/go.mod h1:Lkhr2QTgcoYBhxARU7kJRO8SyVlgUuEkcJO1Y8MCluA=
github.com/redis/rueidis/rueidiscompat v1.0.69 h1:IWVYY9lXdjNO3do2VpJT7aDFi8zbCUuQxZB6E2Grahs=
@@ -1855,8 +1839,8 @@ github.com/sasha-s/go-deadlock v0.3.1 h1:sqv7fDNShgjcaxkO0JNcOAlr8B9+cV5Ey/OB71e
github.com/sasha-s/go-deadlock v0.3.1/go.mod h1:F73l+cr82YSh10GxyRI6qZiCgK64VaZjwesgfQ1/iLM=
github.com/schollz/progressbar/v3 v3.19.0 h1:Ea18xuIRQXLAUidVDox3AbwfUhD0/1IvohyTutOIFoc=
github.com/schollz/progressbar/v3 v3.19.0/go.mod h1:IsO3lpbaGuzh8zIMzgY3+J8l4C8GjO0Y9S69eFvNsec=
github.com/seaweedfs/cockroachdb-parser v0.0.0-20251021184156-909763b17138 h1:bX1vBF7GQjPeFQsCAZ8gCQGS/nJQnekL7gZ4Qg/pF4E=
github.com/seaweedfs/cockroachdb-parser v0.0.0-20251021184156-909763b17138/go.mod h1:JSKCh6uCHBz91lQYFYHCyTrSVIPge4SUFVn28iwMNB0=
github.com/seaweedfs/cockroachdb-parser v0.0.0-20260225204133-2f342c5ea564 h1:TgxPraf1NmF6XTcUG53ULpLQrKvhtUJxQ3hyekxSDNQ=
github.com/seaweedfs/cockroachdb-parser v0.0.0-20260225204133-2f342c5ea564/go.mod h1:JSKCh6uCHBz91lQYFYHCyTrSVIPge4SUFVn28iwMNB0=
github.com/seaweedfs/go-fuse/v2 v2.9.1 h1:gnKmfrKreCRGJmekGz5WMnNZqXEf9s9+V2hdWQdvx88=
github.com/seaweedfs/go-fuse/v2 v2.9.1/go.mod h1:zABdmWEa6A0bwaBeEOBUeUkGIZlxUhcdv+V1Dcc/U/I=
github.com/seaweedfs/goexif v1.0.3 h1:ve/OjI7dxPW8X9YQsv3JuVMaxEyF9Rvfd04ouL+Bz30=
@@ -1876,7 +1860,6 @@ github.com/shibumi/go-pathspec v1.3.0/go.mod h1:Xutfslp817l2I1cZvgcfeMQJG5QnU2lh
github.com/shirou/gopsutil/v4 v4.26.1 h1:TOkEyriIXk2HX9d4isZJtbjXbEjf5qyKPAzbzY0JWSo=
github.com/shirou/gopsutil/v4 v4.26.1/go.mod h1:medLI9/UNAb0dOI9Q3/7yWSqKkj00u+1tgY8nvv41pc=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
github.com/sirupsen/logrus v1.5.0/go.mod h1:+F7Ogzej0PZc/94MaYx/nvG9jOFMD2osvC3s+Squfpo=
github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88=
@@ -1907,7 +1890,6 @@ github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/cobra v1.10.1 h1:lJeBwCfmrnXthfAupyUTzJ/J4Nc1RsHC/mSRU2dll/s=
github.com/spf13/cobra v1.10.1/go.mod h1:7SmJGaTHFVBY0jW4NXGluQoLvhqFQM+6XSKD+P4XaB0=
github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
@@ -2005,12 +1987,12 @@ github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS
github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08=
github.com/twmb/murmur3 v1.1.8 h1:8Yt9taO/WN3l08xErzjeschgZU2QSrwm1kclYq+0aRg=
github.com/twmb/murmur3 v1.1.8/go.mod h1:Qq/R7NUyOfr65zD+6Q5IHKsJLwP7exErjN6lyyq3OSQ=
github.com/twpayne/go-geom v1.4.1 h1:LeivFqaGBRfyg0XJJ9pkudcptwhSSrYN9KZUW6HcgdA=
github.com/twpayne/go-geom v1.4.1/go.mod h1:k/zktXdL+qnA6OgKsdEGUTA17jbQ2ZPTUa3CCySuGpE=
github.com/twpayne/go-kml v1.5.2 h1:rFMw2/EwgkVssGS2MT6YfWSPZz6BgcJkLxQ53jnE8rQ=
github.com/twpayne/go-kml v1.5.2/go.mod h1:kz8jAiIz6FIdU2Zjce9qGlVtgFYES9vt7BTPBHf5jl4=
github.com/twpayne/go-polyline v1.0.0/go.mod h1:ICh24bcLYBX8CknfvNPKqoTbe+eg+MX1NPyJmSBo7pU=
github.com/twpayne/go-waypoint v0.0.0-20200706203930-b263a7f6e4e8/go.mod h1:qj5pHncxKhu9gxtZEYWypA/z097sxhFlbTyOyt9gcnU=
github.com/twpayne/go-geom v1.6.1 h1:iLE+Opv0Ihm/ABIcvQFGIiFBXd76oBIar9drAwHFhR4=
github.com/twpayne/go-geom v1.6.1/go.mod h1:Kr+Nly6BswFsKM5sd31YaoWS5PeDDH2NftJTK7Gd028=
github.com/twpayne/go-kml/v3 v3.2.1 h1:xkTIJ7KMnHGKpHGf30e4XS3UT8o/5jD62hmdGJPf7Io=
github.com/twpayne/go-kml/v3 v3.2.1/go.mod h1:lPWoJR3nQAdePBy3SrnniLdBLVQX0hlxrcziCx9XgT0=
github.com/tyler-smith/go-bip39 v1.1.0 h1:5eUemwrMargf3BSLRRCalXT93Ns6pQJIjYQN2nyfOP8=
github.com/tyler-smith/go-bip39 v1.1.0/go.mod h1:gUYDtqQw1JS3ZJ8UWVcGTGqqr6YIN3CWg+kkNaLt55U=
github.com/tylertreat/BoomFilters v0.0.0-20210315201527-1a82519a3e43 h1:QEePdg0ty2r0t1+qwfZmQ4OOl/MB2UXIeJSpIZv56lg=
github.com/tylertreat/BoomFilters v0.0.0-20210315201527-1a82519a3e43/go.mod h1:OYRfF6eb5wY9VRFkXJH8FFBi3plw2v+giaIu7P054pM=
github.com/ugorji/go/codec v1.3.0 h1:Qd2W2sQawAfG8XSvzwhBeoGq71zXOC/Q1E9y/wUcsUA=
@@ -2214,13 +2196,13 @@ golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8U
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200323165209-0ec3e9974c59/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.4.0/go.mod h1:3quD/ATkf6oY+rnes5c3ExXTbLc8mueNue5/DoinL80=
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU=
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
@@ -2260,8 +2242,8 @@ golang.org/x/image v0.0.0-20210607152325-775e3b0c77b9/go.mod h1:023OzeP/+EPmXeap
golang.org/x/image v0.0.0-20210628002857-a66eb6448b8d/go.mod h1:023OzeP/+EPmXeapQh35lcL3II3LrY8Ic+EFFKVhULM=
golang.org/x/image v0.0.0-20211028202545-6944b10bf410/go.mod h1:023OzeP/+EPmXeapQh35lcL3II3LrY8Ic+EFFKVhULM=
golang.org/x/image v0.0.0-20220302094943-723b81ca9867/go.mod h1:023OzeP/+EPmXeapQh35lcL3II3LrY8Ic+EFFKVhULM=
golang.org/x/image v0.35.0 h1:LKjiHdgMtO8z7Fh18nGY6KDcoEtVfsgLDPeLyguqb7I=
golang.org/x/image v0.35.0/go.mod h1:MwPLTVgvxSASsxdLzKrl8BRFuyqMyGhLwmC+TO1Sybk=
golang.org/x/image v0.36.0 h1:Iknbfm1afbgtwPTmHnS2gTM/6PPZfH+z2EFuOkSbqwc=
golang.org/x/image v0.36.0/go.mod h1:YsWD2TyyGKiIX1kZlu9QfKIsQ4nAAK9bdgdrIsE7xy4=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@@ -2312,7 +2294,6 @@ golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLL
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190628185345-da137c7871d7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191003171128-d98b1b443823/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
@@ -2442,7 +2423,6 @@ golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200121082415-34d275377bf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -2589,7 +2569,6 @@ golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBn
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190624222133-a101b041ded4/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
@@ -2960,7 +2939,6 @@ gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.3/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.7/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
@@ -2970,7 +2948,6 @@ gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C
gopkg.in/yaml.v3 v3.0.0/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gotest.tools/v3 v3.0.2/go.mod h1:3SzNCllyD9/Y+b5r9JIKQ474KzkZyqLqEfYqMsX94Bk=
gotest.tools/v3 v3.5.2 h1:7koQfIKdy+I8UTetycgUqXWSDwpgv193Ka+qRsmBY8Q=
gotest.tools/v3 v3.5.2/go.mod h1:LtdLGcnqToBH83WByAAi/wiwSFCArdFIUV/xxN4pcjA=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
@@ -3047,8 +3024,8 @@ modernc.org/opt v0.1.4/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
modernc.org/sqlite v1.18.1/go.mod h1:6ho+Gow7oX5V+OiOQ6Tr4xeqbx13UZ6t+Fw9IRUG4d4=
modernc.org/sqlite v1.44.3 h1:+39JvV/HWMcYslAwRxHb8067w+2zowvFOUrOWIy9PjY=
modernc.org/sqlite v1.44.3/go.mod h1:CzbrU2lSB1DKUusvwGz7rqEKIq+NUd8GWuBBZDs9/nA=
modernc.org/sqlite v1.46.1 h1:eFJ2ShBLIEnUWlLy12raN0Z1plqmFX9Qe3rjQTKt6sU=
modernc.org/sqlite v1.46.1/go.mod h1:CzbrU2lSB1DKUusvwGz7rqEKIq+NUd8GWuBBZDs9/nA=
modernc.org/strutil v1.1.0/go.mod h1:lstksw84oURvj9y3tn8lGvRxyRC1S2+g5uuIzNfIOBs=
modernc.org/strutil v1.1.1/go.mod h1:DE+MQQ/hjKBZS2zNInV5hhcipt5rLPWkmpbGeW5mmdw=
modernc.org/strutil v1.1.3/go.mod h1:MEHNA7PdEnEwLvspRMtWTNnp2nnyvMfkimT1NKNAGbw=
+2 -2
View File
@@ -1,6 +1,6 @@
apiVersion: v1
description: SeaweedFS
name: seaweedfs
appVersion: "4.12"
appVersion: "4.13"
# Dev note: Trigger a helm chart release by `git tag -a helm-<version>`
version: 4.0.412
version: 4.0.413
+30 -12
View File
@@ -212,8 +212,9 @@ To enable workers, add the following to your values.yaml:
worker:
enabled: true
replicas: 2 # Scale based on workload
capabilities: "vacuum,balance,erasure_coding" # Tasks this worker can handle
maxConcurrent: 3 # Maximum concurrent tasks per worker
jobType: "vacuum,volume_balance,erasure_coding" # Job types this worker can handle
maxDetect: 1 # Maximum concurrent detection requests
maxExecute: 4 # Maximum concurrent execution jobs per worker
# Working directory for task execution
# Default: "/tmp/seaweedfs-worker"
@@ -248,14 +249,14 @@ worker:
memory: "2Gi"
```
### Worker Capabilities
### Worker Job Types
Workers can be configured with different capabilities:
Workers can be configured with different job types:
- **vacuum**: Reclaim deleted file space
- **balance**: Balance volumes across volume servers
- **volume_balance**: Balance volumes across volume servers
- **erasure_coding**: Handle erasure coding operations
You can configure workers with all capabilities or create specialized worker pools with specific capabilities.
You can configure workers with all job types or create specialized worker pools with specific job types.
### Worker Deployment Strategy
@@ -264,11 +265,11 @@ For production deployments, consider:
1. **Multiple Workers**: Deploy 2+ worker replicas for high availability
2. **Resource Allocation**: Workers need sufficient CPU/memory for maintenance tasks
3. **Storage**: Workers need temporary storage for vacuum and balance operations (size depends on volume size)
4. **Specialized Workers**: Create separate worker deployments for different capabilities if needed
4. **Specialized Workers**: Create separate worker deployments for different job types if needed
Example specialized worker configuration:
For specialized worker pools, deploy separate Helm releases with different capabilities:
For specialized worker pools, deploy separate Helm releases with different job types:
**values-worker-vacuum.yaml** (for vacuum operations):
```yaml
@@ -287,8 +288,8 @@ admin:
worker:
enabled: true
replicas: 2
capabilities: "vacuum"
maxConcurrent: 2
jobType: "vacuum"
maxExecute: 2
# REQUIRED: Point to the admin service of your main SeaweedFS release
# Replace <namespace> with the namespace where your main seaweedfs is deployed
# Example: If deploying in namespace "production":
@@ -313,8 +314,8 @@ admin:
worker:
enabled: true
replicas: 1
capabilities: "balance"
maxConcurrent: 1
jobType: "volume_balance"
maxExecute: 1
# REQUIRED: Point to the admin service of your main SeaweedFS release
# Replace <namespace> with the namespace where your main seaweedfs is deployed
# Example: If deploying in namespace "production":
@@ -323,6 +324,7 @@ worker:
```
Deploy the specialized workers as separate releases:
### Specialized Worker Deployment
```bash
# Deploy vacuum workers
helm install seaweedfs-worker-vacuum seaweedfs/seaweedfs -f values-worker-vacuum.yaml
@@ -331,6 +333,22 @@ helm install seaweedfs-worker-vacuum seaweedfs/seaweedfs -f values-worker-vacuum
helm install seaweedfs-worker-balance seaweedfs/seaweedfs -f values-worker-balance.yaml
```
## OpenShift Support
SeaweedFS can be deployed on OpenShift or any cluster enforcing the Kubernetes "restricted" Pod Security Standard. By default, OpenShift blocks containers that run as root or use `hostPath` volumes.
To deploy on OpenShift, use the provided `openshift-values.yaml` which overrides the default configuration to:
1. Use `PersistentVolumeClaims` instead of `hostPath`.
2. Enable `runAsNonRoot` and omit hardcoded UIDs to allow OpenShift to assign valid UIDs automatically.
3. Apply appropriate `seccompProfile` and drop capabilities.
Usage:
```bash
helm install seaweedfs seaweedfs/seaweedfs \
-n seaweedfs --create-namespace \
-f openshift-values.yaml
```
## Enterprise
For enterprise users, please visit [seaweedfs.com](https://seaweedfs.com) for the SeaweedFS Enterprise Edition,
+116
View File
@@ -0,0 +1,116 @@
# openshift-values.yaml
#
# Example overrides for deploying SeaweedFS on OpenShift (or any cluster
# enforcing the Kubernetes "restricted" Pod Security Standard).
#
# OpenShift's default "restricted" SCC blocks containers that:
# - Run as UID 0 (root)
# - Request privilege escalation
# - Use hostPath volumes
# - Omit a seccompProfile
#
# These overrides satisfy all four requirements by:
# 1. Replacing hostPath volumes with PersistentVolumeClaims (or emptyDir for logs)
# 2. Enabling runAsNonRoot: true. By omitting runAsUser, OpenShift will
# automatically assign a valid UID from the namespace's allocated range.
# 3. Dropping all Linux capabilities and setting allowPrivilegeEscalation: false
# 4. Enabling RuntimeDefault seccompProfile
#
# Usage:
# helm install seaweedfs seaweedfs/seaweedfs \
# -n seaweedfs --create-namespace \
# -f openshift-values.yaml
#
# Adjust storageClass and sizes to match your cluster's available StorageClasses.
# On OpenShift you can discover them with: oc get storageclass
master:
data:
type: "persistentVolumeClaim"
size: "10Gi"
storageClass: "" # leave empty to use the cluster default StorageClass
logs:
type: "emptyDir" # avoids hostPath; use persistentVolumeClaim if you need log persistence
podSecurityContext:
enabled: true
# On OpenShift, we omit runAsUser/runAsGroup/fsGroup to let the admission
# controller assign them automatically based on the namespace's SCC.
runAsNonRoot: true
containerSecurityContext:
enabled: true
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
volume:
dataDirs:
- name: data1
type: "persistentVolumeClaim"
size: "100Gi"
storageClass: "" # leave empty to use the cluster default StorageClass
maxVolumes: 0
logs:
type: "emptyDir"
podSecurityContext:
enabled: true
# On OpenShift, we omit runAsUser/runAsGroup/fsGroup to let the admission
# controller assign them automatically based on the namespace's SCC.
runAsNonRoot: true
containerSecurityContext:
enabled: true
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
filer:
data:
type: "persistentVolumeClaim"
size: "25Gi"
storageClass: "" # leave empty to use the cluster default StorageClass
logs:
type: "emptyDir"
podSecurityContext:
enabled: true
# On OpenShift, we omit runAsUser/runAsGroup/fsGroup to let the admission
# controller assign them automatically based on the namespace's SCC.
runAsNonRoot: true
containerSecurityContext:
enabled: true
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
# S3 gateway (if enabled)
s3:
podSecurityContext:
enabled: true
# On OpenShift, we omit runAsUser/runAsGroup/fsGroup to let the admission
# controller assign them automatically based on the namespace's SCC.
runAsNonRoot: true
containerSecurityContext:
enabled: true
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
@@ -6,6 +6,10 @@ metadata:
name: {{ .Values.cosi.bucketClassName }}
driverName: {{ .Values.cosi.driverName }}
deletionPolicy: Delete
{{- with .Values.cosi.bucketClassParameters }}
parameters:
{{- toYaml . | nindent 2 }}
{{- end }}
---
kind: BucketAccessClass
apiVersion: objectstorage.k8s.io/v1alpha1
@@ -10,10 +10,21 @@
{{- if and .Values.s3.reuseLegacySecret $existingSecret }}
{{- $reuse = true }}
{{- end }}
{{- $access_key_admin := include "getOrGeneratePassword" (dict "namespace" .Release.Namespace "secretName" $secretName "key" "admin_access_key_id" "length" 20 "existingSecret" (ternary $existingSecret nil $reuse)) -}}
{{- $secret_key_admin := include "getOrGeneratePassword" (dict "namespace" .Release.Namespace "secretName" $secretName "key" "admin_secret_access_key" "length" 40 "existingSecret" (ternary $existingSecret nil $reuse)) -}}
{{- $access_key_read := include "getOrGeneratePassword" (dict "namespace" .Release.Namespace "secretName" $secretName "key" "read_access_key_id" "length" 20 "existingSecret" (ternary $existingSecret nil $reuse)) -}}
{{- $secret_key_read := include "getOrGeneratePassword" (dict "namespace" .Release.Namespace "secretName" $secretName "key" "read_secret_access_key" "length" 40 "existingSecret" (ternary $existingSecret nil $reuse)) -}}
{{- $creds := .Values.s3.credentials | default dict -}}
{{- $adminCreds := $creds.admin | default dict -}}
{{- $access_key_admin := $adminCreds.accessKey -}}
{{- $secret_key_admin := $adminCreds.secretKey -}}
{{- if not (and $access_key_admin $secret_key_admin) -}}
{{- $access_key_admin = include "getOrGeneratePassword" (dict "namespace" .Release.Namespace "secretName" $secretName "key" "admin_access_key_id" "length" 20 "existingSecret" (ternary $existingSecret nil $reuse)) -}}
{{- $secret_key_admin = include "getOrGeneratePassword" (dict "namespace" .Release.Namespace "secretName" $secretName "key" "admin_secret_access_key" "length" 40 "existingSecret" (ternary $existingSecret nil $reuse)) -}}
{{- end -}}
{{- $readCreds := $creds.read | default dict -}}
{{- $access_key_read := $readCreds.accessKey -}}
{{- $secret_key_read := $readCreds.secretKey -}}
{{- if not (and $access_key_read $secret_key_read) -}}
{{- $access_key_read = include "getOrGeneratePassword" (dict "namespace" .Release.Namespace "secretName" $secretName "key" "read_access_key_id" "length" 20 "existingSecret" (ternary $existingSecret nil $reuse)) -}}
{{- $secret_key_read = include "getOrGeneratePassword" (dict "namespace" .Release.Namespace "secretName" $secretName "key" "read_secret_access_key" "length" 40 "existingSecret" (ternary $existingSecret nil $reuse)) -}}
{{- end -}}
apiVersion: v1
kind: Secret
type: Opaque
@@ -43,4 +54,4 @@ stringData:
s3_auditLogConfig.json: |
{{ toJson .Values.s3.auditLogConfig | nindent 4 }}
{{- end }}
{{- end }}
{{- end }}
@@ -6,6 +6,7 @@ metadata:
namespace: {{ .Release.Namespace }}
labels:
app.kubernetes.io/name: {{ template "seaweedfs.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: sftp
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
@@ -135,7 +135,9 @@ Inject extra environment vars in the format key:value, if populated
{{- $repositoryName := default .Values.image.repository .Values.global.repository | toString -}}
{{- $name := .Values.global.imageName | toString -}}
{{- $tag := default .Chart.AppVersion .Values.image.tag | toString -}}
{{- if $repositoryName -}}
{{- if .Values.image.repository -}}
{{- $name = $repositoryName -}}
{{- else if $repositoryName -}}
{{- $name = printf "%s/%s" (trimSuffix "/" $repositoryName) (base $name) -}}
{{- end -}}
{{- if $registryName -}}
@@ -3,6 +3,15 @@
{{- $s3Enabled := false }}
{{- $enableAuth := false }}
{{- $existingConfigSecret := "" }}
{{- $bucketsFolder := "/buckets" }}
{{- $bucketEnvVars := merge (dict) (.Values.global.extraEnvironmentVars | default dict) }}
{{- if .Values.allInOne.enabled }}
{{- $bucketEnvVars = merge (.Values.allInOne.extraEnvironmentVars | default dict) $bucketEnvVars }}
{{- else }}
{{- $bucketEnvVars = merge (.Values.filer.extraEnvironmentVars | default dict) $bucketEnvVars }}
{{- end }}
{{- $bucketsFolder = default $bucketsFolder (get $bucketEnvVars "WEED_FILER_BUCKETS_FOLDER") }}
{{- $bucketsFolder = trimSuffix "/" $bucketsFolder }}
{{- /* Check allInOne mode first */}}
{{- if .Values.allInOne.enabled }}
@@ -108,15 +117,52 @@ spec:
wait_for_service "http://$WEED_CLUSTER_SW_MASTER{{ .Values.master.readinessProbe.httpGet.path }}"
wait_for_service "http://$WEED_CLUSTER_SW_FILER{{ .Values.filer.readinessProbe.httpGet.path }}"
{{- end }}
set -o pipefail
{{- range $createBuckets }}
if /bin/echo "s3.bucket.list" | /usr/bin/weed shell | awk '{print $1}' | grep -Fxq "{{ .name }}"; then
echo "Bucket '{{ .name }}' already exists, skipping creation."
{{- $bucketName := .name }}
{{- $bucketLock := or .lock .objectLock .withLock }}
bucket_list=$(/bin/echo 's3.bucket.list' | /usr/bin/weed shell) || { echo "Error listing s3 buckets"; exit 1; }
if echo "$bucket_list" | awk '{print $1}' | grep -Fxq "{{ $bucketName }}"; then
echo "Bucket '{{ $bucketName }}' already exists, skipping creation."
else
echo "Creating bucket '{{ .name }}'..."
/bin/echo "s3.bucket.create --name {{ .name }}" | /usr/bin/weed shell
echo "Creating bucket '{{ $bucketName }}'..."
/bin/echo 's3.bucket.create --name {{ $bucketName }}{{- if $bucketLock }} --withLock{{- end }}' | /usr/bin/weed shell
fi
{{- end }}
{{- range $createBuckets }}
{{- $bucketLock := or .lock .objectLock .withLock }}
{{- if $bucketLock }}
/bin/echo 's3.bucket.lock -name {{ .name }} -enable' | /usr/bin/weed shell
{{- end }}
{{- end }}
{{- range $createBuckets }}
{{- $bucketVersioning := "" }}
{{- if kindIs "bool" .versioning }}
{{- if .versioning }}
{{- $bucketVersioning = "Enabled" }}
{{- end }}
{{- else if kindIs "string" .versioning }}
{{- $versioningLower := lower .versioning }}
{{- if eq $versioningLower "enabled" "enable" "true" }}
{{- $bucketVersioning = "Enabled" }}
{{- else if eq $versioningLower "suspended" "disable" "false" }}
{{- $bucketVersioning = "Suspended" }}
{{- else if or (eq .versioning "Enabled") (eq .versioning "Suspended") }}
{{- $bucketVersioning = .versioning }}
{{- else }}
{{- fail (printf "Invalid versioning value for bucket %s: %s. Must be 'Enabled' or 'Suspended'" .name .versioning) }}
{{- end }}
{{- end }}
{{- if $bucketVersioning }}
/bin/echo 's3.bucket.versioning -name {{ .name }} -status {{ $bucketVersioning }}' | /usr/bin/weed shell
{{- end }}
{{- end }}
{{- range $createBuckets }}
{{- if .ttl }}
/bin/echo 'fs.configure -locationPrefix={{ $bucketsFolder }}/{{ .name }}/ -ttl={{ .ttl }} -apply' | /usr/bin/weed shell
{{- end }}
{{- end }}
{{- range $createBuckets }}
{{- if .anonymousRead }}
/bin/echo \
"s3.configure --user anonymous \
@@ -3,6 +3,10 @@ kind: ServiceAccount
metadata:
name: {{ include "seaweedfs.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
{{- with .Values.global.serviceAccountAnnotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
labels:
app.kubernetes.io/name: {{ template "seaweedfs.name" . }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
@@ -12,6 +12,7 @@ metadata:
namespace: {{ $.Release.Namespace }}
labels:
app.kubernetes.io/name: {{ template "seaweedfs.name" $ }}
app.kubernetes.io/instance: {{ $.Release.Name }}
app.kubernetes.io/component: {{ $volumeName }}
helm.sh/chart: {{ $.Chart.Name }}-{{ $.Chart.Version | replace "+" "_" }}
app.kubernetes.io/managed-by: {{ $.Release.Service }}
@@ -136,11 +136,15 @@ spec:
{{- else }}
-admin={{ template "seaweedfs.fullname" . }}-admin.{{ .Release.Namespace }}:{{ .Values.admin.port }}{{ if .Values.admin.grpcPort }}.{{ .Values.admin.grpcPort }}{{ end }} \
{{- end }}
-capabilities={{ .Values.worker.capabilities }} \
-maxConcurrent={{ .Values.worker.maxConcurrent }} \
-workingDir={{ .Values.worker.workingDir }}{{- if or .Values.worker.metricsPort .Values.worker.extraArgs }} \{{ end }}
-jobType={{ .Values.worker.jobType }} \
-maxDetect={{ .Values.worker.maxDetect }} \
-maxExecute={{ .Values.worker.maxExecute }} \
-workingDir={{ .Values.worker.workingDir }}{{- if or .Values.worker.metricsPort .Values.worker.metricsIp .Values.worker.extraArgs }} \{{ end }}
{{- if .Values.worker.metricsPort }}
-metricsPort={{ .Values.worker.metricsPort }}{{- if .Values.worker.extraArgs }} \{{ end }}
-metricsPort={{ .Values.worker.metricsPort }}{{- if or .Values.worker.metricsIp .Values.worker.extraArgs }} \{{ end }}
{{- end }}
{{- if .Values.worker.metricsIp }}
-metricsIp={{ .Values.worker.metricsIp }}{{- if .Values.worker.extraArgs }} \{{ end }}
{{- end }}
{{- range $index, $arg := .Values.worker.extraArgs }}
{{ $arg }}{{- if lt $index (sub (len $.Values.worker.extraArgs) 1) }} \{{ end }}
+48 -6
View File
@@ -20,6 +20,7 @@ global:
filerRead: false
# we will use this serviceAccountName for all ClusterRoles/ClusterRoleBindings
serviceAccountName: "seaweedfs"
serviceAccountAnnotations: {}
automountServiceAccountToken: true
certificates:
duration: 87600h
@@ -890,12 +891,22 @@ filer:
# set to the name of an existing kubernetes Secret with the s3 json config file
# should have a secret key called seaweedfs_s3_config with an inline json configure
existingConfigSecret: null
# To provide explicit credentials for the S3 gateway, set them under
# the top-level s3.credentials key (not filer.s3.credentials).
# The s3-secret.yaml template only reads from .Values.s3.credentials.
# See: s3.credentials.admin.accessKey, s3.credentials.read.accessKey
auditLogConfig: {}
# You may specify buckets to be created during the install or upgrade process.
# Buckets may be exposed publicly by setting `anonymousRead` to `true`
# ttl format: [1-255][m|h|d|w|M|y] (e.g., 7d)
# objectLock enables S3 Object Lock (irreversible, forces versioning)
# versioning: Enabled or Suspended (or true to enable)
# createBuckets:
# - name: bucket-a
# anonymousRead: true
# ttl: 7d
# objectLock: true
# versioning: Enabled
# - name: bucket-b
# anonymousRead: false
@@ -918,12 +929,28 @@ s3:
# set to the name of an existing kubernetes Secret with the s3 json config file
# should have a secret key called seaweedfs_s3_config with an inline json config
existingConfigSecret: null
# Optionally provide explicit credentials for the S3 gateway.
# When set, these are used in the generated s3 secret instead of
# auto-generating random credentials.
# credentials:
# admin:
# accessKey: ""
# secretKey: ""
# read:
# accessKey: ""
# secretKey: ""
auditLogConfig: {}
# You may specify buckets to be created during the install or upgrade process.
# Buckets may be exposed publicly by setting `anonymousRead` to `true`
# ttl format: [1-255][m|h|d|w|M|y] (e.g., 7d)
# objectLock enables S3 Object Lock (irreversible, forces versioning)
# versioning: Enabled or Suspended (or true to enable)
# createBuckets:
# - name: bucket-a
# anonymousRead: true
# ttl: 7d
# objectLock: true
# versioning: Enabled
# - name: bucket-b
# anonymousRead: false
@@ -1270,17 +1297,20 @@ worker:
replicas: 1
loggingOverrideLevel: null
metricsPort: 9327
metricsIp: "" # If empty, defaults to 0.0.0.0
# Admin server to connect to
adminServer: ""
# Worker capabilities - comma-separated list
# Available: vacuum, balance, erasure_coding
# Default: "vacuum,balance,erasure_coding" (all capabilities)
capabilities: "vacuum,balance,erasure_coding"
# Worker job types - comma-separated list
# Available: vacuum, volume_balance, erasure_coding
jobType: "vacuum,volume_balance,erasure_coding"
# Maximum number of concurrent tasks
maxConcurrent: 3
# Maximum number of concurrent detection requests
maxDetect: 1
# Maximum number of concurrent execution jobs
maxExecute: 4
# Working directory for task execution
workingDir: "/tmp/seaweedfs-worker"
@@ -1418,12 +1448,22 @@ allInOne:
# Set to the name of an existing kubernetes Secret with the s3 json config file
# should have a secret key called seaweedfs_s3_config with an inline json config
existingConfigSecret: null
# To provide explicit credentials for the S3 gateway, set them under
# the top-level s3.credentials key (not allInOne.s3.credentials).
# The s3-secret.yaml template only reads from .Values.s3.credentials.
# See: s3.credentials.admin.accessKey, s3.credentials.read.accessKey
auditLogConfig: null # S3 audit log configuration (null inherits from s3.auditLogConfig)
# You may specify buckets to be created during the install process.
# Buckets may be exposed publicly by setting `anonymousRead` to `true`
# ttl format: [1-255][m|h|d|w|M|y] (e.g., 7d)
# objectLock enables S3 Object Lock (irreversible, forces versioning)
# versioning: Enabled or Suspended (or true to enable)
# createBuckets:
# - name: bucket-a
# anonymousRead: true
# ttl: 7d
# objectLock: true
# versioning: Enabled
# - name: bucket-b
# anonymousRead: false
@@ -1589,6 +1629,8 @@ cosi:
image: "ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.1.2"
driverName: "seaweedfs.objectstorage.k8s.io"
bucketClassName: "seaweedfs"
# Optional parameters to pass to the default BucketClass (e.g., diskType for tiered storage)
bucketClassParameters: {}
endpoint: ""
region: ""
-7
View File
@@ -1,7 +0,0 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "weed-volume-rs"
version = "0.1.0"
-6
View File
@@ -1,6 +0,0 @@
[package]
name = "weed-volume-rs"
version = "0.1.0"
edition = "2021"
[dependencies]
-245
View File
@@ -1,245 +0,0 @@
# Rust Volume Server Parity Implementation Plan
## Objective
Implement a native Rust volume server that replicates Go volume-server behavior for HTTP and gRPC APIs, so it can become a drop-in replacement validated by existing integration suites.
## Current Focus (2026-02-16)
- Program focus is now Rust implementation parity, not broad test expansion.
- `test/volume_server` is treated as the parity gate.
- Existing Rust launcher modes (`exec`, `proxy`) are transition tools; they are not the final target.
## Current Status
- Rust crate and launcher are in place.
- Integration harness can run:
- Go master + Go volume (default)
- Go master + Rust launcher (`VOLUME_SERVER_IMPL=rust`)
- Rust launcher `proxy` mode has full-suite integration pass while delegating backend handlers to Go.
- Rust launcher `native` mode is wired as the default Rust entrypoint and currently bootstraps via supervised Go backend delegation.
- Native Rust HTTP handlers now serve control/surface paths in `native` mode:
- `/status`, `/healthz`
- `OPTIONS` admin/public behavior (method allow-list + CORS preflight headers)
- `/ui/index.html` with config-driven JWT/access-ui gating
- `/favicon.ico` and `/seaweedfsstatic/*`
- public non-read methods (`POST`/`PUT`/`DELETE`/unsupported verbs) as `200` no-op passthrough parity
- admin unsupported verbs as `400` parity
- Native Rust control-surface parity now also includes:
- `/healthz` status mirroring from backend state transitions (e.g. leave/stopping => `503`)
- absolute-form HTTP request-target normalization before native route matching
- Native Rust HTTP data-path prevalidation now includes:
- early malformed vid/fid rejection (`400`) for GET/HEAD/POST/PUT fid-route shapes before delegation
- slash-form fid-route parsing for `/{vid}/{fid}` and `/{vid}/{fid}/{filename}` with reserved-path exclusions (`/status`, `/healthz`, `/ui/index.html`, `/stats/*`, static assets)
- write-path error prevalidation for fid routes:
- malformed multipart form-data requests without a boundary => `400`
- requests containing `Content-MD5` header => `400` mismatch parity branch
- request bodies exceeding configured `-fileSizeLimitMB` (by `Content-Length`) => `400`
- Native Rust API/storage handlers are not implemented yet.
## Parity Exit Criteria
1. Native mode passes:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc`
2. CI runs native Rust mode integration coverage (at least smoke, then expanded shards).
3. Rust mode defaults to native behavior for integration harness.
4. Go-backend delegation is removed (or retained only as explicit fallback mode).
## Architecture Workstreams
### A. Runtime and Configuration Parity
- [x] Add `native` runtime mode in `weed-volume-rs` (bootstrap delegation path).
- [ ] Parse and honor volume-server CLI/config flags used by integration harness:
- [ ] network/bind ports (`-ip`, `-port`, `-port.grpc`, `-port.public`)
- [ ] master target/config dir/read mode/throttling/JWT-related config
- [ ] size/timeout controls and maintenance state defaults
- [ ] Implement graceful lifecycle behavior (signals, shutdown, readiness).
### B. Native HTTP Surface
- [ ] Admin/control endpoints:
- [x] `GET /status` (native Rust in `native` mode)
- [x] `GET /healthz` (native Rust in `native` mode)
- [x] `OPTIONS` admin/public method+CORS control behavior (native Rust in `native` mode)
- [x] static/UI endpoints currently exercised
- [ ] Data read path parity:
- [ ] fid parsing/path variants
- [ ] conditional headers (`If-Modified-Since`, `If-None-Match`)
- [ ] range handling (single/multi/invalid)
- [ ] deleted reads, auth checks, read-mode branches
- [ ] chunk-manifest and compression/image transformation branches
- [ ] Data write/delete parity:
- [ ] write success/unchanged/error paths
- [ ] replication and file-size-limit paths
- [ ] delete and chunk-manifest delete branches
- [ ] Method/CORS/public-port parity for split admin/public behavior.
### C. Native gRPC Surface
- [ ] Control-plane RPCs:
- [ ] `GetState`, `SetState`, `VolumeServerStatus`, `Ping`, `VolumeServerLeave`
- [ ] admin lifecycle: allocate/mount/unmount/delete/configure/readonly/writable
- [ ] Data RPCs:
- [ ] `ReadNeedleBlob`, `ReadNeedleMeta`, `WriteNeedleBlob`
- [ ] `BatchDelete`, `ReadAllNeedles`
- [ ] sync/copy/receive and status endpoints
- [ ] Stream RPCs:
- [ ] tail sender/receiver
- [ ] vacuum streams
- [ ] query streams
- [ ] Advanced families:
- [ ] erasure coding RPC set
- [ ] tiering/remote fetch
- [ ] scrub/query mode matrix
### D. Storage Compatibility Layer
- [ ] Implement volume data/index handling compatible with Go on-disk format.
- [ ] Preserve cookie/checksum/timestamp semantics used by tests.
- [ ] Match read/write/delete consistency and error mapping behavior.
- [ ] Ensure EC metadata/data-path compatibility with existing files.
### E. Operational Hardening
- [ ] Deterministic startup/readiness and shutdown semantics.
- [ ] Log/error parity sufficient for debugging and CI triage.
- [ ] Concurrency/timeout behavior alignment for throttling and streams.
- [ ] Performance baseline checks vs Go for key flows.
## Milestone Plan
### M0 (Completed): Harness + Launcher Transition
- [x] Rust launcher integrated into harness.
- [x] Proxy mode full-suite validation with Go backend delegation.
### M1: Native Skeleton (Control Plane First)
- [ ] `native` mode boots and serves:
- [x] `/status`, `/healthz`
- [ ] `GetState`, `SetState`, `VolumeServerStatus`, `Ping`, `VolumeServerLeave`
- Gate:
- [x] targeted HTTP/grpc control tests pass in `native` mode (delegated backend path).
### M2: Native Core Data Paths
- [ ] Native HTTP read/write/delete baseline parity.
- [ ] Native gRPC data baseline parity (`Read/WriteNeedle*`, `BatchDelete`, `ReadAllNeedles`).
- Gate:
- core HTTP and gRPC data suites pass in `native` mode.
### M3: Native Stream + Copy/Sync
- [ ] Tail/copy/receive/sync paths in native mode.
- Gate:
- stream/copy families pass in `native` mode.
### M4: Native Advanced Feature Families
- [ ] EC, tiering, scrub/query advanced branches.
- Gate:
- full `/test/volume_server/http` and `/test/volume_server/grpc` pass in `native` mode.
### M5: CI/Cutover
- [x] Add/expand native-mode CI jobs (smoke matrix includes `native`).
- [x] Make native mode default for Rust integration runs.
- [ ] Keep `exec`/`proxy` only as explicit fallback modes during rollout.
## Immediate Next Steps
1. Implement minimal native gRPC state/ping RPC handlers (`GetState`, `SetState`, `VolumeServerStatus`, `Ping`, `VolumeServerLeave`).
2. Expand native HTTP parity from control/static surface into data-path handlers (read/write/delete variants) while preserving current native control behavior.
3. Keep rerunning native-mode integration suites as each delegated branch is replaced.
4. Add mismatch triage notes for each API moved from delegation to native implementation.
## Risk Register
- On-disk format mismatch risk:
- Mitigation: implement format-level compatibility tests early (idx/dat/needle encoding).
- Behavioral drift in edge branches:
- Mitigation: use integration suite failures as primary truth; only add tests for newly discovered untracked branches.
- Stream/concurrency semantic mismatch:
- Mitigation: stabilize with focused interruption/timeout parity tests.
## Progress Log
- Date: 2026-02-15
- Change: Added Rust launcher integration (`exec`) and harness wiring.
- Validation: Rust launcher mode passed smoke and full integration suites while delegating to Go backend.
- Commits: `7beab85c2`, `880c2e1da`, `63d08e8a9`, `d402573ea`, `3bd20e6a1`, `6ce4d7ede`
- Date: 2026-02-15
- Change: Added Rust proxy supervisor mode and validated full integration suite.
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=proxy go test -count=1 ./test/volume_server/http`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=proxy go test -count=1 ./test/volume_server/grpc`
- Commits: `a7f50d23b`, `548b3d9a3`
- Date: 2026-02-16
- Change: Re-focused plan from test expansion to native Rust implementation parity.
- Validation basis: latest Rust proxy full-suite pass keeps regression baseline stable while native implementation starts.
- Commits: `14c863dbf`
- Date: 2026-02-16
- Change: Added native Rust launcher mode bootstrap, set Rust launcher default mode to `native`, and expanded CI Rust smoke matrix to include `native`.
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc`
- `env VOLUME_SERVER_IMPL=rust go test -count=1 ./test/volume_server/http -run '^TestAdminStatusAndHealthz$'`
- `env VOLUME_SERVER_IMPL=rust go test -count=1 ./test/volume_server/grpc -run '^TestStateAndStatusRPCs$'`
- Commits: `70ddbee37`, `61befd10f`, `2e65966c0`
- Date: 2026-02-16
- Change: Implemented first native Rust HTTP handlers in `native` mode for `/status` and `/healthz` on the admin listener while preserving proxy delegation for other APIs.
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestAdminStatusAndHealthz$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc`
- Commits: `7e6e0261a`
- Date: 2026-02-16
- Change: Implemented native Rust `OPTIONS` handling for admin and public listeners in `native` mode, including method allow-list and origin-driven CORS response parity used by integration tests.
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestOptionsMethodsByPort$|^TestOptionsWithOriginIncludesCorsHeaders$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc`
- Commits: `fbff2cb39`
- Date: 2026-02-16
- Change: Added native Rust UI/static/public-surface handlers in `native` mode:
- `/ui/index.html` with config-driven JWT/access-ui gating parity
- `/favicon.ico` and `/seaweedfsstatic/*` static endpoint parity
- public non-read methods as `200` no-op passthrough parity
- admin unsupported methods as `400` parity
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestAdminStatusAndHealthz$|^TestUiIndexNotExposedWhenJwtSigningEnabled$|^TestUiIndexExposedWhenJwtSigningEnabledAndAccessUITrue$|^TestStaticAssetEndpoints$|^TestStaticAssetEndpointsOnPublicPort$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestOptionsMethodsByPort$|^TestOptionsWithOriginIncludesCorsHeaders$|^TestPublicPortReadOnlyMethodBehavior$|^TestCorsAndUnsupportedMethodBehavior$|^TestUnsupportedMethodTraceParity$|^TestUnsupportedMethodPropfindParity$|^TestUnsupportedMethodConnectParity$|^TestUnsupportedMethodMkcolParity$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc`
- Commits: `23e4497b2`
- Date: 2026-02-16
- Change: Hardened native Rust HTTP control-path parity:
- `/healthz` now mirrors backend health status (`200`/`503`) in native mode
- absolute-form request-target parsing (e.g. `GET http://host/path HTTP/1.1`) now normalizes to route path parity
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestAdminStatusAndHealthz$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http/...`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc/...`
- Commits: `d6ff6ed6d`
- Date: 2026-02-16
- Change: Added native malformed fid-route validation in `native` mode:
- GET/HEAD/POST/PUT requests matching fid URL shapes now return Rust-native `400 Bad Request` for invalid vid/fid tokens
- valid fid-route requests still delegate to backend handlers for storage/data execution paths
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestInvalidReadPathReturnsBadRequest$|^TestWriteInvalidVidAndFidReturnBadRequest$|^TestMalformedVidFidPathReturnsBadRequest$|^TestDownloadLimitInvalidVidWhileOverLimitReturnsBadRequest$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http/...`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc/...`
- Commits: `1ce0174b2`
- Date: 2026-02-16
- Change: Added native Rust write-path prevalidation branches for fid routes in `native` mode:
- multipart form uploads without boundary now return native `400`
- `Content-MD5` requests now return native `400` mismatch branch
- `Content-Length` over configured `-fileSizeLimitMB` now returns native `400` limit branch
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestUploadReadRangeHeadDeleteRoundTrip$|^TestWriteMalformedMultipartAndMD5Mismatch$|^TestWriteRejectsPayloadOverFileSizeLimit$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http/...`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc/...`
- Commits: `94cefd6f4`
- Date: 2026-02-16
- Change: Broadened native fid-route parser to cover slash-form URLs:
- `/{vid}/{fid}` and `/{vid}/{fid}/{filename}` now flow through native malformed vid/fid validation
- added explicit non-fid exclusions for control/stats/static paths to preserve route parity
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestAdminStatusAndHealthz$|^TestReadPathShapesAndIfModifiedSince$|^TestMalformedVidFidPathReturnsBadRequest$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http/...`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc/...`
- Commits: `e66983e9b`
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -148,7 +148,7 @@ func waitForServer(url string, timeout time.Duration) bool {
func testProtobufMarshaling() error {
// Test protobuf marshaling/unmarshaling
testData := &proto.TelemetryData{
ClusterId: "test-cluster-12345",
TopologyId: "test-cluster-12345",
Version: "test-3.45",
Os: "linux/amd64",
VolumeServerCount: 2,
@@ -2,13 +2,11 @@ package admin_dockertest
import (
"bytes"
crand "crypto/rand"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"math/rand"
"net/http"
"net/url"
"os"
"os/exec"
"path/filepath"
@@ -161,129 +159,95 @@ func TestEcEndToEnd(t *testing.T) {
client := &http.Client{}
// 1. Configure Global Maintenance (Scan Interval = 1s) via API
t.Log("Configuring Global Maintenance via API...")
// 1. Configure plugin job types for fast EC detection/execution.
t.Log("Configuring plugin job types via API...")
// 1.1 Fetch current config
req, _ := http.NewRequest("GET", AdminUrl+"/api/maintenance/config", nil)
// Disable volume balance to reduce interference for this EC-focused test.
balanceConfig := map[string]interface{}{
"job_type": "volume_balance",
"admin_runtime": map[string]interface{}{
"enabled": false,
},
}
jsonBody, err := json.Marshal(balanceConfig)
if err != nil {
t.Fatalf("Failed to marshal volume_balance config: %v", err)
}
req, err := http.NewRequest("PUT", AdminUrl+"/api/plugin/job-types/volume_balance/config", bytes.NewBuffer(jsonBody))
if err != nil {
t.Fatalf("Failed to create volume_balance config request: %v", err)
}
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
t.Fatalf("Failed to get global config: %v", err)
t.Fatalf("Failed to update volume_balance config: %v", err)
}
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
t.Fatalf("Failed to get global config (status %d): %s", resp.StatusCode, string(body))
}
var globalConfig map[string]interface{}
if err := json.NewDecoder(resp.Body).Decode(&globalConfig); err != nil {
t.Fatalf("Failed to decode global config: %v", err)
t.Fatalf("Failed to update volume_balance config (status %d): %s", resp.StatusCode, string(body))
}
resp.Body.Close()
// 1.2 Modify config
globalConfig["enabled"] = true
globalConfig["scan_interval_seconds"] = 1
// Ensure policy structure exists
if globalConfig["policy"] == nil {
globalConfig["policy"] = map[string]interface{}{}
ecConfig := map[string]interface{}{
"job_type": "erasure_coding",
"admin_runtime": map[string]interface{}{
"enabled": true,
"detection_interval_seconds": 1,
"global_execution_concurrency": 4,
"per_worker_execution_concurrency": 4,
"max_jobs_per_detection": 100,
},
"worker_config_values": map[string]interface{}{
"quiet_for_seconds": map[string]interface{}{
"int64_value": "1",
},
"min_interval_seconds": map[string]interface{}{
"int64_value": "1",
},
"min_size_mb": map[string]interface{}{
"int64_value": "1",
},
"fullness_ratio": map[string]interface{}{
"double_value": 0.0001,
},
},
}
policy, _ := globalConfig["policy"].(map[string]interface{})
// Ensure task_policies structure exists
if policy["task_policies"] == nil {
policy["task_policies"] = map[string]interface{}{}
jsonBody, err = json.Marshal(ecConfig)
if err != nil {
t.Fatalf("Failed to marshal erasure_coding config: %v", err)
}
taskPolicies, _ := policy["task_policies"].(map[string]interface{})
// Disable balance tasks to avoid interference with EC test
if taskPolicies["balance"] == nil {
taskPolicies["balance"] = map[string]interface{}{}
req, err = http.NewRequest("PUT", AdminUrl+"/api/plugin/job-types/erasure_coding/config", bytes.NewBuffer(jsonBody))
if err != nil {
t.Fatalf("Failed to create erasure_coding config request: %v", err)
}
balancePolicy, _ := taskPolicies["balance"].(map[string]interface{})
balancePolicy["enabled"] = false
// Set global max concurrent
policy["global_max_concurrent"] = 4
globalConfig["policy"] = policy
// Explicitly set required fields
requiredFields := map[string]float64{
"worker_timeout_seconds": 300,
"task_timeout_seconds": 7200,
"retry_delay_seconds": 900,
"cleanup_interval_seconds": 86400,
"task_retention_seconds": 604800,
"max_retries": 3,
}
for field, val := range requiredFields {
if _, ok := globalConfig[field]; !ok || globalConfig[field] == 0 {
globalConfig[field] = val
}
}
// 1.3 Update config
jsonBody, _ := json.Marshal(globalConfig)
req, _ = http.NewRequest("PUT", AdminUrl+"/api/maintenance/config", bytes.NewBuffer(jsonBody))
req.Header.Set("Content-Type", "application/json")
resp, err = client.Do(req)
if err != nil {
t.Fatalf("Failed to update global config: %v", err)
t.Fatalf("Failed to update erasure_coding config: %v", err)
}
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
t.Fatalf("Failed to update global config (status %d): %s", resp.StatusCode, string(body))
t.Fatalf("Failed to update erasure_coding config (status %d): %s", resp.StatusCode, string(body))
}
resp.Body.Close()
// 2. Configure EC Task (Short intervals) via Form API
t.Log("Configuring EC Task via Form API...")
formData := url.Values{}
formData.Set("enabled", "true")
formData.Set("scan_interval_seconds", "1")
formData.Set("repeat_interval_seconds", "1")
formData.Set("check_interval_seconds", "1")
formData.Set("max_concurrent", "4")
formData.Set("quiet_for_seconds_value", "1")
formData.Set("quiet_for_seconds_unit", "seconds")
formData.Set("min_size_mb", "1")
formData.Set("fullness_ratio", "0.0001")
req, _ = http.NewRequest("POST", AdminUrl+"/maintenance/config/erasure_coding", strings.NewReader(formData.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
resp, err = client.Do(req)
if err != nil {
t.Fatalf("Failed to update EC config: %v", err)
}
if resp.StatusCode != 200 && resp.StatusCode != 303 {
body, _ := io.ReadAll(resp.Body)
t.Fatalf("Failed to update EC config (status %d): %s", resp.StatusCode, string(body))
}
resp.Body.Close()
t.Log("EC Task Configuration updated")
// 3. Restart Admin to pick up Global Config (Scan Interval)
if len(runningCmds) > 0 {
adminCmd := runningCmds[len(runningCmds)-1]
t.Log("Restarting Admin Server to apply configuration...")
stopWeed(t, adminCmd)
time.Sleep(10 * time.Second)
startWeed(t, "admin_restarted", "admin", "-master=localhost:9333", "-port=23646", "-port.grpc=33646", "-dataDir=./tmp/admin")
waitForUrl(t, AdminUrl+"/health", 60)
}
// 4. Upload a file
// 2. Upload a file
fileSize := 5 * 1024 * 1024
data := make([]byte, fileSize)
rand.Read(data)
crand.Read(data)
fileName := fmt.Sprintf("ec_test_file_%d", time.Now().Unix())
t.Logf("Uploading %d bytes file %s to Filer...", fileSize, fileName)
uploadUrl := FilerUrl + "/" + fileName
var uploadErr error
for i := 0; i < 10; i++ {
req, _ := http.NewRequest("PUT", uploadUrl, bytes.NewBuffer(data))
req, err := http.NewRequest("PUT", uploadUrl, bytes.NewBuffer(data))
if err != nil {
uploadErr = err
t.Logf("Upload attempt %d failed to create request: %v", i+1, err)
time.Sleep(2 * time.Second)
continue
}
resp, err := client.Do(req)
if err == nil {
if resp.StatusCode == 201 {
@@ -306,17 +270,17 @@ func TestEcEndToEnd(t *testing.T) {
}
t.Log("Upload successful")
// 5. Verify EC Encoding
// 3. Verify EC Encoding
t.Log("Waiting for EC encoding (checking Master topology)...")
startTime := time.Now()
ecVerified := false
var lastBody []byte
for time.Since(startTime) < 300*time.Second {
// 5.1 Check Master Topology
// 3.1 Check Master Topology
resp, err := http.Get(MasterUrl + "/dir/status")
if err == nil {
lastBody, _ = ioutil.ReadAll(resp.Body)
lastBody, _ = io.ReadAll(resp.Body)
resp.Body.Close()
// Check total EC shards
@@ -336,8 +300,8 @@ func TestEcEndToEnd(t *testing.T) {
}
}
// 5.2 Debug: Check workers and tasks
wResp, wErr := http.Get(AdminUrl + "/api/maintenance/workers")
// 3.2 Debug: Check workers and jobs
wResp, wErr := http.Get(AdminUrl + "/api/plugin/workers")
workerCount := 0
if wErr == nil {
var workers []interface{}
@@ -346,7 +310,7 @@ func TestEcEndToEnd(t *testing.T) {
workerCount = len(workers)
}
tResp, tErr := http.Get(AdminUrl + "/api/maintenance/tasks")
tResp, tErr := http.Get(AdminUrl + "/api/plugin/jobs?limit=1000")
taskCount := 0
if tErr == nil {
var tasks []interface{}
@@ -1,5 +1,4 @@
//go:build foundationdb
// +build foundationdb
package foundationdb
@@ -1,5 +1,4 @@
//go:build foundationdb
// +build foundationdb
package foundationdb
-6
View File
@@ -19,9 +19,6 @@ services:
start_period: 10s
seaweedfs-volume:
build:
context: ../../../docker
dockerfile: Dockerfile.local
image: seaweedfs:local
container_name: seaweedfs-spark-volume
ports:
@@ -43,9 +40,6 @@ services:
start_period: 10s
seaweedfs-filer:
build:
context: ../../../docker
dockerfile: Dockerfile.local
image: seaweedfs:local
container_name: seaweedfs-spark-filer
ports:
+1 -1
View File
@@ -116,7 +116,7 @@
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.11.4</version>
<version>1.11.5</version>
</dependency>
<!-- Apache ZooKeeper - Fix CVEs -->
+4 -2
View File
@@ -16,8 +16,9 @@ RUN go mod download
# Copy source code
COPY . .
# Build the weed binary with Kafka gateway support
# Build the weed binaries with Kafka gateway support
RUN CGO_ENABLED=1 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o weed ./weed
RUN CGO_ENABLED=1 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o weed-sql ./cmd/weed-sql
# Final stage
FROM alpine:latest
@@ -32,8 +33,9 @@ RUN addgroup -g 1000 seaweedfs && \
# Set working directory
WORKDIR /usr/bin
# Copy binary from builder
# Copy binaries from builder
COPY --from=builder /app/weed .
COPY --from=builder /app/weed-sql .
# Create data directory
RUN mkdir -p /data && chown seaweedfs:seaweedfs /data
+86 -69
View File
@@ -8,7 +8,7 @@ require (
github.com/seaweedfs/seaweedfs v0.0.0-00010101000000-000000000000
github.com/segmentio/kafka-go v0.4.49
github.com/stretchr/testify v1.11.1
google.golang.org/grpc v1.77.0
google.golang.org/grpc v1.78.0
)
replace github.com/seaweedfs/seaweedfs => ../../
@@ -17,14 +17,15 @@ require (
cloud.google.com/go/auth v0.17.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
cloud.google.com/go/compute/metadata v0.9.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.20.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.2 // indirect
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.3 // indirect
github.com/Azure/go-ntlmssp v0.1.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 // indirect
github.com/Files-com/files-sdk-go/v3 v3.2.218 // indirect
github.com/FilenCloudDienste/filen-sdk-go v0.0.37 // indirect
github.com/Files-com/files-sdk-go/v3 v3.2.264 // indirect
github.com/IBM/go-sdk-core/v5 v5.21.0 // indirect
github.com/Max-Sum/base32768 v0.0.0-20230304063302-18e6ce5945fd // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
@@ -35,51 +36,58 @@ require (
github.com/ProtonMail/go-srp v0.0.7 // indirect
github.com/ProtonMail/gopenpgp/v2 v2.9.0 // indirect
github.com/PuerkitoBio/goquery v1.10.3 // indirect
github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0 // indirect
github.com/abbot/go-http-auth v0.4.0 // indirect
github.com/anchore/go-lzo v0.1.0 // indirect
github.com/andybalholm/brotli v1.2.0 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/aws/aws-sdk-go v1.55.8 // indirect
github.com/aws/aws-sdk-go-v2 v1.41.0 // indirect
github.com/aws/aws-sdk-go-v2 v1.41.1 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4 // indirect
github.com/aws/aws-sdk-go-v2/config v1.32.6 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.19.6 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.3 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16 // indirect
github.com/aws/aws-sdk-go-v2/config v1.32.7 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.19.7 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.17 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.17 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.17 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.15 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.16 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.6 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.16 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.15 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.93.0 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.41.5 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.7 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.17 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.16 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.95.0 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.5 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.9 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.13 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.41.6 // indirect
github.com/aws/smithy-go v1.24.0 // indirect
github.com/bahlo/generic-list-go v0.2.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/boombuler/barcode v1.1.0 // indirect
github.com/bradenaw/juniper v0.15.3 // indirect
github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 // indirect
github.com/buengese/sgzip v0.1.1 // indirect
github.com/bufbuild/protocompile v0.14.1 // indirect
github.com/buger/jsonparser v1.1.1 // indirect
github.com/calebcase/tmpfile v1.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/chilts/sid v0.0.0-20190607042430-660e94789ec9 // indirect
github.com/cloudflare/circl v1.6.1 // indirect
github.com/cloudinary/cloudinary-go/v2 v2.12.0 // indirect
github.com/clipperhouse/stringish v0.1.1 // indirect
github.com/clipperhouse/uax29/v2 v2.3.0 // indirect
github.com/cloudflare/circl v1.6.3 // indirect
github.com/cloudinary/cloudinary-go/v2 v2.13.0 // indirect
github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc // indirect
github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc // indirect
github.com/cognusion/imaging v1.0.2 // indirect
github.com/colinmarc/hdfs/v2 v2.4.0 // indirect
github.com/coreos/go-semver v0.3.1 // indirect
github.com/coreos/go-systemd/v22 v22.5.0 // indirect
github.com/coreos/go-systemd/v22 v22.6.0 // indirect
github.com/creasty/defaults v1.8.0 // indirect
github.com/cronokirby/saferith v0.33.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/diskfs/go-diskfs v1.7.0 // indirect
github.com/dromara/dongle v1.0.1 // indirect
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/eapache/go-resiliency v1.7.0 // indirect
@@ -91,25 +99,26 @@ require (
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/flynn/noise v1.1.0 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.9 // indirect
github.com/gabriel-vasile/mimetype v1.4.11 // indirect
github.com/geoffgarside/ber v1.2.0 // indirect
github.com/go-chi/chi/v5 v5.2.2 // indirect
github.com/go-chi/chi/v5 v5.2.5 // indirect
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 // indirect
github.com/go-git/go-billy/v5 v5.6.2 // indirect
github.com/go-jose/go-jose/v4 v4.1.3 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-openapi/errors v0.22.2 // indirect
github.com/go-openapi/strfmt v0.23.0 // indirect
github.com/go-openapi/errors v0.22.4 // indirect
github.com/go-openapi/strfmt v0.25.0 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.27.0 // indirect
github.com/go-playground/validator/v10 v10.28.0 // indirect
github.com/go-resty/resty/v2 v2.16.5 // indirect
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/gofrs/flock v0.12.1 // indirect
github.com/gofrs/flock v0.13.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
github.com/golang-jwt/jwt/v5 v5.3.0 // indirect
github.com/golang-jwt/jwt/v5 v5.3.1 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/golang/snappy v1.0.0 // indirect
github.com/google/btree v1.1.3 // indirect
@@ -124,78 +133,83 @@ require (
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-retryablehttp v0.7.8 // indirect
github.com/hashicorp/go-uuid v1.0.3 // indirect
github.com/henrybear327/Proton-API-Bridge v1.0.0 // indirect
github.com/henrybear327/go-proton-api v1.0.0 // indirect
github.com/internxt/rclone-adapter v0.0.0-20260213125353-6f59c89fcb7c // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
github.com/jcmturner/gofork v1.7.6 // indirect
github.com/jcmturner/goidentity/v6 v6.0.1 // indirect
github.com/jcmturner/gokrb5/v8 v8.4.4 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/jhump/protoreflect v1.17.0 // indirect
github.com/jhump/protoreflect v1.18.0 // indirect
github.com/jhump/protoreflect/v2 v2.0.0-beta.1 // indirect
github.com/jlaffaye/ftp v0.2.1-0.20240918233326-1b970516f5d3 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/jtolds/gls v4.20.0+incompatible // indirect
github.com/jtolio/noiseconn v0.0.0-20231127013910-f6d9ecbf1de7 // indirect
github.com/jzelinskie/whirlpool v0.0.0-20201016144138-0675e54bb004 // indirect
github.com/karlseguin/ccache/v2 v2.0.8 // indirect
github.com/klauspost/compress v1.18.2 // indirect
github.com/klauspost/compress v1.18.4 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/klauspost/reedsolomon v1.12.6 // indirect
github.com/klauspost/reedsolomon v1.13.0 // indirect
github.com/koofr/go-httpclient v0.0.0-20240520111329-e20f8f203988 // indirect
github.com/koofr/go-koofrclient v0.0.0-20221207135200-cbd7fc9ad6a6 // indirect
github.com/kr/fs v0.1.0 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/lanrat/extsort v1.4.0 // indirect
github.com/lanrat/extsort v1.4.2 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/lpar/date v1.0.0 // indirect
github.com/lufia/plan9stats v0.0.0-20250317134145-8bc96cf8fc35 // indirect
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3 // indirect
github.com/mailru/easyjson v0.9.1 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/mattn/go-runewidth v0.0.19 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/ncw/swift/v2 v2.0.4 // indirect
github.com/ncw/swift/v2 v2.0.5 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/oracle/oci-go-sdk/v65 v65.98.0 // indirect
github.com/oracle/oci-go-sdk/v65 v65.104.0 // indirect
github.com/orcaman/concurrent-map/v2 v2.0.1 // indirect
github.com/panjf2000/ants/v2 v2.11.3 // indirect
github.com/parquet-go/parquet-go v0.26.3 // indirect
github.com/parquet-go/bitpack v1.0.0 // indirect
github.com/parquet-go/jsonlite v1.0.0 // indirect
github.com/parquet-go/parquet-go v0.26.4 // indirect
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/pengsrc/go-shared v0.2.1-0.20190131101655-1999055a4a14 // indirect
github.com/peterh/liner v1.2.2 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
github.com/petermattis/goid v0.0.0-20260113132338-7c7de50cc741 // indirect
github.com/pierrec/lz4/v4 v4.1.25 // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pkg/sftp v1.13.10 // indirect
github.com/pkg/xattr v0.4.12 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/pquerna/otp v1.5.0 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.66.1 // indirect
github.com/prometheus/common v0.67.2 // indirect
github.com/prometheus/procfs v0.19.2 // indirect
github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8 // indirect
github.com/rclone/rclone v1.71.2 // indirect
github.com/quic-go/qpack v0.6.0 // indirect
github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11 // indirect
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18 // indirect
github.com/rclone/rclone v1.73.1 // indirect
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 // indirect
github.com/rdleal/intervalst v1.5.0 // indirect
github.com/relvacode/iso8601 v1.6.0 // indirect
github.com/relvacode/iso8601 v1.7.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rfjakob/eme v1.1.2 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect
github.com/sagikazarmark/locafero v0.11.0 // indirect
github.com/samber/lo v1.51.0 // indirect
github.com/samber/lo v1.52.0 // indirect
github.com/seaweedfs/goexif v1.0.3 // indirect
github.com/shirou/gopsutil/v4 v4.25.11 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/shirou/gopsutil/v4 v4.26.1 // indirect
github.com/sirupsen/logrus v1.9.4-0.20230606125235-dd1b4c2e81af // indirect
github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966 // indirect
github.com/smarty/assertions v1.16.0 // indirect
github.com/sony/gobreaker v1.0.0 // indirect
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
github.com/spacemonkeygo/monkit/v3 v3.0.24 // indirect
github.com/spacemonkeygo/monkit/v3 v3.0.25-0.20251022131615-eb24eb109368 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
@@ -203,15 +217,18 @@ require (
github.com/spiffe/go-spiffe/v2 v2.6.0 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965 // indirect
github.com/t3rm1n4l/go-mega v0.0.0-20250926104142-ccb8d3498e6c // indirect
github.com/t3rm1n4l/go-mega v0.0.0-20251031123324-a804aaa87491 // indirect
github.com/tklauser/go-sysconf v0.3.16 // indirect
github.com/tklauser/numcpus v0.11.0 // indirect
github.com/tyler-smith/go-bip39 v1.1.0 // indirect
github.com/tylertreat/BoomFilters v0.0.0-20210315201527-1a82519a3e43 // indirect
github.com/ulikunitz/xz v0.5.15 // indirect
github.com/unknwon/goconfig v1.0.0 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/viant/ptrie v1.0.1 // indirect
github.com/wk8/go-ordered-map/v2 v2.1.8 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
@@ -220,24 +237,24 @@ require (
github.com/zeebo/blake3 v0.2.4 // indirect
github.com/zeebo/errs v1.4.0 // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
go.etcd.io/bbolt v1.4.2 // indirect
go.etcd.io/bbolt v1.4.3 // indirect
go.mongodb.org/mongo-driver v1.17.6 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0 // indirect
go.opentelemetry.io/otel v1.38.0 // indirect
go.opentelemetry.io/otel/metric v1.38.0 // indirect
go.opentelemetry.io/otel/trace v1.38.0 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/crypto v0.46.0 // indirect
golang.org/x/exp v0.0.0-20250811191247-51f88131bc50 // indirect
golang.org/x/image v0.34.0 // indirect
golang.org/x/net v0.48.0 // indirect
golang.org/x/crypto v0.48.0 // indirect
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 // indirect
golang.org/x/image v0.36.0 // indirect
golang.org/x/net v0.49.0 // indirect
golang.org/x/oauth2 v0.34.0 // indirect
golang.org/x/sync v0.19.0 // indirect
golang.org/x/sys v0.39.0 // indirect
golang.org/x/term v0.38.0 // indirect
golang.org/x/text v0.32.0 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/term v0.40.0 // indirect
golang.org/x/text v0.34.0 // indirect
golang.org/x/time v0.14.0 // indirect
google.golang.org/api v0.258.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20251213004720-97cd9d5aeac2 // indirect
@@ -250,7 +267,7 @@ require (
modernc.org/mathutil v1.7.1 // indirect
moul.io/http2curl/v2 v2.3.0 // indirect
sigs.k8s.io/yaml v1.6.0 // indirect
storj.io/common v0.0.0-20250808122759-804533d519c1 // indirect
storj.io/common v0.0.0-20251107171817-6221ae45072c // indirect
storj.io/drpc v0.0.35-0.20250513201419-f7819ea69b55 // indirect
storj.io/eventkit v0.0.0-20250410172343-61f26d3de156 // indirect
storj.io/infectious v0.0.2 // indirect
+216 -182
View File
@@ -13,8 +13,7 @@ cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKV
cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=
cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc=
cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY=
cloud.google.com/go/auth v0.16.5 h1:mFWNQ2FEVWAliEQWpAdH80omXFokmrnbDhUS9cBywsI=
cloud.google.com/go/auth v0.16.5/go.mod h1:utzRfHMP+Vv0mpOkTRQoWD2q3BatTOoWbA7gCc2dUhQ=
cloud.google.com/go/auth v0.17.0 h1:74yCm7hCj2rUyyAocqnFzsAYXgJhrG26XCFimrc/Kz4=
cloud.google.com/go/auth v0.17.0/go.mod h1:6wv/t5/6rOPAX4fJiRjKkJCvswLwdet7G8+UGXt7nCQ=
cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIiLpZnkHRbnc=
cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c=
@@ -38,8 +37,8 @@ cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohl
cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs=
cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.20.0 h1:JXg2dwJUmPB9JmtVmdEB16APJ7jurfbY5jnfXpJoRMc=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.20.0/go.mod h1:YD5h/ldMsG0XiIw7PdyNhLxaM317eFh5yNLccNfGdyw=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0 h1:fou+2+WFTib47nS+nz/ozhEBnvU96bKHy6LjRsY4E28=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0/go.mod h1:t76Ruy8AHvUAC8GfMWJMa0ElSbuIcO03NLpynfbgsPA=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1 h1:Hk5QBxZQC1jb2Fwj6mpzme37xbCDdNTxU7O9eb5+LB4=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1/go.mod h1:IYus9qsFobWIc2YVwe/WPjcnyCkPKtnHAqUYeebc8z0=
github.com/Azure/azure-sdk-for-go/sdk/azidentity/cache v0.3.2 h1:yz1bePFlP5Vws5+8ez6T3HWXPmwOK7Yvq8QxDBD3SKY=
@@ -48,20 +47,22 @@ github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 h1:9iefClla7iYpfYWdzPCRDo
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2/go.mod h1:XtLgD3ZD34DAaVIIAyG3objl5DynM3CQ/vMcbBNJZGI=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1 h1:/Zt+cDPnpC3OVDm/JKLOs7M2DKmLRIIp3XIx9pHHiig=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1/go.mod h1:Ng3urmn6dYe8gnbCMoHHVl5APYz2txho3koEkV2o2HA=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3 h1:ZJJNFaQ86GVKQ9ehwqyAFE6pIfyicpuJ8IkVaPBc6/4=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3/go.mod h1:URuDvhmATVKqHBH9/0nOiNKk0+YcwfQ3WkK5PqHKxc8=
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.2 h1:l3SabZmNuXCMCbQUIeR4W6/N4j8SeH/lwX+a6leZhHo=
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.2/go.mod h1:k+mEZ4f1pVqZTRqtSDW2AhZ/3wT5qLpsUA75C/k7dtE=
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 h1:mFRzDkZVAjdal+s7s0MwaRv9igoPqLRdzOLzw/8Xvq8=
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358/go.mod h1:chxPXzSsl7ZWRAuOIE23GDNzjWuZquvFlgA8xmpunjU=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4 h1:jWQK1GI+LeGGUKBADtcH2rRqPxYB1Ljwms5gFA2LqrM=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4/go.mod h1:8mwH4klAm9DUgR2EEHyEEAQlRDvLPyg5fQry3y+cDew=
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.3 h1:sxgSqOB9CDToiaVFpxuvb5wGgGqWa3lCShcm5o0n3bE=
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.3/go.mod h1:XdED8i399lEVblYHTZM8eXaP07gv4Z58IL6ueMlVlrg=
github.com/Azure/go-ntlmssp v0.1.0 h1:DjFo6YtWzNqNvQdrwEyr/e4nhU3vRiwenz5QX7sFz+A=
github.com/Azure/go-ntlmssp v0.1.0/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1 h1:WJTmL004Abzc5wDB5VtZG2PJk5ndYDgVacGqfirKxjM=
github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1/go.mod h1:tCcJZ0uHAmvjsVYzEFivsRTN00oz5BEsRgQHu5JZ9WE=
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 h1:XRzhVemXdgvJqCH0sFfrBUTnUJSBrBf7++ypk+twtRs=
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0/go.mod h1:HKpQxkWaGLJ+D/5H8QRpyQXA1eKjxkFlOMwck5+33Jk=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/Files-com/files-sdk-go/v3 v3.2.218 h1:tIvcbHXNY/bq+Sno6vajOJOxhe5XbU59Fa1ohOybK+s=
github.com/Files-com/files-sdk-go/v3 v3.2.218/go.mod h1:E0BaGQbcMUcql+AfubCR/iasWKBxX5UZPivnQGC2z0M=
github.com/FilenCloudDienste/filen-sdk-go v0.0.37 h1:W8S9TrAyZ4//3PXsU6+Bi+fe/6uIL986GyS7PVzIDL4=
github.com/FilenCloudDienste/filen-sdk-go v0.0.37/go.mod h1:0cBhKXQg49XbKZZfk5TCDa3sVLP+xMxZTWL+7KY0XR0=
github.com/Files-com/files-sdk-go/v3 v3.2.264 h1:lMHTplAYI9FtmCo/QOcpRxmPA5REVAct1r2riQmDQKw=
github.com/Files-com/files-sdk-go/v3 v3.2.264/go.mod h1:wGqkOzRu/ClJibvDgcfuJNAqI2nLhe8g91tPlDKRCdE=
github.com/IBM/go-sdk-core/v5 v5.21.0 h1:DUnYhvC4SoC8T84rx5omnhY3+xcQg/Whyoa3mDPIMkk=
github.com/IBM/go-sdk-core/v5 v5.21.0/go.mod h1:Q3BYO6iDA2zweQPDGbNTtqft5tDcEpm6RTuqMlPcvbw=
github.com/IBM/sarama v1.46.0 h1:+YTM1fNd6WKMchlnLKRUB5Z0qD4M8YbvwIIPLvJD53s=
@@ -89,63 +90,69 @@ github.com/ProtonMail/gopenpgp/v2 v2.9.0 h1:ruLzBmwe4dR1hdnrsEJ/S7psSBmV15gFttFU
github.com/ProtonMail/gopenpgp/v2 v2.9.0/go.mod h1:IldDyh9Hv1ZCCYatTuuEt1XZJ0OPjxLpTarDfglih7s=
github.com/PuerkitoBio/goquery v1.10.3 h1:pFYcNSqHxBD06Fpj/KsbStFRsgRATgnf3LeXiUkhzPo=
github.com/PuerkitoBio/goquery v1.10.3/go.mod h1:tMUX0zDMHXYlAQk6p35XxQMqMweEKB7iK7iLNd4RH4Y=
github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0 h1:iLDOF0rdGTrol/q8OfPIIs5kLD8XvA2q75o6Uq/tgak=
github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0/go.mod h1:DrEWcQJjz7t5iF2duaiyhg4jyoF0kxOD6LtECNGkZ/Q=
github.com/aalpar/deheap v0.0.0-20210914013432-0cc84d79dec3 h1:hhdWprfSpFbN7lz3W1gM40vOgvSh1WCSMxYD6gGB4Hs=
github.com/aalpar/deheap v0.0.0-20210914013432-0cc84d79dec3/go.mod h1:XaUnRxSCYgL3kkgX0QHIV0D+znljPIDImxlv2kbGv0Y=
github.com/abbot/go-http-auth v0.4.0 h1:QjmvZ5gSC7jm3Zg54DqWE/T5m1t2AfDu6QlXJT0EVT0=
github.com/abbot/go-http-auth v0.4.0/go.mod h1:Cz6ARTIzApMJDzh5bRMSUou6UMSp0IEXg9km/ci7TJM=
github.com/anchore/go-lzo v0.1.0 h1:NgAacnzqPeGH49Ky19QKLBZEuFRqtTG9cdaucc3Vncs=
github.com/anchore/go-lzo v0.1.0/go.mod h1:3kLx0bve2oN1iDwgM1U5zGku1Tfbdb0No5qp1eL1fIk=
github.com/andybalholm/brotli v1.2.0 h1:ukwgCxwYrmACq68yiUqwIWnGY0cTPox/M94sVwToPjQ=
github.com/andybalholm/brotli v1.2.0/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM=
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc h1:LoL75er+LKDHDUfU5tRvFwxH0LjPpZN8OoG8Ll+liGU=
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc/go.mod h1:w648aMHEgFYS6xb0KVMMtZ2uMeemhiKCuD2vj6gY52A=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 h1:DklsrG3dyBCFEj5IhUbnKptjxatkF07cF2ak3yi77so=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=
github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
github.com/aws/aws-sdk-go-v2 v1.41.0 h1:tNvqh1s+v0vFYdA1xq0aOJH+Y5cRyZ5upu6roPgPKd4=
github.com/aws/aws-sdk-go-v2 v1.41.0/go.mod h1:MayyLB8y+buD9hZqkCW3kX1AKq07Y5pXxtgB+rRFhz0=
github.com/aws/aws-sdk-go-v2 v1.41.1 h1:ABlyEARCDLN034NhxlRUSZr4l71mh+T5KAeGh6cerhU=
github.com/aws/aws-sdk-go-v2 v1.41.1/go.mod h1:MayyLB8y+buD9hZqkCW3kX1AKq07Y5pXxtgB+rRFhz0=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4 h1:489krEF9xIGkOaaX3CE/Be2uWjiXrkCH6gUX+bZA/BU=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4/go.mod h1:IOAPF6oT9KCsceNTvvYMNHy0+kMF8akOjeDvPENWxp4=
github.com/aws/aws-sdk-go-v2/config v1.32.6 h1:hFLBGUKjmLAekvi1evLi5hVvFQtSo3GYwi+Bx4lpJf8=
github.com/aws/aws-sdk-go-v2/config v1.32.6/go.mod h1:lcUL/gcd8WyjCrMnxez5OXkO3/rwcNmvfno62tnXNcI=
github.com/aws/aws-sdk-go-v2/credentials v1.19.6 h1:F9vWao2TwjV2MyiyVS+duza0NIRtAslgLUM0vTA1ZaE=
github.com/aws/aws-sdk-go-v2/credentials v1.19.6/go.mod h1:SgHzKjEVsdQr6Opor0ihgWtkWdfRAIwxYzSJ8O85VHY=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 h1:80+uETIWS1BqjnN9uJ0dBUaETh+P1XwFy5vwHwK5r9k=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16/go.mod h1:wOOsYuxYuB/7FlnVtzeBYRcjSRtQpAW0hCP7tIULMwo=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.18.4 h1:0SzCLoPRSK3qSydsaFQWugP+lOBCTPwfcBOm6222+UA=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.18.4/go.mod h1:JAet9FsBHjfdI+TnMBX4ModNNaQHAd3dc/Bk+cNsxeM=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.3/go.mod h1:br7KA6edAAqDGUYJ+zVVPAyMrPhnN+zdt17yTUT6FPw=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16 h1:rgGwPzb82iBYSvHMHXc8h9mRoOUBZIGFgKb9qniaZZc=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16/go.mod h1:L/UxsGeKpGoIj6DxfhOWHWQ/kGKcd4I1VncE4++IyKA=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16 h1:1jtGzuV7c82xnqOVfx2F0xmJcOw5374L7N6juGW6x6U=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16/go.mod h1:M2E5OQf+XLe+SZGmmpaI2yy+J326aFf6/+54PoxSANc=
github.com/aws/aws-sdk-go-v2/config v1.32.7 h1:vxUyWGUwmkQ2g19n7JY/9YL8MfAIl7bTesIUykECXmY=
github.com/aws/aws-sdk-go-v2/config v1.32.7/go.mod h1:2/Qm5vKUU/r7Y+zUk/Ptt2MDAEKAfUtKc1+3U1Mo3oY=
github.com/aws/aws-sdk-go-v2/credentials v1.19.7 h1:tHK47VqqtJxOymRrNtUXN5SP/zUTvZKeLx4tH6PGQc8=
github.com/aws/aws-sdk-go-v2/credentials v1.19.7/go.mod h1:qOZk8sPDrxhf+4Wf4oT2urYJrYt3RejHSzgAquYeppw=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.17 h1:I0GyV8wiYrP8XpA70g1HBcQO1JlQxCMTW9npl5UbDHY=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.17/go.mod h1:tyw7BOl5bBe/oqvoIeECFJjMdzXoa/dfVz3QQ5lgHGA=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.4 h1:2fjfz3/G9BRvIKuNZ655GwzpklC2kEH0cowZQGO7uBg=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.4/go.mod h1:Ymws824lvMypLFPwyyUXM52SXuGgxpu0+DISLfKvB+c=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.17 h1:xOLELNKGp2vsiteLsvLPwxC+mYmO6OZ8PYgiuPJzF8U=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.17/go.mod h1:5M5CI3D12dNOtH3/mk6minaRwI2/37ifCURZISxA/IQ=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.17 h1:WWLqlh79iO48yLkj1v3ISRNiv+3KdQoZ6JWyfcsyQik=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.17/go.mod h1:EhG22vHRrvF8oXSTYStZhJc1aUgKtnJe+aOiFEV90cM=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 h1:WKuaxf++XKWlHWu9ECbMlha8WOEGm0OUEZqm4K/Gcfk=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4/go.mod h1:ZWy7j6v1vWGmPReu0iSGvRiise4YI5SkR3OHKTZ6Wuc=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.15 h1:NLYTEyZmVZo0Qh183sC8nC+ydJXOOeIL/qI/sS3PdLY=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.15/go.mod h1:Z803iB3B0bc8oJV8zH2PERLRfQUJ2n2BXISpsA4+O1M=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.16 h1:CjMzUs78RDDv4ROu3JnJn/Ig1r6ZD7/T2DXLLRpejic=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.16/go.mod h1:uVW4OLBqbJXSHJYA9svT9BluSvvwbzLQ2Crf6UPzR3c=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4 h1:0ryTNEdJbzUCEWkVXEXoqlXV72J5keC1GvILMOuD00E=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4/go.mod h1:HQ4qwNZh32C3CBeO6iJLQlgtMzqeG17ziAA/3KDJFow=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.6 h1:P1MU/SuhadGvg2jtviDXPEejU3jBNhoeeAlRadHzvHI=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.6/go.mod h1:5KYaMG6wmVKMFBSfWoyG/zH8pWwzQFnKgpoSRlXHKdQ=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.16 h1:oHjJHeUy0ImIV0bsrX0X91GkV5nJAyv1l1CC9lnO0TI=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.16/go.mod h1:iRSNGgOYmiYwSCXxXaKb9HfOEj40+oTKn8pTxMlYkRM=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.15 h1:wsSQ4SVz5YE1crz0Ap7VBZrV4nNqZt4CIBBT8mnwoNc=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.15/go.mod h1:I7sditnFGtYMIqPRU1QoHZAUrXkGp4SczmlLwrNPlD0=
github.com/aws/aws-sdk-go-v2/service/s3 v1.93.0 h1:IrbE3B8O9pm3lsg96AXIN5MXX4pECEuExh/A0Du3AuI=
github.com/aws/aws-sdk-go-v2/service/s3 v1.93.0/go.mod h1:/sJLzHtiiZvs6C1RbxS/anSAFwZD6oC6M/kotQzOiLw=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 h1:HpI7aMmJ+mm1wkSHIA2t5EaFFv5EFYXePW30p1EIrbQ=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.4/go.mod h1:C5RdGMYGlfM0gYq/tifqgn4EbyX99V15P2V3R+VHbQU=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 h1:aM/Q24rIlS3bRAhTyFurowU8A0SMyGDtEOY/l/s/1Uw=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.8/go.mod h1:+fWt2UHSb4kS7Pu8y+BMBvJF0EWx+4H0hzNwtDNRTrg=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 h1:AHDr0DaHIAo8c9t1emrzAlVDFp+iMMKnPdYy6XO4MCE=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12/go.mod h1:GQ73XawFFiWxyWXMHWfhiomvP3tXtdNar/fi8z18sx0=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.5 h1:SciGFVNZ4mHdm7gpD1dgZYnCuVdX1s+lFTg4+4DOy70=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.5/go.mod h1:iW40X4QBmUxdP+fZNOpfmkdMZqsovezbAeO+Ubiv2pk=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.7 h1:DIBqIrJ7hv+e4CmIk2z3pyKT+3B6qVMgRsawHiR3qso=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.7/go.mod h1:vLm00xmBke75UmpNvOcZQ/Q30ZFjbczeLFqGx5urmGo=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.17 h1:RuNSMoozM8oXlgLG/n6WLaFGoea7/CddrCfIiSA+xdY=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.17/go.mod h1:F2xxQ9TZz5gDWsclCtPQscGpP0VUOc8RqgFM3vDENmU=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.16 h1:NSbvS17MlI2lurYgXnCOLvCFX38sBW4eiVER7+kkgsU=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.16/go.mod h1:SwT8Tmqd4sA6G1qaGdzWCJN99bUmPGHfRwwq3G5Qb+A=
github.com/aws/aws-sdk-go-v2/service/s3 v1.95.0 h1:MIWra+MSq53CFaXXAywB2qg9YvVZifkk6vEGl/1Qor0=
github.com/aws/aws-sdk-go-v2/service/s3 v1.95.0/go.mod h1:79S2BdqCJpScXZA2y+cpZuocWsjGjJINyXnOsf5DTz8=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.5 h1:VrhDvQib/i0lxvr3zqlUwLwJP4fpmpyD9wYG1vfSu+Y=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.5/go.mod h1:k029+U8SY30/3/ras4G/Fnv/b88N4mAfliNn08Dem4M=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.9 h1:v6EiMvhEYBoHABfbGB4alOYmCIrcgyPPiBE1wZAEbqk=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.9/go.mod h1:yifAsgBxgJWn3ggx70A3urX2AN49Y5sJTD1UQFlfqBw=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.13 h1:gd84Omyu9JLriJVCbGApcLzVR3XtmC4ZDPcAI6Ftvds=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.13/go.mod h1:sTGThjphYE4Ohw8vJiRStAcu3rbjtXRsdNB0TvZ5wwo=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.6 h1:5fFjR/ToSOzB2OQ/XqWpZBmNvmP/pJ1jOWYlFDJTjRQ=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.6/go.mod h1:qgFDZQSD/Kys7nJnVqYlWKnh0SSdMjAi0uSwON4wgYQ=
github.com/aws/smithy-go v1.24.0 h1:LpilSUItNPFr1eY85RYgTIg5eIEPtvFbskaFcmmIUnk=
github.com/aws/smithy-go v1.24.0/go.mod h1:LEj2LM3rBRQJxPZTB4KuzZkaZYnZPnvgIhb4pu07mx0=
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
github.com/boombuler/barcode v1.1.0 h1:ChaYjBR63fr4LFyGn8E8nt7dBSt3MiU3zMOZqFvVkHo=
github.com/boombuler/barcode v1.1.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
github.com/bradenaw/juniper v0.15.3 h1:RHIAMEDTpvmzV1wg1jMAHGOoI2oJUSPx3lxRldXnFGo=
github.com/bradenaw/juniper v0.15.3/go.mod h1:UX4FX57kVSaDp4TPqvSjkAAewmRFAfXf27BOs5z9dq8=
github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 h1:GKTyiRCL6zVf5wWaqKnf+7Qs6GbEPfd4iMOitWzXJx8=
@@ -154,11 +161,13 @@ github.com/buengese/sgzip v0.1.1 h1:ry+T8l1mlmiWEsDrH/YHZnCVWD2S3im1KLsyO+8ZmTU=
github.com/buengese/sgzip v0.1.1/go.mod h1:i5ZiXGF3fhV7gL1xaRRL1nDnmpNj0X061FQzOS8VMas=
github.com/bufbuild/protocompile v0.14.1 h1:iA73zAf/fyljNjQKwYzUHD6AD4R8KMasmwa/FBatYVw=
github.com/bufbuild/protocompile v0.14.1/go.mod h1:ppVdAIhbr2H8asPk6k4pY7t9zB1OU5DoEw9xY/FUi1c=
github.com/buger/jsonparser v1.1.1 h1:2PnMjfWD7wBILjqQbt530v576A/cAbQvEW9gGIpYMUs=
github.com/buger/jsonparser v1.1.1/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0=
github.com/bwesterb/go-ristretto v1.2.0/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0=
github.com/bytedance/sonic v1.14.0 h1:/OfKt8HFw0kh2rj8N0F6C/qPGRESq0BbaNZgcNXXzQQ=
github.com/bytedance/sonic v1.14.0/go.mod h1:WoEbx8WTcFJfzCe0hbmyTGrfjt8PzNEBdxlNUO24NhA=
github.com/bytedance/sonic/loader v0.3.0 h1:dskwH8edlzNMctoruo8FPTJDF3vLtDT0sXZwvZJyqeA=
github.com/bytedance/sonic/loader v0.3.0/go.mod h1:N8A3vUdtUebEY2/VQC0MyhYeKUFosQU6FxH2JmUe6VI=
github.com/bytedance/sonic v1.13.2 h1:8/H1FempDZqC4VqjptGo14QQlJx8VdZJegxs6wwfqpQ=
github.com/bytedance/sonic v1.13.2/go.mod h1:o68xyaF9u2gvVBuGHPlUVCy+ZfmNNO5ETf1+KgkJhz4=
github.com/bytedance/sonic/loader v0.2.4 h1:ZWCw4stuXUsn1/+zQDqeE7JKP+QO47tz7QCNan80NzY=
github.com/bytedance/sonic/loader v0.2.4/go.mod h1:N8A3vUdtUebEY2/VQC0MyhYeKUFosQU6FxH2JmUe6VI=
github.com/calebcase/tmpfile v1.0.3 h1:BZrOWZ79gJqQ3XbAQlihYZf/YCV0H4KPIdM5K5oMpJo=
github.com/calebcase/tmpfile v1.0.3/go.mod h1:UAUc01aHeC+pudPagY/lWvt2qS9ZO5Zzof6/tIUzqeI=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
@@ -170,17 +179,21 @@ github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWR
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/clipperhouse/stringish v0.1.1 h1:+NSqMOr3GR6k1FdRhhnXrLfztGzuG+VuFDfatpWHKCs=
github.com/clipperhouse/stringish v0.1.1/go.mod h1:v/WhFtE1q0ovMta2+m+UbpZ+2/HEXNWYXQgCt4hdOzA=
github.com/clipperhouse/uax29/v2 v2.3.0 h1:SNdx9DVUqMoBuBoW3iLOj4FQv3dN5mDtuqwuhIGpJy4=
github.com/clipperhouse/uax29/v2 v2.3.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g=
github.com/cloudflare/circl v1.1.0/go.mod h1:prBCrKB9DV4poKZY1l9zBXg2QJY7mvgRvtMxxK7fi4I=
github.com/cloudflare/circl v1.6.1 h1:zqIqSPIndyBh1bjLVVDHMPpVKqp8Su/V+6MeDzzQBQ0=
github.com/cloudflare/circl v1.6.1/go.mod h1:uddAzsPgqdMAYatqJ0lsjX1oECcQLIlRpzZh3pJrofs=
github.com/cloudinary/cloudinary-go/v2 v2.12.0 h1:uveBJeNpJztKDwFW/B+Wuklq584hQmQXlo+hGTSOGZ8=
github.com/cloudinary/cloudinary-go/v2 v2.12.0/go.mod h1:ireC4gqVetsjVhYlwjUJwKTbZuWjEIynbR9zQTlqsvo=
github.com/cloudflare/circl v1.6.3 h1:9GPOhQGF9MCYUeXyMYlqTR6a5gTrgR/fBLXvUgtVcg8=
github.com/cloudflare/circl v1.6.3/go.mod h1:2eXP6Qfat4O/Yhh8BznvKnJ+uzEoTQ6jVKJRn81BiS4=
github.com/cloudinary/cloudinary-go/v2 v2.13.0 h1:ugiQwb7DwpWQnete2AZkTh94MonZKmxD7hDGy1qTzDs=
github.com/cloudinary/cloudinary-go/v2 v2.13.0/go.mod h1:ireC4gqVetsjVhYlwjUJwKTbZuWjEIynbR9zQTlqsvo=
github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc h1:t8YjNUCt1DimB4HCIXBztwWMhgxr5yG5/YaRl9Afdfg=
github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc/go.mod h1:CgWpFCFWzzEA5hVkhAc6DZZzGd3czx+BblvOzjmg6KA=
github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc h1:0xCWmFKBmarCqqqLeM7jFBSw/Or81UEElFqO8MY+GDs=
github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc/go.mod h1:uvR42Hb/t52HQd7x5/ZLzZEK8oihrFpgnodIJ1vte2E=
github.com/cloudwego/base64x v0.1.6 h1:t11wG9AECkCDk5fMSoxmufanudBtJ+/HemLstXDLI2M=
github.com/cloudwego/base64x v0.1.6/go.mod h1:OFcloc187FXDaYHvrNIjxSe8ncn0OOM8gEHfghB2IPU=
github.com/cloudwego/base64x v0.1.5 h1:XPciSp1xaq2VCSt6lF0phncD4koWyULpl5bUxbfCyP4=
github.com/cloudwego/base64x v0.1.5/go.mod h1:0zlkT4Wn5C6NdauXdJRhSKRlJvmclQ1hhJgA0rcu/8w=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cognusion/imaging v1.0.2 h1:BQwBV8V8eF3+dwffp8Udl9xF1JKh5Z0z5JkJwAi98Mc=
github.com/cognusion/imaging v1.0.2/go.mod h1:mj7FvH7cT2dlFogQOSUQRtotBxJ4gFQ2ySMSmBm5dSk=
@@ -188,8 +201,8 @@ github.com/colinmarc/hdfs/v2 v2.4.0 h1:v6R8oBx/Wu9fHpdPoJJjpGSUxo8NhHIwrwsfhFvU9
github.com/colinmarc/hdfs/v2 v2.4.0/go.mod h1:0NAO+/3knbMx6+5pCv+Hcbaz4xn/Zzbn9+WIib2rKVI=
github.com/coreos/go-semver v0.3.1 h1:yi21YpKnrx1gt5R+la8n5WgS0kCrsPp33dmEyHReZr4=
github.com/coreos/go-semver v0.3.1/go.mod h1:irMmmIw/7yzSRPWryHsK7EYSg09caPQL03VsM8rvUec=
github.com/coreos/go-systemd/v22 v22.5.0 h1:RrqgGjYQKalulkV8NGVIfkXQf6YYmOyiJKk8iXXhfZs=
github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
github.com/coreos/go-systemd/v22 v22.6.0 h1:aGVa/v8B7hpb0TKl0MWoAavPDmHvobFe5R5zn0bCJWo=
github.com/coreos/go-systemd/v22 v22.6.0/go.mod h1:iG+pp635Fo7ZmV/j14KUcmEyWF+0X7Lua8rrTWzYgWU=
github.com/creasty/defaults v1.8.0 h1:z27FJxCAa0JKt3utc0sCImAEb+spPucmKoOdLHvHYKk=
github.com/creasty/defaults v1.8.0/go.mod h1:iGzKe6pbEHnpMPtfDXZEr0NVxWnPTjb1bbDy08fPzYM=
github.com/cronokirby/saferith v0.33.0 h1:TgoQlfsD4LIwx71+ChfRcIpjkw+RPOapDEVxa+LhwLo=
@@ -200,8 +213,14 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/diskfs/go-diskfs v1.7.0 h1:vonWmt5CMowXwUc79jWyGrf2DIMeoOjkLlMnQYGVOs8=
github.com/diskfs/go-diskfs v1.7.0/go.mod h1:LhQyXqOugWFRahYUSw47NyZJPezFzB9UELwhpszLP/k=
github.com/djherbis/times v1.6.0 h1:w2ctJ92J8fBvWPxugmXIv7Nz7Q3iDMKNx9v5ocVH20c=
github.com/djherbis/times v1.6.0/go.mod h1:gOHeRAz2h+VJNZ5Gmc/o7iD9k4wW7NMVqieYCY99oc0=
github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI=
github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ=
github.com/dromara/dongle v1.0.1 h1:si/7UP/EXxnFVZok1cNos70GiMGxInAYMilHQFP5dJs=
github.com/dromara/dongle v1.0.1/go.mod h1:ebFhTaDgxaDIKppycENTWlBsxz8mWCPWOLnsEgDpMv4=
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5 h1:FT+t0UEDykcor4y3dMVKXIiWJETBpRgERYTGlmMd7HU=
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5/go.mod h1:rSS3kM9XMzSQ6pw91Qgd6yB5jdt70N4OdtrAf74As5M=
github.com/dsnet/try v0.0.3 h1:ptR59SsrcFUYbT/FhAbKTV6iLkeD6O18qfIWRml2fqI=
@@ -216,10 +235,13 @@ github.com/eapache/queue v1.1.0 h1:YOEu7KNc61ntiQlcEeUIoDTJ2o8mQznoNvUhiigpIqc=
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
github.com/ebitengine/purego v0.9.1 h1:a/k2f2HQU3Pi399RPW1MOaZyhKJL9w/xFpKAg4q1s0A=
github.com/ebitengine/purego v0.9.1/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
github.com/elliotwutingfeng/asciiset v0.0.0-20230602022725-51bbb787efab h1:h1UgjJdAAhj+uPL68n7XASS6bU+07ZX1WJvVS2eyoeY=
github.com/elliotwutingfeng/asciiset v0.0.0-20230602022725-51bbb787efab/go.mod h1:GLo/8fDswSAniFG+BFIaiSPcK610jyzgEhWYPQwuQdw=
github.com/emersion/go-message v0.18.2 h1:rl55SQdjd9oJcIoQNhubD2Acs1E6IzlZISRTK7x/Lpg=
github.com/emersion/go-message v0.18.2/go.mod h1:XpJyL70LwRvq2a8rVbHXikPgKj8+aI0kGdHlg16ibYA=
github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff h1:4N8wnS3f1hNHSmFD5zgFkWCyA4L1kCDkImPAtK7D6tg=
github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff/go.mod h1:HMJKR5wlh/ziNp+sHEDV2ltblO4JD2+IdDOWtGcQBTM=
github.com/emmansun/gmsm v0.15.5/go.mod h1:2m4jygryohSWkaSduFErgCwQKab5BNjURoFrn2DNwyU=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
@@ -237,20 +259,22 @@ github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7z
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/gabriel-vasile/mimetype v1.4.9 h1:5k+WDwEsD9eTLL8Tz3L0VnmVh9QxGjRmjBvAG7U/oYY=
github.com/gabriel-vasile/mimetype v1.4.9/go.mod h1:WnSQhFKJuBlRyLiKohA/2DtIlPFAbguNaG7QCHcyGok=
github.com/gabriel-vasile/mimetype v1.4.11 h1:AQvxbp830wPhHTqc1u7nzoLT+ZFxGY7emj5DR5DYFik=
github.com/gabriel-vasile/mimetype v1.4.11/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
github.com/geoffgarside/ber v1.2.0 h1:/loowoRcs/MWLYmGX9QtIAbA+V/FrnVLsMMPhwiRm64=
github.com/geoffgarside/ber v1.2.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNecGT85ZCc=
github.com/gin-contrib/sse v1.1.0 h1:n0w2GMuUpWDVp7qSpvze6fAu9iRxJY4Hmj6AmBOU05w=
github.com/gin-contrib/sse v1.1.0/go.mod h1:hxRZ5gVpWMT7Z0B0gSNYqqsSCNIJMjzvm6fqCz9vjwM=
github.com/gin-contrib/sse v1.0.0 h1:y3bT1mUWUxDpW4JLQg/HnTqV4rozuW4tC9eFKTxYI9E=
github.com/gin-contrib/sse v1.0.0/go.mod h1:zNuFdwarAygJBht0NTKiSi3jRf6RbqeILZ9Sp6Slhe0=
github.com/gin-gonic/gin v1.11.0 h1:OW/6PLjyusp2PPXtyxKHU0RbX6I/l28FTdDlae5ueWk=
github.com/gin-gonic/gin v1.11.0/go.mod h1:+iq/FyxlGzII0KHiBGjuNn4UNENUlKbGlNmc+W50Dls=
github.com/go-chi/chi/v5 v5.2.2 h1:CMwsvRVTbXVytCk1Wd72Zy1LAsAh9GxMmSNWLHCG618=
github.com/go-chi/chi/v5 v5.2.2/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 h1:JnrjqG5iR07/8k7NqrLNilRsl3s1EPRQEGvbPyOce68=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348/go.mod h1:Czxo/d1g948LtrALAZdL04TL/HnkopquAjxYUuI02bo=
github.com/go-errors/errors v1.5.1 h1:ZwEMSLRCapFLflTpT7NKaAc7ukJ8ZPEjzlxt8rPN8bk=
github.com/go-errors/errors v1.5.1/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
github.com/go-git/go-billy/v5 v5.6.2 h1:6Q86EsPXMa7c3YZ3aLAQsMA0VlWmy43r6FHqa/UNbRM=
github.com/go-git/go-billy/v5 v5.6.2/go.mod h1:rcFC2rAsp/erv7CMz9GczHcuD0D32fWzH+MJAU+jaUU=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
@@ -264,18 +288,20 @@ github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE=
github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78=
github.com/go-openapi/errors v0.22.2 h1:rdxhzcBUazEcGccKqbY1Y7NS8FDcMyIRr0934jrYnZg=
github.com/go-openapi/errors v0.22.2/go.mod h1:+n/5UdIqdVnLIJ6Q9Se8HNGUXYaY6CN8ImWzfi/Gzp0=
github.com/go-openapi/strfmt v0.23.0 h1:nlUS6BCqcnAk0pyhi9Y+kdDVZdZMHfEKQiS4HaMgO/c=
github.com/go-openapi/strfmt v0.23.0/go.mod h1:NrtIpfKtWIygRkKVsxh7XQMDQW5HKQl6S5ik2elW+K4=
github.com/go-openapi/errors v0.22.4 h1:oi2K9mHTOb5DPW2Zjdzs/NIvwi2N3fARKaTJLdNabaM=
github.com/go-openapi/errors v0.22.4/go.mod h1:z9S8ASTUqx7+CP1Q8dD8ewGH/1JWFFLX/2PmAYNQLgk=
github.com/go-openapi/strfmt v0.25.0 h1:7R0RX7mbKLa9EYCTHRcCuIPcaqlyQiWNPTXwClK0saQ=
github.com/go-openapi/strfmt v0.25.0/go.mod h1:nNXct7OzbwrMY9+5tLX4I21pzcmE6ccMGXl3jFdPfn8=
github.com/go-openapi/testify/v2 v2.0.2 h1:X999g3jeLcoY8qctY/c/Z8iBHTbwLz7R2WXd6Ub6wls=
github.com/go-openapi/testify/v2 v2.0.2/go.mod h1:HCPmvFFnheKK2BuwSA0TbbdxJ3I16pjwMkYkP4Ywn54=
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
github.com/go-playground/locales v0.14.1/go.mod h1:hxrqLVvrK65+Rwrd5Fc6F2O76J/NuW9t0sjnWqG1slY=
github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJnYK9S473LQFuzCbDbfSFY=
github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
github.com/go-playground/validator/v10 v10.27.0 h1:w8+XrWVMhGkxOaaowyKH35gFydVHOvC0/uWoy2Fzwn4=
github.com/go-playground/validator/v10 v10.27.0/go.mod h1:I5QpIEbmr8On7W0TktmJAumgzX4CA1XNl4ZmDuVHKKo=
github.com/go-playground/validator/v10 v10.28.0 h1:Q7ibns33JjyW48gHkuFT91qX48KG0ktULL6FgHdG688=
github.com/go-playground/validator/v10 v10.28.0/go.mod h1:GoI6I1SjPBh9p7ykNE/yj3fFYbyDOpwMn5KXd+m2hUU=
github.com/go-resty/resty/v2 v2.16.5 h1:hBKqmWrr7uRc3euHVqmh1HTHcKn99Smr7o5spptdhTM=
github.com/go-resty/resty/v2 v2.16.5/go.mod h1:hkJtXbA2iKHzJheXYvQ8snQES5ZLGKMwQ07xAwp/fiA=
github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
@@ -286,16 +312,14 @@ github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/goccy/go-yaml v1.18.0 h1:8W7wMFS12Pcas7KU+VVkaiCng+kG8QiFeFwzFb+rwuw=
github.com/goccy/go-yaml v1.18.0/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
github.com/gofrs/flock v0.8.1/go.mod h1:F1TvTiK9OcQqauNUHlbJvyl9Qa1QvF/gOUDKA14jxHU=
github.com/gofrs/flock v0.12.1 h1:MTLVXXHf8ekldpJk3AKicLij9MdwOWkZ+a/jHHZby9E=
github.com/gofrs/flock v0.12.1/go.mod h1:9zxTsyu5xtJ9DK+1tFZyibEV7y3uwDxPPfbxeeHCoD0=
github.com/gofrs/flock v0.13.0 h1:95JolYOvGMqeH31+FC7D2+uULf6mG61mEZ/A8dRYMzw=
github.com/gofrs/flock v0.13.0/go.mod h1:jxeyy9R1auM5S6JYDBhDt+E2TCo7DkratH4Pgi8P+Z0=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXeUI=
github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang-jwt/jwt/v5 v5.3.0 h1:pv4AsKCKKZuqlgs5sUmn4x8UlGa0kEVt/puTpKx9vvo=
github.com/golang-jwt/jwt/v5 v5.3.0/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
@@ -355,15 +379,14 @@ github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0=
github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/enterprise-certificate-proxy v0.3.6 h1:GW/XbdyBFQ8Qe+YAmFU9uHLo7OnF5tL52HFAgMmyrf4=
github.com/googleapis/enterprise-certificate-proxy v0.3.6/go.mod h1:MkHOF77EYAE7qfSuSS9PU6g4Nt4e11cnsDUowfwewLA=
github.com/googleapis/enterprise-certificate-proxy v0.3.7 h1:zrn2Ee/nWmHulBx5sAVrGgAa0f2/R35S4DJwfFaUPFQ=
github.com/googleapis/enterprise-certificate-proxy v0.3.7/go.mod h1:MkHOF77EYAE7qfSuSS9PU6g4Nt4e11cnsDUowfwewLA=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/googleapis/gax-go/v2 v2.15.0 h1:SyjDc1mGgZU5LncH8gimWo9lW1DtIfPibOG81vgd/bo=
github.com/googleapis/gax-go/v2 v2.15.0/go.mod h1:zVVkkxAQHa1RQpg9z2AUCMnKhi0Qld9rcmyfL1OZhoc=
github.com/gopherjs/gopherjs v0.0.0-20181103185306-d547d1d9531e h1:JKmoR8x90Iww1ks85zJ1lfDGgIiMDuIptTOhJq+zKyg=
github.com/gopherjs/gopherjs v0.0.0-20181103185306-d547d1d9531e/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/gopherjs/gopherjs v1.17.2 h1:fQnZVsXk8uxXIStYb0N4bGk7jeyTalG/wsZjQ25dO0g=
github.com/gopherjs/gopherjs v1.17.2/go.mod h1:pRRIvn/QzFLrKfvEz3qUuEhtE/zLCWfreZ6J5gM2i+k=
github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
github.com/gorilla/schema v1.4.1 h1:jUg5hUjCSDZpNGLuXQOgIWGdlgrIdYvgQ0wZtdK1M3E=
@@ -390,14 +413,14 @@ github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/C
github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/henrybear327/Proton-API-Bridge v1.0.0 h1:gjKAaWfKu++77WsZTHg6FUyPC5W0LTKWQciUm8PMZb0=
github.com/henrybear327/Proton-API-Bridge v1.0.0/go.mod h1:gunH16hf6U74W2b9CGDaWRadiLICsoJ6KRkSt53zLts=
github.com/henrybear327/go-proton-api v1.0.0 h1:zYi/IbjLwFAW7ltCeqXneUGJey0TN//Xo851a/BgLXw=
github.com/henrybear327/go-proton-api v1.0.0/go.mod h1:w63MZuzufKcIZ93pwRgiOtxMXYafI8H74D77AxytOBc=
github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/internxt/rclone-adapter v0.0.0-20260213125353-6f59c89fcb7c h1:r+KtxPyrhsYeNbsfeqTfEM8xRdwgV6LuNhLZxpXecb4=
github.com/internxt/rclone-adapter v0.0.0-20260213125353-6f59c89fcb7c/go.mod h1:vdPya4AIcDjvng4ViaAzqjegJf0VHYpYHQguFx5xBp0=
github.com/jcmturner/aescts/v2 v2.0.0 h1:9YKLH6ey7H4eDBXW8khjYslgyqG2xZikXP0EQFKrle8=
github.com/jcmturner/aescts/v2 v2.0.0/go.mod h1:AiaICIRyfYg35RUkr8yESTqvSy7csK90qZ5xfvvsoNs=
github.com/jcmturner/dnsutils/v2 v2.0.0 h1:lltnkeZGL0wILNvrNiVCR6Ro5PGU/SeBvVO/8c/iPbo=
@@ -410,8 +433,10 @@ github.com/jcmturner/gokrb5/v8 v8.4.4 h1:x1Sv4HaTpepFkXbt2IkL29DXRf8sOfZXo8eRKh6
github.com/jcmturner/gokrb5/v8 v8.4.4/go.mod h1:1btQEpgT6k+unzCwX1KdWMEwPPkkgBtP+F6aCACiMrs=
github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY=
github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc=
github.com/jhump/protoreflect v1.17.0 h1:qOEr613fac2lOuTgWN4tPAtLL7fUSbuJL5X5XumQh94=
github.com/jhump/protoreflect v1.17.0/go.mod h1:h9+vUUL38jiBzck8ck+6G/aeMX8Z4QUY/NiJPwPNi+8=
github.com/jhump/protoreflect v1.18.0 h1:TOz0MSR/0JOZ5kECB/0ufGnC2jdsgZ123Rd/k4Z5/2w=
github.com/jhump/protoreflect v1.18.0/go.mod h1:ezWcltJIVF4zYdIFM+D/sHV4Oh5LNU08ORzCGfwvTz8=
github.com/jhump/protoreflect/v2 v2.0.0-beta.1 h1:Dw1rslK/VotaUGYsv53XVWITr+5RCPXfvvlGrM/+B6w=
github.com/jhump/protoreflect/v2 v2.0.0-beta.1/go.mod h1:D9LBEowZyv8/iSu97FU2zmXG3JxVTmNw21mu63niFzU=
github.com/jlaffaye/ftp v0.2.1-0.20240918233326-1b970516f5d3 h1:ZxO6Qr2GOXPdcW80Mcn3nemvilMPvpWqxrNfK2ZnNNs=
github.com/jlaffaye/ftp v0.2.1-0.20240918233326-1b970516f5d3/go.mod h1:dvLUr/8Fs9a2OBrEnCC5duphbkz/k/mSy5OkXg3PAgI=
github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
@@ -436,12 +461,12 @@ github.com/keybase/go-keychain v0.0.1 h1:way+bWYa6lDppZoZcgMbYsvC7GxljxrskdNInRt
github.com/keybase/go-keychain v0.0.1/go.mod h1:PdEILRW3i9D8JcdM+FmY6RwkHGnhHxXwkPPMeUgOK1k=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/klauspost/compress v1.18.4 h1:RPhnKRAQ4Fh8zU2FY/6ZFDwTVTxgJ/EMydqSTzE9a2c=
github.com/klauspost/compress v1.18.4/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/klauspost/reedsolomon v1.12.6 h1:8pqE9aECQG/ZFitiUD1xK/E83zwosBAZtE3UbuZM8TQ=
github.com/klauspost/reedsolomon v1.12.6/go.mod h1:ggJT9lc71Vu+cSOPBlxGvBN6TfAS77qB4fp8vJ05NSA=
github.com/klauspost/reedsolomon v1.13.0 h1:E0Cmgf2kMuhZTj6eefnvpKC4/Q4jhCi9YIjcZjK4arc=
github.com/klauspost/reedsolomon v1.13.0/go.mod h1:ggJT9lc71Vu+cSOPBlxGvBN6TfAS77qB4fp8vJ05NSA=
github.com/koofr/go-httpclient v0.0.0-20240520111329-e20f8f203988 h1:CjEMN21Xkr9+zwPmZPaJJw+apzVbjGL5uK/6g9Q2jGU=
github.com/koofr/go-httpclient v0.0.0-20240520111329-e20f8f203988/go.mod h1:/agobYum3uo/8V6yPVnq+R82pyVGCeuWW5arT4Txn8A=
github.com/koofr/go-koofrclient v0.0.0-20221207135200-cbd7fc9ad6a6 h1:FHVoZMOVRA+6/y4yRlbiR3WvsrOcKBd/f64H7YiWR2U=
@@ -458,35 +483,37 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/lanrat/extsort v1.4.0 h1:jysS/Tjnp7mBwJ6NG8SY+XYFi8HF3LujGbqY9jOWjco=
github.com/lanrat/extsort v1.4.0/go.mod h1:hceP6kxKPKebjN1RVrDBXMXXECbaI41Y94tt6MDazc4=
github.com/lanrat/extsort v1.4.2 h1:akbLIdo4PhNZtvjpaWnbXtGMmLtnGzXplkzfgl+XTTY=
github.com/lanrat/extsort v1.4.2/go.mod h1:hceP6kxKPKebjN1RVrDBXMXXECbaI41Y94tt6MDazc4=
github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
github.com/linkedin/goavro/v2 v2.14.1 h1:/8VjDpd38PRsy02JS0jflAu7JZPfJcGTwqWgMkFS2iI=
github.com/linkedin/goavro/v2 v2.14.1/go.mod h1:KXx+erlq+RPlGSPmLF7xGo6SAbh8sCQ53x064+ioxhk=
github.com/lpar/date v1.0.0 h1:bq/zVqFTUmsxvd/CylidY4Udqpr9BOFrParoP6p0x/I=
github.com/lpar/date v1.0.0/go.mod h1:KjYe0dDyMQTgpqcUz4LEIeM5VZwhggjVx/V2dtc8NSo=
github.com/lufia/plan9stats v0.0.0-20250317134145-8bc96cf8fc35 h1:PpXWgLPs+Fqr325bN2FD2ISlRRztXibcX6e8f5FR5Dc=
github.com/lufia/plan9stats v0.0.0-20250317134145-8bc96cf8fc35/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg=
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3 h1:PwQumkgq4/acIiZhtifTV5OUqqiP82UAl0h87xj/l9k=
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg=
github.com/mailru/easyjson v0.9.1 h1:LbtsOm5WAswyWbvTEOqhypdPeZzHavpZx96/n553mR8=
github.com/mailru/easyjson v0.9.1/go.mod h1:1+xMtQp2MRNVL/V1bOzuP3aP8VNwRW55fQUto+XFtTU=
github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-runewidth v0.0.3/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
github.com/mattn/go-runewidth v0.0.16 h1:E5ScNMtiwvlvB5paMFdw9p4kSQzbXFikJ5SQO6TULQc=
github.com/mattn/go-runewidth v0.0.16/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mattn/go-runewidth v0.0.19 h1:v++JhqYnZuu5jSKrk9RbgF5v4CGUjqRfBm05byFGLdw=
github.com/mattn/go-runewidth v0.0.19/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4 h1:BpfhmLKZf+SjVanKKhCgf3bg+511DmU9eDQTen7LLbY=
github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/moby/sys/mountinfo v0.7.2 h1:1shs6aH5s4o5H2zQLn796ADW1wMrIwHsyJ2v9KouLrg=
github.com/moby/sys/mountinfo v0.7.2/go.mod h1:1YOa8w8Ih7uW0wALDUgT1dTTSBrZ+HiBLGws92L2RU4=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/ncw/swift/v2 v2.0.4 h1:hHWVFxn5/YaTWAASmn4qyq2p6OyP/Hm3vMLzkjEqR7w=
github.com/ncw/swift/v2 v2.0.4/go.mod h1:cbAO76/ZwcFrFlHdXPjaqWZ9R7Hdar7HpjRXBfbjigk=
github.com/ncw/swift/v2 v2.0.5 h1:9o5Gsd7bInAFEqsGPcaUdsboMbqf8lnNtxqWKFT9iz8=
github.com/ncw/swift/v2 v2.0.5/go.mod h1:cbAO76/ZwcFrFlHdXPjaqWZ9R7Hdar7HpjRXBfbjigk=
github.com/nxadm/tail v1.4.11 h1:8feyoE3OzPrcshW5/MJ4sGESc5cqmGkGCWlco4l0bqY=
github.com/nxadm/tail v1.4.11/go.mod h1:OTaG3NK980DZzxbRq6lEuzgU+mug70nY11sMd4JXXHc=
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
@@ -500,15 +527,18 @@ github.com/onsi/ginkgo/v2 v2.23.3/go.mod h1:zXTP6xIp3U8aVuXN8ENK9IXRaTjFnpVB9mGm
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/onsi/gomega v1.37.0 h1:CdEG8g0S133B4OswTDC/5XPSzE1OeP29QOioj2PID2Y=
github.com/onsi/gomega v1.37.0/go.mod h1:8D9+Txp43QWKhM24yyOBEdpkzN8FvJyAwecBgsU4KU0=
github.com/oracle/oci-go-sdk/v65 v65.98.0 h1:ZKsy97KezSiYSN1Fml4hcwjpO+wq01rjBkPqIiUejVc=
github.com/oracle/oci-go-sdk/v65 v65.98.0/go.mod h1:RGiXfpDDmRRlLtqlStTzeBjjdUNXyqm3KXKyLCm3A/Q=
github.com/oracle/oci-go-sdk/v65 v65.104.0 h1:l9awEvzWvxmYhy/97A0hZ87pa7BncYXmcO/S8+rvgK0=
github.com/oracle/oci-go-sdk/v65 v65.104.0/go.mod h1:oB8jFGVc/7/zJ+DbleE8MzGHjhs2ioCz5stRTdZdIcY=
github.com/orcaman/concurrent-map/v2 v2.0.1 h1:jOJ5Pg2w1oeB6PeDurIYf6k9PQ+aTITr/6lP/L/zp6c=
github.com/orcaman/concurrent-map/v2 v2.0.1/go.mod h1:9Eq3TG2oBe5FirmYWQfYO5iH1q0Jv47PLaNK++uCdOM=
github.com/panjf2000/ants/v2 v2.11.3 h1:AfI0ngBoXJmYOpDh9m516vjqoUu2sLrIVgppI9TZVpg=
github.com/panjf2000/ants/v2 v2.11.3/go.mod h1:8u92CYMUc6gyvTIw8Ru7Mt7+/ESnJahz5EVtqfrilek=
github.com/parquet-go/parquet-go v0.25.1 h1:l7jJwNM0xrk0cnIIptWMtnSnuxRkwq53S+Po3KG8Xgo=
github.com/parquet-go/parquet-go v0.25.1/go.mod h1:AXBuotO1XiBtcqJb/FKFyjBG4aqa3aQAAWF3ZPzCanY=
github.com/parquet-go/parquet-go v0.26.3/go.mod h1:h9GcSt41Knf5qXI1tp1TfR8bDBUtvdUMzSKe26aZcHk=
github.com/parquet-go/bitpack v1.0.0 h1:AUqzlKzPPXf2bCdjfj4sTeacrUwsT7NlcYDMUQxPcQA=
github.com/parquet-go/bitpack v1.0.0/go.mod h1:XnVk9TH+O40eOOmvpAVZ7K2ocQFrQwysLMnc6M/8lgs=
github.com/parquet-go/jsonlite v1.0.0 h1:87QNdi56wOfsE5bdgas0vRzHPxfJgzrXGml1zZdd7VU=
github.com/parquet-go/jsonlite v1.0.0/go.mod h1:nDjpkpL4EOtqs6NQugUsi0Rleq9sW/OtC1NnZEnxzF0=
github.com/parquet-go/parquet-go v0.26.4 h1:zJ3l8ef5WJZE2m63pKwyEJ2BhyDlgS0PfOEhuCQQU2A=
github.com/parquet-go/parquet-go v0.26.4/go.mod h1:h9GcSt41Knf5qXI1tp1TfR8bDBUtvdUMzSKe26aZcHk=
github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
@@ -517,8 +547,10 @@ github.com/pengsrc/go-shared v0.2.1-0.20190131101655-1999055a4a14 h1:XeOYlK9W1uC
github.com/pengsrc/go-shared v0.2.1-0.20190131101655-1999055a4a14/go.mod h1:jVblp62SafmidSkvWrXyxAme3gaTfEtWwRPGz5cpvHg=
github.com/peterh/liner v1.2.2 h1:aJ4AOodmL+JxOZZEL2u9iJf8omNRpqHc/EbrK+3mAXw=
github.com/peterh/liner v1.2.2/go.mod h1:xFwJyiKIXJZUKItq5dGHZSTBRAuG/CpeNpWLyiNRNwI=
github.com/pierrec/lz4/v4 v4.1.22 h1:cKFw6uJDK+/gfw5BcDL0JL5aBsAFdsIT18eRtLj7VIU=
github.com/pierrec/lz4/v4 v4.1.22/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/petermattis/goid v0.0.0-20260113132338-7c7de50cc741 h1:KPpdlQLZcHfTMQRi6bFQ7ogNO0ltFT4PmtwTLW4W+14=
github.com/petermattis/goid v0.0.0-20260113132338-7c7de50cc741/go.mod h1:pxMtw7cyUw6B2bRH0ZBANSPg+AoSud1I1iyJHI69jH4=
github.com/pierrec/lz4/v4 v4.1.25 h1:kocOqRffaIbU5djlIBr7Wh+cx82C0vtFb0fOurZHqD0=
github.com/pierrec/lz4/v4 v4.1.25/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c/go.mod h1:7rwL4CYBLnjLxUqIJNnCWiEdr3bn6IUYi15bNlnbCCU=
github.com/pkg/diff v0.0.0-20200914180035-5b29258ca4f7/go.mod h1:zO8QMzTeZd5cpnIkz/Gn6iK0jDfGicM1nynOkkPIl28=
@@ -533,13 +565,15 @@ github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRI
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/pquerna/otp v1.5.0 h1:NMMR+WrmaqXU4EzdGJEE1aUUI0AMRzsp96fFFWNPwxs=
github.com/pquerna/otp v1.5.0/go.mod h1:dkJfzwRKNiegxyNb54X/3fLwhCynbMspSyWKnvi1AEg=
github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs=
github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA=
github.com/prometheus/common v0.67.2 h1:PcBAckGFTIHt2+L3I33uNRTlKTplNzFctXcWhPyAEN8=
github.com/prometheus/common v0.67.2/go.mod h1:63W3KZb1JOKgcjlIr64WW/LvFGAqKPj0atm+knVGEko=
github.com/prometheus/procfs v0.19.2 h1:zUMhqEW66Ex7OXIiDkll3tl9a1ZdilUOd/F6ZXw4Vws=
github.com/prometheus/procfs v0.19.2/go.mod h1:M0aotyiemPhBCM0z5w87kL22CxfcH05ZpYlu+b4J7mw=
github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8 h1:Y258uzXU/potCYnQd1r6wlAnoMB68BiCkCcCnKx1SH8=
@@ -548,21 +582,22 @@ github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
github.com/quic-go/quic-go v0.57.0 h1:AsSSrrMs4qI/hLrKlTH/TGQeTMY0ib1pAOX7vA3AdqE=
github.com/quic-go/quic-go v0.57.0/go.mod h1:ly4QBAjHA2VhdnxhojRsCUOeJwKYg+taDlos92xb1+s=
github.com/rclone/rclone v1.71.2 h1:3Jk5xNPFrZhVABRuN/OPvApuZQddpE2tkhYMuEn1Ud4=
github.com/rclone/rclone v1.71.2/go.mod h1:dCK9FzPDlpkbQJ9M7MmWsmv3X5nibfWe+ogJXu6gSgM=
github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11 h1:4MI2alxM/Ye2gIRBlYf28JGWTipZ4Zz7yAziPKrttjs=
github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11/go.mod h1:3HLX7dwZgvB7nt+Yl/xdzVPcargQ1yBmJEUg3n+jMKM=
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18 h1:Lc+d3ISfQaMJKWZOE7z4ZSY4RVmdzbn1B0IM8xN18qM=
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18/go.mod h1:LB2kCEaZMzNn3ocdz+qYfxXmuLxxN0ka62KJd2x53Bc=
github.com/rclone/rclone v1.73.1 h1:FCGZUd7sXO8+nIpbuXGPkd+f23o/ZACefrHtRGDAiDM=
github.com/rclone/rclone v1.73.1/go.mod h1:T9HeSXv/SI9qJFkYLqsjY/5ljIFWJTVmTrG7H/4JrPY=
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 h1:bsUq1dX0N8AOIL7EB/X911+m4EHsnWEHeJ0c+3TTBrg=
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/rdleal/intervalst v1.5.0 h1:SEB9bCFz5IqD1yhfH1Wv8IBnY/JQxDplwkxHjT6hamU=
github.com/rdleal/intervalst v1.5.0/go.mod h1:xO89Z6BC+LQDH+IPQQw/OESt5UADgFD41tYMUINGpxQ=
github.com/relvacode/iso8601 v1.6.0 h1:eFXUhMJN3Gz8Rcq82f9DTMW0svjtAVuIEULglM7QHTU=
github.com/relvacode/iso8601 v1.6.0/go.mod h1:FlNp+jz+TXpyRqgmM7tnzHHzBnz776kmAH2h3sZCn0I=
github.com/relvacode/iso8601 v1.7.0 h1:BXy+V60stMP6cpswc+a93Mq3e65PfXCgDFfhvNNGrdo=
github.com/relvacode/iso8601 v1.7.0/go.mod h1:FlNp+jz+TXpyRqgmM7tnzHHzBnz776kmAH2h3sZCn0I=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/rfjakob/eme v1.1.2 h1:SxziR8msSOElPayZNFfQw4Tjx/Sbaeeh3eRvrHVMUs4=
github.com/rfjakob/eme v1.1.2/go.mod h1:cVvpasglm/G3ngEfcfT/Wt0GwhkuO32pf/poW6Nyk1k=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
@@ -570,18 +605,18 @@ github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 h1:OkMGxebDj
github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06/go.mod h1:+ePHsJ1keEjQtpvf9HHw0f4ZeJ0TLRsxhunSI2hYJSs=
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
github.com/samber/lo v1.51.0 h1:kysRYLbHy/MB7kQZf5DSN50JHmMsNEdeY24VzJFu7wI=
github.com/samber/lo v1.51.0/go.mod h1:4+MXEGsJzbKGaUEQFKBq2xtfuznW9oz/WrgyzMzRoM0=
github.com/samber/lo v1.52.0 h1:Rvi+3BFHES3A8meP33VPAxiBZX/Aws5RxrschYGjomw=
github.com/samber/lo v1.52.0/go.mod h1:4+MXEGsJzbKGaUEQFKBq2xtfuznW9oz/WrgyzMzRoM0=
github.com/seaweedfs/goexif v1.0.3 h1:ve/OjI7dxPW8X9YQsv3JuVMaxEyF9Rvfd04ouL+Bz30=
github.com/seaweedfs/goexif v1.0.3/go.mod h1:Oni780Z236sXpIQzk1XoJlTwqrJ02smEin9zQeff7Fk=
github.com/segmentio/kafka-go v0.4.49 h1:GJiNX1d/g+kG6ljyJEoi9++PUMdXGAxb7JGPiDCuNmk=
github.com/segmentio/kafka-go v0.4.49/go.mod h1:Y1gn60kzLEEaW28YshXyk2+VCUKbJ3Qr6DrnT3i4+9E=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/shirou/gopsutil/v4 v4.25.11 h1:X53gB7muL9Gnwwo2evPSE+SfOrltMoR6V3xJAXZILTY=
github.com/shirou/gopsutil/v4 v4.25.11/go.mod h1:EivAfP5x2EhLp2ovdpKSozecVXn1TmuG7SMzs/Wh4PU=
github.com/shirou/gopsutil/v4 v4.26.1 h1:TOkEyriIXk2HX9d4isZJtbjXbEjf5qyKPAzbzY0JWSo=
github.com/shirou/gopsutil/v4 v4.26.1/go.mod h1:medLI9/UNAb0dOI9Q3/7yWSqKkj00u+1tgY8nvv41pc=
github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/sirupsen/logrus v1.9.4-0.20230606125235-dd1b4c2e81af h1:Sp5TG9f7K39yfB+If0vjp97vuT74F72r8hfRpP8jLU0=
github.com/sirupsen/logrus v1.9.4-0.20230606125235-dd1b4c2e81af/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966 h1:JIAuq3EEf9cgbU6AtGPK4CTG3Zf6CKMNqf0MHTggAUA=
github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966/go.mod h1:sUM3LWHvSMaG192sy56D9F7CNvL7jUJVXoqM1QKLnog=
github.com/smarty/assertions v1.16.0 h1:EvHNkdRA4QHMrn75NZSoUQ/mAUXAYWfatfB01yTCzfY=
@@ -590,17 +625,18 @@ github.com/smartystreets/goconvey v1.8.1 h1:qGjIddxOk4grTu9JPOU31tVfq3cNdBlNa5sS
github.com/smartystreets/goconvey v1.8.1/go.mod h1:+/u4qLyY6x1jReYOp7GOM2FSt8aP9CzCZL03bI28W60=
github.com/snabb/httpreaderat v1.0.1 h1:whlb+vuZmyjqVop8x1EKOg05l2NE4z9lsMMXjmSUCnY=
github.com/snabb/httpreaderat v1.0.1/go.mod h1:lpbGrKDWF37yvRbtRvQsbesS6Ty5c83t8ztannPoMsA=
github.com/sony/gobreaker v0.5.0/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY=
github.com/sony/gobreaker v1.0.0 h1:feX5fGGXSl3dYd4aHZItw+FpHLvvoaqkawKjVNiFMNQ=
github.com/sony/gobreaker v1.0.0/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8/go.mod h1:3n1Cwaq1E1/1lhQhtRK2ts/ZwZEhjcQeJQ1RuC6Q/8U=
github.com/spacemonkeygo/monkit/v3 v3.0.24 h1:cKixJ+evHnfJhWNyIZjBy5hoW8LTWmrJXPo18tzLNrk=
github.com/spacemonkeygo/monkit/v3 v3.0.24/go.mod h1:XkZYGzknZwkD0AKUnZaSXhRiVTLCkq7CWVa3IsE72gA=
github.com/spacemonkeygo/monkit/v3 v3.0.25-0.20251022131615-eb24eb109368 h1:GyYC5Ntqk/yy9lEIGE7chdIvt4zP44taycwd9YDSGdc=
github.com/spacemonkeygo/monkit/v3 v3.0.25-0.20251022131615-eb24eb109368/go.mod h1:XkZYGzknZwkD0AKUnZaSXhRiVTLCkq7CWVa3IsE72gA=
github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/cobra v1.10.1 h1:lJeBwCfmrnXthfAupyUTzJ/J4Nc1RsHC/mSRU2dll/s=
github.com/spf13/cobra v1.10.1/go.mod h1:7SmJGaTHFVBY0jW4NXGluQoLvhqFQM+6XSKD+P4XaB0=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
@@ -622,15 +658,14 @@ github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.7.5/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965 h1:1oFLiOyVl+W7bnBzGhf7BbIv9loSFQcieWWYIjLqcAw=
github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965/go.mod h1:9OrXJhf154huy1nPWmuSrkgjPUtUNhA+Zmy+6AESzuA=
github.com/t3rm1n4l/go-mega v0.0.0-20250926104142-ccb8d3498e6c h1:BLopNCyqewbE8+BtlIp/Juzu8AJGxz0gHdGADnsblVc=
github.com/t3rm1n4l/go-mega v0.0.0-20250926104142-ccb8d3498e6c/go.mod h1:ykucQyiE9Q2qx1wLlEtZkkNn1IURib/2O+Mvd25i1Fo=
github.com/t3rm1n4l/go-mega v0.0.0-20251031123324-a804aaa87491 h1:rrGZv6xYk37hx0tW2sYfgbO0PqStbHqz6Bq6oc9Hurg=
github.com/t3rm1n4l/go-mega v0.0.0-20251031123324-a804aaa87491/go.mod h1:ykucQyiE9Q2qx1wLlEtZkkNn1IURib/2O+Mvd25i1Fo=
github.com/tailscale/depaware v0.0.0-20210622194025-720c4b409502/go.mod h1:p9lPsd+cx33L3H9nNoecRRxPssFKUwwI50I3pZ0yT+8=
github.com/tklauser/go-sysconf v0.3.16 h1:frioLaCQSsF5Cy1jgRBrzr6t502KIIwQ0MArYICU0nA=
github.com/tklauser/go-sysconf v0.3.16/go.mod h1:/qNL9xxDhc7tx3HSRsLWNnuzbVfh3e7gh/BmM179nYI=
@@ -638,10 +673,14 @@ github.com/tklauser/numcpus v0.11.0 h1:nSTwhKH5e1dMNsCdVBukSZrURJRoHbSEQjdEbY+9R
github.com/tklauser/numcpus v0.11.0/go.mod h1:z+LwcLq54uWZTX0u/bGobaV34u6V7KNlTZejzM6/3MQ=
github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS4MhqMhdFk5YI=
github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08=
github.com/tyler-smith/go-bip39 v1.1.0 h1:5eUemwrMargf3BSLRRCalXT93Ns6pQJIjYQN2nyfOP8=
github.com/tyler-smith/go-bip39 v1.1.0/go.mod h1:gUYDtqQw1JS3ZJ8UWVcGTGqqr6YIN3CWg+kkNaLt55U=
github.com/tylertreat/BoomFilters v0.0.0-20210315201527-1a82519a3e43 h1:QEePdg0ty2r0t1+qwfZmQ4OOl/MB2UXIeJSpIZv56lg=
github.com/tylertreat/BoomFilters v0.0.0-20210315201527-1a82519a3e43/go.mod h1:OYRfF6eb5wY9VRFkXJH8FFBi3plw2v+giaIu7P054pM=
github.com/ugorji/go/codec v1.3.0 h1:Qd2W2sQawAfG8XSvzwhBeoGq71zXOC/Q1E9y/wUcsUA=
github.com/ugorji/go/codec v1.3.0/go.mod h1:pRBVtBSKl77K30Bv8R2P+cLSGaTtex6fsA2Wjqmfxj4=
github.com/ugorji/go/codec v1.2.12 h1:9LC83zGrHhuUA9l16C9AHXAqEV/2wBQ4nkvumAE65EE=
github.com/ugorji/go/codec v1.2.12/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg=
github.com/ulikunitz/xz v0.5.15 h1:9DNdB5s+SgV3bQ2ApL10xRc35ck0DuIX/isZvIk+ubY=
github.com/ulikunitz/xz v0.5.15/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
github.com/unknwon/goconfig v1.0.0 h1:rS7O+CmUdli1T+oDm7fYj1MwqNWtEJfNj+FqcUHML8U=
github.com/unknwon/goconfig v1.0.0/go.mod h1:qu2ZQ/wcC/if2u32263HTVC39PeOQRSmidQk3DuDFQ8=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
@@ -652,6 +691,8 @@ github.com/viant/ptrie v1.0.1 h1:3fFC8XqCSchf11sCSS5sbb8eGDNEP2g2Hj96lNdHlZY=
github.com/viant/ptrie v1.0.1/go.mod h1:Y+mwwNCIUgFrCZcrG4/QChfi4ubvnNBsyrENBIgigu0=
github.com/viant/toolbox v0.34.5 h1:szWNPiGHjo8Dd4v2a59saEhG31DRL2Xf3aJ0ZtTSuqc=
github.com/viant/toolbox v0.34.5/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM=
github.com/wk8/go-ordered-map/v2 v2.1.8 h1:5h/BUHu93oj4gIdvHHHGsScSTMijfx5PeYkE/fJgbpc=
github.com/wk8/go-ordered-map/v2 v2.1.8/go.mod h1:5nJHM5DyteebpVlHnWMV0rPz6Zp7+xBAnxjb1X5vnTw=
github.com/wsxiaoys/terminal v0.0.0-20160513160801-0940f3fc43a0 h1:3UeQBvD0TFrlVjOeLOBz+CPAI8dnbqNSVwUwRrkp7vQ=
github.com/wsxiaoys/terminal v0.0.0-20160513160801-0940f3fc43a0/go.mod h1:IXCdmsXIht47RaVFLEdVnh1t+pgYtTAhQGj73kz+2DM=
github.com/xanzy/ssh-agent v0.3.3 h1:+/15pJfg/RsTxqYcX6fHqOXZwwMP+2VyYWJeWM2qQFM=
@@ -662,8 +703,9 @@ github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY=
github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3kKLN4=
github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8=
github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM=
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f h1:J9EGpcZtP0E/raorCMxlFGSTBrsSlaDGf3jU/qvAE2c=
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU=
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb h1:zGWFAtiMcyryUHoUjUJX0/lt1H2+i2Ka2n+D3DImSNo=
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU=
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 h1:EzJWgHovont7NscjpAxXsDA8S8BMYve8Y5+7cuRE7R0=
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415/go.mod h1:GwrjFmJcFw6At/Gs6z4yjiIwzuJ1/+UwLxMQDVQXShQ=
github.com/xeipuuv/gojsonschema v1.2.0 h1:LhYJRs+L4fBtjZUfuSZIKGeVu0QRy8e5Xi7D17UxZ74=
@@ -691,8 +733,8 @@ github.com/zeebo/pcg v1.0.1 h1:lyqfGeWiv4ahac6ttHs+I5hwtH/+1mrhlCtVNQM2kHo=
github.com/zeebo/pcg v1.0.1/go.mod h1:09F0S9iiKrwn9rlI5yjLkmrug154/YRW6KnnXVDM/l4=
github.com/zeebo/xxh3 v1.0.2 h1:xZmwmqxHZA8AI603jOQ0tMqmBr9lPeFwGg6d+xy9DC0=
github.com/zeebo/xxh3 v1.0.2/go.mod h1:5NWz9Sef7zIDm2JHfFlcQvNekmcEl9ekUZQQKCYaDcA=
go.etcd.io/bbolt v1.4.2 h1:IrUHp260R8c+zYx/Tm8QZr04CX+qWS5PGfPdevhdm1I=
go.etcd.io/bbolt v1.4.2/go.mod h1:Is8rSHO/b4f3XigBC0lL0+4FwAQv3HXEEIgFMuKHceM=
go.etcd.io/bbolt v1.4.3 h1:dEadXpI6G79deX5prL3QRNP6JB8UxVkqo4UPnHaNXJo=
go.etcd.io/bbolt v1.4.3/go.mod h1:tKQlpPaYCVFctUIgFKFnAlvbmB3tpy1vkTnDWohtc0E=
go.mongodb.org/mongo-driver v1.17.6 h1:87JUG1wZfWsr6rIz3ZmpH90rL5tea7O3IHuSwHUpsss=
go.mongodb.org/mongo-driver v1.17.6/go.mod h1:Hy04i7O2kC4RS06ZrhPRqj/u4DTYkFDAAccj+rVKqgQ=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
@@ -702,8 +744,8 @@ go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0 h1:Hf9xI/XLML9ElpiHVDNwvqI0hIFlzV8dgIr35kV1kRU=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0/go.mod h1:NfchwuyNoMcZ5MLHwPrODwUF1HWCXWrL31s8gSAdIKY=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0 h1:RbKq8BG0FI8OiXhBfcRtqqHcZcka+gU3cskNuf05R18=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0/go.mod h1:h06DGIukJOevXaj/xrNjhi/2098RZzcLTbc0jDAUbsg=
go.opentelemetry.io/otel v1.38.0 h1:RkfdswUDRimDg0m2Az18RKOsnI8UDzppJAtj01/Ymk8=
go.opentelemetry.io/otel v1.38.0/go.mod h1:zcmtmQ1+YmQM9wrNsTGV/q/uyusom3P8RxwExxkZhjM=
go.opentelemetry.io/otel/metric v1.38.0 h1:Kl6lzIYGAh5M159u9NgiRkmoMKjvbsKtYRwgfrA6WpA=
@@ -716,12 +758,12 @@ go.opentelemetry.io/otel/trace v1.38.0 h1:Fxk5bKrDZJUH+AMyyIXGcFAPah0oRcT+LuNtJr
go.opentelemetry.io/otel/trace v1.38.0/go.mod h1:j1P9ivuFsTceSWe1oY+EeW3sc+Pp42sO++GHkg4wwhs=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
go.yaml.in/yaml/v2 v2.4.3 h1:6gvOSjQoTB3vt1l+CU+tSyi/HOjfOjRLJ4YwYZGwRO0=
go.yaml.in/yaml/v2 v2.4.3/go.mod h1:zSxWcmIDjOzPXpjlTTbAsKokqkDNAVtZO0WOMiT90s8=
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/arch v0.20.0 h1:dx1zTU0MAE98U+TQ8BLl7XsJbgze2WnNKF/8tGp/Q6c=
golang.org/x/arch v0.20.0/go.mod h1:bdwinDaKcfZUGpH09BB7ZmOfhalA8lQdzl62l8gGWsk=
golang.org/x/arch v0.14.0 h1:z9JUEZWr8x4rR0OU6c4/4t6E6jOZ8/QBS2bBYBm4tx4=
golang.org/x/arch v0.14.0/go.mod h1:FEVrYAQjsQXMVJ1nsMoVVXPZg6p2JE2mx8psSWTDQys=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
@@ -730,16 +772,15 @@ golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPh
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.4.0/go.mod h1:3quD/ATkf6oY+rnes5c3ExXTbLc8mueNue5/DoinL80=
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU=
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.22.0/go.mod h1:vr6Su+7cTlO45qkww3VDJlzDn0ctJvRgYbC2NvXHt+M=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/crypto v0.33.0/go.mod h1:bVdXmD7IV/4GdElGPozy6U7lWdRXA4qyRVGJV57uQ5M=
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -750,12 +791,12 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
golang.org/x/exp v0.0.0-20250811191247-51f88131bc50 h1:3yiSh9fhy5/RhCSntf4Sy0Tnx50DmMpQ4MQdKKk4yg4=
golang.org/x/exp v0.0.0-20250811191247-51f88131bc50/go.mod h1:rT6SFzZ7oxADUDx58pcaKFTcZ+inxAa9fTrYx/uVYwg=
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 h1:mgKeJMpvi0yx/sU5GsxQ7p6s2wtOnGAHZWCHUM4KGzY=
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546/go.mod h1:j/pmGrbnkbPtQfxEe5D0VQhZC6qKbfKifgD0oM7sR70=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/image v0.34.0 h1:33gCkyw9hmwbZJeZkct8XyR11yH889EQt/QH4VmXMn8=
golang.org/x/image v0.34.0/go.mod h1:2RNFBZRB+vnwwFil8GkMdRvrJOFd1AzdZI6vOY+eJVU=
golang.org/x/image v0.36.0 h1:Iknbfm1afbgtwPTmHnS2gTM/6PPZfH+z2EFuOkSbqwc=
golang.org/x/image v0.36.0/go.mod h1:YsWD2TyyGKiIX1kZlu9QfKIsQ4nAAK9bdgdrIsE7xy4=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@@ -811,6 +852,7 @@ golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwY
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.3.0/go.mod h1:MBQ8lrhLObU/6UmLb4fmbmk5OcyYmqtbGd/9yIeKjEE=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
@@ -819,16 +861,15 @@ golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
golang.org/x/net v0.49.0 h1:eeHFmOGUTtaaPSGNmjBKpbng9MulQsJURQUAfUwY++o=
golang.org/x/net v0.49.0/go.mod h1:/ysNB2EvaqvesRkuLAyjI1ycPZlQHM3q01F02UY/MV8=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.32.0 h1:jsCblLleRMDrxMN29H3z/k1KliIvpLgCkE6R8FXXNgY=
golang.org/x/oauth2 v0.32.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
golang.org/x/oauth2 v0.34.0 h1:hqK/t4AKgbqWkdkcAeI8XLmbK+4m4G5YeQRrmiotGlw=
golang.org/x/oauth2 v0.34.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -845,7 +886,6 @@ golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.11.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -890,31 +930,29 @@ golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.19.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.3.0/go.mod h1:q750SLmJuPmVoN1blW3UFBPREJfb1KmY3vwxfr+nFDA=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.19.0/go.mod h1:2CuTdWZ7KHSQwUzKva0cbMg6q2DMI3Mmxp+gKJbskEk=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/term v0.29.0/go.mod h1:6bl4lRlvVuDgSf3179VpIxBF0o10JUpXWOnI7nErv7s=
golang.org/x/term v0.38.0 h1:PQ5pkm/rLO6HnxFR7N2lJHOZX6Kez5Y1gDSJla6jo7Q=
golang.org/x/term v0.38.0/go.mod h1:bSEAKrOT1W+VSu9TSCMtoGEOUcKxOKgl3LE5QEF/xVg=
golang.org/x/term v0.40.0 h1:36e4zGLqU4yhjlmxEaagx2KuYbJq3EwY8K943ZsHcvg=
golang.org/x/term v0.40.0/go.mod h1:w2P8uVp06p2iyKKuvXIm7N/y0UCRt3UfJTfZ7oOpglM=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -922,6 +960,7 @@ golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.5.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
@@ -929,14 +968,12 @@ golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/text v0.22.0/go.mod h1:YRoo4H8PVmsu+E3Ou7cqLVH8oXWIHVoX0jqUWALQhfY=
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.12.0 h1:ScB/8o8olJvc+CQPWrK3fPZNfh7qgwCrY0zJmoEQLSE=
golang.org/x/time v0.12.0/go.mod h1:CDIdPxbZBQxdj6cxyCIdrNogrJKMJ7pr37NYpMcMDSg=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -985,8 +1022,8 @@ golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/tools v0.39.0 h1:ik4ho21kwuQln40uelmciQPp9SipgNDdrafrYA4TmQQ=
golang.org/x/tools v0.39.0/go.mod h1:JnefbkDPyD8UU2kI5fuf8ZX4/yUeh9W877ZeBONxUqQ=
golang.org/x/tools v0.41.0 h1:a9b8iMweWG+S0OBnlU36rzLp20z1Rp10w+IY2czHTQc=
golang.org/x/tools v0.41.0/go.mod h1:XSY6eDqxVNiYgezAVqqCeihT4j1U2CCsqvH3WhQpnlg=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -1009,8 +1046,7 @@ google.golang.org/api v0.24.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0M
google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
google.golang.org/api v0.29.0/go.mod h1:Lcubydp8VUV7KeIHD9z2Bys/sm/vGKnG1UHuDBSrHWM=
google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz5138Fc=
google.golang.org/api v0.247.0 h1:tSd/e0QrUlLsrwMKmkbQhYVa109qIintOls2Wh6bngc=
google.golang.org/api v0.247.0/go.mod h1:r1qZOPmxXffXg6xS5uhx16Fa/UFY8QU/K4bfKrnvovM=
google.golang.org/api v0.258.0 h1:IKo1j5FBlN74fe5isA2PVozN3Y5pwNKriEgAXPOkDAc=
google.golang.org/api v0.258.0/go.mod h1:qhOMTQEZ6lUps63ZNq9jhODswwjkjYYguA7fA3TBFww=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
@@ -1047,12 +1083,11 @@ google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7Fc
google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20250715232539-7130f93afb79 h1:Nt6z9UHqSlIdIGJdz6KhTIs2VRx/iOsA5iE8bmQNcxs=
google.golang.org/genproto v0.0.0-20250715232539-7130f93afb79/go.mod h1:kTmlBHMPqR5uCZPBvwa2B18mvubkjyY3CRLI0c6fj0s=
google.golang.org/genproto/googleapis/api v0.0.0-20251022142026-3a174f9686a8 h1:mepRgnBZa07I4TRuomDE4sTIYieg/osKmzIf4USdWS4=
google.golang.org/genproto/googleapis/api v0.0.0-20251022142026-3a174f9686a8/go.mod h1:fDMmzKV90WSg1NbozdqrE64fkuTv6mlq2zxo9ad+3yo=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251111163417-95abcf5c77ba h1:UKgtfRM7Yh93Sya0Fo8ZzhDP4qBckrrxEr2oF5UIVb8=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251111163417-95abcf5c77ba/go.mod h1:7i2o+ce6H/6BluujYR+kqX3GKH+dChPTQU19wjRPiGk=
google.golang.org/genproto v0.0.0-20250922171735-9219d122eba9 h1:LvZVVaPE0JSqL+ZWb6ErZfnEOKIqqFWUJE2D0fObSmc=
google.golang.org/genproto v0.0.0-20250922171735-9219d122eba9/go.mod h1:QFOrLhdAe2PsTp3vQY4quuLKTi9j3XG3r6JPPaw7MSc=
google.golang.org/genproto/googleapis/api v0.0.0-20251124214823-79d6a2a48846 h1:ZdyUkS9po3H7G0tuh955QVyyotWvOD4W0aEapeGeUYk=
google.golang.org/genproto/googleapis/api v0.0.0-20251124214823-79d6a2a48846/go.mod h1:Fk4kyraUvqD7i5H6S43sj2W98fbZa75lpZz/eUyhfO0=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251213004720-97cd9d5aeac2 h1:2I6GHUeJ/4shcDpoUlLs/2WPnhg7yJwvXtqcMJt9liA=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251213004720-97cd9d5aeac2/go.mod h1:7i2o+ce6H/6BluujYR+kqX3GKH+dChPTQU19wjRPiGk=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
@@ -1066,8 +1101,8 @@ google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKa
google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk=
google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.31.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.77.0 h1:wVVY6/8cGA6vvffn+wWK5ToddbgdU3d8MNENr4evgXM=
google.golang.org/grpc v1.77.0/go.mod h1:z0BY1iVj0q8E1uSQCjL9cppRj+gnZjzDnzV0dHhrNig=
google.golang.org/grpc v1.78.0 h1:K1XZG/yGDJnzMdd/uZHAkVqJE+xIDOcmdSFZkBUicNc=
google.golang.org/grpc v1.78.0/go.mod h1:I47qjTo4OKbMkjA/aOOwxDIiPSBofUtQUI5EfpWvW7U=
google.golang.org/grpc/examples v0.0.0-20250407062114-b368379ef8f6 h1:ExN12ndbJ608cboPYflpTny6mXSzPrDLh0iTaVrRrds=
google.golang.org/grpc/examples v0.0.0-20250407062114-b368379ef8f6/go.mod h1:6ytKWczdvnpnO+m+JiG9NjEDzR1FJfsnmJdG7B8QVZ8=
google.golang.org/grpc/security/advancedtls v1.0.0 h1:/KQ7VP/1bs53/aopk9QhuPyFAp9Dm9Ejix3lzYkCrDA=
@@ -1082,8 +1117,7 @@ google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2
google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4=
google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
google.golang.org/protobuf v1.36.10 h1:AYd7cD/uASjIL6Q9LiTjz8JLcrh/88q5UObnmY3aOOE=
google.golang.org/protobuf v1.36.10/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@@ -1121,8 +1155,8 @@ rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
sigs.k8s.io/yaml v1.6.0 h1:G8fkbMSAFqgEFgh4b1wmtzDnioxFCUgTZhlbj5P9QYs=
sigs.k8s.io/yaml v1.6.0/go.mod h1:796bPqUfzR/0jLAl6XjHl3Ck7MiyVv8dbTdyT3/pMf4=
storj.io/common v0.0.0-20250808122759-804533d519c1 h1:z7ZjU+TlPZ2Lq2S12hT6+Fr7jFsBxPMrPBH4zZpZuUA=
storj.io/common v0.0.0-20250808122759-804533d519c1/go.mod h1:YNr7/ty6CmtpG5C9lEPtPXK3hOymZpueCb9QCNuPMUY=
storj.io/common v0.0.0-20251107171817-6221ae45072c h1:UDXSrdeLJe3QFouavSW10fYdpclK0YNu3KvQHzqq2+k=
storj.io/common v0.0.0-20251107171817-6221ae45072c/go.mod h1:XNX7uykja6aco92y2y8RuqaXIDRPpt1YA2OQDKlKEUk=
storj.io/drpc v0.0.35-0.20250513201419-f7819ea69b55 h1:8OE12DvUnB9lfZcHe7IDGsuhjrY9GBAr964PVHmhsro=
storj.io/drpc v0.0.35-0.20250513201419-f7819ea69b55/go.mod h1:Y9LZaa8esL1PW2IDMqJE7CFSNq7d5bQ3RI7mGPtmKMg=
storj.io/eventkit v0.0.0-20250410172343-61f26d3de156 h1:5MZ0CyMbG6Pi0rRzUWVG6dvpXjbBYEX2oyXuj+tT+sk=
+1 -2
View File
@@ -14,8 +14,7 @@ sleep 2
echo "Querying messages..."
cd /Users/chrislu/go/src/github.com/seaweedfs/seaweedfs/test/kafka/kafka-client-loadtest
docker compose exec kafka-gateway /usr/local/bin/weed sql \
docker compose exec kafka-gateway /usr/bin/weed-sql \
-master=seaweedfs-master:9333 \
-database=kafka \
-query="SELECT id, timestamp, producer_id, counter, user_id, event_type FROM \"test-json-topic\" LIMIT 5;"
@@ -0,0 +1,285 @@
package erasure_coding_test
import (
"context"
"fmt"
"testing"
"time"
pluginworkers "github.com/seaweedfs/seaweedfs/test/plugin_workers"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/worker_pb"
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
ecstorage "github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
"github.com/stretchr/testify/require"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/protobuf/proto"
)
type topologySpec struct {
name string
dataCenters int
racksPerDC int
nodesPerRack int
diskTypes []string
replicas int
collection string
}
type detectionCase struct {
name string
topology topologySpec
adminCollectionFilter string
expectProposals bool
}
func TestErasureCodingDetectionAcrossTopologies(t *testing.T) {
cases := []detectionCase{
{
name: "single-dc-multi-rack",
topology: topologySpec{
name: "single-dc-multi-rack",
dataCenters: 1,
racksPerDC: 2,
nodesPerRack: 7,
diskTypes: []string{"hdd"},
replicas: 1,
collection: "ec-test",
},
expectProposals: true,
},
{
name: "multi-dc",
topology: topologySpec{
name: "multi-dc",
dataCenters: 2,
racksPerDC: 1,
nodesPerRack: 7,
diskTypes: []string{"hdd"},
replicas: 1,
collection: "ec-test",
},
expectProposals: true,
},
{
name: "multi-dc-multi-rack",
topology: topologySpec{
name: "multi-dc-multi-rack",
dataCenters: 2,
racksPerDC: 2,
nodesPerRack: 4,
diskTypes: []string{"hdd"},
replicas: 1,
collection: "ec-test",
},
expectProposals: true,
},
{
name: "mixed-disk-types",
topology: topologySpec{
name: "mixed-disk-types",
dataCenters: 1,
racksPerDC: 2,
nodesPerRack: 7,
diskTypes: []string{"hdd", "ssd"},
replicas: 1,
collection: "ec-test",
},
expectProposals: true,
},
{
name: "multi-replica-volume",
topology: topologySpec{
name: "multi-replica-volume",
dataCenters: 1,
racksPerDC: 2,
nodesPerRack: 7,
diskTypes: []string{"hdd"},
replicas: 3,
collection: "ec-test",
},
expectProposals: true,
},
{
name: "collection-filter-match",
topology: topologySpec{
name: "collection-filter-match",
dataCenters: 1,
racksPerDC: 2,
nodesPerRack: 7,
diskTypes: []string{"hdd"},
replicas: 1,
collection: "filtered",
},
adminCollectionFilter: "filtered",
expectProposals: true,
},
{
name: "collection-filter-mismatch",
topology: topologySpec{
name: "collection-filter-mismatch",
dataCenters: 1,
racksPerDC: 2,
nodesPerRack: 7,
diskTypes: []string{"hdd"},
replicas: 1,
collection: "filtered",
},
adminCollectionFilter: "other",
expectProposals: false,
},
{
name: "insufficient-disks",
topology: topologySpec{
name: "insufficient-disks",
dataCenters: 1,
racksPerDC: 1,
nodesPerRack: 2,
diskTypes: []string{"hdd"},
replicas: 1,
collection: "ec-test",
},
expectProposals: false,
},
}
for _, tc := range cases {
tc := tc
t.Run(tc.name, func(t *testing.T) {
volumeID := uint32(7)
response := buildVolumeListResponse(t, tc.topology, volumeID)
master := pluginworkers.NewMasterServer(t, response)
dialOption := grpc.WithTransportCredentials(insecure.NewCredentials())
handler := pluginworker.NewErasureCodingHandler(dialOption, t.TempDir())
harness := pluginworkers.NewHarness(t, pluginworkers.HarnessConfig{
WorkerOptions: pluginworker.WorkerOptions{
GrpcDialOption: dialOption,
},
Handlers: []pluginworker.JobHandler{handler},
})
harness.WaitForJobType("erasure_coding")
if tc.adminCollectionFilter != "" {
err := harness.Plugin().SaveJobTypeConfig(&plugin_pb.PersistedJobTypeConfig{
JobType: "erasure_coding",
AdminConfigValues: map[string]*plugin_pb.ConfigValue{
"collection_filter": {
Kind: &plugin_pb.ConfigValue_StringValue{StringValue: tc.adminCollectionFilter},
},
},
})
require.NoError(t, err)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
proposals, err := harness.Plugin().RunDetection(ctx, "erasure_coding", &plugin_pb.ClusterContext{
MasterGrpcAddresses: []string{master.Address()},
}, 10)
require.NoError(t, err)
if !tc.expectProposals {
require.Empty(t, proposals)
return
}
require.NotEmpty(t, proposals)
proposal := proposals[0]
require.Equal(t, "erasure_coding", proposal.JobType)
paramsValue := proposal.Parameters["task_params_pb"]
require.NotNil(t, paramsValue)
params := &worker_pb.TaskParams{}
require.NoError(t, proto.Unmarshal(paramsValue.GetBytesValue(), params))
require.NotEmpty(t, params.Sources)
require.Len(t, params.Targets, ecstorage.TotalShardsCount)
})
}
}
func buildVolumeListResponse(t *testing.T, spec topologySpec, volumeID uint32) *master_pb.VolumeListResponse {
t.Helper()
volumeSizeLimitMB := uint64(100)
volumeSize := uint64(90) * 1024 * 1024
volumeModifiedAt := time.Now().Add(-10 * time.Minute).Unix()
diskTypes := spec.diskTypes
if len(diskTypes) == 0 {
diskTypes = []string{"hdd"}
}
replicas := spec.replicas
if replicas <= 0 {
replicas = 1
}
collection := spec.collection
if collection == "" {
collection = "ec-test"
}
var dataCenters []*master_pb.DataCenterInfo
nodeIndex := 0
replicasPlaced := 0
for dc := 0; dc < spec.dataCenters; dc++ {
var racks []*master_pb.RackInfo
for rack := 0; rack < spec.racksPerDC; rack++ {
var nodes []*master_pb.DataNodeInfo
for n := 0; n < spec.nodesPerRack; n++ {
nodeIndex++
address := fmt.Sprintf("127.0.0.1:%d", 20000+nodeIndex)
diskType := diskTypes[(nodeIndex-1)%len(diskTypes)]
diskInfo := &master_pb.DiskInfo{
DiskId: 0,
MaxVolumeCount: 100,
VolumeCount: 0,
VolumeInfos: []*master_pb.VolumeInformationMessage{},
}
if replicasPlaced < replicas {
diskInfo.VolumeCount = 1
diskInfo.VolumeInfos = append(diskInfo.VolumeInfos, &master_pb.VolumeInformationMessage{
Id: volumeID,
Collection: collection,
DiskId: 0,
Size: volumeSize,
DeletedByteCount: 0,
ModifiedAtSecond: volumeModifiedAt,
ReplicaPlacement: 1,
ReadOnly: false,
})
replicasPlaced++
}
nodes = append(nodes, &master_pb.DataNodeInfo{
Id: address,
Address: address,
DiskInfos: map[string]*master_pb.DiskInfo{diskType: diskInfo},
})
}
racks = append(racks, &master_pb.RackInfo{
Id: fmt.Sprintf("rack-%d", rack+1),
DataNodeInfos: nodes,
})
}
dataCenters = append(dataCenters, &master_pb.DataCenterInfo{
Id: fmt.Sprintf("dc-%d", dc+1),
RackInfos: racks,
})
}
return &master_pb.VolumeListResponse{
VolumeSizeLimitMb: volumeSizeLimitMB,
TopologyInfo: &master_pb.TopologyInfo{
DataCenterInfos: dataCenters,
},
}
}
@@ -0,0 +1,83 @@
package erasure_coding_test
import (
"context"
"fmt"
"os"
"path/filepath"
"testing"
"time"
pluginworkers "github.com/seaweedfs/seaweedfs/test/plugin_workers"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
ecstorage "github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
"github.com/stretchr/testify/require"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
func TestErasureCodingExecutionEncodesShards(t *testing.T) {
volumeID := uint32(123)
datSize := 1 * 1024 * 1024
dialOption := grpc.WithTransportCredentials(insecure.NewCredentials())
handler := pluginworker.NewErasureCodingHandler(dialOption, t.TempDir())
harness := pluginworkers.NewHarness(t, pluginworkers.HarnessConfig{
WorkerOptions: pluginworker.WorkerOptions{
GrpcDialOption: dialOption,
},
Handlers: []pluginworker.JobHandler{handler},
})
harness.WaitForJobType("erasure_coding")
sourceServer := pluginworkers.NewVolumeServer(t, "")
pluginworkers.WriteTestVolumeFiles(t, sourceServer.BaseDir(), volumeID, datSize)
targetServers := make([]*pluginworkers.VolumeServer, 0, ecstorage.TotalShardsCount)
targetAddresses := make([]string, 0, ecstorage.TotalShardsCount)
for i := 0; i < ecstorage.TotalShardsCount; i++ {
target := pluginworkers.NewVolumeServer(t, "")
targetServers = append(targetServers, target)
targetAddresses = append(targetAddresses, target.Address())
}
job := &plugin_pb.JobSpec{
JobId: fmt.Sprintf("ec-job-%d", volumeID),
JobType: "erasure_coding",
Parameters: map[string]*plugin_pb.ConfigValue{
"volume_id": {
Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: int64(volumeID)},
},
"collection": {
Kind: &plugin_pb.ConfigValue_StringValue{StringValue: "ec-test"},
},
"source_server": {
Kind: &plugin_pb.ConfigValue_StringValue{StringValue: sourceServer.Address()},
},
"target_servers": {
Kind: &plugin_pb.ConfigValue_StringList{StringList: &plugin_pb.StringList{Values: targetAddresses}},
},
},
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
result, err := harness.Plugin().ExecuteJob(ctx, job, nil, 1)
require.NoError(t, err)
require.NotNil(t, result)
require.True(t, result.Success)
require.GreaterOrEqual(t, sourceServer.MarkReadonlyCount(), 1)
require.GreaterOrEqual(t, len(sourceServer.DeleteRequests()), 1)
for shardID := 0; shardID < ecstorage.TotalShardsCount; shardID++ {
targetIndex := shardID % len(targetServers)
target := targetServers[targetIndex]
expected := filepath.Join(target.BaseDir(), fmt.Sprintf("%d.ec%02d", volumeID, shardID))
info, err := os.Stat(expected)
require.NoErrorf(t, err, "missing shard file %s", expected)
require.Greater(t, info.Size(), int64(0))
}
}
@@ -0,0 +1,123 @@
package erasure_coding_test
import (
"context"
"fmt"
"testing"
"time"
pluginworkers "github.com/seaweedfs/seaweedfs/test/plugin_workers"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
"github.com/stretchr/testify/require"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
func TestErasureCodingDetectionLargeTopology(t *testing.T) {
const (
rackCount = 100
serverCount = 1000
volumesPerNode = 300
volumeSizeLimit = uint64(100)
)
if serverCount%rackCount != 0 {
t.Fatalf("serverCount (%d) must be divisible by rackCount (%d)", serverCount, rackCount)
}
nodesPerRack := serverCount / rackCount
eligibleSize := uint64(90) * 1024 * 1024
ineligibleSize := uint64(10) * 1024 * 1024
modifiedAt := time.Now().Add(-10 * time.Minute).Unix()
volumeID := uint32(1)
dataCenters := make([]*master_pb.DataCenterInfo, 0, 1)
racks := make([]*master_pb.RackInfo, 0, rackCount)
for rack := 0; rack < rackCount; rack++ {
nodes := make([]*master_pb.DataNodeInfo, 0, nodesPerRack)
for node := 0; node < nodesPerRack; node++ {
address := fmt.Sprintf("10.0.%d.%d:8080", rack, node+1)
volumes := make([]*master_pb.VolumeInformationMessage, 0, volumesPerNode)
for v := 0; v < volumesPerNode; v++ {
size := ineligibleSize
if volumeID%2 == 0 {
size = eligibleSize
}
volumes = append(volumes, &master_pb.VolumeInformationMessage{
Id: volumeID,
Collection: "ec-bulk",
DiskId: 0,
Size: size,
DeletedByteCount: 0,
ModifiedAtSecond: modifiedAt,
ReplicaPlacement: 1,
ReadOnly: false,
})
volumeID++
}
diskInfo := &master_pb.DiskInfo{
DiskId: 0,
MaxVolumeCount: int64(volumesPerNode + 10),
VolumeCount: int64(volumesPerNode),
VolumeInfos: volumes,
}
nodes = append(nodes, &master_pb.DataNodeInfo{
Id: address,
Address: address,
DiskInfos: map[string]*master_pb.DiskInfo{"hdd": diskInfo},
})
}
racks = append(racks, &master_pb.RackInfo{
Id: fmt.Sprintf("rack-%d", rack+1),
DataNodeInfos: nodes,
})
}
dataCenters = append(dataCenters, &master_pb.DataCenterInfo{
Id: "dc-1",
RackInfos: racks,
})
response := &master_pb.VolumeListResponse{
VolumeSizeLimitMb: volumeSizeLimit,
TopologyInfo: &master_pb.TopologyInfo{
DataCenterInfos: dataCenters,
},
}
master := pluginworkers.NewMasterServer(t, response)
dialOption := grpc.WithTransportCredentials(insecure.NewCredentials())
handler := pluginworker.NewErasureCodingHandler(dialOption, t.TempDir())
harness := pluginworkers.NewHarness(t, pluginworkers.HarnessConfig{
WorkerOptions: pluginworker.WorkerOptions{
GrpcDialOption: dialOption,
},
Handlers: []pluginworker.JobHandler{handler},
})
harness.WaitForJobType("erasure_coding")
totalVolumes := serverCount * volumesPerNode
expectedEligible := totalVolumes / 2
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
start := time.Now()
proposals, err := harness.Plugin().RunDetection(ctx, "erasure_coding", &plugin_pb.ClusterContext{
MasterGrpcAddresses: []string{master.Address()},
}, 0)
duration := time.Since(start)
require.NoError(t, err)
require.GreaterOrEqual(t, len(proposals), 10, "should detect at least some proposals")
t.Logf("large topology detection completed in %s (proposals=%d, eligible=%d)", duration, len(proposals), expectedEligible)
if len(proposals) < expectedEligible {
t.Logf("large topology detection stopped early: %d proposals vs %d eligible", len(proposals), expectedEligible)
}
}
+90
View File
@@ -0,0 +1,90 @@
package pluginworkers
import (
"context"
"net"
"sync"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"google.golang.org/grpc"
"google.golang.org/protobuf/proto"
"testing"
)
// MasterServer provides a stub master gRPC service for topology responses.
type MasterServer struct {
master_pb.UnimplementedSeaweedServer
t *testing.T
server *grpc.Server
listener net.Listener
address string
mu sync.RWMutex
response *master_pb.VolumeListResponse
}
// NewMasterServer starts a stub master server that serves the provided response.
func NewMasterServer(t *testing.T, response *master_pb.VolumeListResponse) *MasterServer {
t.Helper()
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen master: %v", err)
}
server := pb.NewGrpcServer()
ms := &MasterServer{
t: t,
server: server,
listener: listener,
address: listener.Addr().String(),
response: response,
}
master_pb.RegisterSeaweedServer(server, ms)
go func() {
_ = server.Serve(listener)
}()
t.Cleanup(func() {
ms.Shutdown()
})
return ms
}
// Address returns the gRPC address of the master server.
func (m *MasterServer) Address() string {
return m.address
}
// SetVolumeListResponse updates the response served by VolumeList.
func (m *MasterServer) SetVolumeListResponse(response *master_pb.VolumeListResponse) {
m.mu.Lock()
defer m.mu.Unlock()
m.response = response
}
// VolumeList returns the configured topology response.
func (m *MasterServer) VolumeList(ctx context.Context, req *master_pb.VolumeListRequest) (*master_pb.VolumeListResponse, error) {
m.mu.RLock()
defer m.mu.RUnlock()
if m.response == nil {
return &master_pb.VolumeListResponse{}, nil
}
return proto.Clone(m.response).(*master_pb.VolumeListResponse), nil
}
// Shutdown stops the master gRPC server.
func (m *MasterServer) Shutdown() {
if m.server != nil {
m.server.GracefulStop()
}
if m.listener != nil {
_ = m.listener.Close()
}
}
+335
View File
@@ -0,0 +1,335 @@
package pluginworkers
import (
"context"
"fmt"
"io"
"net"
"os"
"path/filepath"
"sync"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
"google.golang.org/grpc"
)
// VolumeServer provides a minimal volume server for erasure coding tests.
type VolumeServer struct {
volume_server_pb.UnimplementedVolumeServerServer
t *testing.T
server *grpc.Server
listener net.Listener
address string
baseDir string
mu sync.Mutex
receivedFiles map[string]uint64
mountRequests []*volume_server_pb.VolumeEcShardsMountRequest
deleteRequests []*volume_server_pb.VolumeDeleteRequest
markReadonlyCalls int
vacuumGarbageRatio float64
vacuumCheckCalls int
vacuumCompactCalls int
vacuumCommitCalls int
vacuumCleanupCalls int
volumeCopyCalls int
volumeMountCalls int
tailReceiverCalls int
}
// NewVolumeServer starts a test volume server using the provided base directory.
func NewVolumeServer(t *testing.T, baseDir string) *VolumeServer {
t.Helper()
if baseDir == "" {
baseDir = t.TempDir()
}
if err := os.MkdirAll(baseDir, 0755); err != nil {
t.Fatalf("create volume base dir: %v", err)
}
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen volume server: %v", err)
}
grpcPort := listener.Addr().(*net.TCPAddr).Port
server := pb.NewGrpcServer()
vs := &VolumeServer{
t: t,
server: server,
listener: listener,
address: fmt.Sprintf("127.0.0.1:0.%d", grpcPort),
baseDir: baseDir,
receivedFiles: make(map[string]uint64),
}
volume_server_pb.RegisterVolumeServerServer(server, vs)
go func() {
_ = server.Serve(listener)
}()
t.Cleanup(func() {
vs.Shutdown()
})
return vs
}
// Address returns the gRPC address of the volume server.
func (v *VolumeServer) Address() string {
return v.address
}
// BaseDir returns the base directory used by the server.
func (v *VolumeServer) BaseDir() string {
return v.baseDir
}
// ReceivedFiles returns a snapshot of received files and byte counts.
func (v *VolumeServer) ReceivedFiles() map[string]uint64 {
v.mu.Lock()
defer v.mu.Unlock()
out := make(map[string]uint64, len(v.receivedFiles))
for key, value := range v.receivedFiles {
out[key] = value
}
return out
}
// SetVacuumGarbageRatio sets the garbage ratio returned by VacuumVolumeCheck.
func (v *VolumeServer) SetVacuumGarbageRatio(ratio float64) {
v.mu.Lock()
defer v.mu.Unlock()
v.vacuumGarbageRatio = ratio
}
// VacuumStats returns the vacuum RPC call counts.
func (v *VolumeServer) VacuumStats() (check, compact, commit, cleanup int) {
v.mu.Lock()
defer v.mu.Unlock()
return v.vacuumCheckCalls, v.vacuumCompactCalls, v.vacuumCommitCalls, v.vacuumCleanupCalls
}
// BalanceStats returns the balance RPC call counts.
func (v *VolumeServer) BalanceStats() (copyCalls, mountCalls, tailCalls int) {
v.mu.Lock()
defer v.mu.Unlock()
return v.volumeCopyCalls, v.volumeMountCalls, v.tailReceiverCalls
}
// MountRequests returns recorded mount requests.
func (v *VolumeServer) MountRequests() []*volume_server_pb.VolumeEcShardsMountRequest {
v.mu.Lock()
defer v.mu.Unlock()
out := make([]*volume_server_pb.VolumeEcShardsMountRequest, len(v.mountRequests))
copy(out, v.mountRequests)
return out
}
// DeleteRequests returns recorded delete requests.
func (v *VolumeServer) DeleteRequests() []*volume_server_pb.VolumeDeleteRequest {
v.mu.Lock()
defer v.mu.Unlock()
out := make([]*volume_server_pb.VolumeDeleteRequest, len(v.deleteRequests))
copy(out, v.deleteRequests)
return out
}
// MarkReadonlyCount returns the number of readonly calls.
func (v *VolumeServer) MarkReadonlyCount() int {
v.mu.Lock()
defer v.mu.Unlock()
return v.markReadonlyCalls
}
// Shutdown stops the volume server.
func (v *VolumeServer) Shutdown() {
if v.server != nil {
v.server.GracefulStop()
}
if v.listener != nil {
_ = v.listener.Close()
}
}
func (v *VolumeServer) filePath(volumeID uint32, ext string) string {
return filepath.Join(v.baseDir, fmt.Sprintf("%d%s", volumeID, ext))
}
func (v *VolumeServer) CopyFile(req *volume_server_pb.CopyFileRequest, stream volume_server_pb.VolumeServer_CopyFileServer) error {
if req == nil {
return fmt.Errorf("copy file request is nil")
}
path := v.filePath(req.VolumeId, req.Ext)
file, err := os.Open(path)
if err != nil {
if req.IgnoreSourceFileNotFound {
return nil
}
return err
}
defer file.Close()
buf := make([]byte, 64*1024)
for {
n, readErr := file.Read(buf)
if n > 0 {
if err := stream.Send(&volume_server_pb.CopyFileResponse{FileContent: buf[:n]}); err != nil {
return err
}
}
if readErr == io.EOF {
break
}
if readErr != nil {
return readErr
}
}
return nil
}
func (v *VolumeServer) ReceiveFile(stream volume_server_pb.VolumeServer_ReceiveFileServer) error {
var (
info *volume_server_pb.ReceiveFileInfo
file *os.File
bytesWritten uint64
filePath string
)
defer func() {
if file != nil {
_ = file.Close()
}
}()
for {
req, err := stream.Recv()
if err == io.EOF {
if info == nil {
return stream.SendAndClose(&volume_server_pb.ReceiveFileResponse{Error: "missing file info"})
}
v.mu.Lock()
v.receivedFiles[filePath] = bytesWritten
v.mu.Unlock()
return stream.SendAndClose(&volume_server_pb.ReceiveFileResponse{BytesWritten: bytesWritten})
}
if err != nil {
return err
}
if reqInfo := req.GetInfo(); reqInfo != nil {
info = reqInfo
filePath = v.filePath(info.VolumeId, info.Ext)
if err := os.MkdirAll(filepath.Dir(filePath), 0755); err != nil {
return err
}
file, err = os.Create(filePath)
if err != nil {
return err
}
continue
}
chunk := req.GetFileContent()
if len(chunk) == 0 {
continue
}
if file == nil {
return fmt.Errorf("file info not received")
}
n, writeErr := file.Write(chunk)
if writeErr != nil {
return writeErr
}
bytesWritten += uint64(n)
}
}
func (v *VolumeServer) VolumeEcShardsMount(ctx context.Context, req *volume_server_pb.VolumeEcShardsMountRequest) (*volume_server_pb.VolumeEcShardsMountResponse, error) {
v.mu.Lock()
v.mountRequests = append(v.mountRequests, req)
v.mu.Unlock()
return &volume_server_pb.VolumeEcShardsMountResponse{}, nil
}
func (v *VolumeServer) VolumeDelete(ctx context.Context, req *volume_server_pb.VolumeDeleteRequest) (*volume_server_pb.VolumeDeleteResponse, error) {
v.mu.Lock()
v.deleteRequests = append(v.deleteRequests, req)
v.mu.Unlock()
if req != nil {
_ = os.Remove(v.filePath(req.VolumeId, ".dat"))
_ = os.Remove(v.filePath(req.VolumeId, ".idx"))
}
return &volume_server_pb.VolumeDeleteResponse{}, nil
}
func (v *VolumeServer) VolumeMarkReadonly(ctx context.Context, req *volume_server_pb.VolumeMarkReadonlyRequest) (*volume_server_pb.VolumeMarkReadonlyResponse, error) {
v.mu.Lock()
v.markReadonlyCalls++
v.mu.Unlock()
return &volume_server_pb.VolumeMarkReadonlyResponse{}, nil
}
func (v *VolumeServer) VacuumVolumeCheck(ctx context.Context, req *volume_server_pb.VacuumVolumeCheckRequest) (*volume_server_pb.VacuumVolumeCheckResponse, error) {
v.mu.Lock()
v.vacuumCheckCalls++
ratio := v.vacuumGarbageRatio
v.mu.Unlock()
return &volume_server_pb.VacuumVolumeCheckResponse{GarbageRatio: ratio}, nil
}
func (v *VolumeServer) VacuumVolumeCompact(req *volume_server_pb.VacuumVolumeCompactRequest, stream volume_server_pb.VolumeServer_VacuumVolumeCompactServer) error {
v.mu.Lock()
v.vacuumCompactCalls++
v.mu.Unlock()
return stream.Send(&volume_server_pb.VacuumVolumeCompactResponse{ProcessedBytes: 1024})
}
func (v *VolumeServer) VacuumVolumeCommit(ctx context.Context, req *volume_server_pb.VacuumVolumeCommitRequest) (*volume_server_pb.VacuumVolumeCommitResponse, error) {
v.mu.Lock()
v.vacuumCommitCalls++
v.mu.Unlock()
return &volume_server_pb.VacuumVolumeCommitResponse{}, nil
}
func (v *VolumeServer) VacuumVolumeCleanup(ctx context.Context, req *volume_server_pb.VacuumVolumeCleanupRequest) (*volume_server_pb.VacuumVolumeCleanupResponse, error) {
v.mu.Lock()
v.vacuumCleanupCalls++
v.mu.Unlock()
return &volume_server_pb.VacuumVolumeCleanupResponse{}, nil
}
func (v *VolumeServer) VolumeCopy(req *volume_server_pb.VolumeCopyRequest, stream volume_server_pb.VolumeServer_VolumeCopyServer) error {
v.mu.Lock()
v.volumeCopyCalls++
v.mu.Unlock()
if err := stream.Send(&volume_server_pb.VolumeCopyResponse{ProcessedBytes: 1024}); err != nil {
return err
}
return stream.Send(&volume_server_pb.VolumeCopyResponse{LastAppendAtNs: uint64(time.Now().UnixNano())})
}
func (v *VolumeServer) VolumeMount(ctx context.Context, req *volume_server_pb.VolumeMountRequest) (*volume_server_pb.VolumeMountResponse, error) {
v.mu.Lock()
v.volumeMountCalls++
v.mu.Unlock()
return &volume_server_pb.VolumeMountResponse{}, nil
}
func (v *VolumeServer) VolumeTailReceiver(ctx context.Context, req *volume_server_pb.VolumeTailReceiverRequest) (*volume_server_pb.VolumeTailReceiverResponse, error) {
v.mu.Lock()
v.tailReceiverCalls++
v.mu.Unlock()
return &volume_server_pb.VolumeTailReceiverResponse{}, nil
}
+171
View File
@@ -0,0 +1,171 @@
package pluginworkers
import (
"context"
"fmt"
"net"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/weed/admin/plugin"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
"github.com/stretchr/testify/require"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
// HarnessConfig configures the shared plugin worker test harness.
type HarnessConfig struct {
PluginOptions plugin.Options
WorkerOptions pluginworker.WorkerOptions
Handlers []pluginworker.JobHandler
}
// Harness manages an in-process plugin admin server and worker.
type Harness struct {
t *testing.T
pluginSvc *plugin.Plugin
adminServer *grpc.Server
adminListener net.Listener
adminGrpcAddr string
worker *pluginworker.Worker
workerCtx context.Context
workerCancel context.CancelFunc
workerDone chan struct{}
}
// NewHarness starts a plugin admin gRPC server and a worker connected to it.
func NewHarness(t *testing.T, cfg HarnessConfig) *Harness {
t.Helper()
pluginOpts := cfg.PluginOptions
if pluginOpts.DataDir == "" {
pluginOpts.DataDir = t.TempDir()
}
pluginSvc, err := plugin.New(pluginOpts)
require.NoError(t, err)
listener, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
adminServer := pb.NewGrpcServer()
plugin_pb.RegisterPluginControlServiceServer(adminServer, pluginSvc)
go func() {
_ = adminServer.Serve(listener)
}()
adminGrpcAddr := listener.Addr().String()
adminPort := listener.Addr().(*net.TCPAddr).Port
adminAddr := fmt.Sprintf("127.0.0.1:0.%d", adminPort)
workerOpts := cfg.WorkerOptions
if workerOpts.AdminServer == "" {
workerOpts.AdminServer = adminAddr
}
if workerOpts.GrpcDialOption == nil {
workerOpts.GrpcDialOption = grpc.WithTransportCredentials(insecure.NewCredentials())
}
if workerOpts.WorkerID == "" {
workerOpts.WorkerID = "plugin-worker-test"
}
if workerOpts.WorkerVersion == "" {
workerOpts.WorkerVersion = "test"
}
if workerOpts.WorkerAddress == "" {
workerOpts.WorkerAddress = "127.0.0.1"
}
if len(cfg.Handlers) > 0 {
workerOpts.Handlers = cfg.Handlers
}
worker, err := pluginworker.NewWorker(workerOpts)
require.NoError(t, err)
workerCtx, workerCancel := context.WithCancel(context.Background())
workerDone := make(chan struct{})
go func() {
defer close(workerDone)
_ = worker.Run(workerCtx)
}()
harness := &Harness{
t: t,
pluginSvc: pluginSvc,
adminServer: adminServer,
adminListener: listener,
adminGrpcAddr: adminGrpcAddr,
worker: worker,
workerCtx: workerCtx,
workerCancel: workerCancel,
workerDone: workerDone,
}
require.Eventually(t, func() bool {
return len(pluginSvc.ListWorkers()) > 0
}, 5*time.Second, 50*time.Millisecond)
t.Cleanup(func() {
harness.Shutdown()
})
return harness
}
// Plugin exposes the underlying admin plugin service.
func (h *Harness) Plugin() *plugin.Plugin {
return h.pluginSvc
}
// AdminGrpcAddress returns the gRPC address for the admin server.
func (h *Harness) AdminGrpcAddress() string {
return h.adminGrpcAddr
}
// WaitForJobType waits until a worker with the given capability is registered.
func (h *Harness) WaitForJobType(jobType string) {
h.t.Helper()
require.Eventually(h.t, func() bool {
workers := h.pluginSvc.ListWorkers()
for _, worker := range workers {
if worker == nil || worker.Capabilities == nil {
continue
}
if _, ok := worker.Capabilities[jobType]; ok {
return true
}
}
return false
}, 5*time.Second, 50*time.Millisecond)
}
// Shutdown stops the worker and admin server.
func (h *Harness) Shutdown() {
if h.workerCancel != nil {
h.workerCancel()
}
if h.workerDone != nil {
select {
case <-h.workerDone:
case <-time.After(2 * time.Second):
}
}
if h.adminServer != nil {
h.adminServer.GracefulStop()
}
if h.adminListener != nil {
_ = h.adminListener.Close()
}
if h.pluginSvc != nil {
h.pluginSvc.Shutdown()
}
}
@@ -0,0 +1,104 @@
package vacuum_test
import (
"context"
"testing"
"time"
pluginworkers "github.com/seaweedfs/seaweedfs/test/plugin_workers"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/worker_pb"
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
"github.com/stretchr/testify/require"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/protobuf/proto"
)
func TestVacuumDetectionIntegration(t *testing.T) {
volumeID := uint32(101)
source := pluginworkers.NewVolumeServer(t, "")
response := buildVacuumVolumeListResponse(t, source.Address(), volumeID, 0.6)
master := pluginworkers.NewMasterServer(t, response)
dialOption := grpc.WithTransportCredentials(insecure.NewCredentials())
handler := pluginworker.NewVacuumHandler(dialOption, 1)
harness := pluginworkers.NewHarness(t, pluginworkers.HarnessConfig{
WorkerOptions: pluginworker.WorkerOptions{
GrpcDialOption: dialOption,
},
Handlers: []pluginworker.JobHandler{handler},
})
harness.WaitForJobType("vacuum")
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
proposals, err := harness.Plugin().RunDetection(ctx, "vacuum", &plugin_pb.ClusterContext{
MasterGrpcAddresses: []string{master.Address()},
}, 10)
require.NoError(t, err)
require.NotEmpty(t, proposals)
proposal := proposals[0]
require.Equal(t, "vacuum", proposal.JobType)
paramsValue := proposal.Parameters["task_params_pb"]
require.NotNil(t, paramsValue)
params := &worker_pb.TaskParams{}
require.NoError(t, proto.Unmarshal(paramsValue.GetBytesValue(), params))
require.NotEmpty(t, params.Sources)
require.NotNil(t, params.GetVacuumParams())
}
func buildVacuumVolumeListResponse(t *testing.T, serverAddress string, volumeID uint32, garbageRatio float64) *master_pb.VolumeListResponse {
t.Helper()
volumeSizeLimitMB := uint64(100)
volumeSize := uint64(90) * 1024 * 1024
deletedBytes := uint64(float64(volumeSize) * garbageRatio)
volumeModifiedAt := time.Now().Add(-48 * time.Hour).Unix()
diskInfo := &master_pb.DiskInfo{
DiskId: 0,
MaxVolumeCount: 100,
VolumeCount: 1,
VolumeInfos: []*master_pb.VolumeInformationMessage{
{
Id: volumeID,
Collection: "vac-test",
DiskId: 0,
Size: volumeSize,
DeletedByteCount: deletedBytes,
ModifiedAtSecond: volumeModifiedAt,
ReplicaPlacement: 1,
ReadOnly: false,
},
},
}
node := &master_pb.DataNodeInfo{
Id: serverAddress,
Address: serverAddress,
DiskInfos: map[string]*master_pb.DiskInfo{"hdd": diskInfo},
}
return &master_pb.VolumeListResponse{
VolumeSizeLimitMb: volumeSizeLimitMB,
TopologyInfo: &master_pb.TopologyInfo{
DataCenterInfos: []*master_pb.DataCenterInfo{
{
Id: "dc-1",
RackInfos: []*master_pb.RackInfo{
{
Id: "rack-1",
DataNodeInfos: []*master_pb.DataNodeInfo{node},
},
},
},
},
},
}
}
@@ -0,0 +1,62 @@
package vacuum_test
import (
"context"
"fmt"
"testing"
"time"
pluginworkers "github.com/seaweedfs/seaweedfs/test/plugin_workers"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
"github.com/stretchr/testify/require"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
func TestVacuumExecutionIntegration(t *testing.T) {
volumeID := uint32(202)
dialOption := grpc.WithTransportCredentials(insecure.NewCredentials())
handler := pluginworker.NewVacuumHandler(dialOption, 1)
harness := pluginworkers.NewHarness(t, pluginworkers.HarnessConfig{
WorkerOptions: pluginworker.WorkerOptions{
GrpcDialOption: dialOption,
},
Handlers: []pluginworker.JobHandler{handler},
})
harness.WaitForJobType("vacuum")
source := pluginworkers.NewVolumeServer(t, "")
source.SetVacuumGarbageRatio(0.6)
job := &plugin_pb.JobSpec{
JobId: fmt.Sprintf("vacuum-job-%d", volumeID),
JobType: "vacuum",
Parameters: map[string]*plugin_pb.ConfigValue{
"volume_id": {
Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: int64(volumeID)},
},
"server": {
Kind: &plugin_pb.ConfigValue_StringValue{StringValue: source.Address()},
},
"collection": {
Kind: &plugin_pb.ConfigValue_StringValue{StringValue: "vac-test"},
},
},
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
result, err := harness.Plugin().ExecuteJob(ctx, job, nil, 1)
require.NoError(t, err)
require.NotNil(t, result)
require.True(t, result.Success)
checkCalls, compactCalls, commitCalls, cleanupCalls := source.VacuumStats()
require.GreaterOrEqual(t, checkCalls, 2)
require.GreaterOrEqual(t, compactCalls, 1)
require.GreaterOrEqual(t, commitCalls, 1)
require.GreaterOrEqual(t, cleanupCalls, 1)
}
@@ -0,0 +1,129 @@
package volume_balance_test
import (
"context"
"testing"
"time"
pluginworkers "github.com/seaweedfs/seaweedfs/test/plugin_workers"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/worker_pb"
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
"github.com/stretchr/testify/require"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/protobuf/proto"
)
func TestVolumeBalanceDetectionIntegration(t *testing.T) {
response := buildBalanceVolumeListResponse(t)
master := pluginworkers.NewMasterServer(t, response)
dialOption := grpc.WithTransportCredentials(insecure.NewCredentials())
handler := pluginworker.NewVolumeBalanceHandler(dialOption)
harness := pluginworkers.NewHarness(t, pluginworkers.HarnessConfig{
WorkerOptions: pluginworker.WorkerOptions{
GrpcDialOption: dialOption,
},
Handlers: []pluginworker.JobHandler{handler},
})
harness.WaitForJobType("volume_balance")
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
proposals, err := harness.Plugin().RunDetection(ctx, "volume_balance", &plugin_pb.ClusterContext{
MasterGrpcAddresses: []string{master.Address()},
}, 10)
require.NoError(t, err)
require.Len(t, proposals, 1)
proposal := proposals[0]
require.Equal(t, "volume_balance", proposal.JobType)
paramsValue := proposal.Parameters["task_params_pb"]
require.NotNil(t, paramsValue)
params := &worker_pb.TaskParams{}
require.NoError(t, proto.Unmarshal(paramsValue.GetBytesValue(), params))
require.NotEmpty(t, params.Sources)
require.NotEmpty(t, params.Targets)
}
func buildBalanceVolumeListResponse(t *testing.T) *master_pb.VolumeListResponse {
t.Helper()
volumeSizeLimitMB := uint64(100)
volumeModifiedAt := time.Now().Add(-2 * time.Hour).Unix()
overloadedVolumes := make([]*master_pb.VolumeInformationMessage, 0, 10)
for i := 0; i < 10; i++ {
volumeID := uint32(1000 + i)
overloadedVolumes = append(overloadedVolumes, &master_pb.VolumeInformationMessage{
Id: volumeID,
Collection: "balance",
DiskId: 0,
Size: 20 * 1024 * 1024,
DeletedByteCount: 0,
ModifiedAtSecond: volumeModifiedAt,
ReplicaPlacement: 1,
ReadOnly: false,
})
}
underloadedVolumes := []*master_pb.VolumeInformationMessage{
{
Id: 2000,
Collection: "balance",
DiskId: 0,
Size: 20 * 1024 * 1024,
DeletedByteCount: 0,
ModifiedAtSecond: volumeModifiedAt,
ReplicaPlacement: 1,
ReadOnly: false,
},
}
overloadedDisk := &master_pb.DiskInfo{
DiskId: 0,
MaxVolumeCount: 100,
VolumeCount: int64(len(overloadedVolumes)),
VolumeInfos: overloadedVolumes,
}
underloadedDisk := &master_pb.DiskInfo{
DiskId: 0,
MaxVolumeCount: 100,
VolumeCount: int64(len(underloadedVolumes)),
VolumeInfos: underloadedVolumes,
}
overloadedNode := &master_pb.DataNodeInfo{
Id: "10.0.0.1:8080",
Address: "10.0.0.1:8080",
DiskInfos: map[string]*master_pb.DiskInfo{"hdd": overloadedDisk},
}
underloadedNode := &master_pb.DataNodeInfo{
Id: "10.0.0.2:8080",
Address: "10.0.0.2:8080",
DiskInfos: map[string]*master_pb.DiskInfo{"hdd": underloadedDisk},
}
rack := &master_pb.RackInfo{
Id: "rack-1",
DataNodeInfos: []*master_pb.DataNodeInfo{overloadedNode, underloadedNode},
}
return &master_pb.VolumeListResponse{
VolumeSizeLimitMb: volumeSizeLimitMB,
TopologyInfo: &master_pb.TopologyInfo{
DataCenterInfos: []*master_pb.DataCenterInfo{
{
Id: "dc-1",
RackInfos: []*master_pb.RackInfo{rack},
},
},
},
}
}
@@ -0,0 +1,67 @@
package volume_balance_test
import (
"context"
"fmt"
"testing"
"time"
pluginworkers "github.com/seaweedfs/seaweedfs/test/plugin_workers"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
"github.com/stretchr/testify/require"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
func TestVolumeBalanceExecutionIntegration(t *testing.T) {
volumeID := uint32(303)
dialOption := grpc.WithTransportCredentials(insecure.NewCredentials())
handler := pluginworker.NewVolumeBalanceHandler(dialOption)
harness := pluginworkers.NewHarness(t, pluginworkers.HarnessConfig{
WorkerOptions: pluginworker.WorkerOptions{
GrpcDialOption: dialOption,
},
Handlers: []pluginworker.JobHandler{handler},
})
harness.WaitForJobType("volume_balance")
source := pluginworkers.NewVolumeServer(t, "")
target := pluginworkers.NewVolumeServer(t, "")
job := &plugin_pb.JobSpec{
JobId: fmt.Sprintf("balance-job-%d", volumeID),
JobType: "volume_balance",
Parameters: map[string]*plugin_pb.ConfigValue{
"volume_id": {
Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: int64(volumeID)},
},
"collection": {
Kind: &plugin_pb.ConfigValue_StringValue{StringValue: "balance"},
},
"source_server": {
Kind: &plugin_pb.ConfigValue_StringValue{StringValue: source.Address()},
},
"target_server": {
Kind: &plugin_pb.ConfigValue_StringValue{StringValue: target.Address()},
},
},
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
result, err := harness.Plugin().ExecuteJob(ctx, job, nil, 1)
require.NoError(t, err)
require.NotNil(t, result)
require.True(t, result.Success)
require.GreaterOrEqual(t, source.MarkReadonlyCount(), 1)
require.GreaterOrEqual(t, len(source.DeleteRequests()), 1)
copyCalls, mountCalls, tailCalls := target.BalanceStats()
require.GreaterOrEqual(t, copyCalls, 1)
require.GreaterOrEqual(t, mountCalls, 1)
require.GreaterOrEqual(t, tailCalls, 1)
}
+49
View File
@@ -0,0 +1,49 @@
package pluginworkers
import (
"fmt"
"math/rand"
"os"
"path/filepath"
"testing"
"github.com/seaweedfs/seaweedfs/weed/storage/types"
)
// WriteTestVolumeFiles creates a minimal .dat/.idx pair for the given volume.
func WriteTestVolumeFiles(t *testing.T, baseDir string, volumeID uint32, datSize int) (string, string) {
t.Helper()
if err := os.MkdirAll(baseDir, 0755); err != nil {
t.Fatalf("create volume dir: %v", err)
}
datPath := filepath.Join(baseDir, volumeFilename(volumeID, ".dat"))
idxPath := filepath.Join(baseDir, volumeFilename(volumeID, ".idx"))
data := make([]byte, datSize)
rng := rand.New(rand.NewSource(99))
_, _ = rng.Read(data)
if err := os.WriteFile(datPath, data, 0644); err != nil {
t.Fatalf("write dat file: %v", err)
}
entry := make([]byte, types.NeedleMapEntrySize)
idEnd := types.NeedleIdSize
offsetEnd := idEnd + types.OffsetSize
sizeEnd := offsetEnd + types.SizeSize
types.NeedleIdToBytes(entry[:idEnd], types.NeedleId(1))
types.OffsetToBytes(entry[idEnd:offsetEnd], types.ToOffset(0))
types.SizeToBytes(entry[offsetEnd:sizeEnd], types.Size(datSize))
if err := os.WriteFile(idxPath, entry, 0644); err != nil {
t.Fatalf("write idx file: %v", err)
}
return datPath, idxPath
}
func volumeFilename(volumeID uint32, ext string) string {
return fmt.Sprintf("%d%s", volumeID, ext)
}
+5 -3
View File
@@ -13,8 +13,9 @@ RUN go mod download
# Copy source code
COPY . .
# Build the weed binary without CGO
# Build the weed binaries without CGO
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags "-s -w" -o weed ./weed/
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags "-s -w" -o weed-db ./cmd/weed-db
# Final stage - minimal runtime image
FROM alpine:latest
@@ -24,11 +25,12 @@ RUN apk --no-cache add ca-certificates netcat-openbsd curl
WORKDIR /root/
# Copy the weed binary from builder stage
# Copy the binaries from builder stage
COPY --from=builder /app/weed .
COPY --from=builder /app/weed-db .
# Make it executable
RUN chmod +x ./weed
RUN chmod +x ./weed ./weed-db
# Expose ports
EXPOSE 9333 8888 8333 8085 9533 5432
+1 -1
View File
@@ -280,7 +280,7 @@ This test setup proves:
- Comprehensive error handling
### ✅ Performance and Scalability
- Direct SQL engine integration (same as `weed sql`)
- Direct SQL engine integration (same as `weed-sql`)
- No translation overhead for real queries
- Efficient data access from stored formats
- Scalable architecture with service discovery
+41 -30
View File
@@ -1,3 +1,6 @@
//go:build ignore
// +build ignore
package main
import (
@@ -11,9 +14,22 @@ import (
_ "github.com/lib/pq"
)
const defaultPostgresHost = "postgres-server"
func getEnv(key, def string) string {
if value, exists := os.LookupEnv(key); exists {
return value
}
return def
}
func main() {
clientMain()
}
func clientMain() {
// Get PostgreSQL connection details from environment
host := getEnv("POSTGRES_HOST", "localhost")
host := getEnv("POSTGRES_HOST", defaultPostgresHost)
port := getEnv("POSTGRES_PORT", "5432")
user := getEnv("POSTGRES_USER", "seaweedfs")
dbname := getEnv("POSTGRES_DB", "default")
@@ -102,7 +118,7 @@ func testSystemInfo(db *sql.DB) error {
}
// Use individual connections for each query to avoid protocol issues
connStr := getEnv("POSTGRES_HOST", "postgres-server")
connStr := getEnv("POSTGRES_HOST", defaultPostgresHost)
port := getEnv("POSTGRES_PORT", "5432")
user := getEnv("POSTGRES_USER", "seaweedfs")
dbname := getEnv("POSTGRES_DB", "logs")
@@ -118,12 +134,12 @@ func testSystemInfo(db *sql.DB) error {
log.Printf(" Query '%s' failed to connect: %v", q.query, err)
continue
}
defer tempDB.Close()
var result string
err = tempDB.QueryRow(q.query).Scan(&result)
if err != nil {
log.Printf(" Query '%s' failed: %v", q.query, err)
tempDB.Close()
continue
}
log.Printf(" %s: %s", q.name, result)
@@ -313,7 +329,7 @@ func testDatabaseSwitching(db *sql.DB) error {
databases := []string{"analytics", "ecommerce", "logs"}
// Use fresh connections to avoid protocol issues
connStr := getEnv("POSTGRES_HOST", "postgres-server")
connStr := getEnv("POSTGRES_HOST", defaultPostgresHost)
port := getEnv("POSTGRES_PORT", "5432")
user := getEnv("POSTGRES_USER", "seaweedfs")
@@ -328,7 +344,6 @@ func testDatabaseSwitching(db *sql.DB) error {
log.Printf(" Could not connect to '%s': %v", dbName, err)
continue
}
defer tempDB.Close()
// Test the connection by executing a simple query
var newDB string
@@ -372,7 +387,7 @@ func testSystemColumns(db *sql.DB) error {
// Use fresh connection to avoid protocol state issues
connStr := fmt.Sprintf("host=%s port=%s user=%s dbname=%s sslmode=disable",
getEnv("POSTGRES_HOST", "postgres-server"),
getEnv("POSTGRES_HOST", defaultPostgresHost),
getEnv("POSTGRES_PORT", "5432"),
getEnv("POSTGRES_USER", "seaweedfs"),
getEnv("POSTGRES_DB", "logs"))
@@ -382,7 +397,6 @@ func testSystemColumns(db *sql.DB) error {
log.Printf(" Could not create connection: %v", err)
continue
}
defer tempDB.Close()
// First check if table exists and has data (safer than COUNT which was causing crashes)
rows, err := tempDB.Query(fmt.Sprintf("SELECT id FROM %s LIMIT 1", table))
@@ -444,7 +458,6 @@ func testComplexQueries(db *sql.DB) error {
log.Printf(" Could not create connection: %v", err)
continue
}
defer tempDB.Close()
// Test basic SELECT with LIMIT (avoid COUNT which was causing crashes)
rows, err := tempDB.Query(fmt.Sprintf("SELECT id FROM %s LIMIT 5", table))
@@ -469,38 +482,36 @@ func testComplexQueries(db *sql.DB) error {
// Test WHERE clause with known ID (safer than arbitrary conditions)
testID := ids[0]
rows, err = tempDB.Query(fmt.Sprintf("SELECT id FROM %s WHERE id = %d", table, testID))
if err == nil {
if err != nil {
log.Printf(" WHERE query failed: %v", err)
tempDB.Close()
return fmt.Errorf("WHERE clause test failed for table '%s'", table)
}
foundMatch := false
if rows.Next() {
var foundID int64
if rows.Next() {
if err := rows.Scan(&foundID); err == nil && foundID == testID {
log.Printf(" ✓ WHERE clause working: found record with ID %d", foundID)
}
if err := rows.Scan(&foundID); err == nil && foundID == testID {
log.Printf(" ✓ WHERE clause working: found record with ID %d", foundID)
foundMatch = true
}
rows.Close()
}
rows.Close()
if !foundMatch {
tempDB.Close()
return fmt.Errorf("WHERE clause test failed: no matching record found for ID %d in table '%s'", testID, table)
}
log.Printf(" ✓ Complex queries test passed for '%s'", table)
tempDB.Close()
return nil
} else {
tempDB.Close()
return fmt.Errorf("no records found in table '%s' - cannot test WHERE clause", table)
}
tempDB.Close()
}
log.Println(" Complex queries test completed - avoided crash-prone patterns")
return nil
}
func stringOrNull(ns sql.NullString) string {
if ns.Valid {
return ns.String
}
return "NULL"
}
func getEnv(key, defaultValue string) string {
if value, exists := os.LookupEnv(key); exists {
return value
}
return defaultValue
}
+1 -1
View File
@@ -54,7 +54,7 @@ services:
seaweedfs:
condition: service_healthy
command: >
./weed db
./weed-db
-host=0.0.0.0
-port=5432
-master=seaweedfs:9333
+1 -1
View File
@@ -322,7 +322,7 @@ func discoverFiler(masterHTTPAddress string) (string, error) {
// Use the first available filer and convert HTTP address to gRPC
filerHTTPAddress := resp.ClusterNodes[0].Address
httpAddr := pb.ServerAddress(filerHTTPAddress)
httpAddr = pb.ServerAddress(filerHTTPAddress)
return httpAddr.ToGrpcAddress(), nil
}
Executable → Regular
View File
Executable → Regular
View File
+7 -4
View File
@@ -12,6 +12,7 @@ FILER_PORT := 8888
TEST_TIMEOUT := 10m
TEST_PATTERN := TestCORS
SERVER_DIR := test-mini-data
S3_CONFIG := s3_test_config.json
# Default target
help:
@@ -80,13 +81,15 @@ start-server: check-deps
@echo "🔍 DEBUG: Creating volume directory..."
@mkdir -p $(SERVER_DIR)
@echo "🔍 DEBUG: Launching SeaweedFS S3 server in background..."
@echo "🔍 DEBUG: Command: $(WEED_BINARY) mini -dir=$(SERVER_DIR) -s3.port=$(S3_PORT) -s3.config=$(S3_CONFIG)"
@$(WEED_BINARY) mini \
@echo "🔍 DEBUG: Command: AWS_ACCESS_KEY_ID=some_access_key1 AWS_SECRET_ACCESS_KEY=some_secret_key1 $(WEED_BINARY) mini -dir=$(SERVER_DIR) -s3.port=$(S3_PORT) -s3.config=$(S3_CONFIG)"
@env AWS_ACCESS_KEY_ID=some_access_key1 \
AWS_SECRET_ACCESS_KEY=some_secret_key1 \
$(WEED_BINARY) mini \
-dir=$(SERVER_DIR) \
-s3.port=$(S3_PORT) \
-s3.config=$(S3_CONFIG) \
> weed-test.log 2>&1 & \
echo $$! > weed-test.pid
echo $$! > weed-server.pid
@echo "Waiting for S3 server to be ready..."
@for i in $$(seq 1 30); do \
@@ -97,7 +100,7 @@ start-server: check-deps
sleep 1; \
done; \
echo "S3 server failed to start"; \
exit 1 > weed-server.pid
exit 1
@echo "🔍 DEBUG: Server PID: $$(cat weed-server.pid 2>/dev/null || echo 'PID file not found')"
@echo "🔍 DEBUG: Checking if PID is still running..."
@sleep 2
+27
View File
@@ -0,0 +1,27 @@
{
"identities": [
{
"name": "anonymous",
"actions": [
"Read",
"List"
]
},
{
"name": "admin",
"credentials": [
{
"accessKey": "some_access_key1",
"secretKey": "some_secret_key1"
}
],
"actions": [
"Admin",
"Read",
"List",
"Tagging",
"Write"
]
}
]
}
+111
View File
@@ -4,6 +4,7 @@ import (
"testing"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/service/iam"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -202,6 +203,116 @@ func TestIAMPolicyManagement(t *testing.T) {
})
})
t.Run("managed_policy_crud_lifecycle", func(t *testing.T) {
policyDoc := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"s3:GetObject","Resource":"arn:aws:s3:::*"}]}`
policyNames := []string{"test-managed-policy-lifecycle-a", "test-managed-policy-lifecycle-b"}
policyArns := make([]*string, 0, len(policyNames))
for _, policyName := range policyNames {
createResp, err := iamClient.CreatePolicy(&iam.CreatePolicyInput{
PolicyName: aws.String(policyName),
PolicyDocument: aws.String(policyDoc),
})
require.NoError(t, err)
policyArns = append(policyArns, createResp.Policy.Arn)
}
t.Cleanup(func() {
for _, policyArn := range policyArns {
_, _ = iamClient.DeletePolicy(&iam.DeletePolicyInput{PolicyArn: policyArn})
}
})
listResp, err := iamClient.ListPolicies(&iam.ListPoliciesInput{})
require.NoError(t, err)
foundByName := map[string]bool{}
for _, policy := range listResp.Policies {
if policy.PolicyName != nil {
foundByName[*policy.PolicyName] = true
}
}
for _, policyName := range policyNames {
assert.True(t, foundByName[policyName], "policy %s should be listed", policyName)
}
getResp, err := iamClient.GetPolicy(&iam.GetPolicyInput{PolicyArn: policyArns[0]})
require.NoError(t, err)
require.NotNil(t, getResp.Policy)
assert.Equal(t, policyNames[0], aws.StringValue(getResp.Policy.PolicyName))
assert.Equal(t, aws.StringValue(policyArns[0]), aws.StringValue(getResp.Policy.Arn))
_, err = iamClient.DeletePolicy(&iam.DeletePolicyInput{PolicyArn: policyArns[0]})
require.NoError(t, err)
_, err = iamClient.GetPolicy(&iam.GetPolicyInput{PolicyArn: policyArns[0]})
require.Error(t, err)
awsErr, ok := err.(awserr.Error)
require.True(t, ok)
assert.Equal(t, iam.ErrCodeNoSuchEntityException, awsErr.Code())
listAfterDeleteResp, err := iamClient.ListPolicies(&iam.ListPoliciesInput{})
require.NoError(t, err)
deletedPolicyFound := false
remainingPolicyFound := false
for _, policy := range listAfterDeleteResp.Policies {
if policy.PolicyName == nil {
continue
}
if *policy.PolicyName == policyNames[0] {
deletedPolicyFound = true
}
if *policy.PolicyName == policyNames[1] {
remainingPolicyFound = true
}
}
assert.False(t, deletedPolicyFound, "deleted policy should no longer be listed")
assert.True(t, remainingPolicyFound, "remaining policy should still be listed")
policyArns[0] = nil
})
t.Run("managed_policy_versions", func(t *testing.T) {
policyName := "test-managed-policy-version"
policyDoc := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"s3:ListBucket","Resource":"*"}]}`
createResp, err := iamClient.CreatePolicy(&iam.CreatePolicyInput{
PolicyName: aws.String(policyName),
PolicyDocument: aws.String(policyDoc),
})
require.NoError(t, err)
t.Cleanup(func() {
_, _ = iamClient.DeletePolicy(&iam.DeletePolicyInput{PolicyArn: createResp.Policy.Arn})
})
listVersionsResp, err := iamClient.ListPolicyVersions(&iam.ListPolicyVersionsInput{
PolicyArn: createResp.Policy.Arn,
})
require.NoError(t, err)
require.NotEmpty(t, listVersionsResp.Versions)
assert.Equal(t, "v1", aws.StringValue(listVersionsResp.Versions[0].VersionId))
assert.Equal(t, true, aws.BoolValue(listVersionsResp.Versions[0].IsDefaultVersion))
getVersionResp, err := iamClient.GetPolicyVersion(&iam.GetPolicyVersionInput{
PolicyArn: createResp.Policy.Arn,
VersionId: aws.String("v1"),
})
require.NoError(t, err)
require.NotNil(t, getVersionResp.PolicyVersion)
assert.Equal(t, "v1", aws.StringValue(getVersionResp.PolicyVersion.VersionId))
assert.Contains(t, aws.StringValue(getVersionResp.PolicyVersion.Document), "s3:ListBucket")
_, err = iamClient.GetPolicyVersion(&iam.GetPolicyVersionInput{
PolicyArn: createResp.Policy.Arn,
VersionId: aws.String("v2"),
})
require.Error(t, err)
awsErr, ok := err.(awserr.Error)
require.True(t, ok)
assert.Equal(t, iam.ErrCodeNoSuchEntityException, awsErr.Code())
})
t.Run("user_inline_policy", func(t *testing.T) {
userName := "test-user-policy"
_, err := iamClient.CreateUser(&iam.CreateUserInput{
+2 -5
View File
@@ -1,7 +1,6 @@
package example
import (
"os"
"testing"
"time"
@@ -22,10 +21,8 @@ func TestIAMOperations(t *testing.T) {
// Set credentials before starting cluster
accessKey := "testkey123"
secretKey := "testsecret456"
os.Setenv("AWS_ACCESS_KEY_ID", accessKey)
os.Setenv("AWS_SECRET_ACCESS_KEY", secretKey)
defer os.Unsetenv("AWS_ACCESS_KEY_ID")
defer os.Unsetenv("AWS_SECRET_ACCESS_KEY")
t.Setenv("AWS_ACCESS_KEY_ID", accessKey)
t.Setenv("AWS_SECRET_ACCESS_KEY", secretKey)
// Create and start test cluster
cluster, err := startMiniCluster(t)
+351 -4
View File
@@ -3,6 +3,8 @@ package example
import (
"bytes"
"context"
"crypto/md5"
"encoding/base64"
"fmt"
"math/rand"
"net"
@@ -15,6 +17,7 @@ import (
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
@@ -68,6 +71,22 @@ func TestS3Integration(t *testing.T) {
testPutObject(t, cluster)
})
t.Run("UploadPart", func(t *testing.T) {
testPutPartWithChecksum(t, cluster)
})
t.Run("PutObjectWithChecksum", func(t *testing.T) {
testPutObjectWithChecksum(t, cluster)
})
t.Run("UploadPartWithChecksum", func(t *testing.T) {
testUploadPartWithChecksum(t, cluster)
})
t.Run("PutObjectWithChecksumAndSSEC", func(t *testing.T) {
testPutObjectWithChecksumAndSSEC(t, cluster)
})
t.Run("GetObject", func(t *testing.T) {
testGetObject(t, cluster)
})
@@ -97,8 +116,9 @@ func findAvailablePort() (int, error) {
return addr.Port, nil
}
// startMiniCluster starts a weed mini instance directly without exec
func startMiniCluster(t *testing.T) (*TestCluster, error) {
// startMiniCluster starts a weed mini instance directly without exec.
// Extra flags (e.g. "-s3.allowDeleteBucketNotEmpty=false") can be appended via extraArgs.
func startMiniCluster(t *testing.T, extraArgs ...string) (*TestCluster, error) {
// Find available ports
masterPort, err := findAvailablePort()
if err != nil {
@@ -146,6 +166,14 @@ func startMiniCluster(t *testing.T) (*TestCluster, error) {
return nil, fmt.Errorf("failed to create security.toml: %v", err)
}
// Set environment variables for admin credentials safely for this test
if os.Getenv("AWS_ACCESS_KEY_ID") == "" {
t.Setenv("AWS_ACCESS_KEY_ID", "admin")
}
if os.Getenv("AWS_SECRET_ACCESS_KEY") == "" {
t.Setenv("AWS_SECRET_ACCESS_KEY", "admin")
}
// Start weed mini in a goroutine by calling the command directly
cluster.wg.Add(1)
go func() {
@@ -165,7 +193,7 @@ func startMiniCluster(t *testing.T) (*TestCluster, error) {
// Configure args for mini command
// Note: When running via 'go test', os.Args[0] is the test binary
// We need to make it look like we're running 'weed mini'
os.Args = []string{
os.Args = append([]string{
"weed",
"-dir=" + testDir,
"-master.port=" + strconv.Itoa(masterPort),
@@ -178,7 +206,7 @@ func startMiniCluster(t *testing.T) (*TestCluster, error) {
"-ip=127.0.0.1",
"-master.peers=none", // Faster startup
"-s3.iam.readOnly=false", // Enable IAM write operations for tests
}
}, extraArgs...)
// Suppress most logging during tests
glog.MaxSize = 1024 * 1024
@@ -336,6 +364,282 @@ func testPutObject(t *testing.T, cluster *TestCluster) {
t.Logf("✓ Put object: %s/%s (%d bytes)", bucketName, objectKey, len(objectData))
}
func createTestBucket(t *testing.T, cluster *TestCluster, prefix string) string {
bucketName := prefix + randomString(8)
_, err := cluster.s3Client.CreateBucket(&s3.CreateBucketInput{
Bucket: aws.String(bucketName),
})
require.NoError(t, err)
time.Sleep(100 * time.Millisecond)
return bucketName
}
// generateSSECKey returns a 32-byte key as a raw string (what the SDK expects
// for SSECustomerKey) and its base64-encoded MD5 (for SSECustomerKeyMD5).
func generateSSECKey() (keyRaw, keyMD5B64 string) {
rng := rand.New(rand.NewSource(time.Now().UnixNano()))
key := make([]byte, 32)
for i := range key {
key[i] = byte(rng.Intn(256))
}
keyRaw = string(key)
keyHash := md5.Sum(key)
keyMD5B64 = base64.StdEncoding.EncodeToString(keyHash[:])
return
}
func testPutObjectWithChecksum(t *testing.T, cluster *TestCluster) {
bucketName := createTestBucket(t, cluster, "test-put-checksum-")
objectKey := "test-checksummed-object.txt"
objectData := "Hello, SeaweedFS S3!"
correctMD5 := calculateMd5(objectData)
incorrectMD5 := calculateMd5(objectData + "incorrect")
// Put object with incorrect MD5 should be rejected
_, err := cluster.s3Client.PutObject(&s3.PutObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
Body: bytes.NewReader([]byte(objectData)),
ContentMD5: aws.String(incorrectMD5),
})
assertBadDigestError(t, err, "PutObject should fail with incorrect MD5")
t.Logf("✓ Put object with incorrect MD5 rejected: %s/%s", bucketName, objectKey)
// Put object with correct MD5 should succeed
_, err = cluster.s3Client.PutObject(&s3.PutObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
Body: bytes.NewReader([]byte(objectData)),
ContentMD5: aws.String(correctMD5),
})
require.NoError(t, err, "Failed to put object")
// Verify object exists
headResp, err := cluster.s3Client.HeadObject(&s3.HeadObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
})
require.NoError(t, err)
assert.NotNil(t, headResp.ContentLength)
assert.Equal(t, int64(len(objectData)), aws.Int64Value(headResp.ContentLength))
t.Logf("✓ Put object with correct MD5: %s/%s (%d bytes)", bucketName, objectKey, len(objectData))
}
// putObjectSSEC sends a PutObject request with SSE-C headers over HTTP.
// The AWS SDK v1 refuses to send SSE-C keys over plain HTTP, so we use the
// low-level Request API and clear the Validate handlers to bypass that check.
// We use Clear() because the specific validator is internal and not easily removable by name.
func putObjectSSEC(client *s3.S3, input *s3.PutObjectInput) (*s3.PutObjectOutput, error) {
req, output := client.PutObjectRequest(input)
req.Handlers.Validate.Clear()
err := req.Send()
return output, err
}
func headObjectSSEC(client *s3.S3, input *s3.HeadObjectInput) (*s3.HeadObjectOutput, error) {
req, output := client.HeadObjectRequest(input)
req.Handlers.Validate.Clear()
err := req.Send()
return output, err
}
func testPutObjectWithChecksumAndSSEC(t *testing.T, cluster *TestCluster) {
bucketName := createTestBucket(t, cluster, "test-put-checksum-ssec-")
objectKey := "test-checksummed-ssec-object.txt"
objectData := "Hello, SeaweedFS S3 with SSE-C!"
correctMD5 := calculateMd5(objectData)
incorrectMD5 := calculateMd5(objectData + "incorrect")
keyRaw, keyMD5B64 := generateSSECKey()
// Put object with SSE-C and incorrect MD5 should be rejected
_, err := putObjectSSEC(cluster.s3Client, &s3.PutObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
Body: bytes.NewReader([]byte(objectData)),
ContentMD5: aws.String(incorrectMD5),
SSECustomerAlgorithm: aws.String("AES256"),
SSECustomerKey: aws.String(keyRaw),
SSECustomerKeyMD5: aws.String(keyMD5B64),
})
assertBadDigestError(t, err, "PutObject with SSE-C should fail with incorrect MD5")
t.Logf("Put object with SSE-C and incorrect MD5 rejected: %s/%s", bucketName, objectKey)
// Put object with SSE-C and correct MD5 should succeed
_, err = putObjectSSEC(cluster.s3Client, &s3.PutObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
Body: bytes.NewReader([]byte(objectData)),
ContentMD5: aws.String(correctMD5),
SSECustomerAlgorithm: aws.String("AES256"),
SSECustomerKey: aws.String(keyRaw),
SSECustomerKeyMD5: aws.String(keyMD5B64),
})
require.NoError(t, err, "Failed to put object with SSE-C and correct MD5")
// Verify object exists (SSE-C requires the key for HeadObject too)
headResp, err := headObjectSSEC(cluster.s3Client, &s3.HeadObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
SSECustomerAlgorithm: aws.String("AES256"),
SSECustomerKey: aws.String(keyRaw),
SSECustomerKeyMD5: aws.String(keyMD5B64),
})
require.NoError(t, err)
assert.NotNil(t, headResp.ContentLength)
assert.Equal(t, int64(len(objectData)), aws.Int64Value(headResp.ContentLength))
t.Logf("Put object with SSE-C and correct MD5: %s/%s (%d bytes)", bucketName, objectKey, len(objectData))
}
func testUploadPartWithChecksum(t *testing.T, cluster *TestCluster) {
bucketName := createTestBucket(t, cluster, "test-upload-part-checksum-")
objectKey := "test-multipart-checksum.txt"
objectData := "Hello, SeaweedFS S3 Multipart!"
// Initiate multipart upload
initResp, err := cluster.s3Client.CreateMultipartUpload(&s3.CreateMultipartUploadInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
})
require.NoError(t, err)
uploadID := initResp.UploadId
correctMD5 := calculateMd5(objectData)
incorrectMD5 := calculateMd5(objectData + "incorrect")
// Upload part with incorrect MD5
_, err = cluster.s3Client.UploadPart(&s3.UploadPartInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
PartNumber: aws.Int64(1),
UploadId: uploadID,
Body: bytes.NewReader([]byte(objectData)),
ContentMD5: aws.String(incorrectMD5),
})
assertBadDigestError(t, err, "UploadPart should fail with incorrect MD5")
// Upload part with correct MD5
partResp, err := cluster.s3Client.UploadPart(&s3.UploadPartInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
PartNumber: aws.Int64(1),
UploadId: uploadID,
Body: bytes.NewReader([]byte(objectData)),
ContentMD5: aws.String(correctMD5),
})
require.NoError(t, err, "Failed to upload part with correct MD5")
// Complete multipart upload
_, err = cluster.s3Client.CompleteMultipartUpload(&s3.CompleteMultipartUploadInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
UploadId: uploadID,
MultipartUpload: &s3.CompletedMultipartUpload{
Parts: []*s3.CompletedPart{
{
ETag: partResp.ETag,
PartNumber: aws.Int64(1),
},
},
},
})
require.NoError(t, err, "Failed to complete multipart upload")
// Verify object exists
headResp, err := cluster.s3Client.HeadObject(&s3.HeadObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
})
require.NoError(t, err)
assert.Equal(t, int64(len(objectData)), aws.Int64Value(headResp.ContentLength))
t.Logf("✓ Multipart upload with checksum successful: %s/%s", bucketName, objectKey)
}
func testPutPartWithChecksum(t *testing.T, cluster *TestCluster) {
bucketName := createTestBucket(t, cluster, "test-put-checksum-")
objectKey := "test-checksummed-part.txt"
partData := "Hello, SeaweedFS S3!"
correctMD5 := calculateMd5(partData)
incorrectMD5 := calculateMd5(partData + "incorrect")
createResp, err := cluster.s3Client.CreateMultipartUpload(&s3.CreateMultipartUploadInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
})
require.NoError(t, err)
uploadID := createResp.UploadId
partBody := []byte(partData)
_, err = cluster.s3Client.UploadPart(&s3.UploadPartInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
UploadId: uploadID,
PartNumber: aws.Int64(1),
Body: bytes.NewReader(partBody),
ContentMD5: aws.String(incorrectMD5),
})
assertBadDigestError(t, err, "UploadPart should fail with incorrect MD5")
uploadResp, err := cluster.s3Client.UploadPart(&s3.UploadPartInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
UploadId: uploadID,
PartNumber: aws.Int64(1),
Body: bytes.NewReader(partBody),
ContentMD5: aws.String(correctMD5),
})
require.NoError(t, err)
_, err = cluster.s3Client.CompleteMultipartUpload(&s3.CompleteMultipartUploadInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
UploadId: uploadID,
MultipartUpload: &s3.CompletedMultipartUpload{
Parts: []*s3.CompletedPart{
{
ETag: uploadResp.ETag,
PartNumber: aws.Int64(1),
},
},
},
})
require.NoError(t, err, "Failed to complete multipart upload")
// Verify object exists
headResp, err := cluster.s3Client.HeadObject(&s3.HeadObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
})
require.NoError(t, err)
assert.Equal(t, int64(len(partData)), aws.Int64Value(headResp.ContentLength))
t.Logf("✓ UploadPart with MD5 validation: %s/%s", bucketName, objectKey)
}
func calculateMd5(objectData string) string {
dataBytes := []byte(objectData)
hash := md5.Sum(dataBytes)
return base64.StdEncoding.EncodeToString(hash[:])
}
func assertBadDigestError(t *testing.T, err error, description string) {
require.Error(t, err, description)
var awsErr awserr.Error
require.ErrorAs(t, err, &awsErr)
assert.Equal(t, "BadDigest", awsErr.Code())
}
func testGetObject(t *testing.T, cluster *TestCluster) {
bucketName := "test-get-" + randomString(8)
objectKey := "test-data.txt"
@@ -476,6 +780,49 @@ func testDeleteBucket(t *testing.T, cluster *TestCluster) {
t.Logf("✓ Deleted bucket: %s", bucketName)
}
func TestS3DeleteBucketNotEmpty(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
cluster, err := startMiniCluster(t, "-s3.allowDeleteBucketNotEmpty=false")
require.NoError(t, err)
defer cluster.Stop()
t.Run("DeleteNonEmptyBucketFails", func(t *testing.T) {
bucketName := createTestBucket(t, cluster, "test-notempty-")
objectKey := "keep-me.txt"
// Put an object so the bucket is non-empty
_, err := cluster.s3Client.PutObject(&s3.PutObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
Body: bytes.NewReader([]byte("data")),
})
require.NoError(t, err)
// Attempt to delete the non-empty bucket — must fail with BucketNotEmpty (409)
_, err = cluster.s3Client.DeleteBucket(&s3.DeleteBucketInput{
Bucket: aws.String(bucketName),
})
require.Error(t, err, "deleting a non-empty bucket should fail")
var awsErr awserr.Error
require.ErrorAs(t, err, &awsErr)
assert.Equal(t, "BucketNotEmpty", awsErr.Code(),
"expected BucketNotEmpty error code, got %s: %s", awsErr.Code(), awsErr.Message())
})
t.Run("DeleteEmptyBucketSucceeds", func(t *testing.T) {
bucketName := createTestBucket(t, cluster, "test-empty-")
// Delete the empty bucket — should succeed even with the flag
_, err := cluster.s3Client.DeleteBucket(&s3.DeleteBucketInput{
Bucket: aws.String(bucketName),
})
require.NoError(t, err, "deleting an empty bucket should succeed")
})
}
// randomString generates a random string for unique naming
func randomString(length int) string {
const charset = "abcdefghijklmnopqrstuvwxyz0123456789"
+351
View File
@@ -2,6 +2,7 @@ package policy
import (
"context"
"errors"
"fmt"
"net"
"net/http"
@@ -14,6 +15,12 @@ import (
"testing"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/iam"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/seaweedfs/seaweedfs/weed/command"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb"
@@ -130,6 +137,291 @@ func TestS3PolicyShellRevised(t *testing.T) {
}
}
func TestS3IAMAttachDetachUserPolicy(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
cluster, err := startMiniCluster(t)
require.NoError(t, err)
defer cluster.Stop()
time.Sleep(500 * time.Millisecond)
policyName := uniqueName("managed-policy")
policyArn := fmt.Sprintf("arn:aws:iam:::policy/%s", policyName)
policyContent := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"*","Resource":"*"}]}`
tmpPolicyFile, err := os.CreateTemp("", "test_policy_attach_*.json")
require.NoError(t, err)
defer os.Remove(tmpPolicyFile.Name())
_, err = tmpPolicyFile.WriteString(policyContent)
require.NoError(t, err)
require.NoError(t, tmpPolicyFile.Close())
weedCmd := "weed"
masterAddr := string(pb.NewServerAddress("127.0.0.1", cluster.masterPort, cluster.masterGrpcPort))
filerAddr := string(pb.NewServerAddress("127.0.0.1", cluster.filerPort, cluster.filerGrpcPort))
execShell(t, weedCmd, masterAddr, filerAddr, fmt.Sprintf("s3.policy -put -name=%s -file=%s", policyName, tmpPolicyFile.Name()))
iamClient := newIAMClient(t, cluster.s3Endpoint)
userName := uniqueName("iam-user")
_, err = iamClient.CreateUser(&iam.CreateUserInput{UserName: aws.String(userName)})
require.NoError(t, err)
_, err = iamClient.AttachUserPolicy(&iam.AttachUserPolicyInput{
UserName: aws.String(userName),
PolicyArn: aws.String(policyArn),
})
require.NoError(t, err)
listOut, err := iamClient.ListAttachedUserPolicies(&iam.ListAttachedUserPoliciesInput{
UserName: aws.String(userName),
})
require.NoError(t, err)
require.True(t, attachedPolicyContains(listOut.AttachedPolicies, policyName))
_, err = iamClient.DetachUserPolicy(&iam.DetachUserPolicyInput{
UserName: aws.String(userName),
PolicyArn: aws.String(policyArn),
})
require.NoError(t, err)
listOut, err = iamClient.ListAttachedUserPolicies(&iam.ListAttachedUserPoliciesInput{
UserName: aws.String(userName),
})
require.NoError(t, err)
require.False(t, attachedPolicyContains(listOut.AttachedPolicies, policyName))
_, err = iamClient.AttachUserPolicy(&iam.AttachUserPolicyInput{
UserName: aws.String(userName),
PolicyArn: aws.String("arn:aws:iam:::policy/does-not-exist"),
})
require.Error(t, err)
if awsErr, ok := err.(awserr.Error); ok {
require.Equal(t, iam.ErrCodeNoSuchEntityException, awsErr.Code())
}
}
func TestS3IAMListPoliciesAndGetPolicy(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
cluster, err := startMiniCluster(t)
require.NoError(t, err)
defer cluster.Stop()
time.Sleep(500 * time.Millisecond)
policyName := uniqueName("managed-policy")
policyArn := fmt.Sprintf("arn:aws:iam:::policy/%s", policyName)
policyContent := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"s3:ListAllMyBuckets","Resource":"*"}]}`
iamClient := newIAMClient(t, cluster.s3Endpoint)
_, err = iamClient.CreatePolicy(&iam.CreatePolicyInput{
PolicyName: aws.String(policyName),
PolicyDocument: aws.String(policyContent),
})
require.NoError(t, err)
listOut, err := iamClient.ListPolicies(&iam.ListPoliciesInput{})
require.NoError(t, err)
require.True(t, managedPolicyContains(listOut.Policies, policyName))
getOut, err := iamClient.GetPolicy(&iam.GetPolicyInput{PolicyArn: aws.String(policyArn)})
require.NoError(t, err)
require.NotNil(t, getOut.Policy)
require.NotNil(t, getOut.Policy.PolicyName)
require.Equal(t, policyName, *getOut.Policy.PolicyName)
missingArn := fmt.Sprintf("arn:aws:iam:::policy/%s", uniqueName("missing"))
_, err = iamClient.GetPolicy(&iam.GetPolicyInput{PolicyArn: aws.String(missingArn)})
require.Error(t, err)
var awsErr awserr.Error
require.True(t, errors.As(err, &awsErr))
require.Equal(t, iam.ErrCodeNoSuchEntityException, awsErr.Code())
}
func TestS3IAMDeletePolicyInUse(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
cluster, err := startMiniCluster(t)
require.NoError(t, err)
defer cluster.Stop()
time.Sleep(500 * time.Millisecond)
policyName := uniqueName("managed-delete-policy")
policyArn := fmt.Sprintf("arn:aws:iam:::policy/%s", policyName)
policyContent := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"s3:*","Resource":"*"}]}`
iamClient := newIAMClient(t, cluster.s3Endpoint)
_, err = iamClient.CreatePolicy(&iam.CreatePolicyInput{
PolicyName: aws.String(policyName),
PolicyDocument: aws.String(policyContent),
})
require.NoError(t, err)
userName := uniqueName("iam-user-delete-policy")
_, err = iamClient.CreateUser(&iam.CreateUserInput{UserName: aws.String(userName)})
require.NoError(t, err)
_, err = iamClient.AttachUserPolicy(&iam.AttachUserPolicyInput{
UserName: aws.String(userName),
PolicyArn: aws.String(policyArn),
})
require.NoError(t, err)
_, err = iamClient.DeletePolicy(&iam.DeletePolicyInput{PolicyArn: aws.String(policyArn)})
require.Error(t, err)
var awsErr awserr.Error
require.True(t, errors.As(err, &awsErr))
require.Equal(t, iam.ErrCodeDeleteConflictException, awsErr.Code())
}
func TestS3MultipartOperationsInheritPutObjectPermissions(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
cluster, err := startMiniCluster(t)
require.NoError(t, err)
defer cluster.Stop()
time.Sleep(500 * time.Millisecond)
// Create a policy with only s3:PutObject permission
// This should implicitly allow multipart upload operations (CreateMultipartUpload,
// UploadPart, CompleteMultipartUpload, AbortMultipartUpload, ListBucketMultipartUploads, ListMultipartUploadParts)
policyName := uniqueName("putobject-policy")
policyArn := fmt.Sprintf("arn:aws:iam:::policy/%s", policyName)
policyContent := `{
"Version":"2012-10-17",
"Statement":[{
"Effect":"Allow",
"Action":"s3:PutObject",
"Resource":"*"
}]
}`
iamClient := newIAMClient(t, cluster.s3Endpoint)
_, err = iamClient.CreatePolicy(&iam.CreatePolicyInput{
PolicyName: aws.String(policyName),
PolicyDocument: aws.String(policyContent),
})
require.NoError(t, err)
// Create a user and attach the policy
userName := uniqueName("multipart-user")
_, err = iamClient.CreateUser(&iam.CreateUserInput{UserName: aws.String(userName)})
require.NoError(t, err)
_, err = iamClient.CreateAccessKey(&iam.CreateAccessKeyInput{UserName: aws.String(userName)})
require.NoError(t, err)
_, err = iamClient.AttachUserPolicy(&iam.AttachUserPolicyInput{
UserName: aws.String(userName),
PolicyArn: aws.String(policyArn),
})
require.NoError(t, err)
// Create S3 client with user credentials (using admin credentials for now, test will still validate permissions)
// In a real scenario, we'd use the user's access key
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1"),
Endpoint: aws.String(cluster.s3Endpoint),
DisableSSL: aws.Bool(true),
S3ForcePathStyle: aws.Bool(true),
Credentials: credentials.NewStaticCredentials("admin", "admin", ""),
})
require.NoError(t, err)
s3Client := newS3Client(sess)
bucketName := uniqueName("multipart-test-bucket")
objectKey := "test-object"
// Create bucket
_, err = s3Client.CreateBucket(&s3.CreateBucketInput{Bucket: aws.String(bucketName)})
require.NoError(t, err)
// Test multipart upload operations with s3:PutObject permission
// These operations should all succeed since multipart operations are part of s3:PutObject
// 1. CreateMultipartUpload
createOut, err := s3Client.CreateMultipartUpload(&s3.CreateMultipartUploadInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
})
require.NoError(t, err)
require.NotNil(t, createOut.UploadId)
uploadID := *createOut.UploadId
// 2. UploadPart
partBody := "test part data"
uploadPartOut, err := s3Client.UploadPart(&s3.UploadPartInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
PartNumber: aws.Int64(1),
UploadId: aws.String(uploadID),
Body: strings.NewReader(partBody),
})
require.NoError(t, err)
require.NotNil(t, uploadPartOut.ETag)
// 3. ListParts
listPartsOut, err := s3Client.ListParts(&s3.ListPartsInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
UploadId: aws.String(uploadID),
})
require.NoError(t, err)
require.Equal(t, 1, len(listPartsOut.Parts))
// 4. CompleteMultipartUpload
completeOut, err := s3Client.CompleteMultipartUpload(&s3.CompleteMultipartUploadInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
UploadId: aws.String(uploadID),
MultipartUpload: &s3.CompletedMultipartUpload{
Parts: []*s3.CompletedPart{
{
ETag: uploadPartOut.ETag,
PartNumber: aws.Int64(1),
},
},
},
})
require.NoError(t, err)
require.NotNil(t, completeOut.ETag)
// Test AbortMultipartUpload with a new upload
createOut2, err := s3Client.CreateMultipartUpload(&s3.CreateMultipartUploadInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey + "-abort"),
})
require.NoError(t, err)
uploadID2 := *createOut2.UploadId
_, err = s3Client.AbortMultipartUpload(&s3.AbortMultipartUploadInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey + "-abort"),
UploadId: aws.String(uploadID2),
})
require.NoError(t, err)
// Test ListMultipartUploads
listUploadsOut, err := s3Client.ListMultipartUploads(&s3.ListMultipartUploadsInput{
Bucket: aws.String(bucketName),
})
require.NoError(t, err)
// After aborting, there should be no active uploads
require.Equal(t, 0, len(listUploadsOut.Uploads))
}
func execShell(t *testing.T, weedCmd, master, filer, shellCmd string) string {
// weed shell -master=... -filer=...
args := []string{"shell", "-master=" + master, "-filer=" + filer}
@@ -145,6 +437,56 @@ func execShell(t *testing.T, weedCmd, master, filer, shellCmd string) string {
return string(out)
}
func newIAMClient(t *testing.T, endpoint string) *iam.IAM {
t.Helper()
accessKey := os.Getenv("AWS_ACCESS_KEY_ID")
secretKey := os.Getenv("AWS_SECRET_ACCESS_KEY")
if accessKey == "" {
accessKey = "admin"
}
if secretKey == "" {
secretKey = "admin"
}
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1"),
Endpoint: aws.String(endpoint),
DisableSSL: aws.Bool(true),
S3ForcePathStyle: aws.Bool(true),
Credentials: credentials.NewStaticCredentials(accessKey, secretKey, ""),
})
require.NoError(t, err)
return iam.New(sess)
}
func newS3Client(sess *session.Session) *s3.S3 {
return s3.New(sess)
}
func attachedPolicyContains(policies []*iam.AttachedPolicy, policyName string) bool {
for _, policy := range policies {
if policy.PolicyName != nil && *policy.PolicyName == policyName {
return true
}
}
return false
}
func managedPolicyContains(policies []*iam.Policy, policyName string) bool {
for _, policy := range policies {
if policy.PolicyName != nil && *policy.PolicyName == policyName {
return true
}
}
return false
}
func uniqueName(prefix string) string {
return fmt.Sprintf("%s-%s", prefix, strconv.FormatInt(time.Now().UnixNano(), 36))
}
// --- Test setup helpers ---
func findAvailablePort() (int, error) {
@@ -216,6 +558,14 @@ enabled = true
err = os.WriteFile(credentialToml, []byte(credentialConfig), 0644)
require.NoError(t, err)
// Set environment variables for admin credentials safely for this test
if os.Getenv("AWS_ACCESS_KEY_ID") == "" {
t.Setenv("AWS_ACCESS_KEY_ID", "admin")
}
if os.Getenv("AWS_SECRET_ACCESS_KEY") == "" {
t.Setenv("AWS_SECRET_ACCESS_KEY", "admin")
}
cluster.wg.Add(1)
go func() {
defer cluster.wg.Done()
@@ -242,6 +592,7 @@ enabled = true
"-master.volumeSizeLimitMB=32",
"-ip=127.0.0.1",
"-master.peers=none",
"-s3.iam.readOnly=false",
}
glog.MaxSize = 1024 * 1024
for _, cmd := range command.Commands {
+12
View File
@@ -0,0 +1,12 @@
FROM alpine:3.20
RUN apk add --no-cache curl && \
addgroup -S seaweed && \
adduser -S seaweed -G seaweed
COPY weed /usr/bin/weed
RUN chmod +x /usr/bin/weed && \
chown seaweed:seaweed /usr/bin/weed && \
mkdir -p /etc/seaweedfs /data/filerldb2 && \
chown -R seaweed:seaweed /etc/seaweedfs /data && \
chmod 755 /data /etc/seaweedfs /data/filerldb2
WORKDIR /data
USER seaweed
+79
View File
@@ -0,0 +1,79 @@
# S3 Proxy Signature Verification Test
Integration test that verifies S3 signature verification works correctly when
SeaweedFS is deployed behind a reverse proxy (nginx).
## What it tests
- S3 operations (create bucket, put/get/head/list/delete) through an nginx
reverse proxy with `X-Forwarded-Host`, `X-Forwarded-Port`, and
`X-Forwarded-Proto` headers
- SeaweedFS configured with `-s3.externalUrl=http://localhost:9000` so the
signature verification uses the client-facing host instead of the internal
backend address
## Architecture
```text
AWS CLI (signs with Host: localhost:9000)
|
v
nginx (:9000)
| proxy_pass → seaweedfs:8333
| Sets: X-Forwarded-Host: localhost
| X-Forwarded-Port: 9000
| X-Forwarded-Proto: http
v
SeaweedFS S3 (:8333, -s3.externalUrl=http://localhost:9000)
| externalHost = "localhost:9000" (parsed at startup)
| extractHostHeader() returns "localhost:9000"
| Matches what AWS CLI signed with
v
Signature verification succeeds
```
**Note:** When `-s3.externalUrl` is configured, direct access to the backend
port (8333) will fail signature verification because the client signs with a
different Host header than what `externalUrl` specifies. This is expected —
all S3 traffic should go through the proxy.
## Prerequisites
- Docker and Docker Compose
- AWS CLI v2 (on host or via Docker, see below)
## Running
```bash
# Build the weed binary first (from repo root):
cd /path/to/seaweedfs
go build -o test/s3/proxy_signature/weed ./weed
cd test/s3/proxy_signature
# Start services
docker compose up -d --build
# Option A: Run test with aws CLI installed locally
./test.sh
# Option B: Run test without aws CLI (uses Docker container)
docker run --rm --network host --entrypoint "" amazon/aws-cli:latest \
bash < test.sh
# Tear down
docker compose down
# Clean up the weed binary
rm -f weed
```
## Troubleshooting
If signature verification fails through the proxy, check:
1. nginx is setting `X-Forwarded-Host` and `X-Forwarded-Port` correctly
2. SeaweedFS is started with `-s3.externalUrl` matching the client endpoint
3. The AWS CLI endpoint URL matches the proxy address
You can also set the `S3_EXTERNAL_URL` environment variable instead of the
`-s3.externalUrl` flag.
@@ -0,0 +1,28 @@
services:
seaweedfs:
build:
context: .
dockerfile: Dockerfile
command: >
/usr/bin/weed mini
-s3.config=/etc/seaweedfs/s3.json
-s3.externalUrl=http://localhost:9000
-ip=seaweedfs
volumes:
- ./s3.json:/etc/seaweedfs/s3.json:ro
healthcheck:
test: ["CMD", "curl", "-sf", "http://seaweedfs:8333/status"]
interval: 3s
timeout: 2s
retries: 20
start_period: 5s
nginx:
image: nginx:alpine
ports:
- "9000:9000"
volumes:
- ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
depends_on:
seaweedfs:
condition: service_healthy
+23
View File
@@ -0,0 +1,23 @@
server {
listen 9000;
server_name localhost;
# Allow large uploads
client_max_body_size 64m;
location / {
proxy_pass http://seaweedfs:8333;
# Standard reverse proxy headers — this is what Kong, Traefik, etc. do
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_buffering off;
proxy_request_buffering off;
}
}
+20
View File
@@ -0,0 +1,20 @@
{
"identities": [
{
"name": "test_admin",
"credentials": [
{
"accessKey": "test_access_key",
"secretKey": "test_secret_key"
}
],
"actions": [
"Admin",
"Read",
"List",
"Tagging",
"Write"
]
}
]
}
+132
View File
@@ -0,0 +1,132 @@
#!/usr/bin/env bash
#
# Integration test for S3 signature verification behind a reverse proxy.
#
# Usage:
# # With aws CLI installed locally:
# docker compose up -d --build && ./test.sh && docker compose down
#
# # Without aws CLI (runs test inside a container):
# docker compose up -d --build
# docker run --rm --network host --entrypoint "" amazon/aws-cli:latest \
# bash < test.sh
# docker compose down
#
# This script tests S3 operations through an nginx reverse proxy to verify
# that signature verification works correctly when SeaweedFS is configured
# with -s3.externalUrl=http://localhost:9000.
#
set -euo pipefail
PROXY_ENDPOINT="http://localhost:9000"
ACCESS_KEY="test_access_key"
SECRET_KEY="test_secret_key"
REGION="us-east-1"
BUCKET="test-proxy-sig-$$"
RED='\033[0;31m'
GREEN='\033[0;32m'
NC='\033[0m'
pass() { echo -e "${GREEN}PASS${NC}: $1"; }
fail() { echo -e "${RED}FAIL${NC}: $1"; exit 1; }
# Helper: run aws s3api command against a given endpoint
s3() {
local endpoint="$1"
shift
aws s3api \
--endpoint-url "$endpoint" \
--region "$REGION" \
--no-verify-ssl \
"$@" 2>&1
}
export AWS_ACCESS_KEY_ID="$ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="$SECRET_KEY"
echo "=== S3 Proxy Signature Verification Test ==="
echo ""
echo "Testing S3 access through nginx reverse proxy at $PROXY_ENDPOINT"
echo "SeaweedFS configured with -s3.externalUrl=http://localhost:9000"
echo "AWS CLI signs requests with Host: localhost:9000"
echo ""
# Wait for proxy to be ready
echo "Waiting for nginx proxy to be ready..."
for i in $(seq 1 30); do
# Use aws CLI for health check if curl is missing
if command -v curl >/dev/null 2>&1; then
http_code=$(curl -s -o /dev/null -w "%{http_code}" "$PROXY_ENDPOINT/" 2>/dev/null || echo "000")
case $http_code in
200|403|405) break ;;
esac
else
if aws s3api list-buckets --endpoint-url "$PROXY_ENDPOINT" --no-sign-request >/dev/null 2>&1; then
break
fi
fi
if [ "$i" -eq 30 ]; then
fail "Proxy did not become ready in time"
fi
echo "Waiting for proxy $i/30..."
sleep 1
done
echo "Proxy is ready."
echo ""
# --- Test 1: Bucket operations through proxy ---
echo "--- Test 1: Bucket operations through proxy ---"
s3 "$PROXY_ENDPOINT" create-bucket --bucket "$BUCKET" > /dev/null \
&& pass "create-bucket" \
|| fail "create-bucket — signature verification likely failed"
s3 "$PROXY_ENDPOINT" list-buckets > /dev/null \
&& pass "list-buckets" \
|| fail "list-buckets"
echo ""
# --- Test 2: Object CRUD through proxy ---
echo "--- Test 2: Object CRUD through proxy ---"
echo "hello-from-proxy" > /tmp/test-proxy-sig.txt
s3 "$PROXY_ENDPOINT" put-object --bucket "$BUCKET" --key "test.txt" --body /tmp/test-proxy-sig.txt > /dev/null \
&& pass "put-object" \
|| fail "put-object"
s3 "$PROXY_ENDPOINT" head-object --bucket "$BUCKET" --key "test.txt" > /dev/null \
&& pass "head-object" \
|| fail "head-object"
s3 "$PROXY_ENDPOINT" list-objects-v2 --bucket "$BUCKET" > /dev/null \
&& pass "list-objects-v2" \
|| fail "list-objects-v2"
s3 "$PROXY_ENDPOINT" get-object --bucket "$BUCKET" --key "test.txt" /tmp/test-proxy-sig-get.txt > /dev/null \
&& pass "get-object" \
|| fail "get-object"
# Verify content round-trip
CONTENT=$(cat /tmp/test-proxy-sig-get.txt)
if [ "$CONTENT" = "hello-from-proxy" ]; then
pass "content integrity (round-trip)"
else
fail "content mismatch: got \"$CONTENT\", expected \"hello-from-proxy\""
fi
echo ""
# --- Test 3: Delete operations through proxy ---
echo "--- Test 3: Delete through proxy ---"
s3 "$PROXY_ENDPOINT" delete-object --bucket "$BUCKET" --key "test.txt" > /dev/null \
&& pass "delete-object" \
|| fail "delete-object"
s3 "$PROXY_ENDPOINT" delete-bucket --bucket "$BUCKET" > /dev/null \
&& pass "delete-bucket" \
|| fail "delete-bucket"
echo ""
# Cleanup temp files
rm -f /tmp/test-proxy-sig.txt /tmp/test-proxy-sig-get.txt
echo "=== All tests passed ==="
-1
View File
@@ -1,5 +1,4 @@
//go:build integration
// +build integration
package sse
Binary file not shown.
Binary file not shown.
+131 -83
View File
@@ -6,6 +6,7 @@ import (
"bytes"
"context"
"encoding/json"
"flag"
"fmt"
"io"
"net"
@@ -18,6 +19,37 @@ import (
"time"
)
// sharedEnv is the single TestEnvironment shared across all tests in this package.
var sharedEnv *TestEnvironment
// TestMain starts one weed mini instance for the whole package and tears it down
// after all tests have run.
func TestMain(m *testing.M) {
flag.Parse()
if os.Getenv("SHORT") != "" || testing.Short() {
// Let tests self-skip when run with -short.
os.Exit(m.Run())
}
env, err := newTestEnvironmentForMain()
if err != nil {
fmt.Fprintf(os.Stderr, "SKIP: setup failed: %v\n", err)
os.Exit(0) // Skip all tests rather than fail
}
sharedEnv = env
if startErr := sharedEnv.startSeaweedFSForMain(); startErr != nil {
fmt.Fprintf(os.Stderr, "SKIP: weed mini failed to start: %v\n", startErr)
sharedEnv.cleanupForMain()
os.Exit(0)
}
code := m.Run()
sharedEnv.cleanupForMain()
os.Exit(code)
}
// TestEnvironment contains the test environment configuration
type TestEnvironment struct {
seaweedDir string
@@ -54,17 +86,15 @@ func getFreePort() (int, net.Listener, error) {
return addr.Port, listener, nil
}
// NewTestEnvironment creates a new test environment
func NewTestEnvironment(t *testing.T) *TestEnvironment {
t.Helper()
// newTestEnvironmentForMain creates a TestEnvironment without calling t.Fatalf so it
// can be used from TestMain (which has no *testing.T).
func newTestEnvironmentForMain() (*TestEnvironment, error) {
// Find the SeaweedFS root directory
wd, err := os.Getwd()
if err != nil {
t.Fatalf("Failed to get working directory: %v", err)
return nil, fmt.Errorf("get working directory: %w", err)
}
// Navigate up to find the SeaweedFS root (contains go.mod)
seaweedDir := wd
for i := 0; i < 5; i++ {
if _, err := os.Stat(filepath.Join(seaweedDir, "go.mod")); err == nil {
@@ -76,82 +106,93 @@ func NewTestEnvironment(t *testing.T) *TestEnvironment {
// Check for weed binary
weedBinary := filepath.Join(seaweedDir, "weed", "weed")
if _, err := os.Stat(weedBinary); os.IsNotExist(err) {
// Try system PATH
weedBinary = "weed"
if _, err := exec.LookPath(weedBinary); err != nil {
t.Skip("weed binary not found, skipping integration test")
return nil, fmt.Errorf("weed binary not found")
}
}
// Create temporary data directory
dataDir, err := os.MkdirTemp("", "seaweed-iceberg-test-*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
return nil, fmt.Errorf("create temp dir: %w", err)
}
// Allocate free ephemeral ports for each service
var listeners []net.Listener
defer func() {
closeListeners := func() {
for _, l := range listeners {
l.Close()
}
}()
}
var l net.Listener
s3Port, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for S3: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for S3: %w", err)
}
listeners = append(listeners, l)
icebergPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Iceberg: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Iceberg: %w", err)
}
listeners = append(listeners, l)
s3GrpcPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for S3 gRPC: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for S3 gRPC: %w", err)
}
listeners = append(listeners, l)
masterPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Master: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Master: %w", err)
}
listeners = append(listeners, l)
masterGrpcPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Master gRPC: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Master gRPC: %w", err)
}
listeners = append(listeners, l)
filerPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Filer: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Filer: %w", err)
}
listeners = append(listeners, l)
filerGrpcPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Filer gRPC: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Filer gRPC: %w", err)
}
listeners = append(listeners, l)
volumePort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Volume: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Volume: %w", err)
}
listeners = append(listeners, l)
volumeGrpcPort, l, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for Volume gRPC: %v", err)
closeListeners()
return nil, fmt.Errorf("get free port for Volume gRPC: %w", err)
}
listeners = append(listeners, l)
// Release the port reservations so weed mini can bind to them
closeListeners()
return &TestEnvironment{
seaweedDir: seaweedDir,
weedBinary: weedBinary,
@@ -166,13 +207,11 @@ func NewTestEnvironment(t *testing.T) *TestEnvironment {
volumePort: volumePort,
volumeGrpcPort: volumeGrpcPort,
dockerAvailable: hasDocker(),
}
}, nil
}
// StartSeaweedFS starts a SeaweedFS mini cluster
func (env *TestEnvironment) StartSeaweedFS(t *testing.T) {
t.Helper()
// startSeaweedFSForMain starts weed mini without a *testing.T (for use in TestMain).
func (env *TestEnvironment) startSeaweedFSForMain() error {
ctx, cancel := context.WithCancel(context.Background())
env.weedCancel = cancel
@@ -182,7 +221,8 @@ func (env *TestEnvironment) StartSeaweedFS(t *testing.T) {
for _, dir := range []string{masterDir, filerDir, volumeDir} {
if err := os.MkdirAll(dir, 0755); err != nil {
t.Fatalf("Failed to create directory %s: %v", dir, err)
cancel()
return fmt.Errorf("create directory %s: %w", dir, err)
}
}
@@ -203,13 +243,30 @@ func (env *TestEnvironment) StartSeaweedFS(t *testing.T) {
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
t.Fatalf("Failed to start SeaweedFS: %v", err)
cancel()
return fmt.Errorf("start SeaweedFS: %w", err)
}
env.weedProcess = cmd
// Wait for services to be ready
if !env.waitForService(fmt.Sprintf("http://127.0.0.1:%d/v1/config", env.icebergPort), 30*time.Second) {
t.Fatalf("Iceberg REST API did not become ready")
cancel()
cmd.Wait()
return fmt.Errorf("Iceberg REST API did not become ready")
}
return nil
}
// cleanupForMain stops SeaweedFS and cleans up resources (no *testing.T needed).
func (env *TestEnvironment) cleanupForMain() {
if env.weedCancel != nil {
env.weedCancel()
}
if env.weedProcess != nil {
time.Sleep(2 * time.Second)
env.weedProcess.Wait()
}
if env.dataDir != "" {
os.RemoveAll(env.dataDir)
}
}
@@ -230,25 +287,6 @@ func (env *TestEnvironment) waitForService(url string, timeout time.Duration) bo
return false
}
// Cleanup stops SeaweedFS and cleans up resources
func (env *TestEnvironment) Cleanup(t *testing.T) {
t.Helper()
if env.weedCancel != nil {
env.weedCancel()
}
if env.weedProcess != nil {
// Give process time to shut down gracefully
time.Sleep(2 * time.Second)
env.weedProcess.Wait()
}
if env.dataDir != "" {
os.RemoveAll(env.dataDir)
}
}
// IcebergURL returns the Iceberg REST Catalog URL
func (env *TestEnvironment) IcebergURL() string {
return fmt.Sprintf("http://127.0.0.1:%d", env.icebergPort)
@@ -260,10 +298,7 @@ func TestIcebergConfig(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
env.StartSeaweedFS(t)
env := sharedEnv
// Test GET /v1/config
resp, err := http.Get(env.IcebergURL() + "/v1/config")
@@ -294,16 +329,14 @@ func TestIcebergNamespaces(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
env.StartSeaweedFS(t)
env := sharedEnv
// Create the default table bucket first via S3
createTableBucket(t, env, "warehouse")
bucketName := "warehouse-ns-" + randomSuffix()
createTableBucket(t, env, bucketName)
// Test GET /v1/namespaces (should return empty list initially)
resp, err := http.Get(env.IcebergURL() + "/v1/namespaces")
resp, err := http.Get(env.IcebergURL() + icebergPath(bucketName, "/v1/namespaces"))
if err != nil {
t.Fatalf("Failed to list namespaces: %v", err)
}
@@ -321,16 +354,14 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
env := sharedEnv
bucketName := "warehouse-stage-" + randomSuffix()
createTableBucket(t, env, bucketName)
env.StartSeaweedFS(t)
createTableBucket(t, env, "warehouse")
namespace := "stage_ns"
namespace := "stage_ns_" + randomSuffix()
tableName := "orders"
status, _, err := doIcebergJSONRequest(env, http.MethodPost, "/v1/namespaces", map[string]any{
status, _, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, "/v1/namespaces"), map[string]any{
"namespace": []string{namespace},
})
if err != nil {
@@ -340,7 +371,7 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) {
t.Fatalf("Create namespace status = %d, want 200 or 409", status)
}
status, badReqResp, err := doIcebergJSONRequest(env, http.MethodPost, fmt.Sprintf("/v1/namespaces/%s/tables", namespace), map[string]any{
status, badReqResp, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables", namespace)), map[string]any{
"stage-create": true,
})
if err != nil {
@@ -358,7 +389,7 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) {
t.Fatalf("error.message = %v, want it to include %q", errorObj["message"], "table name is required")
}
status, stageResp, err := doIcebergJSONRequest(env, http.MethodPost, fmt.Sprintf("/v1/namespaces/%s/tables", namespace), map[string]any{
status, stageResp, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables", namespace)), map[string]any{
"name": tableName,
"stage-create": true,
})
@@ -373,7 +404,7 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) {
t.Fatalf("stage metadata-location = %q, want suffix /metadata/v1.metadata.json", stageLocation)
}
status, _, err = doIcebergJSONRequest(env, http.MethodGet, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName), nil)
status, _, err = doIcebergJSONRequest(env, http.MethodGet, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), nil)
if err != nil {
t.Fatalf("Load staged table request failed: %v", err)
}
@@ -381,7 +412,7 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) {
t.Fatalf("Load staged table status = %d, want 404", status)
}
status, commitResp, err := doIcebergJSONRequest(env, http.MethodPost, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName), map[string]any{
status, commitResp, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), map[string]any{
"requirements": []map[string]any{
{"type": "assert-create"},
},
@@ -398,7 +429,7 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) {
t.Fatalf("final metadata-location = %q, want suffix /metadata/v1.metadata.json", commitLocation)
}
status, loadResp, err := doIcebergJSONRequest(env, http.MethodGet, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName), nil)
status, loadResp, err := doIcebergJSONRequest(env, http.MethodGet, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), nil)
if err != nil {
t.Fatalf("Load finalized table request failed: %v", err)
}
@@ -417,16 +448,14 @@ func TestCommitMissingTableWithoutAssertCreate(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
env := sharedEnv
bucketName := "warehouse-missing-" + randomSuffix()
createTableBucket(t, env, bucketName)
env.StartSeaweedFS(t)
createTableBucket(t, env, "warehouse")
namespace := "stage_missing_assert_ns"
namespace := "stage_missing_assert_ns_" + randomSuffix()
tableName := "missing_table"
status, _, err := doIcebergJSONRequest(env, http.MethodPost, "/v1/namespaces", map[string]any{
status, _, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, "/v1/namespaces"), map[string]any{
"namespace": []string{namespace},
})
if err != nil {
@@ -436,7 +465,7 @@ func TestCommitMissingTableWithoutAssertCreate(t *testing.T) {
t.Fatalf("Create namespace status = %d, want 200 or 409", status)
}
status, _, err = doIcebergJSONRequest(env, http.MethodPost, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName), map[string]any{
status, _, err = doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), map[string]any{
"requirements": []any{},
"updates": []any{},
})
@@ -491,6 +520,21 @@ func doIcebergJSONRequest(env *TestEnvironment, method, path string, payload any
return resp.StatusCode, decoded, nil
}
// icebergPath inserts the table bucket prefix into Iceberg REST API paths.
// For example, "/v1/namespaces" with prefix "my-bucket" becomes
// "/v1/my-bucket/namespaces".
func icebergPath(prefix, path string) string {
if prefix == "" {
return path
}
const base = "/v1/"
if !strings.HasPrefix(path, base) {
return path
}
withPrefix := base + prefix + "/" + strings.TrimPrefix(path, base)
return withPrefix
}
// createTableBucket creates a table bucket via the S3Tables REST API
func createTableBucket(t *testing.T, env *TestEnvironment, bucketName string) {
t.Helper()
@@ -511,13 +555,20 @@ func createTableBucket(t *testing.T, env *TestEnvironment, bucketName string) {
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
t.Logf("Create table bucket %s response: status=%d, body=%s", bucketName, resp.StatusCode, string(body))
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusConflict {
body, _ := io.ReadAll(resp.Body)
t.Fatalf("Failed to create table bucket %s, status %d: %s", bucketName, resp.StatusCode, body)
}
t.Logf("Created table bucket %s", bucketName)
}
// randomSuffix returns a short random hex suffix for unique resource naming.
func randomSuffix() string {
return fmt.Sprintf("%x", time.Now().UnixNano()&0xffffffff)
}
// TestDuckDBIntegration tests Iceberg catalog operations using DuckDB
// This test requires Docker to be available
func TestDuckDBIntegration(t *testing.T) {
@@ -525,15 +576,12 @@ func TestDuckDBIntegration(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
env := sharedEnv
if !env.dockerAvailable {
t.Skip("Docker not available, skipping DuckDB integration test")
}
env.StartSeaweedFS(t)
// Create a temporary SQL file for DuckDB to execute
sqlFile := filepath.Join(env.dataDir, "test.sql")
sqlContent := fmt.Sprintf(`
+64 -7
View File
@@ -26,17 +26,14 @@ func TestPyIcebergRestCatalog(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
env := sharedEnv
if !env.dockerAvailable {
t.Skip("Docker not available, skipping PyIceberg integration test")
}
env.StartSeaweedFS(t)
// Create the test bucket first
bucketName := "pyiceberg-compat-test"
bucketName := "pyiceberg-compat-test-" + randomSuffix()
createTableBucket(t, env, bucketName)
// Build the test working directory path
@@ -56,8 +53,6 @@ func TestPyIcebergRestCatalog(t *testing.T) {
cmd := exec.Command("docker", "run", "--rm",
"--add-host", "host.docker.internal:host-gateway",
"-e", fmt.Sprintf("AWS_ACCESS_KEY_ID=%s", "test"),
"-e", fmt.Sprintf("AWS_SECRET_ACCESS_KEY=%s", "test"),
"-e", fmt.Sprintf("AWS_ENDPOINT_URL=%s", s3Endpoint),
"-v", fmt.Sprintf("%s:/app:ro", testDir),
"iceberg-rest-test",
@@ -78,3 +73,65 @@ func TestPyIcebergRestCatalog(t *testing.T) {
t.Errorf("PyIceberg test failed: %v", err)
}
}
// TestPyIcebergRestCatalogAuthenticated tests the Iceberg REST Catalog using PyIceberg with authentication.
// This test uses the default admin credentials that SeaweedFS creates on startup.
func TestPyIcebergRestCatalogAuthenticated(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
if !env.dockerAvailable {
t.Skip("Docker not available, skipping PyIceberg integration test")
}
// Use default admin credentials
testAccessKey := "admin"
testSecretKey := "admin"
// Create the test bucket first (using unauthenticated request, which works with DefaultAllow)
bucketName := "pyiceberg-auth-test-" + randomSuffix()
createTableBucket(t, env, bucketName)
// Build the test working directory path
testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog")
// Run PyIceberg test using Docker with authentication
catalogURL := fmt.Sprintf("http://host.docker.internal:%d", env.icebergPort)
s3Endpoint := fmt.Sprintf("http://host.docker.internal:%d", env.s3Port)
warehouse := fmt.Sprintf("s3://%s/", bucketName)
// Build the test image first for faster repeated runs
buildCmd := exec.Command("docker", "build", "-t", "iceberg-rest-test", "-f", "Dockerfile.pyiceberg", ".")
buildCmd.Dir = testDir
if out, err := buildCmd.CombinedOutput(); err != nil {
t.Fatalf("Failed to build test image: %v\n%s", err, string(out))
}
cmd := exec.Command("docker", "run", "--rm",
"--add-host", "host.docker.internal:host-gateway",
"-e", fmt.Sprintf("AWS_ENDPOINT_URL=%s", s3Endpoint),
"-v", fmt.Sprintf("%s:/app:ro", testDir),
"iceberg-rest-test",
"python3", "/app/test_rest_catalog_auth.py",
"--catalog-url", catalogURL,
"--warehouse", warehouse,
"--prefix", bucketName,
"--access-key", testAccessKey,
"--secret-key", testSecretKey,
)
cmd.Dir = testDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
t.Logf("Running PyIceberg REST catalog test with authentication...")
t.Logf(" Catalog URL: %s", catalogURL)
t.Logf(" Warehouse: %s", warehouse)
t.Logf(" Access Key: %s", testAccessKey)
if err := cmd.Run(); err != nil {
t.Errorf("PyIceberg authenticated test failed: %v", err)
}
}
@@ -0,0 +1,36 @@
package catalog
import (
"fmt"
"io"
"net/http"
"testing"
)
// verifyTableBucketMetadata verifies that a table bucket was created with proper metadata
func verifyTableBucketMetadata(t *testing.T, env *TestEnvironment, bucketName string) {
t.Helper()
// Use S3Tables REST API to get the bucket
endpoint := fmt.Sprintf("http://localhost:%d/buckets/%s", env.s3Port, bucketName)
req, err := http.NewRequest(http.MethodGet, endpoint, nil)
if err != nil {
t.Fatalf("Failed to create request: %v", err)
}
req.Header.Set("Content-Type", "application/x-amz-json-1.1")
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("Failed to get table bucket %s: %v", bucketName, err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
t.Logf("Get table bucket %s response: status=%d, body=%s", bucketName, resp.StatusCode, string(body))
if resp.StatusCode != http.StatusOK {
t.Fatalf("Failed to get table bucket %s, status %d: %s", bucketName, resp.StatusCode, body)
}
t.Logf("Verified table bucket %s exists with metadata", bucketName)
}
@@ -201,6 +201,7 @@ def main():
"uri": args.catalog_url,
"warehouse": args.warehouse,
"prefix": args.prefix,
"s3.anonymous": "true", # Disable AWS request signing for unauthenticated access
}
)
print(f"Successfully connected to catalog on attempt {attempt + 1}")
@@ -0,0 +1,223 @@
#!/usr/bin/env python3
"""
Iceberg REST Catalog Compatibility Test for SeaweedFS (Authenticated)
This script tests the Iceberg REST Catalog API compatibility with authentication.
Usage:
python3 test_rest_catalog_auth.py --catalog-url http://localhost:8182 \\
--access-key admin --secret-key admin
Requirements:
pip install pyiceberg[s3fs]
"""
import argparse
import sys
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import (
IntegerType,
LongType,
StringType,
NestedField,
)
from pyiceberg.exceptions import (
NamespaceAlreadyExistsError,
NoSuchNamespaceError,
TableAlreadyExistsError,
NoSuchTableError,
)
def test_config_endpoint(catalog):
"""Test that the catalog config endpoint returns valid configuration."""
print("Testing /v1/config endpoint...")
# The catalog is already loaded which means config endpoint worked
print(" /v1/config endpoint working")
return True
def test_namespace_operations(catalog, prefix):
"""Test namespace CRUD operations."""
print("Testing namespace operations...")
namespace = (f"{prefix.replace('-', '_')}_auth_test_ns",)
# List initial namespaces
namespaces = catalog.list_namespaces()
print(f" Initial namespaces: {namespaces}")
# Create namespace
try:
catalog.create_namespace(namespace)
print(f" Created namespace: {namespace}")
except NamespaceAlreadyExistsError:
print(f" ! Namespace already exists: {namespace}")
# List namespaces (should include our new one)
namespaces = catalog.list_namespaces()
if namespace in namespaces:
print(" Namespace appears in list")
else:
print(f" Namespace not found in list: {namespaces}")
return False
# Get namespace properties
try:
props = catalog.load_namespace_properties(namespace)
print(f" Loaded namespace properties: {props}")
except NoSuchNamespaceError:
print(f" Failed to load namespace properties")
return False
return True
def test_table_operations(catalog, prefix):
"""Test table CRUD operations."""
print("Testing table operations...")
namespace = (f"{prefix.replace('-', '_')}_auth_test_ns",)
table_name = "auth_test_table"
table_id = namespace + (table_name,)
# Define a simple schema
schema = Schema(
NestedField(field_id=1, name="id", field_type=LongType(), required=True),
NestedField(field_id=2, name="name", field_type=StringType(), required=False),
NestedField(field_id=3, name="age", field_type=IntegerType(), required=False),
)
# Create table
try:
table = catalog.create_table(
identifier=table_id,
schema=schema,
)
print(f" Created table: {table_id}")
except TableAlreadyExistsError:
print(f" ! Table already exists: {table_id}")
_ = catalog.load_table(table_id)
# List tables
tables = catalog.list_tables(namespace)
if table_name in [t[1] for t in tables]:
print(" Table appears in list")
else:
print(f" Table not found in list: {tables}")
return False
# Load table
try:
loaded_table = catalog.load_table(table_id)
print(f" Loaded table: {loaded_table.name()}")
print(f" Schema: {loaded_table.schema()}")
print(f" Location: {loaded_table.location()}")
except NoSuchTableError:
print(f" Failed to load table")
return False
return True
def test_cleanup(catalog, prefix):
"""Test table and namespace deletion."""
print("Testing cleanup operations...")
namespace = (f"{prefix.replace('-', '_')}_auth_test_ns",)
table_id = namespace + ("auth_test_table",)
# Drop table
try:
catalog.drop_table(table_id)
print(f" Dropped table: {table_id}")
except NoSuchTableError:
print(f" ! Table already deleted: {table_id}")
# Drop namespace
try:
catalog.drop_namespace(namespace)
print(f" Dropped namespace: {namespace}")
except NoSuchNamespaceError:
print(f" ! Namespace already deleted: {namespace}")
except Exception as e:
print(f" ? Namespace drop error (may be expected): {e}")
return True
def main():
parser = argparse.ArgumentParser(description="Test Iceberg REST Catalog with authentication")
parser.add_argument("--catalog-url", required=True, help="Iceberg REST Catalog URL")
parser.add_argument("--warehouse", default="s3://iceberg-test/", help="Warehouse location")
parser.add_argument("--prefix", required=True, help="Table bucket prefix")
parser.add_argument("--access-key", required=True, help="AWS Access Key ID")
parser.add_argument("--secret-key", required=True, help="AWS Secret Access Key")
parser.add_argument("--skip-cleanup", action="store_true", help="Skip cleanup at the end")
args = parser.parse_args()
print(f"Connecting to Iceberg REST Catalog at: {args.catalog_url}")
print(f"Warehouse: {args.warehouse}")
print(f"Prefix: {args.prefix}")
print(f"Using authenticated access with key: {args.access_key}")
print()
# Load the REST catalog with authentication
import time
max_retries = 10
catalog = None
for attempt in range(max_retries):
try:
catalog = load_catalog(
"rest",
**{
"type": "rest",
"uri": args.catalog_url,
"warehouse": args.warehouse,
"prefix": args.prefix,
"s3.access-key-id": args.access_key,
"s3.secret-access-key": args.secret_key,
}
)
print(f"Successfully connected to catalog on attempt {attempt + 1}")
break
except Exception as e:
if attempt < max_retries - 1:
print(f" Attempt {attempt + 1} failed, retrying in 2s... ({e})")
time.sleep(2)
else:
print(f" All {max_retries} attempts failed.")
raise e
# Run tests
tests = [
("Config Endpoint", lambda: test_config_endpoint(catalog)),
("Namespace Operations", lambda: test_namespace_operations(catalog, args.prefix)),
("Table Operations", lambda: test_table_operations(catalog, args.prefix)),
]
if not args.skip_cleanup:
tests.append(("Cleanup", lambda: test_cleanup(catalog, args.prefix)))
passed = 0
failed = 0
for name, test_fn in tests:
print(f"\n{'='*50}")
try:
if test_fn():
passed += 1
print(f"PASSED: {name}")
else:
failed += 1
print(f"FAILED: {name}")
except Exception as e:
failed += 1
print(f"ERROR in {name}: {e}")
print(f"\n{'='*50}")
print(f"Results: {passed} passed, {failed} failed")
return 0 if failed == 0 else 1
if __name__ == "__main__":
sys.exit(main())
@@ -40,6 +40,7 @@ type TestEnvironment struct {
dockerAvailable bool
accessKey string
secretKey string
closers []io.Closer
}
func TestTrinoIcebergCatalog(t *testing.T) {
@@ -118,30 +119,66 @@ func NewTestEnvironment(t *testing.T) *TestEnvironment {
bindIP := testutil.FindBindIP()
masterPort, masterGrpcPort := mustFreePortPair(t, "Master")
volumePort, volumeGrpcPort := mustFreePortPair(t, "Volume")
filerPort, filerGrpcPort := mustFreePortPair(t, "Filer")
s3Port, s3GrpcPort := mustFreePortPair(t, "S3")
icebergPort := mustFreePort(t, "Iceberg")
return &TestEnvironment{
seaweedDir: seaweedDir,
weedBinary: weedBinary,
dataDir: dataDir,
bindIP: bindIP,
s3Port: s3Port,
s3GrpcPort: s3GrpcPort,
icebergPort: icebergPort,
masterPort: masterPort,
masterGrpcPort: masterGrpcPort,
filerPort: filerPort,
filerGrpcPort: filerGrpcPort,
volumePort: volumePort,
volumeGrpcPort: volumeGrpcPort,
dockerAvailable: hasDocker(),
accessKey: "AKIAIOSFODNN7EXAMPLE",
secretKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
env := &TestEnvironment{
seaweedDir: seaweedDir,
weedBinary: weedBinary,
dataDir: dataDir,
bindIP: bindIP,
closers: []io.Closer{},
}
env.masterPort, env.masterGrpcPort = env.mustFreePortPair("Master")
env.volumePort, env.volumeGrpcPort = env.mustFreePortPair("Volume")
env.filerPort, env.filerGrpcPort = env.mustFreePortPair("Filer")
env.s3Port, env.s3GrpcPort = env.mustFreePortPair("S3")
env.icebergPort = env.mustFreePort("Iceberg")
env.dockerAvailable = hasDocker()
env.accessKey = "AKIAIOSFODNN7EXAMPLE"
env.secretKey = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
return env
}
func (env *TestEnvironment) mustFreePort(name string) int {
port, closer, err := getFreePort()
if err != nil {
panic(fmt.Sprintf("Failed to get free port for %s: %v", name, err))
}
env.closers = append(env.closers, closer)
return port
}
func (env *TestEnvironment) mustFreePortPair(name string) (int, int) {
httpPort, httpCloser, grpcPort, grpcCloser, err := findAvailablePortPair()
if err != nil {
panic(fmt.Sprintf("Failed to get free port pair for %s: %v", name, err))
}
env.closers = append(env.closers, httpCloser, grpcCloser)
return httpPort, grpcPort
}
func mustFreePort(t *testing.T, name string) int {
t.Helper()
port, closer, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for %s: %v", name, err)
}
closer.Close()
return port
}
func mustFreePortPair(t *testing.T, name string) (int, int) {
t.Helper()
httpPort, httpCloser, grpcPort, grpcCloser, err := findAvailablePortPair()
if err != nil {
t.Fatalf("Failed to get free port pair for %s: %v", name, err)
}
httpCloser.Close()
grpcCloser.Close()
return httpPort, grpcPort
}
func (env *TestEnvironment) StartSeaweedFS(t *testing.T) {
@@ -161,6 +198,12 @@ func (env *TestEnvironment) StartSeaweedFS(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
env.weedCancel = cancel
// Close all port listeners right before starting the weed process
for _, closer := range env.closers {
closer.Close()
}
env.closers = nil
cmd := exec.CommandContext(ctx, env.weedBinary, "mini",
"-master.port", fmt.Sprintf("%d", env.masterPort),
"-master.port.grpc", fmt.Sprintf("%d", env.masterGrpcPort),
@@ -471,47 +514,27 @@ func hasDocker() bool {
return cmd.Run() == nil
}
func mustFreePort(t *testing.T, name string) int {
t.Helper()
port, err := getFreePort()
func findAvailablePortPair() (int, io.Closer, int, io.Closer, error) {
httpPort, httpCloser, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for %s: %v", name, err)
return 0, nil, 0, nil, err
}
return port
grpcPort, grpcCloser, err := getFreePort()
if err != nil {
httpCloser.Close()
return 0, nil, 0, nil, err
}
return httpPort, httpCloser, grpcPort, grpcCloser, nil
}
func mustFreePortPair(t *testing.T, name string) (int, int) {
t.Helper()
httpPort, grpcPort, err := findAvailablePortPair()
if err != nil {
t.Fatalf("Failed to get free port pair for %s: %v", name, err)
}
return httpPort, grpcPort
}
func findAvailablePortPair() (int, int, error) {
httpPort, err := getFreePort()
if err != nil {
return 0, 0, err
}
grpcPort, err := getFreePort()
if err != nil {
return 0, 0, err
}
return httpPort, grpcPort, nil
}
func getFreePort() (int, error) {
func getFreePort() (int, io.Closer, error) {
listener, err := net.Listen("tcp", "0.0.0.0:0")
if err != nil {
return 0, err
return 0, nil, err
}
defer listener.Close()
addr := listener.Addr().(*net.TCPAddr)
return addr.Port, nil
return addr.Port, listener, nil
}
func randomString(length int) string {
@@ -12,33 +12,57 @@ import (
"time"
cryptorand "crypto/rand"
"sync"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"flag"
"github.com/seaweedfs/seaweedfs/weed/command"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
flag "github.com/seaweedfs/seaweedfs/weed/util/fla9"
)
var (
miniClusterMutex sync.Mutex
)
// TestMain starts a single default weed mini cluster for the whole package and
// tears it down after all tests have completed. Tests that require a different
// cluster configuration (e.g. TestS3TablesCreateBucketIAMPolicy) start their
// own cluster independently.
func TestMain(m *testing.M) {
flag.Parse()
if testing.Short() {
// Tests self-skip with t.Skip when -short is set; no cluster needed.
os.Exit(m.Run())
}
// Create a temporary T-less context so we can use t.TempDir-equivalent.
testDir, err := os.MkdirTemp("", "seaweed-s3tables-shared-*")
if err != nil {
fmt.Fprintf(os.Stderr, "SKIP: failed to create shared temp dir: %v\n", err)
os.Exit(0)
}
cluster, err := startMiniClusterInDir(testDir, nil)
if err != nil {
fmt.Fprintf(os.Stderr, "SKIP: failed to start shared weed mini cluster: %v\n", err)
os.RemoveAll(testDir)
os.Exit(0)
}
sharedCluster = cluster
code := m.Run()
sharedCluster.Stop()
os.RemoveAll(testDir)
os.Exit(code)
}
func TestS3TablesIntegration(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
// Create and start test cluster
cluster, err := startMiniCluster(t)
require.NoError(t, err)
defer cluster.Stop()
// Create S3 Tables client
client := NewS3TablesClient(cluster.s3Endpoint, testRegion, testAccessKey, testSecretKey)
// Re-use the shared cluster started by TestMain.
client := NewS3TablesClient(sharedCluster.s3Endpoint, testRegion, testAccessKey, testSecretKey)
// Run test suite
t.Run("TableBucketLifecycle", func(t *testing.T) {
@@ -70,6 +94,92 @@ func TestS3TablesIntegration(t *testing.T) {
})
}
func TestS3TablesCreateBucketIAMPolicy(t *testing.T) {
if testing.Short() {
t.Skip("Skipping IAM integration test in short mode")
}
t.Setenv("AWS_ACCESS_KEY_ID", "env-admin")
t.Setenv("AWS_SECRET_ACCESS_KEY", "env-secret")
allowedBucket := "tables-allowed"
deniedBucket := "tables-denied"
iamConfigDir := t.TempDir()
iamConfigPath := filepath.Join(iamConfigDir, "iam_config.json")
iamConfig := fmt.Sprintf(`{
"sts": {
"tokenDuration": "1h",
"maxSessionLength": "12h",
"issuer": "seaweedfs-sts",
"signingKey": "%s"
},
"accounts": [
{
"id": "%s",
"displayName": "tables-integration"
}
],
"identities": [
{
"name": "admin",
"credentials": [
{
"accessKey": "%s",
"secretKey": "%s"
}
],
"account": {
"id": "%s",
"displayName": "tables-integration"
},
"policyNames": ["S3TablesBucketPolicy"]
}
],
"policy": {
"defaultEffect": "Deny",
"storeType": "memory"
},
"policies": [
{
"name": "S3TablesBucketPolicy",
"document": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3tables:CreateTableBucket"],
"Resource": [
"arn:aws:s3tables:*:*:bucket/%s",
"arn:aws:s3:::%s"
]
}
]
}
}
]
}`, testIAMSigningKey, testAccountID, testAccessKey, testSecretKey, testAccountID, allowedBucket, allowedBucket)
require.NoError(t, os.WriteFile(iamConfigPath, []byte(iamConfig), 0644))
cluster, err := startMiniClusterWithExtraArgs(t, []string{
"-s3.config=" + iamConfigPath,
"-s3.iam.config=" + iamConfigPath,
})
require.NoError(t, err, "failed to start cluster with IAM config")
defer cluster.Stop()
client := NewS3TablesClient(cluster.s3Endpoint, testRegion, testAccessKey, testSecretKey)
_, err = client.CreateTableBucket(deniedBucket, nil)
require.Error(t, err, "denied bucket creation should fail")
assert.Contains(t, err.Error(), "AccessDenied")
allowedResp, err := client.CreateTableBucket(allowedBucket, nil)
require.NoError(t, err, "allowed bucket creation should succeed")
defer func() {
_ = client.DeleteTableBucket(allowedResp.ARN)
}()
}
func testTableBucketLifecycle(t *testing.T, client *S3TablesClient) {
bucketName := "test-bucket-" + randomString(8)
@@ -508,11 +618,12 @@ func findAvailablePorts(n int) ([]int, error) {
return ports, nil
}
// startMiniCluster starts a weed mini instance directly without exec
func startMiniCluster(t *testing.T) (*TestCluster, error) {
// Find available ports
// We need 8 unique ports: Master(2), Volume(2), Filer(2), S3(2)
ports, err := findAvailablePorts(8)
// startMiniClusterInDir starts a weed mini instance using testDir as the data
// directory. It does not require a *testing.T so it can be called from TestMain.
// extraArgs are appended to the default mini command flags.
func startMiniClusterInDir(testDir string, extraArgs []string) (*TestCluster, error) {
// We need 10 unique ports: Master(2), Volume(2), Filer(2), S3(2), Admin(2)
ports, err := findAvailablePorts(10)
if err != nil {
return nil, fmt.Errorf("failed to find available ports: %v", err)
}
@@ -525,8 +636,8 @@ func startMiniCluster(t *testing.T) (*TestCluster, error) {
filerGrpcPort := ports[5]
s3Port := ports[6]
s3GrpcPort := ports[7]
// Create temporary directory for test data
testDir := t.TempDir()
adminPort := ports[8]
adminGrpcPort := ports[9]
// Ensure no configuration file from previous runs
configFile := filepath.Join(testDir, "mini.options")
@@ -537,7 +648,6 @@ func startMiniCluster(t *testing.T) (*TestCluster, error) {
s3Endpoint := fmt.Sprintf("http://127.0.0.1:%d", s3Port)
cluster := &TestCluster{
t: t,
dataDir: testDir,
ctx: ctx,
cancel: cancel,
@@ -550,22 +660,25 @@ func startMiniCluster(t *testing.T) (*TestCluster, error) {
// Create empty security.toml to disable JWT authentication in tests
securityToml := filepath.Join(testDir, "security.toml")
err = os.WriteFile(securityToml, []byte("# Empty security config for testing\n"), 0644)
if err != nil {
if err = os.WriteFile(securityToml, []byte("# Empty security config for testing\n"), 0644); err != nil {
cancel()
return nil, fmt.Errorf("failed to create security.toml: %v", err)
}
// Ensure AWS credentials are set (don't use t.Setenv here — we are in TestMain).
if os.Getenv("AWS_ACCESS_KEY_ID") == "" {
os.Setenv("AWS_ACCESS_KEY_ID", "admin")
}
if os.Getenv("AWS_SECRET_ACCESS_KEY") == "" {
os.Setenv("AWS_SECRET_ACCESS_KEY", "admin")
}
// Start weed mini in a goroutine by calling the command directly
cluster.wg.Add(1)
go func() {
defer cluster.wg.Done()
// Protect global state mutation with a mutex
miniClusterMutex.Lock()
defer miniClusterMutex.Unlock()
// Save current directory and args
// Save current directory and args, restore on exit.
oldDir, _ := os.Getwd()
oldArgs := os.Args
defer func() {
@@ -573,21 +686,24 @@ func startMiniCluster(t *testing.T) (*TestCluster, error) {
os.Args = oldArgs
}()
// Change to test directory so mini picks up security.toml
// Change to test directory so mini picks up security.toml.
os.Chdir(testDir)
// Configure args for mini command
os.Args = []string{
"weed",
baseArgs := []string{
"-dir=" + testDir,
"-master.dir=" + testDir,
"-master.port=" + strconv.Itoa(masterPort),
"-master.port.grpc=" + strconv.Itoa(masterGrpcPort),
"-volume.port=" + strconv.Itoa(volumePort),
"-volume.port.grpc=" + strconv.Itoa(volumeGrpcPort),
"-volume.port.public=" + strconv.Itoa(volumePort),
"-volume.publicUrl=127.0.0.1:" + strconv.Itoa(volumePort),
"-filer.port=" + strconv.Itoa(filerPort),
"-filer.port.grpc=" + strconv.Itoa(filerGrpcPort),
"-s3.port=" + strconv.Itoa(s3Port),
"-s3.port.grpc=" + strconv.Itoa(s3GrpcPort),
"-admin.port=" + strconv.Itoa(adminPort),
"-admin.port.grpc=" + strconv.Itoa(adminGrpcPort),
"-webdav.port=0", // Disable WebDAV
"-admin.ui=false", // Disable admin UI
"-master.volumeSizeLimitMB=32", // Small volumes for testing
@@ -595,6 +711,10 @@ func startMiniCluster(t *testing.T) (*TestCluster, error) {
"-master.peers=none", // Faster startup
"-s3.iam.readOnly=false", // Enable IAM write operations for tests
}
if len(extraArgs) > 0 {
baseArgs = append(baseArgs, extraArgs...)
}
os.Args = append([]string{"weed"}, baseArgs...)
// Suppress most logging during tests
glog.MaxSize = 1024 * 1024
@@ -613,14 +733,34 @@ func startMiniCluster(t *testing.T) (*TestCluster, error) {
}()
// Wait for S3 service to be ready
err = waitForS3Ready(cluster.s3Endpoint, 30*time.Second)
if err != nil {
if err = waitForS3Ready(cluster.s3Endpoint, 30*time.Second); err != nil {
cancel()
return nil, fmt.Errorf("S3 service failed to start: %v", err)
}
cluster.isRunning = true
return cluster, nil
}
// startMiniClusterWithExtraArgs starts a weed mini instance for a single test.
// It uses t.TempDir() for data isolation and t.Setenv for credential scoping.
func startMiniClusterWithExtraArgs(t *testing.T, extraArgs []string) (*TestCluster, error) {
t.Helper()
testDir := t.TempDir()
// Scope credentials to the test so they are restored after test completion.
if os.Getenv("AWS_ACCESS_KEY_ID") == "" {
t.Setenv("AWS_ACCESS_KEY_ID", "admin")
}
if os.Getenv("AWS_SECRET_ACCESS_KEY") == "" {
t.Setenv("AWS_SECRET_ACCESS_KEY", "admin")
}
cluster, err := startMiniClusterInDir(testDir, extraArgs)
if err != nil {
return nil, err
}
cluster.t = t
t.Logf("Test cluster started successfully at %s", cluster.s3Endpoint)
return cluster, nil
}
@@ -646,22 +786,15 @@ func (c *TestCluster) Stop() {
case <-done:
// Goroutine finished
case <-timer.C:
// Timeout - goroutine doesn't respond to context cancel
// This may indicate the mini cluster didn't shut down cleanly
c.t.Log("Warning: Test cluster shutdown timed out after 2 seconds")
}
// Reset the global cmdMini flags to prevent state leakage to other tests
for _, cmd := range command.Commands {
if cmd.Name() == "mini" {
// Reset flags to defaults
cmd.Flag.VisitAll(func(f *flag.Flag) {
// Reset to default value
f.Value.Set(f.DefValue)
})
break
// Timeout - goroutine doesn't respond to context cancel.
// This may indicate the mini cluster didn't shut down cleanly.
if c.t != nil {
c.t.Log("Warning: Test cluster shutdown timed out after 2 seconds")
} else {
fmt.Println("Warning: Test cluster shutdown timed out after 2 seconds")
}
}
}
// waitForS3Ready waits for the S3 service to be ready
+10 -4
View File
@@ -8,6 +8,11 @@ import (
"time"
)
// sharedCluster is the single default TestCluster shared across all tests
// that do not require a specialised cluster configuration.
// It is initialised by TestMain and must not be modified by individual tests.
var sharedCluster *TestCluster
// TestCluster manages the weed mini instance for integration testing
type TestCluster struct {
t *testing.T
@@ -46,8 +51,9 @@ func NewS3TablesClient(endpoint, region, accessKey, secretKey string) *S3TablesC
// Test configuration constants
const (
testRegion = "us-west-2"
testAccessKey = "admin"
testSecretKey = "admin"
testAccountID = "111122223333"
testRegion = "us-west-2"
testAccessKey = "admin"
testSecretKey = "admin"
testAccountID = "111122223333"
testIAMSigningKey = "dGVzdC1zaWduaW5nLWtleS1mb3Itc3RzLWludGVncmF0aW9uLXRlc3Rz"
)
-205
View File
@@ -3,12 +3,6 @@
## Goal
Create a Go integration test suite under `test/volume_server` that validates **drop-in behavior parity** for the Volume Server HTTP and gRPC APIs, so a Rust rewrite can be verified against the current Go behavior.
## Current Program Focus (2026-02-16)
- Primary execution focus has shifted to implementing native Rust volume-server parity.
- This integration suite is now the parity gate for Rust implementation work.
- New tests should be added only when native Rust implementation reveals uncovered Go behavior that is not yet captured.
- Rust implementation roadmap lives in `rust/volume_server/DEV_PLAN.md`.
## Hard Requirements
- Tests live under `test/volume_server`.
- Tests are written in Go.
@@ -1133,202 +1127,3 @@ Update this section during implementation:
- Profiles covered: P1.
- Gaps introduced/remaining: `499` cancellation and transport-interruption branches remain pending.
- Commit: `d1e5f390a`
- Date: 2026-02-15
- Change: Added Rust volume-server migration bootstrap wiring.
- APIs covered: Harness now supports Go master + selectable volume binary (`VOLUME_SERVER_IMPL=rust` or `VOLUME_SERVER_BINARY`), with Rust-mode smoke coverage for representative HTTP/gRPC integration tests.
- Profiles covered: P1 smoke and default Go matrix unchanged.
- Gaps introduced/remaining: native Rust handler implementations are tracked in `rust/volume_server/DEV_PLAN.md`; current Rust mode is a compatibility launcher phase.
- Commit: `7beab85c2`, `880c2e1da`, `63d08e8a9`, `d402573ea`
- Date: 2026-02-15
- Change: Validated Rust-mode compatibility path with full integration suites.
- APIs covered: full `/test/volume_server/http` and `/test/volume_server/grpc` suites executed successfully with `VOLUME_SERVER_IMPL=rust`.
- Profiles covered: existing integration matrix in Rust launcher mode.
- Gaps introduced/remaining: native Rust endpoint and storage/RPC implementation remains pending; current mode delegates execution to Go volume server.
- Commit: `6ce4d7ede`
- Date: 2026-02-15
- Change: Added Rust proxy-supervision launcher mode and validated all volume-server integration suites in proxy mode.
- APIs covered: full `/test/volume_server/http` and `/test/volume_server/grpc` suites executed successfully with `VOLUME_SERVER_IMPL=rust` and `VOLUME_SERVER_RUST_MODE=proxy`.
- Profiles covered: existing integration matrix in Rust proxy launcher mode.
- Gaps introduced/remaining: native Rust endpoint/storage/RPC logic remains pending; current proxy mode still delegates backend handlers to Go.
- Commit: `a7f50d23b`
- Date: 2026-02-15
- Change: Expanded GitHub CI Rust smoke job to run both launcher modes (`exec` and `proxy`).
- APIs covered: representative HTTP/gRPC smoke flows now validated in CI for both Rust launcher execution paths.
- Profiles covered: CI smoke profile for P1 representative calls.
- Gaps introduced/remaining: CI still runs Rust mode as smoke scope; full-suite Rust-mode CI remains optional due runtime cost.
- Commit: `548b3d9a3`
- Date: 2026-02-15
- Change: Added JWT UI override integration coverage with explicit `access.ui=true` profile support.
- APIs covered: `/ui/index.html` behavior under JWT profile now verifies both default auth-gated path (`401`) and explicit UI-enable override path (`200` with rendered UI).
- Profiles covered: P3 and P3+`access.ui=true`.
- Gaps introduced/remaining: UI exposure behavior under JWT profiles is now covered for both secured and override branches.
- Commit: `de974c05d`
- Date: 2026-02-15
- Change: Added oversized upload integration coverage using explicit `-fileSizeLimitMB` profile configuration.
- APIs covered: HTTP write path now verifies payloads larger than configured file-size limit are rejected (`400`) with limit-related error context.
- Profiles covered: P1-derived profile with `FileSizeLimitMB=1`.
- Gaps introduced/remaining: file-size limit rejection branch is now covered; replicate-write failure and cancellation (`499`) branches remain pending.
- Commit: `4d61cbdee`
- Date: 2026-02-15
- Change: Added gRPC EC-only metadata-read unsupported-path coverage.
- APIs covered: `ReadNeedleMeta` now verifies the explicit unsupported branch when a volume is unmounted and only EC shards are mounted.
- Profiles covered: P1 with EC shard generation/mount lifecycle setup.
- Gaps introduced/remaining: low-level read metadata fault-injection and transport interruption branches remain pending.
- Commit: `37bf9b5eb`
- Date: 2026-02-15
- Change: Added deterministic replicated-write failure coverage for unmet replication requirements.
- APIs covered: HTTP write path now verifies replication error handling (`500`) and no-local-commit outcome (`404` on follow-up read) when volume replication is configured (`001`) but replica set cannot be satisfied.
- Profiles covered: P1-derived single-node profile with per-volume replication override.
- Gaps introduced/remaining: replicated-write failure branch is now covered; cancellation (`499`) and deeper transport fault-injection branches remain pending.
- Commit: `4835d3443`
- Date: 2026-02-15
- Change: Added EC-backed `BatchDelete` positive-path integration coverage.
- APIs covered: `BatchDelete` now validates successful EC-needle deletion (`202`) with post-delete `VolumeEcShardRead` deleted-marker verification (`IsDeleted=true`).
- Profiles covered: P1 with EC shard generate/mount lifecycle; includes test-local gRPC offset proxy for EC internal dial-path parity.
- Gaps introduced/remaining: EC `BatchDelete` success branch is now covered; deeper distributed failure-injection permutations remain pending.
- Commit: `1bb40b6bc`
- Date: 2026-02-15
- Change: Expanded HTTP conditional-header matrix for `HEAD` requests with combined header scenarios.
- APIs covered: `HEAD` now verifies `If-Modified-Since` precedence over mismatched `If-None-Match`, and `If-None-Match` precedence when IMS is stale but ETag matches.
- Profiles covered: P1.
- Gaps introduced/remaining: combined conditional-header precedence paths for `HEAD`/`GET` are now explicitly covered.
- Commit: `ed23e290f`
- Date: 2026-02-15
- Change: Added multi-volume success-stream coverage for `ReadAllNeedles`.
- APIs covered: `ReadAllNeedles` now verifies happy-path streaming across two existing volumes in one request, with per-volume payload validation.
- Profiles covered: P1.
- Gaps introduced/remaining: multi-volume stream boundary coverage is now present for successful reads; transport interruption branches remain pending.
- Commit: `ab95a6ef1`
- Date: 2026-02-15
- Change: Expanded split-port unsupported HTTP method matrix with `MKCOL` coverage.
- APIs covered: admin/public parity for `MKCOL` (`400` on admin, passthrough `200` on public) with post-call data-integrity verification.
- Profiles covered: P2.
- Gaps introduced/remaining: unsupported-method sampling now covers `PATCH`, `TRACE`, `PROPFIND`, `CONNECT`, and `MKCOL`; cancellation/transport fault paths remain the main unaddressed area.
- Commit: `2ab30900d`
- Date: 2026-02-15
- Change: Hardened framework port allocation to keep volume ports within `10000..55535`.
- APIs covered: test harness now guarantees valid `admin+10000` gRPC offset ranges, eliminating EC batch-delete flake scenarios that rely on offset dialing.
- Profiles covered: all profiles using shared framework port allocation.
- Gaps introduced/remaining: no new API gaps; transport cancellation/fault-injection branches remain the primary uncovered area.
- Commit: `90e82b15c`
- Date: 2026-02-15
- Change: Re-validated full Rust proxy-mode integration suite after latest HTTP/gRPC coverage additions and framework hardening.
- APIs covered: full `/test/volume_server/http` and `/test/volume_server/grpc` packages.
- Profiles covered: existing matrix in Rust proxy launcher mode.
- Gaps introduced/remaining: native Rust endpoint/storage/RPC implementation remains pending (current Rust mode still delegates to Go backend handlers).
- Commit: `6bb9d8bac`
- Date: 2026-02-15
- Change: Added tail sender stream-cancellation interruption coverage.
- APIs covered: `VolumeTailSender` now verifies client-side context cancellation behavior (`codes.Canceled`) after stream start/heartbeat.
- Profiles covered: P1.
- Gaps introduced/remaining: tail transport interruption coverage is now partially closed; receiver-side interruption and deeper network fault injection remain pending.
- Commit: `27a80f760`
- Date: 2026-02-15
- Change: Added receiver-side tail stream interruption coverage for unavailable source node.
- APIs covered: `VolumeTailReceiver` now verifies source connect/dial failure branch when `SourceVolumeServer` is unreachable.
- Profiles covered: P1.
- Gaps introduced/remaining: both sender and receiver transport interruption branches are now covered at integration level; deeper injected network faults remain pending.
- Commit: `a5864c3eb`
- Date: 2026-02-15
- Change: Expanded query CSV matrix with explicit CSV-payload parity coverage.
- APIs covered: `Query` now verifies current CSV-input behavior (no streamed rows / immediate EOF) even when source needle content is valid CSV text.
- Profiles covered: P1.
- Gaps introduced/remaining: CSV parsing/selection implementation still absent in current server behavior; integration parity now locks the current no-output semantics for both JSON and CSV payload shapes.
- Commit: `a12dd5f8d`
- Date: 2026-02-15
- Change: Expanded ping unreachable-target matrix with volume-server target coverage.
- APIs covered: `Ping` now explicitly verifies unreachable-target error wrapping for `target_type=volumeServer`, alongside existing master/filer variants.
- Profiles covered: P1.
- Gaps introduced/remaining: ping target-type matrix is covered for success and unreachable branches across master/filer/volume-server.
- Commit: `b9fbb85af`
- Date: 2026-02-15
- Change: Expanded deleted-read HTTP matrix with `HEAD` parity coverage.
- APIs covered: `HEAD` with `readDeleted=true` now validates current behavior (`200`, empty body, payload-size `Content-Length`) after delete.
- Profiles covered: P1.
- Gaps introduced/remaining: deleted-read parity now covers both `GET` and `HEAD` semantics on local-volume path.
- Commit: `cc80ad364`
- Date: 2026-02-16
- Change: Shifted planning priority to native Rust implementation parity.
- APIs covered: no new API additions in this plan entry; integration suite remains the validation gate.
- Profiles covered: unchanged.
- Gaps introduced/remaining: primary remaining gap is native Rust handler/storage/RPC implementation replacing Go backend delegation.
- Commit: `14c863dbf`
- Date: 2026-02-16
- Change: Added Rust launcher `native` mode bootstrap, made it the default Rust launcher mode, and expanded CI Rust smoke matrix coverage to include `native`.
- APIs covered: full `/test/volume_server/http` and `/test/volume_server/grpc` packages re-validated in `VOLUME_SERVER_RUST_MODE=native`; default Rust launcher path (`VOLUME_SERVER_IMPL=rust`) smoke-validated for HTTP and gRPC control tests.
- Profiles covered: existing HTTP/gRPC matrix in native launcher mode; default-mode smoke checks on P1 control flows.
- Gaps introduced/remaining: native Rust API/storage/RPC handlers still pending; current `native` mode remains a delegation bootstrap while parity replacement proceeds.
- Commit: `70ddbee37`, `61befd10f`, `2e65966c0`
- Date: 2026-02-16
- Change: Implemented first native Rust HTTP control handlers in `native` mode for `/status` and `/healthz` (admin listener path), with proxy fallback retained for remaining endpoints.
- APIs covered: `/status` and `/healthz` now served by Rust-native code path in `VOLUME_SERVER_RUST_MODE=native`; full HTTP and gRPC integration suites re-validated.
- Profiles covered: full existing HTTP/gRPC matrix under native mode.
- Gaps introduced/remaining: gRPC control/data APIs and most HTTP data/static/auth paths remain delegated and require incremental native replacement.
- Commit: `7e6e0261a`
- Date: 2026-02-16
- Change: Implemented native Rust HTTP `OPTIONS` handling for admin/public listeners in `VOLUME_SERVER_RUST_MODE=native`.
- APIs covered: admin/public `OPTIONS` method allow-list and CORS origin handling now served from Rust-native path; full HTTP and gRPC packages re-validated.
- Profiles covered: P2 split-port CORS/OPTIONS profiles plus full existing matrix in native mode.
- Gaps introduced/remaining: static/UI/auth and data-path HTTP handlers, plus gRPC control/data/stream handlers, remain delegated and pending native replacement.
- Commit: `fbff2cb39`
- Date: 2026-02-16
- Change: Implemented broader native Rust HTTP surface in `native` mode:
- `/ui/index.html` auth-gating parity (JWT + access.ui handling)
- static asset paths (`/favicon.ico`, `/seaweedfsstatic/*`)
- public non-read method no-op parity (`200`) and admin unsupported-method rejection parity (`400`)
- APIs covered: UI/static/admin/public control-surface behavior now served by Rust-native path; full `/test/volume_server/http` and `/test/volume_server/grpc` packages re-validated.
- Profiles covered: P1/P2/P3 variants touched by UI/static/CORS/method behavior, plus full matrix in native mode.
- Gaps introduced/remaining: HTTP data-path handlers and all gRPC handlers remain delegated and still require native replacement.
- Commit: `23e4497b2`
- Date: 2026-02-16
- Change: Improved native Rust control-path parity for health/status and request-target parsing.
- APIs covered: native `/healthz` now mirrors backend service health transitions (`200`/`503`) and native route matching now handles absolute-form HTTP request targets before path dispatch.
- Profiles covered: full existing HTTP/gRPC integration matrix in native mode (`VOLUME_SERVER_IMPL=rust`, `VOLUME_SERVER_RUST_MODE=native`).
- Gaps introduced/remaining: core HTTP data handlers and all gRPC RPC handlers are still delegated and remain the primary native implementation gap.
- Commit: `d6ff6ed6d`
- Date: 2026-02-16
- Change: Added Rust-native malformed fid-route validation ahead of delegated data handlers.
- APIs covered: GET/HEAD/POST/PUT fid-shaped paths now return native `400` for invalid vid/fid tokens (including invalid-read and invalid-write path variants) while valid routes continue through delegated storage handlers.
- Profiles covered: full existing HTTP/gRPC integration matrix in native mode, plus targeted invalid-path parity runs.
- Gaps introduced/remaining: data-path success branches and all gRPC handler bodies remain delegated; native replacement work continues on those core execution paths.
- Commit: `1ce0174b2`
- Date: 2026-02-16
- Change: Added Rust-native write error prevalidation for delegated fid routes.
- APIs covered: native path now rejects malformed multipart boundary requests, `Content-MD5` writes, and over-limit `Content-Length` writes (using `-fileSizeLimitMB`) with `400` parity behavior before delegation.
- Profiles covered: P1 default and P1 with explicit file-size limit profile, plus full existing HTTP/gRPC native-mode matrix revalidation.
- Gaps introduced/remaining: positive write/read/delete data handlers and all gRPC handler bodies are still delegated; native implementation continues incrementally.
- Commit: `94cefd6f4`
- Date: 2026-02-16
- Change: Broadened Rust-native malformed-fid handling to slash-form read/write URL shapes with reserved-route exclusions.
- APIs covered: `/{vid}/{fid}` and `/{vid}/{fid}/{filename}` malformed cases now hit native `400` validation path, while `/status`, `/healthz`, `/ui/index.html`, `/stats/*`, and static asset routes remain excluded from fid parsing.
- Profiles covered: full existing HTTP/gRPC native-mode matrix, plus targeted admin/read-path malformed-slash validation run.
- Gaps introduced/remaining: successful data-path read/write/delete handlers and all gRPC method bodies remain delegated and are still the primary native implementation gap.
- Commit: `e66983e9b`
+1 -4
View File
@@ -1,10 +1,7 @@
.PHONY: test-volume-server test-volume-server-short test-volume-server-rust-smoke
.PHONY: test-volume-server test-volume-server-short
test-volume-server:
go test ./test/volume_server/... -v
test-volume-server-short:
go test ./test/volume_server/... -short -v
test-volume-server-rust-smoke:
VOLUME_SERVER_IMPL=rust go test ./test/volume_server/http ./test/volume_server/grpc -run 'TestAdminStatusAndHealthz|TestUploadReadRangeHeadDeleteRoundTrip|TestStateAndStatusRPCs|TestVolumeSyncStatusAndReadVolumeFileStatus' -v
+1 -5
View File
@@ -15,10 +15,6 @@ If a `weed` binary is not found, the harness will build one automatically.
## Optional environment variables
- `WEED_BINARY`: explicit path to the `weed` executable (disables auto-build).
- `VOLUME_SERVER_IMPL`: select volume-server implementation (`go` default, `rust` for Rust launcher mode).
- `VOLUME_SERVER_BINARY`: explicit path to the volume-server executable (overrides `VOLUME_SERVER_IMPL`).
- `VOLUME_SERVER_RUST_MODE`: Rust launcher mode (`native` default, `exec` for direct Go delegation, `proxy` for Rust front proxy + Go backend).
- `VOLUME_SERVER_RUST_REBUILD=1`: force rebuild of Rust volume-server binary in Rust mode.
- `VOLUME_SERVER_IT_KEEP_LOGS=1`: keep temporary test directories and process logs.
## Current scope (Phase 0)
@@ -28,4 +24,4 @@ If a `weed` binary is not found, the harness will build one automatically.
- Initial HTTP admin endpoint checks
- Initial gRPC state/status checks
More API coverage is tracked in `test/volume_server/DEV_PLAN.md`.
More API coverage is tracked in `/Users/chris/dev/seaweedfs2/test/volume_server/DEV_PLAN.md`.
+25 -136
View File
@@ -32,12 +32,11 @@ type Cluster struct {
testingTB testing.TB
profile matrix.Profile
weedBinary string
volumeBinary string
baseDir string
configDir string
logsDir string
keepLogs bool
weedBinary string
baseDir string
configDir string
logsDir string
keepLogs bool
masterPort int
masterGrpcPort int
@@ -55,9 +54,9 @@ type Cluster struct {
func StartSingleVolumeCluster(t testing.TB, profile matrix.Profile) *Cluster {
t.Helper()
weedBinary, volumeBinary, err := FindOrBuildServerBinaries()
weedBinary, err := FindOrBuildWeedBinary()
if err != nil {
t.Fatalf("resolve server binaries: %v", err)
t.Fatalf("resolve weed binary: %v", err)
}
baseDir, keepLogs, err := newWorkDir()
@@ -93,7 +92,6 @@ func StartSingleVolumeCluster(t testing.TB, profile matrix.Profile) *Cluster {
testingTB: t,
profile: profile,
weedBinary: weedBinary,
volumeBinary: volumeBinary,
baseDir: baseDir,
configDir: configDir,
logsDir: logsDir,
@@ -208,11 +206,8 @@ func (c *Cluster) startVolume(dataDir string) error {
if c.profile.InflightDownloadTimeout > 0 {
args = append(args, "-inflightDownloadDataTimeout="+c.profile.InflightDownloadTimeout.String())
}
if c.profile.FileSizeLimitMB > 0 {
args = append(args, "-fileSizeLimitMB="+strconv.Itoa(c.profile.FileSizeLimitMB))
}
c.volumeCmd = exec.Command(c.volumeBinary, args...)
c.volumeCmd = exec.Command(c.weedBinary, args...)
c.volumeCmd.Dir = c.baseDir
c.volumeCmd.Stdout = logFile
c.volumeCmd.Stderr = logFile
@@ -269,46 +264,18 @@ func stopProcess(cmd *exec.Cmd) {
}
func allocatePorts(count int) ([]int, error) {
const minPort = 10000
const maxPort = 55535
const host = "127.0.0.1"
rangeSize := maxPort - minPort + 1
listeners := make([]net.Listener, 0, count)
ports := make([]int, 0, count)
seen := make(map[int]struct{}, count)
closeAll := func() {
for _, ll := range listeners {
_ = ll.Close()
}
}
startOffset := int(time.Now().UnixNano() % int64(rangeSize))
for i := 0; i < count; i++ {
found := false
for offset := 0; offset < rangeSize; offset++ {
port := minPort + (startOffset+offset)%rangeSize
if _, exists := seen[port]; exists {
continue
l, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
for _, ll := range listeners {
_ = ll.Close()
}
l, err := net.Listen("tcp", net.JoinHostPort(host, strconv.Itoa(port)))
if err != nil {
continue
}
seen[port] = struct{}{}
listeners = append(listeners, l)
ports = append(ports, port)
found = true
startOffset = (startOffset + offset + 1) % rangeSize
break
}
if !found {
closeAll()
return nil, fmt.Errorf("unable to allocate %d ports within range [%d,%d]", count, minPort, maxPort)
return nil, err
}
listeners = append(listeners, l)
ports = append(ports, l.Addr().(*net.TCPAddr).Port)
}
for _, l := range listeners {
_ = l.Close()
@@ -359,13 +326,6 @@ func writeSecurityConfig(configDir string, profile matrix.Profile) error {
b.WriteString("\"\n")
b.WriteString("expires_after_seconds = 60\n")
}
if profile.AccessUI {
if b.Len() > 0 {
b.WriteString("\n")
}
b.WriteString("[access]\n")
b.WriteString("ui = true\n")
}
if b.Len() == 0 {
b.WriteString("# optional security config generated for integration tests\n")
}
@@ -381,12 +341,17 @@ func FindOrBuildWeedBinary() (string, error) {
return "", fmt.Errorf("WEED_BINARY is set but not executable: %s", fromEnv)
}
repoRoot, err := detectRepoRoot()
if err != nil {
return "", err
repoRoot := ""
if _, file, _, ok := runtime.Caller(0); ok {
repoRoot = filepath.Clean(filepath.Join(filepath.Dir(file), "..", "..", ".."))
candidate := filepath.Join(repoRoot, "weed", "weed")
if isExecutableFile(candidate) {
return candidate, nil
}
}
if candidate := filepath.Join(repoRoot, "weed", "weed"); isExecutableFile(candidate) {
return candidate, nil
if repoRoot == "" {
return "", errors.New("unable to detect repository root")
}
binDir := filepath.Join(os.TempDir(), "seaweedfs_volume_server_it_bin")
@@ -412,82 +377,6 @@ func FindOrBuildWeedBinary() (string, error) {
return binPath, nil
}
// FindOrBuildServerBinaries returns master and volume executables.
// Master always runs from the Go weed binary; volume can be switched via env.
func FindOrBuildServerBinaries() (masterBinary string, volumeBinary string, err error) {
masterBinary, err = FindOrBuildWeedBinary()
if err != nil {
return "", "", err
}
volumeBinary, err = FindOrBuildVolumeServerBinary(masterBinary)
if err != nil {
return "", "", err
}
return masterBinary, volumeBinary, nil
}
// FindOrBuildVolumeServerBinary resolves the executable used for volume-server processes.
//
// Behavior:
// - `VOLUME_SERVER_BINARY=/path/to/bin`: use explicit executable path.
// - `VOLUME_SERVER_IMPL=rust`: build/use Rust volume server launcher.
// - default: use the same Go `weed` binary.
func FindOrBuildVolumeServerBinary(defaultBinary string) (string, error) {
if fromEnv := os.Getenv("VOLUME_SERVER_BINARY"); fromEnv != "" {
if isExecutableFile(fromEnv) {
return fromEnv, nil
}
return "", fmt.Errorf("VOLUME_SERVER_BINARY is set but not executable: %s", fromEnv)
}
impl := strings.ToLower(strings.TrimSpace(os.Getenv("VOLUME_SERVER_IMPL")))
if impl == "" || impl == "go" {
return defaultBinary, nil
}
if impl != "rust" {
return "", fmt.Errorf("unsupported VOLUME_SERVER_IMPL %q (supported: go, rust)", impl)
}
repoRoot, err := detectRepoRoot()
if err != nil {
return "", err
}
return FindOrBuildRustVolumeServerBinary(repoRoot)
}
// FindOrBuildRustVolumeServerBinary builds the Rust volume server launcher when needed.
func FindOrBuildRustVolumeServerBinary(repoRoot string) (string, error) {
manifestPath := filepath.Join(repoRoot, "rust", "volume_server", "Cargo.toml")
if _, err := os.Stat(manifestPath); err != nil {
return "", fmt.Errorf("rust volume server manifest not found at %s: %w", manifestPath, err)
}
targetDir := filepath.Join(os.TempDir(), "seaweedfs_volume_server_it_rust_target")
binPath := filepath.Join(targetDir, "release", "weed-volume-rs")
if isExecutableFile(binPath) && os.Getenv("VOLUME_SERVER_RUST_REBUILD") != "1" {
return binPath, nil
}
cmd := exec.Command("cargo", "build", "--release", "--manifest-path", manifestPath, "--target-dir", targetDir)
var out bytes.Buffer
cmd.Stdout = &out
cmd.Stderr = &out
if err := cmd.Run(); err != nil {
return "", fmt.Errorf("build rust volume server binary: %w\n%s", err, out.String())
}
if !isExecutableFile(binPath) {
return "", fmt.Errorf("built rust volume server binary is not executable: %s", binPath)
}
return binPath, nil
}
func detectRepoRoot() (string, error) {
if _, file, _, ok := runtime.Caller(0); ok {
return filepath.Clean(filepath.Join(filepath.Dir(file), "..", "..", "..")), nil
}
return "", errors.New("unable to detect repository root")
}
func isExecutableFile(path string) bool {
info, err := os.Stat(path)
if err != nil || info.IsDir() {
+6 -284
View File
@@ -1,295 +1,17 @@
package framework
import (
"fmt"
"net"
"os"
"os/exec"
"path/filepath"
"strconv"
"sync"
"testing"
"github.com/seaweedfs/seaweedfs/test/volume_server/matrix"
)
type DualVolumeCluster struct {
testingTB testing.TB
profile matrix.Profile
weedBinary string
volumeBinary string
baseDir string
configDir string
logsDir string
keepLogs bool
masterPort int
masterGrpcPort int
volumePort0 int
volumeGrpcPort0 int
volumePubPort0 int
volumePort1 int
volumeGrpcPort1 int
volumePubPort1 int
masterCmd *exec.Cmd
volumeCmd0 *exec.Cmd
volumeCmd1 *exec.Cmd
cleanupOnce sync.Once
}
// DualVolumeCluster is deprecated. Use MultiVolumeCluster instead.
// For backward compatibility, it's a type alias for MultiVolumeCluster.
type DualVolumeCluster = MultiVolumeCluster
// StartDualVolumeCluster starts a cluster with 2 volume servers.
// Deprecated: Use StartMultiVolumeCluster(t, profile, 2) directly.
func StartDualVolumeCluster(t testing.TB, profile matrix.Profile) *DualVolumeCluster {
t.Helper()
weedBinary, volumeBinary, err := FindOrBuildServerBinaries()
if err != nil {
t.Fatalf("resolve server binaries: %v", err)
}
baseDir, keepLogs, err := newWorkDir()
if err != nil {
t.Fatalf("create temp test directory: %v", err)
}
configDir := filepath.Join(baseDir, "config")
logsDir := filepath.Join(baseDir, "logs")
masterDataDir := filepath.Join(baseDir, "master")
volumeDataDir0 := filepath.Join(baseDir, "volume0")
volumeDataDir1 := filepath.Join(baseDir, "volume1")
for _, dir := range []string{configDir, logsDir, masterDataDir, volumeDataDir0, volumeDataDir1} {
if mkErr := os.MkdirAll(dir, 0o755); mkErr != nil {
t.Fatalf("create %s: %v", dir, mkErr)
}
}
if err = writeSecurityConfig(configDir, profile); err != nil {
t.Fatalf("write security config: %v", err)
}
masterPort, masterGrpcPort, err := allocateMasterPortPair()
if err != nil {
t.Fatalf("allocate master port pair: %v", err)
}
ports, err := allocatePorts(6)
if err != nil {
t.Fatalf("allocate volume ports: %v", err)
}
c := &DualVolumeCluster{
testingTB: t,
profile: profile,
weedBinary: weedBinary,
volumeBinary: volumeBinary,
baseDir: baseDir,
configDir: configDir,
logsDir: logsDir,
keepLogs: keepLogs,
masterPort: masterPort,
masterGrpcPort: masterGrpcPort,
volumePort0: ports[0],
volumeGrpcPort0: ports[1],
volumePubPort0: ports[0],
volumePort1: ports[2],
volumeGrpcPort1: ports[3],
volumePubPort1: ports[2],
}
if profile.SplitPublicPort {
c.volumePubPort0 = ports[4]
c.volumePubPort1 = ports[5]
}
if err = c.startMaster(masterDataDir); err != nil {
c.Stop()
t.Fatalf("start master: %v", err)
}
if err = c.waitForHTTP(c.MasterURL() + "/dir/status"); err != nil {
masterLog := c.tailLog("master.log")
c.Stop()
t.Fatalf("wait for master readiness: %v\nmaster log tail:\n%s", err, masterLog)
}
if err = c.startVolume(0, volumeDataDir0); err != nil {
masterLog := c.tailLog("master.log")
c.Stop()
t.Fatalf("start first volume server: %v\nmaster log tail:\n%s", err, masterLog)
}
if err = c.waitForHTTP(c.VolumeAdminURL(0) + "/status"); err != nil {
volumeLog := c.tailLog("volume0.log")
c.Stop()
t.Fatalf("wait for first volume readiness: %v\nvolume log tail:\n%s", err, volumeLog)
}
if err = c.waitForTCP(c.VolumeGRPCAddress(0)); err != nil {
volumeLog := c.tailLog("volume0.log")
c.Stop()
t.Fatalf("wait for first volume grpc readiness: %v\nvolume log tail:\n%s", err, volumeLog)
}
if err = c.startVolume(1, volumeDataDir1); err != nil {
volumeLog := c.tailLog("volume0.log")
c.Stop()
t.Fatalf("start second volume server: %v\nfirst volume log tail:\n%s", err, volumeLog)
}
if err = c.waitForHTTP(c.VolumeAdminURL(1) + "/status"); err != nil {
volumeLog := c.tailLog("volume1.log")
c.Stop()
t.Fatalf("wait for second volume readiness: %v\nvolume log tail:\n%s", err, volumeLog)
}
if err = c.waitForTCP(c.VolumeGRPCAddress(1)); err != nil {
volumeLog := c.tailLog("volume1.log")
c.Stop()
t.Fatalf("wait for second volume grpc readiness: %v\nvolume log tail:\n%s", err, volumeLog)
}
t.Cleanup(func() {
c.Stop()
})
return c
}
func (c *DualVolumeCluster) Stop() {
if c == nil {
return
}
c.cleanupOnce.Do(func() {
stopProcess(c.volumeCmd1)
stopProcess(c.volumeCmd0)
stopProcess(c.masterCmd)
if !c.keepLogs && !c.testingTB.Failed() {
_ = os.RemoveAll(c.baseDir)
} else if c.baseDir != "" {
c.testingTB.Logf("volume server integration logs kept at %s", c.baseDir)
}
})
}
func (c *DualVolumeCluster) startMaster(dataDir string) error {
logFile, err := os.Create(filepath.Join(c.logsDir, "master.log"))
if err != nil {
return err
}
args := []string{
"-config_dir=" + c.configDir,
"master",
"-ip=127.0.0.1",
"-port=" + strconv.Itoa(c.masterPort),
"-port.grpc=" + strconv.Itoa(c.masterGrpcPort),
"-mdir=" + dataDir,
"-peers=none",
"-volumeSizeLimitMB=" + strconv.Itoa(testVolumeSizeLimitMB),
"-defaultReplication=000",
}
c.masterCmd = exec.Command(c.weedBinary, args...)
c.masterCmd.Dir = c.baseDir
c.masterCmd.Stdout = logFile
c.masterCmd.Stderr = logFile
return c.masterCmd.Start()
}
func (c *DualVolumeCluster) startVolume(index int, dataDir string) error {
logName := fmt.Sprintf("volume%d.log", index)
logFile, err := os.Create(filepath.Join(c.logsDir, logName))
if err != nil {
return err
}
volumePort := c.volumePort0
volumeGrpcPort := c.volumeGrpcPort0
volumePubPort := c.volumePubPort0
if index == 1 {
volumePort = c.volumePort1
volumeGrpcPort = c.volumeGrpcPort1
volumePubPort = c.volumePubPort1
}
args := []string{
"-config_dir=" + c.configDir,
"volume",
"-ip=127.0.0.1",
"-port=" + strconv.Itoa(volumePort),
"-port.grpc=" + strconv.Itoa(volumeGrpcPort),
"-port.public=" + strconv.Itoa(volumePubPort),
"-dir=" + dataDir,
"-max=16",
"-master=127.0.0.1:" + strconv.Itoa(c.masterPort),
"-readMode=" + c.profile.ReadMode,
"-concurrentUploadLimitMB=" + strconv.Itoa(c.profile.ConcurrentUploadLimitMB),
"-concurrentDownloadLimitMB=" + strconv.Itoa(c.profile.ConcurrentDownloadLimitMB),
}
if c.profile.InflightUploadTimeout > 0 {
args = append(args, "-inflightUploadDataTimeout="+c.profile.InflightUploadTimeout.String())
}
if c.profile.InflightDownloadTimeout > 0 {
args = append(args, "-inflightDownloadDataTimeout="+c.profile.InflightDownloadTimeout.String())
}
cmd := exec.Command(c.volumeBinary, args...)
cmd.Dir = c.baseDir
cmd.Stdout = logFile
cmd.Stderr = logFile
if err = cmd.Start(); err != nil {
return err
}
if index == 1 {
c.volumeCmd1 = cmd
} else {
c.volumeCmd0 = cmd
}
return nil
}
func (c *DualVolumeCluster) waitForHTTP(url string) error {
return (&Cluster{}).waitForHTTP(url)
}
func (c *DualVolumeCluster) waitForTCP(addr string) error {
return (&Cluster{}).waitForTCP(addr)
}
func (c *DualVolumeCluster) tailLog(logName string) string {
return (&Cluster{logsDir: c.logsDir}).tailLog(logName)
}
func (c *DualVolumeCluster) MasterAddress() string {
return net.JoinHostPort("127.0.0.1", strconv.Itoa(c.masterPort))
}
func (c *DualVolumeCluster) MasterURL() string {
return "http://" + c.MasterAddress()
}
func (c *DualVolumeCluster) VolumeAdminAddress(index int) string {
if index == 1 {
return net.JoinHostPort("127.0.0.1", strconv.Itoa(c.volumePort1))
}
return net.JoinHostPort("127.0.0.1", strconv.Itoa(c.volumePort0))
}
func (c *DualVolumeCluster) VolumePublicAddress(index int) string {
if index == 1 {
return net.JoinHostPort("127.0.0.1", strconv.Itoa(c.volumePubPort1))
}
return net.JoinHostPort("127.0.0.1", strconv.Itoa(c.volumePubPort0))
}
func (c *DualVolumeCluster) VolumeGRPCAddress(index int) string {
if index == 1 {
return net.JoinHostPort("127.0.0.1", strconv.Itoa(c.volumeGrpcPort1))
}
return net.JoinHostPort("127.0.0.1", strconv.Itoa(c.volumeGrpcPort0))
}
func (c *DualVolumeCluster) VolumeAdminURL(index int) string {
return "http://" + c.VolumeAdminAddress(index)
}
func (c *DualVolumeCluster) VolumePublicURL(index int) string {
return "http://" + c.VolumePublicAddress(index)
return StartMultiVolumeCluster(t, profile, 2)
}
@@ -0,0 +1,303 @@
package framework
import (
"fmt"
"net"
"os"
"os/exec"
"path/filepath"
"strconv"
"sync"
"testing"
"github.com/seaweedfs/seaweedfs/test/volume_server/matrix"
)
type MultiVolumeCluster struct {
testingTB testing.TB
profile matrix.Profile
weedBinary string
baseDir string
configDir string
logsDir string
keepLogs bool
volumeServerCount int
masterPort int
masterGrpcPort int
volumePorts []int
volumeGrpcPorts []int
volumePubPorts []int
masterCmd *exec.Cmd
volumeCmds []*exec.Cmd
cleanupOnce sync.Once
}
// StartMultiVolumeCluster starts a cluster with a specified number of volume servers
func StartMultiVolumeCluster(t testing.TB, profile matrix.Profile, serverCount int) *MultiVolumeCluster {
t.Helper()
if serverCount < 1 {
t.Fatalf("serverCount must be at least 1, got %d", serverCount)
}
weedBinary, err := FindOrBuildWeedBinary()
if err != nil {
t.Fatalf("resolve weed binary: %v", err)
}
baseDir, keepLogs, err := newWorkDir()
if err != nil {
t.Fatalf("create temp test directory: %v", err)
}
configDir := filepath.Join(baseDir, "config")
logsDir := filepath.Join(baseDir, "logs")
masterDataDir := filepath.Join(baseDir, "master")
// Create directories for master and all volume servers
dirs := []string{configDir, logsDir, masterDataDir}
for i := 0; i < serverCount; i++ {
dirs = append(dirs, filepath.Join(baseDir, fmt.Sprintf("volume%d", i)))
}
for _, dir := range dirs {
if mkErr := os.MkdirAll(dir, 0o755); mkErr != nil {
t.Fatalf("create %s: %v", dir, mkErr)
}
}
if err = writeSecurityConfig(configDir, profile); err != nil {
t.Fatalf("write security config: %v", err)
}
masterPort, masterGrpcPort, err := allocateMasterPortPair()
if err != nil {
t.Fatalf("allocate master port pair: %v", err)
}
// Allocate ports for all volume servers (3 ports per server: admin, grpc, public)
// If SplitPublicPort is true, we need an additional port per server
portsPerServer := 3
if profile.SplitPublicPort {
portsPerServer = 4
}
totalPorts := serverCount * portsPerServer
ports, err := allocatePorts(totalPorts)
if err != nil {
t.Fatalf("allocate volume ports: %v", err)
}
c := &MultiVolumeCluster{
testingTB: t,
profile: profile,
weedBinary: weedBinary,
baseDir: baseDir,
configDir: configDir,
logsDir: logsDir,
keepLogs: keepLogs,
volumeServerCount: serverCount,
masterPort: masterPort,
masterGrpcPort: masterGrpcPort,
volumePorts: make([]int, serverCount),
volumeGrpcPorts: make([]int, serverCount),
volumePubPorts: make([]int, serverCount),
volumeCmds: make([]*exec.Cmd, serverCount),
}
// Assign ports to each volume server
for i := 0; i < serverCount; i++ {
baseIdx := i * portsPerServer
c.volumePorts[i] = ports[baseIdx]
c.volumeGrpcPorts[i] = ports[baseIdx+1]
// Assign public port, using baseIdx+3 if SplitPublicPort, else baseIdx+2
pubPortIdx := baseIdx + 2
if profile.SplitPublicPort {
pubPortIdx = baseIdx + 3
}
c.volumePubPorts[i] = ports[pubPortIdx]
}
// Start master
if err = c.startMaster(masterDataDir); err != nil {
c.Stop()
t.Fatalf("start master: %v", err)
}
if err = c.waitForHTTP(c.MasterURL() + "/dir/status"); err != nil {
masterLog := c.tailLog("master.log")
c.Stop()
t.Fatalf("wait for master readiness: %v\nmaster log tail:\n%s", err, masterLog)
}
// Start all volume servers
for i := 0; i < serverCount; i++ {
volumeDataDir := filepath.Join(baseDir, fmt.Sprintf("volume%d", i))
if err = c.startVolume(i, volumeDataDir); err != nil {
// Log current server's log for debugging startup failures
volumeLog := fmt.Sprintf("volume%d.log", i)
c.Stop()
t.Fatalf("start volume server %d: %v\nvolume log tail:\n%s", i, err, c.tailLog(volumeLog))
}
if err = c.waitForHTTP(c.VolumeAdminURL(i) + "/status"); err != nil {
volumeLog := fmt.Sprintf("volume%d.log", i)
c.Stop()
t.Fatalf("wait for volume server %d readiness: %v\nvolume log tail:\n%s", i, err, c.tailLog(volumeLog))
}
if err = c.waitForTCP(c.VolumeGRPCAddress(i)); err != nil {
volumeLog := fmt.Sprintf("volume%d.log", i)
c.Stop()
t.Fatalf("wait for volume server %d grpc readiness: %v\nvolume log tail:\n%s", i, err, c.tailLog(volumeLog))
}
}
t.Cleanup(func() {
c.Stop()
})
return c
}
// StartTripleVolumeCluster is a convenience wrapper that starts a cluster with 3 volume servers
func StartTripleVolumeCluster(t testing.TB, profile matrix.Profile) *MultiVolumeCluster {
return StartMultiVolumeCluster(t, profile, 3)
}
func (c *MultiVolumeCluster) Stop() {
if c == nil {
return
}
c.cleanupOnce.Do(func() {
// Stop volume servers in reverse order
for i := len(c.volumeCmds) - 1; i >= 0; i-- {
stopProcess(c.volumeCmds[i])
}
stopProcess(c.masterCmd)
if !c.keepLogs && !c.testingTB.Failed() {
_ = os.RemoveAll(c.baseDir)
} else if c.baseDir != "" {
c.testingTB.Logf("volume server integration logs kept at %s", c.baseDir)
}
})
}
func (c *MultiVolumeCluster) startMaster(dataDir string) error {
logFile, err := os.Create(filepath.Join(c.logsDir, "master.log"))
if err != nil {
return err
}
args := []string{
"-config_dir=" + c.configDir,
"master",
"-ip=127.0.0.1",
"-port=" + strconv.Itoa(c.masterPort),
"-port.grpc=" + strconv.Itoa(c.masterGrpcPort),
"-mdir=" + dataDir,
"-peers=none",
"-volumeSizeLimitMB=" + strconv.Itoa(testVolumeSizeLimitMB),
"-defaultReplication=000",
}
c.masterCmd = exec.Command(c.weedBinary, args...)
c.masterCmd.Dir = c.baseDir
c.masterCmd.Stdout = logFile
c.masterCmd.Stderr = logFile
return c.masterCmd.Start()
}
func (c *MultiVolumeCluster) startVolume(index int, dataDir string) error {
logName := fmt.Sprintf("volume%d.log", index)
logFile, err := os.Create(filepath.Join(c.logsDir, logName))
if err != nil {
return err
}
args := []string{
"-config_dir=" + c.configDir,
"volume",
"-ip=127.0.0.1",
"-port=" + strconv.Itoa(c.volumePorts[index]),
"-port.grpc=" + strconv.Itoa(c.volumeGrpcPorts[index]),
"-port.public=" + strconv.Itoa(c.volumePubPorts[index]),
"-dir=" + dataDir,
"-max=16",
"-master=127.0.0.1:" + strconv.Itoa(c.masterPort),
"-readMode=" + c.profile.ReadMode,
"-concurrentUploadLimitMB=" + strconv.Itoa(c.profile.ConcurrentUploadLimitMB),
"-concurrentDownloadLimitMB=" + strconv.Itoa(c.profile.ConcurrentDownloadLimitMB),
}
if c.profile.InflightUploadTimeout > 0 {
args = append(args, "-inflightUploadDataTimeout="+c.profile.InflightUploadTimeout.String())
}
if c.profile.InflightDownloadTimeout > 0 {
args = append(args, "-inflightDownloadDataTimeout="+c.profile.InflightDownloadTimeout.String())
}
cmd := exec.Command(c.weedBinary, args...)
cmd.Dir = c.baseDir
cmd.Stdout = logFile
cmd.Stderr = logFile
if err = cmd.Start(); err != nil {
return err
}
c.volumeCmds[index] = cmd
return nil
}
func (c *MultiVolumeCluster) waitForHTTP(url string) error {
return (&Cluster{}).waitForHTTP(url)
}
func (c *MultiVolumeCluster) waitForTCP(addr string) error {
return (&Cluster{}).waitForTCP(addr)
}
func (c *MultiVolumeCluster) tailLog(logName string) string {
return (&Cluster{logsDir: c.logsDir}).tailLog(logName)
}
func (c *MultiVolumeCluster) MasterAddress() string {
return net.JoinHostPort("127.0.0.1", strconv.Itoa(c.masterPort))
}
func (c *MultiVolumeCluster) MasterURL() string {
return "http://" + c.MasterAddress()
}
func (c *MultiVolumeCluster) VolumeAdminAddress(index int) string {
if index < 0 || index >= len(c.volumePorts) {
return ""
}
return net.JoinHostPort("127.0.0.1", strconv.Itoa(c.volumePorts[index]))
}
func (c *MultiVolumeCluster) VolumePublicAddress(index int) string {
if index < 0 || index >= len(c.volumePubPorts) {
return ""
}
return net.JoinHostPort("127.0.0.1", strconv.Itoa(c.volumePubPorts[index]))
}
func (c *MultiVolumeCluster) VolumeGRPCAddress(index int) string {
if index < 0 || index >= len(c.volumeGrpcPorts) {
return ""
}
return net.JoinHostPort("127.0.0.1", strconv.Itoa(c.volumeGrpcPorts[index]))
}
func (c *MultiVolumeCluster) VolumeAdminURL(index int) string {
return "http://" + c.VolumeAdminAddress(index)
}
func (c *MultiVolumeCluster) VolumePublicURL(index int) string {
return "http://" + c.VolumePublicAddress(index)
}
func (c *MultiVolumeCluster) BaseDir() string {
return c.baseDir
}
+2 -10
View File
@@ -14,26 +14,18 @@ import (
func AllocateVolume(t testing.TB, client volume_server_pb.VolumeServerClient, volumeID uint32, collection string) {
t.Helper()
AllocateVolumeWithReplication(t, client, volumeID, collection, "000")
}
func AllocateVolumeWithReplication(t testing.TB, client volume_server_pb.VolumeServerClient, volumeID uint32, collection, replication string) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if replication == "" {
replication = "000"
}
_, err := client.AllocateVolume(ctx, &volume_server_pb.AllocateVolumeRequest{
VolumeId: volumeID,
Collection: collection,
Replication: replication,
Replication: "000",
Version: uint32(needle.GetCurrentVersion()),
})
if err != nil {
t.Fatalf("allocate volume %d (replication=%s): %v", volumeID, replication, err)
t.Fatalf("allocate volume %d: %v", volumeID, err)
}
}
@@ -388,17 +388,6 @@ func TestPingUnknownAndUnreachableTargetPaths(t *testing.T) {
if !strings.Contains(err.Error(), "ping filer") {
t.Fatalf("Ping filer unreachable error mismatch: %v", err)
}
_, err = grpcClient.Ping(ctx, &volume_server_pb.PingRequest{
TargetType: cluster.VolumeServerType,
Target: "127.0.0.1:1.1",
})
if err == nil {
t.Fatalf("Ping volume server target should fail when target is unreachable")
}
if !strings.Contains(err.Error(), "ping volumeServer") {
t.Fatalf("Ping volume server unreachable error mismatch: %v", err)
}
}
func TestPingMasterTargetSuccess(t *testing.T) {
@@ -3,10 +3,7 @@ package volume_server_grpc_test
import (
"bytes"
"context"
"io"
"net"
"net/http"
"strconv"
"strings"
"testing"
"time"
@@ -265,141 +262,3 @@ func TestBatchDeleteRejectsChunkManifestNeedles(t *testing.T) {
t.Fatalf("chunk manifest should not be deleted by BatchDelete reject path, got %d", readResp.StatusCode)
}
}
func TestBatchDeleteEcNeedleSuccess(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
cluster := framework.StartSingleVolumeCluster(t, matrix.P1())
conn, client := framework.DialVolumeServer(t, cluster.VolumeGRPCAddress())
defer conn.Close()
if stopProxy := maybeStartGrpcOffsetProxy(t, cluster.VolumeAdminAddress(), cluster.VolumeGRPCAddress()); stopProxy != nil {
t.Cleanup(stopProxy)
}
const volumeID = uint32(34)
const needleID = uint64(930001)
const cookie = uint32(0x6677BBCC)
framework.AllocateVolume(t, client, volumeID, "")
httpClient := framework.NewHTTPClient()
fid := framework.NewFileID(volumeID, needleID, cookie)
uploadResp := framework.UploadBytes(t, httpClient, cluster.VolumeAdminURL(), fid, []byte("batch-delete-ec-success"))
_ = framework.ReadAllAndClose(t, uploadResp)
if uploadResp.StatusCode != http.StatusCreated {
t.Fatalf("upload expected 201, got %d", uploadResp.StatusCode)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
_, err := client.VolumeEcShardsGenerate(ctx, &volume_server_pb.VolumeEcShardsGenerateRequest{
VolumeId: volumeID,
Collection: "",
})
if err != nil {
t.Fatalf("VolumeEcShardsGenerate failed: %v", err)
}
_, err = client.VolumeEcShardsMount(ctx, &volume_server_pb.VolumeEcShardsMountRequest{
VolumeId: volumeID,
Collection: "",
ShardIds: []uint32{0, 1, 2, 3, 4, 5, 6, 7, 8, 9},
})
if err != nil {
t.Fatalf("VolumeEcShardsMount failed: %v", err)
}
deleteResp, err := client.BatchDelete(ctx, &volume_server_pb.BatchDeleteRequest{FileIds: []string{fid}})
if err != nil {
t.Fatalf("BatchDelete EC needle should return response, got grpc error: %v", err)
}
if len(deleteResp.GetResults()) != 1 {
t.Fatalf("BatchDelete EC needle expected one result, got %d", len(deleteResp.GetResults()))
}
if deleteResp.GetResults()[0].GetStatus() != http.StatusAccepted {
t.Fatalf("BatchDelete EC needle expected status 202, got %d error=%q", deleteResp.GetResults()[0].GetStatus(), deleteResp.GetResults()[0].GetError())
}
if deleteResp.GetResults()[0].GetSize() == 0 {
t.Fatalf("BatchDelete EC needle expected non-zero deleted size")
}
deletedStream, err := client.VolumeEcShardRead(ctx, &volume_server_pb.VolumeEcShardReadRequest{
VolumeId: volumeID,
ShardId: 0,
FileKey: needleID,
Offset: 0,
Size: 1,
})
if err != nil {
t.Fatalf("VolumeEcShardRead deleted-check start failed: %v", err)
}
deletedMsg, err := deletedStream.Recv()
if err != nil {
t.Fatalf("VolumeEcShardRead deleted-check recv failed: %v", err)
}
if !deletedMsg.GetIsDeleted() {
t.Fatalf("VolumeEcShardRead expected IsDeleted=true after EC batch delete")
}
}
func maybeStartGrpcOffsetProxy(t testing.TB, adminAddr, actualGrpcAddr string) func() {
t.Helper()
host, portText, err := net.SplitHostPort(adminAddr)
if err != nil {
t.Fatalf("split admin address %q: %v", adminAddr, err)
}
port, err := strconv.Atoi(portText)
if err != nil {
t.Fatalf("parse admin port %q: %v", portText, err)
}
expectedGrpcAddr := net.JoinHostPort(host, strconv.Itoa(port+10000))
if expectedGrpcAddr == actualGrpcAddr {
return nil
}
listener, err := net.Listen("tcp", expectedGrpcAddr)
if err != nil {
t.Fatalf("listen grpc-offset proxy %s -> %s: %v", expectedGrpcAddr, actualGrpcAddr, err)
}
stopped := make(chan struct{})
go func() {
for {
conn, acceptErr := listener.Accept()
if acceptErr != nil {
select {
case <-stopped:
return
default:
continue
}
}
go proxyBidirectional(conn, actualGrpcAddr)
}
}()
return func() {
close(stopped)
_ = listener.Close()
}
}
func proxyBidirectional(inbound net.Conn, targetAddr string) {
outbound, err := net.Dial("tcp", targetAddr)
if err != nil {
_ = inbound.Close()
return
}
go func() {
_, _ = io.Copy(outbound, inbound)
_ = outbound.Close()
}()
_, _ = io.Copy(inbound, outbound)
_ = inbound.Close()
}
-64
View File
@@ -2,7 +2,6 @@ package volume_server_grpc_test
import (
"context"
"net/http"
"strings"
"testing"
"time"
@@ -145,66 +144,3 @@ func TestReadNeedleBlobAndMetaInvalidOffsets(t *testing.T) {
t.Fatalf("ReadNeedleMeta should fail for invalid offset")
}
}
func TestReadNeedleMetaUnsupportedForEcMountedVolume(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
clusterHarness := framework.StartSingleVolumeCluster(t, matrix.P1())
conn, grpcClient := framework.DialVolumeServer(t, clusterHarness.VolumeGRPCAddress())
defer conn.Close()
const volumeID = uint32(93)
const needleID = uint64(880002)
const cookie = uint32(0x11AA22BB)
framework.AllocateVolume(t, grpcClient, volumeID, "")
httpClient := framework.NewHTTPClient()
fid := framework.NewFileID(volumeID, needleID, cookie)
uploadResp := framework.UploadBytes(t, httpClient, clusterHarness.VolumeAdminURL(), fid, []byte("ec-meta-unsupported"))
_ = framework.ReadAllAndClose(t, uploadResp)
if uploadResp.StatusCode != http.StatusCreated {
t.Fatalf("upload expected 201, got %d", uploadResp.StatusCode)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
_, err := grpcClient.VolumeEcShardsGenerate(ctx, &volume_server_pb.VolumeEcShardsGenerateRequest{
VolumeId: volumeID,
Collection: "",
})
if err != nil {
t.Fatalf("VolumeEcShardsGenerate failed: %v", err)
}
_, err = grpcClient.VolumeEcShardsMount(ctx, &volume_server_pb.VolumeEcShardsMountRequest{
VolumeId: volumeID,
Collection: "",
ShardIds: []uint32{0, 1, 2, 3, 4, 5, 6, 7, 8, 9},
})
if err != nil {
t.Fatalf("VolumeEcShardsMount failed: %v", err)
}
_, err = grpcClient.VolumeUnmount(ctx, &volume_server_pb.VolumeUnmountRequest{
VolumeId: volumeID,
})
if err != nil {
t.Fatalf("VolumeUnmount failed: %v", err)
}
_, err = grpcClient.ReadNeedleMeta(ctx, &volume_server_pb.ReadNeedleMetaRequest{
VolumeId: volumeID,
NeedleId: needleID,
Offset: 0,
Size: 128,
})
if err == nil {
t.Fatalf("ReadNeedleMeta should fail when only EC shards are mounted")
}
if !strings.Contains(strings.ToLower(err.Error()), "ec shards is not supported") {
t.Fatalf("ReadNeedleMeta EC-only error mismatch: %v", err)
}
}
@@ -3,7 +3,6 @@ package volume_server_grpc_test
import (
"context"
"io"
"net/http"
"strings"
"testing"
"time"
@@ -231,71 +230,6 @@ func TestReadAllNeedlesExistingThenMissingVolumeAbortsStream(t *testing.T) {
}
}
func TestReadAllNeedlesStreamsAcrossMultipleVolumes(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
clusterHarness := framework.StartSingleVolumeCluster(t, matrix.P1())
conn, grpcClient := framework.DialVolumeServer(t, clusterHarness.VolumeGRPCAddress())
defer conn.Close()
const volumeIDA = uint32(86)
const volumeIDB = uint32(87)
const needleIDA = uint64(445561)
const needleIDB = uint64(445562)
framework.AllocateVolume(t, grpcClient, volumeIDA, "")
framework.AllocateVolume(t, grpcClient, volumeIDB, "")
client := framework.NewHTTPClient()
fidA := framework.NewFileID(volumeIDA, needleIDA, 0xAA11CC22)
fidB := framework.NewFileID(volumeIDB, needleIDB, 0xBB22DD33)
payloadA := "read-all-multi-volume-a"
payloadB := "read-all-multi-volume-b"
uploadA := framework.UploadBytes(t, client, clusterHarness.VolumeAdminURL(), fidA, []byte(payloadA))
_ = framework.ReadAllAndClose(t, uploadA)
if uploadA.StatusCode != http.StatusCreated {
t.Fatalf("upload A expected 201, got %d", uploadA.StatusCode)
}
uploadB := framework.UploadBytes(t, client, clusterHarness.VolumeAdminURL(), fidB, []byte(payloadB))
_ = framework.ReadAllAndClose(t, uploadB)
if uploadB.StatusCode != http.StatusCreated {
t.Fatalf("upload B expected 201, got %d", uploadB.StatusCode)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
stream, err := grpcClient.ReadAllNeedles(ctx, &volume_server_pb.ReadAllNeedlesRequest{
VolumeIds: []uint32{volumeIDA, volumeIDB},
})
if err != nil {
t.Fatalf("ReadAllNeedles start failed: %v", err)
}
seen := map[uint64]string{}
for {
msg, recvErr := stream.Recv()
if recvErr == io.EOF {
break
}
if recvErr != nil {
t.Fatalf("ReadAllNeedles recv failed: %v", recvErr)
}
if msg.GetNeedleId() == needleIDA || msg.GetNeedleId() == needleIDB {
seen[msg.GetNeedleId()] = string(msg.GetNeedleBlob())
}
}
if got := seen[needleIDA]; got != payloadA {
t.Fatalf("ReadAllNeedles missing/mismatched payload for volume A needle: got %q want %q", got, payloadA)
}
if got := seen[needleIDB]; got != payloadB {
t.Fatalf("ReadAllNeedles missing/mismatched payload for volume B needle: got %q want %q", got, payloadB)
}
}
func copyFileBytes(t testing.TB, grpcClient volume_server_pb.VolumeServerClient, req *volume_server_pb.CopyFileRequest) []byte {
t.Helper()
@@ -0,0 +1,285 @@
package volume_server_grpc_test
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/test/volume_server/framework"
"github.com/seaweedfs/seaweedfs/test/volume_server/matrix"
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
"github.com/seaweedfs/seaweedfs/weed/storage"
"github.com/seaweedfs/seaweedfs/weed/storage/backend"
"github.com/seaweedfs/seaweedfs/weed/storage/idx"
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
"github.com/seaweedfs/seaweedfs/weed/storage/types"
"github.com/seaweedfs/seaweedfs/weed/util"
)
func TestVolumeCopyReturnsPreciseLastAppendTimestamp(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
cluster := framework.StartDualVolumeCluster(t, matrix.P1())
sourceConn, sourceClient := framework.DialVolumeServer(t, cluster.VolumeGRPCAddress(0))
defer sourceConn.Close()
destConn, destClient := framework.DialVolumeServer(t, cluster.VolumeGRPCAddress(1))
defer destConn.Close()
const volumeID = uint32(999)
framework.AllocateVolume(t, sourceClient, volumeID, "")
httpClient := framework.NewHTTPClient()
fid := framework.NewFileID(volumeID, 1, 0x42)
payload := []byte("move-tail-timestamp-payload")
uploadResp := framework.UploadBytes(t, httpClient, cluster.VolumeAdminURL(0), fid, payload)
_ = framework.ReadAllAndClose(t, uploadResp)
if uploadResp.StatusCode != http.StatusCreated {
t.Fatalf("source upload failed: status %d", uploadResp.StatusCode)
}
sourceDir := filepath.Join(cluster.BaseDir(), "volume0")
datPath := storage.VolumeFileName(sourceDir, "", int(volumeID)) + ".dat"
futureTime := time.Now().Add(2 * time.Hour)
if err := os.Chtimes(datPath, futureTime, futureTime); err != nil {
t.Fatalf("set future dat timestamp: %v", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
sourceDataNode := cluster.VolumeAdminAddress(0) + "." + strings.Split(cluster.VolumeGRPCAddress(0), ":")[1]
copyStream, err := destClient.VolumeCopy(ctx, &volume_server_pb.VolumeCopyRequest{
VolumeId: volumeID,
SourceDataNode: sourceDataNode,
})
if err != nil {
t.Fatalf("VolumeCopy start failed: %v", err)
}
var lastAppendAtNs uint64
for {
resp, recvErr := copyStream.Recv()
if recvErr == io.EOF {
break
}
if recvErr != nil {
t.Fatalf("VolumeCopy recv failed: %v", recvErr)
}
if ts := resp.GetLastAppendAtNs(); ts > 0 {
lastAppendAtNs = ts
}
}
if lastAppendAtNs == 0 {
t.Fatalf("volume copy did not return a last append timestamp")
}
destDir := filepath.Join(cluster.BaseDir(), "volume1")
actualLastAppend := readLastAppendAtNs(t, destDir, volumeID)
if actualLastAppend == 0 {
t.Fatalf("failed to compute last append timestamp from destination files")
}
if lastAppendAtNs != actualLastAppend {
t.Fatalf("last append timestamp mismatch: got %d, actual %d", lastAppendAtNs, actualLastAppend)
}
}
func readLastAppendAtNs(t testing.TB, volumeDir string, volumeID uint32) uint64 {
t.Helper()
baseName := storage.VolumeFileName(volumeDir, "", int(volumeID))
idxPath := baseName + ".idx"
idxFile, err := os.Open(idxPath)
if err != nil {
t.Fatalf("open idx file %s: %v", idxPath, err)
}
defer idxFile.Close()
stat, err := idxFile.Stat()
if err != nil {
t.Fatalf("stat idx file %s: %v", idxPath, err)
}
if stat.Size() == 0 {
return 0
}
if stat.Size()%int64(types.NeedleMapEntrySize) != 0 {
t.Fatalf("unexpected idx file size %d", stat.Size())
}
buf := make([]byte, types.NeedleMapEntrySize)
if _, err := idxFile.ReadAt(buf, stat.Size()-int64(types.NeedleMapEntrySize)); err != nil {
t.Fatalf("read idx entry: %v", err)
}
_, offset, _ := idx.IdxFileEntry(buf)
if offset.IsZero() {
return 0
}
datPath := baseName + ".dat"
datFile, err := os.Open(datPath)
if err != nil {
t.Fatalf("open dat file %s: %v", datPath, err)
}
defer datFile.Close()
datBackend := backend.NewDiskFile(datFile)
n, _, _, err := needle.ReadNeedleHeader(datBackend, needle.GetCurrentVersion(), offset.ToActualOffset())
if err != nil {
t.Fatalf("read needle header: %v", err)
}
tailOffset := offset.ToActualOffset() + int64(types.NeedleHeaderSize) + int64(n.Size)
tail := make([]byte, needle.NeedleChecksumSize+types.TimestampSize)
readCount, readErr := datBackend.ReadAt(tail, tailOffset)
if readErr == io.EOF && readCount == len(tail) {
readErr = nil
}
if readErr != nil {
t.Fatalf("read needle tail: %v", readErr)
}
return util.BytesToUint64(tail[needle.NeedleChecksumSize : needle.NeedleChecksumSize+types.TimestampSize])
}
func TestVolumeMoveHandlesInFlightWrites(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
cluster := framework.StartDualVolumeCluster(t, matrix.P1())
sourceConn, sourceClient := framework.DialVolumeServer(t, cluster.VolumeGRPCAddress(0))
defer sourceConn.Close()
destConn, destClient := framework.DialVolumeServer(t, cluster.VolumeGRPCAddress(1))
defer destConn.Close()
const volumeID = uint32(988)
framework.AllocateVolume(t, sourceClient, volumeID, "")
httpClient := framework.NewHTTPClient()
fid := framework.NewFileID(volumeID, 7, 0xABCDEF01)
payload := []byte("volume-move-live-payload")
uploadResp := framework.UploadBytes(t, httpClient, cluster.VolumeAdminURL(0), fid, payload)
_ = framework.ReadAllAndClose(t, uploadResp)
if uploadResp.StatusCode != http.StatusCreated {
t.Fatalf("initial upload failed: %d", uploadResp.StatusCode)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
sourceDataNode := cluster.VolumeAdminAddress(0) + "." + strings.Split(cluster.VolumeGRPCAddress(0), ":")[1]
copyStream, err := destClient.VolumeCopy(ctx, &volume_server_pb.VolumeCopyRequest{
VolumeId: volumeID,
SourceDataNode: sourceDataNode,
})
if err != nil {
t.Fatalf("VolumeCopy start failed: %v", err)
}
var lastAppendAtNs uint64
for {
resp, recvErr := copyStream.Recv()
if recvErr == io.EOF {
break
}
if recvErr != nil {
t.Fatalf("VolumeCopy recv failed: %v", recvErr)
}
if ts := resp.GetLastAppendAtNs(); ts > 0 {
lastAppendAtNs = ts
}
}
if lastAppendAtNs == 0 {
t.Fatalf("volume copy did not return a last append timestamp")
}
type written struct {
fid string
data []byte
}
var writesMu sync.Mutex
var writes []written
var uploadErr error
writeCtx, writeCancel := context.WithCancel(context.Background())
var writerWG sync.WaitGroup
writerWG.Add(1)
go func() {
defer writerWG.Done()
client := framework.NewHTTPClient()
for i := 0; i < 12; i++ {
select {
case <-writeCtx.Done():
return
default:
}
livePayload := []byte("live-data-" + fmt.Sprintf("%02d", i))
liveFid := framework.NewFileID(volumeID, uint64(2000+i), 0xAAAA1111+uint32(i))
resp := framework.UploadBytes(t, client, cluster.VolumeAdminURL(0), liveFid, livePayload)
_ = framework.ReadAllAndClose(t, resp)
if resp.StatusCode != http.StatusCreated {
uploadErr = fmt.Errorf("live upload failed: %d", resp.StatusCode)
return
}
writesMu.Lock()
writes = append(writes, written{fid: liveFid, data: livePayload})
writesMu.Unlock()
time.Sleep(250 * time.Millisecond)
}
}()
tailCtx, tailCancel := context.WithTimeout(context.Background(), 60*time.Second)
defer tailCancel()
_, err = destClient.VolumeTailReceiver(tailCtx, &volume_server_pb.VolumeTailReceiverRequest{
VolumeId: volumeID,
SourceVolumeServer: sourceDataNode,
SinceNs: lastAppendAtNs,
IdleTimeoutSeconds: 3,
})
if err != nil {
writeCancel()
writerWG.Wait()
t.Fatalf("VolumeTailReceiver failed: %v", err)
}
writeCancel()
writerWG.Wait()
if uploadErr != nil {
t.Fatalf("upload goroutine error: %v", uploadErr)
}
writesMu.Lock()
sampleCount := len(writes)
if sampleCount == 0 {
writesMu.Unlock()
t.Fatal("no live writes captured")
}
sample := writes
if sampleCount > 3 {
sample = writes[sampleCount-3:]
}
writesMu.Unlock()
httpCheckClient := framework.NewHTTPClient()
for _, w := range sample {
resp := framework.ReadBytes(t, httpCheckClient, cluster.VolumeAdminURL(1), w.fid)
body := framework.ReadAllAndClose(t, resp)
if resp.StatusCode != http.StatusOK {
t.Fatalf("dest read %s status %d", w.fid, resp.StatusCode)
}
if !bytes.Equal(body, w.data) {
t.Fatalf("dest read body mismatch for %s: %q vs %q", w.fid, string(body), string(w.data))
}
}
}
@@ -280,54 +280,6 @@ func TestQueryJsonSuccessAndCsvNoOutput(t *testing.T) {
}
}
func TestQueryCsvInputWithCsvPayloadStillReturnsNoRows(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
clusterHarness := framework.StartSingleVolumeCluster(t, matrix.P1())
conn, grpcClient := framework.DialVolumeServer(t, clusterHarness.VolumeGRPCAddress())
defer conn.Close()
const volumeID = uint32(67)
const needleID = uint64(777004)
const cookie = uint32(0xDCDCADAD)
framework.AllocateVolume(t, grpcClient, volumeID, "")
csvPayload := []byte("name,score\nalice,12\nbob,3\n")
httpClient := framework.NewHTTPClient()
fid := framework.NewFileID(volumeID, needleID, cookie)
uploadResp := framework.UploadBytes(t, httpClient, clusterHarness.VolumeAdminURL(), fid, csvPayload)
_ = framework.ReadAllAndClose(t, uploadResp)
if uploadResp.StatusCode != 201 {
t.Fatalf("upload expected 201, got %d", uploadResp.StatusCode)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
csvStream, err := grpcClient.Query(ctx, &volume_server_pb.QueryRequest{
FromFileIds: []string{fid},
Selections: []string{"score"},
Filter: &volume_server_pb.QueryRequest_Filter{
Field: "score",
Operand: ">",
Value: "10",
},
InputSerialization: &volume_server_pb.QueryRequest_InputSerialization{
CsvInput: &volume_server_pb.QueryRequest_InputSerialization_CSVInput{},
},
})
if err != nil {
t.Fatalf("Query csv start failed: %v", err)
}
_, err = csvStream.Recv()
if err != io.EOF {
t.Fatalf("Query csv with csv payload expected EOF with no rows, got: %v", err)
}
}
func TestQueryJsonNoMatchReturnsEmptyStripe(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")

Some files were not shown because too many files have changed in this diff Show More