- Implement TestMain-based auto server management for all 13 S3 test suites
- Fix import paths to use full module names instead of relative imports
- Update Makefiles to use correct build targets and binary detection
- Enhance testutil with improved binary discovery and credential handling
- Fix test session creation to use explicit AWS credentials
- Validate infrastructure with successful test execution
- Update documentation with implementation details and usage instructions
All S3 test suites now support:
- make test-with-server: Auto-managed server lifecycle
- make test-external: Use existing server (for CI/CD)
- Consistent build and dependency checking
- Proper error handling and logging
- Change imports from absolute module path 'github.com/seaweedfs/seaweedfs/test/s3/testutil'
to relative path '../testutil'
- Fixes module resolution issues when running tests from individual directories
- All test directories now compile and load testutil correctly
- Enables tests to run with 'go test ./...' from any test subdirectory
- Add test-with-server target to acl, basic, delete, etag Makefiles
- Simplify test-with-server to use TestMain-based server auto-management
- Update tagging, cors, filer_group, retention, remote_cache, sse Makefiles
- Add TEST_PATTERN support for CI/CD parameterized testing
- Remove complex manual server management from test-with-server
- All targets now delegate to TestMain for consistent, simplified server lifecycle
- Backward compatible with existing make targets for manual testing
- Added s3_test_main.go with auto-managed weed mini server to:
- acl, basic, copying, cors, delete, etag, filer_group, iam
- remote_cache, retention, sse, tagging (12 additional test suites)
- All tests now support automatic server lifecycle management via TestMain
- Backward compatible with USE_EXTERNAL_SERVER=true for CI pipelines
- No changes required to existing test files
- All 13 S3 test suites now use consistent weed mini infrastructure
This enables:
- Tests to run independently with self-managed server
- Simplified CI/CD pipelines
- Reduced infrastructure complexity
- Consistent test infrastructure across all S3 tests
* s3: fix remote object not caching
* s3: address review comments for remote object caching
- Fix leading slash in object name by using strings.TrimPrefix
- Return cached entry from CacheRemoteObjectToLocalCluster to get updated local chunk locations
- Reuse existing helper function instead of inline gRPC call
* s3/filer: add singleflight deduplication for remote object caching
- Add singleflight.Group to FilerServer to deduplicate concurrent cache operations
- Wrap CacheRemoteObjectToLocalCluster with singleflight to ensure only one
caching operation runs per object when multiple clients request the same file
- Add early-return check for already-cached objects
- S3 API calls filer gRPC with timeout and graceful fallback on error
- Clear negative bucket cache when bucket is created via weed shell
- Add integration tests for remote cache with singleflight deduplication
This benefits all clients (S3, HTTP, Hadoop) accessing remote-mounted objects
by preventing redundant cache operations and improving concurrent access performance.
Fixes: https://github.com/seaweedfs/seaweedfs/discussions/7599
* fix: data race in concurrent remote object caching
- Add mutex to protect chunks slice from concurrent append
- Add mutex to protect fetchAndWriteErr from concurrent read/write
- Fix incorrect error check (was checking assignResult.Error instead of parseErr)
- Rename inner variable to avoid shadowing fetchAndWriteErr
* fix: address code review comments
- Remove duplicate remote caching block in GetObjectHandler, keep only singleflight version
- Add mutex protection for concurrent chunk slice and error access (data race fix)
- Use lazy initialization for S3 client in tests to avoid panic during package load
- Fix markdown linting: add language specifier to code fence, blank lines around tables
- Add 'all' target to Makefile as alias for test-with-server
- Remove unused 'util' import
* style: remove emojis from test files
* fix: add defensive checks and sort chunks by offset
- Add nil check and type assertion check for singleflight result
- Sort chunks by offset after concurrent fetching to maintain file order
* fix: improve test diagnostics and path normalization
- runWeedShell now returns error for better test diagnostics
- Add all targets to .PHONY in Makefile (logs-primary, logs-remote, health)
- Strip leading slash from normalizedObject to avoid double slashes in path
---------
Co-authored-by: chrislu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Chris Lu
