mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-30 12:47:00 +00:00
Chris LuandGitHub
2a97e08caa
* s3: enforce object lock when deleting a directory marker The key "dir/" is deleted the unversioned way, ahead of the branches that enforce Object Lock, so a principal with plain delete permission could remove a key the gateway was reporting as COMPLIANCE-retained -- retention set through PutObjectRetention is stored on the directory entry and served back by GetObjectRetention, only the delete ignored it. The same path also takes any key ending in "/" regardless of size, while a PUT only makes a marker of one up to 1KiB. A larger one is a genuine versioned object, and deleting it here dropped its whole history after the versioned delete of the same key had been refused. Enforce in the marker delete itself, so the single, versioned and multi-object delete paths are all covered. * s3: apply object lock headers on a directory marker PUT The trailing-slash branch runs before the versioning and Object Lock handling, so it accepted x-amz-object-lock-* headers and stored none of them: a bucket owner could believe a key was retained while nothing recorded it, and an invalid mode or a past retention date that a regular key rejects came back 200 here. Validate the headers the way the regular path does, store what they ask for beside the owner the same callback already sets, and refuse to replace a key that is already retained. * s3: check every version a marker delete would remove The marker delete clears any history under the key in one recursive removal, while the lock check ahead of it resolves the latest version only. A version retained under an unretained one was taken with the rest, so enforce against each version the removal covers. * test: pin the marker lock refusals to AccessDenied A bare require.Error passes on any failure, including one that has nothing to do with the lock. Assert the code, the key the batch delete reports, and that the marker survives each refusal. * s3: check the history entries a version list leaves out The version list skips an entry without a version id, while the removal takes it with the rest, so an entry an older build left unnamed escaped the check. Walk the history directly instead, and refuse when an unnamed entry is still under a retention or a legal hold of its own. * s3: let a governance bypass reach an unnamed history entry The unnamed branch refused every active retention, so a caller allowed to bypass governance could not clear one, which the named path lets through. Refuse a legal hold and compliance mode as before, and take the bypass into account for governance. * s3: keep the object lock decision in one place The unnamed history entry had to repeat the retention and legal hold rules inline because the enforcement helper only takes a key to look up. Split the part that judges an entry out of it and call that from both. * s3: guard a marker PUT on the entry it replaces The overwrite check resolved the key's latest version, but mkdir builds a fresh entry for the marker itself, dropping the lock metadata the old one carried. Once the key had a history, an unlocked version answered for a retained marker and a plain PUT replaced it. Judge the entry the write is about to replace instead; a versioned write of the same key still adds a version, which is its own to allow. * s3: guard a marker delete on the entry it removes The check ran against the key rather than the entry, so once the key had a history it answered with a version and the retention recorded on the marker itself went unseen. Judge the entry that is about to be removed, the same way the PUT side now does; the versions under it are still covered by the walk that follows. * s3: take the object write lock for a marker PUT The overwrite check read the entry that the mkdir after it replaces, so two marker PUTs could both pass while one was still unlocked. The marker delete already runs under this lock; hold it across the check and the mkdir so the entry cannot change in between, and so the two paths are serialized against each other.
SeaweedFS S3 Object Retention Tests
This directory contains comprehensive tests for SeaweedFS S3 Object Retention functionality, including Object Lock, Legal Hold, and WORM (Write Once Read Many) capabilities.
Overview
The test suite validates AWS S3-compatible object retention features including:
- Object Retention: GOVERNANCE and COMPLIANCE modes with retain-until-date
- Legal Hold: Independent protection that can be applied/removed
- Object Lock Configuration: Bucket-level default retention policies
- WORM Integration: Compatibility with legacy WORM functionality
- Version-specific Retention: Different retention policies per object version
- Enforcement: Protection against deletion and overwriting
Test Files
s3_retention_test.go- Core retention functionality testss3_worm_integration_test.go- WORM integration and advanced scenariostest_config.json- Test configuration (endpoints, credentials)Makefile- Comprehensive test automationgo.mod- Go module dependencies
Prerequisites
- Go 1.21 or later
- SeaweedFS binary built (
make build-weed) - AWS SDK Go v2
- Testify testing framework
Quick Start
1. Build and Start Server
# Build SeaweedFS and start test server
make start-server
2. Run Tests
# Run core retention tests
make test-retention-quick
# Run all retention tests
make test-retention
# Run WORM integration tests
make test-retention-worm
# Run all tests with managed server
make test-with-server
3. Cleanup
make clean
Test Categories
Core Retention Tests
TestBasicRetentionWorkflow- Basic GOVERNANCE mode retentionTestRetentionModeCompliance- COMPLIANCE mode (immutable)TestLegalHoldWorkflow- Legal hold on/off functionalityTestObjectLockConfiguration- Bucket object lock settings
Advanced Tests
TestRetentionWithVersions- Version-specific retention policiesTestRetentionAndLegalHoldCombination- Multiple protection typesTestExpiredRetention- Post-expiration behaviorTestRetentionErrorCases- Error handling and edge cases
WORM Integration Tests
TestWORMRetentionIntegration- New retention + legacy WORMTestWORMLegacyCompatibility- Backward compatibilityTestRetentionOverwriteProtection- Prevent overwritesTestRetentionBulkOperations- Bulk delete with retentionTestRetentionWithMultipartUpload- Multipart upload retentionTestRetentionExtendedAttributes- Extended attribute storageTestRetentionBucketDefaults- Default retention applicationTestRetentionConcurrentOperations- Concurrent operation safety
Individual Test Targets
Run specific test categories:
# Basic functionality
make test-basic-retention
make test-compliance-retention
make test-legal-hold
# Advanced features
make test-retention-versions
make test-retention-combination
make test-expired-retention
# WORM integration
make test-worm-integration
make test-worm-legacy
make test-retention-bulk
Configuration
Server Configuration
The tests use these default settings:
- S3 Port: 8333
- Test timeout: 15 minutes
- Volume directory:
./test-volume-data
Test Configuration (test_config.json)
{
"endpoint": "http://localhost:8333",
"access_key": "some_access_key1",
"secret_key": "some_secret_key1",
"region": "us-east-1",
"bucket_prefix": "test-retention-",
"use_ssl": false,
"skip_verify_ssl": true
}
Expected Behavior
GOVERNANCE Mode
- Objects protected until retain-until-date
- Can be bypassed with
x-amz-bypass-governance-retentionheader - Supports time extension (not reduction)
COMPLIANCE Mode
- Objects immutably protected until retain-until-date
- Cannot be bypassed or shortened
- Strictest protection level
Legal Hold
- Independent ON/OFF protection
- Can coexist with retention policies
- Must be explicitly removed to allow deletion
Version Support
- Each object version can have individual retention
- Applies to both versioned and non-versioned buckets
- Version-specific retention retrieval
Development
Running in Development Mode
# Start server for development
make dev-start
# Run quick test
make dev-test
Code Quality
# Format code
make fmt
# Run linter
make lint
# Generate coverage report
make coverage
Performance Testing
# Run benchmarks
make benchmark-retention
Troubleshooting
Server Won't Start
# Check if port is in use
netstat -tlnp | grep 8333
# View server logs
make logs
# Force cleanup
make clean
Test Failures
# Run with verbose output
go test -v -timeout=15m .
# Run specific test
go test -v -run TestBasicRetentionWorkflow .
# Check server health
make health-check
Dependencies
# Install/update dependencies
make install-deps
# Check dependency status
make check-deps
Integration with SeaweedFS
These tests validate the retention implementation in:
weed/s3api/s3api_object_retention.go- Core retention logicweed/s3api/s3api_object_handlers_retention.go- HTTP handlersweed/s3api/s3_constants/extend_key.go- Extended attribute keysweed/s3api/s3err/s3api_errors.go- Error definitionsweed/s3api/s3api_object_handlers_delete.go- Deletion enforcementweed/s3api/s3api_object_handlers_put.go- Upload enforcement
AWS CLI Compatibility
The retention implementation supports standard AWS CLI commands:
# Set object retention
aws s3api put-object-retention \
--bucket mybucket \
--key myobject \
--retention Mode=GOVERNANCE,RetainUntilDate=2024-12-31T23:59:59Z
# Get object retention
aws s3api get-object-retention \
--bucket mybucket \
--key myobject
# Set legal hold
aws s3api put-object-legal-hold \
--bucket mybucket \
--key myobject \
--legal-hold Status=ON
# Configure bucket object lock
aws s3api put-object-lock-configuration \
--bucket mybucket \
--object-lock-configuration ObjectLockEnabled=Enabled,Rule='{DefaultRetention={Mode=GOVERNANCE,Days=30}}'
Contributing
When adding new retention tests:
- Follow existing test patterns
- Use descriptive test names
- Include both positive and negative test cases
- Test error conditions
- Update this README with new test descriptions
- Add appropriate Makefile targets for new test categories