mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-29 12:17:08 +00:00

Files

T

Chris LuandGitHub 863fec6c3f S3: let a key that is a prefix of other keys be an object (#10912 )

* filer: keep the sentinel when CreateEntry reports an update failure

CreateEntry flattened the error UpdateEntry wraps, so errors.Is stopped
matching and ErrExistingIsDirectory and ErrExistingIsFile never reached
the S3 mapper, which answered a retryable 500 instead.

* s3: let a key that is a prefix of other keys be an object

S3 keys are flat, so "a/b" and "a/b/c" are independent objects that
coexist in either write order. The filer stores a key as a path, so one
of them has to live on the directory the other is nested under.

Writing the nested key first refused the prefix key outright. Writing it
second promoted the file to a directory, which kept its data but lost the
key: an empty object left nothing to recognise it by and disappeared, and
one with data listed under a trailing slash it never had.

Mark the directory that carries such a key, and write the object onto it
when the path is already a directory. The mark makes an empty prefix
object visible to listings and readable by GET and HEAD, keeps the empty
folder cleaner off it, and lists it under the key it was written with.
Deleting the key strips the mark back off along with the data.

* filer: keep a TTL off a directory that stands for an object

An expired entry is deleted a row at a time, so expiring a directory
removes it and leaves everything under it unreachable. Promoting a file
to a directory carried its TTL across, and a promoted file is exactly the
one that has keys nested under it.

Drop the TTL on promotion, and leave one an older build wrote alone. The
lifecycle worker still expires the object, through the delete that leaves
the directory behind.

* s3: delete the null version of a key other keys are nested under

The routed delete cannot remove an entry that other keys live under, and
answered a retryable 500 rather than falling back to the lock path the
unversioned delete already falls back to. That path then looked the entry
up under the bucket with the whole key as its name, so the demote wrote it
back one directory too high and failed as not found.

Fall back on any non-precondition error, and split the key before deleting
it. Trailing-slash directory markers with children reach the same delete.

* filer: keep the sentinel when MkFile and Mkdir report a create failure

Same flattening one layer out: every mkFile caller lost the sentinel, so
a CopyObject onto a key that other keys are nested under answered a
retryable 500 where a PutObject of the same key answers 409.

* s3: copy and rename a key that other keys are nested under

Such a key is stored on the directory those keys live in, and copy and
rename both refused it: the source lookup maps every directory entry to
NoSuchKey, so a key a plain GET serves could not be copied or moved, and
the destination side refused it as a directory conflict.

The source is read through a view of the entry as the object it names.
The destination is written the way a PutObject of that key writes it. A
rename at either end copies the object's own data across and strips it off
the source key rather than going through AtomicRenameEntry, which moves a
directory by moving everything under it - the nested keys are not part of
what is being renamed.

2026-08-24 15:10:34 -07:00

Makefile

test(s3): aim collection force-delete at the master the suite actually runs (#10404 )

2026-07-22 21:37:59 -07:00

README.md

More efficient copy object (#6665 )

2025-07-11 18:51:32 -07:00

s3_copying_test.go

test(s3): force-drop collection after deleteBucket in tagging/versioning/cors/copying (#9270 )

2026-04-28 22:13:42 -07:00

s3_rename_test.go

S3: let a key that is a prefix of other keys be an object (#10912 )

2026-08-24 15:10:34 -07:00

test_config.json

More efficient copy object (#6665 )

2025-07-11 18:51:32 -07:00

README.md

SeaweedFS S3 Copying Tests

This directory contains comprehensive Go tests for SeaweedFS S3 copying functionality, converted from the failing Python tests in the s3-tests repository.

Overview

These tests verify that SeaweedFS correctly implements S3 operations, starting with basic put/get operations and progressing to advanced copy operations, including:

Basic S3 Operations: Put/Get operations, bucket management, and metadata handling
Basic object copying: within the same bucket
Cross-bucket copying: across different buckets
Multipart copy operations: for large files
Conditional copy operations: ETag-based conditional copying
Metadata handling: during copy operations
ACL handling: during copy operations

Test Coverage

Basic S3 Operations (Run First)

TestBasicPutGet: Tests fundamental S3 put/get operations with various object types
TestBasicBucketOperations: Tests bucket creation, listing, and deletion
TestBasicLargeObject: Tests handling of larger objects (up to 10MB)

Basic Copy Operations

TestObjectCopySameBucket: Tests copying objects within the same bucket
TestObjectCopyDiffBucket: Tests copying objects to different buckets
TestObjectCopyCannedAcl: Tests copying with ACL settings
TestObjectCopyRetainingMetadata: Tests metadata preservation during copy

Multipart Copy Operations

TestMultipartCopySmall: Tests multipart copying of small files
TestMultipartCopyWithoutRange: Tests multipart copying without range specification
TestMultipartCopySpecialNames: Tests multipart copying with special character names
TestMultipartCopyMultipleSizes: Tests multipart copying with various file sizes

Conditional Copy Operations

TestCopyObjectIfMatchGood: Tests copying with matching ETag condition
TestCopyObjectIfMatchFailed: Tests copying with non-matching ETag condition (should fail)
TestCopyObjectIfNoneMatchFailed: Tests copying with non-matching ETag condition (should succeed)
TestCopyObjectIfNoneMatchGood: Tests copying with matching ETag condition (should fail)

Requirements

Go 1.19+: Required for AWS SDK v2 and modern Go features
SeaweedFS Binary: Built from source (../../../weed/weed)
Free Ports: 8333 (S3), 8888 (Filer), 8080 (Volume), 9333 (Master)
Dependencies: Uses the main repository's go.mod with existing AWS SDK v2 and testify dependencies

Quick Start

1. Build SeaweedFS

cd ../../../
make

2. Run Tests

# Run basic S3 operations first (recommended)
make test-basic

# Run all tests (starts with basic, then copy tests)
make test

# Run quick tests only
make test-quick

# Run multipart tests only
make test-multipart

# Run conditional tests only
make test-conditional

Available Make Targets

Basic Test Execution

make test-basic - Run basic S3 put/get operations (recommended first)
make test - Run all S3 tests (starts with basic, then copying)
make test-quick - Run quick tests only (basic copying)
make test-full - Run full test suite including large files
make test-multipart - Run multipart copying tests only
make test-conditional - Run conditional copying tests only

Server Management

make start-seaweedfs - Start SeaweedFS server for testing
make stop-seaweedfs - Stop SeaweedFS server
make manual-start - Start server for manual testing
make manual-stop - Stop server and clean up

Debugging

make debug-logs - Show recent log entries from all services
make debug-status - Show process and port status
make check-binary - Verify SeaweedFS binary exists

Performance Testing

make benchmark - Run performance benchmarks
make stress - Run stress tests with multiple iterations
make perf - Run performance tests with large files

Cleanup

make clean - Clean up test artifacts and temporary files

Configuration

The tests use the following default configuration:

{
  "endpoint": "http://localhost:8333",
  "access_key": "some_access_key1",
  "secret_key": "some_secret_key1",
  "region": "us-east-1",
  "bucket_prefix": "test-copying-",
  "use_ssl": false,
  "skip_verify_ssl": true
}

You can modify these values in test_config.json or by setting environment variables:

export SEAWEEDFS_BINARY=/path/to/weed
export S3_PORT=8333
export FILER_PORT=8888
export VOLUME_PORT=8080
export MASTER_PORT=9333
export TEST_TIMEOUT=10m
export VOLUME_MAX_SIZE_MB=50

Note: The volume size limit is set to 50MB to ensure proper testing of volume boundaries and multipart operations.

Test Details

TestBasicPutGet

Tests fundamental S3 put/get operations with various object types:
- Simple text objects
- Empty objects
- Binary objects (1KB random data)
- Objects with metadata and content-type
Verifies ETag consistency between put and get operations
Tests metadata preservation

TestBasicBucketOperations

Tests bucket creation and existence verification
Tests object listing in buckets
Tests object creation and listing with directory-like prefixes
Tests bucket deletion and cleanup
Verifies proper error handling for operations on non-existent buckets

TestBasicLargeObject

Tests handling of progressively larger objects:
- 1KB, 10KB, 100KB, 1MB, 5MB, 10MB
Verifies data integrity for large objects
Tests memory handling and streaming for large files
Ensures proper handling up to the 50MB volume limit

TestObjectCopySameBucket

Creates a bucket with a source object
Copies the object to a different key within the same bucket
Verifies the copied object has the same content

TestObjectCopyDiffBucket

Creates source and destination buckets
Copies an object from source to destination bucket
Verifies the copied object has the same content

TestObjectCopyCannedAcl

Tests copying with ACL settings (public-read)
Tests metadata replacement during copy with ACL
Verifies both basic copying and metadata handling

TestObjectCopyRetainingMetadata

Tests with different file sizes (3 bytes, 1MB)
Verifies metadata and content-type preservation
Checks that all metadata is correctly copied

TestMultipartCopySmall

Tests multipart copy with 1-byte files
Uses range-based copying (bytes=0-0)
Verifies multipart upload completion

TestMultipartCopyWithoutRange

Tests multipart copy without specifying range
Should copy entire source object
Verifies correct content length and data

TestMultipartCopySpecialNames

Tests with special character names: " ", "_", "__", "?versionId"
Verifies proper URL encoding and handling
Each special name is tested in isolation

TestMultipartCopyMultipleSizes

Tests with various copy sizes:
- 5MB (single part)
- 5MB + 100KB (multi-part)
- 5MB + 600KB (multi-part)
- 10MB + 100KB (multi-part)
- 10MB + 600KB (multi-part)
- 10MB (exact multi-part boundary)
Uses 5MB part size for all copies
Verifies data integrity across all sizes

TestCopyObjectIfMatchGood

Tests conditional copy with matching ETag
Should succeed when ETag matches
Verifies successful copy operation

TestCopyObjectIfMatchFailed

Tests conditional copy with non-matching ETag
Should fail with precondition error
Verifies proper error handling

TestCopyObjectIfNoneMatchFailed

Tests conditional copy with non-matching ETag for IfNoneMatch
Should succeed when ETag doesn't match
Verifies successful copy operation

TestCopyObjectIfNoneMatchGood

Tests conditional copy with matching ETag for IfNoneMatch
Should fail with precondition error
Verifies proper error handling

Expected Behavior

These tests verify that SeaweedFS correctly implements:

Basic S3 Operations: Standard PutObject, GetObject, ListBuckets, ListObjects APIs
Bucket Management: Bucket creation, deletion, and listing
Object Storage: Binary and text data storage with metadata
Large Object Handling: Efficient storage and retrieval of large files
Basic S3 Copy Operations: Standard CopyObject API
Multipart Copy Operations: UploadPartCopy API with range support
Conditional Operations: ETag-based conditional copying
Metadata Handling: Proper metadata preservation and replacement
ACL Handling: Access control list management during copy
Error Handling: Proper error responses for invalid operations

Troubleshooting

Common Issues

Port Already in Use
```
make stop-seaweedfs
make clean
```
SeaweedFS Binary Not Found
```
cd ../../../
make
```
Test Timeouts
```
export TEST_TIMEOUT=30m
make test
```
Permission Denied
```
sudo make clean
```

Debug Information

# Check server status
make debug-status

# View recent logs
make debug-logs

# Manual server start for investigation
make manual-start
# ... perform manual testing ...
make manual-stop

Log Locations

When running tests, logs are stored in:

Master: /tmp/seaweedfs-master.log
Volume: /tmp/seaweedfs-volume.log
Filer: /tmp/seaweedfs-filer.log
S3: /tmp/seaweedfs-s3.log

Contributing

When adding new tests:

Follow the existing naming convention (TestXxxYyy)
Use the helper functions for common operations
Add cleanup with defer deleteBucket(t, client, bucketName)
Include error checking with require.NoError(t, err)
Use assertions with assert.Equal(t, expected, actual)
Add the test to the appropriate Make target

Performance Notes

TestMultipartCopyMultipleSizes is the most resource-intensive test
Large file tests may take several minutes to complete
Memory usage scales with file sizes being tested
Network latency affects multipart copy performance

Integration with CI/CD

For automated testing:

# Basic validation (recommended first)
make test-basic

# Quick validation
make ci-test

# Full validation
make test-full

# Performance validation
make perf

The tests are designed to be self-contained and can run in containerized environments.