mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-21 14:46:58 +00:00
Chris LuandGitHub
525900dfe4
* fix(s3api): backfill missing per-chunk SSE-S3 metadata at completion When a part of an SSE-S3 multipart upload lands with SseType=NONE on its chunks (e.g. a transient failure to apply SSE-S3 setup in PutObjectPart), the completed object inherits NONE-tagged chunks and detectPrimarySSEType then misses the chunked SSE-S3 encryption. The read path falls through to the unencrypted serve and GET returns ciphertext, producing the SHA mismatch reported in #8908. Recover at completion using the base IV and key data the upload directory recorded at CreateMultipartUpload: - extractMultipartSSES3Info validates upload-entry metadata up front and hard-fails completion if the base IV or key data are malformed; serializing chunk metadata we then could not decrypt is worse than rejecting the upload. - completedMultipartChunk re-derives a per-chunk IV from baseIV + chunk.Offset (matching what putToFiler would have written) and serializes per-chunk SSE-S3 metadata when the chunk has no tag. Existing per-chunk metadata is left alone; we cannot recover an already-derived IV from the upload-entry alone. The IV formula intentionally has no partNumber term: putToFiler hardcodes partOffset=0 when it calls handleSSES3MultipartEncryption for every part, so each chunk's encryption IV is calculateIVWithOffset(baseIV, chunk.Offset_part_local). PartOffsetMultiplier is defined in s3_constants but is not consumed by the encryption path. Adopting (partNumber-1)*PartOffsetMultiplier + chunk.Offset would produce IVs that fail to decrypt the bytes on disk - a stronger failure mode than the bug being fixed. Tests pin this: - TestCompletedMultipartChunkBackfilledIVDecryptsActualCiphertext runs the round trip across the encryption boundary: encrypt parts with CreateSSES3EncryptedReaderWithBaseIV (the call putToFiler uses), drop chunk metadata to reproduce #8908, backfill, decrypt with backfilled IV, assert plaintext intact. - TestCompletedMultipartChunkRejectsPartNumberMultiplierFormula constructs the IV the partNumber formula would produce and shows it does not decrypt the actual ciphertext. This commit covers the chunk-level recovery only. The companion fix for the object-level Extended attributes (SeaweedFSSSES3Key / X-Amz-Server-Side-Encryption) follows separately. * fix(s3api): backfill canonical SSE-S3 attributes onto multipart object The previous commit ensures every chunk of an SSE-S3 multipart upload carries SseType=SSE_S3 with a per-chunk IV, so the multipart-direct read path can decrypt. The completed object's Extended map can still miss the canonical pair detectPrimarySSEType and IsSSES3EncryptedInternal look at: - X-Amz-Server-Side-Encryption (the AmzServerSideEncryption header detectPrimarySSEType reads on inline / small-object reads) - x-seaweedfs-sse-s3-key (SeaweedFSSSES3Key, required by IsSSES3EncryptedInternal and by the read-path key lookup) When a part of the upload was written by a path that did not set those (the same #8908 race that produced the NONE chunks), copySSEHeadersFromFirstPart finds nothing to copy and the final entry ends up with only the multipart-init keys (SeaweedFSSSES3Encryption / BaseIV / KeyData). The read path then mis-detects the object as unencrypted. applyMultipartSSES3HeadersFromUploadEntry writes the canonical pair from the multipart-init metadata in all three completion paths (versioned, suspended, non-versioned), only when the keys are missing so a healthy first part still wins. extractMultipartSSES3Info already ran in prepareMultipartCompletionState, so the data is reused without re-decoding. Tests: TestApplyMultipartSSES3HeadersFromUploadEntry covers backfill, do-not-clobber, and nil-info no-op cases. * fix(s3api): drop double IV adjustment in SSE-KMS chunk view decrypt decryptSSEKMSChunkView was pre-adjusting the SSE-KMS chunk IV (calculateIVWithOffset(baseIV, ChunkOffset)) and then handing the adjusted IV to CreateSSEKMSDecryptedReader, which itself runs calculateIVWithOffset(IV, ChunkOffset) on whatever it receives. The offset was being applied twice for any chunk with a non-zero ChunkOffset, corrupting the keystream for range reads that cross multipart chunk boundaries. Pass the raw SSE-KMS key (with base IV and the original ChunkOffset field) into CreateSSEKMSDecryptedReader so the offset is applied exactly once, and remove the now-dead intra-block skip that was compensating for the double adjustment. Add an anti-test inside TestSSEKMSDecryptChunkView_RequiresOffsetAdjustment that decrypts the same ciphertext with a deliberately double-adjusted IV and asserts the output is corrupted, so any regression that re-introduces the double application fails the unit test. * test(s3): cover multipart SSE across chunk-spanning parts and ranges Adds an integration subtest "Multipart Parts Larger Than Internal Chunks Across SSE Types" to TestSSEMultipartUploadIntegration that exercises the end-to-end S3 path for the bugs fixed in this branch: - Two-part multipart upload with each part larger than the 8MB internal SeaweedFS chunk, so each part itself spans multiple underlying chunks. - Subtests for SSE-C, SSE-KMS, explicit SSE-S3, and bucket-default SSE-S3 - the four paths multipart parts can take through the SSE pipeline. - Each subtest does a full GET (verifying every byte and the response Content-Length / SSE response headers) plus a 129-byte range read straddling the 8MB internal chunk boundary, which is the path that produced the SSE-KMS double-IV corruption (fix in the previous commit) and the SSE-S3 chunk-tag loss (fix in the earlier commits). Factored the request shape behind multipartSSEOptions / uploadAndVerifyMultipartSSEObject so all four SSE flavors share the same upload+verify code; only the SSE-specific input/output configuration differs per subtest. * test(s3): abort orphan multipart uploads on test failure Address coderabbit nitpick on uploadAndVerifyMultipartSSEObject. The helper used require.NoError after CreateMultipartUpload, UploadPart and CompleteMultipartUpload, so a failure in any of those (or in the later GET / range read on a still-incomplete upload) called t.Fatal without aborting the in-flight MPU, leaving an orphan upload in the bucket. Harmless in CI where the data dir is wiped on shutdown, but a real annoyance when iterating locally and a textbook AWS S3 caveat in production. Register a t.Cleanup that calls AbortMultipartUpload unless a "completed" flag was set right after a successful CompleteMultipartUpload. Use context.Background for the abort call since the parent ctx may already be cancelled at cleanup time, and t.Logf the abort error rather than failing the test so the original failure remains visible in the run output.
S3 Server-Side Encryption (SSE) Integration Tests
This directory contains comprehensive integration tests for SeaweedFS S3 API Server-Side Encryption functionality. These tests validate the complete end-to-end encryption/decryption pipeline from S3 API requests through filer metadata storage.
Overview
The SSE integration tests cover three main encryption methods:
- SSE-C (Customer-Provided Keys): Client provides encryption keys via request headers
- SSE-KMS (Key Management Service): Server manages encryption keys through a KMS provider
- SSE-S3 (Server-Managed Keys): Server automatically manages encryption keys
🆕 Real KMS Integration
The tests now include real KMS integration with OpenBao, providing:
- ✅ Actual encryption/decryption operations (not mock keys)
- ✅ Multiple KMS keys for different security levels
- ✅ Per-bucket KMS configuration testing
- ✅ Performance benchmarking with real KMS operations
See README_KMS.md for detailed KMS integration documentation.
Why Integration Tests Matter
These integration tests were created to address a critical gap in test coverage that previously existed. While the SeaweedFS codebase had comprehensive unit tests for SSE components, it lacked integration tests that validated the complete request flow:
Client Request → S3 API → Filer Storage → Metadata Persistence → Retrieval → Decryption
The Bug These Tests Would Have Caught
A critical bug was discovered where:
- ✅ S3 API correctly encrypted data and sent metadata headers to the filer
- ❌ Filer did not process SSE metadata headers, losing all encryption metadata
- ❌ Objects could be encrypted but never decrypted (metadata was lost)
Unit tests passed because they tested components in isolation, but the integration was broken. These integration tests specifically validate that:
- Encryption metadata is correctly sent to the filer
- Filer properly processes and stores the metadata
- Objects can be successfully retrieved and decrypted
- Copy operations preserve encryption metadata
- Multipart uploads maintain encryption consistency
Test Structure
Core Integration Tests
Basic Functionality
TestSSECIntegrationBasic- Basic SSE-C PUT/GET cycleTestSSEKMSIntegrationBasic- Basic SSE-KMS PUT/GET cycle
Data Size Validation
TestSSECIntegrationVariousDataSizes- SSE-C with various data sizes (0B to 1MB)TestSSEKMSIntegrationVariousDataSizes- SSE-KMS with various data sizes
Object Copy Operations
TestSSECObjectCopyIntegration- SSE-C object copying (key rotation, encryption changes)TestSSEKMSObjectCopyIntegration- SSE-KMS object copying
Multipart Uploads
TestSSEMultipartUploadIntegration- SSE multipart uploads for large objects
Error Conditions
TestSSEErrorConditions- Invalid keys, malformed requests, error handling
Performance Tests
BenchmarkSSECThroughput- SSE-C performance benchmarkingBenchmarkSSEKMSThroughput- SSE-KMS performance benchmarking
Running Tests
Prerequisites
-
Build SeaweedFS: Ensure the
weedbinary is built and available in PATHcd /path/to/seaweedfs make -
Dependencies: Tests use AWS SDK Go v2 and testify - these are handled by Go modules
Quick Test
Run basic SSE integration tests:
make test-basic
Comprehensive Testing
Run all SSE integration tests:
make test
Specific Test Categories
make test-ssec # SSE-C tests only
make test-ssekms # SSE-KMS tests only
make test-copy # Copy operation tests
make test-multipart # Multipart upload tests
make test-errors # Error condition tests
Performance Testing
make benchmark # Performance benchmarks
make perf # Various data size performance tests
KMS Integration Testing
make setup-openbao # Set up OpenBao KMS
make test-with-kms # Run all SSE tests with real KMS
make test-ssekms-integration # Run SSE-KMS with OpenBao only
make clean-kms # Clean up KMS environment
Development Testing
make manual-start # Start SeaweedFS for manual testing
# ... run manual tests ...
make manual-stop # Stop and cleanup
Test Configuration
Default Configuration
The tests use these default settings:
- S3 Endpoint:
http://127.0.0.1:8333 - Access Key:
some_access_key1 - Secret Key:
some_secret_key1 - Region:
us-east-1 - Bucket Prefix:
test-sse-
Custom Configuration
Override defaults via environment variables:
S3_PORT=8444 FILER_PORT=8889 make test
Test Environment
Each test run:
- Starts a complete SeaweedFS cluster (master, volume, filer, s3)
- Configures KMS support for SSE-KMS tests
- Creates temporary buckets with unique names
- Runs tests with real HTTP requests
- Cleans up all test artifacts
Test Data Coverage
Data Sizes Tested
- 0 bytes: Empty files (edge case)
- 1 byte: Minimal data
- 16 bytes: Single AES block
- 31 bytes: Just under two blocks
- 32 bytes: Exactly two blocks
- 100 bytes: Small file
- 1 KB: Small text file
- 8 KB: Medium file
- 64 KB: Large file
- 1 MB: Very large file
Encryption Key Scenarios
- SSE-C: Random 256-bit keys, key rotation, wrong keys
- SSE-KMS: Various key IDs, encryption contexts, bucket keys
- Copy Operations: Same key, different keys, encryption transitions
Critical Test Scenarios
Metadata Persistence Validation
The integration tests specifically validate scenarios that would catch metadata storage bugs:
// 1. Upload with SSE-C
client.PutObject(..., SSECustomerKey: key) // ← Metadata sent to filer
// 2. Retrieve with SSE-C
client.GetObject(..., SSECustomerKey: key) // ← Metadata retrieved from filer
// 3. Verify decryption works
assert.Equal(originalData, decryptedData) // ← Would fail if metadata lost
Content-Length Validation
Tests verify that Content-Length headers are correct, which would catch bugs related to IV handling:
assert.Equal(int64(originalSize), resp.ContentLength) // ← Would catch IV-in-stream bugs
Debugging
View Logs
make debug-logs # Show recent log entries
make debug-status # Show process and port status
Manual Testing
make manual-start # Start SeaweedFS
# Test with S3 clients, curl, etc.
make manual-stop # Cleanup
Integration Test Benefits
These integration tests provide:
- End-to-End Validation: Complete request pipeline testing
- Metadata Persistence: Validates filer storage/retrieval of encryption metadata
- Real Network Communication: Uses actual HTTP requests and responses
- Production-Like Environment: Full SeaweedFS cluster with all components
- Regression Protection: Prevents critical integration bugs
- Performance Baselines: Benchmarking for performance monitoring
Continuous Integration
For CI/CD pipelines, use:
make ci-test # Quick tests suitable for CI
make stress # Stress testing for stability validation
Key Differences from Unit Tests
| Aspect | Unit Tests | Integration Tests |
|---|---|---|
| Scope | Individual functions | Complete request pipeline |
| Dependencies | Mocked/simulated | Real SeaweedFS cluster |
| Network | None | Real HTTP requests |
| Storage | In-memory | Real filer database |
| Metadata | Manual simulation | Actual storage/retrieval |
| Speed | Fast (milliseconds) | Slower (seconds) |
| Coverage | Component logic | System integration |
Conclusion
These integration tests ensure that SeaweedFS SSE functionality works correctly in production-like environments. They complement the existing unit tests by validating that all components work together properly, providing confidence that encryption/decryption operations will succeed for real users.
Most importantly, these tests would have immediately caught the critical filer metadata storage bug that was previously undetected, demonstrating the crucial importance of integration testing for distributed systems.