Problem: SigV4 chunked upload checksum trailer parsing rejected mixed-case checksum header names even though HTTP header field names are case-insensitive.
Root cause: extractChecksumAlgorithm compared the x-amz-trailer value and trailer header key against exact lowercase strings.
Fix: Trim and lowercase checksum trailer header names before matching supported checksum algorithms.
Reproduction: go test ./weed/s3api -run TestExtractChecksumAlgorithmIsCaseInsensitive -count=1 with X-Amz-Checksum-Crc32; before the fix it returned unsupported checksum algorithm.
Validation: gofmt -w weed/s3api/chunked_reader_v4.go weed/s3api/chunked_reader_v4_test.go; git diff --check; go test ./weed/s3api -run TestExtractChecksumAlgorithmIsCaseInsensitive -count=1; go test ./weed/s3api -count=1
Co-authored-by: Codex <noreply@openai.com>
A bearer-token client whose SDK appends a CRC32 trailer sends an
unsigned-streaming PUT (STREAMING-UNSIGNED-PAYLOAD-TRAILER) with no SigV4
signature, so getRequestAuthType classifies it as authTypeStreamingUnsigned.
The auth dispatch ignored the bearer token and fell back to anonymous, and
newChunkedReader tried to verify the bearer token as a SigV4 seed signature
and failed, so the body could not be decoded either.
Dispatch the streaming-unsigned auth on whatever credential is present
(SigV4 / JWT / anonymous), and skip the SigV4 seed-signature recompute for
JWT requests in the chunked reader.
2026-05-28 18:10:24 -07:00
Chris LuGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* s3: support STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER for signed chunked uploads with checksums
When AWS SDK v2 clients upload with both chunked encoding and checksum
validation enabled, they use the x-amz-content-sha256 header value of
STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER instead of the simpler
STREAMING-AWS4-HMAC-SHA256-PAYLOAD.
This caused the chunked reader to not be properly activated, resulting
in chunk-signature metadata being stored as part of the file content.
Changes:
- Add streamingSignedPayloadTrailer constant for the new header value
- Update isRequestSignStreamingV4() to recognize this header
- Update newChunkedReader() to handle this streaming type
- Update calculateSeedSignature() to accept this header
- Add unit test for signed streaming upload with trailer
Fixes issue where Quarkus/AWS SDK v2 uploads with checksum validation
resulted in corrupted file content containing chunk-signature data.
* address review comments: add trailer signature to test, fix constant alignment
* test: separate canonical trailer text (\n) from on-wire format (\r\n)
* test: add negative test for invalid trailer signature
* refactor: check HTTP method first in streaming auth checks (fail-fast)
* test: handle crc32 Write error return for completeness
* refactor: extract createTrailerStreamingRequest helper to reduce test duplication
* fmt
* docs: clarify test comment about trailer signature validation status
* refactor: calculate chunk data length dynamically instead of hardcoding
* Update weed/s3api/chunked_reader_v4_test.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix: use current time for signatures instead of hardcoded past date
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Fix issue #6847: S3 chunked encoding includes headers in stored content
- Add hasTrailer flag to s3ChunkedReader to track trailer presence
- Update state transition logic to properly handle trailers in unsigned streaming
- Enhance parseChunkChecksum to handle multiple trailer lines
- Skip checksum verification for unsigned streaming uploads
- Add test case for mixed format handling (unsigned headers with signed chunks)
- Remove redundant CRLF reading in trailer processing
This fixes the issue where chunk-signature and x-amz headers were appearing
in stored file content when using chunked encoding with newer AWS SDKs.
* Fix checksum validation for unsigned streaming uploads
- Always validate checksum for data integrity regardless of signing
- Correct checksum value in test case
- Addresses PR review feedback about checksum verification
* Add warning log when multiple checksum headers found in trailer
- Log a warning when multiple valid checksum headers appear in trailers
- Uses last checksum header as suggested by CodeRabbit reviewer
- Improves debugging for edge cases with multiple checksum algorithms
* Improve trailer parsing robustness in parseChunkChecksum
- Remove redundant trimTrailingWhitespace call since readChunkLine already trims
- Use bytes.TrimSpace for both key and value to handle whitespace around colon separator
- Follows HTTP header specifications for optional whitespace around separators
- Addresses Gemini Code Assist review feedback
* Added continue statements after all state transitions in the state machine to ensure immediate state processing
* simplify
* remove redundant continue clause
* ensure wrong signature
* This handles the case where we have unsigned streaming (no cred) but chunks contain signatures
* Update chunked_reader_v4.go
* address comments
2025-08-11 10:31:01 -07:00
Chris LuGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This change is caching HMAC hashers for repeated use in subsequent
requests and chunks, so they don't have to be initialized from
scratch every time.
On my local computer this gives me ~5-6 times faster signature
calculation and ~5-6.5% more throughput in S3 requests. The smaller
the payload the better the throughput gets.
Chris LuandGitHub
Tom CrassetandGitHub
NinaLuaandGitHub
Patrick SchmidtandChris Lu
Ryan RussellandGitHub
Konstantin Lebedev