Fixes#1683Fixes#1684Fixes#1685Fixes#1690Fixes#1691Fixes#1692Fixes#1694Fixes#1695
This PR primarily focuses on error handling and checksum calculation for signed streaming requests of type `STREAMING-AWS4-HMAC-SHA256-PAYLOAD` and `STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER`.
It fixes the error type returned when the payload encoding is incorrect: the correct `IncompleteBody` error is now returned.
Chunk size validation has been added, enforcing the rule that only the final chunk may be smaller than 8192 bytes.
The `x-amz-trailer` header value is now validated against the checksum trailer present in the payload. For `STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER` requests, if no checksum trailer is provided in the payload, the gateway now returns an `IncompleteBody` error.
If there is a mismatch between the `x-amz-trailer` header and the checksum trailer in the payload, or if the checksum header key in the payload is invalid, a `MalformedTrailer` error is returned.
The `x-amz-decoded-content-length` header value is now compared against the actual decoded payload length, and an `IncompleteBody` error is returned if there is a mismatch.
Finally, the double checksum calculation issue has been fixed. For `STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER` requests, the trailing checksum is now parsed from the request payload and stored in the backend, instead of being recalculated by the backend.
Fixes#1676
`x-amz-decoded-content-length` in streaming uploads specifies the number of actual data-payload bytes, with encoding characters removed. If the value does not match the actual payload after decoding, now an `IncompleteBody` error is returned.
Fixes#1600Fixes#1603Fixes#1607Fixes#1626Fixes#1632Fixes#1652Fixes#1653Fixes#1656Fixes#1657Fixes#1659
This PR focuses mainly on unsigned streaming payload **trailer request payload parsing** and **checksum calculation**. For streaming uploads, there are essentially two ways to specify checksums:
1. via `x-amz-checksum-*` headers,
2. via `x-amz-trailer`,
or none — in which case the checksum should default to **crc64nvme**.
Previously, the implementation calculated the checksum only from `x-amz-checksum-*` headers. Now, `x-amz-trailer` is also treated as a checksum-related header and indicates the checksum algorithm for streaming requests. If `x-amz-trailer` is present, the payload must include a trailing checksum; otherwise, an error is returned.
`x-amz-trailer` and any `x-amz-checksum-*` header **cannot** be used together — doing so results in an error.
If `x-amz-sdk-checksum-algorithm` is specified, then either `x-amz-trailer` or one of the `x-amz-checksum-*` headers must also be present, and the algorithms must match. If they don’t, an error is returned.
The old implementation used to return an internal error when no `x-amz-trailer` was received in streaming requests or when the payload didn’t contain a trailer. This is now fixed.
Checksum calculation used to happen twice in the gateway (once in the chunk reader and once in the backend). A new `ChecksumReader` is introduced to prevent double computation, and the trailing checksum is now read by the backend from the chunk reader. The logic for stacking `io.Reader`s in the Fiber context is preserved, but extended: once a `ChecksumReader` is stacked, all following `io.Reader`s are wrapped with `MockChecksumReader`, which simply delegates to the underlying checksum reader. In the backend, a simple type assertion on `io.Reader` provides the necessary checksum metadata (algorithm, value, etc.).
Fixes#1554Fixes#1423
The gateway previously ignored the `x-amz-content-sha256` header for anonymous unsigned requests to public buckets. This PR adds hash calculation for this header and correctly handles special payload types.
It also fixes the case where a signed streaming payload (`STREAMING-AWS4-HMAC-SHA256-PAYLOAD...`) is used with anonymous requests. In this scenario, the gateway now returns a specific "not supported" error, consistent with S3 behavior.
The debuglogger should be a top level module since we expect
all modules within the project to make use of this. If its
hidden in s3api, then contributors are less likely to make
use of this outside of s3api.
Fixes#1352
Adds a validation check step in `SigV4` authentication for `x-amz-content-sh256` to check it to be either a valid sha256 hash or a special payload type(UNSIGNED-PAYLOAD, STREAMING-UNSIGNED-PAYLOAD-TRAILER...).
This implementation introduces **public buckets**, which are accessible without signature-based authentication.
There are two ways to grant public access to a bucket:
* **Bucket ACLs**
* **Bucket Policies**
Only `Get` and `List` operations are permitted on public buckets. All **write operations** require authentication, regardless of whether public access is granted through an ACL or a policy.
The implementation includes an `AuthorizePublicBucketAccess` middleware, which checks if public access has been granted to the bucket. If so, authentication middlewares are skipped. For unauthenticated requests, appropriate errors are returned based on the specific S3 action.
---
**1. Bucket-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test"
}
]
}
```
**2. Object-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test/*"
}
]
}
```
**3. Both Bucket and Object-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test"
},
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test/*"
}
]
}
```
---
```sh
aws s3api create-bucket --bucket test --object-ownership BucketOwnerPreferred
aws s3api put-bucket-acl --bucket test --acl public-read
```
Closes#1221
Adds debug logging for `signed`/`unsigned` chunk readers.
Adds the `debuglogger.Infof` log method, which prints out green info logs with `[INFO]:` prefix.
The debug logging inclues some chunk details: size, signature, trailers. It also prints out stash/release stash operations.
The error cases are logged with standart yellow `[DEBUG]:` prefix.
The `String to sign` block in signed chunk reader is logged in purple horizontal borders with title.
Fixes#961Fixes#1248
The gateway should return a `MissingContentLength` error if the `Content-Length` HTTP header is missing for upload operations (`PutObject`, `UploadPart`).
The second fix involves enforcing a maximum object size limit of `5 * 1024 * 1024 * 1024` bytes (5 GB) by validating the value of the `Content-Length` header. If the value exceeds this limit, the gateway should return an `EntityTooLarge` error.
Added missing debug logs in the `front-end` and `utility` functions.
Enhanced debug logging with the following improvements:
- Each debug message is now prefixed with [DEBUG] and appears in color.
- The full request URL is printed at the beginning of each debug log block.
- Request/response details are wrapped in framed sections for better readability.
- Headers are displayed in a colored box.
- XML request/response bodies are pretty-printed with indentation and color.
Closes#1159Fixes#1161
Implements signed chunk encoding with trailers in the gateway.
The signed encoding (both with and without trailers) is now handled by the `ChunkReader`.
Fixes the `ChunkReader` implementation to validate encoding headers byte by byte.
The chunk encoding with trailers follows the general signed chunk encoding pattern, but the final chunk includes the trailing signature (`x-amz-trailing-signature`) and the checksum header (`x-amz-checksum-x`, where `x` can be `crc32`, `crc32c`, `sha1`, `sha256`, or `crc64nvme`).
Adds validation for the `X-Amz-Trailer` header.
Make the code scanners happy with a bounds check before we do the
integer conversion from int64 to int, since this can overflow on
32 bit platforms.
Best error to return here is a signature error since this is a
client problem and the chunk headers are considered part of the
request signature.
Fixes#512. For chunked uploads, we parse the chunk headers in place
and then move the data payload up on the buffer to overwrite the
chunk headers for the real data stream.
For the special case where the chunk header was truncated in the
current read buffer, the partial header is stashed in a temporary
byte slice. The following read will contain the remainder of the
header that we can put together and parse.
We were correctly parsing this, but we forgot that the data offset
is calculated based on the start of the header. But the special
case where part of the header was stashed means we were incorrectly
calculating the data offset into the read buffer.
Easy fix to just remove the stash size from the data offset return
value.
As described in
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
this adds support for reading from a chunked upload encoded request
body. The chunked reader modifies the data stream to remove the
chunk encoding while validating the chunk signatures in line. This
allows the upper layers to get just the object data stream.
