Fixes#1601
Unsigned streaming upload trailers (`STREAMING-UNSIGNED-PAYLOAD-TRAILER`) is allowed only for `PutObject` and `UploadPart`. For all other actions, the gateway now returns an `InvalidRequest` error for the `x-amz-content-sha256` header.
Fixes#1676
`x-amz-decoded-content-length` in streaming uploads specifies the number of actual data-payload bytes, with encoding characters removed. If the value does not match the actual payload after decoding, now an `IncompleteBody` error is returned.
Fixes#1665
S3 enforces a validation rule for unsigned streaming payload trailer uploads: all chunk sizes must be greater than 8192 bytes except for the final chunk.
This fix adds a check in the unsigned chunk reader that validates chunk sizes by comparing each chunk size to the previous one.
Fixes#1666Fixes#1660
Unsigned streaming payload trailers have strict encoding rules for the request body. If the body isn’t encoded correctly, the expected `IncompleteBody` API error is now returned. Incorrect encoding includes things like invalid chunk sizes, missing delimiters, or malformed `\r\n` sequences.
Fixes#1612
`x-amz-copy-source` is rejected with an **InvalidArgument** error in S3 for all HTTP methods other than **PUT** (i.e., **GET**, **POST**, **HEAD**, and **DELETE**). For **POST** requests, the behavior is slightly different: the error is returned only when the **uploadId** query parameter is present; otherwise, **MethodNotAllowed** is returned. This behavior applies to both bucket-level and object-level operations.
Fixes#1597
S3 returns a specific error when calling an object GET operation (e.g., `bucket/object/key?uploads`) with the `?uploads` query parameter. It’s not the standard `MethodNotAllowed` error. This PR adds support for handling this specific error route.
Closes#1635
Some S3 actions have dedicated bucket policy actions and require explicit policy permissions when operating on object versions. These actions were missing in the gateway: `GetObjectVersionTagging`, `PutObjectVersionTagging`, `DeleteObjectVersionTagging`, `DeleteObjectVersion`, and `GetObjectVersionAttributes`.
The logic for these actions is straightforward — if the incoming request includes the `versionId` query parameter, S3 enforces the corresponding bucket policy action that includes `version`.
This PR adds support for these missing actions in the gateway.
Fixes#1630
S3 returns `InvalidArgument: Invalid version id specified` for invalid version IDs in object-level actions that accept `versionId` as a query parameter. The `versionId` in S3 follows a specific structure, and if the input string doesn’t match this structure, the error is returned. In the gateway, the `versionId` is generated using the `ulid` package, which also has a defined structure. This PR adds validation for object-level operations that work with object versions by using the ULID parser.
These actions include: `HeadObject`, `GetObject`, `PutObjectTagging`, `GetObjectTagging`, `DeleteObjectTagging`, `PutObjectLegalHold`, `GetObjectLegalHold`, `PutObjectRetention`, `GetObjectRetention`, `DeleteObject`, `CopyObject`, `UploadPartCopy`, and `GetObjectAttributes`.
Fixes#1616
Some object-level actions in the gateway that work with object versions used to return `InvalidVersionId` when the specified object version did not exist. The logic has now been fixed, and they correctly return `NoSuchVersion`. These actions include: `HeadObject`, `GetObject`, `PutObjectLegalHold`, `GetObjectLegalHold`, `PutObjectRetention`, and `GetObjectRetention`.
Closes#1343
Object version tagging support was previously missing in the gateway. The support is added with this PR. If versioning is not enabled at the gateway level and a user attempts to put, get, or delete object version tags, the gateway returns an `InvalidArgument`(Invalid versionId)
Fixes#1379
Adds validation for bucket canned ACLs in `CreateBucket` and `PutBucketAcl`. The gateway supports three values: `private`, `public-read`, and `public-read-write`. All other values (including `authenticated-read`, which is not supported) are considered invalid and result in an `InvalidArgument` error with an empty error message.
Closes#1595
This implementation diverges from AWS S3 behavior. The `CreateBucket` request body is no longer ignored. Based on the S3 request body schema, the gateway parses only the `LocationConstraint` and `Tags` fields. If the `LocationConstraint` does not match the gateway’s region, it returns an `InvalidLocationConstraint` error.
In AWS S3, tagging during bucket creation is supported only for directory buckets. The gateway extends this support to general-purpose buckets.
If the request body is malformed, the gateway returns a `MalformedXML` error.
Fixes#1598
`PutObject` and `UploadPart` accept x-amz-checksum-* calculated checksum headers and `x-amz-sdk-checksum-algorithm`. If the checksum algorithm specified in sdk algorithm doesn't match the one in x-amz-checksum-*, it now returns the correct error message: `Value for x-amz-sdk-checksum-algorithm header is invalid.`.
Fixes#1482
The metadata keys should always be converted to lowercase in `PutObject`, `CreateMultipartUpload`, and `CopyObject`. This implementation converts the metadata keys to lowercase in the front end, ensuring they are stored in lowercase in the backend.
Closes#1549Fixes#1593Fixes#1521Fixes#1427Fixes#1311Fixes#1301Fixes#1040
This PR primarily focuses on checksum calculation within the gateway, but it also includes several related fixes and improvements.
It introduces a middleware responsible for handling and calculating checksums for the `x-amz-checksum-*` headers and `Content-MD5`. The middleware is applied only to actions that expect a request body or checksum headers. It also enforces validation for actions that require a non-empty request body, returning an error if the body is missing. Similarly, it returns an error for actions where at least one checksum header (`Content-MD5` or `x-amz-checksum-*`) is required but none is provided.
The implementation is based on [https://gist.github.com/niksis02/eec3198f03e561a0998d67af75c648d7](the reference table), tested directly against S3:
It also fixes the error case where the `x-amz-sdk-checksum-algorithm` header is present but no corresponding `x-amz-checksum-*` or `x-amz-trailer` header is included.
Additionally, the PR improves validation for the `x-amz-content-sha256` header. For actions that require this header, an error is now returned when it’s missing. For actions that don’t require it, the middleware no longer enforces its presence. Following the common S3 pattern, the header remains mandatory for admin routes.
Finally, the `x-amz-content-sha256` header is now optional for anonymous requests, as it is not required in that case.
Fixes#1579
S3 enforces a specific rule for validating bucket and object tag key/value names. This PR integrates the regexp pattern used by S3 for tag validation.
Official S3 documentation for tag validation rules: [AWS S3 Tag](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_Tag.html)
There are two types of tagging inputs for buckets and objects:
1. **On existing buckets/objects** — used in the `PutObjectTagging` and `PutBucketTagging` actions, where tags are provided in the request body.
2. **On object creation** — used in the `PutObject`, `CreateMultipartUpload`, and `CopyObject` actions, where tags are provided in the request headers and must be URL-encoded.
This implementation ensures correct validation for both types of tag inputs.
Fixes#1364
When a completely malformed request is sent to the gateway, Fiber/Fasthttp fails to parse the request, and the code execution never reaches the routers or handlers. Instead, the error is caught by the global error handler. These kinds of errors (malformed requests that fail during request parsing) are prefixed with **"error when reading request headers"** in Fiber. The implementation adds a check in the global error handler for this specific error message and returns an S3-like XML **BadRequest** error instead.
Fixes#1359
The composite checksums in **CompleteMultipartUpload** generally follow the format `checksum-<number_of_parts>`. Previously, the gateway treated composite checksums as regular checksums without distinguishing between the two formats.
In S3, the `x-amz-checksum-*` headers accept both plain checksum values and the `checksum-<number_of_parts>` format. However, after a successful `CompleteMultipartUpload` request, the final checksum is always stored with the part number included.
This implementation adds support for parsing both formats—checksums with and without the part number. From now on, composite checksums are consistently stored with the part number included.
Additionally, two integration tests are added:
* One verifies the final composite checksum with part numbers.
* Another ensures invalid composite checksums are correctly rejected.
Fixes#1329
Fixes the checksum type/algorithm mismatch error in `CreateMultipartUpload`. The algorithm an type were messed in the error description. It also adds an integration test to target the unsupported checksum type/algorithm pairs.
Fixes#1428
The `x-amz-expected-bucket-owner` header in S3 specifies the account ID of the expected bucket owner. If the account ID provided does not match the actual owner of the bucket, the request fails with an HTTP 403 Forbidden (AccessDenied) error. If the provided account ID is not 12 characters long, S3 returns a 400 Bad Request error.
In our case, we expect the header to contain the bucket owner’s access key ID, and we skip validation errors related to the access key ID, since there is no validation mechanism for user access key IDs. If the provided value does not match the bucket owner’s access key ID, the gateway returns an AccessDenied error.
A few integration tests are added for random actions, as this feature applies to all actions, but it is unnecessary to add test cases for every single one.
Closes#1536
Adds bucket policy version support. Two versions are supported: **2008-10-17** and **2012-10-17**. If the `Version` field is omitted in the bucket policy document, it defaults to **2008-10-17**. However, if an empty string (`""`) is provided, it is considered invalid.
Closes#1566
When an object is locked and bucket versioning is not configured at the gateway level, any object overwrite request should be rejected with an object locked error. The `PutObject` operation already follows this behavior, but `CopyObject` and `CompleteMultipartUpload` were missing this check. This change introduces the locking mechanism for `CopyObject` and `CompleteMultipartUpload` operations.
Fixes#1574
When versioning is enabled at the gateway level and object lock is enabled for a bucket, any overwrite request on a locked object should succeed since it results in the creation of a new object version. This PR fixes the logic by adding a bucket versioning status check in `CheckObjectAccess`.
This change introduces concurrent execution for integration tests. It adds a mechanism to run tests either synchronously or in parallel, controlled by a new flag. By default, tests continue to run in synchronous mode to maintain predictable behavior during local development. In GitHub Actions, the tests are now executed in parallel mode to significantly reduce overall runtime.
The implementation uses a semaphore-based concurrency control to limit the number of parallel test executions and ensures graceful shutdown through context cancellation. This approach improves test performance while keeping the system stable and backward compatible.
Fixes#1565Fixes#1561Fixes#1300
This PR focuses on three main changes:
1. **Prioritizing object-level lock configuration over bucket-level default retention**
When an object is uploaded with a specific retention configuration, it takes precedence over the bucket’s default retention set via `PutObjectLockConfiguration`. If the object’s retention expires, the object must become available for write operations, even if the bucket-level default retention is still active.
2. **Preventing object lock configuration from being disabled once enabled**
To align with AWS S3 behavior, once object lock is enabled for a bucket, it can no longer be disabled. Previously, sending an empty `Enabled` field in the payload would disable object lock. Now, this behavior is removed—an empty `Enabled` field will result in a `MalformedXML` error.
This creates a challenge for integration tests that need to clean up locked objects in order to delete the bucket. To handle this, a method has been implemented that:
* Removes any legal hold if present.
* Applies a temporary retention with a "retain until" date set 3 seconds ahead.
* Waits for 3 seconds before deleting the object and bucket.
3. **Allowing object lock to be enabled on existing buckets via `PutObjectLockConfiguration`**
Object lock can now be enabled on an existing bucket if it wasn’t enabled at creation time.
* If versioning is enabled at the gateway level, the behavior matches AWS S3: object lock can only be enabled when bucket versioning status is `Enabled`.
* If versioning is not enabled at the gateway level, object lock can always be enabled on existing buckets via `PutObjectLockConfiguration`.
* In Azure (which does not support bucket versioning), enabling object lock is always allowed.
This change also fixes the error message returned in this scenario for better clarity.
Fixes#1559Fixes#1330
This PR focuses on three main changes:
1. **Fix object lock error codes and descriptions**
When an object was WORM-protected and delete/overwrite was disallowed due to object lock configurations, the gateway incorrectly returned the `s3.ErrObjectLocked` error code and description. These have now been corrected.
2. **Update `PutObjectRetention` behavior**
Previously, when an object already had a retention mode set, the gateway only allowed modifications if the mode was changed from `GOVERNANCE` to `COMPLIANCE`, and only when the user had the `s3:BypassGovernanceRetention` permission.
The logic has been updated: if the existing retention mode is the same as the one being applied, the operation is now allowed regardless of other factors.
3. **Fix error checks in integration tests (AWS SDK regression)**
Due to an AWS SDK regression, integration tests were previously limited to checking partial error descriptions. This issue seems to be resolved for some actions (though the ticket is still open: https://github.com/aws/aws-sdk-go-v2/issues/2921). Error checks have been reverted back to full description comparisons where possible.
Fixes#1554Fixes#1423
The gateway previously ignored the `x-amz-content-sha256` header for anonymous unsigned requests to public buckets. This PR adds hash calculation for this header and correctly handles special payload types.
It also fixes the case where a signed streaming payload (`STREAMING-AWS4-HMAC-SHA256-PAYLOAD...`) is used with anonymous requests. In this scenario, the gateway now returns a specific "not supported" error, consistent with S3 behavior.
Closes#1525
* Adds validation for the `Content-MD5` header.
* If the header value is invalid, the gateway now returns an `InvalidDigest` error.
* If the value is valid but does not match the payload, it returns a `BadDigest` error.
* Adds integration test cases for `PutBucketCors` with `Content-MD5`.
Fixes#1540Fixes#1538Fixes#1513Fixes#1425
Fixes SigV4 authentication and presigned URL error handling. Adds two sets of errors in the `s3err` package for these authentication mechanisms.
* Adds a check to return a custom "not supported" error when `X-Amz-Security-Token` is present in presigned URLs.
* Adds a check to return a custom "not supported" error when the `AWS4-ECDSA-P256-SHA256` algorithm is used in presigned URLs.
Fixes#1520
Removes the incorrect logic for HeadObject returning successful response, when querying an incomplete multipart upload.
Implements the logic to return `NotImplemented` error if `GetObject`/`HeadObject` is attempted with `partNumber` in azure and posix backends. The front-end part is preserved to be used in s3 proxy backend.
Closes#821
**Implements conditional operations across object APIs:**
* **PutObject** and **CompleteMultipartUpload**:
Supports conditional writes with `If-Match` and `If-None-Match` headers (ETag comparisons).
Evaluation is based on an existing object with the same key in the bucket. The operation is allowed only if the preconditions are satisfied. If no object exists for the key, these headers are ignored.
* **CopyObject** and **UploadPartCopy**:
Adds conditional reads on the copy source object with the following headers:
* `x-amz-copy-source-if-match`
* `x-amz-copy-source-if-none-match`
* `x-amz-copy-source-if-modified-since`
* `x-amz-copy-source-if-unmodified-since`
The first two are ETag comparisons, while the latter two compare against the copy source’s `LastModified` timestamp.
* **AbortMultipartUpload**:
Supports the `x-amz-if-match-initiated-time` header, which is true only if the multipart upload’s initialization time matches.
* **DeleteObject**:
Adds support for:
* `If-Match` (ETag comparison)
* `x-amz-if-match-last-modified-time` (LastModified comparison)
* `x-amz-if-match-size` (object size comparison)
Additionally, this PR updates precondition date parsing logic to support both **RFC1123** and **RFC3339** formats. Dates set in the future are ignored, matching AWS S3 behavior.
Closes#882
Implements conditional reads for `GetObject` and `HeadObject` in the gateway for both POSIX and Azure backends. The behavior is controlled by the `If-Match`, `If-None-Match`, `If-Modified-Since`, and `If-Unmodified-Since` request headers, where the first two perform ETag comparisons and the latter two compare against the object’s `LastModified` date. No validation is performed for invalid ETags or malformed date formats, and precondition date headers are expected to follow RFC1123; otherwise, they are ignored.
The Integration tests cover all possible combinations of conditional headers, ensuring the feature is 100% AWS S3–compatible.
GetObject allows overriding response headers with the following
paramters:
response-cache-control
response-content-disposition
response-content-encoding
response-content-language
response-content-type
response-expires
This is only valid for signed (and pre-singed) requests. An error
is returned for anonymous requests if these are set.
More info on the GetObject overrides can be found in the GetObject
API reference.
This also clarifies the naming of the AccessOptions IsPublicBucket
to IsPublicRequest to indicate this is a public access request
and not just accessing a bucket that allows public access.
Fixes#1501
GetBucketLocation is being deprecated by AWS, but is still used
by some clients. We don't need any backend handlers for this since
the region is managed by the frontend. All we need is to test for
bucket existence, so we can use HeadBucket for this.
Fixes#1499
Fixes#1488
Adds full wildcard (`*`) and single-character (`?`) support for bucket policy actions, fixes resource detection with wildcards, and includes unit tests for `bucket_policy_actions`, `bucket_policy_effect`, and `bucket_policy_principals`.
This changes the marker/continuation token from the object name
to the marker from the azure list objects pager. This is needed
because passing the object name as the token to the azure next
call causes the Azure API to throw 400 Bad Request with
InvalidQueryParameterValue. So we have to use the azure marker
for compatibility with the azure API pager.
To do this we have to align the s3 list objects request to the
Azure ListBlobsHierarchyPager. The v2 requests have an optional
startafter where we will have to page through the azure blobs
to find the correct starting point, but after this we will
only return with the single paginated results form the Azure
pager to maintain the correct markers all the way through to
Azure.
The ListObjects (non V2) assumes that the marker must be an object
name, so for this case we have to page through the azure listings
for each call to find the correct starting point. This makes the
V2 method far more efficient, but maintains correctness for the
ListObjects.
Also remove continuation token string checks in the integration
tests since this is supposed to be an opaque token that the
client should not care about. This will help to maintain the
tests for mutliple backend types.
Fixes#1457
Closes#1454
Adds the implementation of [S3 GetBucketPolicyStatus action](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketPolicyStatus.html). The implementation goes to front-end. Front-End loads the bucket policy and checks if it grants public access to all users.
A bucket policy document `is public` only when `Principal` contains `*`(all users): only when it grants access to `ALL` users.
