Closes#1731
Implements the admin `CreateBucket` (`PATCH /:bucket/create`) endpoint and CLI command, which create a new bucket with the provided owner access key ID. The endpoint internally calls the S3 `CreateBucket` API, storing the new owner information in the request context under the `bucket-owner` key. This value is then retrieved by the S3 API layer and the backends.
The endpoint uses the custom `x-vgw-owner` HTTP header to pass the bucket owner access key ID.
The admin CLI command mirrors `aws s3api create-bucket` and supports all flags implemented by the gateway (for example, `--create-bucket-configuration`, `--acl`, `--object-ownership`, etc.).
Fixes#1733Fixes#1734
The `x-amz-object-lock-retain-until-date` request header appears in the `PutObject`, `CopyObject`, and `CreateMultipartUpload` operations. This PR fixes the two types of error codes and messages returned when the header value is invalid or expired and adds the corresponding integration tests.
The `x-amz-if-match-size` parsing debug log used to appear for all `DeleteObject` calls when the header was missing. An empty-string check was missing, which led to attempting to parse an empty string into an `int64`, causing a failure and triggering the debug log. This check has now been added, and the debug log is emitted only when the header is present and contains an invalid `int64` value.
Closes#1714
There is a `Location` field in the `CompleteMultipartUpload` result that represents the newly created object URL. This PR adds this property to the `CompleteMultipartUpload` response, generating it dynamically in either host-style or path-style format, depending on the gateway configuration.
Fixes#1710
The `If-Match` and `If-None-Match` precondition header values represent object ETags. ETags are generally quoted; however, S3 evaluates precondition headers equivalently whether the ETag is quoted or not, comparing only the underlying value and ignoring the quotes if present.
The new implementation trims quotes from the ETag in both the input precondition header and the object metadata, ensuring that comparisons are performed purely on the ETag value and are insensitive to quoting.
Fixes#1683Fixes#1684Fixes#1685Fixes#1690Fixes#1691Fixes#1692Fixes#1694Fixes#1695
This PR primarily focuses on error handling and checksum calculation for signed streaming requests of type `STREAMING-AWS4-HMAC-SHA256-PAYLOAD` and `STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER`.
It fixes the error type returned when the payload encoding is incorrect: the correct `IncompleteBody` error is now returned.
Chunk size validation has been added, enforcing the rule that only the final chunk may be smaller than 8192 bytes.
The `x-amz-trailer` header value is now validated against the checksum trailer present in the payload. For `STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER` requests, if no checksum trailer is provided in the payload, the gateway now returns an `IncompleteBody` error.
If there is a mismatch between the `x-amz-trailer` header and the checksum trailer in the payload, or if the checksum header key in the payload is invalid, a `MalformedTrailer` error is returned.
The `x-amz-decoded-content-length` header value is now compared against the actual decoded payload length, and an `IncompleteBody` error is returned if there is a mismatch.
Finally, the double checksum calculation issue has been fixed. For `STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER` requests, the trailing checksum is now parsed from the request payload and stored in the backend, instead of being recalculated by the backend.
Fixes#1676
`x-amz-decoded-content-length` in streaming uploads specifies the number of actual data-payload bytes, with encoding characters removed. If the value does not match the actual payload after decoding, now an `IncompleteBody` error is returned.
Fixes#1665
S3 enforces a validation rule for unsigned streaming payload trailer uploads: all chunk sizes must be greater than 8192 bytes except for the final chunk.
This fix adds a check in the unsigned chunk reader that validates chunk sizes by comparing each chunk size to the previous one.
Fixes#1666Fixes#1660
Unsigned streaming payload trailers have strict encoding rules for the request body. If the body isn’t encoded correctly, the expected `IncompleteBody` API error is now returned. Incorrect encoding includes things like invalid chunk sizes, missing delimiters, or malformed `\r\n` sequences.
Fixes#1600Fixes#1603Fixes#1607Fixes#1626Fixes#1632Fixes#1652Fixes#1653Fixes#1656Fixes#1657Fixes#1659
This PR focuses mainly on unsigned streaming payload **trailer request payload parsing** and **checksum calculation**. For streaming uploads, there are essentially two ways to specify checksums:
1. via `x-amz-checksum-*` headers,
2. via `x-amz-trailer`,
or none — in which case the checksum should default to **crc64nvme**.
Previously, the implementation calculated the checksum only from `x-amz-checksum-*` headers. Now, `x-amz-trailer` is also treated as a checksum-related header and indicates the checksum algorithm for streaming requests. If `x-amz-trailer` is present, the payload must include a trailing checksum; otherwise, an error is returned.
`x-amz-trailer` and any `x-amz-checksum-*` header **cannot** be used together — doing so results in an error.
If `x-amz-sdk-checksum-algorithm` is specified, then either `x-amz-trailer` or one of the `x-amz-checksum-*` headers must also be present, and the algorithms must match. If they don’t, an error is returned.
The old implementation used to return an internal error when no `x-amz-trailer` was received in streaming requests or when the payload didn’t contain a trailer. This is now fixed.
Checksum calculation used to happen twice in the gateway (once in the chunk reader and once in the backend). A new `ChecksumReader` is introduced to prevent double computation, and the trailing checksum is now read by the backend from the chunk reader. The logic for stacking `io.Reader`s in the Fiber context is preserved, but extended: once a `ChecksumReader` is stacked, all following `io.Reader`s are wrapped with `MockChecksumReader`, which simply delegates to the underlying checksum reader. In the backend, a simple type assertion on `io.Reader` provides the necessary checksum metadata (algorithm, value, etc.).
Fixes#1630
S3 returns `InvalidArgument: Invalid version id specified` for invalid version IDs in object-level actions that accept `versionId` as a query parameter. The `versionId` in S3 follows a specific structure, and if the input string doesn’t match this structure, the error is returned. In the gateway, the `versionId` is generated using the `ulid` package, which also has a defined structure. This PR adds validation for object-level operations that work with object versions by using the ULID parser.
These actions include: `HeadObject`, `GetObject`, `PutObjectTagging`, `GetObjectTagging`, `DeleteObjectTagging`, `PutObjectLegalHold`, `GetObjectLegalHold`, `PutObjectRetention`, `GetObjectRetention`, `DeleteObject`, `CopyObject`, `UploadPartCopy`, and `GetObjectAttributes`.
Fixes#1598
`PutObject` and `UploadPart` accept x-amz-checksum-* calculated checksum headers and `x-amz-sdk-checksum-algorithm`. If the checksum algorithm specified in sdk algorithm doesn't match the one in x-amz-checksum-*, it now returns the correct error message: `Value for x-amz-sdk-checksum-algorithm header is invalid.`.
Fixes#1482
The metadata keys should always be converted to lowercase in `PutObject`, `CreateMultipartUpload`, and `CopyObject`. This implementation converts the metadata keys to lowercase in the front end, ensuring they are stored in lowercase in the backend.
Closes#1549Fixes#1593Fixes#1521Fixes#1427Fixes#1311Fixes#1301Fixes#1040
This PR primarily focuses on checksum calculation within the gateway, but it also includes several related fixes and improvements.
It introduces a middleware responsible for handling and calculating checksums for the `x-amz-checksum-*` headers and `Content-MD5`. The middleware is applied only to actions that expect a request body or checksum headers. It also enforces validation for actions that require a non-empty request body, returning an error if the body is missing. Similarly, it returns an error for actions where at least one checksum header (`Content-MD5` or `x-amz-checksum-*`) is required but none is provided.
The implementation is based on [https://gist.github.com/niksis02/eec3198f03e561a0998d67af75c648d7](the reference table), tested directly against S3:
It also fixes the error case where the `x-amz-sdk-checksum-algorithm` header is present but no corresponding `x-amz-checksum-*` or `x-amz-trailer` header is included.
Additionally, the PR improves validation for the `x-amz-content-sha256` header. For actions that require this header, an error is now returned when it’s missing. For actions that don’t require it, the middleware no longer enforces its presence. Following the common S3 pattern, the header remains mandatory for admin routes.
Finally, the `x-amz-content-sha256` header is now optional for anonymous requests, as it is not required in that case.
Fixes#1579
S3 enforces a specific rule for validating bucket and object tag key/value names. This PR integrates the regexp pattern used by S3 for tag validation.
Official S3 documentation for tag validation rules: [AWS S3 Tag](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_Tag.html)
There are two types of tagging inputs for buckets and objects:
1. **On existing buckets/objects** — used in the `PutObjectTagging` and `PutBucketTagging` actions, where tags are provided in the request body.
2. **On object creation** — used in the `PutObject`, `CreateMultipartUpload`, and `CopyObject` actions, where tags are provided in the request headers and must be URL-encoded.
This implementation ensures correct validation for both types of tag inputs.
Fixes#1359
The composite checksums in **CompleteMultipartUpload** generally follow the format `checksum-<number_of_parts>`. Previously, the gateway treated composite checksums as regular checksums without distinguishing between the two formats.
In S3, the `x-amz-checksum-*` headers accept both plain checksum values and the `checksum-<number_of_parts>` format. However, after a successful `CompleteMultipartUpload` request, the final checksum is always stored with the part number included.
This implementation adds support for parsing both formats—checksums with and without the part number. From now on, composite checksums are consistently stored with the part number included.
Additionally, two integration tests are added:
* One verifies the final composite checksum with part numbers.
* Another ensures invalid composite checksums are correctly rejected.
Fixes#1484
Removes response header name normalization to prevent Fiber from converting them to camel case. Also fixes the `HeadBucket` response headers by changing their capital letters to lowercase and corrects the `x-amz-meta` headers to use lowercase instead of camel case.
Some systems may choose to allow non-aws compliant bucket names
and/or handle the bucket naem validation in the backend instead.
This adds the option to turn off the strict bucket name validation
checks in the frontend API handlers.
When frontend bucket name validation is disabled, we need to do
sanity checks for posix compliant names in the posix/scoutfs
backends. This is automatically enabled when strict bucket
name validation is disabled.
Fixes#1564
Fiber includes a built-in panic recovery middleware that catches panics in route handlers and middlewares, preventing the server from crashing and allowing it to recover. Alongside this, a stack trace handler has been implemented to store system panics in the context locals (stack).
Both the S3 API server and the Admin server use a global error handler to catch unexpected exceptions and recovered panics. The middleware’s logic is to log the panic or internal error and return an S3-style internal server error response.
Additionally, dedicated **Panic** and **InternalError** loggers have been added to the `s3api` debug logger to record system panics and internal errors in the console.
Fixes#1554Fixes#1423
The gateway previously ignored the `x-amz-content-sha256` header for anonymous unsigned requests to public buckets. This PR adds hash calculation for this header and correctly handles special payload types.
It also fixes the case where a signed streaming payload (`STREAMING-AWS4-HMAC-SHA256-PAYLOAD...`) is used with anonymous requests. In this scenario, the gateway now returns a specific "not supported" error, consistent with S3 behavior.
Closes#1525
* Adds validation for the `Content-MD5` header.
* If the header value is invalid, the gateway now returns an `InvalidDigest` error.
* If the value is valid but does not match the payload, it returns a `BadDigest` error.
* Adds integration test cases for `PutBucketCors` with `Content-MD5`.
Fixes#1540Fixes#1538Fixes#1513Fixes#1425
Fixes SigV4 authentication and presigned URL error handling. Adds two sets of errors in the `s3err` package for these authentication mechanisms.
* Adds a check to return a custom "not supported" error when `X-Amz-Security-Token` is present in presigned URLs.
* Adds a check to return a custom "not supported" error when the `AWS4-ECDSA-P256-SHA256` algorithm is used in presigned URLs.
Closes#821
**Implements conditional operations across object APIs:**
* **PutObject** and **CompleteMultipartUpload**:
Supports conditional writes with `If-Match` and `If-None-Match` headers (ETag comparisons).
Evaluation is based on an existing object with the same key in the bucket. The operation is allowed only if the preconditions are satisfied. If no object exists for the key, these headers are ignored.
* **CopyObject** and **UploadPartCopy**:
Adds conditional reads on the copy source object with the following headers:
* `x-amz-copy-source-if-match`
* `x-amz-copy-source-if-none-match`
* `x-amz-copy-source-if-modified-since`
* `x-amz-copy-source-if-unmodified-since`
The first two are ETag comparisons, while the latter two compare against the copy source’s `LastModified` timestamp.
* **AbortMultipartUpload**:
Supports the `x-amz-if-match-initiated-time` header, which is true only if the multipart upload’s initialization time matches.
* **DeleteObject**:
Adds support for:
* `If-Match` (ETag comparison)
* `x-amz-if-match-last-modified-time` (LastModified comparison)
* `x-amz-if-match-size` (object size comparison)
Additionally, this PR updates precondition date parsing logic to support both **RFC1123** and **RFC3339** formats. Dates set in the future are ignored, matching AWS S3 behavior.
The following panic was triggered when mc client (that uses
chunked uploads) would upload a 171164 byte file. This likely
could have been hit with other sizes as well, but this size
was able to reliably reproduce the issue.
panic: runtime error: slice bounds out of range [:2] with capacity 1
goroutine 66 [running]:
github.com/versity/versitygw/s3api/utils.(*ChunkReader).parseChunkHeaderBytes(0x14000276200, {0x14000167fff?, 0x14000103180?, 0x200000003?})
versitygw/s3api/utils/signed-chunk-reader.go:372 +0xe54
github.com/versity/versitygw/s3api/utils.(*ChunkReader).parseAndRemoveChunkInfo(0x14000276200, {0x14000167fff, 0x1, 0x1})
versitygw/s3api/utils/signed-chunk-reader.go:251 +0x50
github.com/versity/versitygw/s3api/utils.(*ChunkReader).Read(0x14000276200, {0x14000160000, 0x14000056c00?, 0x8000})
versitygw/s3api/utils/signed-chunk-reader.go:126 +0x188
io.(*teeReader).Read(0x140000b09c0, {0x14000160000, 0x105e7b368?, 0x8000})
/usr/local/go/src/io/io.go:628 +0x34
...
The reproducer is:
% truncate -s 171764 testfile
% mc cp testfile gwtest/mybucket/testfile
mc: <ERROR> Failed to copy `/Users/ben/repo/s3perf/tools/testfile`. Put "http://127.0.0.1:7070/mybucket/testfile": dial tcp 127.0.0.1:7070: connect: connection refused
The panic can happen because the capacity of header ([]byte) at
the point of the debuglog line can be less than 2, but we were
trying to always send the first 2 bytes to the debug log.
The debuglogger should be a top level module since we expect
all modules within the project to make use of this. If its
hidden in s3api, then contributors are less likely to make
use of this outside of s3api.
Closes#882
Implements conditional reads for `GetObject` and `HeadObject` in the gateway for both POSIX and Azure backends. The behavior is controlled by the `If-Match`, `If-None-Match`, `If-Modified-Since`, and `If-Unmodified-Since` request headers, where the first two perform ETag comparisons and the latter two compare against the object’s `LastModified` date. No validation is performed for invalid ETags or malformed date formats, and precondition date headers are expected to follow RFC1123; otherwise, they are ignored.
The Integration tests cover all possible combinations of conditional headers, ensuring the feature is 100% AWS S3–compatible.
GetObject allows overriding response headers with the following
paramters:
response-cache-control
response-content-disposition
response-content-encoding
response-content-language
response-content-type
response-expires
This is only valid for signed (and pre-singed) requests. An error
is returned for anonymous requests if these are set.
More info on the GetObject overrides can be found in the GetObject
API reference.
This also clarifies the naming of the AccessOptions IsPublicBucket
to IsPublicRequest to indicate this is a public access request
and not just accessing a bucket that allows public access.
Fixes#1501
Fixes#1345
The previous implementation incorrectly parsed the `x-amz-sdk-checksum-algorithm` header for the `CompleteMultipartUpload` operation, even though this header is not expected and should be ignored. It also mistakenly treated the `x-amz-checksum-algorithm` header as an invalid value for `x-amz-checksum-x`.
The updated implementation only parses the `x-amz-sdk-checksum-algorithm` header for `PutObject` and `UploadPart` operations. Additionally, `x-amz-checksum-algorithm` and `x-amz-checksum-type` headers are now correctly ignored when parsing the precalculated checksum headers (`x-amz-checksum-x`).
Fixes#1339
`x-amz-checksum-type` and `x-amz-checksum-algorithm` request headers should be case insensitive in `CreateMultipartUpload`.
The changes include parsing the header values to upper case before validating and passing to back-end. `x-amz-checksum-type` response header was added in`CreateMultipartUpload`, which was missing before.
Fixes#1352
Adds a validation check step in `SigV4` authentication for `x-amz-content-sh256` to check it to be either a valid sha256 hash or a special payload type(UNSIGNED-PAYLOAD, STREAMING-UNSIGNED-PAYLOAD-TRAILER...).
Fixes#1388Fixes#1389Fixes#1390Fixes#1401
Adds the `x-amz-copy-source` header validation for `CopyObject` and `UploadPartCopy` in front-end.
The error:
```
ErrInvalidCopySource: {
Code: "InvalidArgument",
Description: "Copy Source must mention the source bucket and key: sourcebucket/sourcekey.",
HTTPStatusCode: http.StatusBadRequest,
},
```
is now deprecated.
The conditional read/write headers validation in `CopyObject` should come with #821 and #822.
Closes#908
This PR introduces a new routing system integrated with Fiber. It matches each S3 action to a route using middleware utility functions (e.g., URL query match, request header match). Each S3 action is mapped to a dedicated route in the Fiber router. This functionality cannot be achieved using standard Fiber methods, as Fiber lacks the necessary tooling for such dynamic routing.
Additionally, this PR implements a generic response handler to manage responses from the backend. This abstraction helps isolate the controller from the data layer and Fiber-specific response logic.
With this approach, controller unit testing becomes simpler and more effective.
An update to fasthttp has deprecated the VisitAll() method
for an iterator function All() that can be used to range over
all headers.
This should fix the staticcheck warnings for calling the
deprecated function.
The CRC32, CRC32c, and CRC64NVME data integrity checksums support calculating
the composite full object values for multi-part uploads using the checksum
and length of the individual parts.
Previously, we were reading all of the part data to recalculate the full
object checksum values during the complete multipart upload call. This
disabled the optimized copy_file_range() for certain filesystems such as
XFS because the part data was being read. If the data is not read, and
the file handle is passed directly to io.Copy(), then the filesystem is
allowed to optimize the copying of the data from the source to destination
files.
This now allows both the optimized copy_file_range() optimizations as well
as the data integrity features enabled for support composite checksum types.
This implementation introduces **public buckets**, which are accessible without signature-based authentication.
There are two ways to grant public access to a bucket:
* **Bucket ACLs**
* **Bucket Policies**
Only `Get` and `List` operations are permitted on public buckets. All **write operations** require authentication, regardless of whether public access is granted through an ACL or a policy.
The implementation includes an `AuthorizePublicBucketAccess` middleware, which checks if public access has been granted to the bucket. If so, authentication middlewares are skipped. For unauthenticated requests, appropriate errors are returned based on the specific S3 action.
---
**1. Bucket-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test"
}
]
}
```
**2. Object-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test/*"
}
]
}
```
**3. Both Bucket and Object-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test"
},
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test/*"
}
]
}
```
---
```sh
aws s3api create-bucket --bucket test --object-ownership BucketOwnerPreferred
aws s3api put-bucket-acl --bucket test --acl public-read
```
This adds an object name validation util to check if the object
path would resolve to a path outside of the bucket directory.
S3 returns Bad Request for these type of paths:
% aws s3api put-object --bucket mybucket --key test/../../hello
An error occurred (400) when calling the PutObject operation: Bad Request
Closes#803
Implements host-style bucket addressing in the gateway. This feature can be enabled by running the gateway with the `--virtual-domain` flag and specifying a virtual domain name.
Example:
```bash
./versitygw -a user -s secret --virtual-domain localhost:7070 posix /tmp/vgw
```
The implementation follows this approach: it introduces a middleware (`HostStyleParser`) that parses the bucket name from the `Host` header and appends it to the URL path. This effectively transforms the request into a path-style bucket addressing format, which the gateway already supports. With this design, the gateway can handle both path-style and host-style requests when running in host-style mode.
For local testing, one can either set up a local DNS server to wildcard-match all subdomains of a specified domain and resolve them to the local IP address, or manually add entries to `/etc/hosts` to resolve bucket-prefixed hosts to the server IP (e.g., `127.0.0.1`).
