Previously, if no object checksum type/algorithm was specified when initiating a multipart upload, the CompleteMultipartUpload request would compute the final object’s CRC64NVME checksum but not persist it. This logic has now been fixed, and in the scenario described above the checksum is stored on the final object. There should no longer be any case where a CompleteMultipartUpload request finishes without persisting the final object checksum.
Add data-integrity checksum support in `PutObject` in the POSIX backend for directory objects. Since the only way to upload a directory object is via `PutObject`, this logic validates and stores the checksum of the empty payload. Support for `GetObject` has also been added to retrieve and return directory-object checksums.
Fixes#1870Fixes#1863
A validation has been added to **PutBucketCors** for `CORSRule.AllowedOrigins`. The `AllowedOrigins` list can no longer be empty—otherwise a **MalformedXML** error is returned. Additionally, each origin is now validated to ensure it does not contain more than one wildcard.
A similar validation has been added for `AllowedMethods`. The list must not be empty, or a **MalformedXML** error is returned. Previously, empty method values (e.g., `[]string{""}`) were incorrectly treated as valid. This has been fixed, and an **UnsupportedCORSMethod** error is now returned.
Fixes#1869
Generally, when object ownership is not explicitly specified during bucket creation, it defaults to `BucketOwnerEnforced`. With `BucketOwnerEnforced`, ACLs are disabled and any attempt to set one results in an `InvalidBucketAclWithObjectOwnership` error.
However, there is an edge case. When the `private` canned ACL is used during bucket creation—which is effectively the default ACL for all buckets—`BucketOwnerEnforced` is still permitted. Moreover, if no explicit object ownership is specified together with the `private` canned ACL, the ownership defaults to `BucketOwnerPreferred`.
This fix also resolves the issue with rclone bucket creation, since rclone sends `x-amz-acl: private` by default:
```
rclone mkdir vgw:test
```
Fixes#1849
If no `Content-Type` is provided during object upload, S3 defaults it to `application/octet-stream`. This behavior was missing in the gateway, causing backends to persist an empty `Content-Type`, which Fiber then overrides with its default `text/plain`. The behavior has now been corrected for the object upload operations: `PutObject`, `CreateMultipartUpload`, and `CopyObject`.
The latest version of the go sdk has deprecate the s3 manager in
favor of the new transfermanager. This updates to the new
transfermanager functionality for the two places we were using
the manager, iam s3 and integration tests.
This also removes the pinning of nats nkeys since they have fixed
the tags in their repo now. And general cleanup of the go.mod.
Fixes#1835
If-Match in DeleteObject is a precondition header that compares the client-provided ETag with the server-side ETag before deleting the object. Previously, the comparison failed when the client sent an unquoted ETag, because server ETags are stored with quotes. The implementation now trims quotes from both the input ETag and the server ETag before comparison to avoid mismatches. Both quoted and unquoted ETags are valid according to S3.
CopyObject was failing with NoSuchKey when source keys contained special
characters like {} or spaces. The X-Amz-Copy-Source header is URL-encoded
by clients, but ParseCopySource wasn't decoding before filesystem access.
Added url.QueryUnescape() to properly decode bucket and object names,
fixing copy operations for keys with special characters.
Fixing this also uncovered an errors with azure blob url encoding with
similar special character handling. Added this fix in for the integration
tests to pass.
Fixes#1832Fixes#1637
Fixes#1809Fixes#1806Fixes#1804Fixes#1794
This PR focuses on correcting so-called "list-limiter" parsing and validation. The affected limiters include: `max-keys`, `max-uploads`, `max-parts`, `max-buckets`, `max-uploads` and `part-number-marker`. When a limiter value is outside the integer range, a specific `InvalidArgument` error is now returned. If the value is a valid integer but negative, a different `InvalidArgument` error is produced.
`max-buckets` has its own validation rules: completely invalid values and values outside the allowed range (`1 <= input <= 10000`) return distinct errors. For `ListObjectVersions`, negative `max-keys` values follow S3’s special-case behavior and return a different `InvalidArgument` error message.
Additionally, `GetObjectAttributes` now follows S3 semantics for `x-amz-max-parts`: S3 ignores invalid values, so the gateway now matches that behavior.
Fixes#1792Fixes#1747Fixes#1797Fixes#1799
This PR primarily introduces delimiter support and several bug fixes for the `ListMultipartUploads` action in the POSIX and Azure backends. Delimiter handling is now implemented — when a delimiter is present in multipart-upload object key names, the backend collects and returns the appropriate common prefixes.
This functionality is achieved by introducing a common multipart-upload lister in the backend package. All backends (Azure, POSIX) now use this lister. The lister accepts a list that is already sorted and filtered by `KeyMarker` and `Prefix`.
Previously, the `KeyMarker` was required to exactly match an existing multipart-upload object key. This restriction is removed. The listing now relies on a lexicographical comparison between the provided `KeyMarker` and existing multipart-upload object keys.
Validation for `UploadIdMarker` is also added to correctly return an `InvalidArgument` error for invalid upload IDs. If `KeyMarker` is missing, the `UploadIdMarker` is ignored entirely. If `KeyMarker` is provided, a valid upload ID is one that matches an upload belonging to *the first object key after the KeyMarker*. For example, if the `KeyMarker` is `foo`, but the provided `UploadIdMarker` corresponds to an upload under `quxx`, it is invalid. It must match one of the uploads for the next object key equal to `foo`.
Finally, this PR fixes multipart-upload sorting. Multipart uploads must be sorted primarily lexicographically by their object key, and secondarily—when multiple uploads share the same object key—by their initiation time in ascending order.
Fixes#1767Fixes#1773
As object ACLs are not supported in the gateway, any attempt to set an ACL during object creation must return a NotImplemented error. A check has now been added to `PutObject`, `CopyObject`, and `CreateMultipartUpload` to detect any ACL-related headers and return a NotImplemented error accordingly.
Fixes#1766Fixes#1750
This PR focuses on two bug fixes:
First, it blocks access to delete `DeleteMarkers` for the following operations by returning a `MethodNotAllowed` error: `PutObjectTagging`, `GetObjectTagging`, `DeleteObjectTagging`, `PutObjectLegalHold`, `GetObjectLegalHold`, `PutObjectRetention`, and `GetObjectRetention`.
Second, it removes the access check that previously prevented deleting a delete marker locked by a bucket default retention rule. A delete marker should always be allowed to be deleted.
In the `SignedStreamingPayloadTrailer_success` integration test, the signing date was not using UTC. This caused an incorrect string-to-sign and signature calculation, because the timestamp used for `x-amz-date` (which is always UTC) differed from the timestamp used for streaming payload signature generation. The test now uses UTC, resolving the issue that occurred when the local time zone differed from UTC, specifically in terms of the `yyyymmdd` date component.
Fixes#1757Fixes#1758
When attempting to delete a non-existing object in a versioning-enabled bucket while specifying a `versionId`, VersityGW previously returned an internal error if the object had a parent file object, and an `InvalidArgument` error if the object did not exist. This PR fixes both behaviors and now returns a successful response that includes the `versionId`.
`getBucketName` in the integration test utilities is responsible for generating unique bucket names using the `test-bucket-` prefix and an atomic integer. The previous implementation performed an atomic `Add` followed by a `Load`, which does not guarantee uniqueness and could result in duplicate bucket names. This has been fixed by removing the `Load` call and relying solely on the return value of the `Add` operation, which provides the updated integer value.
Fixes#1751
When an object lock–related operation is performed on an object in a bucket where Object Lock is not enabled, an `InvalidRequest` error is returned; however, the error message differs for some actions. This PR introduces a new error, `ErrMissingObjectLockConfigurationNoSpaces`, for `PutObject`, `CopyObject`, and `CreateMultipartUpload` to maintain compatibility with S3 in terms of the error message. It also adds the missing integration tests for these actions.
Fixes#1741
An object delete request without a `versionId` results in the creation of a new delete marker in versioning-enabled buckets. Even if the latest object version is locked, a new delete marker must still be created.
This implementation skips the object lock check for delete requests in versioning-enabled buckets when the `versionId` is missing, allowing the delete marker to be created as expected.
Additionally, it introduces a flag in the `createObjVersion` method in POSIX to remove unnecessary xattr attributes from an object after creating a new object version. A delete marker must not carry object-specific attributes such as tagging, legal hold, or retention. Currently, the cleanup is limited to legal hold and retention attributes, but this list will be expanded after fixing issue #1751.
Closes#1731
Implements the admin `CreateBucket` (`PATCH /:bucket/create`) endpoint and CLI command, which create a new bucket with the provided owner access key ID. The endpoint internally calls the S3 `CreateBucket` API, storing the new owner information in the request context under the `bucket-owner` key. This value is then retrieved by the S3 API layer and the backends.
The endpoint uses the custom `x-vgw-owner` HTTP header to pass the bucket owner access key ID.
The admin CLI command mirrors `aws s3api create-bucket` and supports all flags implemented by the gateway (for example, `--create-bucket-configuration`, `--acl`, `--object-ownership`, etc.).
Fixes#1733Fixes#1734
The `x-amz-object-lock-retain-until-date` request header appears in the `PutObject`, `CopyObject`, and `CreateMultipartUpload` operations. This PR fixes the two types of error codes and messages returned when the header value is invalid or expired and adds the corresponding integration tests.
There is some desire to have a web dashboard for the gateway. So
that we dont have to proxy all requests through the webserver
and expose credentials over the wire, the better approach would
be to enable CORS headers to allow browser requests directly to
the s3/admin service.
The default for these headers is off, so that they are only
enabled for instances that specfically want to support this
workload.
Fixes#1643
`GetBucketLocation` in S3 returns empty `LocationConstraint` if the bucket is in the `us-east-1` region. This fix returns empty `LocationConstraint` if the gateway region is `us-east-1`.
Fixes#1654Fixes#1644
CreateBucket `LocationConstraint` rejects empty values with an `InvalidLocationConstraint` error.
The `us-east-1` `LocationConstraint` is considered invalid because it is the default value and must not be present in the `CreateBucketConfiguration` request body.
This PR fixes both issues by returning `InvalidLocationConstraint` in both cases.
Fixes#1688
S3 returns a specific `InvalidRequest` error for the requests with `?versions` query param against `object` resource. The PR implements this error route.
Fixes#1707
The `Expect` HTTP header is ignored by the AWS SDK SigV4 signer and is omitted during signature calculation. As a result, the signature is computed incorrectly when the `Expect` header is included in the signed headers. This PR removes the `Expect` header from the SigV4 ignored headers list in the SDK-derived source code.
Fixes#1698
`PutObjectTagging`, `GetObjectTagging` and `DeleteObjectTagging` return the `x-amz-version-id` in the response headers. The PR adds this header in the responses.
Fixes#1686
GetObjectTagging previously returned a `NoSuchTagSet` error when no object tags were set. This has been fixed, and an empty tag set is now returned instead.
Fixes#1708
This PR focuses on evaluating the `x-amz-if-none-match` precondition header for object PUT operations. If any value other than `*` is provided, a `NotImplemented` error is returned. If `If-Match` is used together with `If-None-Match`, regardless of the value combination, a `NotImplemented` error is returned. When only `If-None-Match: *` is specified, a `PreconditionFailed` error is returned if the object already exists in `PutObject` or `CompleteMultipartUpload`; if the object does not exist, object creation is allowed.
Fixes#1709
If any precondition header is present(`If-Match`, `If-None-Match`) in `PutObject` and `CompleteMultipartUpload` and there's no object in the bucket with the given key, a `NoSuchKey` error is now returned. Previously the headers were simply ignored and new object creation was allowed.
Fixes#1712
A successful `PutBucketPolicy` call returns `204`(No Content) http status. This PR fixes the response status code, by sending correct `204`, insted of `200`.
Closes#1714
There is a `Location` field in the `CompleteMultipartUpload` result that represents the newly created object URL. This PR adds this property to the `CompleteMultipartUpload` response, generating it dynamically in either host-style or path-style format, depending on the gateway configuration.
Fixes#1710
The `If-Match` and `If-None-Match` precondition header values represent object ETags. ETags are generally quoted; however, S3 evaluates precondition headers equivalently whether the ETag is quoted or not, comparing only the underlying value and ignoring the quotes if present.
The new implementation trims quotes from the ETag in both the input precondition header and the object metadata, ensuring that comparisons are performed purely on the ETag value and are insensitive to quoting.
Fixes#1601
Unsigned streaming upload trailers (`STREAMING-UNSIGNED-PAYLOAD-TRAILER`) is allowed only for `PutObject` and `UploadPart`. For all other actions, the gateway now returns an `InvalidRequest` error for the `x-amz-content-sha256` header.
Fixes#1676
`x-amz-decoded-content-length` in streaming uploads specifies the number of actual data-payload bytes, with encoding characters removed. If the value does not match the actual payload after decoding, now an `IncompleteBody` error is returned.
Fixes#1665
S3 enforces a validation rule for unsigned streaming payload trailer uploads: all chunk sizes must be greater than 8192 bytes except for the final chunk.
This fix adds a check in the unsigned chunk reader that validates chunk sizes by comparing each chunk size to the previous one.
niksis02
Ben McClelland