Fixes#1428
The `x-amz-expected-bucket-owner` header in S3 specifies the account ID of the expected bucket owner. If the account ID provided does not match the actual owner of the bucket, the request fails with an HTTP 403 Forbidden (AccessDenied) error. If the provided account ID is not 12 characters long, S3 returns a 400 Bad Request error.
In our case, we expect the header to contain the bucket owner’s access key ID, and we skip validation errors related to the access key ID, since there is no validation mechanism for user access key IDs. If the provided value does not match the bucket owner’s access key ID, the gateway returns an AccessDenied error.
A few integration tests are added for random actions, as this feature applies to all actions, but it is unnecessary to add test cases for every single one.
Fixes#1484
Removes response header name normalization to prevent Fiber from converting them to camel case. Also fixes the `HeadBucket` response headers by changing their capital letters to lowercase and corrects the `x-amz-meta` headers to use lowercase instead of camel case.
Closes#1536
Adds bucket policy version support. Two versions are supported: **2008-10-17** and **2012-10-17**. If the `Version` field is omitted in the bucket policy document, it defaults to **2008-10-17**. However, if an empty string (`""`) is provided, it is considered invalid.
Implements graceful shutdown for the admin and s3api servers. They are shut down before other components (IAM, s3logger, etc.) to allow the servers to properly handle any pending requests while dependencies are still active. The shutdown process is controlled by a context with a 10-second timeout. If it exceeds this duration, all remaining requests are forcefully terminated and the servers are closed.
Closes#1566
When an object is locked and bucket versioning is not configured at the gateway level, any object overwrite request should be rejected with an object locked error. The `PutObject` operation already follows this behavior, but `CopyObject` and `CompleteMultipartUpload` were missing this check. This change introduces the locking mechanism for `CopyObject` and `CompleteMultipartUpload` operations.
Some systems may choose to allow non-aws compliant bucket names
and/or handle the bucket naem validation in the backend instead.
This adds the option to turn off the strict bucket name validation
checks in the frontend API handlers.
When frontend bucket name validation is disabled, we need to do
sanity checks for posix compliant names in the posix/scoutfs
backends. This is automatically enabled when strict bucket
name validation is disabled.
Fixes#1564
Fixes#1574
When versioning is enabled at the gateway level and object lock is enabled for a bucket, any overwrite request on a locked object should succeed since it results in the creation of a new object version. This PR fixes the logic by adding a bucket versioning status check in `CheckObjectAccess`.
This change introduces concurrent execution for integration tests. It adds a mechanism to run tests either synchronously or in parallel, controlled by a new flag. By default, tests continue to run in synchronous mode to maintain predictable behavior during local development. In GitHub Actions, the tests are now executed in parallel mode to significantly reduce overall runtime.
The implementation uses a semaphore-based concurrency control to limit the number of parallel test executions and ensures graceful shutdown through context cancellation. This approach improves test performance while keeping the system stable and backward compatible.
New CLI flags:
- --iam-vault-namespace
- --iam-vault-auth-namespace
- --iam-vault-secret-storage-namespace
Behavior:
- Auth requests use the auth namespace
- KV operations use the secret storage namespace
- If a specific namespace is not set, the shared namespace is used
- With AppRole, different auth and secret namespaces are rejected
Using Docker ENTRYPOINT should allow for configuration of running
versitygw within Docker container similar to how the systemd
service is setup with environment variables.
This also adds the backends azure and plugin to the acceptable
backend options for both docker and systemd.
Fixes#1335
Fixes#1565Fixes#1561Fixes#1300
This PR focuses on three main changes:
1. **Prioritizing object-level lock configuration over bucket-level default retention**
When an object is uploaded with a specific retention configuration, it takes precedence over the bucket’s default retention set via `PutObjectLockConfiguration`. If the object’s retention expires, the object must become available for write operations, even if the bucket-level default retention is still active.
2. **Preventing object lock configuration from being disabled once enabled**
To align with AWS S3 behavior, once object lock is enabled for a bucket, it can no longer be disabled. Previously, sending an empty `Enabled` field in the payload would disable object lock. Now, this behavior is removed—an empty `Enabled` field will result in a `MalformedXML` error.
This creates a challenge for integration tests that need to clean up locked objects in order to delete the bucket. To handle this, a method has been implemented that:
* Removes any legal hold if present.
* Applies a temporary retention with a "retain until" date set 3 seconds ahead.
* Waits for 3 seconds before deleting the object and bucket.
3. **Allowing object lock to be enabled on existing buckets via `PutObjectLockConfiguration`**
Object lock can now be enabled on an existing bucket if it wasn’t enabled at creation time.
* If versioning is enabled at the gateway level, the behavior matches AWS S3: object lock can only be enabled when bucket versioning status is `Enabled`.
* If versioning is not enabled at the gateway level, object lock can always be enabled on existing buckets via `PutObjectLockConfiguration`.
* In Azure (which does not support bucket versioning), enabling object lock is always allowed.
This change also fixes the error message returned in this scenario for better clarity.
Fixes#1559Fixes#1330
This PR focuses on three main changes:
1. **Fix object lock error codes and descriptions**
When an object was WORM-protected and delete/overwrite was disallowed due to object lock configurations, the gateway incorrectly returned the `s3.ErrObjectLocked` error code and description. These have now been corrected.
2. **Update `PutObjectRetention` behavior**
Previously, when an object already had a retention mode set, the gateway only allowed modifications if the mode was changed from `GOVERNANCE` to `COMPLIANCE`, and only when the user had the `s3:BypassGovernanceRetention` permission.
The logic has been updated: if the existing retention mode is the same as the one being applied, the operation is now allowed regardless of other factors.
3. **Fix error checks in integration tests (AWS SDK regression)**
Due to an AWS SDK regression, integration tests were previously limited to checking partial error descriptions. This issue seems to be resolved for some actions (though the ticket is still open: https://github.com/aws/aws-sdk-go-v2/issues/2921). Error checks have been reverted back to full description comparisons where possible.
Fixes#1426
Fiber returns a custom error, if it fails to parse the `Content-Length` header. This implementation adds a check in the fiber global error handler to return an empty `400` Bad Request error, if fiber fails to parse the `Content-Length` header.
Fiber includes a built-in panic recovery middleware that catches panics in route handlers and middlewares, preventing the server from crashing and allowing it to recover. Alongside this, a stack trace handler has been implemented to store system panics in the context locals (stack).
Both the S3 API server and the Admin server use a global error handler to catch unexpected exceptions and recovered panics. The middleware’s logic is to log the panic or internal error and return an S3-style internal server error response.
Additionally, dedicated **Panic** and **InternalError** loggers have been added to the `s3api` debug logger to record system panics and internal errors in the console.
Fixes#1554Fixes#1423
The gateway previously ignored the `x-amz-content-sha256` header for anonymous unsigned requests to public buckets. This PR adds hash calculation for this header and correctly handles special payload types.
It also fixes the case where a signed streaming payload (`STREAMING-AWS4-HMAC-SHA256-PAYLOAD...`) is used with anonymous requests. In this scenario, the gateway now returns a specific "not supported" error, consistent with S3 behavior.
Fixes#1545
`Content-Md5` should be validated/calculated only for the actions containing request body, which are:
* All bucket `PUT` operations(PutBucketTagging, PutBucketVersioning ...)
* All object `PUT` operations(PutObject, UploadPart ...) except for object copy ones(CopyObject, UploadPartCopy)
* Object `POST` operations(CompleteMultipartUpload, RestoreObject ...), but not for `CreateMultipartUpload`, as it doesn't have request body.
* Bucket `POST` operation(DeleteObjects).
The PR removes the `Content-Md5` check from bucket/object GET/HEAD operations and from `PUT`/`POST` operations not expecting request body.
For bucket operations a typical host-style request looks like `bucket.host/`. `HostStyleParser` parses the bucket from host header and appends in the path, by changing the requests to `path-styled`. For bucket operations the original request path is `bucket.host/`, after reconsturction it looks like `/bucket/`: a trailing `/` is added at the end.
The PR adds a check to not append this trailing `/` at the end for bucket operations, to keep consistency with path-style requests.
Closes#1525
* Adds validation for the `Content-MD5` header.
* If the header value is invalid, the gateway now returns an `InvalidDigest` error.
* If the value is valid but does not match the payload, it returns a `BadDigest` error.
* Adds integration test cases for `PutBucketCors` with `Content-MD5`.
Ben McClelland
niksis02
dependabot[bot]
Luke McCrone
Kim Henriksen