Fixes#1364
When a completely malformed request is sent to the gateway, Fiber/Fasthttp fails to parse the request, and the code execution never reaches the routers or handlers. Instead, the error is caught by the global error handler. These kinds of errors (malformed requests that fail during request parsing) are prefixed with **"error when reading request headers"** in Fiber. The implementation adds a check in the global error handler for this specific error message and returns an S3-like XML **BadRequest** error instead.
Fixes#1547
When no checksum is specified during multipart upload initialization, the complete multipart upload request should default to **CRC64NVME FULL_OBJECT**. The checksum will not be stored in the final object metadata, as it is used solely for data integrity verification. Note that although CRC64NVME is composable, it is calculated using the standard hash reader, since the part checksums are missing and the final checksum calculation is instead based directly on the parts data.
Fixes#1359
The composite checksums in **CompleteMultipartUpload** generally follow the format `checksum-<number_of_parts>`. Previously, the gateway treated composite checksums as regular checksums without distinguishing between the two formats.
In S3, the `x-amz-checksum-*` headers accept both plain checksum values and the `checksum-<number_of_parts>` format. However, after a successful `CompleteMultipartUpload` request, the final checksum is always stored with the part number included.
This implementation adds support for parsing both formats—checksums with and without the part number. From now on, composite checksums are consistently stored with the part number included.
Additionally, two integration tests are added:
* One verifies the final composite checksum with part numbers.
* Another ensures invalid composite checksums are correctly rejected.
Fixes#1329
Fixes the checksum type/algorithm mismatch error in `CreateMultipartUpload`. The algorithm an type were messed in the error description. It also adds an integration test to target the unsupported checksum type/algorithm pairs.
Fixes#1428
The `x-amz-expected-bucket-owner` header in S3 specifies the account ID of the expected bucket owner. If the account ID provided does not match the actual owner of the bucket, the request fails with an HTTP 403 Forbidden (AccessDenied) error. If the provided account ID is not 12 characters long, S3 returns a 400 Bad Request error.
In our case, we expect the header to contain the bucket owner’s access key ID, and we skip validation errors related to the access key ID, since there is no validation mechanism for user access key IDs. If the provided value does not match the bucket owner’s access key ID, the gateway returns an AccessDenied error.
A few integration tests are added for random actions, as this feature applies to all actions, but it is unnecessary to add test cases for every single one.
Fixes#1484
Removes response header name normalization to prevent Fiber from converting them to camel case. Also fixes the `HeadBucket` response headers by changing their capital letters to lowercase and corrects the `x-amz-meta` headers to use lowercase instead of camel case.
Closes#1536
Adds bucket policy version support. Two versions are supported: **2008-10-17** and **2012-10-17**. If the `Version` field is omitted in the bucket policy document, it defaults to **2008-10-17**. However, if an empty string (`""`) is provided, it is considered invalid.
Implements graceful shutdown for the admin and s3api servers. They are shut down before other components (IAM, s3logger, etc.) to allow the servers to properly handle any pending requests while dependencies are still active. The shutdown process is controlled by a context with a 10-second timeout. If it exceeds this duration, all remaining requests are forcefully terminated and the servers are closed.
Closes#1566
When an object is locked and bucket versioning is not configured at the gateway level, any object overwrite request should be rejected with an object locked error. The `PutObject` operation already follows this behavior, but `CopyObject` and `CompleteMultipartUpload` were missing this check. This change introduces the locking mechanism for `CopyObject` and `CompleteMultipartUpload` operations.
Some systems may choose to allow non-aws compliant bucket names
and/or handle the bucket naem validation in the backend instead.
This adds the option to turn off the strict bucket name validation
checks in the frontend API handlers.
When frontend bucket name validation is disabled, we need to do
sanity checks for posix compliant names in the posix/scoutfs
backends. This is automatically enabled when strict bucket
name validation is disabled.
Fixes#1564
Fixes#1574
When versioning is enabled at the gateway level and object lock is enabled for a bucket, any overwrite request on a locked object should succeed since it results in the creation of a new object version. This PR fixes the logic by adding a bucket versioning status check in `CheckObjectAccess`.
This change introduces concurrent execution for integration tests. It adds a mechanism to run tests either synchronously or in parallel, controlled by a new flag. By default, tests continue to run in synchronous mode to maintain predictable behavior during local development. In GitHub Actions, the tests are now executed in parallel mode to significantly reduce overall runtime.
The implementation uses a semaphore-based concurrency control to limit the number of parallel test executions and ensures graceful shutdown through context cancellation. This approach improves test performance while keeping the system stable and backward compatible.
New CLI flags:
- --iam-vault-namespace
- --iam-vault-auth-namespace
- --iam-vault-secret-storage-namespace
Behavior:
- Auth requests use the auth namespace
- KV operations use the secret storage namespace
- If a specific namespace is not set, the shared namespace is used
- With AppRole, different auth and secret namespaces are rejected
Using Docker ENTRYPOINT should allow for configuration of running
versitygw within Docker container similar to how the systemd
service is setup with environment variables.
This also adds the backends azure and plugin to the acceptable
backend options for both docker and systemd.
Fixes#1335
Fixes#1565Fixes#1561Fixes#1300
This PR focuses on three main changes:
1. **Prioritizing object-level lock configuration over bucket-level default retention**
When an object is uploaded with a specific retention configuration, it takes precedence over the bucket’s default retention set via `PutObjectLockConfiguration`. If the object’s retention expires, the object must become available for write operations, even if the bucket-level default retention is still active.
2. **Preventing object lock configuration from being disabled once enabled**
To align with AWS S3 behavior, once object lock is enabled for a bucket, it can no longer be disabled. Previously, sending an empty `Enabled` field in the payload would disable object lock. Now, this behavior is removed—an empty `Enabled` field will result in a `MalformedXML` error.
This creates a challenge for integration tests that need to clean up locked objects in order to delete the bucket. To handle this, a method has been implemented that:
* Removes any legal hold if present.
* Applies a temporary retention with a "retain until" date set 3 seconds ahead.
* Waits for 3 seconds before deleting the object and bucket.
3. **Allowing object lock to be enabled on existing buckets via `PutObjectLockConfiguration`**
Object lock can now be enabled on an existing bucket if it wasn’t enabled at creation time.
* If versioning is enabled at the gateway level, the behavior matches AWS S3: object lock can only be enabled when bucket versioning status is `Enabled`.
* If versioning is not enabled at the gateway level, object lock can always be enabled on existing buckets via `PutObjectLockConfiguration`.
* In Azure (which does not support bucket versioning), enabling object lock is always allowed.
This change also fixes the error message returned in this scenario for better clarity.
Fixes#1559Fixes#1330
This PR focuses on three main changes:
1. **Fix object lock error codes and descriptions**
When an object was WORM-protected and delete/overwrite was disallowed due to object lock configurations, the gateway incorrectly returned the `s3.ErrObjectLocked` error code and description. These have now been corrected.
2. **Update `PutObjectRetention` behavior**
Previously, when an object already had a retention mode set, the gateway only allowed modifications if the mode was changed from `GOVERNANCE` to `COMPLIANCE`, and only when the user had the `s3:BypassGovernanceRetention` permission.
The logic has been updated: if the existing retention mode is the same as the one being applied, the operation is now allowed regardless of other factors.
3. **Fix error checks in integration tests (AWS SDK regression)**
Due to an AWS SDK regression, integration tests were previously limited to checking partial error descriptions. This issue seems to be resolved for some actions (though the ticket is still open: https://github.com/aws/aws-sdk-go-v2/issues/2921). Error checks have been reverted back to full description comparisons where possible.
Fixes#1426
Fiber returns a custom error, if it fails to parse the `Content-Length` header. This implementation adds a check in the fiber global error handler to return an empty `400` Bad Request error, if fiber fails to parse the `Content-Length` header.
Fiber includes a built-in panic recovery middleware that catches panics in route handlers and middlewares, preventing the server from crashing and allowing it to recover. Alongside this, a stack trace handler has been implemented to store system panics in the context locals (stack).
Both the S3 API server and the Admin server use a global error handler to catch unexpected exceptions and recovered panics. The middleware’s logic is to log the panic or internal error and return an S3-style internal server error response.
Additionally, dedicated **Panic** and **InternalError** loggers have been added to the `s3api` debug logger to record system panics and internal errors in the console.
Fixes#1554Fixes#1423
The gateway previously ignored the `x-amz-content-sha256` header for anonymous unsigned requests to public buckets. This PR adds hash calculation for this header and correctly handles special payload types.
It also fixes the case where a signed streaming payload (`STREAMING-AWS4-HMAC-SHA256-PAYLOAD...`) is used with anonymous requests. In this scenario, the gateway now returns a specific "not supported" error, consistent with S3 behavior.
