This is part of the thread exhaustion issue (#1815).
This PR introduces:
* A **maximum Fiber HTTP connections limit**
* A middleware that enforces a **hard limit on in-flight HTTP requests**
When the in-flight request limit is reached, the middleware returns an **S3-compatible `503 SlowDown`** error.
The same mechanism is implemented for the **admin server** (both max connections and max in-flight requests).
All limits are configurable via **CLI flags** and **environment variables**, for both the `s3api` server and the `admin` server.
---
| Setting | CLI Flag | Alias | Environment Variable | Default |
| --------------- | ------------------- | ----- | --------------------- | ------- |
| Max Connections | `--max-connections` | `-mc` | `VGW_MAX_CONNECTIONS` | 250000 |
| Max Requests | `--max-requests` | `-mr` | `VGW_MAX_REQUESTS` | 100000 |
---
| Setting | CLI Flag | Alias | Environment Variable | Default |
| --------------- | ------------------------- | ------ | --------------------------- | ------- |
| Max Connections | `--admin-max-connections` | `-amc` | `VGW_ADMIN_MAX_CONNECTIONS` | 250000 |
| Max Requests | `--admin-max-requests` | `-amr` | `VGW_ADMIN_MAX_REQUESTS` | 100000 |
The system.yml file was giving this warning:
Context access might be invalid: SAFE_RUN_SET
The warning occurs because this was trying to access env.SAFE_RUN_SET
in a with: key of an action, but GitHub Actions has restrictions on where
context variables can be accessed.
The env context isn't always guaranteed to be available in the with: key
of actions, especially when it depends on runtime values set in previous
steps.
The recommended fix is to change from $GITHUB_ENV to $GITHUB_OUTPUT
for this case.
Fixes#1852Fixes#1821
Fiber used to return the `text/plain` default `Content-Type` for error responses, because it wasn't explicitly set. Now for all error responses the `application/xml` content type is set.
* Update client.go to support anonymous S3 access
* Update s3.go to make access and secret parameters optional
* Update example.conf for more clear S3 access and secret usage
Fixes#1836
The latest version of the go sdk has deprecate the s3 manager in
favor of the new transfermanager. This updates to the new
transfermanager functionality for the two places we were using
the manager, iam s3 and integration tests.
This also removes the pinning of nats nkeys since they have fixed
the tags in their repo now. And general cleanup of the go.mod.
Fixes#1835
If-Match in DeleteObject is a precondition header that compares the client-provided ETag with the server-side ETag before deleting the object. Previously, the comparison failed when the client sent an unquoted ETag, because server ETags are stored with quotes. The implementation now trims quotes from both the input ETag and the server ETag before comparison to avoid mismatches. Both quoted and unquoted ETags are valid according to S3.
CopyObject was failing with NoSuchKey when source keys contained special
characters like {} or spaces. The X-Amz-Copy-Source header is URL-encoded
by clients, but ParseCopySource wasn't decoding before filesystem access.
Added url.QueryUnescape() to properly decode bucket and object names,
fixing copy operations for keys with special characters.
Fixing this also uncovered an errors with azure blob url encoding with
similar special character handling. Added this fix in for the integration
tests to pass.
Fixes#1832Fixes#1637
The PutBucketObjectLockConfiguration now requires Content-MD5
header to match AWS behavior. This broke the GUI from being able
to set object lock configuration for a bucket.
This fix adds the Content-MD5 header to this request.
Fixes#1809Fixes#1806Fixes#1804Fixes#1794
This PR focuses on correcting so-called "list-limiter" parsing and validation. The affected limiters include: `max-keys`, `max-uploads`, `max-parts`, `max-buckets`, `max-uploads` and `part-number-marker`. When a limiter value is outside the integer range, a specific `InvalidArgument` error is now returned. If the value is a valid integer but negative, a different `InvalidArgument` error is produced.
`max-buckets` has its own validation rules: completely invalid values and values outside the allowed range (`1 <= input <= 10000`) return distinct errors. For `ListObjectVersions`, negative `max-keys` values follow S3’s special-case behavior and return a different `InvalidArgument` error message.
Additionally, `GetObjectAttributes` now follows S3 semantics for `x-amz-max-parts`: S3 ignores invalid values, so the gateway now matches that behavior.
Fixes#1792Fixes#1747Fixes#1797Fixes#1799
This PR primarily introduces delimiter support and several bug fixes for the `ListMultipartUploads` action in the POSIX and Azure backends. Delimiter handling is now implemented — when a delimiter is present in multipart-upload object key names, the backend collects and returns the appropriate common prefixes.
This functionality is achieved by introducing a common multipart-upload lister in the backend package. All backends (Azure, POSIX) now use this lister. The lister accepts a list that is already sorted and filtered by `KeyMarker` and `Prefix`.
Previously, the `KeyMarker` was required to exactly match an existing multipart-upload object key. This restriction is removed. The listing now relies on a lexicographical comparison between the provided `KeyMarker` and existing multipart-upload object keys.
Validation for `UploadIdMarker` is also added to correctly return an `InvalidArgument` error for invalid upload IDs. If `KeyMarker` is missing, the `UploadIdMarker` is ignored entirely. If `KeyMarker` is provided, a valid upload ID is one that matches an upload belonging to *the first object key after the KeyMarker*. For example, if the `KeyMarker` is `foo`, but the provided `UploadIdMarker` corresponds to an upload under `quxx`, it is invalid. It must match one of the uploads for the next object key equal to `foo`.
Finally, this PR fixes multipart-upload sorting. Multipart uploads must be sorted primarily lexicographically by their object key, and secondarily—when multiple uploads share the same object key—by their initiation time in ascending order.
In the refactor for being able to set global CORS headers, the
options router was incorrectly set to use both CORS middlewares
casuing duplicate headers to be set. The ApplyBucketCORS()
middleware is not needed for options since this is already handled
by the CORSOoptions controller.
Fixes#1819
The go mod cache is listing a newer version than the latest tag in
the repo. So we need to pin this older version to prevent dependabot
from updating until the repo tags a newer version.
