CopyObject ignored the destination's `If-Match` and `If-None-Match` headers and always overwrote or created the destination, so a stale `If-Match`, `If-None-Match: *` on an existing key, `If-Match` on a missing key and `If-Match` over a delete marker all returned 200. The controller now parses both headers and passes them to the backend through new `IfMatch` and `IfNoneMatch` fields on `s3response.CopyObjectInput`, which match the SDK's `s3.CopyObjectInput`.
As in S3, the destination is evaluated before the copy source is looked up, so a failing destination precondition wins over a missing source, a failing `x-amz-copy-source-*` precondition and an invalid copy onto itself. Posix checks the destination with `checkPutPreconditions` right after the destination bucket check, then forwards the headers to `PutObject`, which repeats the check under the object publish lock. The unversioned in-place self copy does not go through `PutObject`, so it now takes `lockObjectPublish` and repeats the check itself.
`checkPutPreconditions` now treats a current delete marker as a missing key: `If-Match` returns `NoSuchKey` and `If-None-Match: *` creates a new version. Before, the hidden version's etag was evaluated, which also affected `PutObject` and `CompleteMultipartUpload`.
`EvaluateObjectPutPreconditions` now rejects unsupported headers in S3's order: both headers present, then `If-None-Match` other than `*`, then `If-Match: *`, each with `NotImplemented`. `If-Match: *` was previously compared as a literal etag. The new `NmpAdditionalMessageIfMatch` message is added, and a stray backslash is removed from the `If-None-Match` message.
Azure evaluates the destination preconditions before copying, and s3proxy forwards the headers. `s3:if-match` and `s3:if-none-match` are now populated in the policy condition context for CopyObject, as S3 does, while UploadPartCopy stays excluded.
S3 never lists an empty result for another region: a valid region
other than the endpoint's own gets 400 InvalidArgument, "Requests with
bucket-region specified must be made to the corresponding regional
endpoint", with no ArgumentValue. The gateway is a single regional
endpoint, so it now does the same instead of clearing the listing.
Match the region case-insensitively like S3, and check an empty
bucket-region= too by looking for the query argument itself, since an
empty value is not a valid AWS region either.
ListBuckets ignored bucket-region: a value that is not a region was
accepted, and a region other than the gateway one still listed every
bucket. S3 rejects an invalid value with 400 InvalidArgument, "Argument
value abc is not a valid AWS Region", and lists only the buckets of the
requested region.
Check bucket-region against the AWS region code format, always
accepting the gateway region whatever its name, and return an empty
list for any other region, since every bucket lives in the gateway
region. Enable the REST tests that cover both cases.
Fixes#1923Fixes#1930
* fix: reject ListMultipartUploads requests that also carry uploadId
A bucket GET with both the uploads and uploadId query parameters was
routed to ListMultipartUploads and answered with the upload listing.
S3 rejects the combination with 400 InvalidArgument, "Conflicting query
string parameters: uploadId, uploads".
Add an error route for the combination ahead of ListMultipartUploads,
the way the object level '?uploads' request is rejected, and enable
the REST test that covers it.
Fixes#2155
* fix: reject uploads with uploadId for every bucket and object operation
Requests carrying both the uploads and uploadId query parameters were
only rejected for a bucket GET, so DELETE, PUT, POST and HEAD on the
bucket or object paths still ran real operations. S3 rejects the
combination for every method on both paths, before authentication and
before checking that the bucket exists.
Register the error route for all methods on the bucket and object
routers right after the CORS middleware is set up, ahead of every other
route, and cover each method and path with an integration test.
Add request plumbing for server-side encryption with customer-provided
keys (SSE-C) and stop silently ignoring it on backends that cannot
honor it.
Previously SSE-C headers were dropped before reaching the backend, so
posix stored the data unencrypted and returned success, and later
reads without the key succeeded. Clients believed their data was
encrypted when it was not. Partial or malformed SSE-C header sets were
also accepted instead of failing with 400 as S3 does. Rejecting SSE-C
explicitly is the safe behavior until posix implements it, and azure
cannot faithfully map SSE-C onto customer-provided keys.
This addresses part of #2193: invalid/partial SSE-C headers now fail
with 400. Requiring SSE-C headers on reads of SSE-C objects depends on
posix SSE-C support and remains open.
The changes also reject unsupported SSE-S3, SSE-KMS, and DSSE-KMS
requests instead of silently accepting them. And enforcea TLS for
SSE-C by default, with --disable-sse-c-tls-enforcement as the opt-out
in case the deployment handles TLS in front of the gateway.
An upload-id-marker sent without a key-marker is ignored for the
listing, but the gateway still echoed it back in UploadIdMarker. S3
discards the parameter in that case and returns an empty
UploadIdMarker.
Clear the marker in the controller when no key-marker is given, so
every backend gets the same input and the response no longer reports
a marker that had no effect.
Fixes#2150
A DeleteObjects request whose Delete element names no objects was
answered with 200 and an empty DeleteResult. S3 rejects it with
MalformedXML, as the schema requires at least one Object.
Reject the empty list right after the body is parsed, next to the
1000 key limit check. Update the unit and integration tests that
expected the empty result and enable the REST test that covers it.
Fixes#2109
A request path with an invalid percent escape, such as "/bucket/%zz",
failed url.PathUnescape in the DecodeURL middleware and the raw error
was answered with 500 InternalError. S3 rejects such a request with
400 InvalidURI.
Return InvalidURI from DecodeURL, so the s3 and admin routers answer
with it as well, and let the website handler pass the error through
instead of mapping it on its own.
Fixes#2070
Object routes match subresources with `MatchQueryArgs`, and a request whose subresource has no route falls through to the catch-all `PutObject`, `GetObject` and `DeleteObject` handlers. As a result `PutObjectAnnotation` overwrote the object with the annotation payload or created a missing key, `DeleteObjectAnnotation` deleted the object or permanently deleted the version named by `versionId`, `RenameObject` replaced the destination with an empty object, and `PUT`/`DELETE ?torrent` overwrote or deleted the object.
`PUT ?annotation`, `GET ?annotation&annotationName`, `GET ?annotation` and `DELETE ?annotation` now return `NotImplemented` and follow the existing not-implemented route pattern: `BucketObjectNameValidator`, `AuthorizePublicBucketAccess`, signature verification and `ParseAcl`. Each route has its own metrics action, and the new `s3:PutObjectAnnotation`, `s3:GetObjectAnnotation`, `s3:ListObjectAnnotations` and `s3:DeleteObjectAnnotation` policy actions are also accepted in bucket policies.
`PUT ?renameObject` now returns `NotImplemented`. `?torrent` returns `MethodNotAllowed` with the new `TORRENT` resource type on every method, with `Allow: GET` on all methods except `GET`. Both are answered before authentication, as AWS does, and there is no `s3:RenameObject` policy action.
With <Quiet>true</Quiet> in the Delete request, S3 lists only the
objects it could not delete, so a request where every deletion succeeds
returns an empty DeleteResult. The request body was parsed without the
Quiet element and every deleted object was listed.
Parse Quiet with the objects and drop the Deleted entries from the
merged result when it is set; errors are still reported.
Fixes#2124
An authenticated user with write access to a single bucket could list, write and delete files outside the gateway root on the posix and scoutfs backends. The multipart `uploadId` and the `DeleteObjects` body keys were joined into filesystem paths without validation, and unlike object keys in the request path they never passed the object name validator that rejects `..`. With a `../` upload ID, `ListParts` listed the contents of an outside directory, `UploadPart` and `UploadPartCopy` wrote part files into it, `CompleteMultipartUpload` renamed and removed it, and `AbortMultipartUpload` removed it recursively. A `../` key in `DeleteObjects` removed the outside file, after which the empty-parent cleanup removed emptied directories above the root, and a `/` key removed an empty bucket's directory.
Posix now rejects any upload ID it could not have generated. Posix has always generated upload IDs with `uuid.New().String()`, so `validateUploadID` accepts only a UUID in that exact canonical form and returns `NoSuchUpload` for anything else, which is what AWS returns for a malformed upload ID. The check runs right after the bucket existence check in all five multipart operations that take an upload ID, before any path is built from it.
The `DeleteObjects` controller now checks every key with `IsObjectNameValid`, the same check already applied to request-path, copy-source and POST object keys. If any key is missing or resolves outside the bucket, the whole request is rejected with `BadRequest` before authorization, so no key in the batch is deleted. AWS also fails the whole batch when one key is invalid.
Fixes#1535Fixes#1542Fixes#1740Fixes#1776
S3 requires `Content-MD5`, an `x-amz-checksum-*` header or an `x-amz-checksum-*` trailer on every `PutObject` that carries Object Lock parameters, and rejects the upload otherwise with `InvalidRequest` "Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters". The parameters are an `x-amz-object-lock-legal-hold` header of either status, a retention mode and date, or the bucket's default retention rule, which gives even a plain upload lock parameters. A payload hash, `UNSIGNED-PAYLOAD` and signed chunks don't count as an integrity check. The gateway accepted all of these uploads.
`auth.VerifyWriteObjectLock` reads the bucket's lock configuration, rejects lock headers on a bucket without Object Lock with "Bucket is missing ObjectLockConfiguration", legal hold `OFF` included, and reports whether the write carries lock parameters. `PutObject` calls it when the request has lock headers or no integrity check, so an upload with a checksum and no lock headers pays no extra lookup. `CopyObject` and `CreateMultipartUpload` call it only for lock headers, since S3 requires no integrity check for them.
S3 applies the same rule to `UploadPart`, with "Put Part requests" in the message, when the multipart upload carries lock parameters: its own lock headers, or a default retention rule in force at `CreateMultipartUpload`, which still counts if the rule changes later. posix and azure now mark such an upload when it's created, with the `mp-object-lock` attribute and the `Mpobjectlock` sgwtmp metadata respectively, and `UploadPart` rejects a part of it that has no integrity check. The `UploadPart` controller now passes `Content-MD5` to the backend.
An anonymous `PutObject` carrying lock parameters, including a plain one into a bucket with a default retention rule, is refused with `InvalidArgument` "Put Object requests with Object Lock parameters require AWS Signature Version 4". This comes after the missing configuration and integrity checks, and before the upload itself is authorized, so a private bucket with a default retention rule refuses it too. Anonymous parts of an upload with lock parameters only need an integrity check, and browser-based `POST` uploads are unaffected, as on S3.
Anonymous uploads were authorized for `s3:PutObject` alone, so a public bucket granting only that let anonymous callers tag objects. `AuthorizePublicBucketAccess` now authorizes every action a request carries in one `VerifyPublicAccess` call. A `PutObject` with a non-empty `x-amz-tagging` or a browser-based `POST` with a non-empty tag set needs `s3:PutObjectTagging` as well as `s3:PutObject`. The shared `auth.ObjectUploadActions` derives these actions for both anonymous and signed uploads. Each action is granted on its own, by the bucket policy or, failing that, the bucket ACL, whose WRITE grant covers `s3:PutObject` only, and an explicit public Deny of any one of them denies the request.
Anonymous requests naming an object version were authorized for the plain action. A non-empty `versionId` now makes the public access check authorize the version action instead: `s3:GetObjectVersion` rather than `s3:GetObject` for `GetObject` and `HeadObject`, `s3:DeleteObjectVersion` for `DeleteObject`, `s3:GetObjectVersionAttributes` for `GetObjectAttributes`, and the version tagging actions for the tagging APIs. A public ACL READ grant covers `s3:GetObjectVersion`, but a WRITE grant never covers `s3:DeleteObjectVersion`, which S3 grants to the bucket owner alone.
The "x-amz-object-lock-retain-until-date and x-amz-object-lock-mode must both be supplied" errors no longer end with a period, and the anonymous `CopyObject` and `CreateMultipartUpload` rejections have two spaces before "Please authenticate.", both matching S3.
ListObjects omitted the Marker element when the request had no marker.
S3 returns an empty Marker element in this case, allowing clients to read
the field without a presence check.
The backend converts an empty marker to a nil pointer and encoding/xml omits
nil pointers. Restore the request marker in the controller after the backend
call so the behavior is consistent across backends.
Update the controller, integration, and REST coverage to verify that an
empty Marker element is returned.
Fixes#2191
`POSTObject` authorized only `s3:PutObject`, even when the form's `tagging` field set tags on the uploaded object. `PutObject`, `CopyObject` and `CreateMultipartUpload` also require `s3:PutObjectTagging` when tags are sent, so a caller without that permission could tag objects through `POST` in a way `PutObject` refuses. S3 enforces the permission for `POST` too: a user granted only `s3:PutObject` gets `AccessDenied` for `s3:PutObjectTagging` on the object ARN.
`POSTObject` now adds `s3:PutObjectTagging` to the `verifyAccess` actions when the parsed tag set is not empty, in the same single check as `s3:PutObject`, matching `PutObject`. A `tagging` field with an empty `TagSet` still needs only `s3:PutObject`, as on S3.
The access check now runs after the form is validated, instead of before the `POST` policy is parsed. S3 validates the policy and its conditions, the tagging, the checksum fields, the metadata size and the website redirect location before it authorizes the upload, so a caller missing a permission now gets the validation error for an invalid form instead of `AccessDenied`.
* fix: reject DeleteObjects requests with more than 1000 keys
A DeleteObjects request may name at most 1000 keys; S3 rejects anything
larger with 400 InvalidRequest. The handler parsed the body and passed
every key through authorization and on to the backend, so a 1001-key
request was processed instead of being refused.
Check the count right after the body is parsed, before authorization and
before anything reaches the backend, so an over-limit batch can't be
applied partially. Exactly 1000 keys still succeed, and an empty delete
list keeps its current behavior (200 with an empty result).
Fixes#2196
* fix: return MalformedXML for an oversized DeleteObjects request
A DeleteObjects request naming more than 1000 keys is answered with
MalformedXML, not InvalidRequest. Update the check and the unit and
integration expectations.
---------
Co-authored-by: Tung Lam <lamphamabtung96@gmail.com>
`POSTObject` called `verifyAccess` without an `Object`, and `AuthorizePublicBucketAccess` took the object name from the request path, which is only `/bucket` for a `POST`. So both evaluated `s3:PutObject` against the bucket ARN `arn:aws:s3:::bucket`. `PutBucketPolicy` rejects `s3:PutObject` on a bucket resource, so no valid policy could ever match a `POST` upload. An `Allow` on `arn:aws:s3:::bucket/*` or on a key prefix never applied, and a `Deny` on a key prefix never applied either. A public-write ACL or a broad identity policy could therefore upload through `POST` to keys a `Deny` protects, which `PutObject` refuses. Identity policies evaluated by the standalone IAM service had the same mismatch.
S3 treats `POST` as an alternate form of `PUT` and authorizes it as `s3:PutObject` on the ARN of the object named by the form's `key` field. `POSTObject` now passes that `key` as `Object` to `verifyAccess`, matching `PutObject`. For anonymous `POST` requests, `AuthorizePublicBucketAccess` now uses the `key` that `AuthorizePostObject` has already parsed.
`POSTObject` wrote directly to the backend without the `CheckObjectAccess` object lock check that `PutObject`, `CopyObject` and `CompleteMultipartUpload` perform. In a versioning-disabled bucket, a browser-based `POST` could therefore silently overwrite an object protected by a legal hold, a `COMPLIANCE` retention or a `GOVERNANCE` retention. S3 has no such gap: it allows object lock only on versioned buckets, where an overwrite creates a new version.
`POSTObject` now calls `auth.CheckObjectAccess` with `BypassOverwrite` just before writing to the backend, matching `PutObject`. A legal hold or `COMPLIANCE` retention always blocks the overwrite. A `GOVERNANCE` retention blocks it unless the caller, root included, has an explicit `s3:BypassGovernanceRetention` grant. Versioning-enabled buckets skip the check because the upload creates a new version.
Fixes#2239
Running the gateway with a `--region` other than `us-east-1` turned every request from a client signed for a different region into a dead end. The signing region check correctly rejected the request with `AuthorizationHeaderMalformed`, but the gateway gave the client no usable way to learn which region it should have signed for, so the aws sdks could not re-sign and retry the way they do against a real s3 regional endpoint.
The gap was `x-amz-bucket-region`. S3 reports the expected region in that response header as well as in the error body, because a `HEAD` request carries no body for the client to parse. The gateway set the header only on successful responses, so `HeadBucket` and `HeadObject` signed for the wrong region came back as a bare `400` with an empty body and some sdks region redirect had nothing to work with. Region mismatches now report the gateway region through a `RegionMismatchError` interface that `MalformedAuthError`, `AuthQueryParamError` and `InvalidArgumentError` implement, and the error dispatch points set the header from it alongside the existing `Allow` header handling for `MethodNotAllowed`.
The error bodies were incomplete in the same way. Only the `Authorization` header path carried a `Region` element, while the presigned url and POST form paths reported the mismatch in prose alone. `AuthorizationQueryParametersError` is now backed by a dedicated `AuthQueryParamError` type that adds the element, and `InvalidArgumentError` grew an optional `Region` field for the POST form credential error, both omitted for every other error of those kinds.
The messages themselves now read the way s3 writes them. The region and service names are quoted with apostrophes instead of `%q`, whose double quotes the xml encoder rendered as `"` in the raw response, and the POST form authentication fields are echoed back in `ArgumentName` under their canonical spelling, for example `X-Amz-Credential` rather than the lowercase form the browser submits, while fields outside the signing protocol such as `key` are still reported as submitted.
`ParseContentEncoding` dropped the `aws-chunked` token only for streaming payload types, so a presigned request kept it: such a request is signed as `UNSIGNED-PAYLOAD` and sends no `x-amz-content-sha256` header, and S3 strips the token there. Keep it only when the request declares a non streaming payload type.
S3 applies this to the `Content-Encoding` header before any API sees it, so `CopyObject` and `CreateMultipartUpload` follow the same rule even though neither carries a body to frame. `POSTObject` takes its value from a form field rather than the header and is left alone.
The rejection now reports `Content-Encoding` as the `ArgumentName` and the bare `aws-chunked` token as the `ArgumentValue`, rather than `x-amz-content-sha256` and the payload type, so a request sending `gzip,aws-chunked` gets back just the offending coding, and the message carries S3's trailing period.
Adds integration tests for the three cases: `UnsignedStreamingPayloadTrailer_strips_aws_chunked_content_encoding` for a framed upload where only `aws-chunked` is dropped and the remaining codings keep their order, `PutObject_plain_stores_aws_chunked_content_encoding` for a hex-payload PUT that stores the token as sent, and `PutObject_unsigned_payload_with_aws_chunked_content_encoding` for the full error shape across four header spellings.
- Replace `StripAwsChunkedEncoding` with `ParseContentEncoding`, which drops
the token only when `x-amz-content-sha256` names a streaming payload type,
so a client that sends `aws-chunked` on a hex-payload request keeps it
- Revert the `CopyObject` and `CreateMultipartUpload` call sites: both routes
are registered with `streamBody` false, so a streaming payload type is
rejected before the controller and the token can only be a stored value
- Reject `aws-chunked` combined with `UNSIGNED-PAYLOAD` in the authentication
middleware, beside the existing payload-type validation, with a new
`InvalidArgAwsChunkedUnsignedPayload`
- Add REST tests for the three cases: stripped from a chunked upload, kept on
a hex-payload request, and rejected with `UNSIGNED-PAYLOAD`
- Add `CONTENT_ENCODING` to the PutObject REST script and a
`check_content_encoding` driver for HeadObject
X-Amz-Decoded-Content-Length describes decoded aws-chunked payloads and must
not control a plain request. Using it for a plain UploadPart lets a larger
decoded header drive backend preallocation; the posix multipart path does not
truncate the staged part, so completion publishes a zero-filled tail.
Select the decoded length only for aws-chunked readers. Plain PutObject and
UploadPart now use Content-Length for both short-body validation and backend
allocation.
* fix(s3api): reject a PUT whose body ends before Content-Length
With a plain (non-aws-chunked) request body, a PutObject whose body ends
before Content-Length bytes have arrived is committed as a complete but
shorter object. Real S3 rejects this with IncompleteBody and the object
never becomes visible.
Nothing on that path compares bytes received against Content-Length:
fasthttp reports a connection closed mid-body as a plain io.EOF (the
conversion to io.ErrUnexpectedEOF exists only on the chunked
transfer-encoding branch), the authentication middleware leaves an
UNSIGNED-PAYLOAD body unwrapped, and io.Copy treats io.EOF as a normal
end of stream.
ErrIncompleteBody already exists and is enforced by the chunk readers.
This adds the equivalent check for plain bodies, in the controller so
that every backend is covered, and only when the body is not already an
aws-chunked reader.
POST-Object is deliberately untouched: it is a separate handler whose
ContentLength is an upper bound, so a byte-count check there would
break browser form uploads.
* test(s3api): cover ContentLengthReader
Complete and empty bodies, truncated bodies (including an EOF delivered
together with the final bytes, and one byte per Read), a body longer
than announced, and pass-through of a non-EOF error.
* fix(s3api): check plain bodies against Content-Length, not the decoded length
Review catch: the controller replaces contentLength with
X-Amz-Decoded-Content-Length whenever that header is present, regardless
of payload type, and the previous commit fed that value to
ContentLengthReader. A complete plain upload whose decoded header is
larger than Content-Length was then rejected with IncompleteBody, where
it had succeeded before.
AWS S3 ignores X-Amz-Decoded-Content-Length on a plain body and stores
Content-Length bytes. The decoded length only describes aws-chunked
payloads, and those skip this wrapper anyway, so the check now reads the
raw Content-Length header.
If the header is missing or unparseable the body is left unwrapped: an
aws-chunked request without Content-Length is already rejected earlier
with ErrMissingContentLength, so there is nothing to check here.
* test(integration): cover aborted uploads and the decoded-length case
Three tests, as requested in review.
PutObject_aborted_plain_body plain PUT, body ends early -> no object
PutObject_aborted_streaming_body same for an aws-chunked upload
PutObject_plain_body_with_decoded_length
complete plain PUT carrying
X-Amz-Decoded-Content-Length still
succeeds and stores Content-Length bytes
The abort is driven by a reader that fails partway, so the transport
tears the connection down mid-body - that is what a client that dies or
cancels looks like on the wire. Sending a short body with a normal
reader would not reproduce it: net/http would simply report the
mismatch itself.
The third test is the regression the review found. It fails without the
accompanying fix.
* test(integration): replace the bogus streaming abort test with a real one
The reviewer is right: putObjectAborted always sends
x-amz-content-sha256: UNSIGNED-PAYLOAD, and the gateway decides aws-chunked
from that header alone, so PutObject_aborted_streaming_body took the same
plain path as the test above it. It duplicated the plain test instead of
covering the streaming one.
Dropped it and added UnsignedStreamingPayloadTrailer_aborted_connection in
tests/integration/unsigned_streaming_payload_trailer.go, where it belongs.
UnsignedStreamingPayloadTrailer_incomplete_body already covers malformed and
truncated framing, but every case there is a COMPLETE request. The new test is
the other shape: valid framing whose bytes simply stop arriving.
* fix: return InvalidArgument for an empty website configuration
`PutBucketWebsite` parsed an empty `<WebsiteConfiguration/>` into a zero-value config, and `WebsiteConfiguration.Validate` reported the nil `IndexDocument` as `MalformedXML`. S3 reports the missing index document as an `InvalidArgument` request error naming the argument instead:
InvalidArgument: A value for IndexDocument Suffix must be provided if RedirectAllRequestsTo is empty
That branch now returns the error with `ArgumentName=IndexDocument` and `ArgumentValue=null`, matching the response in the report. Genuinely malformed XML, and a `RedirectAllRequestsTo` that conflicts with the other fields, still return `MalformedXML`.
Fixes#2260
* test: cover an empty website configuration in the PutBucketWebsite integration suite
Add a `PutBucketWebsite_empty_configuration` case that sends the empty `<WebsiteConfiguration/>` payload from the report and asserts the `InvalidArgument` response, including `ArgumentName=IndexDocument` and `ArgumentValue=null`.
The SDK collapses the error into a generic API error that drops the argument fields, so the request is signed by hand and the raw response checked with `checkHTTPResponseApiErr`, the same helper the other argument-field assertions use. The case fails against the previous behaviour with `expected error code to be InvalidArgument, instead got MalformedXML`.
---------
Co-authored-by: Tung Lam <lamphamabtung96@gmail.com>
Closes#2322
The admin API could not be served under a path prefix. Admin requests are SigV4-signed over the full path, so a reverse proxy or Gateway API route cannot strip a prefix before forwarding. That made it impossible to serve the WebUI and the admin API under one hostname. The new `--admin-path-prefix` (`VGW_ADMIN_PATH_PREFIX`) option mounts the admin routes under a single-segment prefix such as `/admin`, on `--admin-port` or, when that is unset, on the S3 port. Admin clients include the prefix in their endpoint URL, which the admin CLI and the WebUI already support.
The prefix is limited to unreserved characters, because clients and the gateway encode other characters differently when signing. When the admin API shares the S3 port, the prefix must also differ from `--webui-s3-prefix`, otherwise the WebUI mount answers the admin requests. Auto-detected WebUI admin gateway URLs and the startup banner now include the prefix. The admin routes served on the S3 port now reuse `S3AdminRouter` instead of a duplicated route list. This also stops the standalone admin server from recording bucket creation under the `ActionAdminListBuckets` action.
The Helm chart now exposes `webui.pathPrefix`, `webui.s3Prefix` and `admin.pathPrefix`. The WebUI gateway lists are also passed when only `webui.s3Prefix` is set, since the WebUI hosted on the S3 port does not require `webui.enabled`.
Internal IAM data can now live on its own volume. `iam.dir` sets the IAM directory, and `iam.persistence` creates or references a dedicated PVC when `iam.enabled` is true and `iam.type` is `internal`. Otherwise IAM data stays in the `iam` subdirectory of the backend data volume, so existing releases are unaffected.
The chart now rejects overlapping storage directories. `iam.dir` must be an absolute path outside `/mnt/data`, `gateway.backend.sidecarDir` and `gateway.backend.versioningDir`, and those three directories must not overlap each other. Deleting a bucket removes `<dir>/<bucket>` from the sidecar and versioning directories, so with nested directories, deleting a suitably named bucket could wipe IAM accounts, object versions or the entire backend data directory.
* fix: omit ObjectLockConfiguration Rule when there is no default retention
ParseBucketLockConfigurationOutput always set Rule, so a bucket with object lock enabled and no default retention answered GET ?object-lock with an empty <Rule></Rule>. AWS S3 omits the element, and the AWS SDK v2 clients that read the rule from it follow the empty element with a malformed request.
Fixes#2397
* test: cover GetObjectLockConfiguration with no default retention in the integration suite and drop the stale non-nil Rule expectation from the controller unit test
---------
Co-authored-by: Tung Lam <lamphamabtung96@gmail.com>
Closes#2304
Adds `versitygw utils convert-posix-dataset` (alias `cpd`), which makes a posix dataset not created by the gateway fully compatible with the posix backend. Every top level directory is treated as a bucket and gets a private ACL owned by `--access-key-id` (defaults to the root `--access`) and `BucketOwnerEnforced` object ownership; root level files are ignored. Metadata is stored in xattrs, or in `--sidecar-dir` when set.
`--calculate-etag` and `--checksum-algorithm` optionally compute the MD5 ETag and the `FULL_OBJECT` checksum of every object in a single streamed read. Objects are converted concurrently, tunable with `--concurrency` and `--read-buffer-size`. Existing bucket and object metadata is never overwritten.
S3 returns `InvalidArgument` when an object action receives a `versionId` query parameter with no value. The gateway silently treated it as an unversioned request instead.
Added a shared `versionId` validation helper and apply it to the object actions that accept the parameter, so malformed requests are rejected up front rather than reaching the backend.
The body stream is now wrapped in a `bodyStreamTracker` before the handler touches it, which remembers the stream's first terminal result rather than asking fasthttp a second, unsafe question. `io.EOF` means the body was read out in full and the connection is still in sync; no terminal result means the handler stopped partway, so the leftovers are drained the way a
`Content-Length` body already was; a framing error means nothing decodable is left and the connection cannot carry another request.
`fasthttp.Request.SetBodyStream` cannot install the wrapper, as it releases the current `*requestStream` back to its pool, so `requestBodyStream` is now the accessor every body reader takes the stream from.
Broken framing no longer gives up on draining either. The connection is closed either way, so a bounded read off the raw socket costs nothing and lets the client finish its write and read the S3 error instead of a reset.
Service the overflow list during normal operation: the worker
publishes and clears it after every channel job, so a burst that
exceeds the channel buffer drains as soon as the sink recovers
instead of accumulating until shutdown.
Make dispatch and the drain sweep share one critical section. An
append either lands before the sweep and is drained, or runs
after the worker exited and publishes inline; the check-then-send
window that could strand a record between the two is closed.
Request publications use the same boundary with drop semantics:
once the drain finished, no owner can guarantee the sinks are
still open, so the record is dropped rather than published.
Never close the metrics datapoint channel. The forwarder exits
through the canceled context after draining the buffer, the
closed flag turns late producers into drops, and no send can race
a closure. Implement the new Manager method in the test mock.
A streamed chunked request can leave bytes queued after the handler returns.
Reusing that connection lets fasthttp parse those bytes as the next request,
allowing a shared upstream proxy connection to mix requests across tenants.
Mark chunked requests Connection: close when the middleware cannot safely
drain them, preventing leftover bytes from crossing the request boundary.
Draining is intentionally skipped because fasthttp's request stream reads past
the terminating chunk when probing for EOF and can block waiting for another
chunk header. The connection-reuse sacrifice is therefore required to avoid both
request desynchronization and delaying the response. Content-Length bodies
retain the existing bounded drain behavior.
A new glaicer mode for the archiving backend needs the storage
class supplied to the backend for put object, create multipart
upload, and browser post object input.
`WithOnListen` tells an embedder when the S3 server is serving, but not
where: the callback takes no arguments, and the server reports the
addresses it bound nowhere else. An embedder that wants an ephemeral port
therefore cannot ask for port 0; it has to pick a free port itself,
release it, and pass it in, which loses to any other process that binds
the same port in between.
Add `WithOnListenAddrs`, which passes the callback the address of every
listener the server bound, in port-specification order, so an embedder can
serve on `127.0.0.1:0` and learn the port the kernel chose. `WithOnListen`
is unchanged. `MultiListener` gains `Addrs`, the every-listener counterpart
of `Addr`, to supply them.
Ben McClellandandGitHub
niksis02
DualFroz
Javier Garcia
Lennard GeißlerandBen McClelland
Barry Loongandniksis02
DavvyyandGitHub
iRioxe
Jihyeon Gim
RaduBerinde