Follow-up to #2486
The AWS SDKs and CLI send `ObjectIdentifier.LastModifiedTime` as an HTTP date, but `s3response.DeleteObjects` decoded it into the SDK's `time.Time` field, which accepts only RFC 3339. Any `DeleteObjects` request that carried a `LastModifiedTime` was therefore rejected as a whole with `InvalidRequest`, and the RFC 3339 form the gateway did accept is one that S3 rejects.
`DeleteObjects` now has its own `UnmarshalXML`. It embeds `types.ObjectIdentifier`, decodes `LastModifiedTime` as a string and parses it with `smithytime.ParseHTTPDate`, falling back to `time.RFC1123Z` for numeric zone offsets. This accepts IMF-fixdate, RFC 850, ANSIC and `+hhmm` offsets and rejects RFC 3339, which matches S3 directory buckets, the only bucket type where S3 evaluates this field.
Errors are reported as S3 reports them. An unparseable date returns `InvalidRequest` with "Invalid value for date attribute: <value>" through the new `s3err.GetInvalidDateAttributeErr`, which echoes the raw value of the first bad date in the body. An empty, whitespace-only or self-closing `LastModifiedTime` returns `MalformedXML`, which takes precedence over an invalid date in any object. The `DeleteObjects` controller now passes S3 errors from the decoder through instead of turning every decode failure into `InvalidRequest`.
fiber `v3.6.0` (gofiber/fiber#4695) normalizes every request path before routing as RFC 3986 Section 6.2.2 describes: it decodes the escapes of unreserved characters and removes `.` and `..` segments, and `ctx.Path()`, `ctx.Params()` and every `ctx.Path(override)` return the normalized path with no way to opt out. `DecodeURL` sets the decoded path back through `ctx.Path()`, so keys were decoded twice (`100%2541` was stored as `100A`, `q%7euser` as `q~user`) and dot segments rerouted requests: `PUT /bucket/.` became `CreateBucket` and `/bucket/../../etc/passwd` named the bucket `etc`.
The new `httpctx.Ctx`, created with `fiber.NewWithCustomCtx`, keeps a path set with `Path()` as it is and has fiber route a copy in which every `%` and the dots of a `.` or `..` segment are `_`. The copy keeps the length of the path, since fiber matches routes on it but takes the route parameters from `Path()` at the matched offsets, so `Params("bucket")` holds the real bucket. The S3, admin and website servers and the RDMA ops tracker use it.
S3 stores keys holding `.` and `..` segments as they are (`a/../b`, `x/.` and `..` are objects of their own) and only rejects, before authentication, a request path whose decoded `..` segments climb above the root, separating the segments at `\` as well as at `/` and skipping the empty ones. `DecodeURL` now rejects such a path with `InvalidURI` and keeps the dot segments in the key otherwise.
`Backend.StoresDotSegmentKeys()` reports whether a backend stores such keys as their own objects: `azure` and `s3proxy` do, while `posix` and the `BackendUnsupported` default don't, since a filesystem resolves the segments to another object or to a path outside the bucket. `IsObjectNameValid` takes the backend and rejects keys holding dot segments for those backends in the request path, the `DeleteObjects` keys, the `PostObject` key and the copy sources, so `posix` now also rejects `a/../b` and `a/./b`, which it stored as `b` and `a/b`. The unused `path.go` is removed.
Fixes#2525
`CompleteMultipartUpload` returned the stored part's ETag in `InvalidPart` errors instead of the ETag the client sent: posix on an ETag mismatch, and azure on a zero-byte part mismatch, where it returned the expected block ID. Azure also returned an empty ETag for a part that was never uploaded. All three now return the client's ETag. Like S3, `GetInvalidPartErr` removes every `"` from it, inner quotes included, and keeps surrounding whitespace.
`SetResponseHeaders` turns off header name normalization and leaves it off, so response header names stay lowercase. Before it runs, the `RequestIDs` middleware stored `X-Amz-Request-Id` and `X-Amz-Id-2`. Afterwards, the `EnsureRequestIDs` call in `ProcessController` stored `x-amz-request-id` and `x-amz-id-2`. Every response with a headers map, success responses included, therefore sent both pairs. `EnsureRequestIDs`, `RequestID` and `HostID` now use `SetCanonical` and `PeekCanonical` with the lowercase names, so the normalization setting doesn't affect them, and each header is sent once in lowercase as S3 does.
ListObjects and ListObjectsV2 ignored the encoding-type query parameter.
Any value was accepted, and with encoding-type=url the response carried
neither an EncodingType element nor encoded key names.
S3 accepts only "url" and answers anything else with InvalidArgument
("Invalid Encoding Method specified in Request", argument encoding-type).
With "url" it sets EncodingType and encodes the key names in the
listing as a query component, "a+ b.txt" becoming "a%2B+b.txt", with
"/" left as is: Key, Prefix, Delimiter and CommonPrefixes in both
operations, Marker and NextMarker in ListObjects, and StartAfter in
ListObjectsV2. The continuation tokens are opaque and are not encoded.
The encoding is applied to a copy of the backend result in the
controllers, so it covers every backend.
Fixes#1985Fixes#1984
The controller now passes the requested attributes to the backend in
ObjectAttributes, so posix and azure only do the extra HeadObject for the
parts count when ObjectParts was asked for. This also lets the s3 backend
reach the upstream service, which rejected the request before because the
field was missing.
The s3 backend now copies TotalPartsCount into PartsCount, leaves out
ObjectParts when the upstream response has none, and sets the part number
markers, which were dropped by inverted strconv.Atoi error checks.
A conditional delete of a key with no current version used to succeed. `DeleteObject` with `If-Match`, and a `DeleteObjects` key with an `ETag`, reported success for a missing key. When the current version was a delete marker, the condition was checked against the ETag the marker kept from the version it hides, so it could pass and create a second marker. S3 fails both with `NoSuchKey` and creates no marker. The posix backend now checks the conditions against the current version in `checkDeletePreconditions`, before its missing-key shortcuts. The azure backend returns `NoSuchKey` for a missing blob instead of skipping the conditions. `ObjectDeletePreconditions.IsSet` reports whether a delete has any condition.
A condition only applies to the current version. `DeleteObject` with a `versionId` and `If-Match` now returns `NotImplemented` with `<Header>If-Match</Header>` and the S3 additional message, ahead of authorization. Before, it checked the condition against that version and deleted the version. In `DeleteObjects`, an empty `<ETag>` fails its key with `InvalidArgument` and a `VersionId` together with an `ETag` fails it with `NotImplemented`, both ahead of the key's authorization.
The `PreconditionFailed` and `NotImplemented` messages drop their trailing period, which S3 doesn't use on any operation.
CopyObject ignored the destination's `If-Match` and `If-None-Match` headers and always overwrote or created the destination, so a stale `If-Match`, `If-None-Match: *` on an existing key, `If-Match` on a missing key and `If-Match` over a delete marker all returned 200. The controller now parses both headers and passes them to the backend through new `IfMatch` and `IfNoneMatch` fields on `s3response.CopyObjectInput`, which match the SDK's `s3.CopyObjectInput`.
As in S3, the destination is evaluated before the copy source is looked up, so a failing destination precondition wins over a missing source, a failing `x-amz-copy-source-*` precondition and an invalid copy onto itself. Posix checks the destination with `checkPutPreconditions` right after the destination bucket check, then forwards the headers to `PutObject`, which repeats the check under the object publish lock. The unversioned in-place self copy does not go through `PutObject`, so it now takes `lockObjectPublish` and repeats the check itself.
`checkPutPreconditions` now treats a current delete marker as a missing key: `If-Match` returns `NoSuchKey` and `If-None-Match: *` creates a new version. Before, the hidden version's etag was evaluated, which also affected `PutObject` and `CompleteMultipartUpload`.
`EvaluateObjectPutPreconditions` now rejects unsupported headers in S3's order: both headers present, then `If-None-Match` other than `*`, then `If-Match: *`, each with `NotImplemented`. `If-Match: *` was previously compared as a literal etag. The new `NmpAdditionalMessageIfMatch` message is added, and a stray backslash is removed from the `If-None-Match` message.
Azure evaluates the destination preconditions before copying, and s3proxy forwards the headers. `s3:if-match` and `s3:if-none-match` are now populated in the policy condition context for CopyObject, as S3 does, while UploadPartCopy stays excluded.
S3 never lists an empty result for another region: a valid region
other than the endpoint's own gets 400 InvalidArgument, "Requests with
bucket-region specified must be made to the corresponding regional
endpoint", with no ArgumentValue. The gateway is a single regional
endpoint, so it now does the same instead of clearing the listing.
Match the region case-insensitively like S3, and check an empty
bucket-region= too by looking for the query argument itself, since an
empty value is not a valid AWS region either.
ListBuckets ignored bucket-region: a value that is not a region was
accepted, and a region other than the gateway one still listed every
bucket. S3 rejects an invalid value with 400 InvalidArgument, "Argument
value abc is not a valid AWS Region", and lists only the buckets of the
requested region.
Check bucket-region against the AWS region code format, always
accepting the gateway region whatever its name, and return an empty
list for any other region, since every bucket lives in the gateway
region. Enable the REST tests that cover both cases.
Fixes#1923Fixes#1930
The bucket default retention was never stored on objects: protection was computed at delete time from the live bucket rule (`CreatedAt` + period), so changing or removing the rule unlocked existing objects, `COMPLIANCE` ones included, and a rule set N days ago stopped protecting even new objects. The posix and azure backends now store the rule's retention, `{Mode, now + Days/Years}` truncated to milliseconds, as the object's own retention whenever a write sets no Object Lock parameters: `PutObject` (which also covers POST, RDMA and `CopyObject`), `CreateMultipartUpload` (the rule and start time are fixed at creation, as S3 does) and the unversioned in-place self-copy, which applied no lock settings before. As on S3, any explicit `x-amz-object-lock-legal-hold`, `ON` or `OFF`, means the default retention is not applied. The bucket-rule fallback in `checkObject` and `BucketLockConfig.CreatedAt` are removed, so objects written before this change lose bucket-derived protection, and an expired retention no longer skips the legal hold check.
`PutObjectLockConfiguration` now rejects a `DefaultRetention` with a mode but no `Days`/`Years` with `InvalidRequest` "DefaultRetention must specify a fixed retention (Days or Years) or a DefaultEventHold.", and a `<Rule>` without `DefaultRetention` with `MalformedXML`. Azure `CopyObject` with the `COPY` metadata directive no longer copies the source's `Objectretention`/`Objectlegalhold` metadata to the destination.
Browser POST uploads now validate and apply the `x-amz-object-lock-mode`, `x-amz-object-lock-retain-until-date` and `x-amz-object-lock-legal-hold` form fields. They require `s3:PutObjectRetention`/`s3:PutObjectLegalHold`, for anonymous uploads too, and return `ErrMissingObjectLockConfigurationNoSpaces` on buckets without Object Lock after authorization. A denied POST names the actions in S3's order: retention, legal hold, tagging, then `s3:PutObject`.
`NoSuchObjectLockConfiguration` now returns 404 without a trailing period. `x-amz-object-lock-retain-until-date` on HEAD and GET has millisecond precision and omits zero milliseconds, `GetObjectRetention` returns `<Retention>` with a `.000Z` date, and posix and azure `GetObject` now return the Object Lock headers.
Add request plumbing for server-side encryption with customer-provided
keys (SSE-C) and stop silently ignoring it on backends that cannot
honor it.
Previously SSE-C headers were dropped before reaching the backend, so
posix stored the data unencrypted and returned success, and later
reads without the key succeeded. Clients believed their data was
encrypted when it was not. Partial or malformed SSE-C header sets were
also accepted instead of failing with 400 as S3 does. Rejecting SSE-C
explicitly is the safe behavior until posix implements it, and azure
cannot faithfully map SSE-C onto customer-provided keys.
This addresses part of #2193: invalid/partial SSE-C headers now fail
with 400. Requiring SSE-C headers on reads of SSE-C objects depends on
posix SSE-C support and remains open.
The changes also reject unsupported SSE-S3, SSE-KMS, and DSSE-KMS
requests instead of silently accepting them. And enforcea TLS for
SSE-C by default, with --disable-sse-c-tls-enforcement as the opt-out
in case the deployment handles TLS in front of the gateway.
An upload-id-marker sent without a key-marker is ignored for the
listing, but the gateway still echoed it back in UploadIdMarker. S3
discards the parameter in that case and returns an empty
UploadIdMarker.
Clear the marker in the controller when no key-marker is given, so
every backend gets the same input and the response no longer reports
a marker that had no effect.
Fixes#2150
A DeleteObjects request whose Delete element names no objects was
answered with 200 and an empty DeleteResult. S3 rejects it with
MalformedXML, as the schema requires at least one Object.
Reject the empty list right after the body is parsed, next to the
1000 key limit check. Update the unit and integration tests that
expected the empty result and enable the REST test that covers it.
Fixes#2109
With <Quiet>true</Quiet> in the Delete request, S3 lists only the
objects it could not delete, so a request where every deletion succeeds
returns an empty DeleteResult. The request body was parsed without the
Quiet element and every deleted object was listed.
Parse Quiet with the objects and drop the Deleted entries from the
merged result when it is set; errors are still reported.
Fixes#2124
An authenticated user with write access to a single bucket could list, write and delete files outside the gateway root on the posix and scoutfs backends. The multipart `uploadId` and the `DeleteObjects` body keys were joined into filesystem paths without validation, and unlike object keys in the request path they never passed the object name validator that rejects `..`. With a `../` upload ID, `ListParts` listed the contents of an outside directory, `UploadPart` and `UploadPartCopy` wrote part files into it, `CompleteMultipartUpload` renamed and removed it, and `AbortMultipartUpload` removed it recursively. A `../` key in `DeleteObjects` removed the outside file, after which the empty-parent cleanup removed emptied directories above the root, and a `/` key removed an empty bucket's directory.
Posix now rejects any upload ID it could not have generated. Posix has always generated upload IDs with `uuid.New().String()`, so `validateUploadID` accepts only a UUID in that exact canonical form and returns `NoSuchUpload` for anything else, which is what AWS returns for a malformed upload ID. The check runs right after the bucket existence check in all five multipart operations that take an upload ID, before any path is built from it.
The `DeleteObjects` controller now checks every key with `IsObjectNameValid`, the same check already applied to request-path, copy-source and POST object keys. If any key is missing or resolves outside the bucket, the whole request is rejected with `BadRequest` before authorization, so no key in the batch is deleted. AWS also fails the whole batch when one key is invalid.
Fixes#1535Fixes#1542Fixes#1740Fixes#1776
S3 requires `Content-MD5`, an `x-amz-checksum-*` header or an `x-amz-checksum-*` trailer on every `PutObject` that carries Object Lock parameters, and rejects the upload otherwise with `InvalidRequest` "Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters". The parameters are an `x-amz-object-lock-legal-hold` header of either status, a retention mode and date, or the bucket's default retention rule, which gives even a plain upload lock parameters. A payload hash, `UNSIGNED-PAYLOAD` and signed chunks don't count as an integrity check. The gateway accepted all of these uploads.
`auth.VerifyWriteObjectLock` reads the bucket's lock configuration, rejects lock headers on a bucket without Object Lock with "Bucket is missing ObjectLockConfiguration", legal hold `OFF` included, and reports whether the write carries lock parameters. `PutObject` calls it when the request has lock headers or no integrity check, so an upload with a checksum and no lock headers pays no extra lookup. `CopyObject` and `CreateMultipartUpload` call it only for lock headers, since S3 requires no integrity check for them.
S3 applies the same rule to `UploadPart`, with "Put Part requests" in the message, when the multipart upload carries lock parameters: its own lock headers, or a default retention rule in force at `CreateMultipartUpload`, which still counts if the rule changes later. posix and azure now mark such an upload when it's created, with the `mp-object-lock` attribute and the `Mpobjectlock` sgwtmp metadata respectively, and `UploadPart` rejects a part of it that has no integrity check. The `UploadPart` controller now passes `Content-MD5` to the backend.
An anonymous `PutObject` carrying lock parameters, including a plain one into a bucket with a default retention rule, is refused with `InvalidArgument` "Put Object requests with Object Lock parameters require AWS Signature Version 4". This comes after the missing configuration and integrity checks, and before the upload itself is authorized, so a private bucket with a default retention rule refuses it too. Anonymous parts of an upload with lock parameters only need an integrity check, and browser-based `POST` uploads are unaffected, as on S3.
Anonymous uploads were authorized for `s3:PutObject` alone, so a public bucket granting only that let anonymous callers tag objects. `AuthorizePublicBucketAccess` now authorizes every action a request carries in one `VerifyPublicAccess` call. A `PutObject` with a non-empty `x-amz-tagging` or a browser-based `POST` with a non-empty tag set needs `s3:PutObjectTagging` as well as `s3:PutObject`. The shared `auth.ObjectUploadActions` derives these actions for both anonymous and signed uploads. Each action is granted on its own, by the bucket policy or, failing that, the bucket ACL, whose WRITE grant covers `s3:PutObject` only, and an explicit public Deny of any one of them denies the request.
Anonymous requests naming an object version were authorized for the plain action. A non-empty `versionId` now makes the public access check authorize the version action instead: `s3:GetObjectVersion` rather than `s3:GetObject` for `GetObject` and `HeadObject`, `s3:DeleteObjectVersion` for `DeleteObject`, `s3:GetObjectVersionAttributes` for `GetObjectAttributes`, and the version tagging actions for the tagging APIs. A public ACL READ grant covers `s3:GetObjectVersion`, but a WRITE grant never covers `s3:DeleteObjectVersion`, which S3 grants to the bucket owner alone.
The "x-amz-object-lock-retain-until-date and x-amz-object-lock-mode must both be supplied" errors no longer end with a period, and the anonymous `CopyObject` and `CreateMultipartUpload` rejections have two spaces before "Please authenticate.", both matching S3.
ListObjects omitted the Marker element when the request had no marker.
S3 returns an empty Marker element in this case, allowing clients to read
the field without a presence check.
The backend converts an empty marker to a nil pointer and encoding/xml omits
nil pointers. Restore the request marker in the controller after the backend
call so the behavior is consistent across backends.
Update the controller, integration, and REST coverage to verify that an
empty Marker element is returned.
Fixes#2191
`POSTObject` authorized only `s3:PutObject`, even when the form's `tagging` field set tags on the uploaded object. `PutObject`, `CopyObject` and `CreateMultipartUpload` also require `s3:PutObjectTagging` when tags are sent, so a caller without that permission could tag objects through `POST` in a way `PutObject` refuses. S3 enforces the permission for `POST` too: a user granted only `s3:PutObject` gets `AccessDenied` for `s3:PutObjectTagging` on the object ARN.
`POSTObject` now adds `s3:PutObjectTagging` to the `verifyAccess` actions when the parsed tag set is not empty, in the same single check as `s3:PutObject`, matching `PutObject`. A `tagging` field with an empty `TagSet` still needs only `s3:PutObject`, as on S3.
The access check now runs after the form is validated, instead of before the `POST` policy is parsed. S3 validates the policy and its conditions, the tagging, the checksum fields, the metadata size and the website redirect location before it authorizes the upload, so a caller missing a permission now gets the validation error for an invalid form instead of `AccessDenied`.
* fix: reject DeleteObjects requests with more than 1000 keys
A DeleteObjects request may name at most 1000 keys; S3 rejects anything
larger with 400 InvalidRequest. The handler parsed the body and passed
every key through authorization and on to the backend, so a 1001-key
request was processed instead of being refused.
Check the count right after the body is parsed, before authorization and
before anything reaches the backend, so an over-limit batch can't be
applied partially. Exactly 1000 keys still succeed, and an empty delete
list keeps its current behavior (200 with an empty result).
Fixes#2196
* fix: return MalformedXML for an oversized DeleteObjects request
A DeleteObjects request naming more than 1000 keys is answered with
MalformedXML, not InvalidRequest. Update the check and the unit and
integration expectations.
---------
Co-authored-by: Tung Lam <lamphamabtung96@gmail.com>
`POSTObject` called `verifyAccess` without an `Object`, and `AuthorizePublicBucketAccess` took the object name from the request path, which is only `/bucket` for a `POST`. So both evaluated `s3:PutObject` against the bucket ARN `arn:aws:s3:::bucket`. `PutBucketPolicy` rejects `s3:PutObject` on a bucket resource, so no valid policy could ever match a `POST` upload. An `Allow` on `arn:aws:s3:::bucket/*` or on a key prefix never applied, and a `Deny` on a key prefix never applied either. A public-write ACL or a broad identity policy could therefore upload through `POST` to keys a `Deny` protects, which `PutObject` refuses. Identity policies evaluated by the standalone IAM service had the same mismatch.
S3 treats `POST` as an alternate form of `PUT` and authorizes it as `s3:PutObject` on the ARN of the object named by the form's `key` field. `POSTObject` now passes that `key` as `Object` to `verifyAccess`, matching `PutObject`. For anonymous `POST` requests, `AuthorizePublicBucketAccess` now uses the `key` that `AuthorizePostObject` has already parsed.
`POSTObject` wrote directly to the backend without the `CheckObjectAccess` object lock check that `PutObject`, `CopyObject` and `CompleteMultipartUpload` perform. In a versioning-disabled bucket, a browser-based `POST` could therefore silently overwrite an object protected by a legal hold, a `COMPLIANCE` retention or a `GOVERNANCE` retention. S3 has no such gap: it allows object lock only on versioned buckets, where an overwrite creates a new version.
`POSTObject` now calls `auth.CheckObjectAccess` with `BypassOverwrite` just before writing to the backend, matching `PutObject`. A legal hold or `COMPLIANCE` retention always blocks the overwrite. A `GOVERNANCE` retention blocks it unless the caller, root included, has an explicit `s3:BypassGovernanceRetention` grant. Versioning-enabled buckets skip the check because the upload creates a new version.
Fixes#2239
Running the gateway with a `--region` other than `us-east-1` turned every request from a client signed for a different region into a dead end. The signing region check correctly rejected the request with `AuthorizationHeaderMalformed`, but the gateway gave the client no usable way to learn which region it should have signed for, so the aws sdks could not re-sign and retry the way they do against a real s3 regional endpoint.
The gap was `x-amz-bucket-region`. S3 reports the expected region in that response header as well as in the error body, because a `HEAD` request carries no body for the client to parse. The gateway set the header only on successful responses, so `HeadBucket` and `HeadObject` signed for the wrong region came back as a bare `400` with an empty body and some sdks region redirect had nothing to work with. Region mismatches now report the gateway region through a `RegionMismatchError` interface that `MalformedAuthError`, `AuthQueryParamError` and `InvalidArgumentError` implement, and the error dispatch points set the header from it alongside the existing `Allow` header handling for `MethodNotAllowed`.
The error bodies were incomplete in the same way. Only the `Authorization` header path carried a `Region` element, while the presigned url and POST form paths reported the mismatch in prose alone. `AuthorizationQueryParametersError` is now backed by a dedicated `AuthQueryParamError` type that adds the element, and `InvalidArgumentError` grew an optional `Region` field for the POST form credential error, both omitted for every other error of those kinds.
The messages themselves now read the way s3 writes them. The region and service names are quoted with apostrophes instead of `%q`, whose double quotes the xml encoder rendered as `"` in the raw response, and the POST form authentication fields are echoed back in `ArgumentName` under their canonical spelling, for example `X-Amz-Credential` rather than the lowercase form the browser submits, while fields outside the signing protocol such as `key` are still reported as submitted.
`ParseContentEncoding` dropped the `aws-chunked` token only for streaming payload types, so a presigned request kept it: such a request is signed as `UNSIGNED-PAYLOAD` and sends no `x-amz-content-sha256` header, and S3 strips the token there. Keep it only when the request declares a non streaming payload type.
S3 applies this to the `Content-Encoding` header before any API sees it, so `CopyObject` and `CreateMultipartUpload` follow the same rule even though neither carries a body to frame. `POSTObject` takes its value from a form field rather than the header and is left alone.
- Replace `StripAwsChunkedEncoding` with `ParseContentEncoding`, which drops
the token only when `x-amz-content-sha256` names a streaming payload type,
so a client that sends `aws-chunked` on a hex-payload request keeps it
- Revert the `CopyObject` and `CreateMultipartUpload` call sites: both routes
are registered with `streamBody` false, so a streaming payload type is
rejected before the controller and the token can only be a stored value
- Reject `aws-chunked` combined with `UNSIGNED-PAYLOAD` in the authentication
middleware, beside the existing payload-type validation, with a new
`InvalidArgAwsChunkedUnsignedPayload`
- Add REST tests for the three cases: stripped from a chunked upload, kept on
a hex-payload request, and rejected with `UNSIGNED-PAYLOAD`
- Add `CONTENT_ENCODING` to the PutObject REST script and a
`check_content_encoding` driver for HeadObject
X-Amz-Decoded-Content-Length describes decoded aws-chunked payloads and must
not control a plain request. Using it for a plain UploadPart lets a larger
decoded header drive backend preallocation; the posix multipart path does not
truncate the staged part, so completion publishes a zero-filled tail.
Select the decoded length only for aws-chunked readers. Plain PutObject and
UploadPart now use Content-Length for both short-body validation and backend
allocation.
* fix(s3api): reject a PUT whose body ends before Content-Length
With a plain (non-aws-chunked) request body, a PutObject whose body ends
before Content-Length bytes have arrived is committed as a complete but
shorter object. Real S3 rejects this with IncompleteBody and the object
never becomes visible.
Nothing on that path compares bytes received against Content-Length:
fasthttp reports a connection closed mid-body as a plain io.EOF (the
conversion to io.ErrUnexpectedEOF exists only on the chunked
transfer-encoding branch), the authentication middleware leaves an
UNSIGNED-PAYLOAD body unwrapped, and io.Copy treats io.EOF as a normal
end of stream.
ErrIncompleteBody already exists and is enforced by the chunk readers.
This adds the equivalent check for plain bodies, in the controller so
that every backend is covered, and only when the body is not already an
aws-chunked reader.
POST-Object is deliberately untouched: it is a separate handler whose
ContentLength is an upper bound, so a byte-count check there would
break browser form uploads.
* test(s3api): cover ContentLengthReader
Complete and empty bodies, truncated bodies (including an EOF delivered
together with the final bytes, and one byte per Read), a body longer
than announced, and pass-through of a non-EOF error.
* fix(s3api): check plain bodies against Content-Length, not the decoded length
Review catch: the controller replaces contentLength with
X-Amz-Decoded-Content-Length whenever that header is present, regardless
of payload type, and the previous commit fed that value to
ContentLengthReader. A complete plain upload whose decoded header is
larger than Content-Length was then rejected with IncompleteBody, where
it had succeeded before.
AWS S3 ignores X-Amz-Decoded-Content-Length on a plain body and stores
Content-Length bytes. The decoded length only describes aws-chunked
payloads, and those skip this wrapper anyway, so the check now reads the
raw Content-Length header.
If the header is missing or unparseable the body is left unwrapped: an
aws-chunked request without Content-Length is already rejected earlier
with ErrMissingContentLength, so there is nothing to check here.
* test(integration): cover aborted uploads and the decoded-length case
Three tests, as requested in review.
PutObject_aborted_plain_body plain PUT, body ends early -> no object
PutObject_aborted_streaming_body same for an aws-chunked upload
PutObject_plain_body_with_decoded_length
complete plain PUT carrying
X-Amz-Decoded-Content-Length still
succeeds and stores Content-Length bytes
The abort is driven by a reader that fails partway, so the transport
tears the connection down mid-body - that is what a client that dies or
cancels looks like on the wire. Sending a short body with a normal
reader would not reproduce it: net/http would simply report the
mismatch itself.
The third test is the regression the review found. It fails without the
accompanying fix.
* test(integration): replace the bogus streaming abort test with a real one
The reviewer is right: putObjectAborted always sends
x-amz-content-sha256: UNSIGNED-PAYLOAD, and the gateway decides aws-chunked
from that header alone, so PutObject_aborted_streaming_body took the same
plain path as the test above it. It duplicated the plain test instead of
covering the streaming one.
Dropped it and added UnsignedStreamingPayloadTrailer_aborted_connection in
tests/integration/unsigned_streaming_payload_trailer.go, where it belongs.
UnsignedStreamingPayloadTrailer_incomplete_body already covers malformed and
truncated framing, but every case there is a COMPLETE request. The new test is
the other shape: valid framing whose bytes simply stop arriving.
* fix: return InvalidArgument for an empty website configuration
`PutBucketWebsite` parsed an empty `<WebsiteConfiguration/>` into a zero-value config, and `WebsiteConfiguration.Validate` reported the nil `IndexDocument` as `MalformedXML`. S3 reports the missing index document as an `InvalidArgument` request error naming the argument instead:
InvalidArgument: A value for IndexDocument Suffix must be provided if RedirectAllRequestsTo is empty
That branch now returns the error with `ArgumentName=IndexDocument` and `ArgumentValue=null`, matching the response in the report. Genuinely malformed XML, and a `RedirectAllRequestsTo` that conflicts with the other fields, still return `MalformedXML`.
Fixes#2260
* test: cover an empty website configuration in the PutBucketWebsite integration suite
Add a `PutBucketWebsite_empty_configuration` case that sends the empty `<WebsiteConfiguration/>` payload from the report and asserts the `InvalidArgument` response, including `ArgumentName=IndexDocument` and `ArgumentValue=null`.
The SDK collapses the error into a generic API error that drops the argument fields, so the request is signed by hand and the raw response checked with `checkHTTPResponseApiErr`, the same helper the other argument-field assertions use. The case fails against the previous behaviour with `expected error code to be InvalidArgument, instead got MalformedXML`.
---------
Co-authored-by: Tung Lam <lamphamabtung96@gmail.com>
* fix: omit ObjectLockConfiguration Rule when there is no default retention
ParseBucketLockConfigurationOutput always set Rule, so a bucket with object lock enabled and no default retention answered GET ?object-lock with an empty <Rule></Rule>. AWS S3 omits the element, and the AWS SDK v2 clients that read the rule from it follow the empty element with a malformed request.
Fixes#2397
* test: cover GetObjectLockConfiguration with no default retention in the integration suite and drop the stale non-nil Rule expectation from the controller unit test
---------
Co-authored-by: Tung Lam <lamphamabtung96@gmail.com>
S3 returns `InvalidArgument` when an object action receives a `versionId` query parameter with no value. The gateway silently treated it as an unversioned request instead.
Added a shared `versionId` validation helper and apply it to the object actions that accept the parameter, so malformed requests are rejected up front rather than reaching the backend.
Service the overflow list during normal operation: the worker
publishes and clears it after every channel job, so a burst that
exceeds the channel buffer drains as soon as the sink recovers
instead of accumulating until shutdown.
Make dispatch and the drain sweep share one critical section. An
append either lands before the sweep and is drained, or runs
after the worker exited and publishes inline; the check-then-send
window that could strand a record between the two is closed.
Request publications use the same boundary with drop semantics:
once the drain finished, no owner can guarantee the sinks are
still open, so the record is dropped rather than published.
Never close the metrics datapoint channel. The forwarder exits
through the canceled context after draining the buffer, the
closed flag turns late producers into drops, and no send can race
a closure. Implement the new Manager method in the test mock.
A new glaicer mode for the archiving backend needs the storage
class supplied to the backend for put object, create multipart
upload, and browser post object input.
S3 explicitly documents this action as owner-only ("To retrieve the versioning state of a bucket, you must be the bucket owner." — https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketVersioning.html), and the handler enforced that with an extra auth.IsAdminOrOwner check on top of VerifyAccess. Real S3 behaves differently: verified against AWS that a bucket policy explicitly denying s3:GetBucketVersioning denies the bucket owner itself, and that an Allow grants the action to a principal that doesn't own the bucket. It goes through ordinary bucket policy/ACL evaluation like any other bucket subresource read, which is what the write side, PutBucketVersioning, already did here. Removes the extra check along with auth.IsAdminOrOwner, which had no other call site.
Ben McClellandandGitHub
niksis02
LuisFigueroaG
Javier GarciaandBen McClelland
dualfroz
Lennard GeißlerandBen McClelland
Barry Loongandniksis02
DavvyyandGitHub
Jihyeon Gim