A conditional delete of a key with no current version used to succeed. `DeleteObject` with `If-Match`, and a `DeleteObjects` key with an `ETag`, reported success for a missing key. When the current version was a delete marker, the condition was checked against the ETag the marker kept from the version it hides, so it could pass and create a second marker. S3 fails both with `NoSuchKey` and creates no marker. The posix backend now checks the conditions against the current version in `checkDeletePreconditions`, before its missing-key shortcuts. The azure backend returns `NoSuchKey` for a missing blob instead of skipping the conditions. `ObjectDeletePreconditions.IsSet` reports whether a delete has any condition.
A condition only applies to the current version. `DeleteObject` with a `versionId` and `If-Match` now returns `NotImplemented` with `<Header>If-Match</Header>` and the S3 additional message, ahead of authorization. Before, it checked the condition against that version and deleted the version. In `DeleteObjects`, an empty `<ETag>` fails its key with `InvalidArgument` and a `VersionId` together with an `ETag` fails it with `NotImplemented`, both ahead of the key's authorization.
The `PreconditionFailed` and `NotImplemented` messages drop their trailing period, which S3 doesn't use on any operation.
DeleteObjects dropped the ETag, Size and LastModifiedTime of each object
when it called DeleteObject, so an object with a stale ETag was still
deleted. The posix and azure backends now forward them to DeleteObject.
EvaluateObjectDeletePreconditions trims the quotes an SDK sends around
the ETag and treats "*" as a match for any ETag.
The REST ETag mismatch test no longer skips.
Taken from #2485, without its posix and integration tests.
Refs #2112
Fixes#2499
An empty tag key returned `InvalidTag` without the `TagKey` element because the field was `omitempty`. S3 always sends `<TagKey>`, empty for an empty key, so it is now always written.
`PutBucketTagging`, `PutObjectTagging`, the POST object `tagging` field and `CreateBucket` tags now share one validator, `backend.ParseTagSet`, which matches S3: a duplicate key error carries `TagKey`, a `Tag` missing its `Key` or `Value` element is `MalformedXML` and is checked across the whole document first, every tag is validated before the tag count limit, and the 128/256 length limits count UTF-16 code units instead of bytes. The duplicate tag regex in `s3api/utils` is removed.
For the `x-amz-tagging` header, the length limits now apply to the URL-decoded key and value, the `InvalidArgument` error echoes the whole header as `ArgumentValue` instead of the unparsed remainder, and more than 10 tags is rejected with `BadRequest`.
Fixes#2489
An unsupported `Authorization` algorithm returned only the algorithm in `ArgumentValue`, while AWS S3 echoes the whole header as received. `sigv4auth.ParseAuthorization` now carries the full header in `ParseError.Input`, which the S3 error mapper returns. Also drop the trailing period from the `InvalidArgAuthHeader` message to match AWS.
The azure backend tracks zero-byte parts separately and reports its own
block id as the ETag when the client ETag doesn't match, so the test
failed against azurite. Upload some data so the part is a regular block,
and send the ETag unquoted so both backends echo it back as sent.
Once a bucket had any policy, `verifyResourceAccess` evaluated the policy alone and ignored the bucket ACL, so a non-root bucket owner kept only what the policy explicitly allowed. A Deny-only policy (e.g. Deny `*` `s3:GetObject` on one prefix) therefore locked the owner out of every operation, including `DeleteBucketPolicy`, and only root/admin could recover. ACL grants also authorized every bucket operation through the controller's `AclPermission`, so a non-owner with a `WRITE` grant could install a bucket policy that locked out the owner and every other grantee.
The policy and the ACL are now combined per action the way AWS does it: an explicit policy Deny wins, and otherwise the action is allowed by either a policy Allow or an ACL grant. The bucket owner keeps everything a policy doesn't explicitly deny and, like an AWS account root user, can always call `GetBucketPolicy`, `PutBucketPolicy` and `DeleteBucketPolicy` even past a Deny. The protection covers only those three APIs; `GetBucketPolicyStatus` and every other action stay deniable, matching AWS.
ACL grants to non-owners now follow AWS's ACL-to-action mapping through the new `aclActionPermissions` table: `READ` grants `s3:ListBucket`, `s3:ListBucketVersions`, `s3:ListBucketMultipartUploads`, `s3:GetObject`, `s3:GetObjectVersion` and `s3:GetObjectAttributes`; `WRITE` grants `s3:PutObject` and `s3:DeleteObject`; `READ_ACP` and `WRITE_ACP` grant reading and writing the bucket and object ACLs. Bucket configuration, the bucket policy, tagging, object lock settings and `DeleteBucket` are the owner's alone unless a policy grants them. `verifyACL` is replaced by `aclAllows`, which derives the required permission from the action, so `AccessOptions.AclPermission` now only feeds the readonly gate. The anonymous path uses the same table in place of `publicACLAllowedActions`, which leaves its behavior unchanged.
`GetBucketPolicyStatus` also reported a Deny-only `Principal: "*"` policy as public; only Allow statements now count toward `IsPublic`.
In posix, when the upload directory rename in `CompleteMultipartUploadWithCopy` fails with `ENOENT`, the call is treated as a retry of an already completed upload. That fallback returned 200 whenever any regular file existed at the target bucket/key, with an ETag computed from the request's part ETags, and never checked that the object came from this upload. Completing with the upload ID of another bucket or key, a random upload ID, or an aborted upload was reported as stored while nothing was written. The later `mp-metadata` upload ID check was unreachable whenever the object existed.
The fallback now succeeds only when the object at the key carries this upload ID in its `mp-metadata` attribute and, outside `--data-integrity-etag` mode, when its stored ETag equals the multipart ETag computed from the request's parts. This matches AWS, which rejects a retry with different part ETags with `NoSuchUpload` but ignores part numbers. Every other case returns `NoSuchUpload`, as AWS does for a foreign, random or aborted upload ID and for a retry after the object was overwritten or deleted. Keys with a trailing slash are rejected up front because `ObjectPath` resolves `foo/` to the file `foo`, and attribute read errors other than a missing attribute are no longer reported as `NoSuchUpload`. A successful retry now returns the stored ETag and the object's `version-id`, matching AWS, which returns the original VersionId on a retry in a versioned bucket.
The posix ListObjectsV2 reported the listing marker as the response
ContinuationToken. The marker is the larger of start-after and the
continuation token, so a request with only start-after got its
start-after value back as a ContinuationToken, and a start-after past
the continuation token replaced it. S3 echoes the continuation token
from the request, and leaves ContinuationToken out when the request
had none.
Return the request continuation token instead of the marker, and
enable the REST test that covers it.
Fixes#2007
CompleteMultipartUpload with a part ETag that is not a hex MD5 returned
a bare InvalidPart error, without the UploadId, PartNumber and ETag
elements. S3 includes them, and the gateway already does for a part
that is missing or whose ETag doesn't match.
Pass the upload ID to ComputeMultipartETagFromPartETags so it can
return the detailed InvalidPart error for the offending part, and
enable the REST test that covers it.
Fixes#1008
CopyObject ignored the destination's `If-Match` and `If-None-Match` headers and always overwrote or created the destination, so a stale `If-Match`, `If-None-Match: *` on an existing key, `If-Match` on a missing key and `If-Match` over a delete marker all returned 200. The controller now parses both headers and passes them to the backend through new `IfMatch` and `IfNoneMatch` fields on `s3response.CopyObjectInput`, which match the SDK's `s3.CopyObjectInput`.
As in S3, the destination is evaluated before the copy source is looked up, so a failing destination precondition wins over a missing source, a failing `x-amz-copy-source-*` precondition and an invalid copy onto itself. Posix checks the destination with `checkPutPreconditions` right after the destination bucket check, then forwards the headers to `PutObject`, which repeats the check under the object publish lock. The unversioned in-place self copy does not go through `PutObject`, so it now takes `lockObjectPublish` and repeats the check itself.
`checkPutPreconditions` now treats a current delete marker as a missing key: `If-Match` returns `NoSuchKey` and `If-None-Match: *` creates a new version. Before, the hidden version's etag was evaluated, which also affected `PutObject` and `CompleteMultipartUpload`.
`EvaluateObjectPutPreconditions` now rejects unsupported headers in S3's order: both headers present, then `If-None-Match` other than `*`, then `If-Match: *`, each with `NotImplemented`. `If-Match: *` was previously compared as a literal etag. The new `NmpAdditionalMessageIfMatch` message is added, and a stray backslash is removed from the `If-None-Match` message.
Azure evaluates the destination preconditions before copying, and s3proxy forwards the headers. `s3:if-match` and `s3:if-none-match` are now populated in the policy condition context for CopyObject, as S3 does, while UploadPartCopy stays excluded.
S3 never lists an empty result for another region: a valid region
other than the endpoint's own gets 400 InvalidArgument, "Requests with
bucket-region specified must be made to the corresponding regional
endpoint", with no ArgumentValue. The gateway is a single regional
endpoint, so it now does the same instead of clearing the listing.
Match the region case-insensitively like S3, and check an empty
bucket-region= too by looking for the query argument itself, since an
empty value is not a valid AWS region either.
ListBuckets ignored bucket-region: a value that is not a region was
accepted, and a region other than the gateway one still listed every
bucket. S3 rejects an invalid value with 400 InvalidArgument, "Argument
value abc is not a valid AWS Region", and lists only the buckets of the
requested region.
Check bucket-region against the AWS region code format, always
accepting the gateway region whatever its name, and return an empty
list for any other region, since every bucket lives in the gateway
region. Enable the REST tests that cover both cases.
Fixes#1923Fixes#1930
With an upload-id-marker the lister ranges over Uploads[startIndex:],
but passed the index of that subslice to hasMoreAfter, which indexes
the full list. A page that ended on the last remaining upload was still
reported as truncated, with next markers pointing at that upload, so a
client had to make one more request to get an empty page.
Iterate over the full list from startIndex, so hasMoreAfter gets the
real position of the next upload.
* fix: reject ListMultipartUploads requests that also carry uploadId
A bucket GET with both the uploads and uploadId query parameters was
routed to ListMultipartUploads and answered with the upload listing.
S3 rejects the combination with 400 InvalidArgument, "Conflicting query
string parameters: uploadId, uploads".
Add an error route for the combination ahead of ListMultipartUploads,
the way the object level '?uploads' request is rejected, and enable
the REST test that covers it.
Fixes#2155
* fix: reject uploads with uploadId for every bucket and object operation
Requests carrying both the uploads and uploadId query parameters were
only rejected for a bucket GET, so DELETE, PUT, POST and HEAD on the
bucket or object paths still ran real operations. S3 rejects the
combination for every method on both paths, before authentication and
before checking that the bucket exists.
Register the error route for all methods on the bucket and object
routers right after the CORS middleware is set up, ahead of every other
route, and cover each method and path with an integration test.
The bucket default retention was never stored on objects: protection was computed at delete time from the live bucket rule (`CreatedAt` + period), so changing or removing the rule unlocked existing objects, `COMPLIANCE` ones included, and a rule set N days ago stopped protecting even new objects. The posix and azure backends now store the rule's retention, `{Mode, now + Days/Years}` truncated to milliseconds, as the object's own retention whenever a write sets no Object Lock parameters: `PutObject` (which also covers POST, RDMA and `CopyObject`), `CreateMultipartUpload` (the rule and start time are fixed at creation, as S3 does) and the unversioned in-place self-copy, which applied no lock settings before. As on S3, any explicit `x-amz-object-lock-legal-hold`, `ON` or `OFF`, means the default retention is not applied. The bucket-rule fallback in `checkObject` and `BucketLockConfig.CreatedAt` are removed, so objects written before this change lose bucket-derived protection, and an expired retention no longer skips the legal hold check.
`PutObjectLockConfiguration` now rejects a `DefaultRetention` with a mode but no `Days`/`Years` with `InvalidRequest` "DefaultRetention must specify a fixed retention (Days or Years) or a DefaultEventHold.", and a `<Rule>` without `DefaultRetention` with `MalformedXML`. Azure `CopyObject` with the `COPY` metadata directive no longer copies the source's `Objectretention`/`Objectlegalhold` metadata to the destination.
Browser POST uploads now validate and apply the `x-amz-object-lock-mode`, `x-amz-object-lock-retain-until-date` and `x-amz-object-lock-legal-hold` form fields. They require `s3:PutObjectRetention`/`s3:PutObjectLegalHold`, for anonymous uploads too, and return `ErrMissingObjectLockConfigurationNoSpaces` on buckets without Object Lock after authorization. A denied POST names the actions in S3's order: retention, legal hold, tagging, then `s3:PutObject`.
`NoSuchObjectLockConfiguration` now returns 404 without a trailing period. `x-amz-object-lock-retain-until-date` on HEAD and GET has millisecond precision and omits zero milliseconds, `GetObjectRetention` returns `<Retention>` with a `.000Z` date, and posix and azure `GetObject` now return the Object Lock headers.
Add request plumbing for server-side encryption with customer-provided
keys (SSE-C) and stop silently ignoring it on backends that cannot
honor it.
Previously SSE-C headers were dropped before reaching the backend, so
posix stored the data unencrypted and returned success, and later
reads without the key succeeded. Clients believed their data was
encrypted when it was not. Partial or malformed SSE-C header sets were
also accepted instead of failing with 400 as S3 does. Rejecting SSE-C
explicitly is the safe behavior until posix implements it, and azure
cannot faithfully map SSE-C onto customer-provided keys.
This addresses part of #2193: invalid/partial SSE-C headers now fail
with 400. Requiring SSE-C headers on reads of SSE-C objects depends on
posix SSE-C support and remains open.
The changes also reject unsupported SSE-S3, SSE-KMS, and DSSE-KMS
requests instead of silently accepting them. And enforcea TLS for
SSE-C by default, with --disable-sse-c-tls-enforcement as the opt-out
in case the deployment handles TLS in front of the gateway.
An upload-id-marker sent without a key-marker is ignored for the
listing, but the gateway still echoed it back in UploadIdMarker. S3
discards the parameter in that case and returns an empty
UploadIdMarker.
Clear the marker in the controller when no key-marker is given, so
every backend gets the same input and the response no longer reports
a marker that had no effect.
Fixes#2150
NextKeyMarker and NextUploadIdMarker were only set for a truncated
listing, so a ListMultipartUploads response that fit in one page
returned them empty. S3 still sets them to the key and upload ID of
the last upload in the response.
Fill both markers from the last listed upload when the listing is not
truncated, including the case where the page ends exactly at
max-uploads. A page of common prefixes only keeps them empty, as
before. Enable the REST test that covers it.
Fixes#2144
The posix and azure backends dropped every upload with a key equal to
the key-marker before handing the list to the lister, which then looked
for the upload-id-marker under the next key. An upload-id-marker that
belongs to the key-marker could never be found, so the request failed
with "Invalid uploadId marker", including when a client paged with the
NextKeyMarker and NextUploadIdMarker the gateway returned.
Keep the uploads of the key-marker itself when an upload-id-marker is
set, so the listing continues right after the given upload, as S3 does.
Enable the REST test that covers it.
Fixes#2149
A DeleteObjects request whose Delete element names no objects was
answered with 200 and an empty DeleteResult. S3 rejects it with
MalformedXML, as the schema requires at least one Object.
Reject the empty list right after the body is parsed, next to the
1000 key limit check. Update the unit and integration tests that
expected the empty result and enable the REST test that covers it.
Fixes#2109
A request path with an invalid percent escape, such as "/bucket/%zz",
failed url.PathUnescape in the DecodeURL middleware and the raw error
was answered with 500 InternalError. S3 rejects such a request with
400 InvalidURI.
Return InvalidURI from DecodeURL, so the s3 and admin routers answer
with it as well, and let the website handler pass the error through
instead of mapping it on its own.
Fixes#2070
Object routes match subresources with `MatchQueryArgs`, and a request whose subresource has no route falls through to the catch-all `PutObject`, `GetObject` and `DeleteObject` handlers. As a result `PutObjectAnnotation` overwrote the object with the annotation payload or created a missing key, `DeleteObjectAnnotation` deleted the object or permanently deleted the version named by `versionId`, `RenameObject` replaced the destination with an empty object, and `PUT`/`DELETE ?torrent` overwrote or deleted the object.
`PUT ?annotation`, `GET ?annotation&annotationName`, `GET ?annotation` and `DELETE ?annotation` now return `NotImplemented` and follow the existing not-implemented route pattern: `BucketObjectNameValidator`, `AuthorizePublicBucketAccess`, signature verification and `ParseAcl`. Each route has its own metrics action, and the new `s3:PutObjectAnnotation`, `s3:GetObjectAnnotation`, `s3:ListObjectAnnotations` and `s3:DeleteObjectAnnotation` policy actions are also accepted in bucket policies.
`PUT ?renameObject` now returns `NotImplemented`. `?torrent` returns `MethodNotAllowed` with the new `TORRENT` resource type on every method, with `Allow: GET` on all methods except `GET`. Both are answered before authentication, as AWS does, and there is no `s3:RenameObject` policy action.
Creating a delete marker in posix keeps the file of the hidden version at the object path and only marks it with the delete marker attribute, so the delete marker kept the modification time of the version it hides. `ListObjectVersions` listed that time for current and noncurrent delete markers, and `HeadObject` and `GetObject` sent it as the `Last-Modified` header of the `MethodNotAllowed` response for a delete marker `versionId`. AWS reports the time the delete marker is created, including for the null delete marker created while versioning is suspended. `DeleteObject` now sets the modification time of the delete marker to the time it's created. Setting the time needs the entry to be owned by the gateway, so it is left as is otherwise, like it is for directory objects.
`ListObjectVersions` also listed the delete markers with the SDK `types.DeleteMarkerEntry`, which has no XML marshaling of its own, so `LastModified` was sent in the local time zone of the gateway and with a sub-second part, while the versions are sent in UTC to the second. The delete markers are now listed with the new `s3response.DeleteMarkerEntry`, which formats `LastModified` the same way as `s3response.ObjectVersion`, and the s3proxy backend converts the upstream delete markers to it.
Deleting the current version of an object by `versionId` makes the newest remaining version current again. Posix copies that version from the versioning directory into a new temporary file and links it in place of the object, but never restored the file's modification time, so `HeadObject`, `ListObjectVersions` and `ListObjectsV2` reported the delete time as the promoted version's `LastModified`. This affected an older version, a version under a deleted delete marker and the null version alike, while AWS keeps the original `LastModified` in all three cases. Directory objects were not affected, as `deleteDirObjectLatestVersion` already restores the directory's mtime.
The temporary file now carries an optional modification time, set with `setModTime` and applied by `link()` right before the file is published: through `/proc/self/fd` before `linkat` for an `O_TMPFILE` file, and after the file is closed and before the rename on the `CreateTemp` fallback and on non-Linux platforms. Both the promotion in `DeleteObject` and `createObjVersion` use it. `createObjVersion` used to call `os.Chtimes` on the temporary file while it was still open for writing, and Windows resets the last write time when a handle the file was written through is closed.
With <Quiet>true</Quiet> in the Delete request, S3 lists only the
objects it could not delete, so a request where every deletion succeeds
returns an empty DeleteResult. The request body was parsed without the
Quiet element and every deleted object was listed.
Parse Quiet with the objects and drop the Deleted entries from the
merged result when it is set; errors are still reported.
Fixes#2124
The `sidecar` metadata storer keeps attributes by path rather than on the file. So a new file published at an object's path inherited every attribute it didn't set itself: tags, user metadata, content headers and `x-amz-website-redirect-location`, plus legal hold and retention on version promotion. This hit overwrites by `PutObject`, `CopyObject` and `CompleteMultipartUpload`, new versions and `PutObject` over a delete marker in versioned buckets, and the version made current again by `DeleteObject` with a `versionId`. A plain `PutObject` over a multipart object also kept its `mp-metadata`, so `HeadObject` and `GetObject` with `partNumber` returned the old `PartsCount` and a `Content-Range` beyond the object size. `xattr` was not affected, since its attributes live on the inode and are replaced along with it.
Add `ReplaceObject(bucket, object, keep)` to `MetadataStorer`. It is a no-op for `XattrMeta` and `NoMeta`. `SideCar` removes the object's attribute files not named in `keep` and leaves directories in place, since they hold the metadata of objects under `<object>/meta/`. Posix calls it in `PutObject` and `CompleteMultipartUpload` after the version snapshot, and in `DeleteObject` when restoring the latest remaining version. This replaces the scattered deletes of the delete marker, `versionId` and object lock attributes. Attributes that every publish rewrites are kept and overwritten in place: the etag, the checksums, the new `versionId`, `mp-metadata` for multipart uploads, and the restored version's attributes on promotion. This way concurrent readers, such as the lock-free `If-Match` fast-fail check, never see them missing.
`PutObject` now re-checks under the publish lock that the object path hasn't become a directory while the body was staged, as `CompleteMultipartUpload` already does. This stops it from removing the attributes of a directory object created by a racing `PUT foo/`.
An authenticated user with write access to a single bucket could list, write and delete files outside the gateway root on the posix and scoutfs backends. The multipart `uploadId` and the `DeleteObjects` body keys were joined into filesystem paths without validation, and unlike object keys in the request path they never passed the object name validator that rejects `..`. With a `../` upload ID, `ListParts` listed the contents of an outside directory, `UploadPart` and `UploadPartCopy` wrote part files into it, `CompleteMultipartUpload` renamed and removed it, and `AbortMultipartUpload` removed it recursively. A `../` key in `DeleteObjects` removed the outside file, after which the empty-parent cleanup removed emptied directories above the root, and a `/` key removed an empty bucket's directory.
Posix now rejects any upload ID it could not have generated. Posix has always generated upload IDs with `uuid.New().String()`, so `validateUploadID` accepts only a UUID in that exact canonical form and returns `NoSuchUpload` for anything else, which is what AWS returns for a malformed upload ID. The check runs right after the bucket existence check in all five multipart operations that take an upload ID, before any path is built from it.
The `DeleteObjects` controller now checks every key with `IsObjectNameValid`, the same check already applied to request-path, copy-source and POST object keys. If any key is missing or resolves outside the bucket, the whole request is rejected with `BadRequest` before authorization, so no key in the batch is deleted. AWS also fails the whole batch when one key is invalid.
Fixes#1535Fixes#1542Fixes#1740Fixes#1776
S3 requires `Content-MD5`, an `x-amz-checksum-*` header or an `x-amz-checksum-*` trailer on every `PutObject` that carries Object Lock parameters, and rejects the upload otherwise with `InvalidRequest` "Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters". The parameters are an `x-amz-object-lock-legal-hold` header of either status, a retention mode and date, or the bucket's default retention rule, which gives even a plain upload lock parameters. A payload hash, `UNSIGNED-PAYLOAD` and signed chunks don't count as an integrity check. The gateway accepted all of these uploads.
`auth.VerifyWriteObjectLock` reads the bucket's lock configuration, rejects lock headers on a bucket without Object Lock with "Bucket is missing ObjectLockConfiguration", legal hold `OFF` included, and reports whether the write carries lock parameters. `PutObject` calls it when the request has lock headers or no integrity check, so an upload with a checksum and no lock headers pays no extra lookup. `CopyObject` and `CreateMultipartUpload` call it only for lock headers, since S3 requires no integrity check for them.
S3 applies the same rule to `UploadPart`, with "Put Part requests" in the message, when the multipart upload carries lock parameters: its own lock headers, or a default retention rule in force at `CreateMultipartUpload`, which still counts if the rule changes later. posix and azure now mark such an upload when it's created, with the `mp-object-lock` attribute and the `Mpobjectlock` sgwtmp metadata respectively, and `UploadPart` rejects a part of it that has no integrity check. The `UploadPart` controller now passes `Content-MD5` to the backend.
An anonymous `PutObject` carrying lock parameters, including a plain one into a bucket with a default retention rule, is refused with `InvalidArgument` "Put Object requests with Object Lock parameters require AWS Signature Version 4". This comes after the missing configuration and integrity checks, and before the upload itself is authorized, so a private bucket with a default retention rule refuses it too. Anonymous parts of an upload with lock parameters only need an integrity check, and browser-based `POST` uploads are unaffected, as on S3.
Anonymous uploads were authorized for `s3:PutObject` alone, so a public bucket granting only that let anonymous callers tag objects. `AuthorizePublicBucketAccess` now authorizes every action a request carries in one `VerifyPublicAccess` call. A `PutObject` with a non-empty `x-amz-tagging` or a browser-based `POST` with a non-empty tag set needs `s3:PutObjectTagging` as well as `s3:PutObject`. The shared `auth.ObjectUploadActions` derives these actions for both anonymous and signed uploads. Each action is granted on its own, by the bucket policy or, failing that, the bucket ACL, whose WRITE grant covers `s3:PutObject` only, and an explicit public Deny of any one of them denies the request.
Anonymous requests naming an object version were authorized for the plain action. A non-empty `versionId` now makes the public access check authorize the version action instead: `s3:GetObjectVersion` rather than `s3:GetObject` for `GetObject` and `HeadObject`, `s3:DeleteObjectVersion` for `DeleteObject`, `s3:GetObjectVersionAttributes` for `GetObjectAttributes`, and the version tagging actions for the tagging APIs. A public ACL READ grant covers `s3:GetObjectVersion`, but a WRITE grant never covers `s3:DeleteObjectVersion`, which S3 grants to the bucket owner alone.
The "x-amz-object-lock-retain-until-date and x-amz-object-lock-mode must both be supplied" errors no longer end with a period, and the anonymous `CopyObject` and `CreateMultipartUpload` rejections have two spaces before "Please authenticate.", both matching S3.
ListObjects omitted the Marker element when the request had no marker.
S3 returns an empty Marker element in this case, allowing clients to read
the field without a presence check.
The backend converts an empty marker to a nil pointer and encoding/xml omits
nil pointers. Restore the request marker in the controller after the backend
call so the behavior is consistent across backends.
Update the controller, integration, and REST coverage to verify that an
empty Marker element is returned.
Fixes#2191
In a bucket with versioning `Suspended`, `CompleteMultipartUpload` only archived the object it replaced when versioning was `Enabled`, so completing an upload over a versioned object overwrote it without a snapshot and that version was lost. The null version in the versioning directory was not removed either, which left a stale `null` version next to the new one. With sidecar metadata, the leftover `versionIdKey` attribute also made the new object report the overwritten version's id.
With versioning `Enabled` and sidecar metadata, the upload's metadata was stored at the object path before the replaced object was archived, so `createObjVersion` copied the new upload's content type, user metadata and other properties into the previous version.
`CompleteMultipartUpload` now archives the replaced object with `snapshotObjVersion` before storing any metadata of the new object, which covers both `Enabled` and `Suspended` buckets. In a `Suspended` bucket it also removes the archived null version and clears the stale `versionIdKey` attribute, matching `PutObject` and S3 behavior.
Some requests need more than one S3 action authorized on the same object. `PutObject`, `CopyObject` and `CreateMultipartUpload` add `s3:PutObjectTagging`, `s3:PutObjectLegalHold` or `s3:PutObjectRetention` depending on the headers sent, and a browser-based `POST` with a `tagging` field adds `s3:PutObjectTagging`. The gateway allowed such a request only when a single policy source allowed every action, because `verifyBucketPolicy` and `aggregateActionDecisions` each reduced their side to one verdict before `objectsAccessErrors` combined them. S3 authorizes each action separately. As a result, a user granted `s3:PutObject` by the bucket policy and `s3:PutObjectTagging` by their identity policy was refused a tagged `PutObject` that S3 allows, and the denial message wrongly said no identity-based policy allows `s3:PutObjectTagging`. Only gateways backed by the standalone IAM service (`--iam-standalone-endpoint`) were affected.
`verifyResourceAccess` now returns one decision per action for each key. It computes them with `decisionFor` and parses the bucket policy once per request instead of once per key. `identityPolicyDecisions` now returns the `EvaluatePolicy` result without reducing it. `objectsAccessErrors` combines the two sources action by action. An explicit deny from the bucket, identity or session policy still wins and is reported as before. Otherwise an action is allowed when either source allows it, and, when a session policy is present, the session policy must also allow it. A request with any action not allowed is denied, and the denial names the first such action in list order, as S3 does. Each request still fetches the bucket policy once and makes one `EvaluatePolicy` round trip, and `VerifyObjectsAccess` still makes one round trip per action group.
niksis02
theluckystrikeandniksis02
Ben McClellandandGitHub
LuisFigueroaG
Luke McCrone
DualFroz
Javier Garcia
Lennard GeißlerandBen McClelland