Fixes#2513
On Windows the posix backend stored a key as `bucket/key` verbatim, but Windows rejects `" * : < > ? \ |` and control characters in file names, so such a PUT failed with `ERROR_INVALID_NAME`, which `isErrNameTooLong` reported as a misleading `KeyTooLongError`. The same mapping did worse elsewhere: `\` acted as a path separator, so `a\b` aliased `a/b` and `..\..\x` resolved outside the bucket because `IsObjectNameValid` only splits on `/`; `a:b` wrote an NTFS alternate data stream; Win32 strips a trailing dot or space, so `foo.` overwrote `foo`; and device names such as `CON` or `nul.txt` opened the device and silently discarded the data.
With `encodeKeys`, enabled by default only on Windows, `encodeKey` stores each key segment under a name Windows accepts: every invalid character, a trailing dot or space, and the first letter of a device name are stored as the private use character `U+F000` plus the character code, the convention Cygwin and WSL use for invalid characters. A key character in `U+F000`-`U+F07F` is stored after `U+F000`, so every stored name belongs to exactly one key. Device names are checked with `filepath.IsLocal`, which asks the system, so `CON.txt` stays a plain file on Windows 11 where it is valid. Keys that need no encoding keep their names, so existing data is unaffected, and nothing changes on other platforms.
Public backend methods keep taking raw keys and encode them once for every path, metadata, version-hash and publish-lock use, while responses, errors and the multipart `objname` attribute keep the raw key and multipart upload directories keep hashing it. Listings walk `bucketFS`, which decodes the entry names, sorts them in key byte order to match S3 and skips names no key is stored under. `HeadObject` passes the raw key to `GetObjectLegalHold` and `GetObjectRetention` to avoid encoding twice, and `NormalizeObjectKey` resolves the encoded key, so policy resources see `a\b` as one literal key.
A conditional delete of a key with no current version used to succeed. `DeleteObject` with `If-Match`, and a `DeleteObjects` key with an `ETag`, reported success for a missing key. When the current version was a delete marker, the condition was checked against the ETag the marker kept from the version it hides, so it could pass and create a second marker. S3 fails both with `NoSuchKey` and creates no marker. The posix backend now checks the conditions against the current version in `checkDeletePreconditions`, before its missing-key shortcuts. The azure backend returns `NoSuchKey` for a missing blob instead of skipping the conditions. `ObjectDeletePreconditions.IsSet` reports whether a delete has any condition.
A condition only applies to the current version. `DeleteObject` with a `versionId` and `If-Match` now returns `NotImplemented` with `<Header>If-Match</Header>` and the S3 additional message, ahead of authorization. Before, it checked the condition against that version and deleted the version. In `DeleteObjects`, an empty `<ETag>` fails its key with `InvalidArgument` and a `VersionId` together with an `ETag` fails it with `NotImplemented`, both ahead of the key's authorization.
The `PreconditionFailed` and `NotImplemented` messages drop their trailing period, which S3 doesn't use on any operation.
DeleteObjects dropped the ETag, Size and LastModifiedTime of each object
when it called DeleteObject, so an object with a stale ETag was still
deleted. The posix and azure backends now forward them to DeleteObject.
EvaluateObjectDeletePreconditions trims the quotes an SDK sends around
the ETag and treats "*" as a match for any ETag.
The REST ETag mismatch test no longer skips.
Taken from #2485, without its posix and integration tests.
Refs #2112
Fixes#2499
An empty tag key returned `InvalidTag` without the `TagKey` element because the field was `omitempty`. S3 always sends `<TagKey>`, empty for an empty key, so it is now always written.
`PutBucketTagging`, `PutObjectTagging`, the POST object `tagging` field and `CreateBucket` tags now share one validator, `backend.ParseTagSet`, which matches S3: a duplicate key error carries `TagKey`, a `Tag` missing its `Key` or `Value` element is `MalformedXML` and is checked across the whole document first, every tag is validated before the tag count limit, and the 128/256 length limits count UTF-16 code units instead of bytes. The duplicate tag regex in `s3api/utils` is removed.
For the `x-amz-tagging` header, the length limits now apply to the URL-decoded key and value, the `InvalidArgument` error echoes the whole header as `ArgumentValue` instead of the unparsed remainder, and more than 10 tags is rejected with `BadRequest`.
Fixes#2489
An unsupported `Authorization` algorithm returned only the algorithm in `ArgumentValue`, while AWS S3 echoes the whole header as received. `sigv4auth.ParseAuthorization` now carries the full header in `ParseError.Input`, which the S3 error mapper returns. Also drop the trailing period from the `InvalidArgAuthHeader` message to match AWS.
The azure backend tracks zero-byte parts separately and reports its own
block id as the ETag when the client ETag doesn't match, so the test
failed against azurite. Upload some data so the part is a regular block,
and send the ETag unquoted so both backends echo it back as sent.
Once a bucket had any policy, `verifyResourceAccess` evaluated the policy alone and ignored the bucket ACL, so a non-root bucket owner kept only what the policy explicitly allowed. A Deny-only policy (e.g. Deny `*` `s3:GetObject` on one prefix) therefore locked the owner out of every operation, including `DeleteBucketPolicy`, and only root/admin could recover. ACL grants also authorized every bucket operation through the controller's `AclPermission`, so a non-owner with a `WRITE` grant could install a bucket policy that locked out the owner and every other grantee.
The policy and the ACL are now combined per action the way AWS does it: an explicit policy Deny wins, and otherwise the action is allowed by either a policy Allow or an ACL grant. The bucket owner keeps everything a policy doesn't explicitly deny and, like an AWS account root user, can always call `GetBucketPolicy`, `PutBucketPolicy` and `DeleteBucketPolicy` even past a Deny. The protection covers only those three APIs; `GetBucketPolicyStatus` and every other action stay deniable, matching AWS.
ACL grants to non-owners now follow AWS's ACL-to-action mapping through the new `aclActionPermissions` table: `READ` grants `s3:ListBucket`, `s3:ListBucketVersions`, `s3:ListBucketMultipartUploads`, `s3:GetObject`, `s3:GetObjectVersion` and `s3:GetObjectAttributes`; `WRITE` grants `s3:PutObject` and `s3:DeleteObject`; `READ_ACP` and `WRITE_ACP` grant reading and writing the bucket and object ACLs. Bucket configuration, the bucket policy, tagging, object lock settings and `DeleteBucket` are the owner's alone unless a policy grants them. `verifyACL` is replaced by `aclAllows`, which derives the required permission from the action, so `AccessOptions.AclPermission` now only feeds the readonly gate. The anonymous path uses the same table in place of `publicACLAllowedActions`, which leaves its behavior unchanged.
`GetBucketPolicyStatus` also reported a Deny-only `Principal: "*"` policy as public; only Allow statements now count toward `IsPublic`.
In posix, when the upload directory rename in `CompleteMultipartUploadWithCopy` fails with `ENOENT`, the call is treated as a retry of an already completed upload. That fallback returned 200 whenever any regular file existed at the target bucket/key, with an ETag computed from the request's part ETags, and never checked that the object came from this upload. Completing with the upload ID of another bucket or key, a random upload ID, or an aborted upload was reported as stored while nothing was written. The later `mp-metadata` upload ID check was unreachable whenever the object existed.
The fallback now succeeds only when the object at the key carries this upload ID in its `mp-metadata` attribute and, outside `--data-integrity-etag` mode, when its stored ETag equals the multipart ETag computed from the request's parts. This matches AWS, which rejects a retry with different part ETags with `NoSuchUpload` but ignores part numbers. Every other case returns `NoSuchUpload`, as AWS does for a foreign, random or aborted upload ID and for a retry after the object was overwritten or deleted. Keys with a trailing slash are rejected up front because `ObjectPath` resolves `foo/` to the file `foo`, and attribute read errors other than a missing attribute are no longer reported as `NoSuchUpload`. A successful retry now returns the stored ETag and the object's `version-id`, matching AWS, which returns the original VersionId on a retry in a versioned bucket.
The posix ListObjectsV2 reported the listing marker as the response
ContinuationToken. The marker is the larger of start-after and the
continuation token, so a request with only start-after got its
start-after value back as a ContinuationToken, and a start-after past
the continuation token replaced it. S3 echoes the continuation token
from the request, and leaves ContinuationToken out when the request
had none.
Return the request continuation token instead of the marker, and
enable the REST test that covers it.
Fixes#2007
CompleteMultipartUpload with a part ETag that is not a hex MD5 returned
a bare InvalidPart error, without the UploadId, PartNumber and ETag
elements. S3 includes them, and the gateway already does for a part
that is missing or whose ETag doesn't match.
Pass the upload ID to ComputeMultipartETagFromPartETags so it can
return the detailed InvalidPart error for the offending part, and
enable the REST test that covers it.
Fixes#1008
CopyObject ignored the destination's `If-Match` and `If-None-Match` headers and always overwrote or created the destination, so a stale `If-Match`, `If-None-Match: *` on an existing key, `If-Match` on a missing key and `If-Match` over a delete marker all returned 200. The controller now parses both headers and passes them to the backend through new `IfMatch` and `IfNoneMatch` fields on `s3response.CopyObjectInput`, which match the SDK's `s3.CopyObjectInput`.
As in S3, the destination is evaluated before the copy source is looked up, so a failing destination precondition wins over a missing source, a failing `x-amz-copy-source-*` precondition and an invalid copy onto itself. Posix checks the destination with `checkPutPreconditions` right after the destination bucket check, then forwards the headers to `PutObject`, which repeats the check under the object publish lock. The unversioned in-place self copy does not go through `PutObject`, so it now takes `lockObjectPublish` and repeats the check itself.
`checkPutPreconditions` now treats a current delete marker as a missing key: `If-Match` returns `NoSuchKey` and `If-None-Match: *` creates a new version. Before, the hidden version's etag was evaluated, which also affected `PutObject` and `CompleteMultipartUpload`.
`EvaluateObjectPutPreconditions` now rejects unsupported headers in S3's order: both headers present, then `If-None-Match` other than `*`, then `If-Match: *`, each with `NotImplemented`. `If-Match: *` was previously compared as a literal etag. The new `NmpAdditionalMessageIfMatch` message is added, and a stray backslash is removed from the `If-None-Match` message.
Azure evaluates the destination preconditions before copying, and s3proxy forwards the headers. `s3:if-match` and `s3:if-none-match` are now populated in the policy condition context for CopyObject, as S3 does, while UploadPartCopy stays excluded.
S3 never lists an empty result for another region: a valid region
other than the endpoint's own gets 400 InvalidArgument, "Requests with
bucket-region specified must be made to the corresponding regional
endpoint", with no ArgumentValue. The gateway is a single regional
endpoint, so it now does the same instead of clearing the listing.
Match the region case-insensitively like S3, and check an empty
bucket-region= too by looking for the query argument itself, since an
empty value is not a valid AWS region either.
ListBuckets ignored bucket-region: a value that is not a region was
accepted, and a region other than the gateway one still listed every
bucket. S3 rejects an invalid value with 400 InvalidArgument, "Argument
value abc is not a valid AWS Region", and lists only the buckets of the
requested region.
Check bucket-region against the AWS region code format, always
accepting the gateway region whatever its name, and return an empty
list for any other region, since every bucket lives in the gateway
region. Enable the REST tests that cover both cases.
Fixes#1923Fixes#1930
With an upload-id-marker the lister ranges over Uploads[startIndex:],
but passed the index of that subslice to hasMoreAfter, which indexes
the full list. A page that ended on the last remaining upload was still
reported as truncated, with next markers pointing at that upload, so a
client had to make one more request to get an empty page.
Iterate over the full list from startIndex, so hasMoreAfter gets the
real position of the next upload.
* fix: reject ListMultipartUploads requests that also carry uploadId
A bucket GET with both the uploads and uploadId query parameters was
routed to ListMultipartUploads and answered with the upload listing.
S3 rejects the combination with 400 InvalidArgument, "Conflicting query
string parameters: uploadId, uploads".
Add an error route for the combination ahead of ListMultipartUploads,
the way the object level '?uploads' request is rejected, and enable
the REST test that covers it.
Fixes#2155
* fix: reject uploads with uploadId for every bucket and object operation
Requests carrying both the uploads and uploadId query parameters were
only rejected for a bucket GET, so DELETE, PUT, POST and HEAD on the
bucket or object paths still ran real operations. S3 rejects the
combination for every method on both paths, before authentication and
before checking that the bucket exists.
Register the error route for all methods on the bucket and object
routers right after the CORS middleware is set up, ahead of every other
route, and cover each method and path with an integration test.
The bucket default retention was never stored on objects: protection was computed at delete time from the live bucket rule (`CreatedAt` + period), so changing or removing the rule unlocked existing objects, `COMPLIANCE` ones included, and a rule set N days ago stopped protecting even new objects. The posix and azure backends now store the rule's retention, `{Mode, now + Days/Years}` truncated to milliseconds, as the object's own retention whenever a write sets no Object Lock parameters: `PutObject` (which also covers POST, RDMA and `CopyObject`), `CreateMultipartUpload` (the rule and start time are fixed at creation, as S3 does) and the unversioned in-place self-copy, which applied no lock settings before. As on S3, any explicit `x-amz-object-lock-legal-hold`, `ON` or `OFF`, means the default retention is not applied. The bucket-rule fallback in `checkObject` and `BucketLockConfig.CreatedAt` are removed, so objects written before this change lose bucket-derived protection, and an expired retention no longer skips the legal hold check.
`PutObjectLockConfiguration` now rejects a `DefaultRetention` with a mode but no `Days`/`Years` with `InvalidRequest` "DefaultRetention must specify a fixed retention (Days or Years) or a DefaultEventHold.", and a `<Rule>` without `DefaultRetention` with `MalformedXML`. Azure `CopyObject` with the `COPY` metadata directive no longer copies the source's `Objectretention`/`Objectlegalhold` metadata to the destination.
Browser POST uploads now validate and apply the `x-amz-object-lock-mode`, `x-amz-object-lock-retain-until-date` and `x-amz-object-lock-legal-hold` form fields. They require `s3:PutObjectRetention`/`s3:PutObjectLegalHold`, for anonymous uploads too, and return `ErrMissingObjectLockConfigurationNoSpaces` on buckets without Object Lock after authorization. A denied POST names the actions in S3's order: retention, legal hold, tagging, then `s3:PutObject`.
`NoSuchObjectLockConfiguration` now returns 404 without a trailing period. `x-amz-object-lock-retain-until-date` on HEAD and GET has millisecond precision and omits zero milliseconds, `GetObjectRetention` returns `<Retention>` with a `.000Z` date, and posix and azure `GetObject` now return the Object Lock headers.
Add request plumbing for server-side encryption with customer-provided
keys (SSE-C) and stop silently ignoring it on backends that cannot
honor it.
Previously SSE-C headers were dropped before reaching the backend, so
posix stored the data unencrypted and returned success, and later
reads without the key succeeded. Clients believed their data was
encrypted when it was not. Partial or malformed SSE-C header sets were
also accepted instead of failing with 400 as S3 does. Rejecting SSE-C
explicitly is the safe behavior until posix implements it, and azure
cannot faithfully map SSE-C onto customer-provided keys.
This addresses part of #2193: invalid/partial SSE-C headers now fail
with 400. Requiring SSE-C headers on reads of SSE-C objects depends on
posix SSE-C support and remains open.
The changes also reject unsupported SSE-S3, SSE-KMS, and DSSE-KMS
requests instead of silently accepting them. And enforcea TLS for
SSE-C by default, with --disable-sse-c-tls-enforcement as the opt-out
in case the deployment handles TLS in front of the gateway.
An upload-id-marker sent without a key-marker is ignored for the
listing, but the gateway still echoed it back in UploadIdMarker. S3
discards the parameter in that case and returns an empty
UploadIdMarker.
Clear the marker in the controller when no key-marker is given, so
every backend gets the same input and the response no longer reports
a marker that had no effect.
Fixes#2150
NextKeyMarker and NextUploadIdMarker were only set for a truncated
listing, so a ListMultipartUploads response that fit in one page
returned them empty. S3 still sets them to the key and upload ID of
the last upload in the response.
Fill both markers from the last listed upload when the listing is not
truncated, including the case where the page ends exactly at
max-uploads. A page of common prefixes only keeps them empty, as
before. Enable the REST test that covers it.
Fixes#2144
The posix and azure backends dropped every upload with a key equal to
the key-marker before handing the list to the lister, which then looked
for the upload-id-marker under the next key. An upload-id-marker that
belongs to the key-marker could never be found, so the request failed
with "Invalid uploadId marker", including when a client paged with the
NextKeyMarker and NextUploadIdMarker the gateway returned.
Keep the uploads of the key-marker itself when an upload-id-marker is
set, so the listing continues right after the given upload, as S3 does.
Enable the REST test that covers it.
Fixes#2149
A DeleteObjects request whose Delete element names no objects was
answered with 200 and an empty DeleteResult. S3 rejects it with
MalformedXML, as the schema requires at least one Object.
Reject the empty list right after the body is parsed, next to the
1000 key limit check. Update the unit and integration tests that
expected the empty result and enable the REST test that covers it.
Fixes#2109
A request path with an invalid percent escape, such as "/bucket/%zz",
failed url.PathUnescape in the DecodeURL middleware and the raw error
was answered with 500 InternalError. S3 rejects such a request with
400 InvalidURI.
Return InvalidURI from DecodeURL, so the s3 and admin routers answer
with it as well, and let the website handler pass the error through
instead of mapping it on its own.
Fixes#2070
Object routes match subresources with `MatchQueryArgs`, and a request whose subresource has no route falls through to the catch-all `PutObject`, `GetObject` and `DeleteObject` handlers. As a result `PutObjectAnnotation` overwrote the object with the annotation payload or created a missing key, `DeleteObjectAnnotation` deleted the object or permanently deleted the version named by `versionId`, `RenameObject` replaced the destination with an empty object, and `PUT`/`DELETE ?torrent` overwrote or deleted the object.
`PUT ?annotation`, `GET ?annotation&annotationName`, `GET ?annotation` and `DELETE ?annotation` now return `NotImplemented` and follow the existing not-implemented route pattern: `BucketObjectNameValidator`, `AuthorizePublicBucketAccess`, signature verification and `ParseAcl`. Each route has its own metrics action, and the new `s3:PutObjectAnnotation`, `s3:GetObjectAnnotation`, `s3:ListObjectAnnotations` and `s3:DeleteObjectAnnotation` policy actions are also accepted in bucket policies.
`PUT ?renameObject` now returns `NotImplemented`. `?torrent` returns `MethodNotAllowed` with the new `TORRENT` resource type on every method, with `Allow: GET` on all methods except `GET`. Both are answered before authentication, as AWS does, and there is no `s3:RenameObject` policy action.
Creating a delete marker in posix keeps the file of the hidden version at the object path and only marks it with the delete marker attribute, so the delete marker kept the modification time of the version it hides. `ListObjectVersions` listed that time for current and noncurrent delete markers, and `HeadObject` and `GetObject` sent it as the `Last-Modified` header of the `MethodNotAllowed` response for a delete marker `versionId`. AWS reports the time the delete marker is created, including for the null delete marker created while versioning is suspended. `DeleteObject` now sets the modification time of the delete marker to the time it's created. Setting the time needs the entry to be owned by the gateway, so it is left as is otherwise, like it is for directory objects.
`ListObjectVersions` also listed the delete markers with the SDK `types.DeleteMarkerEntry`, which has no XML marshaling of its own, so `LastModified` was sent in the local time zone of the gateway and with a sub-second part, while the versions are sent in UTC to the second. The delete markers are now listed with the new `s3response.DeleteMarkerEntry`, which formats `LastModified` the same way as `s3response.ObjectVersion`, and the s3proxy backend converts the upstream delete markers to it.
Ben McClellandandGitHub
niksis02
Luke McCrone
dualfroz
theluckystrikeandniksis02
LuisFigueroaG