Let captureHTTPClient answer with a configurable header and body, and
use it for the GetObjectAttributes tests instead of a separate stub
client that repeated the same client setup.
S3Proxy.GetObjectAttributes dropped VersionId and DeleteMarker from the
upstream response, so the gateway never sent x-amz-version-id or
x-amz-delete-marker for objects behind an s3 backend. Copy them into the
response, and add tests for how the upstream response is mapped,
including the parts count.
Fixes#2511
`CreateMultipartUpload` in the posix backend creates the upload directory under `.sgwtmp/multipart` with `os.MkdirAll` and returned any failure wrapped as `create upload temp dir`, so a full filesystem quota or a full disk came back as a 500 `InternalError` (which SDKs also retry) instead of the error the other write paths return. `PutObject`, `UploadPart`, `UploadPartCopy` and `CompleteMultipartUpload` already map these errors.
`syscall.EDQUOT` from the directory creation now returns `QuotaExceeded` (403) and `syscall.ENOSPC` returns `InsufficientStorage` (507), the same mapping the rest of the posix write paths use.
The posix backend stages multipart uploads and temporary files inside the bucket directory under `.sgwtmp`, but nothing stopped an object key from resolving there. A client with access to the bucket could read the parts of other clients' in-progress uploads with `GetObject`/`HeadObject`, copy them out with `CopyObject`, overwrite or delete them with `PutObject`/`DeleteObject` so the upload failed on completion with `InvalidPart`, and store data that never shows up in listings. Spellings such as `./.sgwtmp/...`, `x/../.sgwtmp/...`, a leading `/` and, on case-insensitive file systems, `.SGWTMP/...` reached the same files.
Add the non-AWS `ReservedObjectKey` error (400) and return it whenever an object key resolves to `.sgwtmp` or a path inside it. The key is resolved the same way `ObjectPath` resolves it, and its first path element is compared case-insensitively. The check runs after the bucket existence check in `PutObject`, `GetObject`, `HeadObject`, `GetObjectAttributes`, `DeleteObject`, `DeleteObjects` (reported per key), `CopyObject` (source and destination), the `UploadPartCopy` source, `CreateMultipartUpload`, `CompleteMultipartUpload`, object tagging, legal hold and retention. The exported `ValidateObjectKey` covers the ScoutFS `GetObject` and `RestoreObject` entry points. `UploadPart`, `ListParts` and `AbortMultipartUpload` only hash the key and stay unchanged, so uploads created with a reserved key before this fix can still be aborted.
`ListObjects` and `ListObjectsV2` also listed the staged parts when the prefix named a directory inside `.sgwtmp`, because the prefix fast path in `quickWalk` compared only the last element of the prefix directory against the skipped directories. `inSkipDir` now checks every element, ignoring case.
The controller now passes the requested attributes to the backend in
ObjectAttributes, so posix and azure only do the extra HeadObject for the
parts count when ObjectParts was asked for. This also lets the s3 backend
reach the upstream service, which rejected the request before because the
field was missing.
The s3 backend now copies TotalPartsCount into PartsCount, leaves out
ObjectParts when the upstream response has none, and sets the part number
markers, which were dropped by inverted strconv.Atoi error checks.
Fixes#2513
On Windows the posix backend stored a key as `bucket/key` verbatim, but Windows rejects `" * : < > ? \ |` and control characters in file names, so such a PUT failed with `ERROR_INVALID_NAME`, which `isErrNameTooLong` reported as a misleading `KeyTooLongError`. The same mapping did worse elsewhere: `\` acted as a path separator, so `a\b` aliased `a/b` and `..\..\x` resolved outside the bucket because `IsObjectNameValid` only splits on `/`; `a:b` wrote an NTFS alternate data stream; Win32 strips a trailing dot or space, so `foo.` overwrote `foo`; and device names such as `CON` or `nul.txt` opened the device and silently discarded the data.
With `encodeKeys`, enabled by default only on Windows, `encodeKey` stores each key segment under a name Windows accepts: every invalid character, a trailing dot or space, and the first letter of a device name are stored as the private use character `U+F000` plus the character code, the convention Cygwin and WSL use for invalid characters. A key character in `U+F000`-`U+F07F` is stored after `U+F000`, so every stored name belongs to exactly one key. Device names are checked with `filepath.IsLocal`, which asks the system, so `CON.txt` stays a plain file on Windows 11 where it is valid. Keys that need no encoding keep their names, so existing data is unaffected, and nothing changes on other platforms.
Public backend methods keep taking raw keys and encode them once for every path, metadata, version-hash and publish-lock use, while responses, errors and the multipart `objname` attribute keep the raw key and multipart upload directories keep hashing it. Listings walk `bucketFS`, which decodes the entry names, sorts them in key byte order to match S3 and skips names no key is stored under. `HeadObject` passes the raw key to `GetObjectLegalHold` and `GetObjectRetention` to avoid encoding twice, and `NormalizeObjectKey` resolves the encoded key, so policy resources see `a\b` as one literal key.
GetObjectAttributes never returned ObjectParts, so a request for it on
an object completed from a multipart upload got no PartsCount. S3
returns <ObjectParts><PartsCount>N</PartsCount></ObjectParts> for these
objects and leaves ObjectParts out for objects that weren't uploaded in
parts.
The posix and azure backends already keep the part layout of completed
uploads, and HeadObject reports the parts count when a part number is
requested. GetObjectAttributes now uses that to fill ObjectParts. The
ObjectParts fields that aren't set are left out of the response, as S3
does.
Fixes#925
The api layer validates a client's LocationConstraint against the gateway
region, but the bucket is created in the s3 backend's own region
(--region / VGW_S3_REGION). They are separate settings, so passing the
validated client constraint through to the backend only worked when the two
regions matched. With the gateway on the default us-east-1 and the backend
on eu-west-1 the client sends no constraint at all, and the backend still
answered
IllegalLocationConstraintException: The unspecified location constraint is
incompatible for the region specific endpoint this request was sent to.
which is the error in #2490.
Set the constraint in S3Proxy.CreateBucket from the backend client's region
instead, next to the existing empty-configuration drop. A backend on
us-east-1 still gets no constraint, and the api layer's rejection of a
client constraint that disagrees with the gateway region is unchanged.
Fixes#2490
A conditional delete of a key with no current version used to succeed. `DeleteObject` with `If-Match`, and a `DeleteObjects` key with an `ETag`, reported success for a missing key. When the current version was a delete marker, the condition was checked against the ETag the marker kept from the version it hides, so it could pass and create a second marker. S3 fails both with `NoSuchKey` and creates no marker. The posix backend now checks the conditions against the current version in `checkDeletePreconditions`, before its missing-key shortcuts. The azure backend returns `NoSuchKey` for a missing blob instead of skipping the conditions. `ObjectDeletePreconditions.IsSet` reports whether a delete has any condition.
A condition only applies to the current version. `DeleteObject` with a `versionId` and `If-Match` now returns `NotImplemented` with `<Header>If-Match</Header>` and the S3 additional message, ahead of authorization. Before, it checked the condition against that version and deleted the version. In `DeleteObjects`, an empty `<ETag>` fails its key with `InvalidArgument` and a `VersionId` together with an `ETag` fails it with `NotImplemented`, both ahead of the key's authorization.
The `PreconditionFailed` and `NotImplemented` messages drop their trailing period, which S3 doesn't use on any operation.
DeleteObjects dropped the ETag, Size and LastModifiedTime of each object
when it called DeleteObject, so an object with a stale ETag was still
deleted. The posix and azure backends now forward them to DeleteObject.
EvaluateObjectDeletePreconditions trims the quotes an SDK sends around
the ETag and treats "*" as a match for any ETag.
The REST ETag mismatch test no longer skips.
Taken from #2485, without its posix and integration tests.
Refs #2112
Fixes#2499
An empty tag key returned `InvalidTag` without the `TagKey` element because the field was `omitempty`. S3 always sends `<TagKey>`, empty for an empty key, so it is now always written.
`PutBucketTagging`, `PutObjectTagging`, the POST object `tagging` field and `CreateBucket` tags now share one validator, `backend.ParseTagSet`, which matches S3: a duplicate key error carries `TagKey`, a `Tag` missing its `Key` or `Value` element is `MalformedXML` and is checked across the whole document first, every tag is validated before the tag count limit, and the 128/256 length limits count UTF-16 code units instead of bytes. The duplicate tag regex in `s3api/utils` is removed.
For the `x-amz-tagging` header, the length limits now apply to the URL-decoded key and value, the `InvalidArgument` error echoes the whole header as `ArgumentValue` instead of the unparsed remainder, and more than 10 tags is rejected with `BadRequest`.
The posix backend publishes an object by linking or renaming a temp file
into place, but never flushes it. An acknowledged PUT can therefore be
lost, or come back empty, after a crash or power loss, depending on the
filesystem's writeback interval.
With --fsync (VGW_FSYNC), the backend fsyncs the temp file, which covers
its data and xattr metadata, before publishing it. After publishing, it
fsyncs the object's directory and each parent up to the bucket, so that
the new entry and any directories created for it are durable. A failed
flush fails the request. The option is off by default, so existing
deployments are unaffected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
In posix, when the upload directory rename in `CompleteMultipartUploadWithCopy` fails with `ENOENT`, the call is treated as a retry of an already completed upload. That fallback returned 200 whenever any regular file existed at the target bucket/key, with an ETag computed from the request's part ETags, and never checked that the object came from this upload. Completing with the upload ID of another bucket or key, a random upload ID, or an aborted upload was reported as stored while nothing was written. The later `mp-metadata` upload ID check was unreachable whenever the object existed.
The fallback now succeeds only when the object at the key carries this upload ID in its `mp-metadata` attribute and, outside `--data-integrity-etag` mode, when its stored ETag equals the multipart ETag computed from the request's parts. This matches AWS, which rejects a retry with different part ETags with `NoSuchUpload` but ignores part numbers. Every other case returns `NoSuchUpload`, as AWS does for a foreign, random or aborted upload ID and for a retry after the object was overwritten or deleted. Keys with a trailing slash are rejected up front because `ObjectPath` resolves `foo/` to the file `foo`, and attribute read errors other than a missing attribute are no longer reported as `NoSuchUpload`. A successful retry now returns the stored ETag and the object's `version-id`, matching AWS, which returns the original VersionId on a retry in a versioned bucket.
The posix ListObjectsV2 reported the listing marker as the response
ContinuationToken. The marker is the larger of start-after and the
continuation token, so a request with only start-after got its
start-after value back as a ContinuationToken, and a start-after past
the continuation token replaced it. S3 echoes the continuation token
from the request, and leaves ContinuationToken out when the request
had none.
Return the request continuation token instead of the marker, and
enable the REST test that covers it.
Fixes#2007
CompleteMultipartUpload with a part ETag that is not a hex MD5 returned
a bare InvalidPart error, without the UploadId, PartNumber and ETag
elements. S3 includes them, and the gateway already does for a part
that is missing or whose ETag doesn't match.
Pass the upload ID to ComputeMultipartETagFromPartETags so it can
return the detailed InvalidPart error for the offending part, and
enable the REST test that covers it.
Fixes#1008
CopyObject ignored the destination's `If-Match` and `If-None-Match` headers and always overwrote or created the destination, so a stale `If-Match`, `If-None-Match: *` on an existing key, `If-Match` on a missing key and `If-Match` over a delete marker all returned 200. The controller now parses both headers and passes them to the backend through new `IfMatch` and `IfNoneMatch` fields on `s3response.CopyObjectInput`, which match the SDK's `s3.CopyObjectInput`.
As in S3, the destination is evaluated before the copy source is looked up, so a failing destination precondition wins over a missing source, a failing `x-amz-copy-source-*` precondition and an invalid copy onto itself. Posix checks the destination with `checkPutPreconditions` right after the destination bucket check, then forwards the headers to `PutObject`, which repeats the check under the object publish lock. The unversioned in-place self copy does not go through `PutObject`, so it now takes `lockObjectPublish` and repeats the check itself.
`checkPutPreconditions` now treats a current delete marker as a missing key: `If-Match` returns `NoSuchKey` and `If-None-Match: *` creates a new version. Before, the hidden version's etag was evaluated, which also affected `PutObject` and `CompleteMultipartUpload`.
`EvaluateObjectPutPreconditions` now rejects unsupported headers in S3's order: both headers present, then `If-None-Match` other than `*`, then `If-Match: *`, each with `NotImplemented`. `If-Match: *` was previously compared as a literal etag. The new `NmpAdditionalMessageIfMatch` message is added, and a stray backslash is removed from the `If-None-Match` message.
Azure evaluates the destination preconditions before copying, and s3proxy forwards the headers. `s3:if-match` and `s3:if-none-match` are now populated in the policy condition context for CopyObject, as S3 does, while UploadPartCopy stays excluded.
With an upload-id-marker the lister ranges over Uploads[startIndex:],
but passed the index of that subslice to hasMoreAfter, which indexes
the full list. A page that ended on the last remaining upload was still
reported as truncated, with next markers pointing at that upload, so a
client had to make one more request to get an empty page.
Iterate over the full list from startIndex, so hasMoreAfter gets the
real position of the next upload.
The bucket default retention was never stored on objects: protection was computed at delete time from the live bucket rule (`CreatedAt` + period), so changing or removing the rule unlocked existing objects, `COMPLIANCE` ones included, and a rule set N days ago stopped protecting even new objects. The posix and azure backends now store the rule's retention, `{Mode, now + Days/Years}` truncated to milliseconds, as the object's own retention whenever a write sets no Object Lock parameters: `PutObject` (which also covers POST, RDMA and `CopyObject`), `CreateMultipartUpload` (the rule and start time are fixed at creation, as S3 does) and the unversioned in-place self-copy, which applied no lock settings before. As on S3, any explicit `x-amz-object-lock-legal-hold`, `ON` or `OFF`, means the default retention is not applied. The bucket-rule fallback in `checkObject` and `BucketLockConfig.CreatedAt` are removed, so objects written before this change lose bucket-derived protection, and an expired retention no longer skips the legal hold check.
`PutObjectLockConfiguration` now rejects a `DefaultRetention` with a mode but no `Days`/`Years` with `InvalidRequest` "DefaultRetention must specify a fixed retention (Days or Years) or a DefaultEventHold.", and a `<Rule>` without `DefaultRetention` with `MalformedXML`. Azure `CopyObject` with the `COPY` metadata directive no longer copies the source's `Objectretention`/`Objectlegalhold` metadata to the destination.
Browser POST uploads now validate and apply the `x-amz-object-lock-mode`, `x-amz-object-lock-retain-until-date` and `x-amz-object-lock-legal-hold` form fields. They require `s3:PutObjectRetention`/`s3:PutObjectLegalHold`, for anonymous uploads too, and return `ErrMissingObjectLockConfigurationNoSpaces` on buckets without Object Lock after authorization. A denied POST names the actions in S3's order: retention, legal hold, tagging, then `s3:PutObject`.
`NoSuchObjectLockConfiguration` now returns 404 without a trailing period. `x-amz-object-lock-retain-until-date` on HEAD and GET has millisecond precision and omits zero milliseconds, `GetObjectRetention` returns `<Retention>` with a `.000Z` date, and posix and azure `GetObject` now return the Object Lock headers.
Add request plumbing for server-side encryption with customer-provided
keys (SSE-C) and stop silently ignoring it on backends that cannot
honor it.
Previously SSE-C headers were dropped before reaching the backend, so
posix stored the data unencrypted and returned success, and later
reads without the key succeeded. Clients believed their data was
encrypted when it was not. Partial or malformed SSE-C header sets were
also accepted instead of failing with 400 as S3 does. Rejecting SSE-C
explicitly is the safe behavior until posix implements it, and azure
cannot faithfully map SSE-C onto customer-provided keys.
This addresses part of #2193: invalid/partial SSE-C headers now fail
with 400. Requiring SSE-C headers on reads of SSE-C objects depends on
posix SSE-C support and remains open.
The changes also reject unsupported SSE-S3, SSE-KMS, and DSSE-KMS
requests instead of silently accepting them. And enforcea TLS for
SSE-C by default, with --disable-sse-c-tls-enforcement as the opt-out
in case the deployment handles TLS in front of the gateway.
NextKeyMarker and NextUploadIdMarker were only set for a truncated
listing, so a ListMultipartUploads response that fit in one page
returned them empty. S3 still sets them to the key and upload ID of
the last upload in the response.
Fill both markers from the last listed upload when the listing is not
truncated, including the case where the page ends exactly at
max-uploads. A page of common prefixes only keeps them empty, as
before. Enable the REST test that covers it.
Fixes#2144
The posix and azure backends dropped every upload with a key equal to
the key-marker before handing the list to the lister, which then looked
for the upload-id-marker under the next key. An upload-id-marker that
belongs to the key-marker could never be found, so the request failed
with "Invalid uploadId marker", including when a client paged with the
NextKeyMarker and NextUploadIdMarker the gateway returned.
Keep the uploads of the key-marker itself when an upload-id-marker is
set, so the listing continues right after the given upload, as S3 does.
Enable the REST test that covers it.
Fixes#2149
ScoutFS delegates buffered object transfer paths to the
POSIX backend, so expose IOBufferSize in ScoutfsOpts and
pass it through to PosixOpts. This lets operators tune
transfer buffering for ScoutFS instead of being limited
to the default.
Uploads copied the request body with io.CopyBuffer, which issues one write
per read. Network reads return arbitrary lengths, so the first short read
produced an unaligned write and the tmpfile silently reopened buffered for
the rest of the object.
- Fill the I/O buffer completely before each O_DIRECT write so only the
final tail is unaligned; write that tail buffered without treating it
as a fallback.
- Allocate I/O buffers 4 KiB aligned and use 4 KiB alignment for O_DIRECT
lengths and offsets to cover 4Kn devices.
- Replace the O_DIRECT read wrapper with a reader that issues aligned
preads into an aligned buffer for full and ranged GETs. sendfile from an
O_DIRECT fd fails with EINVAL after a partial socket write, so O_DIRECT
reads no longer go through sendfile; buffered reads still do.
- Debug log unexpected O_DIRECT fallbacks on both paths.
Creating a delete marker in posix keeps the file of the hidden version at the object path and only marks it with the delete marker attribute, so the delete marker kept the modification time of the version it hides. `ListObjectVersions` listed that time for current and noncurrent delete markers, and `HeadObject` and `GetObject` sent it as the `Last-Modified` header of the `MethodNotAllowed` response for a delete marker `versionId`. AWS reports the time the delete marker is created, including for the null delete marker created while versioning is suspended. `DeleteObject` now sets the modification time of the delete marker to the time it's created. Setting the time needs the entry to be owned by the gateway, so it is left as is otherwise, like it is for directory objects.
`ListObjectVersions` also listed the delete markers with the SDK `types.DeleteMarkerEntry`, which has no XML marshaling of its own, so `LastModified` was sent in the local time zone of the gateway and with a sub-second part, while the versions are sent in UTC to the second. The delete markers are now listed with the new `s3response.DeleteMarkerEntry`, which formats `LastModified` the same way as `s3response.ObjectVersion`, and the s3proxy backend converts the upstream delete markers to it.
Deleting the current version of an object by `versionId` makes the newest remaining version current again. Posix copies that version from the versioning directory into a new temporary file and links it in place of the object, but never restored the file's modification time, so `HeadObject`, `ListObjectVersions` and `ListObjectsV2` reported the delete time as the promoted version's `LastModified`. This affected an older version, a version under a deleted delete marker and the null version alike, while AWS keeps the original `LastModified` in all three cases. Directory objects were not affected, as `deleteDirObjectLatestVersion` already restores the directory's mtime.
The temporary file now carries an optional modification time, set with `setModTime` and applied by `link()` right before the file is published: through `/proc/self/fd` before `linkat` for an `O_TMPFILE` file, and after the file is closed and before the rename on the `CreateTemp` fallback and on non-Linux platforms. Both the promotion in `DeleteObject` and `createObjVersion` use it. `createObjVersion` used to call `os.Chtimes` on the temporary file while it was still open for writing, and Windows resets the last write time when a handle the file was written through is closed.
The `sidecar` metadata storer keeps attributes by path rather than on the file. So a new file published at an object's path inherited every attribute it didn't set itself: tags, user metadata, content headers and `x-amz-website-redirect-location`, plus legal hold and retention on version promotion. This hit overwrites by `PutObject`, `CopyObject` and `CompleteMultipartUpload`, new versions and `PutObject` over a delete marker in versioned buckets, and the version made current again by `DeleteObject` with a `versionId`. A plain `PutObject` over a multipart object also kept its `mp-metadata`, so `HeadObject` and `GetObject` with `partNumber` returned the old `PartsCount` and a `Content-Range` beyond the object size. `xattr` was not affected, since its attributes live on the inode and are replaced along with it.
Add `ReplaceObject(bucket, object, keep)` to `MetadataStorer`. It is a no-op for `XattrMeta` and `NoMeta`. `SideCar` removes the object's attribute files not named in `keep` and leaves directories in place, since they hold the metadata of objects under `<object>/meta/`. Posix calls it in `PutObject` and `CompleteMultipartUpload` after the version snapshot, and in `DeleteObject` when restoring the latest remaining version. This replaces the scattered deletes of the delete marker, `versionId` and object lock attributes. Attributes that every publish rewrites are kept and overwritten in place: the etag, the checksums, the new `versionId`, `mp-metadata` for multipart uploads, and the restored version's attributes on promotion. This way concurrent readers, such as the lock-free `If-Match` fast-fail check, never see them missing.
`PutObject` now re-checks under the publish lock that the object path hasn't become a directory while the body was staged, as `CompleteMultipartUpload` already does. This stops it from removing the attributes of a directory object created by a racing `PUT foo/`.
An authenticated user with write access to a single bucket could list, write and delete files outside the gateway root on the posix and scoutfs backends. The multipart `uploadId` and the `DeleteObjects` body keys were joined into filesystem paths without validation, and unlike object keys in the request path they never passed the object name validator that rejects `..`. With a `../` upload ID, `ListParts` listed the contents of an outside directory, `UploadPart` and `UploadPartCopy` wrote part files into it, `CompleteMultipartUpload` renamed and removed it, and `AbortMultipartUpload` removed it recursively. A `../` key in `DeleteObjects` removed the outside file, after which the empty-parent cleanup removed emptied directories above the root, and a `/` key removed an empty bucket's directory.
Posix now rejects any upload ID it could not have generated. Posix has always generated upload IDs with `uuid.New().String()`, so `validateUploadID` accepts only a UUID in that exact canonical form and returns `NoSuchUpload` for anything else, which is what AWS returns for a malformed upload ID. The check runs right after the bucket existence check in all five multipart operations that take an upload ID, before any path is built from it.
The `DeleteObjects` controller now checks every key with `IsObjectNameValid`, the same check already applied to request-path, copy-source and POST object keys. If any key is missing or resolves outside the bucket, the whole request is rejected with `BadRequest` before authorization, so no key in the batch is deleted. AWS also fails the whole batch when one key is invalid.
Fixes#2439
When the gateway process lacks write permission on the backend filesystem (`EACCES`/`EPERM`), for example when it runs as an unprivileged user over a root, bucket or directory it cannot write, posix write operations returned 500 `InternalError`. S3 clients retry that several times before failing, and the real cause never reaches the client. `PutObject` and `CopyObject` were worse: any failure to create the parent directory or to link the object into place was reported as 409 `ExistingObjectIsDirectory`, with nothing logged.
Add `backend.IsPermissionErr` and `backend.MapPermissionErr`, which replaces a permission error with 403 `AccessDenied` and logs the cause at debug level. Every posix method that writes, and scoutfs `RestoreObject`, defers it on its named error result, so the mapping applies however deeply the error was wrapped. Methods that delegate inherit it: `PutObject`, `UploadPart`, `CompleteMultipartUpload`, the `Delete*` bucket config methods, and `DeleteObjects` for each key. `PutObject` now passes permission errors through its `ExistingObjectIsDirectory` catch-alls.
The sidecar metadata store wrapped its errors with `%v`, which broke the `errors.Is` chain. It now uses `%w`, so sidecar permission failures are mapped the same way as xattr ones.
Only `EACCES` and `EPERM` are matched, not `fs.ErrPermission`. On Windows, `ERROR_ACCESS_DENIED` is also returned for transient delete-pending states where a retry can succeed, so Windows behavior is unchanged.
Fixes#1535Fixes#1542Fixes#1740Fixes#1776
S3 requires `Content-MD5`, an `x-amz-checksum-*` header or an `x-amz-checksum-*` trailer on every `PutObject` that carries Object Lock parameters, and rejects the upload otherwise with `InvalidRequest` "Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters". The parameters are an `x-amz-object-lock-legal-hold` header of either status, a retention mode and date, or the bucket's default retention rule, which gives even a plain upload lock parameters. A payload hash, `UNSIGNED-PAYLOAD` and signed chunks don't count as an integrity check. The gateway accepted all of these uploads.
`auth.VerifyWriteObjectLock` reads the bucket's lock configuration, rejects lock headers on a bucket without Object Lock with "Bucket is missing ObjectLockConfiguration", legal hold `OFF` included, and reports whether the write carries lock parameters. `PutObject` calls it when the request has lock headers or no integrity check, so an upload with a checksum and no lock headers pays no extra lookup. `CopyObject` and `CreateMultipartUpload` call it only for lock headers, since S3 requires no integrity check for them.
S3 applies the same rule to `UploadPart`, with "Put Part requests" in the message, when the multipart upload carries lock parameters: its own lock headers, or a default retention rule in force at `CreateMultipartUpload`, which still counts if the rule changes later. posix and azure now mark such an upload when it's created, with the `mp-object-lock` attribute and the `Mpobjectlock` sgwtmp metadata respectively, and `UploadPart` rejects a part of it that has no integrity check. The `UploadPart` controller now passes `Content-MD5` to the backend.
An anonymous `PutObject` carrying lock parameters, including a plain one into a bucket with a default retention rule, is refused with `InvalidArgument` "Put Object requests with Object Lock parameters require AWS Signature Version 4". This comes after the missing configuration and integrity checks, and before the upload itself is authorized, so a private bucket with a default retention rule refuses it too. Anonymous parts of an upload with lock parameters only need an integrity check, and browser-based `POST` uploads are unaffected, as on S3.
Anonymous uploads were authorized for `s3:PutObject` alone, so a public bucket granting only that let anonymous callers tag objects. `AuthorizePublicBucketAccess` now authorizes every action a request carries in one `VerifyPublicAccess` call. A `PutObject` with a non-empty `x-amz-tagging` or a browser-based `POST` with a non-empty tag set needs `s3:PutObjectTagging` as well as `s3:PutObject`. The shared `auth.ObjectUploadActions` derives these actions for both anonymous and signed uploads. Each action is granted on its own, by the bucket policy or, failing that, the bucket ACL, whose WRITE grant covers `s3:PutObject` only, and an explicit public Deny of any one of them denies the request.
Anonymous requests naming an object version were authorized for the plain action. A non-empty `versionId` now makes the public access check authorize the version action instead: `s3:GetObjectVersion` rather than `s3:GetObject` for `GetObject` and `HeadObject`, `s3:DeleteObjectVersion` for `DeleteObject`, `s3:GetObjectVersionAttributes` for `GetObjectAttributes`, and the version tagging actions for the tagging APIs. A public ACL READ grant covers `s3:GetObjectVersion`, but a WRITE grant never covers `s3:DeleteObjectVersion`, which S3 grants to the bucket owner alone.
The "x-amz-object-lock-retain-until-date and x-amz-object-lock-mode must both be supplied" errors no longer end with a period, and the anonymous `CopyObject` and `CreateMultipartUpload` rejections have two spaces before "Please authenticate.", both matching S3.
In a bucket with versioning `Suspended`, `CompleteMultipartUpload` only archived the object it replaced when versioning was `Enabled`, so completing an upload over a versioned object overwrote it without a snapshot and that version was lost. The null version in the versioning directory was not removed either, which left a stale `null` version next to the new one. With sidecar metadata, the leftover `versionIdKey` attribute also made the new object report the overwritten version's id.
With versioning `Enabled` and sidecar metadata, the upload's metadata was stored at the object path before the replaced object was archived, so `createObjVersion` copied the new upload's content type, user metadata and other properties into the previous version.
`CompleteMultipartUpload` now archives the replaced object with `snapshotObjVersion` before storing any metadata of the new object, which covers both `Enabled` and `Suspended` buckets. In a `Suspended` bucket it also removes the archived null version and clears the stale `versionIdKey` attribute, matching `PutObject` and S3 behavior.
`latestObjVersion` and `fileToObjVersions` placed the null version among the `ulid` versions by comparing file modification times, but a delete marker keeps the data and mtime of the version it hides. A null delete marker created in a versioning-suspended bucket therefore tied with the version below it and lost. Deleting the current version of an object whose history held a null delete marker restored the older version instead of the delete marker, and `ListObjectVersions` listed the delete marker below that version. Directory objects were exposed too, as a directory's mtime changes when objects are created under it.
`createObjVersion` now records the position of the null version when it moves it to the versioning directory. At that moment the null version is the current one, so it is newer than every version already there and older than every version created later. The id of the newest version in the directory is stored in the new `null-version-prev` attribute, and `nullVersionPos` compares it with the `ulid` version ids for both `DeleteObject` and `ListObjectVersions`. Null versions moved before this change carry no attribute and still fall back to the modification time. The attribute is not copied onto the object when a null version is restored.
`fileToObjVersions` also listed the null version without checking the `VersionIdMarker`, so it was repeated on every later page, and a `null` marker never matched, which dropped every version below it. The current version was listed again when it was the marker, so paging with the marker at a current version looped forever. The null version is now listed through `addNullVersion`, which skips it before the marker and continues after it when it is the marker, and the current version is no longer listed again when the marker names it.
Posix `DeleteObject` only compared `versionId` with the object's version when the bucket had a versioning status. A bucket that was never versioned has none, so the request fell through to the plain delete, which ignored `versionId`. Any well-formed `versionId` then permanently deleted the object, on a gateway with or without `--versioning-dir`, and through `DeleteObjects` as well, since it calls `DeleteObject` for each key.
A never-versioned bucket only holds `null` versions, so any other `versionId` names a version that doesn't exist. S3 returns success for that, echoes `x-amz-version-id`, and deletes nothing. `DeleteObject` now does the same, returning early for any `versionId` other than `null`. `versionId=null` still deletes the object.
Fixes#2165
`GetObject` marked an object carrying no `versionIdKey` attribute as the null version and then overwrote that with the empty attribute value, and `HeadObject` never marked it at all, so neither reported a version id for an object put into a versioning-suspended bucket. Both now resolve it through a shared `liveObjVersionId`, which reports `null` only when the bucket has versioning configured so that a bucket that was never versioned keeps reporting no version id at all.
`HeadObject` also resolved an explicit `?versionId=null` into the versioning directory twice, once for the missing attribute and once for the mismatch that followed from it, and answered `404` for a version that `GetObject` served. It now treats the missing attribute as the null version the way `GetObject` does.
`latestObjVersion` took the last version directory entry as the newest, but `os.ReadDir` sorts by name and `null` sorts after every `ulid`. Deleting the current version of an object whose history also held a null version therefore restored `null` rather than the version created immediately before the deleted one. The `ulid` entries keep their name order, which is creation order, and the null version is placed by its modification time, the same rule `fileToObjVersions` already applies.
`CopyObject` treated a destination path equal to the source path as an in-place metadata rewrite, regardless of bucket versioning. On a versioned bucket a self copy therefore edited the current version and returned its existing version id instead of creating a new one. This also defeated object lock: `CheckObjectAccess` skips the retention and legal hold checks for overwrites on version-enabled buckets because an overwrite is expected to create a new version, so a `COMPLIANCE` retained or legal held version had its metadata replaced underneath it. A self copy is now rewritten in place only when the bucket is unversioned, and otherwise goes through the regular copy path, which creates a new version and leaves the one it replaces untouched.
The rejection of a self copy that replaces nothing moved out of the in-place branch and is now applied only when the copy source carries no version id. Naming a version explicitly makes the request a regular copy, which AWS accepts even with the `COPY` metadata directive, while versitygw answered `InvalidRequest`.
Source tagging for a `COPY` tagging directive is now read before `PutObject` writes the destination rather than after. A self copy replaces the source object's attributes, so the later read returned nothing and the tags were dropped from the new version.
`CopyObject` and `UploadPartCopy` never looked at the source's `delete-marker` attribute. A delete marker leaves the data file in place at the object path and only flags it, so a copy whose source resolved to a marker opened that file and succeeded, handing back the data of the version the marker had deleted.
AWS rejects such a copy: `NoSuchKey` when the marker is the current version of the key, and `InvalidRequest` when the marker is named by version id, the latter regardless of whether it is the latest version. Versions the marker hides stay copyable by version id.
Both copy paths now run the resolved source through `checkCopySourceDeleteMarker` once the entry has been validated, returning `NoSuchKey` for an unqualified source and the new `ErrCopySourceDeleteMarker` for one carrying a version id.
LuisFigueroaG
Ben McClellandandGitHub
niksis02
Tung Lam
theluckystrikeandniksis02
Keith KennedyandClaude Opus 5.5
DualFroz