Fixes#2509
Concurrent `CompleteMultipartUpload` calls for the same upload ID could return a 500 `InternalError` on FreeBSD, logging `get object tagging: xattr.get ...inprogress: no such file or directory`. The posix backend claims an upload by renaming `<uploadId>` to `<uploadId>.<etag>.inprogress` and relied on only one concurrent `rename` succeeding. FreeBSD's `kern_renameat` looks up the source and the target in separate unlocked `namei` calls and returns success without doing anything when both resolve to the same vnode. A call that found `<uploadId>` before a sibling renamed it, and the `.inprogress` directory after, therefore claimed the upload as well. Both callers assembled the object, and the first to finish removed the `.inprogress` directory while the second still had to read its attributes. Linux holds the parent directory lock across both lookups, so the race never showed there.
The claim rename now runs under the object's publish lock (`lockObjectPublish`), released right after the rename. Claiming renames of the same upload can no longer overlap: within a process in every `--object-lock-mode`, and across processes with `flock` or `fcntl`. Assembling the parts still happens outside the lock.
Follow-up to #2514, which added PosixOpts.ExcludeDirs (--exclude-dirs,
VGW_EXCLUDE_DIRS) so the listings can skip filesystem control
directories such as ZFS's .zfs.
The walks matched skipped names against every directory entry, files
included. In WalkVersions, returning fs.SkipDir for a file skips the
rest of its parent directory, so a regular file named like an excluded
directory (for example an object "d/.zfs") made ListObjectVersions drop
every sibling that sorted after it. Walk hid the file itself.
Previously only .sgwtmp was skipped, which clients can't create, but
excluded names are chosen by the operator and any client can PUT such a
key. Walk and WalkVersions now skip only directories, so a file with an
excluded name is listed like any other object.
Excluded names were also not reserved: requests could still read and
write objects under an excluded directory that the listings never show.
On ZFS, .zfs is reachable by name even with snapdir=hidden, so GetObject
could read snapshot contents. That bypasses bucket policies scoped to a
key prefix and exposes data that was deleted or overwritten since the
snapshot was taken. On other filesystems, objects written there were
orphans that no listing reports. validateObjectName now rejects, with
ReservedObjectKey, any key that has an excluded name as a directory at
any depth, ignoring case and after resolving "..", "." and repeated
slashes. Every object operation already calls it. A directory-object
key (trailing slash) counts its last element as a directory; a plain
key with an excluded last element is still a valid file. The new
ErrExcludedObjectKey keeps the ReservedObjectKey code with a description
that fits excluded directories, leaving the .sgwtmp error unchanged.
Nothing changes when ExcludeDirs is empty.
posix.New now also rejects "." and ".." as excluded names, since they
can never match a directory entry and would silently do nothing.
fiber `v3.6.0` (gofiber/fiber#4695) normalizes every request path before routing as RFC 3986 Section 6.2.2 describes: it decodes the escapes of unreserved characters and removes `.` and `..` segments, and `ctx.Path()`, `ctx.Params()` and every `ctx.Path(override)` return the normalized path with no way to opt out. `DecodeURL` sets the decoded path back through `ctx.Path()`, so keys were decoded twice (`100%2541` was stored as `100A`, `q%7euser` as `q~user`) and dot segments rerouted requests: `PUT /bucket/.` became `CreateBucket` and `/bucket/../../etc/passwd` named the bucket `etc`.
The new `httpctx.Ctx`, created with `fiber.NewWithCustomCtx`, keeps a path set with `Path()` as it is and has fiber route a copy in which every `%` and the dots of a `.` or `..` segment are `_`. The copy keeps the length of the path, since fiber matches routes on it but takes the route parameters from `Path()` at the matched offsets, so `Params("bucket")` holds the real bucket. The S3, admin and website servers and the RDMA ops tracker use it.
S3 stores keys holding `.` and `..` segments as they are (`a/../b`, `x/.` and `..` are objects of their own) and only rejects, before authentication, a request path whose decoded `..` segments climb above the root, separating the segments at `\` as well as at `/` and skipping the empty ones. `DecodeURL` now rejects such a path with `InvalidURI` and keeps the dot segments in the key otherwise.
`Backend.StoresDotSegmentKeys()` reports whether a backend stores such keys as their own objects: `azure` and `s3proxy` do, while `posix` and the `BackendUnsupported` default don't, since a filesystem resolves the segments to another object or to a path outside the bucket. `IsObjectNameValid` takes the backend and rejects keys holding dot segments for those backends in the request path, the `DeleteObjects` keys, the `PostObject` key and the copy sources, so `posix` now also rejects `a/../b` and `a/./b`, which it stored as `b` and `a/b`. The unused `path.go` is removed.
Let captureHTTPClient answer with a configurable header and body, and
use it for the GetObjectAttributes tests instead of a separate stub
client that repeated the same client setup.
Fixes#2525
`CompleteMultipartUpload` returned the stored part's ETag in `InvalidPart` errors instead of the ETag the client sent: posix on an ETag mismatch, and azure on a zero-byte part mismatch, where it returned the expected block ID. Azure also returned an empty ETag for a part that was never uploaded. All three now return the client's ETag. Like S3, `GetInvalidPartErr` removes every `"` from it, inner quotes included, and keeps surrounding whitespace.
`SetResponseHeaders` turns off header name normalization and leaves it off, so response header names stay lowercase. Before it runs, the `RequestIDs` middleware stored `X-Amz-Request-Id` and `X-Amz-Id-2`. Afterwards, the `EnsureRequestIDs` call in `ProcessController` stored `x-amz-request-id` and `x-amz-id-2`. Every response with a headers map, success responses included, therefore sent both pairs. `EnsureRequestIDs`, `RequestID` and `HostID` now use `SetCanonical` and `PeekCanonical` with the lowercase names, so the normalization setting doesn't affect them, and each header is sent once in lowercase as S3 does.
The previous revision hard-coded .zfs next to .sgwtmp in the
ListObjects, ListObjectsV2 and ListObjectVersions walks. That hides a
directory named .zfs on filesystems that don't have ZFS snapshots, and
it gives an operator no way to exclude a different control directory.
Add PosixOpts.ExcludeDirs and build the effective skip list once in New
from the gateway's own metadata directory plus the configured entries,
so all three listing operations read the same list. The new
--exclude-dirs flag (VGW_EXCLUDE_DIRS) exposes it, so a dataset mounted
with snapdir=visible is handled with --exclude-dirs .zfs.
.zfs isn't in the default list: on a filesystem without ZFS snapshots a
directory with that name is ordinary bucket content. An operator who
needs the skip sets --exclude-dirs .zfs.
S3Proxy.GetObjectAttributes dropped VersionId and DeleteMarker from the
upstream response, so the gateway never sent x-amz-version-id or
x-amz-delete-marker for objects behind an s3 backend. Copy them into the
response, and add tests for how the upstream response is mapped,
including the parts count.
Fixes#2511
`CreateMultipartUpload` in the posix backend creates the upload directory under `.sgwtmp/multipart` with `os.MkdirAll` and returned any failure wrapped as `create upload temp dir`, so a full filesystem quota or a full disk came back as a 500 `InternalError` (which SDKs also retry) instead of the error the other write paths return. `PutObject`, `UploadPart`, `UploadPartCopy` and `CompleteMultipartUpload` already map these errors.
`syscall.EDQUOT` from the directory creation now returns `QuotaExceeded` (403) and `syscall.ENOSPC` returns `InsufficientStorage` (507), the same mapping the rest of the posix write paths use.
The posix backend stages multipart uploads and temporary files inside the bucket directory under `.sgwtmp`, but nothing stopped an object key from resolving there. A client with access to the bucket could read the parts of other clients' in-progress uploads with `GetObject`/`HeadObject`, copy them out with `CopyObject`, overwrite or delete them with `PutObject`/`DeleteObject` so the upload failed on completion with `InvalidPart`, and store data that never shows up in listings. Spellings such as `./.sgwtmp/...`, `x/../.sgwtmp/...`, a leading `/` and, on case-insensitive file systems, `.SGWTMP/...` reached the same files.
Add the non-AWS `ReservedObjectKey` error (400) and return it whenever an object key resolves to `.sgwtmp` or a path inside it. The key is resolved the same way `ObjectPath` resolves it, and its first path element is compared case-insensitively. The check runs after the bucket existence check in `PutObject`, `GetObject`, `HeadObject`, `GetObjectAttributes`, `DeleteObject`, `DeleteObjects` (reported per key), `CopyObject` (source and destination), the `UploadPartCopy` source, `CreateMultipartUpload`, `CompleteMultipartUpload`, object tagging, legal hold and retention. The exported `ValidateObjectKey` covers the ScoutFS `GetObject` and `RestoreObject` entry points. `UploadPart`, `ListParts` and `AbortMultipartUpload` only hash the key and stay unchanged, so uploads created with a reserved key before this fix can still be aborted.
`ListObjects` and `ListObjectsV2` also listed the staged parts when the prefix named a directory inside `.sgwtmp`, because the prefix fast path in `quickWalk` compared only the last element of the prefix directory against the skipped directories. `inSkipDir` now checks every element, ignoring case.
The controller now passes the requested attributes to the backend in
ObjectAttributes, so posix and azure only do the extra HeadObject for the
parts count when ObjectParts was asked for. This also lets the s3 backend
reach the upstream service, which rejected the request before because the
field was missing.
The s3 backend now copies TotalPartsCount into PartsCount, leaves out
ObjectParts when the upstream response has none, and sets the part number
markers, which were dropped by inverted strconv.Atoi error checks.
Fixes#2513
On Windows the posix backend stored a key as `bucket/key` verbatim, but Windows rejects `" * : < > ? \ |` and control characters in file names, so such a PUT failed with `ERROR_INVALID_NAME`, which `isErrNameTooLong` reported as a misleading `KeyTooLongError`. The same mapping did worse elsewhere: `\` acted as a path separator, so `a\b` aliased `a/b` and `..\..\x` resolved outside the bucket because `IsObjectNameValid` only splits on `/`; `a:b` wrote an NTFS alternate data stream; Win32 strips a trailing dot or space, so `foo.` overwrote `foo`; and device names such as `CON` or `nul.txt` opened the device and silently discarded the data.
With `encodeKeys`, enabled by default only on Windows, `encodeKey` stores each key segment under a name Windows accepts: every invalid character, a trailing dot or space, and the first letter of a device name are stored as the private use character `U+F000` plus the character code, the convention Cygwin and WSL use for invalid characters. A key character in `U+F000`-`U+F07F` is stored after `U+F000`, so every stored name belongs to exactly one key. Device names are checked with `filepath.IsLocal`, which asks the system, so `CON.txt` stays a plain file on Windows 11 where it is valid. Keys that need no encoding keep their names, so existing data is unaffected, and nothing changes on other platforms.
Public backend methods keep taking raw keys and encode them once for every path, metadata, version-hash and publish-lock use, while responses, errors and the multipart `objname` attribute keep the raw key and multipart upload directories keep hashing it. Listings walk `bucketFS`, which decodes the entry names, sorts them in key byte order to match S3 and skips names no key is stored under. `HeadObject` passes the raw key to `GetObjectLegalHold` and `GetObjectRetention` to avoid encoding twice, and `NormalizeObjectKey` resolves the encoded key, so policy resources see `a\b` as one literal key.
GetObjectAttributes never returned ObjectParts, so a request for it on
an object completed from a multipart upload got no PartsCount. S3
returns <ObjectParts><PartsCount>N</PartsCount></ObjectParts> for these
objects and leaves ObjectParts out for objects that weren't uploaded in
parts.
The posix and azure backends already keep the part layout of completed
uploads, and HeadObject reports the parts count when a part number is
requested. GetObjectAttributes now uses that to fill ObjectParts. The
ObjectParts fields that aren't set are left out of the response, as S3
does.
Fixes#925
The api layer validates a client's LocationConstraint against the gateway
region, but the bucket is created in the s3 backend's own region
(--region / VGW_S3_REGION). They are separate settings, so passing the
validated client constraint through to the backend only worked when the two
regions matched. With the gateway on the default us-east-1 and the backend
on eu-west-1 the client sends no constraint at all, and the backend still
answered
IllegalLocationConstraintException: The unspecified location constraint is
incompatible for the region specific endpoint this request was sent to.
which is the error in #2490.
Set the constraint in S3Proxy.CreateBucket from the backend client's region
instead, next to the existing empty-configuration drop. A backend on
us-east-1 still gets no constraint, and the api layer's rejection of a
client constraint that disagrees with the gateway region is unchanged.
Fixes#2490
A conditional delete of a key with no current version used to succeed. `DeleteObject` with `If-Match`, and a `DeleteObjects` key with an `ETag`, reported success for a missing key. When the current version was a delete marker, the condition was checked against the ETag the marker kept from the version it hides, so it could pass and create a second marker. S3 fails both with `NoSuchKey` and creates no marker. The posix backend now checks the conditions against the current version in `checkDeletePreconditions`, before its missing-key shortcuts. The azure backend returns `NoSuchKey` for a missing blob instead of skipping the conditions. `ObjectDeletePreconditions.IsSet` reports whether a delete has any condition.
A condition only applies to the current version. `DeleteObject` with a `versionId` and `If-Match` now returns `NotImplemented` with `<Header>If-Match</Header>` and the S3 additional message, ahead of authorization. Before, it checked the condition against that version and deleted the version. In `DeleteObjects`, an empty `<ETag>` fails its key with `InvalidArgument` and a `VersionId` together with an `ETag` fails it with `NotImplemented`, both ahead of the key's authorization.
The `PreconditionFailed` and `NotImplemented` messages drop their trailing period, which S3 doesn't use on any operation.
DeleteObjects dropped the ETag, Size and LastModifiedTime of each object
when it called DeleteObject, so an object with a stale ETag was still
deleted. The posix and azure backends now forward them to DeleteObject.
EvaluateObjectDeletePreconditions trims the quotes an SDK sends around
the ETag and treats "*" as a match for any ETag.
The REST ETag mismatch test no longer skips.
Taken from #2485, without its posix and integration tests.
Refs #2112
Fixes#2499
An empty tag key returned `InvalidTag` without the `TagKey` element because the field was `omitempty`. S3 always sends `<TagKey>`, empty for an empty key, so it is now always written.
`PutBucketTagging`, `PutObjectTagging`, the POST object `tagging` field and `CreateBucket` tags now share one validator, `backend.ParseTagSet`, which matches S3: a duplicate key error carries `TagKey`, a `Tag` missing its `Key` or `Value` element is `MalformedXML` and is checked across the whole document first, every tag is validated before the tag count limit, and the 128/256 length limits count UTF-16 code units instead of bytes. The duplicate tag regex in `s3api/utils` is removed.
For the `x-amz-tagging` header, the length limits now apply to the URL-decoded key and value, the `InvalidArgument` error echoes the whole header as `ArgumentValue` instead of the unparsed remainder, and more than 10 tags is rejected with `BadRequest`.
The posix backend publishes an object by linking or renaming a temp file
into place, but never flushes it. An acknowledged PUT can therefore be
lost, or come back empty, after a crash or power loss, depending on the
filesystem's writeback interval.
With --fsync (VGW_FSYNC), the backend fsyncs the temp file, which covers
its data and xattr metadata, before publishing it. After publishing, it
fsyncs the object's directory and each parent up to the bucket, so that
the new entry and any directories created for it are durable. A failed
flush fails the request. The option is off by default, so existing
deployments are unaffected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
In posix, when the upload directory rename in `CompleteMultipartUploadWithCopy` fails with `ENOENT`, the call is treated as a retry of an already completed upload. That fallback returned 200 whenever any regular file existed at the target bucket/key, with an ETag computed from the request's part ETags, and never checked that the object came from this upload. Completing with the upload ID of another bucket or key, a random upload ID, or an aborted upload was reported as stored while nothing was written. The later `mp-metadata` upload ID check was unreachable whenever the object existed.
The fallback now succeeds only when the object at the key carries this upload ID in its `mp-metadata` attribute and, outside `--data-integrity-etag` mode, when its stored ETag equals the multipart ETag computed from the request's parts. This matches AWS, which rejects a retry with different part ETags with `NoSuchUpload` but ignores part numbers. Every other case returns `NoSuchUpload`, as AWS does for a foreign, random or aborted upload ID and for a retry after the object was overwritten or deleted. Keys with a trailing slash are rejected up front because `ObjectPath` resolves `foo/` to the file `foo`, and attribute read errors other than a missing attribute are no longer reported as `NoSuchUpload`. A successful retry now returns the stored ETag and the object's `version-id`, matching AWS, which returns the original VersionId on a retry in a versioned bucket.
The posix ListObjectsV2 reported the listing marker as the response
ContinuationToken. The marker is the larger of start-after and the
continuation token, so a request with only start-after got its
start-after value back as a ContinuationToken, and a start-after past
the continuation token replaced it. S3 echoes the continuation token
from the request, and leaves ContinuationToken out when the request
had none.
Return the request continuation token instead of the marker, and
enable the REST test that covers it.
Fixes#2007
CompleteMultipartUpload with a part ETag that is not a hex MD5 returned
a bare InvalidPart error, without the UploadId, PartNumber and ETag
elements. S3 includes them, and the gateway already does for a part
that is missing or whose ETag doesn't match.
Pass the upload ID to ComputeMultipartETagFromPartETags so it can
return the detailed InvalidPart error for the offending part, and
enable the REST test that covers it.
Fixes#1008
CopyObject ignored the destination's `If-Match` and `If-None-Match` headers and always overwrote or created the destination, so a stale `If-Match`, `If-None-Match: *` on an existing key, `If-Match` on a missing key and `If-Match` over a delete marker all returned 200. The controller now parses both headers and passes them to the backend through new `IfMatch` and `IfNoneMatch` fields on `s3response.CopyObjectInput`, which match the SDK's `s3.CopyObjectInput`.
As in S3, the destination is evaluated before the copy source is looked up, so a failing destination precondition wins over a missing source, a failing `x-amz-copy-source-*` precondition and an invalid copy onto itself. Posix checks the destination with `checkPutPreconditions` right after the destination bucket check, then forwards the headers to `PutObject`, which repeats the check under the object publish lock. The unversioned in-place self copy does not go through `PutObject`, so it now takes `lockObjectPublish` and repeats the check itself.
`checkPutPreconditions` now treats a current delete marker as a missing key: `If-Match` returns `NoSuchKey` and `If-None-Match: *` creates a new version. Before, the hidden version's etag was evaluated, which also affected `PutObject` and `CompleteMultipartUpload`.
`EvaluateObjectPutPreconditions` now rejects unsupported headers in S3's order: both headers present, then `If-None-Match` other than `*`, then `If-Match: *`, each with `NotImplemented`. `If-Match: *` was previously compared as a literal etag. The new `NmpAdditionalMessageIfMatch` message is added, and a stray backslash is removed from the `If-None-Match` message.
Azure evaluates the destination preconditions before copying, and s3proxy forwards the headers. `s3:if-match` and `s3:if-none-match` are now populated in the policy condition context for CopyObject, as S3 does, while UploadPartCopy stays excluded.
With an upload-id-marker the lister ranges over Uploads[startIndex:],
but passed the index of that subslice to hasMoreAfter, which indexes
the full list. A page that ended on the last remaining upload was still
reported as truncated, with next markers pointing at that upload, so a
client had to make one more request to get an empty page.
Iterate over the full list from startIndex, so hasMoreAfter gets the
real position of the next upload.
The bucket default retention was never stored on objects: protection was computed at delete time from the live bucket rule (`CreatedAt` + period), so changing or removing the rule unlocked existing objects, `COMPLIANCE` ones included, and a rule set N days ago stopped protecting even new objects. The posix and azure backends now store the rule's retention, `{Mode, now + Days/Years}` truncated to milliseconds, as the object's own retention whenever a write sets no Object Lock parameters: `PutObject` (which also covers POST, RDMA and `CopyObject`), `CreateMultipartUpload` (the rule and start time are fixed at creation, as S3 does) and the unversioned in-place self-copy, which applied no lock settings before. As on S3, any explicit `x-amz-object-lock-legal-hold`, `ON` or `OFF`, means the default retention is not applied. The bucket-rule fallback in `checkObject` and `BucketLockConfig.CreatedAt` are removed, so objects written before this change lose bucket-derived protection, and an expired retention no longer skips the legal hold check.
`PutObjectLockConfiguration` now rejects a `DefaultRetention` with a mode but no `Days`/`Years` with `InvalidRequest` "DefaultRetention must specify a fixed retention (Days or Years) or a DefaultEventHold.", and a `<Rule>` without `DefaultRetention` with `MalformedXML`. Azure `CopyObject` with the `COPY` metadata directive no longer copies the source's `Objectretention`/`Objectlegalhold` metadata to the destination.
Browser POST uploads now validate and apply the `x-amz-object-lock-mode`, `x-amz-object-lock-retain-until-date` and `x-amz-object-lock-legal-hold` form fields. They require `s3:PutObjectRetention`/`s3:PutObjectLegalHold`, for anonymous uploads too, and return `ErrMissingObjectLockConfigurationNoSpaces` on buckets without Object Lock after authorization. A denied POST names the actions in S3's order: retention, legal hold, tagging, then `s3:PutObject`.
`NoSuchObjectLockConfiguration` now returns 404 without a trailing period. `x-amz-object-lock-retain-until-date` on HEAD and GET has millisecond precision and omits zero milliseconds, `GetObjectRetention` returns `<Retention>` with a `.000Z` date, and posix and azure `GetObject` now return the Object Lock headers.
Add request plumbing for server-side encryption with customer-provided
keys (SSE-C) and stop silently ignoring it on backends that cannot
honor it.
Previously SSE-C headers were dropped before reaching the backend, so
posix stored the data unencrypted and returned success, and later
reads without the key succeeded. Clients believed their data was
encrypted when it was not. Partial or malformed SSE-C header sets were
also accepted instead of failing with 400 as S3 does. Rejecting SSE-C
explicitly is the safe behavior until posix implements it, and azure
cannot faithfully map SSE-C onto customer-provided keys.
This addresses part of #2193: invalid/partial SSE-C headers now fail
with 400. Requiring SSE-C headers on reads of SSE-C objects depends on
posix SSE-C support and remains open.
The changes also reject unsupported SSE-S3, SSE-KMS, and DSSE-KMS
requests instead of silently accepting them. And enforcea TLS for
SSE-C by default, with --disable-sse-c-tls-enforcement as the opt-out
in case the deployment handles TLS in front of the gateway.
NextKeyMarker and NextUploadIdMarker were only set for a truncated
listing, so a ListMultipartUploads response that fit in one page
returned them empty. S3 still sets them to the key and upload ID of
the last upload in the response.
Fill both markers from the last listed upload when the listing is not
truncated, including the case where the page ends exactly at
max-uploads. A page of common prefixes only keeps them empty, as
before. Enable the REST test that covers it.
Fixes#2144
The posix and azure backends dropped every upload with a key equal to
the key-marker before handing the list to the lister, which then looked
for the upload-id-marker under the next key. An upload-id-marker that
belongs to the key-marker could never be found, so the request failed
with "Invalid uploadId marker", including when a client paged with the
NextKeyMarker and NextUploadIdMarker the gateway returned.
Keep the uploads of the key-marker itself when an upload-id-marker is
set, so the listing continues right after the given upload, as S3 does.
Enable the REST test that covers it.
Fixes#2149
ScoutFS delegates buffered object transfer paths to the
POSIX backend, so expose IOBufferSize in ScoutfsOpts and
pass it through to PosixOpts. This lets operators tune
transfer buffering for ScoutFS instead of being limited
to the default.
Uploads copied the request body with io.CopyBuffer, which issues one write
per read. Network reads return arbitrary lengths, so the first short read
produced an unaligned write and the tmpfile silently reopened buffered for
the rest of the object.
- Fill the I/O buffer completely before each O_DIRECT write so only the
final tail is unaligned; write that tail buffered without treating it
as a fallback.
- Allocate I/O buffers 4 KiB aligned and use 4 KiB alignment for O_DIRECT
lengths and offsets to cover 4Kn devices.
- Replace the O_DIRECT read wrapper with a reader that issues aligned
preads into an aligned buffer for full and ranged GETs. sendfile from an
O_DIRECT fd fails with EINVAL after a partial socket write, so O_DIRECT
reads no longer go through sendfile; buffered reads still do.
- Debug log unexpected O_DIRECT fallbacks on both paths.
Creating a delete marker in posix keeps the file of the hidden version at the object path and only marks it with the delete marker attribute, so the delete marker kept the modification time of the version it hides. `ListObjectVersions` listed that time for current and noncurrent delete markers, and `HeadObject` and `GetObject` sent it as the `Last-Modified` header of the `MethodNotAllowed` response for a delete marker `versionId`. AWS reports the time the delete marker is created, including for the null delete marker created while versioning is suspended. `DeleteObject` now sets the modification time of the delete marker to the time it's created. Setting the time needs the entry to be owned by the gateway, so it is left as is otherwise, like it is for directory objects.
`ListObjectVersions` also listed the delete markers with the SDK `types.DeleteMarkerEntry`, which has no XML marshaling of its own, so `LastModified` was sent in the local time zone of the gateway and with a sub-second part, while the versions are sent in UTC to the second. The delete markers are now listed with the new `s3response.DeleteMarkerEntry`, which formats `LastModified` the same way as `s3response.ObjectVersion`, and the s3proxy backend converts the upstream delete markers to it.
Deleting the current version of an object by `versionId` makes the newest remaining version current again. Posix copies that version from the versioning directory into a new temporary file and links it in place of the object, but never restored the file's modification time, so `HeadObject`, `ListObjectVersions` and `ListObjectsV2` reported the delete time as the promoted version's `LastModified`. This affected an older version, a version under a deleted delete marker and the null version alike, while AWS keeps the original `LastModified` in all three cases. Directory objects were not affected, as `deleteDirObjectLatestVersion` already restores the directory's mtime.
The temporary file now carries an optional modification time, set with `setModTime` and applied by `link()` right before the file is published: through `/proc/self/fd` before `linkat` for an `O_TMPFILE` file, and after the file is closed and before the rename on the `CreateTemp` fallback and on non-Linux platforms. Both the promotion in `DeleteObject` and `createObjVersion` use it. `createObjVersion` used to call `os.Chtimes` on the temporary file while it was still open for writing, and Windows resets the last write time when a handle the file was written through is closed.
The `sidecar` metadata storer keeps attributes by path rather than on the file. So a new file published at an object's path inherited every attribute it didn't set itself: tags, user metadata, content headers and `x-amz-website-redirect-location`, plus legal hold and retention on version promotion. This hit overwrites by `PutObject`, `CopyObject` and `CompleteMultipartUpload`, new versions and `PutObject` over a delete marker in versioned buckets, and the version made current again by `DeleteObject` with a `versionId`. A plain `PutObject` over a multipart object also kept its `mp-metadata`, so `HeadObject` and `GetObject` with `partNumber` returned the old `PartsCount` and a `Content-Range` beyond the object size. `xattr` was not affected, since its attributes live on the inode and are replaced along with it.
Add `ReplaceObject(bucket, object, keep)` to `MetadataStorer`. It is a no-op for `XattrMeta` and `NoMeta`. `SideCar` removes the object's attribute files not named in `keep` and leaves directories in place, since they hold the metadata of objects under `<object>/meta/`. Posix calls it in `PutObject` and `CompleteMultipartUpload` after the version snapshot, and in `DeleteObject` when restoring the latest remaining version. This replaces the scattered deletes of the delete marker, `versionId` and object lock attributes. Attributes that every publish rewrites are kept and overwritten in place: the etag, the checksums, the new `versionId`, `mp-metadata` for multipart uploads, and the restored version's attributes on promotion. This way concurrent readers, such as the lock-free `If-Match` fast-fail check, never see them missing.
`PutObject` now re-checks under the publish lock that the object path hasn't become a directory while the body was staged, as `CompleteMultipartUpload` already does. This stops it from removing the attributes of a directory object created by a racing `PUT foo/`.
An authenticated user with write access to a single bucket could list, write and delete files outside the gateway root on the posix and scoutfs backends. The multipart `uploadId` and the `DeleteObjects` body keys were joined into filesystem paths without validation, and unlike object keys in the request path they never passed the object name validator that rejects `..`. With a `../` upload ID, `ListParts` listed the contents of an outside directory, `UploadPart` and `UploadPartCopy` wrote part files into it, `CompleteMultipartUpload` renamed and removed it, and `AbortMultipartUpload` removed it recursively. A `../` key in `DeleteObjects` removed the outside file, after which the empty-parent cleanup removed emptied directories above the root, and a `/` key removed an empty bucket's directory.
Posix now rejects any upload ID it could not have generated. Posix has always generated upload IDs with `uuid.New().String()`, so `validateUploadID` accepts only a UUID in that exact canonical form and returns `NoSuchUpload` for anything else, which is what AWS returns for a malformed upload ID. The check runs right after the bucket existence check in all five multipart operations that take an upload ID, before any path is built from it.
The `DeleteObjects` controller now checks every key with `IsObjectNameValid`, the same check already applied to request-path, copy-source and POST object keys. If any key is missing or resolves outside the bucket, the whole request is rejected with `BadRequest` before authorization, so no key in the batch is deleted. AWS also fails the whole batch when one key is invalid.
Ben McClellandandGitHub
niksis02
LuisFigueroaG
Tung Lam
theluckystrikeandniksis02
Keith KennedyandClaude Opus 5.5
DualFroz