Object key validation allowed internal parent-directory segments such as `public/../private.txt`. Bucket policy and auth checks evaluated the raw key, so a policy allowing bucket/public/* could match the request while posix backend later resolved the key with `filepath.Join` and accessed `bucket/private.txt`.
Add backend-specific object key normalization to close that mismatch. The Backend interface now exposes `NormalizeObjectKey` so authorization can evaluate resources using the same key shape a backend will use for storage access.
Backends that do not collapse object paths, including Azure and the S3 proxy, inherit `BackendUnsupported.NormalizeObjectKey`. That implementation returns the input key unchanged, avoiding unnecessary normalization and keeping policy evaluation unpolluted for object stores where ../ is part of the key name.
posix/scoutfs normalize keys with filepath.Join so policy resources and request keys are compared after internal dot segments are collapsed.
Bucket policy evaluation now normalizes both the incoming object key and object resource patterns from the policy before matching. Object lock governance bypass policy checks use the same backend normalizer as well, so retention and legal hold authorization cannot diverge from backend path resolution.
Fixes#2123Fixes#2120Fixes#2116Fixes#2111Fixes#2108Fixes#2086Fixes#2085Fixes#2083Fixes#2081Fixes#2080Fixes#2073Fixes#2072Fixes#2071Fixes#2069Fixes#2044Fixes#2043Fixes#2042Fixes#2041Fixes#2040Fixes#2039Fixes#2036Fixes#2035Fixes#2034Fixes#2028Fixes#2020Fixes#1842Fixes#1810Fixes#1780Fixes#1775Fixes#1736Fixes#1705Fixes#1663Fixes#1645Fixes#1583Fixes#1526Fixes#1514Fixes#1493Fixes#1487Fixes#959Fixes#779Closes#823Closes#85
Refactor global S3 error handling around structured error types and centralized XML response generation.
All S3 errors now share the common APIError base for the fields every error has: Code, HTTP status code, and Message. Non-traditional errors that need AWS-compatible XML fields now have dedicated typed errors in the s3err package. Each typed error implements the shared S3Error behavior so controllers and middleware can handle errors consistently while still emitting error-specific XML fields.
Add a dedicated InvalidArgumentError type because InvalidArgument is used widely across request validation, auth, copy source handling, object lock validation, multipart validation, and header parsing. The new InvalidArgument path uses explicit InvalidArgErrorCode constants with predefined descriptions and ArgumentName values, keeping call sites readable while preserving the correct InvalidArgument XML shape and optional ArgumentValue.
New structured errors added in s3err:
- `AccessForbiddenError`: Method, ResourceType
- `BadDigestError`: CalculatedDigest, ExpectedDigest
- `BucketError`: BucketName
- `ContentSHA256MismatchError`: ClientComputedContentSHA256, S3ComputedContentSHA256
- `EntityTooLargeError`: ProposedSize, MaxSizeAllowed
- `EntityTooSmallError`: ProposedSize, MinSizeAllowed
- `ExpiredPresignedURLError`: ServerTime, XAmzExpires, Expires
- `InvalidAccessKeyIdError`: AWSAccessKeyId
- `InvalidArgumentError`: Description, ArgumentName, ArgumentValue
- `InvalidChunkSizeError`: Chunk, BadChunkSize
- `InvalidDigestError`: ContentMD5
- `InvalidLocationConstraintError`: LocationConstraint
- `InvalidPartError`: UploadId, PartNumber, ETag
- `InvalidRangeError`: RangeRequested, ActualObjectSize
- `InvalidTagError`: TagKey, TagValue
- `KeyTooLongError`: Size, MaxSizeAllowed
- `MetadataTooLargeError`: Size, MaxSizeAllowed
- `MethodNotAllowedError`: Method, ResourceType, AllowedMethods
- `NoSuchUploadError`: UploadId
- `NoSuchVersionError`: Key, VersionId
- `NotImplementedError`: Header, AdditionalMessage
- `PreconditionFailedError`: Condition
- `RequestTimeTooSkewedError`: RequestTime, ServerTime, MaxAllowedSkewMilliseconds
- `SignatureDoesNotMatchError`: AWSAccessKeyId, StringToSign, SignatureProvided, StringToSignBytes, CanonicalRequest, CanonicalRequestBytes
Fix CompleteMultipartUpload validation in the Azure backend so missing or empty `ETag` values return the appropriate S3 error instead of allowing a gateway panic.
Fix presigned authentication expiration validation to compare server time in `UTC`, matching the `UTC` timestamp used by presigned URL signing.
Add request ID and host ID support across S3 requests. Each request now receives AWS S3-like identifiers, returned in response headers as `x-amz-request-id` and `x-amz-id-2` and included in all XML error responses as RequestId and HostId. The generated ID structure is designed to resemble AWS S3 request IDs and host IDs.
The request signature calculation/validation for streaming uploads was previously delayed until the request body was fully read, both for Authorization header authentication and presigned URLs.
Now, the signature is validated immediately in the authorization middlewares without reading the request body, since the signature calculation itself does not depend on the request body. Instead, only the `x-amz-content-sha256` SHA-256 hash calculation is delayed.
Directories that exist on the filesystem but were not explicitly created
via S3 (put-object with a key ending in '/') do not have an etag
value. ListObjectsV2 already uses the presence of this attribute to
decide whether to include a directory as an object. GetObject and
HeadObject were not performing this check, so they would successfully
return directories that ListObjectsV2 would not list.
Add the etag attribute check in GetObject and HeadObject: if a
directory path is requested but has no etag xattr, return 404. This
makes all three operations agree on which directories are S3 objects.
Fixes#2130
Store multipart upload metadata through shared backend helpers so POSIX and Azure use the same encode/decode path. POSIX stores raw gzipped JSON in metadata stores, while Azure stores base64-encoded gzip for string metadata. Retrieval falls back to the legacy raw JSON format for existing objects. Storing the mp metadata compressed in posix will guarantee that for any allowed number of parts, the metadata won't exceed the xattr threshold(64KB).
Integrate the new S3 checksum types in the gateway, including `SHA512`, `MD5`, `XXHASH64`, `XXHASH3`, and `XXHASH128`. This adds checksum calculation, validation, schema handling, and test coverage for the expanded checksum support.
These external packages have been used:
- `github.com/zeebo/xxh3` for `XXHASH3` and `XXHASH128`
- `github.com/cespare/xxhash/v2` for `XXHASH64`
Adjust integration tests because `aws-sdk-go-v2/service/s3` does not support automatic checksum calculation for the new checksum algorithms and returns an SDK-level error when only the checksum algorithm is provided. Only precalculated checksum values are acceptable for these checksum types.
References:
- `https://github.com/aws/aws-sdk-go-v2/issues/3404`
- `https://github.com/aws/aws-sdk-go-v2/issues/3403`
Add a new non-AWS error ErrNoSpaceLeftOnDevice (HTTP 507 Insufficient
Storage) to s3err. Update all call sites in the posix backend that
could return ENOSPC and return the new error when the underlying
filesystem has no space remaining.
Fixes#2093
Unlike xattr metadata which is tied to the filesystem object and removed
automatically, sidecar metadata lives in a parallel directory tree and
must be deleted explicitly. Add DeleteAttributes calls after removing
bucket directories, null-version files, and versioned object files.
Also add an os.Stat existence check in GetObjectTagging and
PutObjectTagging when no versionId is given, since sidecar's
StoreAttribute/RetrieveAttribute do not naturally return ErrNotExist
for missing objects the way xattr operations do.
Fixes#2052Fixes#2056Fixes#2057
Previously, GetObject and HeadObject used the request's `Range` header to determine the response status code, which caused incorrect 206 responses for invalid Range header values.
The status is now driven by whether res.ContentRange is set in the response, rather than by the presence of a range in the request. Backends (posix and azure) now set Content-Range for PartNumber=1 on non-multipart objects, skipping zero-size objects where no range applies.
HeadObject was also fixed to return 206 when Content-Range is present, and to only return checksums when the full object is requested.
Closes#1064
Use the multipart ETag as the in-progress directory suffix instead of the static `.inprogress` marker so that concurrent CompleteMultipartUpload calls for the same upload ID are all treated as successful (idempotent) rather than racing, where only one succeeded and the rest returned NoSuchUpload.
After finalizing the multipart upload, store an `mp-metadata` xattr on the assembled object that records the upload ID and cumulative byte offsets for each part. GetObject and HeadObject now use this metadata to serve individual part ranges via the `partNumber` query parameter, returning a successful response instead of returning NotImplemented.
Add two new S3 error codes:
- `ErrInvalidPartNumberRange` (416 RequestedRangeNotSatisfiable) — returned
when the requested part number exceeds the number of parts in the upload.
- `ErrRangeAndPartNumber` (400 BadRequest) — returned when both a Range header
and a partNumber query parameter are specified on the same request.
Closes#1813
We use a specific `versionId` format(`ulid` package) to generate versionIds in posix, which is not compatible to S3. The versionId validation was performed in frontend which is a potential source of failure for s3 proxy configured on an s3 service which doesn't use ulid for versionId generation(e.g. aws S3). These changes move the specific `ulid` versionId validation to posix to not force any specific versionId format in the gateway.
A concurrent PutObject and DeleteObject on the same prefix directory
can race:
PutObject opens an O_TMPFILE in MetaTmpDir (not yet visible in the fs)
DeleteObject removes the last visible object in the prefix directory
and calls removeParents(), which rmdir's the now-empty prefix
directory
PutObject's link() tries to link the fd into a parent directory that
no longer exists
Fix by detecting ENOENT in the final link step (Linkat, Rename, and
MoveFile) and retrying after recreating the parent directory.
Also extract linkatOTmpfile() to consolidate the Linkat+EEXIST→Renameat
logic that was previously inline in link().
Fixes#1988
Closes#1897
Extract the `X-Amz-Source-Expected-Bucket-Owner` header for CopyObject and UploadPartCopy. Verify the source bucket owner in the backend and if the provided access key id doesn't match, return an `AccessDenied` error.
Fixes#1896
Enforces the S3 `5 GiB` copy source size limit across the posix and azure
backends for `CopyObject` and `UploadPartCopy`, returning `InvalidRequest` when
the source object exceeds the threshold.
The limit is now configurable via `--copy-object-threshold`
(`VGW_COPY_OBJECT_THRESHOLD`, default 5 GiB).
A new `--mp-max-parts flag` (`VGW_MP_MAX_PARTS`, default `10000`) has been added to make multipart upload parts number limit configurable.
No integration test has been added, as GitHub Actions cannot reliably
handle large objects.
When two requests raced to complete the same multipart upload, the first
caller to finish would remove the part files and upload directory. The
second caller, already past the initial existence check, would then fail
mid-flight with confusing errors such as ErrInvalidPart or an I/O error
when trying to open a part that no longer exists.
Fix this by atomically renaming the upload directory from <uploadID> to
<uploadID>.inprogress at the very start of CompleteMultipartUploadWithCopy,
before any part data is read. A concurrent caller will now find the
original directory absent and receive a clean NoSuchUpload error. A
deferred rename restores the original name if the complete does not
succeed, allowing the client to retry.
ListMultipartUploads is updated to skip any directories whose name ends
in .inprogress so in-flight completes do not appear as pending uploads.
The scoutfs backend uses the move blocks ioctl when combining parts into the
final multipart upload object. Once a move blocks from any part is successful,
the original data is no longer in the part file. If the multipart upload fails
and retries, future complete multipart upload calls will not have the correct
data within the part files anymore.
To prevent this case, once a move blocks call is successful for an upload, any
future failure for the complete upload is set to auto-abort the upload to force
clients to re-upload the part data again.
When a client provides an invalid or incomplete body for a single-part
upload, the handler returns before the link stage. Factor tempfile
removal into cleanup() to catch all cases.
The standard io.Copy() will attempt to use copy_file_range when available.
However, this can cause problems with certain filesystems. Add an option
that will prevent io.Copy() from being able to use copy_file_range to force
a standard data copy between file descriptors when needed for completing
multipart uploads.
This is a fixup of the codebase using:
go run golang.org/x/tools/go/analysis/passes/modernize/cmd/modernize@latest -fix ./...
This has no bahvior changes, and only updates safe changes for
modern go features.
Fixes#1909
Previously, the mapping between object metadata and posix object was as follows: for each metadata key, we stored a separate xattr with the `user.X-Amz-Meta.<key>` prefix. This resulted in syscall overhead when storing and deleting large numbers of metadata keys.
In addition, very long metadata keys caused failures because most posix filesystems limit xattr key lengths to 127–255 bytes, while S3 does not enforce such a per-key limit.
The logic has now been changed so that all object metadata is stored in a single xattr, `user.metadata`, as a JSON key/value object. For backward compatibility, metadata GET operations still fall back to the old mechanism (`metadata key -> xattr key`) when `user.metadata` is not present.
A new CLI utility has been added to convert all legacy object metadata to the new metadata format within the provided directory.
**Example usage:**
```
versitygw utils convert-xattr-metadata path/to/bucket
```
or
```
versitygw utils cxm path/to/bucket
```
It is recommended to run this command on bucket directories to convert all legacy metadata for every object in the bucket.
Closes#1857
Adds object Tagging support for directory objects in `PutObject` posix. Updates the integration tests to test object metadata and tagging both for file and directory objects.
In sidecar mode, the three StoreAttribute/storeChecksums calls after the
copy loop were using objPath (the source object path) instead of the
destination part's bucket and partPath. This caused checksums and the
internal part-crc64nvme to be written under the source object's sidecar
directory, making them unresolvable when CompleteMultipartUploadWithCopy
tried to retrieve them. All three stores now use *upi.Bucket and partPath,
consistent with the etag store directly below them.
This PR optimizes multipart upload checksum handling. When a checksum algorithm/type is specified at multipart-upload initiation, each `UploadPart` request computes, validates, and stores the corresponding part checksum. During `CompleteMultipartUpload`, the final checksum is derived either via composite checksum calculation or by composing the CRC-family checksums.
When **no** checksum algorithm is specified during multipart-upload initiation, each `UploadPart` may supply a different checksum algorithm for data-integrity verification. To support this scenario, a new mechanism has been implemented: for every `UploadPart`, a **crc64nvme** checksum is always computed.
* If the client uses crc64nvme for the part upload, a single hash reader is used.
* Otherwise, two hash readers are used—one for crc64nvme and one for the user-provided checksum.
The crc64nvme value is stored in part xattrs under `user.part-crc64nvme` and later used during `CompleteMultipartUpload` as a composable checksum source.
In `CompleteMultipartUpload`, the hash reader is entirely removed; the gateway no longer re-reads part data to compute the final checksum. The logic now follows two distinct paths:
1. **Checksum algorithm/type specified at MP initiation**
* All required per-part checksums have already been stored.
* If the checksum type is `FULL_OBJECT`, the gateway uses the composable path.
* If the type is `COMPOSITE`, the gateway follows the checksum-combining path.
2. **No checksum algorithm specified at MP initiation**
* The gateway loads the stored per-part `crc64nvme` values and composes them to compute the final checksum.
The previous `composableCRC` check has been removed because all `FULL_OBJECT` algorithms are inherently composable (`crc32`, `crc32c`, `crc64nvme`). Validation now relies solely on `checksum.Type`.
Previously, if no object checksum type/algorithm was specified when initiating a multipart upload, the CompleteMultipartUpload request would compute the final object’s CRC64NVME checksum but not persist it. This logic has now been fixed, and in the scenario described above the checksum is stored on the final object. There should no longer be any case where a CompleteMultipartUpload request finishes without persisting the final object checksum.
This brings scoutfs in-line with the posix concurrency limiter.
This fixes a hang with scoutfs due to not correctly initializing
the concurrency in posix leading to a concurrency of 0 allowed.
This also adds a sane default to the posix concurrency when not
initialized.
Add data-integrity checksum support in `PutObject` in the POSIX backend for directory objects. Since the only way to upload a directory object is via `PutObject`, this logic validates and stores the checksum of the empty payload. Support for `GetObject` has also been added to retrieve and return directory-object checksums.
Closes#1815
Implements posix actions concurrency limiter. Since posix actions perform filesystem-heavy syscalls, a semaphore-based limiter is introduced to cap the maximum number of concurrent posix actions. When the limit is reached, additional action calls block until a slot becomes available.
For internal posix calls, the `no_acquire_slot` context key is used to prevent acquiring the limiter multiple times within a single action (e.g., PutObject internally calling PutObjectLegalHold).
The posix concurrency limit can be configured via the gateway posix subcommand flag (--concurrency) or the environment variable `VGW_POSIX_CONCURRENCY`. The default value is `5000`.
Fixes#1809Fixes#1806Fixes#1804Fixes#1794
This PR focuses on correcting so-called "list-limiter" parsing and validation. The affected limiters include: `max-keys`, `max-uploads`, `max-parts`, `max-buckets`, `max-uploads` and `part-number-marker`. When a limiter value is outside the integer range, a specific `InvalidArgument` error is now returned. If the value is a valid integer but negative, a different `InvalidArgument` error is produced.
`max-buckets` has its own validation rules: completely invalid values and values outside the allowed range (`1 <= input <= 10000`) return distinct errors. For `ListObjectVersions`, negative `max-keys` values follow S3’s special-case behavior and return a different `InvalidArgument` error message.
Additionally, `GetObjectAttributes` now follows S3 semantics for `x-amz-max-parts`: S3 ignores invalid values, so the gateway now matches that behavior.
Fixes#1792Fixes#1747Fixes#1797Fixes#1799
This PR primarily introduces delimiter support and several bug fixes for the `ListMultipartUploads` action in the POSIX and Azure backends. Delimiter handling is now implemented — when a delimiter is present in multipart-upload object key names, the backend collects and returns the appropriate common prefixes.
This functionality is achieved by introducing a common multipart-upload lister in the backend package. All backends (Azure, POSIX) now use this lister. The lister accepts a list that is already sorted and filtered by `KeyMarker` and `Prefix`.
Previously, the `KeyMarker` was required to exactly match an existing multipart-upload object key. This restriction is removed. The listing now relies on a lexicographical comparison between the provided `KeyMarker` and existing multipart-upload object keys.
Validation for `UploadIdMarker` is also added to correctly return an `InvalidArgument` error for invalid upload IDs. If `KeyMarker` is missing, the `UploadIdMarker` is ignored entirely. If `KeyMarker` is provided, a valid upload ID is one that matches an upload belonging to *the first object key after the KeyMarker*. For example, if the `KeyMarker` is `foo`, but the provided `UploadIdMarker` corresponds to an upload under `quxx`, it is invalid. It must match one of the uploads for the next object key equal to `foo`.
Finally, this PR fixes multipart-upload sorting. Multipart uploads must be sorted primarily lexicographically by their object key, and secondarily—when multiple uploads share the same object key—by their initiation time in ascending order.
In the POSIX `ListParts` implementation, there was a code snippet that loaded the object metadata, even though it wasn’t needed and never used in the response. This redundant code has now been removed.
The previous logic was not allowing put-object on windows when the parent directory did not already exist, and would
not always return the correct error if an ancestor in the path already existed as a file.
The problem is the different behavior of the os.Stat command in Windows compared to *nix in backend/posix/posix.go
in function PutObjectWithPostFunc. The os.Stat returns ENOTDIR on *nix if the parent object is a file instead of a
directory. On Windows, if the parent object does not exist at all, the return code of such os.Stat is ERROR_PATH_NOT_FOUND which is mapped to ENOTDIR. However this is inappropriate in this case. As a result, the
return code of the os.Stat is incorrectly interpreted as if the parent object is a file instead of the parent object does not
exist. Which then leads to a failed upload.
This fix validates the existing parent structure on put to make sure the correct error is returned or the put is successful.
Fixes#1702
Fixes#1766Fixes#1750
This PR focuses on two bug fixes:
First, it blocks access to delete `DeleteMarkers` for the following operations by returning a `MethodNotAllowed` error: `PutObjectTagging`, `GetObjectTagging`, `DeleteObjectTagging`, `PutObjectLegalHold`, `GetObjectLegalHold`, `PutObjectRetention`, and `GetObjectRetention`.
Second, it removes the access check that previously prevented deleting a delete marker locked by a bucket default retention rule. A delete marker should always be allowed to be deleted.
Fixes#1757Fixes#1758
When attempting to delete a non-existing object in a versioning-enabled bucket while specifying a `versionId`, VersityGW previously returned an internal error if the object had a parent file object, and an `InvalidArgument` error if the object did not exist. This PR fixes both behaviors and now returns a successful response that includes the `versionId`.
Fixes#1751
When an object lock–related operation is performed on an object in a bucket where Object Lock is not enabled, an `InvalidRequest` error is returned; however, the error message differs for some actions. This PR introduces a new error, `ErrMissingObjectLockConfigurationNoSpaces`, for `PutObject`, `CopyObject`, and `CreateMultipartUpload` to maintain compatibility with S3 in terms of the error message. It also adds the missing integration tests for these actions.
Fixes#1741
An object delete request without a `versionId` results in the creation of a new delete marker in versioning-enabled buckets. Even if the latest object version is locked, a new delete marker must still be created.
This implementation skips the object lock check for delete requests in versioning-enabled buckets when the `versionId` is missing, allowing the delete marker to be created as expected.
Additionally, it introduces a flag in the `createObjVersion` method in POSIX to remove unnecessary xattr attributes from an object after creating a new object version. A delete marker must not carry object-specific attributes such as tagging, legal hold, or retention. Currently, the cleanup is limited to legal hold and retention attributes, but this list will be expanded after fixing issue #1751.
niksis02
Ben McClellandandGitHub
Anael ORLINSKI
Jakob van SantenandGitHub
khambrechtandGitHub