ParseCopySource indexes into copySourceHeader[0] without checking
for an empty string, causing an index-out-of-range panic.
The three backend callers (azure, posix) pass the x-amz-copy-source
header value directly, so a malformed or missing header propagates
an empty string into ParseCopySource.
Add an empty-string guard at the top of the function that returns
an InvalidArgCopySourceBucket error, consistent with the existing
error returned when the source path has no bucket/object separator.
Add a table-driven test case that reproduces the panic without the
fix and verifies the correct error with the fix.
Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
Object key validation allowed internal parent-directory segments such as `public/../private.txt`. Bucket policy and auth checks evaluated the raw key, so a policy allowing bucket/public/* could match the request while posix backend later resolved the key with `filepath.Join` and accessed `bucket/private.txt`.
Add backend-specific object key normalization to close that mismatch. The Backend interface now exposes `NormalizeObjectKey` so authorization can evaluate resources using the same key shape a backend will use for storage access.
Backends that do not collapse object paths, including Azure and the S3 proxy, inherit `BackendUnsupported.NormalizeObjectKey`. That implementation returns the input key unchanged, avoiding unnecessary normalization and keeping policy evaluation unpolluted for object stores where ../ is part of the key name.
posix/scoutfs normalize keys with filepath.Join so policy resources and request keys are compared after internal dot segments are collapsed.
Bucket policy evaluation now normalizes both the incoming object key and object resource patterns from the policy before matching. Object lock governance bypass policy checks use the same backend normalizer as well, so retention and legal hold authorization cannot diverge from backend path resolution.
Fixes#2123Fixes#2120Fixes#2116Fixes#2111Fixes#2108Fixes#2086Fixes#2085Fixes#2083Fixes#2081Fixes#2080Fixes#2073Fixes#2072Fixes#2071Fixes#2069Fixes#2044Fixes#2043Fixes#2042Fixes#2041Fixes#2040Fixes#2039Fixes#2036Fixes#2035Fixes#2034Fixes#2028Fixes#2020Fixes#1842Fixes#1810Fixes#1780Fixes#1775Fixes#1736Fixes#1705Fixes#1663Fixes#1645Fixes#1583Fixes#1526Fixes#1514Fixes#1493Fixes#1487Fixes#959Fixes#779Closes#823Closes#85
Refactor global S3 error handling around structured error types and centralized XML response generation.
All S3 errors now share the common APIError base for the fields every error has: Code, HTTP status code, and Message. Non-traditional errors that need AWS-compatible XML fields now have dedicated typed errors in the s3err package. Each typed error implements the shared S3Error behavior so controllers and middleware can handle errors consistently while still emitting error-specific XML fields.
Add a dedicated InvalidArgumentError type because InvalidArgument is used widely across request validation, auth, copy source handling, object lock validation, multipart validation, and header parsing. The new InvalidArgument path uses explicit InvalidArgErrorCode constants with predefined descriptions and ArgumentName values, keeping call sites readable while preserving the correct InvalidArgument XML shape and optional ArgumentValue.
New structured errors added in s3err:
- `AccessForbiddenError`: Method, ResourceType
- `BadDigestError`: CalculatedDigest, ExpectedDigest
- `BucketError`: BucketName
- `ContentSHA256MismatchError`: ClientComputedContentSHA256, S3ComputedContentSHA256
- `EntityTooLargeError`: ProposedSize, MaxSizeAllowed
- `EntityTooSmallError`: ProposedSize, MinSizeAllowed
- `ExpiredPresignedURLError`: ServerTime, XAmzExpires, Expires
- `InvalidAccessKeyIdError`: AWSAccessKeyId
- `InvalidArgumentError`: Description, ArgumentName, ArgumentValue
- `InvalidChunkSizeError`: Chunk, BadChunkSize
- `InvalidDigestError`: ContentMD5
- `InvalidLocationConstraintError`: LocationConstraint
- `InvalidPartError`: UploadId, PartNumber, ETag
- `InvalidRangeError`: RangeRequested, ActualObjectSize
- `InvalidTagError`: TagKey, TagValue
- `KeyTooLongError`: Size, MaxSizeAllowed
- `MetadataTooLargeError`: Size, MaxSizeAllowed
- `MethodNotAllowedError`: Method, ResourceType, AllowedMethods
- `NoSuchUploadError`: UploadId
- `NoSuchVersionError`: Key, VersionId
- `NotImplementedError`: Header, AdditionalMessage
- `PreconditionFailedError`: Condition
- `RequestTimeTooSkewedError`: RequestTime, ServerTime, MaxAllowedSkewMilliseconds
- `SignatureDoesNotMatchError`: AWSAccessKeyId, StringToSign, SignatureProvided, StringToSignBytes, CanonicalRequest, CanonicalRequestBytes
Fix CompleteMultipartUpload validation in the Azure backend so missing or empty `ETag` values return the appropriate S3 error instead of allowing a gateway panic.
Fix presigned authentication expiration validation to compare server time in `UTC`, matching the `UTC` timestamp used by presigned URL signing.
Add request ID and host ID support across S3 requests. Each request now receives AWS S3-like identifiers, returned in response headers as `x-amz-request-id` and `x-amz-id-2` and included in all XML error responses as RequestId and HostId. The generated ID structure is designed to resemble AWS S3 request IDs and host IDs.
The request signature calculation/validation for streaming uploads was previously delayed until the request body was fully read, both for Authorization header authentication and presigned URLs.
Now, the signature is validated immediately in the authorization middlewares without reading the request body, since the signature calculation itself does not depend on the request body. Instead, only the `x-amz-content-sha256` SHA-256 hash calculation is delayed.
Directories that exist on the filesystem but were not explicitly created
via S3 (put-object with a key ending in '/') do not have an etag
value. ListObjectsV2 already uses the presence of this attribute to
decide whether to include a directory as an object. GetObject and
HeadObject were not performing this check, so they would successfully
return directories that ListObjectsV2 would not list.
Add the etag attribute check in GetObject and HeadObject: if a
directory path is requested but has no etag xattr, return 404. This
makes all three operations agree on which directories are S3 objects.
Fixes#2130
Store multipart upload metadata through shared backend helpers so POSIX and Azure use the same encode/decode path. POSIX stores raw gzipped JSON in metadata stores, while Azure stores base64-encoded gzip for string metadata. Retrieval falls back to the legacy raw JSON format for existing objects. Storing the mp metadata compressed in posix will guarantee that for any allowed number of parts, the metadata won't exceed the xattr threshold(64KB).
Integrate the new S3 checksum types in the gateway, including `SHA512`, `MD5`, `XXHASH64`, `XXHASH3`, and `XXHASH128`. This adds checksum calculation, validation, schema handling, and test coverage for the expanded checksum support.
These external packages have been used:
- `github.com/zeebo/xxh3` for `XXHASH3` and `XXHASH128`
- `github.com/cespare/xxhash/v2` for `XXHASH64`
Adjust integration tests because `aws-sdk-go-v2/service/s3` does not support automatic checksum calculation for the new checksum algorithms and returns an SDK-level error when only the checksum algorithm is provided. Only precalculated checksum values are acceptable for these checksum types.
References:
- `https://github.com/aws/aws-sdk-go-v2/issues/3404`
- `https://github.com/aws/aws-sdk-go-v2/issues/3403`
The MoveData() requires that all but the last part be 4k aligned.
We accidentally were including the alignment check for the last
part causing large uploads where the total object was not a
multiple of 4k to fallback to copying the last part. For very large
part sizes this was triggering timeouts in some clients.
Add a new non-AWS error ErrNoSpaceLeftOnDevice (HTTP 507 Insufficient
Storage) to s3err. Update all call sites in the posix backend that
could return ENOSPC and return the new error when the underlying
filesystem has no space remaining.
Fixes#2093
Unlike xattr metadata which is tied to the filesystem object and removed
automatically, sidecar metadata lives in a parallel directory tree and
must be deleted explicitly. Add DeleteAttributes calls after removing
bucket directories, null-version files, and versioned object files.
Also add an os.Stat existence check in GetObjectTagging and
PutObjectTagging when no versionId is given, since sidecar's
StoreAttribute/RetrieveAttribute do not naturally return ErrNotExist
for missing objects the way xattr operations do.
Fixes#2052Fixes#2056Fixes#2057
Previously, GetObject and HeadObject used the request's `Range` header to determine the response status code, which caused incorrect 206 responses for invalid Range header values.
The status is now driven by whether res.ContentRange is set in the response, rather than by the presence of a range in the request. Backends (posix and azure) now set Content-Range for PartNumber=1 on non-multipart objects, skipping zero-size objects where no range applies.
HeadObject was also fixed to return 206 when Content-Range is present, and to only return checksums when the full object is requested.
Closes#1064
Use the multipart ETag as the in-progress directory suffix instead of the static `.inprogress` marker so that concurrent CompleteMultipartUpload calls for the same upload ID are all treated as successful (idempotent) rather than racing, where only one succeeded and the rest returned NoSuchUpload.
After finalizing the multipart upload, store an `mp-metadata` xattr on the assembled object that records the upload ID and cumulative byte offsets for each part. GetObject and HeadObject now use this metadata to serve individual part ranges via the `partNumber` query parameter, returning a successful response instead of returning NotImplemented.
Add two new S3 error codes:
- `ErrInvalidPartNumberRange` (416 RequestedRangeNotSatisfiable) — returned
when the requested part number exceeds the number of parts in the upload.
- `ErrRangeAndPartNumber` (400 BadRequest) — returned when both a Range header
and a partNumber query parameter are specified on the same request.
As multipart uploads are translated to blobs in azure blob storage, they were visible in ListObjects(V2) as complete objects. Now the blobs with multipart prefix are filtered out during listing.
The listing logic is rewritten client-side to implement proper S3 semantics: flat blob enumeration with manual delimiter handling, correct truncation (IsTruncated only set when more items genuinely exist beyond maxKeys), and StartAfter/Marker/ContinuationToken applied via the lexicographic max of both constraints in ListObjectsV2.
For the same reason bucket deletion was not allowed. Now multipart objects are explicitly checked on bucket deletion and any pending multipart upload doesn't block the bucket deletion anymore.
On Windows, a file apparently cannot be renamed while an open
handle to it exists.
The previous code used defer tempfile.Close(), which meant the
handle was still open when os.Rename was called, producing:
failed to rename temporary file: The process cannot access
the file because it is being used by another process.
Fix by closing the file explicitly before the rename.
Fixes#2021
The AWS SDK v2 includes Accept-Encoding in SigV4 signed headers
which causes GCS to return a SignatureDoesNotMatch error because
GCS rewrites that header internally before verifying the signature.
Add a --gcs-compatibility / VGW_S3_GCS_COMPATIBILITY option for the s3proxy
backend that injects two Smithy finalize-layer middlewares: one removes
Accept-Encoding from the request immediately before the Signing step, and
a second restores it after signing so the header is still sent on the wire.
see: https://github.com/aws/aws-sdk-go-v2/issues/1816
This can be removed once GCS fixes this incompatibility.
Fixes#2030
When an object has offline blocks, the restore status was incorrectly set to `ongoing-request="false"` instead of omitting the header entirely, which causes s3 clients fail on parsing the x-amz-restore header.
Remove the incorrect `stageNotInProgress` constant and simplify the `requestOngoing` initialization to reflect the correct default.
Closes#1813
We use a specific `versionId` format(`ulid` package) to generate versionIds in posix, which is not compatible to S3. The versionId validation was performed in frontend which is a potential source of failure for s3 proxy configured on an s3 service which doesn't use ulid for versionId generation(e.g. aws S3). These changes move the specific `ulid` versionId validation to posix to not force any specific versionId format in the gateway.
A concurrent PutObject and DeleteObject on the same prefix directory
can race:
PutObject opens an O_TMPFILE in MetaTmpDir (not yet visible in the fs)
DeleteObject removes the last visible object in the prefix directory
and calls removeParents(), which rmdir's the now-empty prefix
directory
PutObject's link() tries to link the fd into a parent directory that
no longer exists
Fix by detecting ENOENT in the final link step (Linkat, Rename, and
MoveFile) and retrying after recreating the parent directory.
Also extract linkatOTmpfile() to consolidate the Linkat+EEXIST→Renameat
logic that was previously inline in link().
Fixes#1988
Closes#1897
Extract the `X-Amz-Source-Expected-Bucket-Owner` header for CopyObject and UploadPartCopy. Verify the source bucket owner in the backend and if the provided access key id doesn't match, return an `AccessDenied` error.
Fixes#1896
Enforces the S3 `5 GiB` copy source size limit across the posix and azure
backends for `CopyObject` and `UploadPartCopy`, returning `InvalidRequest` when
the source object exceeds the threshold.
The limit is now configurable via `--copy-object-threshold`
(`VGW_COPY_OBJECT_THRESHOLD`, default 5 GiB).
A new `--mp-max-parts flag` (`VGW_MP_MAX_PARTS`, default `10000`) has been added to make multipart upload parts number limit configurable.
No integration test has been added, as GitHub Actions cannot reliably
handle large objects.
When two requests raced to complete the same multipart upload, the first
caller to finish would remove the part files and upload directory. The
second caller, already past the initial existence check, would then fail
mid-flight with confusing errors such as ErrInvalidPart or an I/O error
when trying to open a part that no longer exists.
Fix this by atomically renaming the upload directory from <uploadID> to
<uploadID>.inprogress at the very start of CompleteMultipartUploadWithCopy,
before any part data is read. A concurrent caller will now find the
original directory absent and receive a clean NoSuchUpload error. A
deferred rename restores the original name if the complete does not
succeed, allowing the client to retry.
ListMultipartUploads is updated to skip any directories whose name ends
in .inprogress so in-flight completes do not appear as pending uploads.
The scoutfs backend uses the move blocks ioctl when combining parts into the
final multipart upload object. Once a move blocks from any part is successful,
the original data is no longer in the part file. If the multipart upload fails
and retries, future complete multipart upload calls will not have the correct
data within the part files anymore.
To prevent this case, once a move blocks call is successful for an upload, any
future failure for the complete upload is set to auto-abort the upload to force
clients to re-upload the part data again.
When a client provides an invalid or incomplete body for a single-part
upload, the handler returns before the link stage. Factor tempfile
removal into cleanup() to catch all cases.
The standard io.Copy() will attempt to use copy_file_range when available.
However, this can cause problems with certain filesystems. Add an option
that will prevent io.Copy() from being able to use copy_file_range to force
a standard data copy between file descriptors when needed for completing
multipart uploads.
Rewrite the posix Walk implementation to avoid the extra ReadDir per
directory that was noted as a TODO in the old code. The new algorithm
holds all traversal state in a walkState struct and uses processDir to
interleave sibling entries in correct S3 lexicographic order without a
second syscall.
Key changes:
prefix optimisation: jump directly into the deepest matching directory
rather than scanning from the root on every call
marker short-circuit: skip entire subtrees that are lexically before
the marker, making paginated listing faster
Co-authored-by: Ben McClelland <ben.mcclelland@versity.com>
This is a fixup of the codebase using:
go run golang.org/x/tools/go/analysis/passes/modernize/cmd/modernize@latest -fix ./...
This has no bahvior changes, and only updates safe changes for
modern go features.
When access/secret are not provided, let AWS SDK v2 resolve credentials
from the default provider chain (env vars, IRSA, ECS/EC2 roles, etc.)
instead of forcing anonymous credentials.
Add an explicit anonymous credentials option for s3 proxy to force
backend anonymous access.
Fixes#1955
Fixes#1909
Previously, the mapping between object metadata and posix object was as follows: for each metadata key, we stored a separate xattr with the `user.X-Amz-Meta.<key>` prefix. This resulted in syscall overhead when storing and deleting large numbers of metadata keys.
In addition, very long metadata keys caused failures because most posix filesystems limit xattr key lengths to 127–255 bytes, while S3 does not enforce such a per-key limit.
The logic has now been changed so that all object metadata is stored in a single xattr, `user.metadata`, as a JSON key/value object. For backward compatibility, metadata GET operations still fall back to the old mechanism (`metadata key -> xattr key`) when `user.metadata` is not present.
A new CLI utility has been added to convert all legacy object metadata to the new metadata format within the provided directory.
**Example usage:**
```
versitygw utils convert-xattr-metadata path/to/bucket
```
or
```
versitygw utils cxm path/to/bucket
```
It is recommended to run this command on bucket directories to convert all legacy metadata for every object in the bucket.
Sebastien Tardif
niksis02
Ben McClellandandGitHub
Anael ORLINSKI
Jakob van SantenandGitHub
tonyipmandBen McClelland
57_WolveandGitHub