STILL NOT FIXED:
in backend/meta/sidecar.go CommitMetadata()
Between pathIno(dataPath) != myIno returning false (check passes) and os.Rename executing, another goroutine can win link() and install a new inode at dataPath. This goroutine then proceeds to rename its stale attributes over the winner's freshly committed ones.
The comment's safety argument — "the true winner's CommitMetadata will overwrite any partially committed attributes before it finishes" — only holds if the winner runs entirely after this goroutine's renames. If the winner already finished its renames before this goroutine's stale os.Rename executes, the loser's data corrupts the final metadata silently.
This is a fundamental TOCTOU problem. There is no POSIX syscall for "rename only if this inode still owns the destination", so it can't be fixed with filesystem operations alone. The real fix would require serializing CommitMetadata calls per object
----
Previously, StoreAttribute wrote metadata (checksums, etc.) directly to
the final per-object sidecar path during upload. Concurrent uploads of
the same object would race to write into the same directory, causing one
upload's metadata to be silently overwritten by another's data file, or
vice-versa. On Linux, O_TMPFILE fd-number reuse made this worse: after
link() closed the fd, the inode-number slot could be immediately reused
by a different goroutine.
Fix by staging sidecar metadata in a per-upload temporary directory
named .sgwtmp.<pid>.<inode> instead of the final path, then atomically
committing it after the data file has been linked.
tmpSidecarID() identifies an in-flight upload by PID + inode (Unix) or
PID + temp filename (Windows), matching the token produced by
tmpfile.SidecarToken() in the posix backend so staging and commit find
the same directory.
StoreAttribute now writes attributes into the inode-keyed temp sidecar
dir with a retry loop (up to 5 attempts) to handle the inode-reuse edge
case where a concurrent CommitMetadata RemoveAll races with a fresh
MkdirAll on the same directory name.
CommitMetadata moves the staged attributes to the final object path
one-by-one via atomic rename. Before each rename it re-verifies that the
data file's inode still matches the token; if not, a later concurrent
upload won the race and this goroutine aborts cleanly, leaving the
winner's metadata intact.
Integrate x-amz-website-redirect-location across object metadata flows so uploads, copies, multipart creation, HEAD, and GET preserve and return redirect locations, and website hosting applies object-level redirects from the stored value.
Enhances the static website hosting implementation with more complete S3-compatible behavior across request handling, backend storage, validation, CORS, and errors.
Adds dedicated website endpoint handling for GET, HEAD, and OPTIONS requests, including index document resolution, error document serving, redirect-all support, pre-fetch and post-error routing rules, query string preservation in redirects, public access checks before object reads, and method-not-allowed responses.
Improves error handling for website responses by returning S3-compatible HTML error bodies with request IDs, host IDs, x-amz-error-code, x-amz-error-message, and specialized error fields. This also fixes website-related validation errors to return more accurate S3-style error codes and messages, including invalid redirect protocols, invalid HTTP redirect/error codes, conflicting routing rule replacements, routing rule limits, and oversized website configuration requests.
Adds website CORS support for GET, HEAD, and OPTIONS preflight requests, including bucket CORS lookup through website host bucket resolution, allowed origin/method/header validation, exposed header handling, ETag exposure, Vary headers, max-age handling, and CORS access-denied responses.
Adds debug logging around website configuration parsing, validation failures, CORS checks, backend lookup failures, and internal website error paths to make failures easier to diagnose.
Adds compressed website configuration storage so larger configs fit backend metadata limits, including gzip storage for POSIX extended attributes and base64-encoded compressed metadata for Azure. Also adds Azure PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite support.
Adds and expands test coverage for website config validation, S3-compatible HTML error bodies, website routing behavior, public access enforcement, HEAD behavior, CORS handling, PutBucketWebsite limits, and end-to-end website hosting through a Docker-based dnsmasq test setup and CI workflow.
Replace PutBucketWebsite, GetBucketWebsite, DeleteBucketWebsite
NotImplemented test stubs with comprehensive integration tests covering:
- non-existing bucket errors
- validation (empty suffix, suffix with slash, invalid protocol, mutual
exclusion of RedirectAllRequestsTo and IndexDocument)
- successful put/get round-trips for both index+error and redirect-all configs
- delete idempotency and verification
Signed-off-by: Marc Singer <marc@singer.gg>
Add error document serving, routing rules, and integration tests
Implement Features 1 and 2 of S3 static website hosting:
- WebsiteErrorDocument controller wrapper intercepts 4xx errors on
website-enabled buckets and serves the configured error document or
evaluates post-request routing rules (error code match redirects)
- ResolveWebsiteIndex middleware now caches parsed WebsiteConfiguration
in context, handles RedirectAllRequestsTo, evaluates pre-request
routing rules (key prefix match redirects), and rewrites directory
keys for index document
- MatchPreRequestRule and MatchPostRequestRule methods on
WebsiteConfiguration for routing rule evaluation
- 14 unit tests for routing rule matching
- 7 integration tests covering error document, routing rules,
redirect-all, and index document behavior
Signed-off-by: Marc Singer <marc@singer.gg>
Add separate website hosting endpoint with virtual-host routing
Signed-off-by: Marc Singer <marc@singer.gg>
Support catch-all mode for website endpoint when --website-domain is omitted
Signed-off-by: Marc Singer <marc@singer.gg>
Validate required signed headers for both Authorization-header SigV4 requests and presigned URLs. The required signed header set is now `host` plus every incoming header with the `x-amz-` prefix.
During request reconstruction, signed headers and explicitly ignored headers are copied into the generated request used for signature verification. If an incoming `x-amz-*` header is present but missing from the client-provided `SignedHeaders`, return `AccessDenied` with a `HeadersNotSigned` field. The `host` header remains part of the canonical request and signed header calculation.
Previously, a client could sign a request without an S3 control header and then add that header after signing. For example, a presigned `PUT` URL could be generated with only `host` signed, then the actual request could include an unsigned `X-Amz-Tagging` or `X-Amz-Copy-Source` header. Because the verifier reconstructed the request only from `SignedHeaders`, that extra header was omitted from signature calculation and could pass authentication even though it changed the request semantics. This is now rejected with `AccessDenied`.
Expose v4 helper methods for checking required and ignored headers, and update canonical header signing so ignored headers can still be included when a client explicitly lists them in `SignedHeaders`, while `Authorization` remains excluded from signature calculation.
Object key validation allowed internal parent-directory segments such as `public/../private.txt`. Bucket policy and auth checks evaluated the raw key, so a policy allowing bucket/public/* could match the request while posix backend later resolved the key with `filepath.Join` and accessed `bucket/private.txt`.
Add backend-specific object key normalization to close that mismatch. The Backend interface now exposes `NormalizeObjectKey` so authorization can evaluate resources using the same key shape a backend will use for storage access.
Backends that do not collapse object paths, including Azure and the S3 proxy, inherit `BackendUnsupported.NormalizeObjectKey`. That implementation returns the input key unchanged, avoiding unnecessary normalization and keeping policy evaluation unpolluted for object stores where ../ is part of the key name.
posix/scoutfs normalize keys with filepath.Join so policy resources and request keys are compared after internal dot segments are collapsed.
Bucket policy evaluation now normalizes both the incoming object key and object resource patterns from the policy before matching. Object lock governance bypass policy checks use the same backend normalizer as well, so retention and legal hold authorization cannot diverge from backend path resolution.
Fix the gateway panic when validating malformed bucket ownership controls bodies with no rules. The handler now checks the rules count before indexing the first rule.
Validate multipart PostObject key fields with the existing object name rules so path traversal and degenerate names return BadRequest. This prevents crafted object keys from escaping the gateway root.
Distinguish public bucket policy no-match from explicit deny during anonymous access checks. This preserves ACL fallback only for requests that are not allowed by policy, while ensuring a matching Deny statement short-circuits authorization and returns AccessDenied even when a public ACL would otherwise grant access.
Fixes#2123Fixes#2120Fixes#2116Fixes#2111Fixes#2108Fixes#2086Fixes#2085Fixes#2083Fixes#2081Fixes#2080Fixes#2073Fixes#2072Fixes#2071Fixes#2069Fixes#2044Fixes#2043Fixes#2042Fixes#2041Fixes#2040Fixes#2039Fixes#2036Fixes#2035Fixes#2034Fixes#2028Fixes#2020Fixes#1842Fixes#1810Fixes#1780Fixes#1775Fixes#1736Fixes#1705Fixes#1663Fixes#1645Fixes#1583Fixes#1526Fixes#1514Fixes#1493Fixes#1487Fixes#959Fixes#779Closes#823Closes#85
Refactor global S3 error handling around structured error types and centralized XML response generation.
All S3 errors now share the common APIError base for the fields every error has: Code, HTTP status code, and Message. Non-traditional errors that need AWS-compatible XML fields now have dedicated typed errors in the s3err package. Each typed error implements the shared S3Error behavior so controllers and middleware can handle errors consistently while still emitting error-specific XML fields.
Add a dedicated InvalidArgumentError type because InvalidArgument is used widely across request validation, auth, copy source handling, object lock validation, multipart validation, and header parsing. The new InvalidArgument path uses explicit InvalidArgErrorCode constants with predefined descriptions and ArgumentName values, keeping call sites readable while preserving the correct InvalidArgument XML shape and optional ArgumentValue.
New structured errors added in s3err:
- `AccessForbiddenError`: Method, ResourceType
- `BadDigestError`: CalculatedDigest, ExpectedDigest
- `BucketError`: BucketName
- `ContentSHA256MismatchError`: ClientComputedContentSHA256, S3ComputedContentSHA256
- `EntityTooLargeError`: ProposedSize, MaxSizeAllowed
- `EntityTooSmallError`: ProposedSize, MinSizeAllowed
- `ExpiredPresignedURLError`: ServerTime, XAmzExpires, Expires
- `InvalidAccessKeyIdError`: AWSAccessKeyId
- `InvalidArgumentError`: Description, ArgumentName, ArgumentValue
- `InvalidChunkSizeError`: Chunk, BadChunkSize
- `InvalidDigestError`: ContentMD5
- `InvalidLocationConstraintError`: LocationConstraint
- `InvalidPartError`: UploadId, PartNumber, ETag
- `InvalidRangeError`: RangeRequested, ActualObjectSize
- `InvalidTagError`: TagKey, TagValue
- `KeyTooLongError`: Size, MaxSizeAllowed
- `MetadataTooLargeError`: Size, MaxSizeAllowed
- `MethodNotAllowedError`: Method, ResourceType, AllowedMethods
- `NoSuchUploadError`: UploadId
- `NoSuchVersionError`: Key, VersionId
- `NotImplementedError`: Header, AdditionalMessage
- `PreconditionFailedError`: Condition
- `RequestTimeTooSkewedError`: RequestTime, ServerTime, MaxAllowedSkewMilliseconds
- `SignatureDoesNotMatchError`: AWSAccessKeyId, StringToSign, SignatureProvided, StringToSignBytes, CanonicalRequest, CanonicalRequestBytes
Fix CompleteMultipartUpload validation in the Azure backend so missing or empty `ETag` values return the appropriate S3 error instead of allowing a gateway panic.
Fix presigned authentication expiration validation to compare server time in `UTC`, matching the `UTC` timestamp used by presigned URL signing.
Add request ID and host ID support across S3 requests. Each request now receives AWS S3-like identifiers, returned in response headers as `x-amz-request-id` and `x-amz-id-2` and included in all XML error responses as RequestId and HostId. The generated ID structure is designed to resemble AWS S3 request IDs and host IDs.
The request signature calculation/validation for streaming uploads was previously delayed until the request body was fully read, both for Authorization header authentication and presigned URLs.
Now, the signature is validated immediately in the authorization middlewares without reading the request body, since the signature calculation itself does not depend on the request body. Instead, only the `x-amz-content-sha256` SHA-256 hash calculation is delayed.
Directories that exist on the filesystem but were not explicitly created
via S3 (put-object with a key ending in '/') do not have an etag
value. ListObjectsV2 already uses the presence of this attribute to
decide whether to include a directory as an object. GetObject and
HeadObject were not performing this check, so they would successfully
return directories that ListObjectsV2 would not list.
Add the etag attribute check in GetObject and HeadObject: if a
directory path is requested but has no etag xattr, return 404. This
makes all three operations agree on which directories are S3 objects.
Fixes#2130
The gateway currently supports only Signature Version 4 (SigV4) authorization. Deprecated AWS SigV2 requests are now rejected with an AWS-specific `InvalidRequest` error for both Authorization-header requests and query-string requests(presigned URLs).
This also fixes SigV4 Authorization-header handling for date headers. SigV4 accepts two date headers: `Date` and `X-Amz-Date`. `X-Amz-Date` takes precedence, but when it is missing, `Date` should be used. The gateway now uses the `Date` header with lower precedence when `X-Amz-Date` is not present. No SDK integration test was added for this case because the SDK always sets `X-Amz-Date`, and this behavior is not configurable.
Integrate the new S3 checksum types in the gateway, including `SHA512`, `MD5`, `XXHASH64`, `XXHASH3`, and `XXHASH128`. This adds checksum calculation, validation, schema handling, and test coverage for the expanded checksum support.
These external packages have been used:
- `github.com/zeebo/xxh3` for `XXHASH3` and `XXHASH128`
- `github.com/cespare/xxhash/v2` for `XXHASH64`
Adjust integration tests because `aws-sdk-go-v2/service/s3` does not support automatic checksum calculation for the new checksum algorithms and returns an SDK-level error when only the checksum algorithm is provided. Only precalculated checksum values are acceptable for these checksum types.
References:
- `https://github.com/aws/aws-sdk-go-v2/issues/3404`
- `https://github.com/aws/aws-sdk-go-v2/issues/3403`
Fixes#1986
When a client includes tagging, legal hold, or retention headers in a PutObject, CopyObject or CreateMultipartUpload request, the corresponding bucket policy permissions must be verified in addition to s3:PutObject:
`X-Amz-Tagging` - `s3:PutObjectTagging`
`X-Amz-Object-Lock-Legal-Hold` - `s3:PutObjectLegalHold`
`X-Amz-Object-Lock-Mode` - `s3:PutObjectRetention`
Previously, only s3:PutObject was checked, allowing users to set tagging, legal hold, and retention without having the required permissions. Now each action permission is check, if user tries to add them.
For CopyObject these permissions are checked on destination object.
Enough people are making use of sidecar that we need to add
a CI test to make sure we have some coverage with this mode.
This add a couple small functional test fixes found wtih
enabling sidecar tests as well.
The dl.min.io download site has been having stability issues
possibly related to github action runners getting rate limited.
Disable these for now until we can find a better place to host
this client.
Fixes#2052Fixes#2056Fixes#2057
Previously, GetObject and HeadObject used the request's `Range` header to determine the response status code, which caused incorrect 206 responses for invalid Range header values.
The status is now driven by whether res.ContentRange is set in the response, rather than by the presence of a range in the request. Backends (posix and azure) now set Content-Range for PartNumber=1 on non-multipart objects, skipping zero-size objects where no range applies.
HeadObject was also fixed to return 206 when Content-Range is present, and to only return checksums when the full object is requested.
Closes#1064
Use the multipart ETag as the in-progress directory suffix instead of the static `.inprogress` marker so that concurrent CompleteMultipartUpload calls for the same upload ID are all treated as successful (idempotent) rather than racing, where only one succeeded and the rest returned NoSuchUpload.
After finalizing the multipart upload, store an `mp-metadata` xattr on the assembled object that records the upload ID and cumulative byte offsets for each part. GetObject and HeadObject now use this metadata to serve individual part ranges via the `partNumber` query parameter, returning a successful response instead of returning NotImplemented.
Add two new S3 error codes:
- `ErrInvalidPartNumberRange` (416 RequestedRangeNotSatisfiable) — returned
when the requested part number exceeds the number of parts in the upload.
- `ErrRangeAndPartNumber` (400 BadRequest) — returned when both a Range header
and a partNumber query parameter are specified on the same request.
As multipart uploads are translated to blobs in azure blob storage, they were visible in ListObjects(V2) as complete objects. Now the blobs with multipart prefix are filtered out during listing.
The listing logic is rewritten client-side to implement proper S3 semantics: flat blob enumeration with manual delimiter handling, correct truncation (IsTruncated only set when more items genuinely exist beyond maxKeys), and StartAfter/Marker/ContinuationToken applied via the lexicographic max of both constraints in ListObjectsV2.
For the same reason bucket deletion was not allowed. Now multipart objects are explicitly checked on bucket deletion and any pending multipart upload doesn't block the bucket deletion anymore.
Closes#1813
We use a specific `versionId` format(`ulid` package) to generate versionIds in posix, which is not compatible to S3. The versionId validation was performed in frontend which is a potential source of failure for s3 proxy configured on an s3 service which doesn't use ulid for versionId generation(e.g. aws S3). These changes move the specific `ulid` versionId validation to posix to not force any specific versionId format in the gateway.
A concurrent PutObject and DeleteObject on the same prefix directory
can race:
PutObject opens an O_TMPFILE in MetaTmpDir (not yet visible in the fs)
DeleteObject removes the last visible object in the prefix directory
and calls removeParents(), which rmdir's the now-empty prefix
directory
PutObject's link() tries to link the fd into a parent directory that
no longer exists
Fix by detecting ENOENT in the final link step (Linkat, Rename, and
MoveFile) and retrying after recreating the parent directory.
Also extract linkatOTmpfile() to consolidate the Linkat+EEXIST→Renameat
logic that was previously inline in link().
Fixes#1988
Fixes the [comment](https://github.com/versity/versitygw/issues/1648#issuecomment-4175425099)
Removes the unnecessary multipart/form-data boundary normalizing. The boundary prefix(`--`) was trimmed in `NewMultipartParser`, which caused incorrect boundary check for the boundaries starting with 2 dashes(e.g. `----WebKitFormBoundaryABC123`).
Closes#1897
Extract the `X-Amz-Source-Expected-Bucket-Owner` header for CopyObject and UploadPartCopy. Verify the source bucket owner in the backend and if the provided access key id doesn't match, return an `AccessDenied` error.
Ben McClelland
niksis02
Marc Singerandniksis02
Luke McCrone
alice nodelmanandGitHub