When versitygw serves files that were placed on the filesystem outside
of the gateway, those files have no stored etag attribute and are
served with an empty ETag. This causes problems for S3 clients that
rely on ETags being non empty.
The new --default-etag flag (VGW_DEFAULT_ETAG) lets operators specify
a fallback ETag value returned for any object that lacks a stored etag
attribute, making pre-existing filesystem data behave more predictably
under S3 workloads without requiring a metadata migration.
quote part etags across all azure multipart surfaces (uploadpart,
listparts, uploadpartcopy) via a shared quoteETag helper, matching the
s3 contract and the posix backend's GenerateEtag convention. populate
the previously-empty UploadPartCopy CopyPartResult with the quoted etag
and last-modified. normalize the client-supplied etag in
CompleteMultipartUpload with backend.TrimEtag so both quoted and raw
etags are accepted. add an azure-full-flow integration test asserting
part etags are quoted and consistent across uploadpart, listparts and
completemultipartupload.
When uploads hit ENOSPC or EDQUOT, the server was returning the
correct S3 error but could close the connection while unread
request bytes were still in flight, which caused TCP resets and
surfaced as broken pipe/connection reset errors in SDKs instead
of a clean Insufficient Storage response. This change drains the
remaining upload body before returning the error so the response
can be delivered and the connection can close gracefully,
preserving correct client-visible behavior under disk-full and
quota-exceeded conditions.
Fixes#2209
Expose S3 option injection in embed gateway config so
integrators can attach request middleware and other
server behaviors without needing to re-implement
RunVersityGW().
This change adds Windows functional test execution in CI and updates
backend handling so windows filesystem error/path semantics map correctly
to expected S3 outcomes.
The only meta supported on windows right now is sidecar, so the tests
in windows mode also skip sidecar skips.
Future work is to address the skips and/or more clearly document
the unsupported/incompatible behavior on windows.
The windows support will still remain best effort, but these tests
should at least flag when future changes introduce incompatible
behavior on windows.
Fixes#2187
Enforce maximum default retention periods when parsing PutObjectLockConfiguration requests. Reject Days values greater than 36500 and Years values greater than 100 with InvalidArgument errors.
Fixes#2185
Add asterisk handling for `If-Match` and `If-None-Match` read preconditions across `GetObject`, `HeadObject`, `CopyObject`, and `UploadPartCopy`.
`If-Match`: `*` now matches any `ETag`, while `If-None-Match`: `*` returns `304 Not Modified`.
Fixes#2179
Treat a present but empty `Content-MD5` header as an invalid digest instead of handling it as if the header were absent. This makes PUT operations return `InvalidDigest` for empty `Content-MD5` values while preserving existing behavior for missing headers.
Fixes#2180Fixes#2181
Migrate the gateway from Fiber v2 to Fiber v3.3.0 and update the affected server, middleware, handler, controller, and test code for the new APIs.
Replace the deprecated Fiber filesystem middleware used by the WebUI with the Fiber v3 static middleware, serving the embedded WebUI assets from an fs.Sub filesystem.
Fix the request header limit handling regression by adding a temporary handler for Fiber v3/fasthttp small-buffer errors so oversized request headers return the expected regulated S3 error response.
Fix the debuglogger panic by reworking the boxed key/value formatter used for debug request and response dumps. The formatter now handles long header keys and values without producing invalid wrap widths, negative padding, or out-of-range string slices.
Integrate x-amz-website-redirect-location across object metadata flows so uploads, copies, multipart creation, HEAD, and GET preserve and return redirect locations, and website hosting applies object-level redirects from the stored value.
Enhances the static website hosting implementation with more complete S3-compatible behavior across request handling, backend storage, validation, CORS, and errors.
Adds dedicated website endpoint handling for GET, HEAD, and OPTIONS requests, including index document resolution, error document serving, redirect-all support, pre-fetch and post-error routing rules, query string preservation in redirects, public access checks before object reads, and method-not-allowed responses.
Improves error handling for website responses by returning S3-compatible HTML error bodies with request IDs, host IDs, x-amz-error-code, x-amz-error-message, and specialized error fields. This also fixes website-related validation errors to return more accurate S3-style error codes and messages, including invalid redirect protocols, invalid HTTP redirect/error codes, conflicting routing rule replacements, routing rule limits, and oversized website configuration requests.
Adds website CORS support for GET, HEAD, and OPTIONS preflight requests, including bucket CORS lookup through website host bucket resolution, allowed origin/method/header validation, exposed header handling, ETag exposure, Vary headers, max-age handling, and CORS access-denied responses.
Adds debug logging around website configuration parsing, validation failures, CORS checks, backend lookup failures, and internal website error paths to make failures easier to diagnose.
Adds compressed website configuration storage so larger configs fit backend metadata limits, including gzip storage for POSIX extended attributes and base64-encoded compressed metadata for Azure. Also adds Azure PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite support.
Adds and expands test coverage for website config validation, S3-compatible HTML error bodies, website routing behavior, public access enforcement, HEAD behavior, CORS handling, PutBucketWebsite limits, and end-to-end website hosting through a Docker-based dnsmasq test setup and CI workflow.
Replace PutBucketWebsite, GetBucketWebsite, DeleteBucketWebsite
NotImplemented test stubs with comprehensive integration tests covering:
- non-existing bucket errors
- validation (empty suffix, suffix with slash, invalid protocol, mutual
exclusion of RedirectAllRequestsTo and IndexDocument)
- successful put/get round-trips for both index+error and redirect-all configs
- delete idempotency and verification
Signed-off-by: Marc Singer <marc@singer.gg>
Add error document serving, routing rules, and integration tests
Implement Features 1 and 2 of S3 static website hosting:
- WebsiteErrorDocument controller wrapper intercepts 4xx errors on
website-enabled buckets and serves the configured error document or
evaluates post-request routing rules (error code match redirects)
- ResolveWebsiteIndex middleware now caches parsed WebsiteConfiguration
in context, handles RedirectAllRequestsTo, evaluates pre-request
routing rules (key prefix match redirects), and rewrites directory
keys for index document
- MatchPreRequestRule and MatchPostRequestRule methods on
WebsiteConfiguration for routing rule evaluation
- 14 unit tests for routing rule matching
- 7 integration tests covering error document, routing rules,
redirect-all, and index document behavior
Signed-off-by: Marc Singer <marc@singer.gg>
Add separate website hosting endpoint with virtual-host routing
Signed-off-by: Marc Singer <marc@singer.gg>
Support catch-all mode for website endpoint when --website-domain is omitted
Signed-off-by: Marc Singer <marc@singer.gg>
Add S3 bucket website configuration types with XML serialization support
in s3response/website.go. Includes IndexDocument, ErrorDocument,
RedirectAllRequestsTo, and RoutingRules with full validation matching
AWS S3 behavior.
Add corresponding S3 error codes: ErrNoSuchWebsiteConfiguration,
ErrInvalidWebsiteConfiguration, ErrInvalidWebsiteSuffix, and
ErrInvalidWebsiteRedirectCode.
Unit tests cover validation logic, XML round-trip, and parsing.
Signed-off-by: Marc Singer <marc@singer.gg>
Add website backend interface and implementations for posix and s3proxy
Add PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite methods
to the Backend interface with BackendUnsupported stubs that return
ErrNotImplemented.
Posix backend stores website config as a metadata attribute (key:
'website') following the same pattern as CORS. ScoutFS inherits via
embedding.
S3Proxy backend stores website config in the metadata bucket with
prefix 'vgw-meta-website-', consistent with existing ACL/policy/CORS
metadata storage. Returns ErrNoSuchWebsiteConfiguration when not found.
Signed-off-by: Marc Singer <marc@singer.gg>
Add website API controllers and wire into router
Add PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite
controller methods following the same pattern as CORS. Controllers
parse and validate WebsiteConfiguration XML, check IAM authorization,
and delegate to the backend.
Replace the three HandleErrorRoute(ErrNotImplemented) stubs in the
router with the new controller methods. Regenerate the backend mock
to include the new interface methods.
Signed-off-by: Marc Singer <marc@singer.gg>
Add website index document middleware and wire into router
Add ResolveWebsiteIndex middleware that rewrites directory-like object keys
(empty or ending with /) to include the IndexDocument suffix when website
hosting is enabled. Also handles RedirectAllRequestsTo by returning 301.
Wire the middleware into both GetObject and HeadObject handler chains in
the router, positioned after BucketObjectNameValidator and before auth.
Signed-off-by: Marc Singer <marc@singer.gg>
Add nil and length checks before accessing
resp.OwnershipControls.Rules[0]. If the upstream S3 backend returns
a response with nil OwnershipControls or empty Rules, the server
panics with an index-out-of-range error. Return
OwnershipControlsNotFoundError instead, matching the behavior of
the posix and azure backends.
This bug was introduced in commit 7545e623 (2024-06-28) when bucket
ownership controls were first implemented for the s3proxy backend.
Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
Ben McClelland
Ed Espino
Luke McCrone
dependabot[bot]
Mart-Kuc
niksis02
Peter Dahlberg
Marc Singer
Sebastien Tardif