This change adds Windows functional test execution in CI and updates
backend handling so windows filesystem error/path semantics map correctly
to expected S3 outcomes.
The only meta supported on windows right now is sidecar, so the tests
in windows mode also skip sidecar skips.
Future work is to address the skips and/or more clearly document
the unsupported/incompatible behavior on windows.
The windows support will still remain best effort, but these tests
should at least flag when future changes introduce incompatible
behavior on windows.
Fixes#2187
Enforce maximum default retention periods when parsing PutObjectLockConfiguration requests. Reject Days values greater than 36500 and Years values greater than 100 with InvalidArgument errors.
Fixes#2185
Add asterisk handling for `If-Match` and `If-None-Match` read preconditions across `GetObject`, `HeadObject`, `CopyObject`, and `UploadPartCopy`.
`If-Match`: `*` now matches any `ETag`, while `If-None-Match`: `*` returns `304 Not Modified`.
Fixes#2179
Treat a present but empty `Content-MD5` header as an invalid digest instead of handling it as if the header were absent. This makes PUT operations return `InvalidDigest` for empty `Content-MD5` values while preserving existing behavior for missing headers.
Fixes#2180Fixes#2181
Migrate the gateway from Fiber v2 to Fiber v3.3.0 and update the affected server, middleware, handler, controller, and test code for the new APIs.
Replace the deprecated Fiber filesystem middleware used by the WebUI with the Fiber v3 static middleware, serving the embedded WebUI assets from an fs.Sub filesystem.
Fix the request header limit handling regression by adding a temporary handler for Fiber v3/fasthttp small-buffer errors so oversized request headers return the expected regulated S3 error response.
Fix the debuglogger panic by reworking the boxed key/value formatter used for debug request and response dumps. The formatter now handles long header keys and values without producing invalid wrap widths, negative padding, or out-of-range string slices.
Integrate x-amz-website-redirect-location across object metadata flows so uploads, copies, multipart creation, HEAD, and GET preserve and return redirect locations, and website hosting applies object-level redirects from the stored value.
Enhances the static website hosting implementation with more complete S3-compatible behavior across request handling, backend storage, validation, CORS, and errors.
Adds dedicated website endpoint handling for GET, HEAD, and OPTIONS requests, including index document resolution, error document serving, redirect-all support, pre-fetch and post-error routing rules, query string preservation in redirects, public access checks before object reads, and method-not-allowed responses.
Improves error handling for website responses by returning S3-compatible HTML error bodies with request IDs, host IDs, x-amz-error-code, x-amz-error-message, and specialized error fields. This also fixes website-related validation errors to return more accurate S3-style error codes and messages, including invalid redirect protocols, invalid HTTP redirect/error codes, conflicting routing rule replacements, routing rule limits, and oversized website configuration requests.
Adds website CORS support for GET, HEAD, and OPTIONS preflight requests, including bucket CORS lookup through website host bucket resolution, allowed origin/method/header validation, exposed header handling, ETag exposure, Vary headers, max-age handling, and CORS access-denied responses.
Adds debug logging around website configuration parsing, validation failures, CORS checks, backend lookup failures, and internal website error paths to make failures easier to diagnose.
Adds compressed website configuration storage so larger configs fit backend metadata limits, including gzip storage for POSIX extended attributes and base64-encoded compressed metadata for Azure. Also adds Azure PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite support.
Adds and expands test coverage for website config validation, S3-compatible HTML error bodies, website routing behavior, public access enforcement, HEAD behavior, CORS handling, PutBucketWebsite limits, and end-to-end website hosting through a Docker-based dnsmasq test setup and CI workflow.
Replace PutBucketWebsite, GetBucketWebsite, DeleteBucketWebsite
NotImplemented test stubs with comprehensive integration tests covering:
- non-existing bucket errors
- validation (empty suffix, suffix with slash, invalid protocol, mutual
exclusion of RedirectAllRequestsTo and IndexDocument)
- successful put/get round-trips for both index+error and redirect-all configs
- delete idempotency and verification
Signed-off-by: Marc Singer <marc@singer.gg>
Add error document serving, routing rules, and integration tests
Implement Features 1 and 2 of S3 static website hosting:
- WebsiteErrorDocument controller wrapper intercepts 4xx errors on
website-enabled buckets and serves the configured error document or
evaluates post-request routing rules (error code match redirects)
- ResolveWebsiteIndex middleware now caches parsed WebsiteConfiguration
in context, handles RedirectAllRequestsTo, evaluates pre-request
routing rules (key prefix match redirects), and rewrites directory
keys for index document
- MatchPreRequestRule and MatchPostRequestRule methods on
WebsiteConfiguration for routing rule evaluation
- 14 unit tests for routing rule matching
- 7 integration tests covering error document, routing rules,
redirect-all, and index document behavior
Signed-off-by: Marc Singer <marc@singer.gg>
Add separate website hosting endpoint with virtual-host routing
Signed-off-by: Marc Singer <marc@singer.gg>
Support catch-all mode for website endpoint when --website-domain is omitted
Signed-off-by: Marc Singer <marc@singer.gg>
Add S3 bucket website configuration types with XML serialization support
in s3response/website.go. Includes IndexDocument, ErrorDocument,
RedirectAllRequestsTo, and RoutingRules with full validation matching
AWS S3 behavior.
Add corresponding S3 error codes: ErrNoSuchWebsiteConfiguration,
ErrInvalidWebsiteConfiguration, ErrInvalidWebsiteSuffix, and
ErrInvalidWebsiteRedirectCode.
Unit tests cover validation logic, XML round-trip, and parsing.
Signed-off-by: Marc Singer <marc@singer.gg>
Add website backend interface and implementations for posix and s3proxy
Add PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite methods
to the Backend interface with BackendUnsupported stubs that return
ErrNotImplemented.
Posix backend stores website config as a metadata attribute (key:
'website') following the same pattern as CORS. ScoutFS inherits via
embedding.
S3Proxy backend stores website config in the metadata bucket with
prefix 'vgw-meta-website-', consistent with existing ACL/policy/CORS
metadata storage. Returns ErrNoSuchWebsiteConfiguration when not found.
Signed-off-by: Marc Singer <marc@singer.gg>
Add website API controllers and wire into router
Add PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite
controller methods following the same pattern as CORS. Controllers
parse and validate WebsiteConfiguration XML, check IAM authorization,
and delegate to the backend.
Replace the three HandleErrorRoute(ErrNotImplemented) stubs in the
router with the new controller methods. Regenerate the backend mock
to include the new interface methods.
Signed-off-by: Marc Singer <marc@singer.gg>
Add website index document middleware and wire into router
Add ResolveWebsiteIndex middleware that rewrites directory-like object keys
(empty or ending with /) to include the IndexDocument suffix when website
hosting is enabled. Also handles RedirectAllRequestsTo by returning 301.
Wire the middleware into both GetObject and HeadObject handler chains in
the router, positioned after BucketObjectNameValidator and before auth.
Signed-off-by: Marc Singer <marc@singer.gg>
Add nil and length checks before accessing
resp.OwnershipControls.Rules[0]. If the upstream S3 backend returns
a response with nil OwnershipControls or empty Rules, the server
panics with an index-out-of-range error. Return
OwnershipControlsNotFoundError instead, matching the behavior of
the posix and azure backends.
This bug was introduced in commit 7545e623 (2024-06-28) when bucket
ownership controls were first implemented for the s3proxy backend.
Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
Add missing return statements after error checks in sendLog. When
json.Marshal or http.NewRequest fails, the error is logged but
execution continues. If http.NewRequest returns a nil *Request,
the subsequent req.Header.Set call panics with a nil pointer
dereference.
This bug was introduced in PR #129 (2023-07-14) and has been
present for nearly 3 years.
Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
ParseCopySource indexes into copySourceHeader[0] without checking
for an empty string, causing an index-out-of-range panic.
The three backend callers (azure, posix) pass the x-amz-copy-source
header value directly, so a malformed or missing header propagates
an empty string into ParseCopySource.
Add an empty-string guard at the top of the function that returns
an InvalidArgCopySourceBucket error, consistent with the existing
error returned when the source path has no bucket/object separator.
Add a table-driven test case that reproduces the panic without the
fix and verifies the correct error with the fix.
Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
WithOnListen registers a callback invoked once the server is bound and
ready to accept connections. It wraps fiber's existing OnListen hook,
which fires immediately before the first connection is served.
This allows callers to detect readiness deterministically rather than
relying on a fixed sleep or polling a health endpoint.
Example use in tests:
ready := make(chan struct{})
srv, _ := s3api.New(..., s3api.WithOnListen(func() { close(ready) }))
go srv.ServeMultiPort([]string{addr})
<-ready // blocks until the server is accepting connections
When the connection terminates before all bytes read, we were
getting an io.ErrUnexpectedEOF that was not being handled as
a standard io.EOF resulting in an internal error being raised.
Translate io.ErrUnexpectedEOF to io.EOF so that we return the
normal errors for unexpected content. Add a log message so
that its clear the error is due to the connection being
terminated before all data sent and not the fault of the
gateway.
Validate required signed headers for both Authorization-header SigV4 requests and presigned URLs. The required signed header set is now `host` plus every incoming header with the `x-amz-` prefix.
During request reconstruction, signed headers and explicitly ignored headers are copied into the generated request used for signature verification. If an incoming `x-amz-*` header is present but missing from the client-provided `SignedHeaders`, return `AccessDenied` with a `HeadersNotSigned` field. The `host` header remains part of the canonical request and signed header calculation.
Previously, a client could sign a request without an S3 control header and then add that header after signing. For example, a presigned `PUT` URL could be generated with only `host` signed, then the actual request could include an unsigned `X-Amz-Tagging` or `X-Amz-Copy-Source` header. Because the verifier reconstructed the request only from `SignedHeaders`, that extra header was omitted from signature calculation and could pass authentication even though it changed the request semantics. This is now rejected with `AccessDenied`.
Expose v4 helper methods for checking required and ignored headers, and update canonical header signing so ignored headers can still be included when a client explicitly lists them in `SignedHeaders`, while `Authorization` remains excluded from signature calculation.
Browsers throw an opaque TypeError for all network-level failures — CORS
policy violations, TLS/certificate rejections (e.g. self-signed certs), DNS
failures, and unreachable hosts — with no way to distinguish between them.
Asserting "CORS blocked" on every TypeError caused users to chase a CORS
misconfiguration when the real problem was an untrusted certificate or
unreachable gateway. The error message now lists some plausible causes so users
have possible diagnostics regardless of the actual failure mode.
Fixes#2143
The object info modal in the WebUI was always displaying STANDARD as the
storage class regardless of the actual value. The root cause is a browser
CORS restriction: when the WebUI makes a cross-origin HEAD request to the
S3 endpoint, the browser silently drops any response header not listed in
Access-Control-Expose-Headers, causing response.headers.get('x-amz-storage-class')
to return null and the UI to fall back to the hardcoded STANDARD default.
Adding x-amz-storage-class to the default set of exposed headers ensures
the browser makes it available to JavaScript, allowing storage classes such
as GLACIER to be correctly reflected in the UI.
dependabot[bot]
Ben McClelland
Mart-Kuc
Luke McCrone
niksis02
Peter Dahlberg
Marc Singer
Sebastien Tardif
Yevgeniy Miretskiy