Commit Graph

503 Commits

Author SHA1 Message Date
niksis02 12fb5a6594 fix: reject empty Content-MD5 on PUT operations
Fixes #2179

Treat a present but empty `Content-MD5` header as an invalid digest instead of handling it as if the header were absent. This makes PUT operations return `InvalidDigest` for empty `Content-MD5` values while preserving existing behavior for missing headers.
2026-06-15 18:53:42 +04:00
niksis02 4d391cabc8 feat: migrate Fiber to v3.3.0
Fixes #2180
Fixes #2181

Migrate the gateway from Fiber v2 to Fiber v3.3.0 and update the affected server, middleware, handler, controller, and test code for the new APIs.

Replace the deprecated Fiber filesystem middleware used by the WebUI with the Fiber v3 static middleware, serving the embedded WebUI assets from an fs.Sub filesystem.

Fix the request header limit handling regression by adding a temporary handler for Fiber v3/fasthttp small-buffer errors so oversized request headers return the expected regulated S3 error response.

Fix the debuglogger panic by reworking the boxed key/value formatter used for debug request and response dumps. The formatter now handles long header keys and values without producing invalid wrap widths, negative padding, or out-of-range string slices.
2026-06-15 14:48:31 +04:00
niksis02 f08f76fea4 feat: support x-amz-website-redirect-location
Integrate x-amz-website-redirect-location across object metadata flows so uploads, copies, multipart creation, HEAD, and GET preserve and return redirect locations, and website hosting applies object-level redirects from the stored value.
2026-06-10 12:41:55 +04:00
niksis02 1625c5963e feat: improve static website hosting support
Enhances the static website hosting implementation with more complete S3-compatible behavior across request handling, backend storage, validation, CORS, and errors.

Adds dedicated website endpoint handling for GET, HEAD, and OPTIONS requests, including index document resolution, error document serving, redirect-all support, pre-fetch and post-error routing rules, query string preservation in redirects, public access checks before object reads, and method-not-allowed responses.

Improves error handling for website responses by returning S3-compatible HTML error bodies with request IDs, host IDs, x-amz-error-code, x-amz-error-message, and specialized error fields. This also fixes website-related validation errors to return more accurate S3-style error codes and messages, including invalid redirect protocols, invalid HTTP redirect/error codes, conflicting routing rule replacements, routing rule limits, and oversized website configuration requests.

Adds website CORS support for GET, HEAD, and OPTIONS preflight requests, including bucket CORS lookup through website host bucket resolution, allowed origin/method/header validation, exposed header handling, ETag exposure, Vary headers, max-age handling, and CORS access-denied responses.

Adds debug logging around website configuration parsing, validation failures, CORS checks, backend lookup failures, and internal website error paths to make failures easier to diagnose.

Adds compressed website configuration storage so larger configs fit backend metadata limits, including gzip storage for POSIX extended attributes and base64-encoded compressed metadata for Azure. Also adds Azure PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite support.

Adds and expands test coverage for website config validation, S3-compatible HTML error bodies, website routing behavior, public access enforcement, HEAD behavior, CORS handling, PutBucketWebsite limits, and end-to-end website hosting through a Docker-based dnsmasq test setup and CI workflow.
2026-06-10 12:41:55 +04:00
Marc Singer 375c2764d5 Add website integration tests and remove NotImplemented stubs
Replace PutBucketWebsite, GetBucketWebsite, DeleteBucketWebsite
NotImplemented test stubs with comprehensive integration tests covering:
- non-existing bucket errors
- validation (empty suffix, suffix with slash, invalid protocol, mutual
  exclusion of RedirectAllRequestsTo and IndexDocument)
- successful put/get round-trips for both index+error and redirect-all configs
- delete idempotency and verification

Signed-off-by: Marc Singer <marc@singer.gg>

Add error document serving, routing rules, and integration tests

Implement Features 1 and 2 of S3 static website hosting:

- WebsiteErrorDocument controller wrapper intercepts 4xx errors on
  website-enabled buckets and serves the configured error document or
  evaluates post-request routing rules (error code match redirects)
- ResolveWebsiteIndex middleware now caches parsed WebsiteConfiguration
  in context, handles RedirectAllRequestsTo, evaluates pre-request
  routing rules (key prefix match redirects), and rewrites directory
  keys for index document
- MatchPreRequestRule and MatchPostRequestRule methods on
  WebsiteConfiguration for routing rule evaluation
- 14 unit tests for routing rule matching
- 7 integration tests covering error document, routing rules,
  redirect-all, and index document behavior

Signed-off-by: Marc Singer <marc@singer.gg>

Add separate website hosting endpoint with virtual-host routing

Signed-off-by: Marc Singer <marc@singer.gg>

Support catch-all mode for website endpoint when --website-domain is omitted

Signed-off-by: Marc Singer <marc@singer.gg>
2026-06-10 12:41:51 +04:00
Marc Singer ac5c3b4a86 Add WebsiteConfiguration types, validation, and S3 error codes
Add S3 bucket website configuration types with XML serialization support
in s3response/website.go. Includes IndexDocument, ErrorDocument,
RedirectAllRequestsTo, and RoutingRules with full validation matching
AWS S3 behavior.

Add corresponding S3 error codes: ErrNoSuchWebsiteConfiguration,
ErrInvalidWebsiteConfiguration, ErrInvalidWebsiteSuffix, and
ErrInvalidWebsiteRedirectCode.

Unit tests cover validation logic, XML round-trip, and parsing.

Signed-off-by: Marc Singer <marc@singer.gg>

Add website backend interface and implementations for posix and s3proxy

Add PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite methods
to the Backend interface with BackendUnsupported stubs that return
ErrNotImplemented.

Posix backend stores website config as a metadata attribute (key:
'website') following the same pattern as CORS. ScoutFS inherits via
embedding.

S3Proxy backend stores website config in the metadata bucket with
prefix 'vgw-meta-website-', consistent with existing ACL/policy/CORS
metadata storage. Returns ErrNoSuchWebsiteConfiguration when not found.

Signed-off-by: Marc Singer <marc@singer.gg>

Add website API controllers and wire into router

Add PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite
controller methods following the same pattern as CORS. Controllers
parse and validate WebsiteConfiguration XML, check IAM authorization,
and delegate to the backend.

Replace the three HandleErrorRoute(ErrNotImplemented) stubs in the
router with the new controller methods. Regenerate the backend mock
to include the new interface methods.

Signed-off-by: Marc Singer <marc@singer.gg>

Add website index document middleware and wire into router

Add ResolveWebsiteIndex middleware that rewrites directory-like object keys
(empty or ending with /) to include the IndexDocument suffix when website
hosting is enabled. Also handles RedirectAllRequestsTo by returning 301.

Wire the middleware into both GetObject and HeadObject handler chains in
the router, positioned after BucketObjectNameValidator and before auth.

Signed-off-by: Marc Singer <marc@singer.gg>
2026-06-10 12:40:45 +04:00
Ben McClelland aa52aa08d7 Merge pull request #2167 from miretskiy/add-on-listen-hook
s3api: add WithOnListen option for server readiness notification
2026-06-06 09:11:04 -07:00
Yevgeniy Miretskiy 8495cf42b6 s3api: add WithOnListen option for server readiness notification
WithOnListen registers a callback invoked once the server is bound and
ready to accept connections. It wraps fiber's existing OnListen hook,
which fires immediately before the first connection is served.

This allows callers to detect readiness deterministically rather than
relying on a fixed sleep or polling a health endpoint.

Example use in tests:

    ready := make(chan struct{})
    srv, _ := s3api.New(..., s3api.WithOnListen(func() { close(ready) }))
    go srv.ServeMultiPort([]string{addr})
    <-ready // blocks until the server is accepting connections
2026-06-05 07:24:33 -04:00
Ben McClelland e137e8d375 fix: connection early termination resulting in internal error
When the connection terminates before all bytes read, we were
getting an io.ErrUnexpectedEOF that was not being handled as
a standard io.EOF resulting in an internal error being raised.
Translate io.ErrUnexpectedEOF to io.EOF so that we return the
normal errors for unexpected content. Add a log message so
that its clear the error is due to the connection being
terminated before all data sent and not the fault of the
gateway.
2026-06-01 10:12:41 -07:00
Ben McClelland da6aa1c843 Merge pull request #2159 from versity/ben/webui-storageclass
fix: expose x-amz-storage-class in CORS response headers
2026-06-01 09:23:01 -07:00
niksis02 577470214d fix: enforce required SignedHeaders validation for SigV4 requests
Validate required signed headers for both Authorization-header SigV4 requests and presigned URLs. The required signed header set is now `host` plus every incoming header with the `x-amz-` prefix.

During request reconstruction, signed headers and explicitly ignored headers are copied into the generated request used for signature verification. If an incoming `x-amz-*` header is present but missing from the client-provided `SignedHeaders`, return `AccessDenied` with a `HeadersNotSigned` field. The `host` header remains part of the canonical request and signed header calculation.

Previously, a client could sign a request without an S3 control header and then add that header after signing. For example, a presigned `PUT` URL could be generated with only `host` signed, then the actual request could include an unsigned `X-Amz-Tagging` or `X-Amz-Copy-Source` header. Because the verifier reconstructed the request only from `SignedHeaders`, that extra header was omitted from signature calculation and could pass authentication even though it changed the request semantics. This is now rejected with `AccessDenied`.

Expose v4 helper methods for checking required and ignored headers, and update canonical header signing so ignored headers can still be included when a client explicitly lists them in `SignedHeaders`, while `Authorization` remains excluded from signature calculation.
2026-05-30 21:16:26 +04:00
Ben McClelland 0e165edfb1 fix: expose x-amz-storage-class in CORS response headers
The object info modal in the WebUI was always displaying STANDARD as the
storage class regardless of the actual value. The root cause is a browser
CORS restriction: when the WebUI makes a cross-origin HEAD request to the
S3 endpoint, the browser silently drops any response header not listed in
Access-Control-Expose-Headers, causing response.headers.get('x-amz-storage-class')
to return null and the UI to fall back to the hardcoded STANDARD default.

Adding x-amz-storage-class to the default set of exposed headers ensures
the browser makes it available to JavaScript, allowing storage classes such
as GLACIER to be correctly reflected in the UI.
2026-05-29 20:17:56 -07:00
Ben McClelland 2bc4da944f Merge pull request #2153 from versity/sis/policy-key-normalization
fix: normalize object keys during bucket policy evaluation
2026-05-27 20:17:25 -07:00
niksis02 cd0b4e6d9d fix: normalize object keys during bucket policy evaluation
Object key validation allowed internal parent-directory segments such as `public/../private.txt`. Bucket policy and auth checks evaluated the raw key, so a policy allowing bucket/public/* could match the request while posix backend later resolved the key with `filepath.Join` and accessed `bucket/private.txt`.

Add backend-specific object key normalization to close that mismatch. The Backend interface now exposes `NormalizeObjectKey` so authorization can evaluate resources using the same key shape a backend will use for storage access.

Backends that do not collapse object paths, including Azure and the S3 proxy, inherit `BackendUnsupported.NormalizeObjectKey`. That implementation returns the input key unchanged, avoiding unnecessary normalization and keeping policy evaluation unpolluted for object stores where ../ is part of the key name.

posix/scoutfs normalize keys with filepath.Join so policy resources and request keys are compared after internal dot segments are collapsed.

Bucket policy evaluation now normalizes both the incoming object key and object resource patterns from the policy before matching. Object lock governance bypass policy checks use the same backend normalizer as well, so retention and legal hold authorization cannot diverge from backend path resolution.
2026-05-27 22:20:39 +04:00
niksis02 4ef090dbfc fix: fix empty ownership control rules panic
Fix the gateway panic when validating malformed bucket ownership controls bodies with no rules. The handler now checks the rules count before indexing the first rule.
2026-05-23 04:18:06 +04:00
niksis02 eecc1a779c fix: reject invalid PostObject keys
Validate multipart PostObject key fields with the existing object name rules so path traversal and degenerate names return BadRequest. This prevents crafted object keys from escaping the gateway root.
2026-05-22 14:48:31 +04:00
niksis02 9f786b3c2c feat: global error refactoring
Fixes #2123
Fixes #2120
Fixes #2116
Fixes #2111
Fixes #2108
Fixes #2086
Fixes #2085
Fixes #2083
Fixes #2081
Fixes #2080
Fixes #2073
Fixes #2072
Fixes #2071
Fixes #2069
Fixes #2044
Fixes #2043
Fixes #2042
Fixes #2041
Fixes #2040
Fixes #2039
Fixes #2036
Fixes #2035
Fixes #2034
Fixes #2028
Fixes #2020
Fixes #1842
Fixes #1810
Fixes #1780
Fixes #1775
Fixes #1736
Fixes #1705
Fixes #1663
Fixes #1645
Fixes #1583
Fixes #1526
Fixes #1514
Fixes #1493
Fixes #1487
Fixes #959
Fixes #779
Closes #823
Closes #85

Refactor global S3 error handling around structured error types and centralized XML response generation.

All S3 errors now share the common APIError base for the fields every error has: Code, HTTP status code, and Message. Non-traditional errors that need AWS-compatible XML fields now have dedicated typed errors in the s3err package. Each typed error implements the shared S3Error behavior so controllers and middleware can handle errors consistently while still emitting error-specific XML fields.

Add a dedicated InvalidArgumentError type because InvalidArgument is used widely across request validation, auth, copy source handling, object lock validation, multipart validation, and header parsing. The new InvalidArgument path uses explicit InvalidArgErrorCode constants with predefined descriptions and ArgumentName values, keeping call sites readable while preserving the correct InvalidArgument XML shape and optional ArgumentValue.

New structured errors added in s3err:
- `AccessForbiddenError`: Method, ResourceType
- `BadDigestError`: CalculatedDigest, ExpectedDigest
- `BucketError`: BucketName
- `ContentSHA256MismatchError`: ClientComputedContentSHA256, S3ComputedContentSHA256
- `EntityTooLargeError`: ProposedSize, MaxSizeAllowed
- `EntityTooSmallError`: ProposedSize, MinSizeAllowed
- `ExpiredPresignedURLError`: ServerTime, XAmzExpires, Expires
- `InvalidAccessKeyIdError`: AWSAccessKeyId
- `InvalidArgumentError`: Description, ArgumentName, ArgumentValue
- `InvalidChunkSizeError`: Chunk, BadChunkSize
- `InvalidDigestError`: ContentMD5
- `InvalidLocationConstraintError`: LocationConstraint
- `InvalidPartError`: UploadId, PartNumber, ETag
- `InvalidRangeError`: RangeRequested, ActualObjectSize
- `InvalidTagError`: TagKey, TagValue
- `KeyTooLongError`: Size, MaxSizeAllowed
- `MetadataTooLargeError`: Size, MaxSizeAllowed
- `MethodNotAllowedError`: Method, ResourceType, AllowedMethods
- `NoSuchUploadError`: UploadId
- `NoSuchVersionError`: Key, VersionId
- `NotImplementedError`: Header, AdditionalMessage
- `PreconditionFailedError`: Condition
- `RequestTimeTooSkewedError`: RequestTime, ServerTime, MaxAllowedSkewMilliseconds
- `SignatureDoesNotMatchError`: AWSAccessKeyId, StringToSign, SignatureProvided, StringToSignBytes, CanonicalRequest, CanonicalRequestBytes

Fix CompleteMultipartUpload validation in the Azure backend so missing or empty `ETag` values return the appropriate S3 error instead of allowing a gateway panic.

Fix presigned authentication expiration validation to compare server time in `UTC`, matching the `UTC` timestamp used by presigned URL signing.

Add request ID and host ID support across S3 requests. Each request now receives AWS S3-like identifiers, returned in response headers as `x-amz-request-id` and `x-amz-id-2` and included in all XML error responses as RequestId and HostId. The generated ID structure is designed to resemble AWS S3 request IDs and host IDs.

The request signature calculation/validation for streaming uploads was previously delayed until the request body was fully read, both for Authorization header authentication and presigned URLs.
Now, the signature is validated immediately in the authorization middlewares without reading the request body, since the signature calculation itself does not depend on the request body. Instead, only the `x-amz-content-sha256` SHA-256 hash calculation is delayed.
2026-05-21 23:49:34 +04:00
niksis02 e69d073273 fix: reject SigV2 requests
The gateway currently supports only Signature Version 4 (SigV4) authorization. Deprecated AWS SigV2 requests are now rejected with an AWS-specific `InvalidRequest` error for both Authorization-header requests and query-string requests(presigned URLs).

This also fixes SigV4 Authorization-header handling for date headers. SigV4 accepts two date headers: `Date` and `X-Amz-Date`. `X-Amz-Date` takes precedence, but when it is missing, `Date` should be used. The gateway now uses the `Date` header with lower precedence when `X-Amz-Date` is not present. No SDK integration test was added for this case because the SDK always sets `X-Amz-Date`, and this behavior is not configurable.
2026-05-06 23:43:10 +04:00
Ben McClelland 21da1d7e70 Merge pull request #2097 from versity/dependabot/go_modules/dev-dependencies-3789651bc5
chore(deps): bump the dev-dependencies group with 7 updates
2026-05-04 16:24:32 -07:00
niksis02 d2fa265fb8 feat: support sha512, md5, xxhash3, xxhash64, xxhash128 data integrity checksums
Integrate the new S3 checksum types in the gateway, including `SHA512`, `MD5`, `XXHASH64`, `XXHASH3`, and `XXHASH128`. This adds checksum calculation, validation, schema handling, and test coverage for the expanded checksum support.

These external packages have been used:
- `github.com/zeebo/xxh3` for `XXHASH3` and `XXHASH128`
- `github.com/cespare/xxhash/v2` for `XXHASH64`

Adjust integration tests because `aws-sdk-go-v2/service/s3` does not support automatic checksum calculation for the new checksum algorithms and returns an SDK-level error when only the checksum algorithm is provided. Only precalculated checksum values are acceptable for these checksum types.

References:
- `https://github.com/aws/aws-sdk-go-v2/issues/3404`
- `https://github.com/aws/aws-sdk-go-v2/issues/3403`
2026-05-04 08:50:39 -07:00
Ben McClelland 6731179e12 Merge pull request #2107 from versity/sis/unsigned-streaming-payload-caching-fix
fix: remove unsigned chunk reader caching
2026-05-04 08:42:33 -07:00
Yevgeniy Miretskiy d1fba07fe6 feat: add custom route and middleware options
Add options for embedders to register Fiber routes and middleware before the S3 route table is initialized.

WithRoute registers a top-level route with explicit method and path matching. WithMiddleware registers prefix middleware that can handle a request or call ctx.Next() to continue into the S3 stack.

Add coverage for route registration order when a top-level route and catch-all middleware are both configured.
2026-05-04 08:41:16 -07:00
niksis02 27971f2a20 fix: remove unsigned chunk reader caching
Closes #1273

Rewrite UnsignedChunkReader to stream the payload bytes directly into the caller buffer instead of allocating and stashing full chunks. With this implementation, no stash is held by the reader and the chunk reader doesn't allocate any memory.

Make debug logging more descriptive, which records reader state on all error paths and logs read progress whenever a Read call fills the caller buffer.

Some unit tests were added to cover the main moving parts of the reader flow.
2026-05-02 01:14:38 +04:00
Dylan Dellett-Wion e6aa9de052 fix: apply CORS middleware to admin CreateBucket route
The PATCH /:bucket/create admin route was missing
middlewares.ApplyDefaultCORS, while every other admin PATCH route
applies it. The OPTIONS preflight handler already sets CORS headers,
so browsers pass preflight but block the actual response for lacking
Access-Control-Allow-Origin. This caused the WebUI bucket-creation
flow to fail with ERR_FAILED even though the server returned 201.

Fixes #2105. Introduced in #1739 when the endpoint was added.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-04-30 21:06:55 -04:00
niksis02 8d5b2be0b2 fix: check PutObjectTagging/LegalHold/Retention permissions on PutObject,CopyObject and CreateMultipartUpload
Fixes #1986

When a client includes tagging, legal hold, or retention headers in a PutObject, CopyObject or CreateMultipartUpload request, the corresponding bucket policy permissions must be verified in addition to s3:PutObject:

`X-Amz-Tagging` - `s3:PutObjectTagging`
`X-Amz-Object-Lock-Legal-Hold` - `s3:PutObjectLegalHold`
`X-Amz-Object-Lock-Mode` - `s3:PutObjectRetention`

Previously, only s3:PutObject was checked, allowing users to set tagging, legal hold, and retention without having the required permissions. Now each action permission is check, if user tries to add them.

For CopyObject these permissions are checked on destination object.
2026-04-28 01:05:34 +04:00
Ben McClelland 0dc074acbf feat: add --socket-perm option for UNIX socket file permissions
Add a --socket-perm flag (VGW_SOCKET_PERM env var) to control the
file-mode permissions on file-backed UNIX domain sockets. This allows
operators to limit access permission without relying on process umask.
The option applies to S3, admin, and WebUI sockets and has no effect
on TCP/IP addresses or Linux abstract namespace sockets.

Fixes #2010
2026-04-20 19:07:08 -07:00
niksis02 d6fb9547b8 fix: correct 206 Partial Content response status for ranged GetObject and HeadObject
Fixes #2052
Fixes #2056
Fixes #2057

Previously, GetObject and HeadObject used the request's `Range` header to determine the response status code, which caused incorrect 206 responses for invalid Range header values.

The status is now driven by whether res.ContentRange is set in the response, rather than by the presence of a range in the request. Backends (posix and azure) now set Content-Range for PartNumber=1 on non-multipart objects, skipping zero-size objects where no range applies.

HeadObject was also fixed to return 206 when Content-Range is present, and to only return checksums when the full object is requested.
2026-04-21 02:13:04 +04:00
niksis02 62e8cddbc7 fix: make CompleteMultipartUpload idempotent and add part-number support to GetObject/HeadObject
Closes #1064

Use the multipart ETag as the in-progress directory suffix instead of the static `.inprogress` marker so that concurrent CompleteMultipartUpload calls for the same upload ID are all treated as successful (idempotent) rather than racing, where only one succeeded and the rest returned NoSuchUpload.

After finalizing the multipart upload, store an `mp-metadata` xattr on the assembled object that records the upload ID and cumulative byte offsets for each part. GetObject and HeadObject now use this metadata to serve individual part ranges via the `partNumber` query parameter, returning a successful response instead of returning NotImplemented.

Add two new S3 error codes:
- `ErrInvalidPartNumberRange` (416 RequestedRangeNotSatisfiable) — returned
  when the requested part number exceeds the number of parts in the upload.
- `ErrRangeAndPartNumber` (400 BadRequest) — returned when both a Range header
  and a partNumber query parameter are specified on the same request.
2026-04-20 20:45:58 +04:00
Ben McClelland 2b918d585e chore: fix spellings and unused function args
No logic changes, just janitorial cleanup
2026-04-15 10:32:14 -07:00
Ben McClelland 873148a5c4 fix: add Host header to HTTP test requests for fasthttp v1.70.0 compatibility
fasthttp v1.70.0 now enforces the HTTP/1.1 requirement of exactly
one Host header, rejecting requests that omit it. Fix tests that
were failing due to missing host.
2026-04-13 19:13:58 -07:00
niksis02 b473aa0545 fix: move versionId validation to backend
Closes #1813

We use a specific `versionId` format(`ulid` package) to generate versionIds in posix, which is not compatible to S3. The versionId validation was performed in frontend which is a potential source of failure for s3 proxy configured on an s3 service which doesn't use ulid for versionId generation(e.g. aws S3). These changes move the specific `ulid` versionId validation to posix to not force any specific versionId format in the gateway.
2026-04-07 01:56:51 +04:00
Ben McClelland 98a186cc4e Merge pull request #2008 from versity/sis/object-post-boundary-prefix
fix: remove POST object multipart boundary prefix trimming
2026-04-02 08:27:02 -07:00
Ben McClelland 71ae9bf045 Merge pull request #2005 from versity/sis/copyobject-expected-source-bucket-owner
feat: implement x-amz-source-expected-bucket-owner for CopyObject and UploadPartCopy
2026-04-02 08:25:50 -07:00
niksis02 a25408c225 fix: remove POST object multipart boundary prefix trimming
Fixes the [comment](https://github.com/versity/versitygw/issues/1648#issuecomment-4175425099)

Removes the unnecessary multipart/form-data boundary normalizing. The boundary prefix(`--`) was trimmed in `NewMultipartParser`, which caused incorrect boundary check for the boundaries starting with 2 dashes(e.g. `----WebKitFormBoundaryABC123`).
2026-04-02 17:20:23 +04:00
niksis02 052f2364cc feat: implement x-amz-source-expected-bucket-owner for CopyObject and UploadPartCopy
Closes #1897

Extract the `X-Amz-Source-Expected-Bucket-Owner` header for CopyObject and UploadPartCopy. Verify the source bucket owner in the backend and if the provided access key id doesn't match, return an `AccessDenied` error.
2026-04-01 21:44:33 +04:00
niksis02 3d47e7fed1 fix: add x-amz-bucket-region to ListObjects(V2) response headers.
Fixes #1814

The `x-amz-bucket-region` is not mentioned in AWS S3 documentation, however s3 sends it in all ListObjects(V2) successful responses. The header is now added.
2026-04-01 15:22:33 +04:00
Ben McClelland e0209ebab4 Merge pull request #1997 from versity/sis/copyobject-threshold
fix: enforce 5gb copy source object size threshold.
2026-03-31 12:27:11 -07:00
Ben McClelland 89ab7a4f2b Merge pull request #1982 from versity/sis/POST-object
feat: add browser-based POST object upload support
2026-03-31 12:25:00 -07:00
niksis02 59002b2650 feat: implement integration tests for browser-based POST object 2026-03-31 22:47:04 +04:00
niksis02 285d130a47 feat: add browser-based POST object upload support
Closes #1648
Fixes #1980
Fixes #1981

This PR implements browser-based POST object uploads for S3-compatible form uploads. It adds support for handling `multipart/form-data` object uploads submitted from browsers, including streaming multipart parsing so file content is not buffered in memory, POST policy decoding and evaluation, SigV4-based form authorization, and integration with the existing `PutObject` backend flow. The implementation covers the full browser POST upload path, including validation of required form fields, credential scope and request date checks, signature verification, metadata extraction from `x-amz-meta-*` fields, checksum field parsing, object tagging conversion from XML into the query-string format expected by `PutObject`, and browser-compatible success handling through `success_action_status` and `success_action_redirect`. It also wires the new flow into the router and metrics layer and adds POST-specific error handling and debug logging across policy parsing, multipart parsing, and POST authorization. AWS S3 also accepts the `redirect` form field alongside `success_action_redirect`, but since AWS has marked `redirect` as deprecated and is planning to remove it, this gateway intentionally does not support it.
2026-03-31 22:44:54 +04:00
niksis02 bbe246e8ec fix: enforce 5gb copy source object size threshold.
Fixes #1896

Enforces the S3 `5 GiB` copy source size limit across the posix and azure
backends for `CopyObject` and `UploadPartCopy`, returning `InvalidRequest` when
the source object exceeds the threshold.

The limit is now configurable via `--copy-object-threshold`
(`VGW_COPY_OBJECT_THRESHOLD`, default 5 GiB).
A new `--mp-max-parts flag` (`VGW_MP_MAX_PARTS`, default `10000`) has been added to make multipart upload parts number limit configurable.

No integration test has been added, as GitHub Actions cannot reliably
handle large objects.
2026-03-31 22:44:03 +04:00
niksis02 3b17f05d17 feat: support response header overrides in HeadObject
Closes #1967

Add support for response header override query parameters(`response-cache-control`, `response-content-disposition`, `response-content-encoding`, `response-content-language`, `response-content-type`, `response-expires`) in `HeadObject`. Anonymous requests with override params are rejected with `ErrAnonymousResponseHeaders`.
2026-03-31 17:19:03 +04:00
Ben McClelland ffe73834f2 fix: allow list buckets to use default aws region us-east-1
Some SDKs/clients make the assumption that us-east-1 will always
work for bucket listings even if individual buckets are in
different regions. This is how AWS works with allowing full
bucket listing from the default us-east-1 region as well as
any other valid region.

Fixes #1879
2026-03-11 10:07:47 -07:00
Ben McClelland 97cc6bf23b chore: run go modernize tool
This is a fixup of the codebase using:
go run golang.org/x/tools/go/analysis/passes/modernize/cmd/modernize@latest -fix ./...

This has no bahvior changes, and only updates safe changes for
modern go features.
2026-03-10 09:47:37 -07:00
Ben McClelland a152f29113 feat: option to host webui on s3 port
This adds the webui-s3-prefix option to specify a prefix and host
the webui on the same port as the s3 service. Like the health
endpoint, this will mask any bucket with the same name as the
webui prefix.

The benefit of hosting this on the same interface as the s3
service is no longer needing the CORS headers for the browser
access if the webui and s3 access are on the same IP:PORT.
2026-03-10 08:20:19 -07:00
Ben McClelland 710003f341 feat: add unix domain socket listener support to port option
This adds the ability to specify unix domain socket paths for the
service listener with the --port <path> option. Where <path> can
be either a path to a file in a filesystem or prefixed with @ for
an abstract socket name.

Anything not matching the <host>:<port> pattern in the --port
option will be considered a socket filename.
2026-03-07 22:05:41 -08:00
niksis02 21a636b3b5 fix: add request headers and metadata headers limit
Fixes #1606

According to AWS documentation:
> *“The PUT request header is limited to 8 KB in size. Within the PUT request header, the user-defined metadata is limited to 2 KB in size. The size of user-defined metadata is measured by taking the sum of the number of bytes in the UTF-8 encoding of each key and value.”*

Based on this, object metadata size is now limited to **2 KB** for all object upload operations (`PutObject`, `CopyObject`, and `CreateMultipartUpload`).

Fixes handling of metadata HTTP headers when the same header appears multiple times with different casing or even if they are identical. According to S3 behavior, these headers must be merged into a single lower-cased metadata key, with values concatenated using commas.

Example:

```
x-amz-meta-Key: value1
x-amz-meta-kEy: value2
x-amz-meta-keY: value3
```

Translated to:

```
key: value1,value2,value3
```

This PR also introduces an **8 KB limit for request headers**. Although the S3 documentation explicitly mentions the 8 KB limit only for **PUT requests**, in practice this limit applies to **all requests**.

To enforce the header size limit, the Fiber configuration option `ReadBufferSize` is used. This parameter defines the maximum number of bytes read when parsing an incoming request. Note that this limit does not apply strictly to request headers only, since request parsing also includes other parts of the request line (e.g., the HTTP method, protocol string, and version such as `HTTP/1.1`). So `ReadBufferSize` is effectively a limit for request headers size, but not the exact limit.
2026-03-06 23:25:49 +04:00
niksis02 5c918f3682 feat: revert ignore object ACL behavior
The logic to return a `NotImplemented` error on object upload operations, when any ACL header is present has been removed. Now all object ACL headers are by default ignored. The `-noacl` flag is preserved to disabled bucket ACLs.

**Testing**
The Put/Get object ACL tests are moved to `NotImplemented` integration tests group as a default gateway behavior. The existing `_acl_not_supported` tests are modified to expect no error, when ACLs are used on object uploads.
2026-03-02 19:30:57 +04:00
niksis02 5ae791b154 feat: configuration option to disable ACLs
Closes #1847

This PR introduces a global optional gateway CLI flag `--disable-acl` (`VGW_DISABLE_ACL`) to disable ACL handling. When this flag is enabled, the gateway ignores all ACL-related headers, particularly in `CreateBucket`, `PutObject`, `CopyObject`, and `CreateMultipartUpload`.

`GetBucketAcl` behavior is unchanged simply returning the bucket ACL config.
There's no change in object ACL actions(`PutObjectACL`, `GetObjectACL`). They return a`NotImplemented` error as before.

A new custom error is added for PutBucketAcl calls when ACLs are disabled at the gateway level. Its HTTP status code and error code match AWS S3’s behavior, with only a slightly different error message.

In the access-control checker, ACL evaluation is fully bypassed. If ACLs are disabled only the bucket owner gets access to the bucket and all grantee checks are ignored.

The PR also includes minor refactoring of the S3 API server and router. The growing list of parameters passed to the router’s Init method has been consolidated into fields within the router struct, initialized during router construction. Parameters not needed by the S3 server are no longer stored in the server configuration and are instead forwarded directly to the router.
2026-02-27 20:04:13 +04:00
niksis02 6fafc15d08 fix: fixes PutBucketCors CORSRules validation
Fixes #1870
Fixes #1863

A validation has been added to **PutBucketCors** for `CORSRule.AllowedOrigins`. The `AllowedOrigins` list can no longer be empty—otherwise a **MalformedXML** error is returned. Additionally, each origin is now validated to ensure it does not contain more than one wildcard.

A similar validation has been added for `AllowedMethods`. The list must not be empty, or a **MalformedXML** error is returned. Previously, empty method values (e.g., `[]string{""}`) were incorrectly treated as valid. This has been fixed, and an **UnsupportedCORSMethod** error is now returned.
2026-02-24 16:59:38 +04:00