Commit Graph

261 Commits

Author SHA1 Message Date
Ben McClelland 27f04ad5ea feat: add windows functional test coverage and fix some windows behavior
This change adds Windows functional test execution in CI and updates
backend handling so windows filesystem error/path semantics map correctly
to expected S3 outcomes.

The only meta supported on windows right now is sidecar, so the tests
in windows mode also skip sidecar skips.

Future work is to address the skips and/or more clearly document
the unsupported/incompatible behavior on windows.

The windows support will still remain best effort, but these tests
should at least flag when future changes introduce incompatible
behavior on windows.
2026-06-18 11:34:05 -07:00
niksis02 12fb5a6594 fix: reject empty Content-MD5 on PUT operations
Fixes #2179

Treat a present but empty `Content-MD5` header as an invalid digest instead of handling it as if the header were absent. This makes PUT operations return `InvalidDigest` for empty `Content-MD5` values while preserving existing behavior for missing headers.
2026-06-15 18:53:42 +04:00
niksis02 f08f76fea4 feat: support x-amz-website-redirect-location
Integrate x-amz-website-redirect-location across object metadata flows so uploads, copies, multipart creation, HEAD, and GET preserve and return redirect locations, and website hosting applies object-level redirects from the stored value.
2026-06-10 12:41:55 +04:00
niksis02 1625c5963e feat: improve static website hosting support
Enhances the static website hosting implementation with more complete S3-compatible behavior across request handling, backend storage, validation, CORS, and errors.

Adds dedicated website endpoint handling for GET, HEAD, and OPTIONS requests, including index document resolution, error document serving, redirect-all support, pre-fetch and post-error routing rules, query string preservation in redirects, public access checks before object reads, and method-not-allowed responses.

Improves error handling for website responses by returning S3-compatible HTML error bodies with request IDs, host IDs, x-amz-error-code, x-amz-error-message, and specialized error fields. This also fixes website-related validation errors to return more accurate S3-style error codes and messages, including invalid redirect protocols, invalid HTTP redirect/error codes, conflicting routing rule replacements, routing rule limits, and oversized website configuration requests.

Adds website CORS support for GET, HEAD, and OPTIONS preflight requests, including bucket CORS lookup through website host bucket resolution, allowed origin/method/header validation, exposed header handling, ETag exposure, Vary headers, max-age handling, and CORS access-denied responses.

Adds debug logging around website configuration parsing, validation failures, CORS checks, backend lookup failures, and internal website error paths to make failures easier to diagnose.

Adds compressed website configuration storage so larger configs fit backend metadata limits, including gzip storage for POSIX extended attributes and base64-encoded compressed metadata for Azure. Also adds Azure PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite support.

Adds and expands test coverage for website config validation, S3-compatible HTML error bodies, website routing behavior, public access enforcement, HEAD behavior, CORS handling, PutBucketWebsite limits, and end-to-end website hosting through a Docker-based dnsmasq test setup and CI workflow.
2026-06-10 12:41:55 +04:00
Marc Singer 375c2764d5 Add website integration tests and remove NotImplemented stubs
Replace PutBucketWebsite, GetBucketWebsite, DeleteBucketWebsite
NotImplemented test stubs with comprehensive integration tests covering:
- non-existing bucket errors
- validation (empty suffix, suffix with slash, invalid protocol, mutual
  exclusion of RedirectAllRequestsTo and IndexDocument)
- successful put/get round-trips for both index+error and redirect-all configs
- delete idempotency and verification

Signed-off-by: Marc Singer <marc@singer.gg>

Add error document serving, routing rules, and integration tests

Implement Features 1 and 2 of S3 static website hosting:

- WebsiteErrorDocument controller wrapper intercepts 4xx errors on
  website-enabled buckets and serves the configured error document or
  evaluates post-request routing rules (error code match redirects)
- ResolveWebsiteIndex middleware now caches parsed WebsiteConfiguration
  in context, handles RedirectAllRequestsTo, evaluates pre-request
  routing rules (key prefix match redirects), and rewrites directory
  keys for index document
- MatchPreRequestRule and MatchPostRequestRule methods on
  WebsiteConfiguration for routing rule evaluation
- 14 unit tests for routing rule matching
- 7 integration tests covering error document, routing rules,
  redirect-all, and index document behavior

Signed-off-by: Marc Singer <marc@singer.gg>

Add separate website hosting endpoint with virtual-host routing

Signed-off-by: Marc Singer <marc@singer.gg>

Support catch-all mode for website endpoint when --website-domain is omitted

Signed-off-by: Marc Singer <marc@singer.gg>
2026-06-10 12:41:51 +04:00
niksis02 577470214d fix: enforce required SignedHeaders validation for SigV4 requests
Validate required signed headers for both Authorization-header SigV4 requests and presigned URLs. The required signed header set is now `host` plus every incoming header with the `x-amz-` prefix.

During request reconstruction, signed headers and explicitly ignored headers are copied into the generated request used for signature verification. If an incoming `x-amz-*` header is present but missing from the client-provided `SignedHeaders`, return `AccessDenied` with a `HeadersNotSigned` field. The `host` header remains part of the canonical request and signed header calculation.

Previously, a client could sign a request without an S3 control header and then add that header after signing. For example, a presigned `PUT` URL could be generated with only `host` signed, then the actual request could include an unsigned `X-Amz-Tagging` or `X-Amz-Copy-Source` header. Because the verifier reconstructed the request only from `SignedHeaders`, that extra header was omitted from signature calculation and could pass authentication even though it changed the request semantics. This is now rejected with `AccessDenied`.

Expose v4 helper methods for checking required and ignored headers, and update canonical header signing so ignored headers can still be included when a client explicitly lists them in `SignedHeaders`, while `Authorization` remains excluded from signature calculation.
2026-05-30 21:16:26 +04:00
Ben McClelland 2bc4da944f Merge pull request #2153 from versity/sis/policy-key-normalization
fix: normalize object keys during bucket policy evaluation
2026-05-27 20:17:25 -07:00
niksis02 cd0b4e6d9d fix: normalize object keys during bucket policy evaluation
Object key validation allowed internal parent-directory segments such as `public/../private.txt`. Bucket policy and auth checks evaluated the raw key, so a policy allowing bucket/public/* could match the request while posix backend later resolved the key with `filepath.Join` and accessed `bucket/private.txt`.

Add backend-specific object key normalization to close that mismatch. The Backend interface now exposes `NormalizeObjectKey` so authorization can evaluate resources using the same key shape a backend will use for storage access.

Backends that do not collapse object paths, including Azure and the S3 proxy, inherit `BackendUnsupported.NormalizeObjectKey`. That implementation returns the input key unchanged, avoiding unnecessary normalization and keeping policy evaluation unpolluted for object stores where ../ is part of the key name.

posix/scoutfs normalize keys with filepath.Join so policy resources and request keys are compared after internal dot segments are collapsed.

Bucket policy evaluation now normalizes both the incoming object key and object resource patterns from the policy before matching. Object lock governance bypass policy checks use the same backend normalizer as well, so retention and legal hold authorization cannot diverge from backend path resolution.
2026-05-27 22:20:39 +04:00
niksis02 4ef090dbfc fix: fix empty ownership control rules panic
Fix the gateway panic when validating malformed bucket ownership controls bodies with no rules. The handler now checks the rules count before indexing the first rule.
2026-05-23 04:18:06 +04:00
niksis02 eecc1a779c fix: reject invalid PostObject keys
Validate multipart PostObject key fields with the existing object name rules so path traversal and degenerate names return BadRequest. This prevents crafted object keys from escaping the gateway root.
2026-05-22 14:48:31 +04:00
niksis02 ed1ad6b623 fix: honor explicit public bucket policy deny
Distinguish public bucket policy no-match from explicit deny during anonymous access checks. This preserves ACL fallback only for requests that are not allowed by policy, while ensuring a matching Deny statement short-circuits authorization and returns AccessDenied even when a public ACL would otherwise grant access.
2026-05-22 00:01:24 +04:00
niksis02 9f786b3c2c feat: global error refactoring
Fixes #2123
Fixes #2120
Fixes #2116
Fixes #2111
Fixes #2108
Fixes #2086
Fixes #2085
Fixes #2083
Fixes #2081
Fixes #2080
Fixes #2073
Fixes #2072
Fixes #2071
Fixes #2069
Fixes #2044
Fixes #2043
Fixes #2042
Fixes #2041
Fixes #2040
Fixes #2039
Fixes #2036
Fixes #2035
Fixes #2034
Fixes #2028
Fixes #2020
Fixes #1842
Fixes #1810
Fixes #1780
Fixes #1775
Fixes #1736
Fixes #1705
Fixes #1663
Fixes #1645
Fixes #1583
Fixes #1526
Fixes #1514
Fixes #1493
Fixes #1487
Fixes #959
Fixes #779
Closes #823
Closes #85

Refactor global S3 error handling around structured error types and centralized XML response generation.

All S3 errors now share the common APIError base for the fields every error has: Code, HTTP status code, and Message. Non-traditional errors that need AWS-compatible XML fields now have dedicated typed errors in the s3err package. Each typed error implements the shared S3Error behavior so controllers and middleware can handle errors consistently while still emitting error-specific XML fields.

Add a dedicated InvalidArgumentError type because InvalidArgument is used widely across request validation, auth, copy source handling, object lock validation, multipart validation, and header parsing. The new InvalidArgument path uses explicit InvalidArgErrorCode constants with predefined descriptions and ArgumentName values, keeping call sites readable while preserving the correct InvalidArgument XML shape and optional ArgumentValue.

New structured errors added in s3err:
- `AccessForbiddenError`: Method, ResourceType
- `BadDigestError`: CalculatedDigest, ExpectedDigest
- `BucketError`: BucketName
- `ContentSHA256MismatchError`: ClientComputedContentSHA256, S3ComputedContentSHA256
- `EntityTooLargeError`: ProposedSize, MaxSizeAllowed
- `EntityTooSmallError`: ProposedSize, MinSizeAllowed
- `ExpiredPresignedURLError`: ServerTime, XAmzExpires, Expires
- `InvalidAccessKeyIdError`: AWSAccessKeyId
- `InvalidArgumentError`: Description, ArgumentName, ArgumentValue
- `InvalidChunkSizeError`: Chunk, BadChunkSize
- `InvalidDigestError`: ContentMD5
- `InvalidLocationConstraintError`: LocationConstraint
- `InvalidPartError`: UploadId, PartNumber, ETag
- `InvalidRangeError`: RangeRequested, ActualObjectSize
- `InvalidTagError`: TagKey, TagValue
- `KeyTooLongError`: Size, MaxSizeAllowed
- `MetadataTooLargeError`: Size, MaxSizeAllowed
- `MethodNotAllowedError`: Method, ResourceType, AllowedMethods
- `NoSuchUploadError`: UploadId
- `NoSuchVersionError`: Key, VersionId
- `NotImplementedError`: Header, AdditionalMessage
- `PreconditionFailedError`: Condition
- `RequestTimeTooSkewedError`: RequestTime, ServerTime, MaxAllowedSkewMilliseconds
- `SignatureDoesNotMatchError`: AWSAccessKeyId, StringToSign, SignatureProvided, StringToSignBytes, CanonicalRequest, CanonicalRequestBytes

Fix CompleteMultipartUpload validation in the Azure backend so missing or empty `ETag` values return the appropriate S3 error instead of allowing a gateway panic.

Fix presigned authentication expiration validation to compare server time in `UTC`, matching the `UTC` timestamp used by presigned URL signing.

Add request ID and host ID support across S3 requests. Each request now receives AWS S3-like identifiers, returned in response headers as `x-amz-request-id` and `x-amz-id-2` and included in all XML error responses as RequestId and HostId. The generated ID structure is designed to resemble AWS S3 request IDs and host IDs.

The request signature calculation/validation for streaming uploads was previously delayed until the request body was fully read, both for Authorization header authentication and presigned URLs.
Now, the signature is validated immediately in the authorization middlewares without reading the request body, since the signature calculation itself does not depend on the request body. Instead, only the `x-amz-content-sha256` SHA-256 hash calculation is delayed.
2026-05-21 23:49:34 +04:00
Ben McClelland 2c0844ad88 fix: ignore implicit directories for Get/HeadObject
Directories that exist on the filesystem but were not explicitly created
via S3 (put-object with a key ending in '/') do not have an etag
value. ListObjectsV2 already uses the presence of this attribute to
decide whether to include a directory as an object. GetObject and
HeadObject were not performing this check, so they would successfully
return directories that ListObjectsV2 would not list.

Add the etag attribute check in GetObject and HeadObject: if a
directory path is requested but has no etag xattr, return 404. This
makes all three operations agree on which directories are S3 objects.

Fixes #2130
2026-05-12 18:56:00 -07:00
niksis02 e69d073273 fix: reject SigV2 requests
The gateway currently supports only Signature Version 4 (SigV4) authorization. Deprecated AWS SigV2 requests are now rejected with an AWS-specific `InvalidRequest` error for both Authorization-header requests and query-string requests(presigned URLs).

This also fixes SigV4 Authorization-header handling for date headers. SigV4 accepts two date headers: `Date` and `X-Amz-Date`. `X-Amz-Date` takes precedence, but when it is missing, `Date` should be used. The gateway now uses the `Date` header with lower precedence when `X-Amz-Date` is not present. No SDK integration test was added for this case because the SDK always sets `X-Amz-Date`, and this behavior is not configurable.
2026-05-06 23:43:10 +04:00
Ben McClelland bb3cdd9cb6 fix: skip integration tests not compatible in sidecar
A few tests are not yet compatible with sidecar mode. Add a test
option to skip these when running sidecar tests.
2026-05-04 16:05:29 -07:00
niksis02 d2fa265fb8 feat: support sha512, md5, xxhash3, xxhash64, xxhash128 data integrity checksums
Integrate the new S3 checksum types in the gateway, including `SHA512`, `MD5`, `XXHASH64`, `XXHASH3`, and `XXHASH128`. This adds checksum calculation, validation, schema handling, and test coverage for the expanded checksum support.

These external packages have been used:
- `github.com/zeebo/xxh3` for `XXHASH3` and `XXHASH128`
- `github.com/cespare/xxhash/v2` for `XXHASH64`

Adjust integration tests because `aws-sdk-go-v2/service/s3` does not support automatic checksum calculation for the new checksum algorithms and returns an SDK-level error when only the checksum algorithm is provided. Only precalculated checksum values are acceptable for these checksum types.

References:
- `https://github.com/aws/aws-sdk-go-v2/issues/3404`
- `https://github.com/aws/aws-sdk-go-v2/issues/3403`
2026-05-04 08:50:39 -07:00
niksis02 8d5b2be0b2 fix: check PutObjectTagging/LegalHold/Retention permissions on PutObject,CopyObject and CreateMultipartUpload
Fixes #1986

When a client includes tagging, legal hold, or retention headers in a PutObject, CopyObject or CreateMultipartUpload request, the corresponding bucket policy permissions must be verified in addition to s3:PutObject:

`X-Amz-Tagging` - `s3:PutObjectTagging`
`X-Amz-Object-Lock-Legal-Hold` - `s3:PutObjectLegalHold`
`X-Amz-Object-Lock-Mode` - `s3:PutObjectRetention`

Previously, only s3:PutObject was checked, allowing users to set tagging, legal hold, and retention without having the required permissions. Now each action permission is check, if user tries to add them.

For CopyObject these permissions are checked on destination object.
2026-04-28 01:05:34 +04:00
niksis02 d6fb9547b8 fix: correct 206 Partial Content response status for ranged GetObject and HeadObject
Fixes #2052
Fixes #2056
Fixes #2057

Previously, GetObject and HeadObject used the request's `Range` header to determine the response status code, which caused incorrect 206 responses for invalid Range header values.

The status is now driven by whether res.ContentRange is set in the response, rather than by the presence of a range in the request. Backends (posix and azure) now set Content-Range for PartNumber=1 on non-multipart objects, skipping zero-size objects where no range applies.

HeadObject was also fixed to return 206 when Content-Range is present, and to only return checksums when the full object is requested.
2026-04-21 02:13:04 +04:00
Ben McClelland 8533bc1b60 Merge pull request #2054 from versity/sis/racing-complete-mp
fix: make CompleteMultipartUpload idempotent and add part-number support to GetObject/HeadObject
2026-04-20 10:23:09 -07:00
niksis02 62e8cddbc7 fix: make CompleteMultipartUpload idempotent and add part-number support to GetObject/HeadObject
Closes #1064

Use the multipart ETag as the in-progress directory suffix instead of the static `.inprogress` marker so that concurrent CompleteMultipartUpload calls for the same upload ID are all treated as successful (idempotent) rather than racing, where only one succeeded and the rest returned NoSuchUpload.

After finalizing the multipart upload, store an `mp-metadata` xattr on the assembled object that records the upload ID and cumulative byte offsets for each part. GetObject and HeadObject now use this metadata to serve individual part ranges via the `partNumber` query parameter, returning a successful response instead of returning NotImplemented.

Add two new S3 error codes:
- `ErrInvalidPartNumberRange` (416 RequestedRangeNotSatisfiable) — returned
  when the requested part number exceeds the number of parts in the upload.
- `ErrRangeAndPartNumber` (400 BadRequest) — returned when both a Range header
  and a partNumber query parameter are specified on the same request.
2026-04-20 20:45:58 +04:00
niksis02 b905554e06 fix: fix azure multipart upload objects masking
As multipart uploads are translated to blobs in azure blob storage, they were visible in ListObjects(V2) as complete objects. Now the blobs with multipart prefix are filtered out during listing.

The listing logic is rewritten client-side to implement proper S3 semantics: flat blob enumeration with manual delimiter handling, correct truncation (IsTruncated only set when more items genuinely exist beyond maxKeys), and StartAfter/Marker/ContinuationToken applied via the lexicographic max of both constraints in ListObjectsV2.

For the same reason bucket deletion was not allowed. Now multipart objects are explicitly checked on bucket deletion and any pending multipart upload doesn't block the bucket deletion anymore.
2026-04-16 23:56:18 +04:00
Ben McClelland 2b918d585e chore: fix spellings and unused function args
No logic changes, just janitorial cleanup
2026-04-15 10:32:14 -07:00
Ben McClelland 1fca33e738 Merge pull request #2006 from versity/ben/racing-put-delete
fix: retry link on ENOENT caused by racing DeleteObject
2026-04-02 09:57:48 -07:00
Ben McClelland 98a186cc4e Merge pull request #2008 from versity/sis/object-post-boundary-prefix
fix: remove POST object multipart boundary prefix trimming
2026-04-02 08:27:02 -07:00
Ben McClelland c26012905c fix: retry link on ENOENT caused by racing DeleteObject
A concurrent PutObject and DeleteObject on the same prefix directory
can race:
PutObject opens an O_TMPFILE in MetaTmpDir (not yet visible in the fs)
DeleteObject removes the last visible object in the prefix directory
and calls removeParents(), which rmdir's the now-empty prefix
directory
PutObject's link() tries to link the fd into a parent directory that
no longer exists

Fix by detecting ENOENT in the final link step (Linkat, Rename, and
MoveFile) and retrying after recreating the parent directory.

Also extract linkatOTmpfile() to consolidate the Linkat+EEXIST→Renameat
logic that was previously inline in link().

Fixes #1988
2026-04-02 08:14:44 -07:00
niksis02 a25408c225 fix: remove POST object multipart boundary prefix trimming
Fixes the [comment](https://github.com/versity/versitygw/issues/1648#issuecomment-4175425099)

Removes the unnecessary multipart/form-data boundary normalizing. The boundary prefix(`--`) was trimmed in `NewMultipartParser`, which caused incorrect boundary check for the boundaries starting with 2 dashes(e.g. `----WebKitFormBoundaryABC123`).
2026-04-02 17:20:23 +04:00
niksis02 052f2364cc feat: implement x-amz-source-expected-bucket-owner for CopyObject and UploadPartCopy
Closes #1897

Extract the `X-Amz-Source-Expected-Bucket-Owner` header for CopyObject and UploadPartCopy. Verify the source bucket owner in the backend and if the provided access key id doesn't match, return an `AccessDenied` error.
2026-04-01 21:44:33 +04:00
Ben McClelland 89ab7a4f2b Merge pull request #1982 from versity/sis/POST-object
feat: add browser-based POST object upload support
2026-03-31 12:25:00 -07:00
niksis02 59002b2650 feat: implement integration tests for browser-based POST object 2026-03-31 22:47:04 +04:00
niksis02 3b17f05d17 feat: support response header overrides in HeadObject
Closes #1967

Add support for response header override query parameters(`response-cache-control`, `response-content-disposition`, `response-content-encoding`, `response-content-language`, `response-content-type`, `response-expires`) in `HeadObject`. Anonymous requests with override params are rejected with `ErrAnonymousResponseHeaders`.
2026-03-31 17:19:03 +04:00
Ben McClelland 13b3dc5267 fix: serialize concurrent CompleteMultipartUpload calls via rename
When two requests raced to complete the same multipart upload, the first
caller to finish would remove the part files and upload directory. The
second caller, already past the initial existence check, would then fail
mid-flight with confusing errors such as ErrInvalidPart or an I/O error
when trying to open a part that no longer exists.

Fix this by atomically renaming the upload directory from <uploadID> to
<uploadID>.inprogress at the very start of CompleteMultipartUploadWithCopy,
before any part data is read. A concurrent caller will now find the
original directory absent and receive a clean NoSuchUpload error. A
deferred rename restores the original name if the complete does not
succeed, allowing the client to retry.

ListMultipartUploads is updated to skip any directories whose name ends
in .inprogress so in-flight completes do not appear as pending uploads.
2026-03-27 16:14:20 -07:00
niksis02 21a636b3b5 fix: add request headers and metadata headers limit
Fixes #1606

According to AWS documentation:
> *“The PUT request header is limited to 8 KB in size. Within the PUT request header, the user-defined metadata is limited to 2 KB in size. The size of user-defined metadata is measured by taking the sum of the number of bytes in the UTF-8 encoding of each key and value.”*

Based on this, object metadata size is now limited to **2 KB** for all object upload operations (`PutObject`, `CopyObject`, and `CreateMultipartUpload`).

Fixes handling of metadata HTTP headers when the same header appears multiple times with different casing or even if they are identical. According to S3 behavior, these headers must be merged into a single lower-cased metadata key, with values concatenated using commas.

Example:

```
x-amz-meta-Key: value1
x-amz-meta-kEy: value2
x-amz-meta-keY: value3
```

Translated to:

```
key: value1,value2,value3
```

This PR also introduces an **8 KB limit for request headers**. Although the S3 documentation explicitly mentions the 8 KB limit only for **PUT requests**, in practice this limit applies to **all requests**.

To enforce the header size limit, the Fiber configuration option `ReadBufferSize` is used. This parameter defines the maximum number of bytes read when parsing an incoming request. Note that this limit does not apply strictly to request headers only, since request parsing also includes other parts of the request line (e.g., the HTTP method, protocol string, and version such as `HTTP/1.1`). So `ReadBufferSize` is effectively a limit for request headers size, but not the exact limit.
2026-03-06 23:25:49 +04:00
niksis02 5c918f3682 feat: revert ignore object ACL behavior
The logic to return a `NotImplemented` error on object upload operations, when any ACL header is present has been removed. Now all object ACL headers are by default ignored. The `-noacl` flag is preserved to disabled bucket ACLs.

**Testing**
The Put/Get object ACL tests are moved to `NotImplemented` integration tests group as a default gateway behavior. The existing `_acl_not_supported` tests are modified to expect no error, when ACLs are used on object uploads.
2026-03-02 19:30:57 +04:00
Ben McClelland da82e5e247 fix: add missing tests to group-tests map
These tests were missing in the tests map to run the individual
tests:
ListBuckets_empty_success
CompleteMultipartUpload_incorrect_part_number
2026-02-28 16:39:43 -08:00
Ben McClelland 0ad928a4d8 Merge pull request #1894 from versity/sis/getobject-directory-object-checksum
feat: adds checksums for directory objects in posix
2026-02-28 09:39:51 -08:00
niksis02 5ae791b154 feat: configuration option to disable ACLs
Closes #1847

This PR introduces a global optional gateway CLI flag `--disable-acl` (`VGW_DISABLE_ACL`) to disable ACL handling. When this flag is enabled, the gateway ignores all ACL-related headers, particularly in `CreateBucket`, `PutObject`, `CopyObject`, and `CreateMultipartUpload`.

`GetBucketAcl` behavior is unchanged simply returning the bucket ACL config.
There's no change in object ACL actions(`PutObjectACL`, `GetObjectACL`). They return a`NotImplemented` error as before.

A new custom error is added for PutBucketAcl calls when ACLs are disabled at the gateway level. Its HTTP status code and error code match AWS S3’s behavior, with only a slightly different error message.

In the access-control checker, ACL evaluation is fully bypassed. If ACLs are disabled only the bucket owner gets access to the bucket and all grantee checks are ignored.

The PR also includes minor refactoring of the S3 API server and router. The growing list of parameters passed to the router’s Init method has been consolidated into fields within the router struct, initialized during router construction. Parameters not needed by the S3 server are no longer stored in the server configuration and are instead forwarded directly to the router.
2026-02-27 20:04:13 +04:00
niksis02 24364754fd feat: adds checksums for directory objects in posix
Add data-integrity checksum support in `PutObject` in the POSIX backend for directory objects. Since the only way to upload a directory object is via `PutObject`, this logic validates and stores the checksum of the empty payload. Support for `GetObject` has also been added to retrieve and return directory-object checksums.
2026-02-26 22:36:56 +04:00
niksis02 6fafc15d08 fix: fixes PutBucketCors CORSRules validation
Fixes #1870
Fixes #1863

A validation has been added to **PutBucketCors** for `CORSRule.AllowedOrigins`. The `AllowedOrigins` list can no longer be empty—otherwise a **MalformedXML** error is returned. Additionally, each origin is now validated to ensure it does not contain more than one wildcard.

A similar validation has been added for `AllowedMethods`. The list must not be empty, or a **MalformedXML** error is returned. Previously, empty method values (e.g., `[]string{""}`) were incorrectly treated as valid. This has been fixed, and an **UnsupportedCORSMethod** error is now returned.
2026-02-24 16:59:38 +04:00
niksis02 7fb3dedecc feat: adds Location, x-amz-bucket-arn response headers in CreateBucket
Closes #1873
2026-02-20 13:02:51 +04:00
Ben McClelland a81f9e5152 Merge pull request #1871 from versity/sis/create-bucket-private-canned-acl
fix: correct private canned ACL behavior on bucket creation
2026-02-18 09:55:36 -08:00
niksis02 f1577fd00b fix: correct private canned ACL behavior on bucket creation
Fixes #1869

Generally, when object ownership is not explicitly specified during bucket creation, it defaults to `BucketOwnerEnforced`. With `BucketOwnerEnforced`, ACLs are disabled and any attempt to set one results in an `InvalidBucketAclWithObjectOwnership` error.

However, there is an edge case. When the `private` canned ACL is used during bucket creation—which is effectively the default ACL for all buckets—`BucketOwnerEnforced` is still permitted. Moreover, if no explicit object ownership is specified together with the `private` canned ACL, the ownership defaults to `BucketOwnerPreferred`.

This fix also resolves the issue with rclone bucket creation, since rclone sends `x-amz-acl: private` by default:

```
rclone mkdir vgw:test
```
2026-02-18 20:32:05 +04:00
niksis02 46bcc8af35 fix: fixes object default Content-Type
Fixes #1849

If no `Content-Type` is provided during object upload, S3 defaults it to `application/octet-stream`. This behavior was missing in the gateway, causing backends to persist an empty `Content-Type`, which Fiber then overrides with its default `text/plain`. The behavior has now been corrected for the object upload operations: `PutObject`, `CreateMultipartUpload`, and `CopyObject`.
2026-02-18 01:44:52 +04:00
Ben McClelland e702a4860a fix: CopyObject with URL-encoded special chars
CopyObject was failing with NoSuchKey when source keys contained special
characters like {} or spaces. The X-Amz-Copy-Source header is URL-encoded
by clients, but ParseCopySource wasn't decoding before filesystem access.

Added url.QueryUnescape() to properly decode bucket and object names,
fixing copy operations for keys with special characters.

Fixing this also uncovered an errors with azure blob url encoding with
similar special character handling. Added this fix in for the integration
tests to pass.

Fixes #1832
Fixes #1637
2026-02-10 14:55:18 -08:00
niksis02 2365f9f1ae fix: fixes list-limiters parsing and validation
Fixes #1809
Fixes #1806
Fixes #1804
Fixes #1794

This PR focuses on correcting so-called "list-limiter" parsing and validation. The affected limiters include: `max-keys`, `max-uploads`, `max-parts`, `max-buckets`, `max-uploads` and `part-number-marker`. When a limiter value is outside the integer range, a specific `InvalidArgument` error is now returned. If the value is a valid integer but negative, a different `InvalidArgument` error is produced.

`max-buckets` has its own validation rules: completely invalid values and values outside the allowed range (`1 <= input <= 10000`) return distinct errors. For `ListObjectVersions`, negative `max-keys` values follow S3’s special-case behavior and return a different `InvalidArgument` error message.

Additionally, `GetObjectAttributes` now follows S3 semantics for `x-amz-max-parts`: S3 ignores invalid values, so the gateway now matches that behavior.
2026-02-06 14:21:56 +04:00
niksis02 2e6794007c feat: adds delimiter support in ListMultipartUploads
Fixes #1792
Fixes #1747
Fixes #1797
Fixes #1799

This PR primarily introduces delimiter support and several bug fixes for the `ListMultipartUploads` action in the POSIX and Azure backends. Delimiter handling is now implemented — when a delimiter is present in multipart-upload object key names, the backend collects and returns the appropriate common prefixes.
This functionality is achieved by introducing a common multipart-upload lister in the backend package. All backends (Azure, POSIX) now use this lister. The lister accepts a list that is already sorted and filtered by `KeyMarker` and `Prefix`.

Previously, the `KeyMarker` was required to exactly match an existing multipart-upload object key. This restriction is removed. The listing now relies on a lexicographical comparison between the provided `KeyMarker` and existing multipart-upload object keys.

Validation for `UploadIdMarker` is also added to correctly return an `InvalidArgument` error for invalid upload IDs. If `KeyMarker` is missing, the `UploadIdMarker` is ignored entirely. If `KeyMarker` is provided, a valid upload ID is one that matches an upload belonging to *the first object key after the KeyMarker*. For example, if the `KeyMarker` is `foo`, but the provided `UploadIdMarker` corresponds to an upload under `quxx`, it is invalid. It must match one of the uploads for the next object key equal to `foo`.

Finally, this PR fixes multipart-upload sorting. Multipart uploads must be sorted primarily lexicographically by their object key, and secondarily—when multiple uploads share the same object key—by their initiation time in ascending order.
2026-02-06 14:16:16 +04:00
niksis02 8569b158f0 fix: return not implemented in object actions, if acl header is present
Fixes #1767
Fixes #1773

As object ACLs are not supported in the gateway, any attempt to set an ACL during object creation must return a NotImplemented error. A check has now been added to `PutObject`, `CopyObject`, and `CreateMultipartUpload` to detect any ACL-related headers and return a NotImplemented error accordingly.
2026-01-23 17:03:03 +04:00
niksis02 86e2b02e55 fix: fixes delete markers access for some actions
Fixes #1766
Fixes #1750

This PR focuses on two bug fixes:

First, it blocks access to delete `DeleteMarkers` for the following operations by returning a `MethodNotAllowed` error: `PutObjectTagging`, `GetObjectTagging`, `DeleteObjectTagging`, `PutObjectLegalHold`, `GetObjectLegalHold`, `PutObjectRetention`, and `GetObjectRetention`.

Second, it removes the access check that previously prevented deleting a delete marker locked by a bucket default retention rule. A delete marker should always be allowed to be deleted.
2026-01-20 16:24:46 +04:00
niksis02 43559e646e fix: fixes non-existing object deletion with versionId
Fixes #1757
Fixes #1758

When attempting to delete a non-existing object in a versioning-enabled bucket while specifying a `versionId`, VersityGW previously returned an internal error if the object had a parent file object, and an `InvalidArgument` error if the object did not exist. This PR fixes both behaviors and now returns a successful response that includes the `versionId`.
2026-01-16 15:00:47 +04:00
niksis02 2a7e76a44f fix: fixes missing bucket object lock config error
Fixes #1751

When an object lock–related operation is performed on an object in a bucket where Object Lock is not enabled, an `InvalidRequest` error is returned; however, the error message differs for some actions. This PR introduces a new error, `ErrMissingObjectLockConfigurationNoSpaces`, for `PutObject`, `CopyObject`, and `CreateMultipartUpload` to maintain compatibility with S3 in terms of the error message. It also adds the missing integration tests for these actions.
2026-01-14 13:41:50 +04:00
niksis02 06f4f0ac15 fix: skips object lock check in DeleteObject without versionId.
Fixes #1741

An object delete request without a `versionId` results in the creation of a new delete marker in versioning-enabled buckets. Even if the latest object version is locked, a new delete marker must still be created.

This implementation skips the object lock check for delete requests in versioning-enabled buckets when the `versionId` is missing, allowing the delete marker to be created as expected.

Additionally, it introduces a flag in the `createObjVersion` method in POSIX to remove unnecessary xattr attributes from an object after creating a new object version. A delete marker must not carry object-specific attributes such as tagging, legal hold, or retention. Currently, the cleanup is limited to legal hold and retention attributes, but this list will be expanded after fixing issue #1751.
2026-01-13 16:50:54 +04:00