This change adds Windows functional test execution in CI and updates
backend handling so windows filesystem error/path semantics map correctly
to expected S3 outcomes.
The only meta supported on windows right now is sidecar, so the tests
in windows mode also skip sidecar skips.
Future work is to address the skips and/or more clearly document
the unsupported/incompatible behavior on windows.
The windows support will still remain best effort, but these tests
should at least flag when future changes introduce incompatible
behavior on windows.
Enhances the static website hosting implementation with more complete S3-compatible behavior across request handling, backend storage, validation, CORS, and errors.
Adds dedicated website endpoint handling for GET, HEAD, and OPTIONS requests, including index document resolution, error document serving, redirect-all support, pre-fetch and post-error routing rules, query string preservation in redirects, public access checks before object reads, and method-not-allowed responses.
Improves error handling for website responses by returning S3-compatible HTML error bodies with request IDs, host IDs, x-amz-error-code, x-amz-error-message, and specialized error fields. This also fixes website-related validation errors to return more accurate S3-style error codes and messages, including invalid redirect protocols, invalid HTTP redirect/error codes, conflicting routing rule replacements, routing rule limits, and oversized website configuration requests.
Adds website CORS support for GET, HEAD, and OPTIONS preflight requests, including bucket CORS lookup through website host bucket resolution, allowed origin/method/header validation, exposed header handling, ETag exposure, Vary headers, max-age handling, and CORS access-denied responses.
Adds debug logging around website configuration parsing, validation failures, CORS checks, backend lookup failures, and internal website error paths to make failures easier to diagnose.
Adds compressed website configuration storage so larger configs fit backend metadata limits, including gzip storage for POSIX extended attributes and base64-encoded compressed metadata for Azure. Also adds Azure PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite support.
Adds and expands test coverage for website config validation, S3-compatible HTML error bodies, website routing behavior, public access enforcement, HEAD behavior, CORS handling, PutBucketWebsite limits, and end-to-end website hosting through a Docker-based dnsmasq test setup and CI workflow.
Replace PutBucketWebsite, GetBucketWebsite, DeleteBucketWebsite
NotImplemented test stubs with comprehensive integration tests covering:
- non-existing bucket errors
- validation (empty suffix, suffix with slash, invalid protocol, mutual
exclusion of RedirectAllRequestsTo and IndexDocument)
- successful put/get round-trips for both index+error and redirect-all configs
- delete idempotency and verification
Signed-off-by: Marc Singer <marc@singer.gg>
Add error document serving, routing rules, and integration tests
Implement Features 1 and 2 of S3 static website hosting:
- WebsiteErrorDocument controller wrapper intercepts 4xx errors on
website-enabled buckets and serves the configured error document or
evaluates post-request routing rules (error code match redirects)
- ResolveWebsiteIndex middleware now caches parsed WebsiteConfiguration
in context, handles RedirectAllRequestsTo, evaluates pre-request
routing rules (key prefix match redirects), and rewrites directory
keys for index document
- MatchPreRequestRule and MatchPostRequestRule methods on
WebsiteConfiguration for routing rule evaluation
- 14 unit tests for routing rule matching
- 7 integration tests covering error document, routing rules,
redirect-all, and index document behavior
Signed-off-by: Marc Singer <marc@singer.gg>
Add separate website hosting endpoint with virtual-host routing
Signed-off-by: Marc Singer <marc@singer.gg>
Support catch-all mode for website endpoint when --website-domain is omitted
Signed-off-by: Marc Singer <marc@singer.gg>
The latest version of the go sdk has deprecate the s3 manager in
favor of the new transfermanager. This updates to the new
transfermanager functionality for the two places we were using
the manager, iam s3 and integration tests.
This also removes the pinning of nats nkeys since they have fixed
the tags in their repo now. And general cleanup of the go.mod.
Fixes#1385
When accessing a specific object version, the user must have the `s3:GetObjectVersion` permission in the bucket policy. The `s3:GetObject` permission alone is not sufficient for a regular user to query object versions using `HeadObject`.
This PR fixes the issue and adds integration tests for both `HeadObject` and `GetObject`. It also includes cleanup in the integration tests by refactoring the creation of user S3 clients, and moves some test user data to the package level to avoid repetition across tests.
This implementation introduces **public buckets**, which are accessible without signature-based authentication.
There are two ways to grant public access to a bucket:
* **Bucket ACLs**
* **Bucket Policies**
Only `Get` and `List` operations are permitted on public buckets. All **write operations** require authentication, regardless of whether public access is granted through an ACL or a policy.
The implementation includes an `AuthorizePublicBucketAccess` middleware, which checks if public access has been granted to the bucket. If so, authentication middlewares are skipped. For unauthenticated requests, appropriate errors are returned based on the specific S3 action.
---
**1. Bucket-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test"
}
]
}
```
**2. Object-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test/*"
}
]
}
```
**3. Both Bucket and Object-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test"
},
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test/*"
}
]
}
```
---
```sh
aws s3api create-bucket --bucket test --object-ownership BucketOwnerPreferred
aws s3api put-bucket-acl --bucket test --acl public-read
```
Closes#803
Implements host-style bucket addressing in the gateway. This feature can be enabled by running the gateway with the `--virtual-domain` flag and specifying a virtual domain name.
Example:
```bash
./versitygw -a user -s secret --virtual-domain localhost:7070 posix /tmp/vgw
```
The implementation follows this approach: it introduces a middleware (`HostStyleParser`) that parses the bucket name from the `Host` header and appends it to the URL path. This effectively transforms the request into a path-style bucket addressing format, which the gateway already supports. With this design, the gateway can handle both path-style and host-style requests when running in host-style mode.
For local testing, one can either set up a local DNS server to wildcard-match all subdomains of a specified domain and resolve them to the local IP address, or manually add entries to `/etc/hosts` to resolve bucket-prefixed hosts to the server IP (e.g., `127.0.0.1`).
This fixes the cases for racing uploads with the same object names.
Before we were making some bad assumptions about what would cause
an error when trying to link/rename the final object name into
the namespace, but missed the case that another upload for the
same name could be racing with this upload and causing an incorrect
error.
This also changes the order of setting metadata to prevent
accidental setting of metadata for the current upload to another
racing upload.
This also fix auth.CheckObjectAccess() when objects are removed
while this runs.
Fixes#854
Ben McClelland
niksis02
Marc Singer
jonaustin09