Add S3 bucket website configuration types with XML serialization support
in s3response/website.go. Includes IndexDocument, ErrorDocument,
RedirectAllRequestsTo, and RoutingRules with full validation matching
AWS S3 behavior.
Add corresponding S3 error codes: ErrNoSuchWebsiteConfiguration,
ErrInvalidWebsiteConfiguration, ErrInvalidWebsiteSuffix, and
ErrInvalidWebsiteRedirectCode.
Unit tests cover validation logic, XML round-trip, and parsing.
Signed-off-by: Marc Singer <marc@singer.gg>
Add website backend interface and implementations for posix and s3proxy
Add PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite methods
to the Backend interface with BackendUnsupported stubs that return
ErrNotImplemented.
Posix backend stores website config as a metadata attribute (key:
'website') following the same pattern as CORS. ScoutFS inherits via
embedding.
S3Proxy backend stores website config in the metadata bucket with
prefix 'vgw-meta-website-', consistent with existing ACL/policy/CORS
metadata storage. Returns ErrNoSuchWebsiteConfiguration when not found.
Signed-off-by: Marc Singer <marc@singer.gg>
Add website API controllers and wire into router
Add PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite
controller methods following the same pattern as CORS. Controllers
parse and validate WebsiteConfiguration XML, check IAM authorization,
and delegate to the backend.
Replace the three HandleErrorRoute(ErrNotImplemented) stubs in the
router with the new controller methods. Regenerate the backend mock
to include the new interface methods.
Signed-off-by: Marc Singer <marc@singer.gg>
Add website index document middleware and wire into router
Add ResolveWebsiteIndex middleware that rewrites directory-like object keys
(empty or ending with /) to include the IndexDocument suffix when website
hosting is enabled. Also handles RedirectAllRequestsTo by returning 301.
Wire the middleware into both GetObject and HeadObject handler chains in
the router, positioned after BucketObjectNameValidator and before auth.
Signed-off-by: Marc Singer <marc@singer.gg>
Object key validation allowed internal parent-directory segments such as `public/../private.txt`. Bucket policy and auth checks evaluated the raw key, so a policy allowing bucket/public/* could match the request while posix backend later resolved the key with `filepath.Join` and accessed `bucket/private.txt`.
Add backend-specific object key normalization to close that mismatch. The Backend interface now exposes `NormalizeObjectKey` so authorization can evaluate resources using the same key shape a backend will use for storage access.
Backends that do not collapse object paths, including Azure and the S3 proxy, inherit `BackendUnsupported.NormalizeObjectKey`. That implementation returns the input key unchanged, avoiding unnecessary normalization and keeping policy evaluation unpolluted for object stores where ../ is part of the key name.
posix/scoutfs normalize keys with filepath.Join so policy resources and request keys are compared after internal dot segments are collapsed.
Bucket policy evaluation now normalizes both the incoming object key and object resource patterns from the policy before matching. Object lock governance bypass policy checks use the same backend normalizer as well, so retention and legal hold authorization cannot diverge from backend path resolution.
Closes#1343
Object version tagging support was previously missing in the gateway. The support is added with this PR. If versioning is not enabled at the gateway level and a user attempts to put, get, or delete object version tags, the gateway returns an `InvalidArgument`(Invalid versionId)
Fixes#1559Fixes#1330
This PR focuses on three main changes:
1. **Fix object lock error codes and descriptions**
When an object was WORM-protected and delete/overwrite was disallowed due to object lock configurations, the gateway incorrectly returned the `s3.ErrObjectLocked` error code and description. These have now been corrected.
2. **Update `PutObjectRetention` behavior**
Previously, when an object already had a retention mode set, the gateway only allowed modifications if the mode was changed from `GOVERNANCE` to `COMPLIANCE`, and only when the user had the `s3:BypassGovernanceRetention` permission.
The logic has been updated: if the existing retention mode is the same as the one being applied, the operation is now allowed regardless of other factors.
3. **Fix error checks in integration tests (AWS SDK regression)**
Due to an AWS SDK regression, integration tests were previously limited to checking partial error descriptions. This issue seems to be resolved for some actions (though the ticket is still open: https://github.com/aws/aws-sdk-go-v2/issues/2921). Error checks have been reverted back to full description comparisons where possible.
Closes#1003
**Changes Introduced:**
1. **S3 Bucket CORS Actions**
* Implemented the following S3 bucket CORS APIs:
* `PutBucketCors` – Configure CORS rules for a bucket.
* `GetBucketCors` – Retrieve the current CORS configuration for a bucket.
* `DeleteBucketCors` – Remove CORS configuration from a bucket.
2. **CORS Preflight Handling**
* Added an `OPTIONS` endpoint to handle browser preflight requests.
* The endpoint evaluates incoming requests against bucket CORS rules and returns the appropriate `Access-Control-*` headers.
3. **CORS Middleware**
* Implemented middleware that:
* Checks if a bucket has CORS configured.
* Detects the `Origin` header in the request.
* Adds the necessary `Access-Control-*` headers to the response when the request matches the bucket CORS configuration.
Add helper util auth.UpdateBucketACLOwner() that sets new
default ACL based on new owner and removes old bucket policy.
The ChangeBucketOwner() remains in the backend.Backend
interface in case there is ever a backend that needs to manage
ownership in some other way than with bucket ACLs. The arguments
are changing to clarify the updated owner. This will break any
plugins implementing the old interface. They should use the new
auth.UpdateBucketACLOwner() or implement the corresponding
change specific for the backend.
Fixes#1276
Creates the custom `s3response.CopyObjectOutput` type to handle the `LastModified` date property formatting correctly. It uses `time.RFC3339` to format the date to match the format that s3 uses.
The xml encoding for the s3.CompleteMultipartUploadOutput response
type was not producing exactly the right field names for the
expected complete multipart upload result.
This change follows the pattern we have had to do for other xml
responses to create our own type that will encode better to the
expected response.
This will change the backend.Backend interface, so plugins and
other backends will have to make the corresponding changes.
Implements the bucket cors s3 actions in FE to return `NotImplemented` error.
Actions implemented:
- `PutBucketCors`
- `GetBucketCors`
- `DeleteBucketCors`
`Note`: no logic is implemented for the actions in any backend and no input or output data validation is added.
Fixes#998Closes#1125Closes#1126Closes#1127
Implements objects meta properties(Content-Disposition, Content-Language, Content-Encoding, Cache-Control, Expires) and tagging besed on the directives(metadata, tagging) in CopyObject in posix and azure backends. The properties/tagging should be coppied from the source object if "COPY" directive is provided and it should be replaced otherwise.
Changes the object copy principle in azure: instead of using the `CopyFromURL` method from azure sdk, it first loads the object then creates one, to be able to compare and store the meta properties.
Closes#1128
Adds `Content-Disposition`, `Content-Language`, `Cache-Control` and `Expires` object meta properties support in posix and azure backends.
Changes the `PutObject` and `CreateMultipartUpload` actions backend input type to custom `s3response` types to be able to store `Expires` as any string.
Changed ListObjectsV2 and ListObjects actions return types from
*s3.ListObjects(V2)Output to s3response.ListObjects(V2)Result.
Changed the listing objects timestamp to RFC3339 to match AWS
S3 objects timestamp.
Fixes#752
The AWS spec for the create multipart upload response is:
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult>
<Bucket>string</Bucket>
<Key>string</Key>
<UploadId>string</UploadId>
</InitiateMultipartUploadResult>
So we need the return type to marshal to this xml format.
Bucket ACLs are now disabled by default the same as AWS.
By default the object ownership is BucketOwnerEnforced
which means that bucket ACLs are disabled. If one attempts
to set bucket ACL the following error is returned both in
the gateway and on AWS:
ErrAclNotSupported: {
Code: "AccessControlListNotSupported",
Description: "The bucket does not allow ACLs",
HTTPStatusCode: http.StatusBadRequest,
},
ACls can be enabled with PutBucketOwnershipControls
Changed bucket canned ACL translation
New backend interface methods:
PutBucketOwnershipControls
GetBucketOwnershipControls
DeleteBucketOwnershipControls
Added these to metrics
This adds the default ACL to the CreateBucket backend method so
that the backend doesn't need to know how to construct and ACL.
This also moves the s3proxy ACLs to a tag key/value because the
gateway ACLs are not the same accounts as the backend s3 server.
TODO: we may need to mask this tag key/value if we add support
for the Get/PutBucketTagging API.
This adds a context to the backend interface calls so that the backend
can enable request cancellation. This change isn't acutally implementing
any backend handling, but just putting the pieces into place to pass the
context to the backend.
The xml encoding of the s3.ListBucketsOutput return type was not giving
correct results when there is only a single bucket. This revives the
old aws xsd schema and generated types that will give more accurate xml
encoding results.
Marc Singer
niksis02
Ben McClelland
jonaustin09
Jon Austin