The error check for the SDK call in `GetObjectAttributes` in the S3 proxy backend was missing, which caused the gateway to panic in all cases where the SDK method returned an error. The error check has now been added so that the method returns an error when the SDK call fails.
Fixes#1649
`GetBucketVersioning` used to be a cause of a panic in s3 proxy backend, because of an inproper error handling. Now the error returned from the sdk method is explitily checked, before returning the response.
Closes#1343
Object version tagging support was previously missing in the gateway. The support is added with this PR. If versioning is not enabled at the gateway level and a user attempts to put, get, or delete object version tags, the gateway returns an `InvalidArgument`(Invalid versionId)
Fixes#1559Fixes#1330
This PR focuses on three main changes:
1. **Fix object lock error codes and descriptions**
When an object was WORM-protected and delete/overwrite was disallowed due to object lock configurations, the gateway incorrectly returned the `s3.ErrObjectLocked` error code and description. These have now been corrected.
2. **Update `PutObjectRetention` behavior**
Previously, when an object already had a retention mode set, the gateway only allowed modifications if the mode was changed from `GOVERNANCE` to `COMPLIANCE`, and only when the user had the `s3:BypassGovernanceRetention` permission.
The logic has been updated: if the existing retention mode is the same as the one being applied, the operation is now allowed regardless of other factors.
3. **Fix error checks in integration tests (AWS SDK regression)**
Due to an AWS SDK regression, integration tests were previously limited to checking partial error descriptions. This issue seems to be resolved for some actions (though the ticket is still open: https://github.com/aws/aws-sdk-go-v2/issues/2921). Error checks have been reverted back to full description comparisons where possible.
Closes#1518
Adds the `x-amz-object-size` header to the `PutObject` response, indicating the size of the uploaded object. This change is applied to the POSIX, Azure, and S3 proxy backends.
The CORS actions were directly proxied in s3 proxy backend. The new implementation stores/retreives/deletes bucket cors configuration in `meta` bucket.
Closes#1003
**Changes Introduced:**
1. **S3 Bucket CORS Actions**
* Implemented the following S3 bucket CORS APIs:
* `PutBucketCors` – Configure CORS rules for a bucket.
* `GetBucketCors` – Retrieve the current CORS configuration for a bucket.
* `DeleteBucketCors` – Remove CORS configuration from a bucket.
2. **CORS Preflight Handling**
* Added an `OPTIONS` endpoint to handle browser preflight requests.
* The endpoint evaluates incoming requests against bucket CORS rules and returns the appropriate `Access-Control-*` headers.
3. **CORS Middleware**
* Implemented middleware that:
* Checks if a bucket has CORS configured.
* Detects the `Origin` header in the request.
* Adds the necessary `Access-Control-*` headers to the response when the request matches the bucket CORS configuration.
Add helper util auth.UpdateBucketACLOwner() that sets new
default ACL based on new owner and removes old bucket policy.
The ChangeBucketOwner() remains in the backend.Backend
interface in case there is ever a backend that needs to manage
ownership in some other way than with bucket ACLs. The arguments
are changing to clarify the updated owner. This will break any
plugins implementing the old interface. They should use the new
auth.UpdateBucketACLOwner() or implement the corresponding
change specific for the backend.
We were incorrctly trying to pass through the admin request
actions through to the backend s3 service in s3proxy. This
was resulting in internal server errors since not all s3
backends would understand these requests. Instead the
gateway needs to handle these requests directly.
Fixes#1381
The http body stream is not a seekable stream, so most operation
retry attempts will fail with an internal server error. This
change tells the s3 client within the gateway to not retry any
requests, and instead let the client of the gateway handle the
error retry.
Fixes#1353
We were using the metadata retrieval to check for existing
buckets during create, and then return either BucketAlreadyExists
or ErrBucketAlreadyOwnedByYou accordingly.
Howver, the metadata retrieval was returning success with a
default ACL when the bucket metadata did not already exist
causing the gateway to always think this bucket existed.
Fix here is to let the metadata retrieval know that we do not
want the default ACL for this case.
This fixes the listing of buckets when multi tenant mode is
enabled with a metadata bucket. The following behavior changes
are fixed:
* prevent listing of metadata bucket by all accounts
* prevent listing of non-owned buckets by user/userplus
* return correct BucketAlreadyExists/BucketAlreadyOwnedByYou
for attempts to create existing bucket
Fixes#1326
There were a couple of cases that would return an error for the
non existing bucket acl instead of treating that as the default
acl.
This also cleans up the backends that were doing their own
acl parsing instead of using the auth.ParseACL() function.
Fixes#1304
Fixes#1276
Creates the custom `s3response.CopyObjectOutput` type to handle the `LastModified` date property formatting correctly. It uses `time.RFC3339` to format the date to match the format that s3 uses.
The xml encoding for the s3.CompleteMultipartUploadOutput response
type was not producing exactly the right field names for the
expected complete multipart upload result.
This change follows the pattern we have had to do for other xml
responses to create our own type that will encode better to the
expected response.
This will change the backend.Backend interface, so plugins and
other backends will have to make the corresponding changes.
Closes#1111
Bucket ACLs and policies are now stored in the meta bucket as objects with the following prefixes:
- `vgw-meta-acl-<bucket-name>`
- `vgw-meta-policy-<bucket-name>`
The name of the meta bucket is provided during S3 proxy initialization. The gateway verifies whether the specified bucket exists; if it does not, an error is returned.
If no meta bucket is provided, the S3 proxy returns default values for ACL and policy actions.
Fixes#998Closes#1125Closes#1126Closes#1127
Implements objects meta properties(Content-Disposition, Content-Language, Content-Encoding, Cache-Control, Expires) and tagging besed on the directives(metadata, tagging) in CopyObject in posix and azure backends. The properties/tagging should be coppied from the source object if "COPY" directive is provided and it should be replaced otherwise.
Changes the object copy principle in azure: instead of using the `CopyFromURL` method from azure sdk, it first loads the object then creates one, to be able to compare and store the meta properties.
Closes#1128
Adds `Content-Disposition`, `Content-Language`, `Cache-Control` and `Expires` object meta properties support in posix and azure backends.
Changes the `PutObject` and `CreateMultipartUpload` actions backend input type to custom `s3response` types to be able to store `Expires` as any string.
We currently store bucket ACLs as tags in the backend S3 service.
Some backend services do not implment tags though. In this case,
we need to return the default bucket ACL for some continued
functionality.
There is still more work to return more correct errors for setting
ACLs when this is not implemented.
The frontend for versitygw wants to manage its own object
retention, and these settings are conflicting with trying
to forward object lock requests to the backend s3 server.
This disables object lock for the s3 backend for now until
we can properly implement this for the backend. We will
likely just need to add metadata to backend buckets and
objects for our object retention settings.
For any uninitialized input values, the s3 sdk expects these to
be set to nil instead of the type zero value. Setting to the zero
value is incorrectly making the server request think we are
trying to set those parameters.
This sets all input parameters to nil for the default values. Many
of these are already nil from the frontend, but this should
protect against a future addition of input params for an api
requests and not remembering to add teh zero value check on in
the s3 backend.
We were previously getting the following error for put-object:
<Error><Code>InvalidArgument</Code><Message>x-amz-object-lock-retain-until-date and x-amz-object-lock-mode must both be supplied</Message><Resource></Resource><RequestId></RequestId><HostId></HostId></Error>
Updated the convertObjects function to initialize the result slice
with a length of 0 and a capacity equal to the length of the input
slice (objs).
This fixes the problem where the results were being appended to
a slice that already contained 0 value entries for the
preallocated length.
The s3 sdk expects the version id and other input options to be
nil when not specified. Otherwise it incorrectly accepts the ""
string as the requested input option.
This just sets the version id and otehr options back to nil if
its "" from the s3api controller.
For s3 proxy backend, the ACLs are stored in bucket tags to not
conflict with the underlying s3 object store ACLs. If the tag
for the gatway ACLs does not exist for the bucket, then we were
incorrectly returning the NoSuchTagSet from teh tag lookup.
Instead, in this case we need to just return the default gateway
bucket ACL, which is the root account ownership.
Side note that there is currently a bug in the go sdk where the
NoSuchTagSet is not defained as an error type, and a workaround
for string matching is required until this is addressed.
https://github.com/aws/aws-sdk-go-v2/issues/2878
It is better if we let the s3response module handle the xml
formatting spec specifics, and let the backends not worry
about how to format the time fields. This should help to
prevent any future backend modifications or additions from
accidental incorrect time formatting.
Changed ListObjectsV2 and ListObjects actions return types from
*s3.ListObjects(V2)Output to s3response.ListObjects(V2)Result.
Changed the listing objects timestamp to RFC3339 to match AWS
S3 objects timestamp.
Fixes#752
The AWS spec for the create multipart upload response is:
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult>
<Bucket>string</Bucket>
<Key>string</Key>
<UploadId>string</UploadId>
</InitiateMultipartUploadResult>
So we need the return type to marshal to this xml format.
Bucket ACLs are now disabled by default the same as AWS.
By default the object ownership is BucketOwnerEnforced
which means that bucket ACLs are disabled. If one attempts
to set bucket ACL the following error is returned both in
the gateway and on AWS:
ErrAclNotSupported: {
Code: "AccessControlListNotSupported",
Description: "The bucket does not allow ACLs",
HTTPStatusCode: http.StatusBadRequest,
},
ACls can be enabled with PutBucketOwnershipControls
Changed bucket canned ACL translation
New backend interface methods:
PutBucketOwnershipControls
GetBucketOwnershipControls
DeleteBucketOwnershipControls
Added these to metrics
This adds the default ACL to the CreateBucket backend method so
that the backend doesn't need to know how to construct and ACL.
This also moves the s3proxy ACLs to a tag key/value because the
gateway ACLs are not the same accounts as the backend s3 server.
TODO: we may need to mask this tag key/value if we add support
for the Get/PutBucketTagging API.
