This checks to see if the common prefix is before the marker and
thus would have been returned in earlier list objects request.
The error case was aws cli listing multiple entries for the same
common prefix when the listing required multiple pagination
requests.
Fixes#778
This fixes put object with setting a content type. If no content
type is set, then we will return a default content type for
following requests. Mutli-part upload appears to be ok.
Also fixed content eincoding and multipart uploads.
Fixes#783
When using the REPLACE directive, we were incorrectly removing the
old metadata on the object due to missing the metadata prefix on
the key. Fix this to remove the correct metadata before setting
new metadata.
Fixes#787
The "x-amz-copy-source" header may start with '/' as observed with
WinSCP. However, '/' is also the separator between the bucket and the
object path in "x-amz-copy-source".
Consider the following code in VerifyObjectCopyAccess():
srcBucket, srcObject, found := strings.Cut(copySource, "/")
If `copySource` starts with '/', then `srcBucket` is set to an empty
string. Later, an error is returned because bucket "" does not exist.
This issue was fixed in the Posix and Azure backends by the following
commit:
* 5e484f2 fix: Fixed CopySource parsing to handle the values starting with '/' in CopyObject action in posix and azure backends.
But the issue was not fixed in `VerifyObjectCopyAccess`.
This commit sanitizes "x-amz-copy-source" right after the header is
extracted in `s3api/controllers/base.go`. This ensures that the
`CopySource` argument passed to the backend functions UploadPartCopy()
and CopyObject() does not start with '/'. Since the backends no longer
need to strip away any leading '/' in `CopySource`, the parts of
commit 5e484f2 modifying the Posix and Azure backends are reverted.
Fixes issue #773.
Signed-off-by: Christophe Vu-Brugier <christophe.vu-brugier@seagate.com>
It is better if we let the s3response module handle the xml
formatting spec specifics, and let the backends not worry
about how to format the time fields. This should help to
prevent any future backend modifications or additions from
accidental incorrect time formatting.
Changed ListObjectsV2 and ListObjects actions return types from
*s3.ListObjects(V2)Output to s3response.ListObjects(V2)Result.
Changed the listing objects timestamp to RFC3339 to match AWS
S3 objects timestamp.
Fixes#752
The posix limits wont exactly match up with the AWS key length
limits because posix has component length limits as well as path
length limits.
This reponds with the aws compatible KeyTooLongError under these
conditions now.
Note that delete object returns success even in the error cases.
Fixes#755
The admin/root user has access to all buckets. So we don't need a
valid ACL defined on the bucket to list these. Treat a non-existing
ACL as an empty ACL for this case.
Fixes#727
The AWS spec for the create multipart upload response is:
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult>
<Bucket>string</Bucket>
<Key>string</Key>
<UploadId>string</UploadId>
</InitiateMultipartUploadResult>
So we need the return type to marshal to this xml format.
The container URL was adding an extra / when the service URL ended
with /. This results in an invalid container URI error from Azure.
This also fixes the container metadata keys to not include "-" that
the Azure service disallows. The metadata values are base64 encoded
to prevent any special chacter handling on Azure service side.
The content type is set for uploads consistent with S3 expectations.
Fixes#736
In copy-object, if the source and destination are the same then
X-Amz-Metadata-Directive must be set to "REPLACE" in order to use
this api call to update the metadata of the object in place.
The default X-Amz-Metadata-Directive is "COPY" if not specified.
"COPY" is only valid if source and destination are not the same
object.
When "REPLACE" selected, metadata does not have to differ for the
call to be successful. The "REPLACE" always sets the incoming
metadata (even if empty or the same as the source).
Fixes#734
The expected response code for deleting non-existing objects is
expected to be 204 (No Content) instead of NoSuchKey. The tests
are updated to validate expected responses.
Fixes#724
The API hanlders and backend were stripping trailing "/" in object
paths. So if an object exists and a request came in for head/get/delete/copy
for that same name but with a trailing "/" indicating request should
be for directory object, the "/" would be stripped and the request
would be handlied for the incorrect file object.
This fix adds in checks to handle the case with the training "/"
in the request.
Fixes#709
For large directories, the treewalk can take longer than the
client request timeout. If the client times out the request
then we need to stop walking the filesystem and just return
the context error.
This should prevent the gateway from consuming system resources
uneccessarily after an incoming request is terminated.
This adds the ability to treat symlinks to directories at the top
level gateway directory as buckets the same as normal directories.
This could be a potential security issue allowing traversal into
other filesystems within the system, so is defaulted to off. This
can be enabled when specifically needed for both posix and scoutfs
backend systems.
Fixes#644
Bucket ACLs are now disabled by default the same as AWS.
By default the object ownership is BucketOwnerEnforced
which means that bucket ACLs are disabled. If one attempts
to set bucket ACL the following error is returned both in
the gateway and on AWS:
ErrAclNotSupported: {
Code: "AccessControlListNotSupported",
Description: "The bucket does not allow ACLs",
HTTPStatusCode: http.StatusBadRequest,
},
ACls can be enabled with PutBucketOwnershipControls
Changed bucket canned ACL translation
New backend interface methods:
PutBucketOwnershipControls
GetBucketOwnershipControls
DeleteBucketOwnershipControls
Added these to metrics
In the posix backend, the path argument to os.RemoveAll() does not
start with the bucket name. Since the path does not exist,
os.RemoveAll() does nothing and uploaded parts are left in the
".sgwtmp" directory.
This commit prefixes the path with the bucket name.
Fixes issue #626.
Signed-off-by: Christophe Vu-Brugier <christophe.vu-brugier@seagate.com>
The restore object api request handler was incorrectly trying to
unmarshal the request body, but for the stadnard (all?) case the
request body is emtpy. We only need the bucket and opbject params
for now.
This also adds a fix to actually honor the enable glacier mode
in scoutfs.
Add meta.MetadataStorer compatibility to scoutfs so that scoutfs
is using the same interface as posix. This fixes the metadata
retrieval and adds the recently supported object lock compatibility
as well.
