The AWS spec for the create multipart upload response is:
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult>
<Bucket>string</Bucket>
<Key>string</Key>
<UploadId>string</UploadId>
</InitiateMultipartUploadResult>
So we need the return type to marshal to this xml format.
The container URL was adding an extra / when the service URL ended
with /. This results in an invalid container URI error from Azure.
This also fixes the container metadata keys to not include "-" that
the Azure service disallows. The metadata values are base64 encoded
to prevent any special chacter handling on Azure service side.
The content type is set for uploads consistent with S3 expectations.
Fixes#736
In copy-object, if the source and destination are the same then
X-Amz-Metadata-Directive must be set to "REPLACE" in order to use
this api call to update the metadata of the object in place.
The default X-Amz-Metadata-Directive is "COPY" if not specified.
"COPY" is only valid if source and destination are not the same
object.
When "REPLACE" selected, metadata does not have to differ for the
call to be successful. The "REPLACE" always sets the incoming
metadata (even if empty or the same as the source).
Fixes#734
The expected response code for deleting non-existing objects is
expected to be 204 (No Content) instead of NoSuchKey. The tests
are updated to validate expected responses.
Fixes#724
The API hanlders and backend were stripping trailing "/" in object
paths. So if an object exists and a request came in for head/get/delete/copy
for that same name but with a trailing "/" indicating request should
be for directory object, the "/" would be stripped and the request
would be handlied for the incorrect file object.
This fix adds in checks to handle the case with the training "/"
in the request.
Fixes#709
For large directories, the treewalk can take longer than the
client request timeout. If the client times out the request
then we need to stop walking the filesystem and just return
the context error.
This should prevent the gateway from consuming system resources
uneccessarily after an incoming request is terminated.
This adds the ability to treat symlinks to directories at the top
level gateway directory as buckets the same as normal directories.
This could be a potential security issue allowing traversal into
other filesystems within the system, so is defaulted to off. This
can be enabled when specifically needed for both posix and scoutfs
backend systems.
Fixes#644
Bucket ACLs are now disabled by default the same as AWS.
By default the object ownership is BucketOwnerEnforced
which means that bucket ACLs are disabled. If one attempts
to set bucket ACL the following error is returned both in
the gateway and on AWS:
ErrAclNotSupported: {
Code: "AccessControlListNotSupported",
Description: "The bucket does not allow ACLs",
HTTPStatusCode: http.StatusBadRequest,
},
ACls can be enabled with PutBucketOwnershipControls
Changed bucket canned ACL translation
New backend interface methods:
PutBucketOwnershipControls
GetBucketOwnershipControls
DeleteBucketOwnershipControls
Added these to metrics
In the posix backend, the path argument to os.RemoveAll() does not
start with the bucket name. Since the path does not exist,
os.RemoveAll() does nothing and uploaded parts are left in the
".sgwtmp" directory.
This commit prefixes the path with the bucket name.
Fixes issue #626.
Signed-off-by: Christophe Vu-Brugier <christophe.vu-brugier@seagate.com>
The restore object api request handler was incorrectly trying to
unmarshal the request body, but for the stadnard (all?) case the
request body is emtpy. We only need the bucket and opbject params
for now.
This also adds a fix to actually honor the enable glacier mode
in scoutfs.
Add meta.MetadataStorer compatibility to scoutfs so that scoutfs
is using the same interface as posix. This fixes the metadata
retrieval and adds the recently supported object lock compatibility
as well.
The complete multipart upload can be optimized in some cases
to not need to copy the full data from parts to the final
object file. If the filesystem supports it, there can be
optimizations to just clone exent references and not have to
actually copy the data.
For io.Copy() to make use of file_copy_range, we have to pass it
the *os.File file handles from the filesystem for the source and
destination. We were previously adding a layer of indirection
that was causing io.Copy() to fallback to full data copy. This
fixes the copy by exposing the underlying fd.
This also skips the falloc for the final object in complete
mutlipart upload, because some filesystems will be able to use
file_copy_range to optimize the copy and may not even need
new data allocations in the final object file.
Note that this only affects posix mode as scoutfs has special
handling for this case that is similar to but not compatible
with copy_file_range using a special ioctl.
Added support to enable object lock on bucket creation in posix and azure
backends.
Implemented the logic to add object legal hold and retention on object creation
in azure and posix backends.
Added the functionality for HeadObject to return object lock related headers.
Added integration tests for these features.
The xattr package has a more universal error type for xattrs
not existing. Use this for better platform compatibility.
This also adds the xattr.XATTR_SUPPORTED check for platform
xattr suport in xattr package.
Fixes#527
When fileystem quota exceeded, the gateway will now return the
error:
S3 error: 403 (QuotaExceeded):
Your request was denied due to quota exceeded.
This will help clients to better detect upload errors due to
quota exceeded.
Fixes#483
