For large directories, the treewalk can take longer than the
client request timeout. If the client times out the request
then we need to stop walking the filesystem and just return
the context error.
This should prevent the gateway from consuming system resources
uneccessarily after an incoming request is terminated.
This adds the ability to treat symlinks to directories at the top
level gateway directory as buckets the same as normal directories.
This could be a potential security issue allowing traversal into
other filesystems within the system, so is defaulted to off. This
can be enabled when specifically needed for both posix and scoutfs
backend systems.
Fixes#644
Bucket ACLs are now disabled by default the same as AWS.
By default the object ownership is BucketOwnerEnforced
which means that bucket ACLs are disabled. If one attempts
to set bucket ACL the following error is returned both in
the gateway and on AWS:
ErrAclNotSupported: {
Code: "AccessControlListNotSupported",
Description: "The bucket does not allow ACLs",
HTTPStatusCode: http.StatusBadRequest,
},
ACls can be enabled with PutBucketOwnershipControls
Changed bucket canned ACL translation
New backend interface methods:
PutBucketOwnershipControls
GetBucketOwnershipControls
DeleteBucketOwnershipControls
Added these to metrics
In the posix backend, the path argument to os.RemoveAll() does not
start with the bucket name. Since the path does not exist,
os.RemoveAll() does nothing and uploaded parts are left in the
".sgwtmp" directory.
This commit prefixes the path with the bucket name.
Fixes issue #626.
Signed-off-by: Christophe Vu-Brugier <christophe.vu-brugier@seagate.com>
Add meta.MetadataStorer compatibility to scoutfs so that scoutfs
is using the same interface as posix. This fixes the metadata
retrieval and adds the recently supported object lock compatibility
as well.
The complete multipart upload can be optimized in some cases
to not need to copy the full data from parts to the final
object file. If the filesystem supports it, there can be
optimizations to just clone exent references and not have to
actually copy the data.
For io.Copy() to make use of file_copy_range, we have to pass it
the *os.File file handles from the filesystem for the source and
destination. We were previously adding a layer of indirection
that was causing io.Copy() to fallback to full data copy. This
fixes the copy by exposing the underlying fd.
This also skips the falloc for the final object in complete
mutlipart upload, because some filesystems will be able to use
file_copy_range to optimize the copy and may not even need
new data allocations in the final object file.
Note that this only affects posix mode as scoutfs has special
handling for this case that is similar to but not compatible
with copy_file_range using a special ioctl.
Added support to enable object lock on bucket creation in posix and azure
backends.
Implemented the logic to add object legal hold and retention on object creation
in azure and posix backends.
Added the functionality for HeadObject to return object lock related headers.
Added integration tests for these features.
When fileystem quota exceeded, the gateway will now return the
error:
S3 error: 403 (QuotaExceeded):
Your request was denied due to quota exceeded.
This will help clients to better detect upload errors due to
quota exceeded.
Fixes#483
We had some duplicated code that we can bring into the backend
package so that we can remove duplications. This moves the mkdir
implementation into backend so that both posix and scoutfs can
call the same implementation.
This adds the default ACL to the CreateBucket backend method so
that the backend doesn't need to know how to construct and ACL.
This also moves the s3proxy ACLs to a tag key/value because the
gateway ACLs are not the same accounts as the backend s3 server.
TODO: we may need to mask this tag key/value if we add support
for the Get/PutBucketTagging API.
Since objects with trailing "/" are mapped to directories in the
posix filesystem, they must not contain data since there is no
place to store that data.
This checks both PutObject and CreateMultipartUpload for invalid
directory object types containing data.
Ben McClellandandGitHub
jonaustin09
Christophe Vu-Brugier
Brian Wheeler
Jon AustinandGitHub