This adds the option to store metadata for objects and buckets
within a specified directory:
bucket: <sidecar dir>/<bucket>/meta/<attribute>
object: <sidecar dir>/bucket/<object>/meta/<attribute>
Example invocation:
./versitygw -a myaccess -s mysecret posix --sidecar /tmp/sidecar /tmp/gw
The attributes are stored by name within the hidden directory.
The posix backend will return ENOTEMPTY when trying to delete a
directory that is not empty. This normally would run successfully
on object systems. So we need to create another non-standard error
for this case. We mainly just don't want to return InternalError
for this case.
Fixes#946
The fileystem will return ENOTDIR if we try to access a file path
where a parent directory within the path already exists as a file.
In this case we need to return a standard 404 no such key since
the request object does not exist within the filesytem.
Fixes#942
From #919, This provides the *os.File handle to io.Copy() for the
case where the full file range is requested. This prevents hiding
the *os.File type for io.Copy() optimizations.
This still requires the change to valyala/fasthttp#1889 to expose
the net.Conn similarly to enable the linux sendfile optimization.
We had 0755 hard coded for newly created directories before. This
adds a user option to configure what the default mode permissions
should be for newly created directories.
Fixes#878
This fixes the cases for racing uploads with the same object names.
Before we were making some bad assumptions about what would cause
an error when trying to link/rename the final object name into
the namespace, but missed the case that another upload for the
same name could be racing with this upload and causing an incorrect
error.
This also changes the order of setting metadata to prevent
accidental setting of metadata for the current upload to another
racing upload.
This also fix auth.CheckObjectAccess() when objects are removed
while this runs.
Fixes#854
The object semantics are that a key that ends with / is different
than a key that does not. However on posix a name that ends with /
is assumed to be a directory, but it still succeeds to access the
directory by name without a trailing /. So we need to explicitly
check if the request is for a non-directory and we are trying to
access a directory.
Fixes#824
This fixes put object with setting a content type. If no content
type is set, then we will return a default content type for
following requests. Mutli-part upload appears to be ok.
Also fixed content eincoding and multipart uploads.
Fixes#783
When using the REPLACE directive, we were incorrectly removing the
old metadata on the object due to missing the metadata prefix on
the key. Fix this to remove the correct metadata before setting
new metadata.
Fixes#787
The "x-amz-copy-source" header may start with '/' as observed with
WinSCP. However, '/' is also the separator between the bucket and the
object path in "x-amz-copy-source".
Consider the following code in VerifyObjectCopyAccess():
srcBucket, srcObject, found := strings.Cut(copySource, "/")
If `copySource` starts with '/', then `srcBucket` is set to an empty
string. Later, an error is returned because bucket "" does not exist.
This issue was fixed in the Posix and Azure backends by the following
commit:
* 5e484f2 fix: Fixed CopySource parsing to handle the values starting with '/' in CopyObject action in posix and azure backends.
But the issue was not fixed in `VerifyObjectCopyAccess`.
This commit sanitizes "x-amz-copy-source" right after the header is
extracted in `s3api/controllers/base.go`. This ensures that the
`CopySource` argument passed to the backend functions UploadPartCopy()
and CopyObject() does not start with '/'. Since the backends no longer
need to strip away any leading '/' in `CopySource`, the parts of
commit 5e484f2 modifying the Posix and Azure backends are reverted.
Fixes issue #773.
Signed-off-by: Christophe Vu-Brugier <christophe.vu-brugier@seagate.com>
It is better if we let the s3response module handle the xml
formatting spec specifics, and let the backends not worry
about how to format the time fields. This should help to
prevent any future backend modifications or additions from
accidental incorrect time formatting.
Changed ListObjectsV2 and ListObjects actions return types from
*s3.ListObjects(V2)Output to s3response.ListObjects(V2)Result.
Changed the listing objects timestamp to RFC3339 to match AWS
S3 objects timestamp.
Fixes#752
The posix limits wont exactly match up with the AWS key length
limits because posix has component length limits as well as path
length limits.
This reponds with the aws compatible KeyTooLongError under these
conditions now.
Note that delete object returns success even in the error cases.
Fixes#755
