This adds a NoMeta MetadataStorer that might be useful for read
only filesystems that are populated with data not from the gateway.
The limitation is that no incoming metadata is stored with the
buckets or objects, and all buckets and objects will only have
default metadata.
This adds the option to store metadata for objects and buckets
within a specified directory:
bucket: <sidecar dir>/<bucket>/meta/<attribute>
object: <sidecar dir>/bucket/<object>/meta/<attribute>
Example invocation:
./versitygw -a myaccess -s mysecret posix --sidecar /tmp/sidecar /tmp/gw
The attributes are stored by name within the hidden directory.
There were two issues that were preventing correct behavior here.
One was that we need to specifically request the container metadata
when listing containers, and then we also need to handle the case
where the container does not include the acl metadata.
This fixes both of these cases by adding in the metadata request
option for this container listing, and will return a default acl
if not provided in the container metadaata.
Fixes#948
We need to treat a prefix that has a parent component as a file
instead of a directory in the filesystem as a non existing prefix
instead of an internal server error. This error was caused
because we were not handling ENOTDIR within the fs walk.
Fixes#949
The posix backend will return ENOTEMPTY when trying to delete a
directory that is not empty. This normally would run successfully
on object systems. So we need to create another non-standard error
for this case. We mainly just don't want to return InternalError
for this case.
Fixes#946
The fileystem will return ENOTDIR if we try to access a file path
where a parent directory within the path already exists as a file.
In this case we need to return a standard 404 no such key since
the request object does not exist within the filesytem.
Fixes#942
The prefix can contain one or more parent directories. In this case it is not necessary to start traversal at the root dir. Instead start the directory traversal at the last directory component within the prefix.
Uses / for path separation for all platforms including ones that have other native os path separators. This ensures client compatibility across server platforms.
Fixes performance issue with ListObjects recursing into all subdirectories, even when using a delimiter. With delimiter, only contents at the top level prefix are returned with all other objects represented below common prefixes. Since the common prefixes don't list all objects separately, there is no need to traverse into the directories below the top level list-objects contents.
Fixes#903
From #919, This provides the *os.File handle to io.Copy() for the
case where the full file range is requested. This prevents hiding
the *os.File type for io.Copy() optimizations.
This still requires the change to valyala/fasthttp#1889 to expose
the net.Conn similarly to enable the linux sendfile optimization.
The latest azurite made a change where the blob metadata must be
explicitly requested when calling NewListBlobsFlatPager(). We were
taking action on metadata iteams, and the tests were failing due
to these always missing without requesting metadata to be included
in the response.
Fix is to enable metadata for the response.
We had 0755 hard coded for newly created directories before. This
adds a user option to configure what the default mode permissions
should be for newly created directories.
Fixes#878
This fixes the cases for racing uploads with the same object names.
Before we were making some bad assumptions about what would cause
an error when trying to link/rename the final object name into
the namespace, but missed the case that another upload for the
same name could be racing with this upload and causing an incorrect
error.
This also changes the order of setting metadata to prevent
accidental setting of metadata for the current upload to another
racing upload.
This also fix auth.CheckObjectAccess() when objects are removed
while this runs.
Fixes#854
The object semantics are that a key that ends with / is different
than a key that does not. However on posix a name that ends with /
is assumed to be a directory, but it still succeeds to access the
directory by name without a trailing /. So we need to explicitly
check if the request is for a non-directory and we are trying to
access a directory.
Fixes#824
This checks to see if the common prefix is before the marker and
thus would have been returned in earlier list objects request.
The error case was aws cli listing multiple entries for the same
common prefix when the listing required multiple pagination
requests.
Fixes#778
