This was previously not including the bucket directory for the
mutlipart temp file cleanup. This fixes leftovers in the tmp
directories after uploading multipart uploads.
Fixes#1004Fixes#1122Fixes#1120
Separates `GetObject` and `UploadPartCopy` range parsing/validation.
`GetObject` returns a successful response if acceptRange is invalid.
Adjusts the range upper limit, if it exceeds the actual objects size for `GetObject`.
Corrects the `ContentRange` in the `GetObject` response.
Fixes the `UploadPartCopy` action copy source range parsing/validation.
`UploadPartCopy` returns `InvalidArgument` if the copy source range is not valid.
Instead of an internal server error, we should be returning
method not allowed when trying to upload to a read only
filesystem or make other modifications that are expected to
fail. This will give clearer feedback to the clients
that this is not expected to work.
Fixes#1062
We currently store bucket ACLs as tags in the backend S3 service.
Some backend services do not implment tags though. In this case,
we need to return the default bucket ACL for some continued
functionality.
There is still more work to return more correct errors for setting
ACLs when this is not implemented.
The frontend for versitygw wants to manage its own object
retention, and these settings are conflicting with trying
to forward object lock requests to the backend s3 server.
This disables object lock for the s3 backend for now until
we can properly implement this for the backend. We will
likely just need to add metadata to backend buckets and
objects for our object retention settings.
For any uninitialized input values, the s3 sdk expects these to
be set to nil instead of the type zero value. Setting to the zero
value is incorrectly making the server request think we are
trying to set those parameters.
This sets all input parameters to nil for the default values. Many
of these are already nil from the frontend, but this should
protect against a future addition of input params for an api
requests and not remembering to add teh zero value check on in
the s3 backend.
We were previously getting the following error for put-object:
<Error><Code>InvalidArgument</Code><Message>x-amz-object-lock-retain-until-date and x-amz-object-lock-mode must both be supplied</Message><Resource></Resource><RequestId></RequestId><HostId></HostId></Error>
- extract method to find top-level dirs, optionally including links to
dirs
- rework ListBuckets and ListBucketsAndOwners to use FileInfo rather
than DirEntry
Fixes#1081
Updated the convertObjects function to initialize the result slice
with a length of 0 and a capacity equal to the length of the input
slice (objs).
This fixes the problem where the results were being appended to
a slice that already contained 0 value entries for the
preallocated length.
The s3 sdk expects the version id and other input options to be
nil when not specified. Otherwise it incorrectly accepts the ""
string as the requested input option.
This just sets the version id and otehr options back to nil if
its "" from the s3api controller.
For s3 proxy backend, the ACLs are stored in bucket tags to not
conflict with the underlying s3 object store ACLs. If the tag
for the gatway ACLs does not exist for the bucket, then we were
incorrectly returning the NoSuchTagSet from teh tag lookup.
Instead, in this case we need to just return the default gateway
bucket ACL, which is the root account ownership.
Side note that there is currently a bug in the go sdk where the
NoSuchTagSet is not defained as an error type, and a workaround
for string matching is required until this is addressed.
https://github.com/aws/aws-sdk-go-v2/issues/2878
This adds a NoMeta MetadataStorer that might be useful for read
only filesystems that are populated with data not from the gateway.
The limitation is that no incoming metadata is stored with the
buckets or objects, and all buckets and objects will only have
default metadata.
This adds the option to store metadata for objects and buckets
within a specified directory:
bucket: <sidecar dir>/<bucket>/meta/<attribute>
object: <sidecar dir>/bucket/<object>/meta/<attribute>
Example invocation:
./versitygw -a myaccess -s mysecret posix --sidecar /tmp/sidecar /tmp/gw
The attributes are stored by name within the hidden directory.
There were two issues that were preventing correct behavior here.
One was that we need to specifically request the container metadata
when listing containers, and then we also need to handle the case
where the container does not include the acl metadata.
This fixes both of these cases by adding in the metadata request
option for this container listing, and will return a default acl
if not provided in the container metadaata.
Fixes#948
We need to treat a prefix that has a parent component as a file
instead of a directory in the filesystem as a non existing prefix
instead of an internal server error. This error was caused
because we were not handling ENOTDIR within the fs walk.
Fixes#949
The posix backend will return ENOTEMPTY when trying to delete a
directory that is not empty. This normally would run successfully
on object systems. So we need to create another non-standard error
for this case. We mainly just don't want to return InternalError
for this case.
Fixes#946
The fileystem will return ENOTDIR if we try to access a file path
where a parent directory within the path already exists as a file.
In this case we need to return a standard 404 no such key since
the request object does not exist within the filesytem.
Fixes#942
The prefix can contain one or more parent directories. In this case it is not necessary to start traversal at the root dir. Instead start the directory traversal at the last directory component within the prefix.
Uses / for path separation for all platforms including ones that have other native os path separators. This ensures client compatibility across server platforms.
Fixes performance issue with ListObjects recursing into all subdirectories, even when using a delimiter. With delimiter, only contents at the top level prefix are returned with all other objects represented below common prefixes. Since the common prefixes don't list all objects separately, there is no need to traverse into the directories below the top level list-objects contents.
Fixes#903
From #919, This provides the *os.File handle to io.Copy() for the
case where the full file range is requested. This prevents hiding
the *os.File type for io.Copy() optimizations.
This still requires the change to valyala/fasthttp#1889 to expose
the net.Conn similarly to enable the linux sendfile optimization.
The latest azurite made a change where the blob metadata must be
explicitly requested when calling NewListBlobsFlatPager(). We were
taking action on metadata iteams, and the tests were failing due
to these always missing without requesting metadata to be included
in the response.
Fix is to enable metadata for the response.
