This adds an example plugin that can be useful for benchmarking
the gateway frontend without adding any storage I/O on the
backend. This can also be a useful template for developers
looking to implement a new plugin for the gateway.
Also add a build verification to the go workflow for every plugin
in the plugins directory.
This adds the ability to specify unix domain socket paths for the
service listener with the --port <path> option. Where <path> can
be either a path to a file in a filesystem or prefixed with @ for
an abstract socket name.
Anything not matching the <host>:<port> pattern in the --port
option will be considered a socket filename.
Fixes#1606
According to AWS documentation:
> *“The PUT request header is limited to 8 KB in size. Within the PUT request header, the user-defined metadata is limited to 2 KB in size. The size of user-defined metadata is measured by taking the sum of the number of bytes in the UTF-8 encoding of each key and value.”*
Based on this, object metadata size is now limited to **2 KB** for all object upload operations (`PutObject`, `CopyObject`, and `CreateMultipartUpload`).
Fixes handling of metadata HTTP headers when the same header appears multiple times with different casing or even if they are identical. According to S3 behavior, these headers must be merged into a single lower-cased metadata key, with values concatenated using commas.
Example:
```
x-amz-meta-Key: value1
x-amz-meta-kEy: value2
x-amz-meta-keY: value3
```
Translated to:
```
key: value1,value2,value3
```
This PR also introduces an **8 KB limit for request headers**. Although the S3 documentation explicitly mentions the 8 KB limit only for **PUT requests**, in practice this limit applies to **all requests**.
To enforce the header size limit, the Fiber configuration option `ReadBufferSize` is used. This parameter defines the maximum number of bytes read when parsing an incoming request. Note that this limit does not apply strictly to request headers only, since request parsing also includes other parts of the request line (e.g., the HTTP method, protocol string, and version such as `HTTP/1.1`). So `ReadBufferSize` is effectively a limit for request headers size, but not the exact limit.
Fix warnings from newer systemd:
Standard output type syslog is obsolete,
automatically updating to journal.
This updates the stdout/stderr to journal output type which is
what is getting set anyways after the syslog type has been
deprecated.
No expected behavior change with this other than quieting
warnings.
Fixes#1909
Previously, the mapping between object metadata and posix object was as follows: for each metadata key, we stored a separate xattr with the `user.X-Amz-Meta.<key>` prefix. This resulted in syscall overhead when storing and deleting large numbers of metadata keys.
In addition, very long metadata keys caused failures because most posix filesystems limit xattr key lengths to 127–255 bytes, while S3 does not enforce such a per-key limit.
The logic has now been changed so that all object metadata is stored in a single xattr, `user.metadata`, as a JSON key/value object. For backward compatibility, metadata GET operations still fall back to the old mechanism (`metadata key -> xattr key`) when `user.metadata` is not present.
A new CLI utility has been added to convert all legacy object metadata to the new metadata format within the provided directory.
**Example usage:**
```
versitygw utils convert-xattr-metadata path/to/bucket
```
or
```
versitygw utils cxm path/to/bucket
```
It is recommended to run this command on bucket directories to convert all legacy metadata for every object in the bucket.
The fifth parameter of request() is useAdminEndpoint (boolean), but
putBucketPolicy was passing a Content-Type header object instead.
This caused useAdminEndpoint to be truthy and contentType to default
to 'application/xml' instead of 'application/json'.
Fixed by passing false for useAdminEndpoint and 'application/json'
as the contentType argument.
Fixes#1928
Use net.JoinHostPort instead of manual string formatting in
buildServiceURLs so IPv6 addresses are properly wrapped in brackets.
This fixes malformed URLs like "http://::1:7070" being presented as
the default server URL in the WebUI login form when listening on an
IPv6 address; they now correctly render as "http://[::1]:7070".
Fixes#1926
When copying between two different Azure blobs, the source download
stream body was only consumed by PutObject but never explicitly closed.
If PutObject or any subsequent step returned an error, the underlying
HTTP connection held by the Azure SDK was never released, leaking both
the connection and any internal SDK retry goroutines attached to it.
Added a deferred close on downloadResp.Body immediately after the
successful DownloadStream call to ensure the body is always drained and
released regardless of the outcome.
Azure's ListContainers Marker parameter requires an opaque internal token
(e.g. /accountname/containername) rather than a plain container name, so
passing MaxResults and our ContinuationToken directly to the Azure API
caused 400 OutOfRangeInput errors. Rework ListBuckets to iterate all Azure
pages client-side, skip entries at or before the ContinuationToken (matching
the posix backend's "start after" semantics), and stop once MaxBuckets items
have been collected, setting ContinuationToken to the last returned bucket
name. This avoids using Azure's NextMarker entirely and correctly handles
both unpaginated and paginated requests.
Azure Storage's StageBlock REST API rejects Content-Length: 0
with InvalidHeaderValue. The tests (PresignedAuth_UploadPart,
UploadPart_success) upload a nil/empty body, which causes the
Azure SDK to send Content-Length: 0. Azurite is lenient and
accepts it; real Azure Storage does not.
Use a new metadata key ("Zerobytesparts") sett on the
.sgwtmp/multipart/<uploadId>/<object-hash> blob to track and
0 length parts.
