As multipart uploads are translated to blobs in azure blob storage, they were visible in ListObjects(V2) as complete objects. Now the blobs with multipart prefix are filtered out during listing.
The listing logic is rewritten client-side to implement proper S3 semantics: flat blob enumeration with manual delimiter handling, correct truncation (IsTruncated only set when more items genuinely exist beyond maxKeys), and StartAfter/Marker/ContinuationToken applied via the lexicographic max of both constraints in ListObjectsV2.
For the same reason bucket deletion was not allowed. Now multipart objects are explicitly checked on bucket deletion and any pending multipart upload doesn't block the bucket deletion anymore.
encodeURIComponent encodes the trailing slash as %2F, producing a path
like /bucket/folder%2F. S3 servers (e.g. Ceph) reject this with a
SignatureDoesNotMatch error because the canonical URI used for signing
differs from what the server reconstructs.
encodeS3Key encodes each path segment individually, leaving slashes as
literal /, which is consistent with all other object-key operations in
the API client.
Fixes#2029
On Windows, a file apparently cannot be renamed while an open
handle to it exists.
The previous code used defer tempfile.Close(), which meant the
handle was still open when os.Rename was called, producing:
failed to rename temporary file: The process cannot access
the file because it is being used by another process.
Fix by closing the file explicitly before the rename.
Fixes#2021
The AWS SDK v2 includes Accept-Encoding in SigV4 signed headers
which causes GCS to return a SignatureDoesNotMatch error because
GCS rewrites that header internally before verifying the signature.
Add a --gcs-compatibility / VGW_S3_GCS_COMPATIBILITY option for the s3proxy
backend that injects two Smithy finalize-layer middlewares: one removes
Accept-Encoding from the request immediately before the Signing step, and
a second restores it after signing so the header is still sent on the wire.
see: https://github.com/aws/aws-sdk-go-v2/issues/1816
This can be removed once GCS fixes this incompatibility.
The urfave/cli/v2 has built-in auto-completion support for bash.
This enables generating the auto completions with a cli option
that can be added to environment setup scripts.
Also adding some distro support scripts for rpm and deb for the
auto completion setup.
To enable auto completion in zsh, this still needs to be added
to user .zshrc:
fpath=(/usr/share/zsh/site-functions $fpath)
autoload -Uz compinit && compinit
Fixes#2015
fasthttp v1.70.0 now enforces the HTTP/1.1 requirement of exactly
one Host header, rejecting requests that omit it. Fix tests that
were failing due to missing host.
Fixes#2030
When an object has offline blocks, the restore status was incorrectly set to `ongoing-request="false"` instead of omitting the header entirely, which causes s3 clients fail on parsing the x-amz-restore header.
Remove the incorrect `stageNotInProgress` constant and simplify the `requestOngoing` initialization to reflect the correct default.
Closes#1813
We use a specific `versionId` format(`ulid` package) to generate versionIds in posix, which is not compatible to S3. The versionId validation was performed in frontend which is a potential source of failure for s3 proxy configured on an s3 service which doesn't use ulid for versionId generation(e.g. aws S3). These changes move the specific `ulid` versionId validation to posix to not force any specific versionId format in the gateway.
Fixes#2009
The webui address validation loop called net.SplitHostPort on every address without first checking whether it was a unix socket path, causing an error for unix domain socket paths. The fix includes skipping the host:port validation for unix socket paths for webui.
A concurrent PutObject and DeleteObject on the same prefix directory
can race:
PutObject opens an O_TMPFILE in MetaTmpDir (not yet visible in the fs)
DeleteObject removes the last visible object in the prefix directory
and calls removeParents(), which rmdir's the now-empty prefix
directory
PutObject's link() tries to link the fd into a parent directory that
no longer exists
Fix by detecting ENOENT in the final link step (Linkat, Rename, and
MoveFile) and retrying after recreating the parent directory.
Also extract linkatOTmpfile() to consolidate the Linkat+EEXIST→Renameat
logic that was previously inline in link().
Fixes#1988
Fixes the [comment](https://github.com/versity/versitygw/issues/1648#issuecomment-4175425099)
Removes the unnecessary multipart/form-data boundary normalizing. The boundary prefix(`--`) was trimmed in `NewMultipartParser`, which caused incorrect boundary check for the boundaries starting with 2 dashes(e.g. `----WebKitFormBoundaryABC123`).
Closes#1897
Extract the `X-Amz-Source-Expected-Bucket-Owner` header for CopyObject and UploadPartCopy. Verify the source bucket owner in the backend and if the provided access key id doesn't match, return an `AccessDenied` error.
Fixes#1814
The `x-amz-bucket-region` is not mentioned in AWS S3 documentation, however s3 sends it in all ListObjects(V2) successful responses. The header is now added.
Closes#1648Fixes#1980Fixes#1981
This PR implements browser-based POST object uploads for S3-compatible form uploads. It adds support for handling `multipart/form-data` object uploads submitted from browsers, including streaming multipart parsing so file content is not buffered in memory, POST policy decoding and evaluation, SigV4-based form authorization, and integration with the existing `PutObject` backend flow. The implementation covers the full browser POST upload path, including validation of required form fields, credential scope and request date checks, signature verification, metadata extraction from `x-amz-meta-*` fields, checksum field parsing, object tagging conversion from XML into the query-string format expected by `PutObject`, and browser-compatible success handling through `success_action_status` and `success_action_redirect`. It also wires the new flow into the router and metrics layer and adds POST-specific error handling and debug logging across policy parsing, multipart parsing, and POST authorization. AWS S3 also accepts the `redirect` form field alongside `success_action_redirect`, but since AWS has marked `redirect` as deprecated and is planning to remove it, this gateway intentionally does not support it.
Fixes#1896
Enforces the S3 `5 GiB` copy source size limit across the posix and azure
backends for `CopyObject` and `UploadPartCopy`, returning `InvalidRequest` when
the source object exceeds the threshold.
The limit is now configurable via `--copy-object-threshold`
(`VGW_COPY_OBJECT_THRESHOLD`, default 5 GiB).
A new `--mp-max-parts flag` (`VGW_MP_MAX_PARTS`, default `10000`) has been added to make multipart upload parts number limit configurable.
No integration test has been added, as GitHub Actions cannot reliably
handle large objects.
Closes#1967
Add support for response header override query parameters(`response-cache-control`, `response-content-disposition`, `response-content-encoding`, `response-content-language`, `response-content-type`, `response-expires`) in `HeadObject`. Anonymous requests with override params are rejected with `ErrAnonymousResponseHeaders`.
Ben McClellandandGitHub
niksis02
Luke McCrone
dependabot[bot]andGitHub
Anael ORLINSKI
