mirrors/versitygw
1
0
Fork 0
mirror of https://github.com/versity/versitygw.git synced 2026-04-17 11:21:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,364 Commits 31 Branches 49 Tags
f8e188a42d0f286dcdb2a5c17c1bd8f6205dfd60
Commit Graph

2364 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
dependabot[bot]
f8e188a42d chore(deps): bump the dev-dependencies group with 11 updates 
Bumps the dev-dependencies group with 11 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.32.13` | `1.32.14` |
| [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) | `1.19.13` | `1.19.14` |
| [github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager](https://github.com/aws/aws-sdk-go-v2) | `0.1.12` | `0.1.14` |
| [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) | `1.97.3` | `1.98.0` |
| [github.com/aws/smithy-go](https://github.com/aws/smithy-go) | `1.24.2` | `1.24.3` |
| [github.com/Azure/azure-sdk-for-go/sdk/internal](https://github.com/Azure/azure-sdk-for-go) | `1.11.2` | `1.12.0` |
| [github.com/AzureAD/microsoft-authentication-library-for-go](https://github.com/AzureAD/microsoft-authentication-library-for-go) | `1.7.0` | `1.7.1` |
| [github.com/andybalholm/brotli](https://github.com/andybalholm/brotli) | `1.2.0` | `1.2.1` |
| [github.com/aws/aws-sdk-go-v2/service/sso](https://github.com/aws/aws-sdk-go-v2) | `1.30.14` | `1.30.15` |
| [github.com/aws/aws-sdk-go-v2/service/ssooidc](https://github.com/aws/aws-sdk-go-v2) | `1.35.18` | `1.35.19` |
| [github.com/mattn/go-runewidth](https://github.com/mattn/go-runewidth) | `0.0.21` | `0.0.22` |


Updates `github.com/aws/aws-sdk-go-v2/config` from 1.32.13 to 1.32.14
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.32.13...config/v1.32.14)

Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.19.13 to 1.19.14
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.19.13...credentials/v1.19.14)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager` from 0.1.12 to 0.1.14
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/feature/s3/transfermanager/v0.1.12...feature/s3/transfermanager/v0.1.14)

Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.97.3 to 1.98.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.97.3...service/s3/v1.98.0)

Updates `github.com/aws/smithy-go` from 1.24.2 to 1.24.3
- [Release notes](https://github.com/aws/smithy-go/releases)
- [Changelog](https://github.com/aws/smithy-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/smithy-go/compare/v1.24.2...v1.24.3)

Updates `github.com/Azure/azure-sdk-for-go/sdk/internal` from 1.11.2 to 1.12.0
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/internal/v1.11.2...sdk/azcore/v1.12.0)

Updates `github.com/AzureAD/microsoft-authentication-library-for-go` from 1.7.0 to 1.7.1
- [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-go/releases)
- [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-go/blob/main/changelog.md)
- [Commits](https://github.com/AzureAD/microsoft-authentication-library-for-go/compare/v1.7.0...v1.7.1)

Updates `github.com/andybalholm/brotli` from 1.2.0 to 1.2.1
- [Commits](https://github.com/andybalholm/brotli/compare/v1.2.0...v1.2.1)

Updates `github.com/aws/aws-sdk-go-v2/service/sso` from 1.30.14 to 1.30.15
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/acm/v1.30.14...service/acm/v1.30.15)

Updates `github.com/aws/aws-sdk-go-v2/service/ssooidc` from 1.35.18 to 1.35.19
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/pi/v1.35.18...service/pi/v1.35.19)

Updates `github.com/mattn/go-runewidth` from 0.0.21 to 0.0.22
- [Commits](https://github.com/mattn/go-runewidth/compare/v0.0.21...v0.0.22)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-version: 1.32.14
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-version: 1.19.14
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager
  dependency-version: 0.1.14
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-version: 1.98.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/smithy-go
  dependency-version: 1.24.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/internal
  dependency-version: 1.12.0
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/AzureAD/microsoft-authentication-library-for-go
  dependency-version: 1.7.1
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/andybalholm/brotli
  dependency-version: 1.2.1
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sso
  dependency-version: 1.30.15
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/ssooidc
  dependency-version: 1.35.19
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/mattn/go-runewidth
  dependency-version: 0.0.22
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-04-06 21:12:54 +00:00
Ben McClelland
09413c5f60 Merge pull request #2014 from versity/sis/webui-unix-socket 
fix: fix webui port validation to allow unix socket paths
 2026-04-03 13:19:52 -07:00
niksis02
b151d4626a fix: fix webui port validation to allow unix socket paths 
Fixes #2009

The webui address validation loop called net.SplitHostPort on every address without first checking whether it was a unix socket path, causing an error for unix domain socket paths. The fix includes skipping the host:port validation for unix socket paths for webui.
 2026-04-03 21:12:22 +04:00
Ben McClelland
1fca33e738 Merge pull request #2006 from versity/ben/racing-put-delete 
fix: retry link on ENOENT caused by racing DeleteObject
v1.4.0 		2026-04-02 09:57:48 -07:00
Ben McClelland
98a186cc4e Merge pull request #2008 from versity/sis/object-post-boundary-prefix 
fix: remove POST object multipart boundary prefix trimming
 2026-04-02 08:27:02 -07:00
Ben McClelland
71ae9bf045 Merge pull request #2005 from versity/sis/copyobject-expected-source-bucket-owner 
feat: implement x-amz-source-expected-bucket-owner for CopyObject and UploadPartCopy
 2026-04-02 08:25:50 -07:00
Ben McClelland
c26012905c fix: retry link on ENOENT caused by racing DeleteObject 
A concurrent PutObject and DeleteObject on the same prefix directory
can race:
PutObject opens an O_TMPFILE in MetaTmpDir (not yet visible in the fs)
DeleteObject removes the last visible object in the prefix directory
and calls removeParents(), which rmdir's the now-empty prefix
directory
PutObject's link() tries to link the fd into a parent directory that
no longer exists

Fix by detecting ENOENT in the final link step (Linkat, Rename, and
MoveFile) and retrying after recreating the parent directory.

Also extract linkatOTmpfile() to consolidate the Linkat+EEXIST→Renameat
logic that was previously inline in link().

Fixes #1988
 2026-04-02 08:14:44 -07:00
niksis02
a25408c225 fix: remove POST object multipart boundary prefix trimming 
Fixes the [comment](https://github.com/versity/versitygw/issues/1648#issuecomment-4175425099)

Removes the unnecessary multipart/form-data boundary normalizing. The boundary prefix(`--`) was trimmed in `NewMultipartParser`, which caused incorrect boundary check for the boundaries starting with 2 dashes(e.g. `----WebKitFormBoundaryABC123`).
 2026-04-02 17:20:23 +04:00
Ben McClelland
a6fffc6372 Merge pull request #2003 from versity/sis/missing-date-header-err-status-code 
fix: change ErrMissingDateHeader http status code to 403
 2026-04-01 20:56:37 -07:00
Ben McClelland
e009f692d4 Merge pull request #2002 from versity/sis/listobjects-bucket-region-header 
fix: add x-amz-bucket-region to ListObjects(V2) response headers.
 2026-04-01 20:55:34 -07:00
Luke McCrone
6b93190922 test: fix response code 2026-04-01 15:13:58 -03:00
niksis02
052f2364cc feat: implement x-amz-source-expected-bucket-owner for CopyObject and UploadPartCopy 
Closes #1897

Extract the `X-Amz-Source-Expected-Bucket-Owner` header for CopyObject and UploadPartCopy. Verify the source bucket owner in the backend and if the provided access key id doesn't match, return an `AccessDenied` error.
 2026-04-01 21:44:33 +04:00
niksis02
90e240d23f fix: change ErrMissingDateHeader http status code to 403 
Fixes #1934

The HTTP status code for `ErrMissingDateHeader` was `400`. Now it's changed to correct `403`.
 2026-04-01 16:21:07 +04:00
niksis02
3d47e7fed1 fix: add x-amz-bucket-region to ListObjects(V2) response headers. 
Fixes #1814

The `x-amz-bucket-region` is not mentioned in AWS S3 documentation, however s3 sends it in all ListObjects(V2) successful responses. The header is now added.
 2026-04-01 15:22:33 +04:00
Ben McClelland
e0209ebab4 Merge pull request #1997 from versity/sis/copyobject-threshold 
fix: enforce 5gb copy source object size threshold.
 2026-03-31 12:27:11 -07:00
Ben McClelland
89ab7a4f2b Merge pull request #1982 from versity/sis/POST-object 
feat: add browser-based POST object upload support
 2026-03-31 12:25:00 -07:00
Ben McClelland
56b9c67905 Merge pull request #2001 from versity/sis/headobject-response-overrides 
feat: support response header overrides in HeadObject
 2026-03-31 12:23:57 -07:00
niksis02
59002b2650 feat: implement integration tests for browser-based POST object 2026-03-31 22:47:04 +04:00
niksis02
285d130a47 feat: add browser-based POST object upload support 
Closes #1648
Fixes #1980
Fixes #1981

This PR implements browser-based POST object uploads for S3-compatible form uploads. It adds support for handling `multipart/form-data` object uploads submitted from browsers, including streaming multipart parsing so file content is not buffered in memory, POST policy decoding and evaluation, SigV4-based form authorization, and integration with the existing `PutObject` backend flow. The implementation covers the full browser POST upload path, including validation of required form fields, credential scope and request date checks, signature verification, metadata extraction from `x-amz-meta-*` fields, checksum field parsing, object tagging conversion from XML into the query-string format expected by `PutObject`, and browser-compatible success handling through `success_action_status` and `success_action_redirect`. It also wires the new flow into the router and metrics layer and adds POST-specific error handling and debug logging across policy parsing, multipart parsing, and POST authorization. AWS S3 also accepts the `redirect` form field alongside `success_action_redirect`, but since AWS has marked `redirect` as deprecated and is planning to remove it, this gateway intentionally does not support it.
 2026-03-31 22:44:54 +04:00
niksis02
bbe246e8ec fix: enforce 5gb copy source object size threshold. 
Fixes #1896

Enforces the S3 `5 GiB` copy source size limit across the posix and azure
backends for `CopyObject` and `UploadPartCopy`, returning `InvalidRequest` when
the source object exceeds the threshold.

The limit is now configurable via `--copy-object-threshold`
(`VGW_COPY_OBJECT_THRESHOLD`, default 5 GiB).
A new `--mp-max-parts flag` (`VGW_MP_MAX_PARTS`, default `10000`) has been added to make multipart upload parts number limit configurable.

No integration test has been added, as GitHub Actions cannot reliably
handle large objects.
 2026-03-31 22:44:03 +04:00
niksis02
3b17f05d17 feat: support response header overrides in HeadObject 
Closes #1967

Add support for response header override query parameters(`response-cache-control`, `response-content-disposition`, `response-content-encoding`, `response-content-language`, `response-content-type`, `response-expires`) in `HeadObject`. Anonymous requests with override params are rejected with `ErrAnonymousResponseHeaders`.
 2026-03-31 17:19:03 +04:00
Ben McClelland
69b23fe139 Merge pull request #2000 from versity/dependabot/go_modules/dev-dependencies-961b21b1c2 2026-03-30 20:42:58 -07:00
Ben McClelland
2d91e0203e Merge pull request #1999 from versity/dependabot/github_actions/sigstore/cosign-installer-4.1.1 2026-03-30 20:42:24 -07:00
dependabot[bot]
fe61e3070b chore(deps): bump the dev-dependencies group with 17 updates 
Bumps the dev-dependencies group with 17 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) | `1.41.4` | `1.41.5` |
| [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.32.12` | `1.32.13` |
| [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) | `1.19.12` | `1.19.13` |
| [github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager](https://github.com/aws/aws-sdk-go-v2) | `0.1.11` | `0.1.12` |
| [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) | `1.97.2` | `1.97.3` |
| [github.com/nats-io/nats.go](https://github.com/nats-io/nats.go) | `1.49.0` | `1.50.0` |
| [github.com/aws/aws-sdk-go-v2/feature/ec2/imds](https://github.com/aws/aws-sdk-go-v2) | `1.18.20` | `1.18.21` |
| [github.com/aws/aws-sdk-go-v2/internal/configsources](https://github.com/aws/aws-sdk-go-v2) | `1.4.20` | `1.4.21` |
| [github.com/aws/aws-sdk-go-v2/internal/endpoints/v2](https://github.com/aws/aws-sdk-go-v2) | `2.7.20` | `2.7.21` |
| [github.com/aws/aws-sdk-go-v2/internal/v4a](https://github.com/aws/aws-sdk-go-v2) | `1.4.21` | `1.4.22` |
| [github.com/aws/aws-sdk-go-v2/service/internal/checksum](https://github.com/aws/aws-sdk-go-v2) | `1.9.12` | `1.9.13` |
| [github.com/aws/aws-sdk-go-v2/service/internal/presigned-url](https://github.com/aws/aws-sdk-go-v2) | `1.13.20` | `1.13.21` |
| [github.com/aws/aws-sdk-go-v2/service/internal/s3shared](https://github.com/aws/aws-sdk-go-v2) | `1.19.20` | `1.19.21` |
| [github.com/aws/aws-sdk-go-v2/service/signin](https://github.com/aws/aws-sdk-go-v2) | `1.0.8` | `1.0.9` |
| [github.com/aws/aws-sdk-go-v2/service/sso](https://github.com/aws/aws-sdk-go-v2) | `1.30.13` | `1.30.14` |
| [github.com/aws/aws-sdk-go-v2/service/ssooidc](https://github.com/aws/aws-sdk-go-v2) | `1.35.17` | `1.35.18` |
| [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) | `1.41.9` | `1.41.10` |


Updates `github.com/aws/aws-sdk-go-v2` from 1.41.4 to 1.41.5
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.41.4...v1.41.5)

Updates `github.com/aws/aws-sdk-go-v2/config` from 1.32.12 to 1.32.13
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.32.12...config/v1.32.13)

Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.19.12 to 1.19.13
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/sqs/v1.19.12...credentials/v1.19.13)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager` from 0.1.11 to 0.1.12
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/feature/s3/transfermanager/v0.1.11...feature/s3/transfermanager/v0.1.12)

Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.97.2 to 1.97.3
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.97.2...service/s3/v1.97.3)

Updates `github.com/nats-io/nats.go` from 1.49.0 to 1.50.0
- [Release notes](https://github.com/nats-io/nats.go/releases)
- [Commits](https://github.com/nats-io/nats.go/compare/v1.49.0...v1.50.0)

Updates `github.com/aws/aws-sdk-go-v2/feature/ec2/imds` from 1.18.20 to 1.18.21
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/config/v1.18.21/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.18.20...config/v1.18.21)

Updates `github.com/aws/aws-sdk-go-v2/internal/configsources` from 1.4.20 to 1.4.21
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/internal/v4a/v1.4.20...internal/v4a/v1.4.21)

Updates `github.com/aws/aws-sdk-go-v2/internal/endpoints/v2` from 2.7.20 to 2.7.21
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/internal/endpoints/v2.7.20...internal/endpoints/v2.7.21)

Updates `github.com/aws/aws-sdk-go-v2/internal/v4a` from 1.4.21 to 1.4.22
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/internal/v4a/v1.4.21...internal/v4a/v1.4.22)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/checksum` from 1.9.12 to 1.9.13
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/grafana/v1.9.12...service/grafana/v1.9.13)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/presigned-url` from 1.13.20 to 1.13.21
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.13.20...credentials/v1.13.21)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/s3shared` from 1.19.20 to 1.19.21
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/codebuild/v1.19.20...service/codebuild/v1.19.21)

Updates `github.com/aws/aws-sdk-go-v2/service/signin` from 1.0.8 to 1.0.9
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/m2/v1.0.8...service/m2/v1.0.9)

Updates `github.com/aws/aws-sdk-go-v2/service/sso` from 1.30.13 to 1.30.14
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/acm/v1.30.13...service/acm/v1.30.14)

Updates `github.com/aws/aws-sdk-go-v2/service/ssooidc` from 1.35.17 to 1.35.18
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/pi/v1.35.17...service/pi/v1.35.18)

Updates `github.com/aws/aws-sdk-go-v2/service/sts` from 1.41.9 to 1.41.10
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/ecs/v1.41.9...service/ecs/v1.41.10)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-version: 1.41.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-version: 1.32.13
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-version: 1.19.13
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager
  dependency-version: 0.1.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-version: 1.97.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/nats-io/nats.go
  dependency-version: 1.50.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/ec2/imds
  dependency-version: 1.18.21
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/internal/configsources
  dependency-version: 1.4.21
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/internal/endpoints/v2
  dependency-version: 2.7.21
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/internal/v4a
  dependency-version: 1.4.22
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/checksum
  dependency-version: 1.9.13
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/presigned-url
  dependency-version: 1.13.21
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/s3shared
  dependency-version: 1.19.21
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/signin
  dependency-version: 1.0.9
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sso
  dependency-version: 1.30.14
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/ssooidc
  dependency-version: 1.35.18
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-version: 1.41.10
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-03-30 22:13:59 +00:00
dependabot[bot]
1ec724b7a4 chore(deps): bump sigstore/cosign-installer from 4.1.0 to 4.1.1 
Bumps [sigstore/cosign-installer](https://github.com/sigstore/cosign-installer) from 4.1.0 to 4.1.1.
- [Release notes](https://github.com/sigstore/cosign-installer/releases)
- [Commits](https://github.com/sigstore/cosign-installer/compare/v4.1.0...v4.1.1)

---
updated-dependencies:
- dependency-name: sigstore/cosign-installer
  dependency-version: 4.1.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-03-30 21:59:18 +00:00
Ben McClelland
2b12a5b81c Merge pull request #1989 from versity/test/rest_list_objects_encoding 
test: REST ListObjects encoding
 2026-03-30 14:43:12 -07:00
Ben McClelland
f6cd991d89 Merge pull request #1993 from versity/ben/racing-mp-complete 
fix: serialize concurrent CompleteMultipartUpload calls via rename
 2026-03-30 09:05:54 -07:00
Luke McCrone
8feed3b333 test: ListObjects encoding-type tests 2026-03-30 10:24:03 -03:00
Ben McClelland
13b3dc5267 fix: serialize concurrent CompleteMultipartUpload calls via rename 
When two requests raced to complete the same multipart upload, the first
caller to finish would remove the part files and upload directory. The
second caller, already past the initial existence check, would then fail
mid-flight with confusing errors such as ErrInvalidPart or an I/O error
when trying to open a part that no longer exists.

Fix this by atomically renaming the upload directory from <uploadID> to
<uploadID>.inprogress at the very start of CompleteMultipartUploadWithCopy,
before any part data is read. A concurrent caller will now find the
original directory absent and receive a clean NoSuchUpload error. A
deferred rename restores the original name if the complete does not
succeed, allowing the client to retry.

ListMultipartUploads is updated to skip any directories whose name ends
in .inprogress so in-flight completes do not appear as pending uploads.
 2026-03-27 16:14:20 -07:00
Ben McClelland
fab9f1f813 Merge pull request #1994 from versity/test/rest_go_and_eval_fix 
test: remove eval, go command generation overhaul
 2026-03-27 15:56:13 -07:00
Ben McClelland
0234f3ecc7 Merge pull request #1992 from versity/ben/cleanup-fd 
fix: cleanup file descriptor leaks with chown fails
 2026-03-27 11:43:31 -07:00
Ben McClelland
52fa3f1f76 Merge pull request #1991 from versity/ben/scoutfs-fail-upload 
fix: abort scoutfs multipart uploads on error after successful moveblocks
 2026-03-27 11:43:17 -07:00
Luke McCrone
ccd0ba2762 test: remove eval, go command generation overhaul 2026-03-27 15:00:56 -03:00
Ben McClelland
495b38a899 fix: abort scoutfs multipart uploads on error after successful moveblocks 
The scoutfs backend uses the move blocks ioctl when combining parts into the
final multipart upload object. Once a move blocks from any part is successful,
the original data is no longer in the part file. If the multipart upload fails
and retries, future complete multipart upload calls will not have the correct
data within the part files anymore.

To prevent this case, once a move blocks call is successful for an upload, any
future failure for the complete upload is set to auto-abort the upload to force
clients to re-upload the part data again.
 2026-03-26 16:28:42 -07:00
Ben McClelland
927d1d668a fix: cleanup file descriptor leaks with chown fails 
We were missing a few cases of cleaning up temp files and file
descriptors in the openTmpFile Chown() error cases.
 2026-03-26 15:45:20 -07:00
Ben McClelland
6c8eba12e3 Merge pull request #1987 from versity/dependabot/go_modules/dev-dependencies-a33a00629d 
chore(deps): bump the dev-dependencies group with 5 updates
 2026-03-24 08:12:38 -07:00
dependabot[bot]
3a8d658a7f chore(deps): bump the dev-dependencies group with 5 updates 
Bumps the dev-dependencies group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager](https://github.com/aws/aws-sdk-go-v2) | `0.1.10` | `0.1.11` |
| [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) | `1.97.1` | `1.97.2` |
| [github.com/go-ldap/ldap/v3](https://github.com/go-ldap/ldap) | `3.4.12` | `3.4.13` |
| [github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream](https://github.com/aws/aws-sdk-go-v2) | `1.7.7` | `1.7.8` |
| [github.com/klauspost/compress](https://github.com/klauspost/compress) | `1.18.4` | `1.18.5` |


Updates `github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager` from 0.1.10 to 0.1.11
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/feature/s3/transfermanager/v0.1.10...feature/s3/transfermanager/v0.1.11)

Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.97.1 to 1.97.2
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.97.1...service/s3/v1.97.2)

Updates `github.com/go-ldap/ldap/v3` from 3.4.12 to 3.4.13
- [Release notes](https://github.com/go-ldap/ldap/releases)
- [Commits](https://github.com/go-ldap/ldap/compare/v3.4.12...v3.4.13)

Updates `github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream` from 1.7.7 to 1.7.8
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/m2/v1.7.7...service/m2/v1.7.8)

Updates `github.com/klauspost/compress` from 1.18.4 to 1.18.5
- [Release notes](https://github.com/klauspost/compress/releases)
- [Commits](https://github.com/klauspost/compress/compare/v1.18.4...v1.18.5)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager
  dependency-version: 0.1.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-version: 1.97.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/go-ldap/ldap/v3
  dependency-version: 3.4.13
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream
  dependency-version: 1.7.8
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/klauspost/compress
  dependency-version: 1.18.5
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-03-23 21:08:32 +00:00
Ben McClelland
271d7377dd Merge pull request #1979 from versity/test/util_cleanup 
test: cleanup of some common tests, bucket and file creation
 2026-03-20 15:35:58 -07:00
Luke McCrone
a96def085d test: test_common.sh updates, bucket/file creation updates, hardcode fixes 2026-03-20 14:09:00 -03:00
Jakob van Santen
22f04312a7 fix: cleanup tempfiles when error prevents calling link 
When a client provides an invalid or incomplete body for a single-part
upload, the handler returns before the link stage.  Factor tempfile
removal into cleanup() to catch all cases.
 2026-03-20 08:25:44 -07:00
Jakob van Santen
56cb36d45a feat: add option to disable copy-file-range for multipart uploads 
The standard io.Copy() will attempt to use copy_file_range when available.
However, this can cause problems with certain filesystems. Add an option
that will prevent io.Copy() from being able to use copy_file_range to force
a standard data copy between file descriptors when needed for completing
multipart uploads.
 2026-03-20 08:23:01 -07:00
Ben McClelland
ae411fa3c1 Merge pull request #1974 from jvansanten/no-truncate 
Always write complete files in sidecar meta provider
 2026-03-19 16:01:50 -07:00
Ben McClelland
88e6396950 Merge pull request #1889 from versity/ben/fast-walk 
feat: implement quicker backend/posix walk algorithm
 2026-03-19 16:00:54 -07:00
tonyipm
33917ad6f3 feat: implement quicker backend/posix walk algorithm 
Rewrite the posix Walk implementation to avoid the extra ReadDir per
directory that was noted as a TODO in the old code.  The new algorithm
holds all traversal state in a walkState struct and uses processDir to
interleave sibling entries in correct S3 lexicographic order without a
second syscall.

Key changes:
prefix optimisation: jump directly into the deepest matching directory
rather than scanning from the root on every call

marker short-circuit: skip entire subtrees that are lexically before
the marker, making paginated listing faster

Co-authored-by: Ben McClelland <ben.mcclelland@versity.com>
 2026-03-18 17:39:16 -07:00
Ben McClelland
df1c312c7a Merge pull request #1968 from versity/test/rest_list_objects 
Test/rest list objects
 2026-03-18 10:25:25 -07:00
Ben McClelland
02f925a84b Merge pull request #1978 from 57-Wolve/57-Wolve-patch-s3-glacier-response 
Update scoutfs glacier stage constants for ongoing requests
 2026-03-18 10:25:06 -07:00
Ben McClelland
766437a66d Merge pull request #1975 from jvansanten/hide-non-bucket-dirs 
Do not expose directories that can't be buckets
 2026-03-18 10:24:50 -07:00
57_Wolve
c32ddfff1a Update stage constants for ongoing requests 
Fix issue with incompatible S3 response for offline and staging status.

Resolves issue with Restic Glacier support.
 2026-03-18 09:46:55 -05:00
Jakob van Santen
d8f927519a Do not expose directories that can't be buckets 2026-03-18 15:33:10 +01:00
Jakob van Santen
0559807783 Always write complete files in sidecar meta provider 
Some filesystems like dCache don't allow file truncation
 2026-03-18 15:29:29 +01:00