Commit Graph
2873 Commits
Author SHA1 Message Date
niksis02 aa1e82b66c fix: require s3:PutObjectTagging for tagged browser-based POST object uploads
`POSTObject` authorized only `s3:PutObject`, even when the form's `tagging` field set tags on the uploaded object. `PutObject`, `CopyObject` and `CreateMultipartUpload` also require `s3:PutObjectTagging` when tags are sent, so a caller without that permission could tag objects through `POST` in a way `PutObject` refuses. S3 enforces the permission for `POST` too: a user granted only `s3:PutObject` gets `AccessDenied` for `s3:PutObjectTagging` on the object ARN.

`POSTObject` now adds `s3:PutObjectTagging` to the `verifyAccess` actions when the parsed tag set is not empty, in the same single check as `s3:PutObject`, matching `PutObject`. A `tagging` field with an empty `TagSet` still needs only `s3:PutObject`, as on S3.

The access check now runs after the form is validated, instead of before the `POST` policy is parsed. S3 validates the policy and its conditions, the tagging, the checksum fields, the metadata size and the website redirect location before it authorizes the upload, so a caller missing a permission now gets the validation error for an invalid form instead of `AccessDenied`.
2026-09-24 22:31:08 +04:00
84e8b0dc56 fix: reject DeleteObjects requests with more than 1000 keys
* fix: reject DeleteObjects requests with more than 1000 keys

A DeleteObjects request may name at most 1000 keys; S3 rejects anything
larger with 400 InvalidRequest. The handler parsed the body and passed
every key through authorization and on to the backend, so a 1001-key
request was processed instead of being refused.

Check the count right after the body is parsed, before authorization and
before anything reaches the backend, so an over-limit batch can't be
applied partially. Exactly 1000 keys still succeed, and an empty delete
list keeps its current behavior (200 with an empty result).

Fixes #2196

* fix: return MalformedXML for an oversized DeleteObjects request

A DeleteObjects request naming more than 1000 keys is answered with
MalformedXML, not InvalidRequest. Update the check and the unit and
integration expectations.

---------

Co-authored-by: Tung Lam <lamphamabtung96@gmail.com>
2026-09-24 08:22:36 -07:00
Ben McClellandandGitHub ca4445323f Merge pull request #2431 from versity/sis/region-mismatch-error-parity
fix: align signing region mismatch errors with s3 regional endpoints
2026-09-24 07:58:19 -07:00
Ben McClellandandGitHub 7a3636ce2a Merge pull request #2436 from versity/sis/null-version-position
fix: place the null version by its recorded position in posix versioning
2026-09-24 07:54:06 -07:00
niksis02 cdbebb06d6 fix: place the null version by its recorded position in posix versioning
`latestObjVersion` and `fileToObjVersions` placed the null version among the `ulid` versions by comparing file modification times, but a delete marker keeps the data and mtime of the version it hides. A null delete marker created in a versioning-suspended bucket therefore tied with the version below it and lost. Deleting the current version of an object whose history held a null delete marker restored the older version instead of the delete marker, and `ListObjectVersions` listed the delete marker below that version. Directory objects were exposed too, as a directory's mtime changes when objects are created under it.

`createObjVersion` now records the position of the null version when it moves it to the versioning directory. At that moment the null version is the current one, so it is newer than every version already there and older than every version created later. The id of the newest version in the directory is stored in the new `null-version-prev` attribute, and `nullVersionPos` compares it with the `ulid` version ids for both `DeleteObject` and `ListObjectVersions`. Null versions moved before this change carry no attribute and still fall back to the modification time. The attribute is not copied onto the object when a null version is restored.

`fileToObjVersions` also listed the null version without checking the `VersionIdMarker`, so it was repeated on every later page, and a `null` marker never matched, which dropped every version below it. The current version was listed again when it was the marker, so paging with the marker at a current version looped forever. The null version is now listed through `addNullVersion`, which skips it before the marker and continues after it when it is the marker, and the current version is no longer listed again when the marker names it.
2026-09-24 16:58:30 +04:00
Ben McClellandandGitHub 724e09502d Merge pull request #2435 from versity/sis/post-object-policy-object-arn
fix: authorize browser-based `POST` object uploads against the object ARN
2026-09-23 18:07:27 -07:00
Ben McClellandandGitHub a9877a25c0 Merge pull request #2434 from versity/sis/post-object-lock
fix: enforce object lock on browser-based `POST` object uploads
2026-09-23 18:05:55 -07:00
Ben McClellandandGitHub 9a63beb706 Merge pull request #2433 from versity/sis/never-versioned-bucket-delete-version
fix: keep the object on DeleteObject with a versionId in a never-versioned bucket
2026-09-23 18:04:02 -07:00
niksis02 0667514220 fix: authorize browser-based POST object uploads against the object ARN
`POSTObject` called `verifyAccess` without an `Object`, and `AuthorizePublicBucketAccess` took the object name from the request path, which is only `/bucket` for a `POST`. So both evaluated `s3:PutObject` against the bucket ARN `arn:aws:s3:::bucket`. `PutBucketPolicy` rejects `s3:PutObject` on a bucket resource, so no valid policy could ever match a `POST` upload. An `Allow` on `arn:aws:s3:::bucket/*` or on a key prefix never applied, and a `Deny` on a key prefix never applied either. A public-write ACL or a broad identity policy could therefore upload through `POST` to keys a `Deny` protects, which `PutObject` refuses. Identity policies evaluated by the standalone IAM service had the same mismatch.

S3 treats `POST` as an alternate form of `PUT` and authorizes it as `s3:PutObject` on the ARN of the object named by the form's `key` field. `POSTObject` now passes that `key` as `Object` to `verifyAccess`, matching `PutObject`. For anonymous `POST` requests, `AuthorizePublicBucketAccess` now uses the `key` that `AuthorizePostObject` has already parsed.
2026-09-23 22:53:54 +04:00
niksis02 1b9fedb7e6 fix: enforce object lock on browser-based POST object uploads
`POSTObject` wrote directly to the backend without the `CheckObjectAccess` object lock check that `PutObject`, `CopyObject` and `CompleteMultipartUpload` perform. In a versioning-disabled bucket, a browser-based `POST` could therefore silently overwrite an object protected by a legal hold, a `COMPLIANCE` retention or a `GOVERNANCE` retention. S3 has no such gap: it allows object lock only on versioned buckets, where an overwrite creates a new version.

`POSTObject` now calls `auth.CheckObjectAccess` with `BypassOverwrite` just before writing to the backend, matching `PutObject`. A legal hold or `COMPLIANCE` retention always blocks the overwrite. A `GOVERNANCE` retention blocks it unless the caller, root included, has an explicit `s3:BypassGovernanceRetention` grant. Versioning-enabled buckets skip the check because the upload creates a new version.
2026-09-23 18:13:25 +04:00
niksis02 7093e91b6e fix: keep the object on DeleteObject with a versionId in a never-versioned bucket
Posix `DeleteObject` only compared `versionId` with the object's version when the bucket had a versioning status. A bucket that was never versioned has none, so the request fell through to the plain delete, which ignored `versionId`. Any well-formed `versionId` then permanently deleted the object, on a gateway with or without `--versioning-dir`, and through `DeleteObjects` as well, since it calls `DeleteObject` for each key.

A never-versioned bucket only holds `null` versions, so any other `versionId` names a version that doesn't exist. S3 returns success for that, echoes `x-amz-version-id`, and deletes nothing. `DeleteObject` now does the same, returning early for any `versionId` other than `null`. `versionId=null` still deletes the object.
2026-09-23 15:17:49 +04:00
niksis02 241571abad fix: align signing region mismatch errors with s3 regional endpoints
Fixes #2239

Running the gateway with a `--region` other than `us-east-1` turned every request from a client signed for a different region into a dead end. The signing region check correctly rejected the request with `AuthorizationHeaderMalformed`, but the gateway gave the client no usable way to learn which region it should have signed for, so the aws sdks could not re-sign and retry the way they do against a real s3 regional endpoint.

The gap was `x-amz-bucket-region`. S3 reports the expected region in that response header as well as in the error body, because a `HEAD` request carries no body for the client to parse. The gateway set the header only on successful responses, so `HeadBucket` and `HeadObject` signed for the wrong region came back as a bare `400` with an empty body and some sdks region redirect had nothing to work with. Region mismatches now report the gateway region through a `RegionMismatchError` interface that `MalformedAuthError`, `AuthQueryParamError` and `InvalidArgumentError` implement, and the error dispatch points set the header from it alongside the existing `Allow` header handling for `MethodNotAllowed`.

The error bodies were incomplete in the same way. Only the `Authorization` header path carried a `Region` element, while the presigned url and POST form paths reported the mismatch in prose alone. `AuthorizationQueryParametersError` is now backed by a dedicated `AuthQueryParamError` type that adds the element, and `InvalidArgumentError` grew an optional `Region` field for the POST form credential error, both omitted for every other error of those kinds.

The messages themselves now read the way s3 writes them. The region and service names are quoted with apostrophes instead of `%q`, whose double quotes the xml encoder rendered as `&#34;` in the raw response, and the POST form authentication fields are echoed back in `ArgumentName` under their canonical spelling, for example `X-Amz-Credential` rather than the lowercase form the browser submits, while fields outside the signing protocol such as `key` are still reported as submitted.
2026-09-23 14:41:29 +04:00
Ben McClellandandGitHub e949c0fdf1 Merge pull request #2429 from versity/dependabot/github_actions/github/codeql-action-4.38.1
chore(deps): bump github/codeql-action from 4.38.0 to 4.38.1
2026-09-22 15:01:11 -07:00
Ben McClellandandGitHub 4ee4ea5979 Merge pull request #2430 from versity/dependabot/go_modules/dev-dependencies-e6a9953bab
chore(deps): bump the dev-dependencies group with 11 updates
2026-09-22 15:00:45 -07:00
Ben McClellandandGitHub 1b231d57fb Merge pull request #2427 from versity/sis/strip-aws-chunked-content-encoding
fix: strip `aws-chunked` from stored `Content-Encoding` on streaming uploads
2026-09-22 15:00:16 -07:00
niksis02 bbe398af72 fix: strip aws-chunked when the request declares no payload type
`ParseContentEncoding` dropped the `aws-chunked` token only for streaming payload types, so a presigned request kept it: such a request is signed as `UNSIGNED-PAYLOAD` and sends no `x-amz-content-sha256` header, and S3 strips the token there. Keep it only when the request declares a non streaming payload type.

S3 applies this to the `Content-Encoding` header before any API sees it, so `CopyObject` and `CreateMultipartUpload` follow the same rule even though neither carries a body to frame. `POSTObject` takes its value from a form field rather than the header and is left alone.
2026-09-22 02:05:00 +04:00
niksis02 888fd5c1ad fix: match S3's Content-Encoding error details for aws-chunked with UNSIGNED-PAYLOAD
The rejection now reports `Content-Encoding` as the `ArgumentName` and the bare `aws-chunked` token as the `ArgumentValue`, rather than `x-amz-content-sha256` and the payload type, so a request sending `gzip,aws-chunked` gets back just the offending coding, and the message carries S3's trailing period.

Adds integration tests for the three cases: `UnsignedStreamingPayloadTrailer_strips_aws_chunked_content_encoding` for a framed upload where only `aws-chunked` is dropped and the remaining codings keep their order, `PutObject_plain_stores_aws_chunked_content_encoding` for a hex-payload PUT that stores the token as sent, and `PutObject_unsigned_payload_with_aws_chunked_content_encoding` for the full error shape across four header spellings.
2026-09-22 02:04:34 +04:00
Barry Loongandniksis02 25f9790c0a fix(s3api): strip aws-chunked only for streaming uploads
- Replace `StripAwsChunkedEncoding` with `ParseContentEncoding`, which drops
  the token only when `x-amz-content-sha256` names a streaming payload type,
  so a client that sends `aws-chunked` on a hex-payload request keeps it
- Revert the `CopyObject` and `CreateMultipartUpload` call sites: both routes
  are registered with `streamBody` false, so a streaming payload type is
  rejected before the controller and the token can only be a stored value
- Reject `aws-chunked` combined with `UNSIGNED-PAYLOAD` in the authentication
  middleware, beside the existing payload-type validation, with a new
  `InvalidArgAwsChunkedUnsignedPayload`
- Add REST tests for the three cases: stripped from a chunked upload, kept on
  a hex-payload request, and rejected with `UNSIGNED-PAYLOAD`
- Add `CONTENT_ENCODING` to the PutObject REST script and a
  `check_content_encoding` driver for HeadObject
2026-09-22 02:03:09 +04:00
Barry Loongandniksis02 370d909372 fix: strip aws-chunked from stored Content-Encoding 2026-09-22 02:03:08 +04:00
dependabot[bot]andGitHub 3daad06d75 chore(deps): bump the dev-dependencies group with 11 updates
Bumps the dev-dependencies group with 11 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.33.4` | `1.33.5` |
| [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) | `1.20.4` | `1.20.5` |
| [github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager](https://github.com/aws/aws-sdk-go-v2) | `0.4.6` | `0.4.7` |
| [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) | `1.50.0` | `1.51.0` |
| [github.com/nats-io/nats.go](https://github.com/nats-io/nats.go) | `1.53.1` | `1.54.0` |
| [github.com/rabbitmq/amqp091-go](https://github.com/rabbitmq/amqp091-go) | `1.14.0` | `1.15.0` |
| [github.com/apache/arrow-go/v18](https://github.com/apache/arrow-go) | `18.7.0` | `18.8.0` |
| [github.com/gofiber/schema](https://github.com/gofiber/schema) | `1.8.6` | `1.8.7` |
| [github.com/gofiber/utils/v2](https://github.com/gofiber/utils) | `2.5.1` | `2.5.2` |
| [github.com/molecule-man/go-brrr](https://github.com/molecule-man/go-brrr) | `1.0.1` | `1.1.1` |
| [github.com/pierrec/lz4/v4](https://github.com/pierrec/lz4) | `4.1.29` | `4.1.30` |


Updates `github.com/aws/aws-sdk-go-v2/config` from 1.33.4 to 1.33.5
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.33.4...config/v1.33.5)

Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.20.4 to 1.20.5
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/mq/v1.20.4...service/mq/v1.20.5)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager` from 0.4.6 to 0.4.7
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/feature/s3/transfermanager/v0.4.6...feature/s3/transfermanager/v0.4.7)

Updates `github.com/aws/aws-sdk-go-v2/service/sts` from 1.50.0 to 1.51.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.50.0...service/s3/v1.51.0)

Updates `github.com/nats-io/nats.go` from 1.53.1 to 1.54.0
- [Release notes](https://github.com/nats-io/nats.go/releases)
- [Commits](https://github.com/nats-io/nats.go/compare/v1.53.1...v1.54.0)

Updates `github.com/rabbitmq/amqp091-go` from 1.14.0 to 1.15.0
- [Release notes](https://github.com/rabbitmq/amqp091-go/releases)
- [Changelog](https://github.com/rabbitmq/amqp091-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rabbitmq/amqp091-go/compare/v1.14.0...v1.15.0)

Updates `github.com/apache/arrow-go/v18` from 18.7.0 to 18.8.0
- [Release notes](https://github.com/apache/arrow-go/releases)
- [Commits](https://github.com/apache/arrow-go/compare/v18.7.0...v18.8.0)

Updates `github.com/gofiber/schema` from 1.8.6 to 1.8.7
- [Release notes](https://github.com/gofiber/schema/releases)
- [Commits](https://github.com/gofiber/schema/compare/v1.8.6...v1.8.7)

Updates `github.com/gofiber/utils/v2` from 2.5.1 to 2.5.2
- [Release notes](https://github.com/gofiber/utils/releases)
- [Commits](https://github.com/gofiber/utils/compare/v2.5.1...v2.5.2)

Updates `github.com/molecule-man/go-brrr` from 1.0.1 to 1.1.1
- [Release notes](https://github.com/molecule-man/go-brrr/releases)
- [Commits](https://github.com/molecule-man/go-brrr/compare/v1.0.1...v1.1.1)

Updates `github.com/pierrec/lz4/v4` from 4.1.29 to 4.1.30
- [Release notes](https://github.com/pierrec/lz4/releases)
- [Commits](https://github.com/pierrec/lz4/compare/v4.1.29...v4.1.30)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-version: 1.33.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-version: 1.20.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager
  dependency-version: 0.4.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-version: 1.51.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/nats-io/nats.go
  dependency-version: 1.54.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/rabbitmq/amqp091-go
  dependency-version: 1.15.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/apache/arrow-go/v18
  dependency-version: 18.8.0
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/gofiber/schema
  dependency-version: 1.8.7
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/gofiber/utils/v2
  dependency-version: 2.5.2
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/molecule-man/go-brrr
  dependency-version: 1.1.1
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/pierrec/lz4/v4
  dependency-version: 4.1.30
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-09-21 21:09:46 +00:00
dependabot[bot]andGitHub 8162344ca8 chore(deps): bump github/codeql-action from 4.38.0 to 4.38.1
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.38.0 to 4.38.1.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v4.38.0...v4.38.1)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.38.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-09-21 21:05:38 +00:00
Ben McClellandandGitHub db9539b472 Merge pull request #2422 from versity/ben/web-explorer-concurrent-uploads
feat: add webui concurrent uploads with progress tracking
2026-09-21 12:37:43 -07:00
Ben McClellandandGitHub 158f66424c Merge pull request #2413 from versity/sis/directory-objects-versioning
feat: add versioning support for directory objects in posix
2026-09-21 10:19:03 -07:00
Ben McClelland 7e9f556252 feat: add webui concurrent uploads with progress tracking
Queue uploads with a concurrency limit, track per-file and aggregate progress,
and preserve batch bucket/prefix context.

Limit concurrent uploads to three. Display aggregate byte progress as a
percentage and bytesDone / bytesTotal, completion and failure totals, and the
number of active uploads in the message box. Display Queued, Uploading, and
Failed files with individual progress bars; completed uploads remain in the
aggregate totals without filling the live file list.

Allow queued and active uploads to be cancelled from the progress widget. Remove
queued uploads immediately, abort active object and multipart part requests, and
abort a multipart upload on the server after its upload ID is available.
2026-09-21 09:52:41 -07:00
niksis02 e1d72907e4 fix: handle the null version id correctly in posix versioning
Fixes #2165

`GetObject` marked an object carrying no `versionIdKey` attribute as the null version and then overwrote that with the empty attribute value, and `HeadObject` never marked it at all, so neither reported a version id for an object put into a versioning-suspended bucket. Both now resolve it through a shared `liveObjVersionId`, which reports `null` only when the bucket has versioning configured so that a bucket that was never versioned keeps reporting no version id at all.

`HeadObject` also resolved an explicit `?versionId=null` into the versioning directory twice, once for the missing attribute and once for the mismatch that followed from it, and answered `404` for a version that `GetObject` served. It now treats the missing attribute as the null version the way `GetObject` does.

`latestObjVersion` took the last version directory entry as the newest, but `os.ReadDir` sorts by name and `null` sorts after every `ulid`. Deleting the current version of an object whose history also held a null version therefore restored `null` rather than the version created immediately before the deleted one. The `ulid` entries keep their name order, which is creation order, and the null version is placed by its modification time, the same rule `fileToObjVersions` already applies.
2026-09-21 20:34:47 +04:00
Ben McClellandandGitHub 97c997cb76 Merge pull request #2428 from versity/sis/cors-before-auth-all-routes
fix: apply bucket CORS right before auth middleware
2026-09-21 09:30:43 -07:00
Ben McClellandandGitHub ff05358f0b Merge pull request #2423 from versity/ben/dashboard-err
fix: hide webui user management when server in single-user mode
2026-09-21 09:16:35 -07:00
Ben McClellandandGitHub e2b7b33bac Merge pull request #2420 from versity/ben/mpu-contentlength
fix: use content-legnth for PutObject and UploadPart allocation sizes
2026-09-21 09:16:14 -07:00
niksis02 d313772685 fix: apply bucket CORS right before auth middleware
Move `applyBucketCORS` directly after `BucketObjectNameValidator` in every route so error responses still include CORS headers.
2026-09-21 17:50:13 +04:00
Sis NikoyanandGitHub 4a5c89200c Merge pull request #2416 from versity/sis/fix-trailing-slash-key-collisions
fix: stop posix mixing up keys that differ only by a trailing slash
2026-09-21 16:58:55 +04:00
Sis NikoyanandGitHub 421888caf3 Merge pull request #2417 from versity/sis/fix-copy-from-delete-marker
fix: stop posix copying data from a delete marker
2026-09-21 16:40:38 +04:00
Sis NikoyanandGitHub 7905ce6c19 Merge pull request #2418 from versity/sis/fix-copy-object-to-itself-versioning
fix: stop posix rewriting the current version on a copy to itself
2026-09-21 15:50:27 +04:00
Ben McClellandandGitHub 38f12c85fa Merge pull request #2421 from versity/ben/rdma-legacy-headers
fix: remove legacy rdma header descriptors
2026-09-20 13:15:06 -07:00
Ben McClelland 3db22f3d38 fix: hide webui user management when server in single-user mode
Single-root deployments reject /list-users with XAdminMethodNotSupported.
Treat that response as a capability signal so the dashboard suppresses
the expected error and removes unsupported user-management controls.
2026-09-20 13:07:11 -07:00
Ben McClelland 9c98084309 fix: remove legacy rdma header descriptors
These headers are not used by the cuda cuObject client/server
interactions, so are not needed in this server implementation.
2026-09-20 12:15:43 -07:00
Ben McClelland 20a5778aac fix: use content-legnth for PutObject and UploadPart allocation sizes
X-Amz-Decoded-Content-Length describes decoded aws-chunked payloads and must
not control a plain request. Using it for a plain UploadPart lets a larger
decoded header drive backend preallocation; the posix multipart path does not
truncate the staged part, so completion publishes a zero-filled tail.

Select the decoded length only for aws-chunked readers. Plain PutObject and
UploadPart now use Content-Length for both short-body validation and backend
allocation.
2026-09-20 11:46:23 -07:00
DavvyyandGitHub d03ac3c299 fix: reject a PutObject whose body ends before Content-Length
* fix(s3api): reject a PUT whose body ends before Content-Length

With a plain (non-aws-chunked) request body, a PutObject whose body ends
before Content-Length bytes have arrived is committed as a complete but
shorter object. Real S3 rejects this with IncompleteBody and the object
never becomes visible.

Nothing on that path compares bytes received against Content-Length:
fasthttp reports a connection closed mid-body as a plain io.EOF (the
conversion to io.ErrUnexpectedEOF exists only on the chunked
transfer-encoding branch), the authentication middleware leaves an
UNSIGNED-PAYLOAD body unwrapped, and io.Copy treats io.EOF as a normal
end of stream.

ErrIncompleteBody already exists and is enforced by the chunk readers.
This adds the equivalent check for plain bodies, in the controller so
that every backend is covered, and only when the body is not already an
aws-chunked reader.

POST-Object is deliberately untouched: it is a separate handler whose
ContentLength is an upper bound, so a byte-count check there would
break browser form uploads.

* test(s3api): cover ContentLengthReader

Complete and empty bodies, truncated bodies (including an EOF delivered
together with the final bytes, and one byte per Read), a body longer
than announced, and pass-through of a non-EOF error.

* fix(s3api): check plain bodies against Content-Length, not the decoded length

Review catch: the controller replaces contentLength with
X-Amz-Decoded-Content-Length whenever that header is present, regardless
of payload type, and the previous commit fed that value to
ContentLengthReader. A complete plain upload whose decoded header is
larger than Content-Length was then rejected with IncompleteBody, where
it had succeeded before.

AWS S3 ignores X-Amz-Decoded-Content-Length on a plain body and stores
Content-Length bytes. The decoded length only describes aws-chunked
payloads, and those skip this wrapper anyway, so the check now reads the
raw Content-Length header.

If the header is missing or unparseable the body is left unwrapped: an
aws-chunked request without Content-Length is already rejected earlier
with ErrMissingContentLength, so there is nothing to check here.

* test(integration): cover aborted uploads and the decoded-length case

Three tests, as requested in review.

PutObject_aborted_plain_body       plain PUT, body ends early -> no object
PutObject_aborted_streaming_body   same for an aws-chunked upload
PutObject_plain_body_with_decoded_length
                                   complete plain PUT carrying
                                   X-Amz-Decoded-Content-Length still
                                   succeeds and stores Content-Length bytes

The abort is driven by a reader that fails partway, so the transport
tears the connection down mid-body - that is what a client that dies or
cancels looks like on the wire. Sending a short body with a normal
reader would not reproduce it: net/http would simply report the
mismatch itself.

The third test is the regression the review found. It fails without the
accompanying fix.

* test(integration): replace the bogus streaming abort test with a real one

The reviewer is right: putObjectAborted always sends
x-amz-content-sha256: UNSIGNED-PAYLOAD, and the gateway decides aws-chunked
from that header alone, so PutObject_aborted_streaming_body took the same
plain path as the test above it. It duplicated the plain test instead of
covering the streaming one.

Dropped it and added UnsignedStreamingPayloadTrailer_aborted_connection in
tests/integration/unsigned_streaming_payload_trailer.go, where it belongs.

UnsignedStreamingPayloadTrailer_incomplete_body already covers malformed and
truncated framing, but every case there is a COMPLETE request. The new test is
the other shape: valid framing whose bytes simply stop arriving.
2026-09-20 11:18:15 -07:00
915fc7a20f fix: return InvalidArgument for an empty website configuration
* fix: return InvalidArgument for an empty website configuration

`PutBucketWebsite` parsed an empty `<WebsiteConfiguration/>` into a zero-value config, and `WebsiteConfiguration.Validate` reported the nil `IndexDocument` as `MalformedXML`. S3 reports the missing index document as an `InvalidArgument` request error naming the argument instead:

    InvalidArgument: A value for IndexDocument Suffix must be provided if RedirectAllRequestsTo is empty

That branch now returns the error with `ArgumentName=IndexDocument` and `ArgumentValue=null`, matching the response in the report. Genuinely malformed XML, and a `RedirectAllRequestsTo` that conflicts with the other fields, still return `MalformedXML`.

Fixes #2260

* test: cover an empty website configuration in the PutBucketWebsite integration suite

Add a `PutBucketWebsite_empty_configuration` case that sends the empty `<WebsiteConfiguration/>` payload from the report and asserts the `InvalidArgument` response, including `ArgumentName=IndexDocument` and `ArgumentValue=null`.

The SDK collapses the error into a generic API error that drops the argument fields, so the request is signed by hand and the raw response checked with `checkHTTPResponseApiErr`, the same helper the other argument-field assertions use. The case fails against the previous behaviour with `expected error code to be InvalidArgument, instead got MalformedXML`.

---------

Co-authored-by: Tung Lam <lamphamabtung96@gmail.com>
2026-09-20 11:04:22 -07:00
Ben McClellandandGitHub f1fa2a2e48 Merge pull request #2405 from iRioxe/fix-cors-check
Hidden actual error message cause of CORS check Error on PUT/<bucket-name>?object-lock=
2026-09-20 11:03:51 -07:00
niksis02 ba02c766ab fix: stop posix rewriting the current version on a copy to itself
`CopyObject` treated a destination path equal to the source path as an in-place metadata rewrite, regardless of bucket versioning. On a versioned bucket a self copy therefore edited the current version and returned its existing version id instead of creating a new one. This also defeated object lock: `CheckObjectAccess` skips the retention and legal hold checks for overwrites on version-enabled buckets because an overwrite is expected to create a new version, so a `COMPLIANCE` retained or legal held version had its metadata replaced underneath it. A self copy is now rewritten in place only when the bucket is unversioned, and otherwise goes through the regular copy path, which creates a new version and leaves the one it replaces untouched.

The rejection of a self copy that replaces nothing moved out of the in-place branch and is now applied only when the copy source carries no version id. Naming a version explicitly makes the request a regular copy, which AWS accepts even with the `COPY` metadata directive, while versitygw answered `InvalidRequest`.

Source tagging for a `COPY` tagging directive is now read before `PutObject` writes the destination rather than after. A self copy replaces the source object's attributes, so the later read returned nothing and the tags were dropped from the new version.
2026-09-20 17:40:19 +04:00
niksis02 d204d2e238 fix: stop posix copying data from a delete marker
`CopyObject` and `UploadPartCopy` never looked at the source's `delete-marker` attribute. A delete marker leaves the data file in place at the object path and only flags it, so a copy whose source resolved to a marker opened that file and succeeded, handing back the data of the version the marker had deleted.

AWS rejects such a copy: `NoSuchKey` when the marker is the current version of the key, and `InvalidRequest` when the marker is named by version id, the latter regardless of whether it is the latest version. Versions the marker hides stay copyable by version id.

Both copy paths now run the resolved source through `checkCopySourceDeleteMarker` once the entry has been validated, returning `NoSuchKey` for an unqualified source and the new `ErrCopySourceDeleteMarker` for one carrying a version id.
2026-09-20 15:35:08 +04:00
niksis02 bd6cfe64a3 fix: stop posix mixing up keys that differ only by a trailing slash
In posix a key and the same key with a trailing slash, such as `foo` and `foo/`, map to one path and share one set of attributes: a file there is the object of `foo`, a directory object is the object of `foo/`. Several operations used the entry at that path without checking that it belongs to the requested key. They now check it through the new `isLiveObject`, `statLiveObject` and `objVersionAttrPath` helpers.

`DeleteObject` of `foo` with a version id of `foo/` treated the directory as the current version of `foo` and removed it. That left the noncurrent versions of `foo/` unlisted and undeletable, so `DeleteBucket` failed with `BucketNotEmpty`. A delete of `foo` without a version id failed with an internal error while trying to version the directory as a file. Both now succeed without touching `foo/`, as for any object that doesn't exist.

`CompleteMultipartUpload` of `foo` cleared the `delete-marker` attribute of `foo/` and then failed with an internal error when linking the object onto the directory, which turned the delete marker back into a live version. It now returns `ExistingObjectIsDirectory` before any attribute is changed, both before the parts are assembled and again under the object publish lock. The idempotent completion path also no longer reports a missing upload as completed just because `foo/` exists.

The object tagging, legal hold and retention APIs read and wrote the attributes of the other key. For example, `PutObjectLegalHold` on `foo` could turn off the legal hold of `foo/`, and the object lock check could block a delete because of the other key's retention. When the requested key has no object, they now return `NoSuchKey` for the current version and `NoSuchVersion` for a specific version. They also resolve the `null` version id to a current null version instead of looking for it in the versioning directory. The object lock check treats `NoSuchVersion` like `NoSuchKey`, since a version that doesn't exist has nothing to protect. Conditional writes no longer evaluate `If-Match` and `If-None-Match` against the other key's ETag. `CreateMultipartUpload` now stores the upload's tagging and object lock settings directly on the upload directory, and `PutObject` of a directory object sets its tagging after the directory gets its ETag.
2026-09-19 22:26:47 +04:00
niksis02 729ced91e9 fix: stop posix publish lock serializing the same key across buckets
`lockObjectPublish` picked the process-local slot by the key's shard alone, while the shared lock file is per bucket and shard. As a result, writes to the same key name in different buckets waited on each other even though their lock files never conflict.

The slot is now chosen from the bucket hash and the shard (`bucketHash[0] ^ shard`). It still depends only on the lock file, never on the full key, so all requests for one lock file keep sharing a slot, which fcntl locks need because they don't exclude within a process.
2026-09-19 03:11:40 +04:00
niksis02 23317c766c feat: add versioning support for directory objects in posix
Directory objects (keys ending in `/`) are now versioned in versioning-enabled posix buckets like regular objects. The directory is kept in place across versions and carries the `version-id` and `delete-marker` attributes, while older versions are stored as empty files in the versioning directory. Only the attributes listed in `dirObjectAttrs` are copied into versions, replaced on a new `PutObject`, and restored when the latest version is deleted, so attributes the directory carries as a parent of other objects are left untouched.

`PutObject` on a directory object now snapshots the current version, starts the new version with a clean attribute set and returns a `VersionId`. `DeleteObject` creates delete markers for directory objects and, when deleting the latest version, restores the previous one onto the directory, or removes the directory (or keeps it as a plain parent of its children) when no versions are left. `ListObjectVersions` lists directory object versions and delete markers and uses the key with the trailing slash for `NextKeyMarker` and marker comparisons, `ListObjects` hides directory delete markers and `GetObject` returns the directory object version id.

`PutObject` on directory objects now applies the object lock legal hold and retention headers, and `CopyObject` and `UploadPartCopy` accept a directory object as a 0-byte copy source.

The versioning integration tests are extended to run their scenarios for directory objects as well, with new tests for directory-object-specific behavior.
2026-09-19 02:10:34 +04:00
Ben McClellandandGitHub 1246a08cfa Merge pull request #2411 from versity/iRioxe/fix-deleted-file-hang
fix: clear stale delete-marker metadata on reupload
2026-09-16 16:27:42 -07:00
iRioxeandBen McClelland 00fa54d391 fix: clear stale delete-marker metadata on reupload
When versioning uses sidecar metadata, deleting the current object can leave
the delete-marker attribute behind after the data file is removed. A later
PUT or multipart completion using the same key then inherits that stale
marker and the object remains hidden from the normal object view.

Clear the marker when publishing a replacement, including orphaned sidecar
and suspended-versioning cases, and remove all object metadata when deleting
the final version without an older version to restore.

Co-authored-by: Ben McClelland <ben.mcclelland@versity.com>
2026-09-16 16:05:03 -07:00
Ben McClellandandGitHub bce4ad4870 Merge pull request #2410 from versity/sis/helm-webui-prefix-iam-storage
feat: admin API path prefix, Helm WebUI prefixes and separate IAM storage
2026-09-16 15:37:41 -07:00
Ben McClellandandGitHub 2e28162d1f Merge pull request #2406 from versity/sis/posix-preexisting-dataset-fixes
fix: treat root-level non-directory entries as non-existing buckets in posix
2026-09-16 15:31:08 -07:00
niksis02 966a6a8650 feat: admin API path prefix, Helm WebUI prefixes and separate IAM storage
Closes #2322

The admin API could not be served under a path prefix. Admin requests are SigV4-signed over the full path, so a reverse proxy or Gateway API route cannot strip a prefix before forwarding. That made it impossible to serve the WebUI and the admin API under one hostname. The new `--admin-path-prefix` (`VGW_ADMIN_PATH_PREFIX`) option mounts the admin routes under a single-segment prefix such as `/admin`, on `--admin-port` or, when that is unset, on the S3 port. Admin clients include the prefix in their endpoint URL, which the admin CLI and the WebUI already support.

The prefix is limited to unreserved characters, because clients and the gateway encode other characters differently when signing. When the admin API shares the S3 port, the prefix must also differ from `--webui-s3-prefix`, otherwise the WebUI mount answers the admin requests. Auto-detected WebUI admin gateway URLs and the startup banner now include the prefix. The admin routes served on the S3 port now reuse `S3AdminRouter` instead of a duplicated route list. This also stops the standalone admin server from recording bucket creation under the `ActionAdminListBuckets` action.

The Helm chart now exposes `webui.pathPrefix`, `webui.s3Prefix` and `admin.pathPrefix`. The WebUI gateway lists are also passed when only `webui.s3Prefix` is set, since the WebUI hosted on the S3 port does not require `webui.enabled`.

Internal IAM data can now live on its own volume. `iam.dir` sets the IAM directory, and `iam.persistence` creates or references a dedicated PVC when `iam.enabled` is true and `iam.type` is `internal`. Otherwise IAM data stays in the `iam` subdirectory of the backend data volume, so existing releases are unaffected.

The chart now rejects overlapping storage directories. `iam.dir` must be an absolute path outside `/mnt/data`, `gateway.backend.sidecarDir` and `gateway.backend.versioningDir`, and those three directories must not overlap each other. Deleting a bucket removes `<dir>/<bucket>` from the sidecar and versioning directories, so with nested directories, deleting a suitably named bucket could wipe IAM accounts, object versions or the entire backend data directory.
2026-09-16 23:09:14 +04:00
niksis02 cc48b7568c fix: treat root-level non-directory entries as non-existing buckets in posix
A posix gateway serving a preexisting dataset treats every directory under the root as a bucket. But the bucket checks only stat the path, so a regular file, FIFO or symlink at the root passed for a bucket. Only `ListBuckets` skipped these entries.

With a root-level file as the bucket, `HeadBucket`, `ListObjects`, `ListObjectsV2`, `ListMultipartUploads`, `GetBucketAcl`, `GetBucketLocation`, `GetBucketVersioning` and the bucket config getters returned a successful response. `PutBucketAcl`, `PutBucketTagging`, `PutBucketPolicy`, `PutBucketCors`, ... and the admin `ChangeBucketOwner` also succeeded, storing gateway metadata on the file itself or in the sidecar directory. `DeleteObject` and `DeleteObjects` reported success. `CreateMultipartUpload`, `UploadPart`, `UploadPartCopy`, `ListParts`, `CompleteMultipartUpload`, `ListObjectVersions` and `DeleteBucket` returned `InternalError` with not a directory. `PutObject` and `CopyObject` returned `ObjectParentIsFile`. `GetObject`, `HeadObject`, `GetObjectAttributes` and the object tagging, legal hold and retention actions returned `NoSuchKey`, and `AbortMultipartUpload` returned `NoSuchUpload`. A symlink loop at the root returned `InternalError` for every action.

Bucket existence is now checked in one place, `doesBucketExist`, and `ScoutFS` uses it too through the exported `DoesBucketExist`. A bucket is a directory under the root, or a symlink to a directory when `--bucketlinks` is enabled. Any other entry returns `NoSuchBucket` for all bucket and object actions. `CreateBucket` on such an entry returns `BucketAlreadyExists` without reading its metadata.

This also enforces `--bucketlinks` for all actions. Before, it only affected `ListBuckets`: a symlinked directory was hidden from the listing, but every other action followed the link and used it as a bucket even with the option disabled. Without `--bucketlinks` such symlinks now return `NoSuchBucket`.
2026-09-16 20:14:09 +04:00