Posix `DeleteObject` only compared `versionId` with the object's version when the bucket had a versioning status. A bucket that was never versioned has none, so the request fell through to the plain delete, which ignored `versionId`. Any well-formed `versionId` then permanently deleted the object, on a gateway with or without `--versioning-dir`, and through `DeleteObjects` as well, since it calls `DeleteObject` for each key.
A never-versioned bucket only holds `null` versions, so any other `versionId` names a version that doesn't exist. S3 returns success for that, echoes `x-amz-version-id`, and deletes nothing. `DeleteObject` now does the same, returning early for any `versionId` other than `null`. `versionId=null` still deletes the object.
`ParseContentEncoding` dropped the `aws-chunked` token only for streaming payload types, so a presigned request kept it: such a request is signed as `UNSIGNED-PAYLOAD` and sends no `x-amz-content-sha256` header, and S3 strips the token there. Keep it only when the request declares a non streaming payload type.
S3 applies this to the `Content-Encoding` header before any API sees it, so `CopyObject` and `CreateMultipartUpload` follow the same rule even though neither carries a body to frame. `POSTObject` takes its value from a form field rather than the header and is left alone.
The rejection now reports `Content-Encoding` as the `ArgumentName` and the bare `aws-chunked` token as the `ArgumentValue`, rather than `x-amz-content-sha256` and the payload type, so a request sending `gzip,aws-chunked` gets back just the offending coding, and the message carries S3's trailing period.
Adds integration tests for the three cases: `UnsignedStreamingPayloadTrailer_strips_aws_chunked_content_encoding` for a framed upload where only `aws-chunked` is dropped and the remaining codings keep their order, `PutObject_plain_stores_aws_chunked_content_encoding` for a hex-payload PUT that stores the token as sent, and `PutObject_unsigned_payload_with_aws_chunked_content_encoding` for the full error shape across four header spellings.
Fixes#2165
`GetObject` marked an object carrying no `versionIdKey` attribute as the null version and then overwrote that with the empty attribute value, and `HeadObject` never marked it at all, so neither reported a version id for an object put into a versioning-suspended bucket. Both now resolve it through a shared `liveObjVersionId`, which reports `null` only when the bucket has versioning configured so that a bucket that was never versioned keeps reporting no version id at all.
`HeadObject` also resolved an explicit `?versionId=null` into the versioning directory twice, once for the missing attribute and once for the mismatch that followed from it, and answered `404` for a version that `GetObject` served. It now treats the missing attribute as the null version the way `GetObject` does.
`latestObjVersion` took the last version directory entry as the newest, but `os.ReadDir` sorts by name and `null` sorts after every `ulid`. Deleting the current version of an object whose history also held a null version therefore restored `null` rather than the version created immediately before the deleted one. The `ulid` entries keep their name order, which is creation order, and the null version is placed by its modification time, the same rule `fileToObjVersions` already applies.
X-Amz-Decoded-Content-Length describes decoded aws-chunked payloads and must
not control a plain request. Using it for a plain UploadPart lets a larger
decoded header drive backend preallocation; the posix multipart path does not
truncate the staged part, so completion publishes a zero-filled tail.
Select the decoded length only for aws-chunked readers. Plain PutObject and
UploadPart now use Content-Length for both short-body validation and backend
allocation.
* fix(s3api): reject a PUT whose body ends before Content-Length
With a plain (non-aws-chunked) request body, a PutObject whose body ends
before Content-Length bytes have arrived is committed as a complete but
shorter object. Real S3 rejects this with IncompleteBody and the object
never becomes visible.
Nothing on that path compares bytes received against Content-Length:
fasthttp reports a connection closed mid-body as a plain io.EOF (the
conversion to io.ErrUnexpectedEOF exists only on the chunked
transfer-encoding branch), the authentication middleware leaves an
UNSIGNED-PAYLOAD body unwrapped, and io.Copy treats io.EOF as a normal
end of stream.
ErrIncompleteBody already exists and is enforced by the chunk readers.
This adds the equivalent check for plain bodies, in the controller so
that every backend is covered, and only when the body is not already an
aws-chunked reader.
POST-Object is deliberately untouched: it is a separate handler whose
ContentLength is an upper bound, so a byte-count check there would
break browser form uploads.
* test(s3api): cover ContentLengthReader
Complete and empty bodies, truncated bodies (including an EOF delivered
together with the final bytes, and one byte per Read), a body longer
than announced, and pass-through of a non-EOF error.
* fix(s3api): check plain bodies against Content-Length, not the decoded length
Review catch: the controller replaces contentLength with
X-Amz-Decoded-Content-Length whenever that header is present, regardless
of payload type, and the previous commit fed that value to
ContentLengthReader. A complete plain upload whose decoded header is
larger than Content-Length was then rejected with IncompleteBody, where
it had succeeded before.
AWS S3 ignores X-Amz-Decoded-Content-Length on a plain body and stores
Content-Length bytes. The decoded length only describes aws-chunked
payloads, and those skip this wrapper anyway, so the check now reads the
raw Content-Length header.
If the header is missing or unparseable the body is left unwrapped: an
aws-chunked request without Content-Length is already rejected earlier
with ErrMissingContentLength, so there is nothing to check here.
* test(integration): cover aborted uploads and the decoded-length case
Three tests, as requested in review.
PutObject_aborted_plain_body plain PUT, body ends early -> no object
PutObject_aborted_streaming_body same for an aws-chunked upload
PutObject_plain_body_with_decoded_length
complete plain PUT carrying
X-Amz-Decoded-Content-Length still
succeeds and stores Content-Length bytes
The abort is driven by a reader that fails partway, so the transport
tears the connection down mid-body - that is what a client that dies or
cancels looks like on the wire. Sending a short body with a normal
reader would not reproduce it: net/http would simply report the
mismatch itself.
The third test is the regression the review found. It fails without the
accompanying fix.
* test(integration): replace the bogus streaming abort test with a real one
The reviewer is right: putObjectAborted always sends
x-amz-content-sha256: UNSIGNED-PAYLOAD, and the gateway decides aws-chunked
from that header alone, so PutObject_aborted_streaming_body took the same
plain path as the test above it. It duplicated the plain test instead of
covering the streaming one.
Dropped it and added UnsignedStreamingPayloadTrailer_aborted_connection in
tests/integration/unsigned_streaming_payload_trailer.go, where it belongs.
UnsignedStreamingPayloadTrailer_incomplete_body already covers malformed and
truncated framing, but every case there is a COMPLETE request. The new test is
the other shape: valid framing whose bytes simply stop arriving.
* fix: return InvalidArgument for an empty website configuration
`PutBucketWebsite` parsed an empty `<WebsiteConfiguration/>` into a zero-value config, and `WebsiteConfiguration.Validate` reported the nil `IndexDocument` as `MalformedXML`. S3 reports the missing index document as an `InvalidArgument` request error naming the argument instead:
InvalidArgument: A value for IndexDocument Suffix must be provided if RedirectAllRequestsTo is empty
That branch now returns the error with `ArgumentName=IndexDocument` and `ArgumentValue=null`, matching the response in the report. Genuinely malformed XML, and a `RedirectAllRequestsTo` that conflicts with the other fields, still return `MalformedXML`.
Fixes#2260
* test: cover an empty website configuration in the PutBucketWebsite integration suite
Add a `PutBucketWebsite_empty_configuration` case that sends the empty `<WebsiteConfiguration/>` payload from the report and asserts the `InvalidArgument` response, including `ArgumentName=IndexDocument` and `ArgumentValue=null`.
The SDK collapses the error into a generic API error that drops the argument fields, so the request is signed by hand and the raw response checked with `checkHTTPResponseApiErr`, the same helper the other argument-field assertions use. The case fails against the previous behaviour with `expected error code to be InvalidArgument, instead got MalformedXML`.
---------
Co-authored-by: Tung Lam <lamphamabtung96@gmail.com>
`CopyObject` treated a destination path equal to the source path as an in-place metadata rewrite, regardless of bucket versioning. On a versioned bucket a self copy therefore edited the current version and returned its existing version id instead of creating a new one. This also defeated object lock: `CheckObjectAccess` skips the retention and legal hold checks for overwrites on version-enabled buckets because an overwrite is expected to create a new version, so a `COMPLIANCE` retained or legal held version had its metadata replaced underneath it. A self copy is now rewritten in place only when the bucket is unversioned, and otherwise goes through the regular copy path, which creates a new version and leaves the one it replaces untouched.
The rejection of a self copy that replaces nothing moved out of the in-place branch and is now applied only when the copy source carries no version id. Naming a version explicitly makes the request a regular copy, which AWS accepts even with the `COPY` metadata directive, while versitygw answered `InvalidRequest`.
Source tagging for a `COPY` tagging directive is now read before `PutObject` writes the destination rather than after. A self copy replaces the source object's attributes, so the later read returned nothing and the tags were dropped from the new version.
`CopyObject` and `UploadPartCopy` never looked at the source's `delete-marker` attribute. A delete marker leaves the data file in place at the object path and only flags it, so a copy whose source resolved to a marker opened that file and succeeded, handing back the data of the version the marker had deleted.
AWS rejects such a copy: `NoSuchKey` when the marker is the current version of the key, and `InvalidRequest` when the marker is named by version id, the latter regardless of whether it is the latest version. Versions the marker hides stay copyable by version id.
Both copy paths now run the resolved source through `checkCopySourceDeleteMarker` once the entry has been validated, returning `NoSuchKey` for an unqualified source and the new `ErrCopySourceDeleteMarker` for one carrying a version id.
In posix a key and the same key with a trailing slash, such as `foo` and `foo/`, map to one path and share one set of attributes: a file there is the object of `foo`, a directory object is the object of `foo/`. Several operations used the entry at that path without checking that it belongs to the requested key. They now check it through the new `isLiveObject`, `statLiveObject` and `objVersionAttrPath` helpers.
`DeleteObject` of `foo` with a version id of `foo/` treated the directory as the current version of `foo` and removed it. That left the noncurrent versions of `foo/` unlisted and undeletable, so `DeleteBucket` failed with `BucketNotEmpty`. A delete of `foo` without a version id failed with an internal error while trying to version the directory as a file. Both now succeed without touching `foo/`, as for any object that doesn't exist.
`CompleteMultipartUpload` of `foo` cleared the `delete-marker` attribute of `foo/` and then failed with an internal error when linking the object onto the directory, which turned the delete marker back into a live version. It now returns `ExistingObjectIsDirectory` before any attribute is changed, both before the parts are assembled and again under the object publish lock. The idempotent completion path also no longer reports a missing upload as completed just because `foo/` exists.
The object tagging, legal hold and retention APIs read and wrote the attributes of the other key. For example, `PutObjectLegalHold` on `foo` could turn off the legal hold of `foo/`, and the object lock check could block a delete because of the other key's retention. When the requested key has no object, they now return `NoSuchKey` for the current version and `NoSuchVersion` for a specific version. They also resolve the `null` version id to a current null version instead of looking for it in the versioning directory. The object lock check treats `NoSuchVersion` like `NoSuchKey`, since a version that doesn't exist has nothing to protect. Conditional writes no longer evaluate `If-Match` and `If-None-Match` against the other key's ETag. `CreateMultipartUpload` now stores the upload's tagging and object lock settings directly on the upload directory, and `PutObject` of a directory object sets its tagging after the directory gets its ETag.
Directory objects (keys ending in `/`) are now versioned in versioning-enabled posix buckets like regular objects. The directory is kept in place across versions and carries the `version-id` and `delete-marker` attributes, while older versions are stored as empty files in the versioning directory. Only the attributes listed in `dirObjectAttrs` are copied into versions, replaced on a new `PutObject`, and restored when the latest version is deleted, so attributes the directory carries as a parent of other objects are left untouched.
`PutObject` on a directory object now snapshots the current version, starts the new version with a clean attribute set and returns a `VersionId`. `DeleteObject` creates delete markers for directory objects and, when deleting the latest version, restores the previous one onto the directory, or removes the directory (or keeps it as a plain parent of its children) when no versions are left. `ListObjectVersions` lists directory object versions and delete markers and uses the key with the trailing slash for `NextKeyMarker` and marker comparisons, `ListObjects` hides directory delete markers and `GetObject` returns the directory object version id.
`PutObject` on directory objects now applies the object lock legal hold and retention headers, and `CopyObject` and `UploadPartCopy` accept a directory object as a 0-byte copy source.
The versioning integration tests are extended to run their scenarios for directory objects as well, with new tests for directory-object-specific behavior.
A posix gateway serving a preexisting dataset treats every directory under the root as a bucket. But the bucket checks only stat the path, so a regular file, FIFO or symlink at the root passed for a bucket. Only `ListBuckets` skipped these entries.
With a root-level file as the bucket, `HeadBucket`, `ListObjects`, `ListObjectsV2`, `ListMultipartUploads`, `GetBucketAcl`, `GetBucketLocation`, `GetBucketVersioning` and the bucket config getters returned a successful response. `PutBucketAcl`, `PutBucketTagging`, `PutBucketPolicy`, `PutBucketCors`, ... and the admin `ChangeBucketOwner` also succeeded, storing gateway metadata on the file itself or in the sidecar directory. `DeleteObject` and `DeleteObjects` reported success. `CreateMultipartUpload`, `UploadPart`, `UploadPartCopy`, `ListParts`, `CompleteMultipartUpload`, `ListObjectVersions` and `DeleteBucket` returned `InternalError` with not a directory. `PutObject` and `CopyObject` returned `ObjectParentIsFile`. `GetObject`, `HeadObject`, `GetObjectAttributes` and the object tagging, legal hold and retention actions returned `NoSuchKey`, and `AbortMultipartUpload` returned `NoSuchUpload`. A symlink loop at the root returned `InternalError` for every action.
Bucket existence is now checked in one place, `doesBucketExist`, and `ScoutFS` uses it too through the exported `DoesBucketExist`. A bucket is a directory under the root, or a symlink to a directory when `--bucketlinks` is enabled. Any other entry returns `NoSuchBucket` for all bucket and object actions. `CreateBucket` on such an entry returns `BucketAlreadyExists` without reading its metadata.
This also enforces `--bucketlinks` for all actions. Before, it only affected `ListBuckets`: a symlinked directory was hidden from the listing, but every other action followed the link and used it as a bucket even with the option disabled. Without `--bucketlinks` such symlinks now return `NoSuchBucket`.
* fix: omit ObjectLockConfiguration Rule when there is no default retention
ParseBucketLockConfigurationOutput always set Rule, so a bucket with object lock enabled and no default retention answered GET ?object-lock with an empty <Rule></Rule>. AWS S3 omits the element, and the AWS SDK v2 clients that read the rule from it follow the empty element with a malformed request.
Fixes#2397
* test: cover GetObjectLockConfiguration with no default retention in the integration suite and drop the stale non-nil Rule expectation from the controller unit test
---------
Co-authored-by: Tung Lam <lamphamabtung96@gmail.com>
S3 returns `InvalidArgument` when an object action receives a `versionId` query parameter with no value. The gateway silently treated it as an unversioned request instead.
Added a shared `versionId` validation helper and apply it to the object actions that accept the parameter, so malformed requests are rejected up front rather than reaching the backend.
Closes#2294
Support AWS's conditional-write condition keys in both S3 bucket policies and IAM identity policies, so a policy can require that an upload or delete carry an `If-Match`/`If-None-Match` precondition.
`requestConditionContext` now populates both keys from the request headers, which covers both policy types at once: the same map is evaluated in-process for bucket policies and shipped to the IAM service for identity policies. Only the three requests whose preconditions the gateway actually enforces contribute - PutObject, CompleteMultipartUpload and DeleteObject. Copies, form uploads, DeleteObjects batches, upload parts and the sub-resource writes all ignore the headers, and a policy must never grant on a precondition that won't be checked; reads are excluded for the same reason, since GET and HEAD take these headers as ordinary HTTP cache preconditions. The value is the ETag with its surrounding quotes stripped, matching what the gateway enforces against.
Bucket policies validate condition keys against a fixed catalogue at `PutBucketPolicy` time, so both keys are added there with the action sets AWS accepts: `s3:if-match` on `s3:PutObject` and `s3:DeleteObject`, `s3:if-none-match` on `s3:PutObject` alone. Identity policies validate only the operator vocabulary, matching AWS, so they need no change.
Bucket policy `Principal` named callers by access key id. Under the standalone IAM service it now names them by AWS-style ARN, as real S3 does: a user ARN, a role ARN (covering every session of that role), an assumed-role ARN (covering one session), the account root ARN or bare account id, or `*`. Every other IAM backend has no ARNs to name anything by and keeps access-key principals unchanged, gated on a new `auth.PrincipalResolver` capability interface that only the standalone client implements.
`auth.Account` carries `Arn` and `RoleArn`, filled at authentication time, so a session can be matched against both its own ARN and its role's. Principals are validated at PutBucketPolicy time through a new `/private/resolve-principals` endpoint, which rejects anything that does not name a live identity with `MalformedPolicy: Invalid principal in policy`.
An `Allow` naming the account root ARN or bare account id delegates to the account's own IAM rather than granting on its own, while a `Deny` naming it denies every principal in the account outright. Denial messages now name the caller by ARN wherever one exists.
Also fixes `aws:PrincipalArn` for assumed-role sessions, which reported the session ARN where AWS reports the role's, and stops an unreachable IAM service being reported as a malformed policy.
`CreateAccessKey`, `UpdateAccessKey`, `DeleteAccessKey` and `ListAccessKeys` required an explicit `UserName`. Real IAM treats it as optional and resolves it from the access key signing the request, so an IAM user managing its own keys had to name itself. They now infer it, matching AWS: only an entirely absent parameter is inferred, while a present-but-empty one stays a `ValidationError`, and the inferred scope is strictly the caller's own user — another user's key id returns `NoSuchEntity` rather than being touched.
A caller with no IAM user of its own gets IAM's own `Must specify userName when calling with non-User credentials` `ValidationError`, shared with `GetUser` as `iamerr.MustSpecifyUserName`. That covers assumed-role sessions and also the gateway's root credential, which is configured rather than stored as an IAM user and so owns no access keys the API could manage — real IAM manages the root account's own keys here, which has no equivalent in this gateway.
The policy middleware resolves the same four actions through `callerOrNamedUserResource`, so the resource-level check targets the caller's own user ARN when UserName is omitted instead of falling back to no resource at all, which would have denied every request authorized by an own-ARN-scoped grant.
Role last-used tracking was missing entirely - `GetRole` returned a `RoleLastUsed` element that nothing ever wrote, rendering the zero time instead of the empty element AWS returns for an unused role - and access key last-used only ever saw the `IAM`/`STS` control plane, so a credential used exclusively against the S3 gateway reported as never used. Roles now record a use whenever a request authenticates with one of their session credentials, through a new `Storer.RecordRoleUsage` mirroring `RecordAccessKeyUsage`, gated on the session's role still being the one it was minted against so a session outliving its role can't attribute its use to a same-named replacement. `LastUsedDate` became a `*time.Time` so an unused role renders as an empty element.
Both records now cover the S3 data plane as well: the gateway sends its configured region and `s3` on evaluate-policy and the IAM service records the caller there, so `GetAccessKeyLastUsed's` `ServiceName` is now iam, sts or s3. That call was chosen over derive-signing-key, which runs before signature verification and takes its region and service from the caller's own `Authorization` header - recording there would let anyone who knows an access key id refresh and poison another identity's audit record. Requests denied by a bucket policy or made against a public bucket are not recorded, since neither reaches identity-policy evaluation. To keep per-request recording affordable, an update is skipped while the stored record has the same service and region and is under a minute old; a change of either is written through immediately.
Assuming a role is not a use, a request denied by an identity policy is, and both successful and denied S3 requests update the record. Also moves the `OIDC-dependent` tests into the `s3-iam-session` group so runoidctests.sh runs a single group.
objectsAccessErrors recorded the first resource-policy Deny and returned immediately, leaving every later key in the same action subset with a nil result — which VerifyObjectsAccess caller reads as "authorized" and sends straight to the backend. An explicit bucket-policy Deny therefore let the keys after it skip authorization entirely and be deleted, including keys that same bucket policy explicitly denied.
The loop now continues rather than returning, so every denied key is settled with its own error. The identity-policy round trip the early return was saving is still skipped, but only when the bucket policy denied every key in the batch, since no identity-policy answer could change any result then. Both loops that follow skip keys already holding an error, so a resource-level explicit deny is never overwritten by an identity-policy result nor flattened to the generic AccessDenied message.
S3 explicitly documents this action as owner-only ("To retrieve the versioning state of a bucket, you must be the bucket owner." — https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketVersioning.html), and the handler enforced that with an extra auth.IsAdminOrOwner check on top of VerifyAccess. Real S3 behaves differently: verified against AWS that a bucket policy explicitly denying s3:GetBucketVersioning denies the bucket owner itself, and that an Allow grants the action to a principal that doesn't own the bucket. It goes through ordinary bucket policy/ACL evaluation like any other bucket subresource read, which is what the write side, PutBucketVersioning, already did here. Removes the extra check along with auth.IsAdminOrOwner, which had no other call site.
* feat(azure): carry Azure blob marker in continuation token for pagination
Azure blob markers are opaque values only Azure may mint, so an S3 object
key can never be passed back to Azure as a listing marker. Paging the
underlying Azure listing with an S3 key therefore failed or re-scanned the
whole prefix on every page.
Introduce azMarkerToken, an opaque S3 continuation token that carries the
Azure marker alongside the last returned key: the Azure marker resumes the
blob listing where it stopped, and the last key filters out already-returned
entries. Tokens are versioned with a "vgw1." prefix; anything without it is
treated as a plain key, so tokens from older versions and hand-crafted
markers keep working. The shared listBlobs helper now backs both ListObjects
and ListObjectsV2, applying the S3 marker and delimiter client side.
ListObjectsV2 pages efficiently via the token; ListObjects (v1) has no token
to carry state and walks the prefix from the start each page.
Add unit tests (token round-trip, marker-resumed pagination, delimiter and
common-prefix handling, multipart filtering) driven by a fake Azure
container, and integration tests covering full and delimited pagination.
Signed-off-by: Nils Leger <nils.leger@getflip.com>
* fix: token is now bound to delimiter too
Signed-off-by: Nils Leger <nils.leger@getflip.com>
---------
Signed-off-by: Nils Leger <nils.leger@getflip.com>
* feat(azure): implement server-side copy with fallback
Add server-side object copy for the Azure backend using StartCopyFromURL, with a
fallback to download+reupload when server-side copy is unavailable. The copy
logic lives in backend/azure/copy.go and handles metadata, tagging and object
lock configurations.
The copy-source SAS service version is configurable via the --copy-sas-version
flag (AZ_COPY_SAS_VERSION) and defaults to the SDK version, so production is
unchanged. Endpoints that lag the SDK's SAS version (e.g. Azurite) cannot verify
a SAS signed with the newer version and can set an older one. On a metadata-COPY,
the internal website-redirect key is dropped from the destination to match the
download+reupload fallback.
Testing:
- Add CopyObject_cross_bucket_server_side_copy, which copies an object with data,
user metadata, content-type and tags across two buckets and verifies all are
preserved and an ETag is returned.
- Configure the Azurite functional-test gateway with AZ_COPY_SAS_VERSION and let
Azurite trust its self-signed test certificate (NODE_EXTRA_CA_CERTS) so it can
fetch the copy source from its own HTTPS endpoint, ensuring CI exercises the
real server-side copy path instead of always falling back.
Signed-off-by: Nils Leger <nils.leger@getflip.com>
* docs: update copyright year
Signed-off-by: Nils Leger <nils.leger@getflip.com>
* fix: always fallback to download+upload whenever there is an error building the server-side copy URL
Signed-off-by: Nils Leger <nils.leger@getflip.com>
---------
Signed-off-by: Nils Leger <nils.leger@getflip.com>
Adds `TagOpenIDConnectProvider`, `UntagOpenIDConnectProvider` and `ListOpenIDConnectProviderTags` to the standalone IAM service, backed by both the internal and Vault storers. They follow the user and role tagging actions in most respects — the tag action merges into the provider's existing tags and rejects a repeated key, untag removal is idempotent, and the tag listing is sorted by key and paginated, with the per-request member count and the per-provider tag total enforced as separate quotas so replacing a tag on a provider already at the 50-tag cap still succeeds — but differ in the one respect IAM itself draws: OIDC provider tag keys are compared exactly, not case-insensitively. On a provider `env` and `ENV` are two independent tags, both may be supplied in a single request, only a byte-identical repeat is a duplicate (reported without the "Tag keys are case insensitive" note the user and role actions carry), and untagging `env` leaves `ENV` in place.
That distinction is now carried by `iamutil.TagKeyCase`, which `ParseTags` uses for duplicate detection and which `mergeTags`, `removeTags` and the tag listing's marker lookup use for key matching. `CreateOpenIDConnectProvider` moves onto the exact comparison too, so a provider created with case-differing tag keys keeps both.
All three actions are authorized against the target provider's ARN, so `aws:ResourceTag/<key>` reads the provider's own tags, and the tag and untag actions populate `aws:RequestTag/<key>` and `aws:TagKeys` respectively, so a tag-scoped policy Condition governs which tags a caller may set or remove. All three report a missing provider with the wording `DeleteOpenIDConnectProvider` uses rather than the one `GetOpenIDConnectProvider` uses, which is why the Vault provider read now takes the not-found error its calling action reports.
The WebGUI gains a Tags section in the OIDC provider manage view, replacing the read-only tag row, and the shared tag editor gains a case-sensitive mode that changes its duplicate-key check, its diffing of an edited set into an untag and tag pair, and the wording of its guidance.
Adds `TagRole`, `UntagRole` and `ListRoleTags` to the standalone IAM service, backed by both the internal and Vault storers, with the same semantics the user tagging actions already have: tag keys are matched case-insensitively but stored case-preserving, TagRole merges into the role's existing tags and rejects duplicate keys, UntagRole removal is idempotent, and ListRoleTags is sorted by key and paginated. The per-request member count and the per-role tag total are enforced as separate quotas, so replacing a tag on a role already at the 50-tag cap still succeeds. Role tags and the tags of a user sharing the same name are independent sets.
All three actions are authorized against the target role's ARN, so `aws:ResourceTag/<key>` reads the role's own tags, and TagRole and UntagRole populate `aws:RequestTag/<key>` and `aws:TagKeys` respectively, so a tag-scoped policy Condition governs which tags a caller may set or remove.
The tag storage helpers are now shared between users and roles: `MaxTagsPerUser` becomes `MaxTagsPerResource`, `ListUserTagsOutput` becomes `ListTagsOutput`, and `paginateTags` takes the marker and page size directly instead of a user-specific input struct.
The WebGUI gains a Tags section in the IAM role manage view, reusing the tag editor the user view already uses, which applies a whole edited tag set as a single UntagRole and TagRole pair.
Also corrects the `roleName` length bound across every role action: it was validated against the 128-character user-lookup limit, where IAM caps role names at 64.
Adds `TagUser`, `UntagUser` and `ListUserTags` to the standalone IAM service, backed by both the internal and Vault storers. Tag keys are matched case-insensitively but stored case-preserving, TagUser merges into the user's existing tags and rejects duplicate keys, UntagUser removal is idempotent, and ListUserTags is sorted by key and paginated. The per-request member count and the per-user tag total are enforced as separate quotas.
All three actions are authorized against the target user's ARN, and TagUser and UntagUser populate aws:RequestTag/<key> and aws:TagKeys respectively, so a tag-scoped policy Condition governs which tags a caller may set or remove.
The WebGUI gains a Tags section in the IAM user manage view, with an editor that applies a whole edited tag set as a single UntagUser and TagUser pair.
Also corrects two error shapes that never matched AWS: a half-supplied tag member now reports a ValidationError naming the member field instead of MissingParameter, and the maxItems bound check reports separate lower- and upper-bound errors across every IAM list action.
Gates bucket listing behind an identity policy, lets browsers reach the standalone IAM API, and turns the WebUI into a dashboard for S3, IAM, or both.
**Bucket listing.** `ListBuckets` is now gated by the new `s3:ListAllMyBuckets` action, evaluated against `arn:aws:s3:::*`. The request names no bucket, so only identity policies apply — there is no resource policy to combine with, which is the same shape `CreateBucket` already had, so both now share one identity-only evaluation path. Root and admin bypass it, and backends with no identity-policy layer keep listing as before since their listing is already narrowed to the caller's own buckets. The action is IAM-only and is deliberately absent from the bucket-policy action list.
**Fixed bucket ownership.** The standalone IAM client has no per-user ownership to express — accounts are all plain users, cannot be enumerated, and access is decided by policy rather than ACL — so it now implements `auth.FixedBucketOwner` and every bucket is owned by root. Bucket creation stops resolving an owner, `ListBuckets` returns every bucket to every caller (what they may then do with one stays a per-request policy decision), and the admin `ChangeBucketOwner` reports method-not-supported. Other IAM backends are untouched.
**IAM service CORS.** `--cors-allow-origin` now applies to the `iam` command: it answers preflights and stamps the CORS headers, mirroring back the requested method and headers rather than enumerating the SigV4 header set. Without it no browser can reach the IAM API at all, so setting `--webui` without it falls back to `*` with a warning. The chart gets `iamServer.corsAllowOrigin`.
**WebUI.** New IAM pages for users, roles and OIDC providers, signing IAM/STS query-form requests directly from the browser. Navigation is capability-gated rather than role-gated: on sign-in the session probes the S3, admin and IAM endpoints independently and each page shows only what those credentials actually reach, so one build serves an IAM-only dashboard, an S3-only dashboard, and a combined one. The login page takes an optional IAM endpoint, seeded from the new `--webui-iam-gateways` (chart: `webui.iamGateways`) — never auto-detected, since the IAM service is a separate process. The WebUI can also be hosted by `versitygw iam` itself, for deployments with no S3 gateway behind it.
**The admin API is ignored once an IAM endpoint is in play.** The IAM service is then the user directory and bucket ownership is fixed, which leaves the admin API no job: the session is given no admin endpoint at all, its login field is hidden, `users.html` redirects to its IAM counterpart, and every admin-only surface stays off screen. Dashboard and Buckets remain available to any S3 session in such a deployment, running on the S3 and IAM APIs alone and surfacing each denial per action instead of redirecting.
Also fixes two WebUI bugs: embedded assets went out with a zero modification time and no `Cache-Control`, so browsers treated them as fresh for centuries and an upgraded gateway served new HTML against stale JS — they now revalidate against an ETag; and the login page's advanced-options section clipped its last field, since it animated to a height named in the stylesheet rather than the one it measures now.
**Usage**
IAM-only dashboard, served by the IAM service:
versitygw iam --port :7076 --webui :8080 --cors-allow-origin http://localhost:8080/
IAM + S3, dashboard served by the IAM service — point it at the gateway with `--webui-gateways`, and let the gateway accept the dashboard's origin:
versitygw iam --port :7076 --webui :8080 --webui-gateways http://localhost:7070/ --cors-allow-origin http://localhost:8080/
versitygw --port :7070 --cors-allow-origin http://localhost:8080/ posix /data
IAM + S3, dashboard served by the S3 gateway — point it at the IAM service with `--webui-iam-gateways`, and let the IAM service accept the dashboard's origin:
versitygw --port :7070 --webui :8080 --webui-iam-gateways http://localhost:7076/ posix /data
versitygw iam --port :7076 --cors-allow-origin http://localhost:8080/
Validate bucket existence before checking the versioning state so a
missing bucket still yields 404 NoSuchBucket. When no versioning
directory is configured, respond with 200 and an empty configuration
instead of VersioningNotConfigured, matching AWS semantics. GUI clients
(e.g. S3 Browser) poll this API while browsing buckets and were
failing on default deployments. PutBucketVersioning still rejects with
VersioningNotConfigured.
Fix a swapped test body between the Get/Put "not configured" cases,
add integration coverage for GetBucketVersioning on a non-existent
bucket, and add backend/posix unit tests guarding the validation order
and the empty 200 response.
Ref: exastor/versitygw!3
Fixes#1327Fixes#1567Closes#2264
Wires the S3 gateway up to the standalone IAM service so identity policies, not just bucket policies and ACLs, are enforced on the S3 data plane. The gateway authenticates SigV4 requests by calling new private derive-signing-key and resolve-identity endpoints on the IAM service instead of holding secrets itself, and evaluates identity policy through the same PolicyEvaluator path added to auth.VerifyAccess, combined with the bucket policy using explicit-deny-wins precedence. The private endpoints are served over their own mTLS listener (new iamapi/private package, genmtlscerts.sh to generate test material, and client-cert support in internal/netutil), separate from the public IAM API. As part of this the vendored aws/signer/v4 package is deleted and replaced by a pure-Go SigV4 implementation in internal/sigv4auth, which now reads canonical request data directly off the fiber.Ctx instead of reconstructing an http.Request, and is shared by both the S3 request-signing verification and the new private-endpoint signing.
DeleteObjects moves from an all-or-nothing authorization check to true partial success: VerifyObjectsAccess evaluates every object in a batch independently against both the identity policy and any object lock, so a denial or a locked object only removes that key from the batch instead of failing the whole request. It also batches the identity-policy round trip and the bucket-policy fetch once per request rather than once per object, and separates plain deletes from versioned ones since a versioned delete needs s3:DeleteObjectVersion rather than s3:DeleteObject. Object lock handling got a few correctness fixes alongside this: a bypass is now modeled as BypassNone/BypassRequested/BypassOverwrite rather than a single bool, because root's blanket ability to override a GOVERNANCE retention should only apply when the client actually asked to bypass it (DeleteObject/DeleteObjects/PutObjectRetention), not when the gateway is silently replacing a locked object via an overwrite, which needs the permission from everyone including root. Retention changes are now correctly classified as an extension (allowed under plain s3:PutObjectRetention) versus a weakening (date or mode change, which needs the bypass permission), and a COMPLIANCE lock can never be weakened by anyone regardless of permissions, matching AWS. Separately, VerifyObjectCopyAccess had a readonly-mode gap: it returned early for root/admin before ever calling VerifyAccess, so the readonly check inside VerifyAccess never ran for them on CopyObject; access checks are now ordered so the readonly gate always applies before any root/admin bypass, for copy as well as every other write path.
Bucket policies also gained Condition block support, via a new shared internal/condition package moved out of the IAM policy package since both bucket and identity policies share the same evaluation semantics. It implements the full AWS operator set — String{Equals,NotEquals,EqualsIgnoreCase,NotEqualsIgnoreCase,Like,NotLike}, Numeric{Equals,NotEquals,LessThan,LessThanEquals,GreaterThan,GreaterThanEquals}, Date{Equals,NotEquals,LessThan,LessThanEquals,GreaterThan,GreaterThanEquals}, Bool, BinaryEquals, Arn{Equals,Like,NotEquals,NotLike}, IpAddress/NotIpAddress, and Null — along with the ForAllValues/ForAnyValue set qualifiers and the IfExists modifier. A new requestConditionContext builds the per-request keys a bucket policy's Condition block can reference — aws:SourceIp, aws:SecureTransport, aws:CurrentTime, aws:EpochTime, aws:UserAgent, aws:Referer, s3:prefix, s3:delimiter, s3:max-keys, s3:x-amz-acl, s3:VersionId — following AWS's own per-action rules for which keys a given S3 operation actually populates. Identity-derived keys such as aws:PrincipalArn and aws:username are deliberately left unwired here, since the gateway has no way to know them; the standalone IAM service fills those in itself when it evaluates an identity policy.
Also added new integration test suites for S3-side IAM: s3_iam_access_control.go and s3_iam_session_access_control.go cover identity-policy enforcement and session-credential requests against real S3 operations, alongside expanded OIDC/web-identity coverage and a new runoidctests.sh runner wired into the OIDC GitHub Actions workflow.
Add IAMAssumeRoleWithWebIdentity_github_oidc_live, the only web-identity test that exercises AssumeRoleWithWebIdentity against a real external OIDC provider end-to-end: GitHub Actions' own issuer, with real discovery-document fetch, JWKS fetch, RS256 signature verification, claims mapping, and session credential issuance. Every other web-identity test in the suite uses a fake token that never reaches real signature verification. The test registers a throwaway OIDC provider and trust role scoped to this repo (via a distinct test audience and repo-scoped sub condition), fetches a real ID token from GitHub's runtime endpoint, assumes the role, and confirms the issued session credentials work with a follow-up GetCallerIdentity call. It cleans up the role and provider unconditionally and skips itself when run outside a GitHub Actions job with id-token: write permission (e.g. local runs or fork PRs, where GitHub downgrades OIDC permissions to read-only).
Add functional-iam-oidc.yml to run this test in CI on push to main and on same-repo pull_request runs, isolated from the full iam suite since it's the only test needing id-token: write.
Add a SKIP counter and skipF() alongside the existing runF/passF/failF, and report it in the final RAN/PASS/FAIL summary, so a test opting out via skipF() (as this one does when OIDC env vars aren't present) is visible instead of silently absent from the count.
Implements the `AssumeRoleWithWebIdentity` and `GetCallerIdentity` STS actions, letting callers exchange an external OIDC token for temporary credentials scoped to an IAM role. Token handling covers JWT claim parsing, issuer/audience resolution (including `azp` override semantics), JWKS fetching and caching with `singleflight`-deduplicated refresh, and rate-limited forced refresh on unrecognized `kid` values. OIDC provider thumbprint fetching now performs a real TLS handshake verified against the system trust store and the provider hostname (previously `InsecureSkipVerify`), since the observed certificate is persisted as a long-lived trust anchor rather than used once and discarded; all discovery-document and JWKS fetches go through an SSRF-safe HTTP client with bounded redirects and response size.
Adds policy `Condition` block evaluation, supporting `String`, `Numeric`, `Date`, `Bool`, `BinaryEquals`, and `IpAddress` operators along with their `IfExists`/`Not` variants and `ForAllValues`/`ForAnyValues` set qualifiers, plus policy variable substitution (e.g. `${aws:username}`) in supported operators. Adds identity-based inline policy evaluation and a new IAM authorization middleware that authorizes each request against action, resource, and condition context together, applying the session-policy-intersects-role-policy semantics for assumed-role sessions.
Adds a new debug logger `--log-level` flag (`silent`/`debug`/`unsafe`), along with a tree-based XML masker that redacts secrets and tokens at the property level in logged request/response bodies instead of skipping the whole body. The old `--debug/VGW_DEBUG` flag is kept as a deprecated alias for `--log-level=debug`, printing a console warning that points users at `--log-level` for finer-grained control.
Fixes a Vault storage bug where CAS (check-and-set) writes always read the current document version as 0 because `kvVersion` asserted metadata as `float64` while the Vault client actually returns `json.Number`, causing every write past the first to be rejected as a concurrent modification. Also adds a constant-time `SecureCompare` for signature/token comparisons in sigv4 auth.
Adds an integration test suite (`iam_access_control.go`) covering IAM access control across user, role, and session identities.
Add support for `CreateOpenIDConnectProvider`, `GetOpenIDConnectProvider`, `ListOpenIDConnectProviders`, `DeleteOpenIDConnectProvider`, `AddClientIDToOpenIDConnectProvider`, `RemoveClientIDFromOpenIDConnectProvider`, and `UpdateOpenIDConnectProviderThumbprint` on both the internal and Vault storage backends, rounding out the standalone IAM service with the same OIDC identity provider management AWS IAM exposes.
CreateOpenIDConnectProvider validates the issuer URL, enforces the client ID and per-provider client ID list limits, and accepts an optional ThumbprintList. When the caller omits ThumbprintList, the provider auto-fetches the thumbprint by opening an outbound TLS connection to the issuer URL and hashing its top-level CA certificate, matching real AWS behavior. This auto-fetch is configurable: it can be turned off with the `--disable-oidc-thumbprint-autofetch` CLI flag (or the `VGW_IAM_DISABLE_OIDC_THUMBPRINT_AUTOFETCH` environment variable) for restricted or air-gapped deployments where the IAM server shouldn't make outbound connections, in which case an omitted ThumbprintList is rejected instead. AddClientIDToOpenIDConnectProvider and RemoveClientIDFromOpenIDConnectProvider manage a provider's client ID list, and UpdateOpenIDConnectProviderThumbprint replaces its thumbprint list, all with the same length and format validation applied at creation time.
Provider ARNs are derived from the issuer URL, and GetOpenIDConnectProvider and DeleteOpenIDConnectProvider resolve providers by ARN, returning NoSuchEntity when a provider doesn't exist. ListOpenIDConnectProviders returns the full set of stored providers. These actions are wired into the IAM API router and given their own XML response types under iamapi/types, with a new iamapi/internal/iamutil package handling URL validation, thumbprint fetching and normalization, and ARN construction shared across the controller methods.
Add support for the `PutRolePolicy`, `GetRolePolicy`, `DeleteRolePolicy`, and `ListRolePolicies` actions in the IAM-compatible gateway service, extending role management with the same inline-policy lifecycle already available for IAM users. `PutRolePolicy` validates the policy name and document, parses the document for AWS-compatible syntax and semantic errors (missing actions/resources, malformed ARNs, disallowed principals, duplicate statement IDs, and so on), and rejects documents once the role's aggregate inline-policy size would exceed `MaxInlinePolicyBytesPerRole` (10240 bytes, distinct from the 2048-byte quota enforced for users). Putting a policy under an existing name overwrites its document in place. `GetRolePolicy` and `DeleteRolePolicy` look up or remove a named inline policy from a role, returning a `NoSuchEntity` error when the role or the policy is not found. `ListRolePolicies` returns a role's inline policy names in sorted order with marker-based pagination.
These actions are implemented for both the internal file-backed store and the Vault-backed store, wired into the IAM API router, and given their own XML response types under `iamapi/types`. A new `NoSuchEntityRolePolicy` error was added to `iamapi/iamerr` to mirror the existing user-policy error.
Adds `CreateRole`, `GetRole`, `ListRoles`, `DeleteRole`, and `UpdateAssumeRolePolicy` to the standalone IAM service, following the same controller/storage patterns established for users. Both the internal filesystem/S3-backed store and the Vault-backed store implement the new `Storer` methods, with role-specific indexing and lookup helpers mirroring the existing user ones.
Role creation requires a trust policy, passed as `AssumeRolePolicyDocument`. A trust policy is a distinct kind of IAM policy document that governs who (or what) is allowed to assume a role, rather than what actions the role itself is permitted to perform. Its grammar is effectively the inverse of an identity policy: `Principal` is required, `Action`/`NotAction` values must carry the `sts:` prefix, and `Resource`/`NotResource` are forbidden. This is implemented in `iamapi/policy/trust.go` as a new validation path alongside the existing identity-policy validation, and is reused by `UpdateAssumeRolePolicy` when replacing a role's trust policy.
Also fixes user name uniqueness enforcement to be case-insensitive, matching AWS IAM behavior, and applies the same case-insensitive handling to role names. The internal store now maintains lowercase name indexes for both users and roles, and the Vault store resolves the canonical stored key via a case-insensitive list-and-compare fallback since Vault's KV paths are case-sensitive.
Add support for AWS-compatible inline identity-based policies on IAM
users, implementing the `PutUserPolicy`, `GetUserPolicy`, `DeleteUserPolicy`, and `ListUserPolicies` actions on both the internal and Vault storage
backends.
- iamapi/policy is a new package that parses and validates policy documents against IAM's parameter-level constraints (max length, allowed charset) and policy grammar (Version, Effect, mutually exclusive Action/NotAction and Resource/NotResource, vendor-prefixed actions, ARN-shaped resources, no Principal/NotPrincipal, unique Sids).
- `PutUserPolicy` creates or replaces a named inline policy on a user, enforcing a 2048-byte aggregate quota across all of a user's inline policies (MaxInlinePolicyBytesPerUser), matching the AWS IAM quota.
- `GetUserPolicy` returns a policy's document RFC 3986 percent-encoded, matching how real IAM encodes the PolicyDocument response element.
- `DeleteUserPolicy` removes a named inline policy from a user.
- `ListUserPolicies` returns a paginated, sorted list of a user's inline policy names, honoring Marker/MaxItems like the other IAM list APIs.
- `DeleteUser` is now rejected with a DeleteConflict error if the user still has inline policies attached, mirroring the existing access-key delete-conflict behavior.
Add `CreateAccessKey`, `UpdateAccessKey`, `DeleteAccessKey`, `ListAccessKeys`, and `GetAccessKeyLastUsed` actions for managing user access keys and retrieving their latest usage details.
Generate AWS-style access key IDs and secrets, validate key identifiers and statuses, enforce per-user key quotas, and prevent deleting users that still own access keys. Persist access keys across internal and Vault storage backends with ownership indexing, pagination, and IAM-compatible errors and XML responses.
Closes#1640
Add a standalone AWS IAM Query API implementation for managing IAM users through standard AWS SDKs and the AWS CLI.
Server usage
Start the IAM server with internal file-backed storage:
mkdir -p /tmp/versitygw-iam
./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --dir /tmp/versitygw-iam
Start the IAM server with Vault KV v2 storage using AppRole:
VGW_IAM_VAULT_ROLE_SECRET=<role-secret> ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --vault-endpoint-url http://127.0.0.1:8200 --vault-auth-method approle --vault-role-id <role-id> --vault-mount-path kv --vault-secret-storage-path iam
Vault authentication also supports root tokens, separate authentication and secret-storage namespaces, custom mount paths, server certificate validation, and mutual TLS client certificates.
Configure the AWS CLI credentials used by the IAM server:
export AWS_ACCESS_KEY_ID=user
export AWS_SECRET_ACCESS_KEY=pass
export AWS_DEFAULT_REGION=us-east-1
Implemented IAM actions
CreateUser creates an IAM user with an AWS-compatible ARN, generated AIDA user ID, creation timestamp, optional path, and tags. It validates usernames, paths, tag limits, reserved tag prefixes, duplicate tag keys, and existing users.
aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob
aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob --path /engineering/ --tags Key=team,Value=storage
GetUser returns a stored user or the root identity when requested without a username through the IAM Query API.
aws --endpoint-url http://127.0.0.1:7070 iam get-user --user-name bob
ListUsers returns users in deterministic username order and supports path filtering, marker-based pagination, and MaxItems limits.
aws --endpoint-url http://127.0.0.1:7070 iam list-users
aws --endpoint-url http://127.0.0.1:7070 iam list-users --path-prefix /engineering/ --max-items 100
UpdateUser updates the username and/or path, recalculates the user ARN, and rejects conflicts with existing users.
aws --endpoint-url http://127.0.0.1:7070 iam update-user --user-name bob --new-user-name robert --new-path /platform/
DeleteUser permanently removes an IAM user and returns AWS-compatible errors for missing users.
aws --endpoint-url http://127.0.0.1:7070 iam delete-user --user-name robert
IAM protocol and authentication
- Support the AWS IAM Query protocol version 2010-05-08 over GET and POST form requests.
- Return AWS-compatible XML responses, error documents, status codes, request IDs, user metadata, and pagination fields.
- Authenticate root credentials with AWS Signature Version 4 for the IAM service in us-east-1.
- Support both Authorization-header and query-string SigV4 authentication.
- Validate credential scope, signed headers, timestamps, clock skew, content length, signatures, and unsupported signature or session-token modes.
- Add IAM-specific validation and error mapping for malformed requests, invalid actions, duplicate entities, missing users, throttling, and internal failures.
Storage implementations
- Add an internal JSON-backed store using iam.json and iam.json.backup with atomic temporary-file replacement, concurrent access protection, stable ordering, pagination, and persistence across restarts.
- Add a Vault KV v2 store with one secret per user, CAS-based duplicate protection, permanent deletion, AppRole reauthentication, namespace support, configurable authentication and KV mounts, root-token authentication, and TLS/mTLS configuration.
- Introduce a common Storer interface and require exactly one storage backend to be configured.
Server and embedding support
- Register the new `versitygw iam` command with environment-variable and CLI configuration for both storage backends.
- Add `embedgw.RunIAMAPI` and `IAMConfig` for embedding the IAM service in Go applications.
Gateway-level internal packages
- Add `internal/iamstore` as a reusable generic file-backed IAM persistence engine and migrate the existing gateway internal IAM service to it.
- Add `internal/sigv4auth` for shared SigV4 header and presigned-query parsing, canonical request generation, signature verification, and structured authentication errors.
- Refactor the S3 authentication paths to use the shared SigV4 implementation while preserving S3-specific error responses.
- Add `internal/httpctx` for shared Fiber context keys and AWS-style request ID handling.
- Add `internal/routekit` for shared query, form, and header route matchers.
- Add `internal/netutil` for reusable certificate storage, hostname-aware listeners, multi-address serving, TLS listeners, and UNIX socket handling.
- Update the custom SigV4 signer to honor an explicitly supplied signed-header list so unrelated headers do not alter IAM signatures.
Testing and CI
- Add AWS IAM SDK-based integration coverage for all supported user actions, header authentication, query authentication, validation, errors, filtering, and pagination.
- Split standalone IAM tests into `versitygw test iam` and retain existing gateway IAM tests under `versitygw test gw-iam`.
- Add unit coverage for controllers, authentication, routing, storage, embedding, listeners, request matching, persistence, and signing behavior.
- Add `runiamtests.sh` to exercise internal storage over HTTP and HTTPS plus Vault storage through AppRole.
- Add a dedicated IAM functional-test workflow with a Vault service and merged runtime coverage reporting.
- Include the IAM test runner in shellcheck and add the AWS IAM SDK dependency.
Introduce an opt-in mode that derives ETags from checksums instead of relying on
MD5 assumptions, aligning with AWS-compatible scenarios where ETag should be
treated as an opaque identifier rather than a content-MD5 signal.
This can significantly reduce CPU load on the server to disable MD5 checksums
when we are already validating data with other checksums.
This change adds --data-integrity-etag and VGW_DATA_INTEGRITY_ETAG support for
both posix and scoutfs backends. In this mode, PUT object ETags, multipart part
ETags, and completed multipart object ETags are checksum-derived.
The copy-source header was being interpreted differently by request
validation and the backend parser. An encoded ?versionId= separator
could be decoded at one layer but treated as part of the object path
at another, which opened a path traversal risk on filesystem-backed
copy operations.
Align backend copy-source parsing with validation by decoding the
header before splitting the versionId suffix, so both layers derive
the same bucket, object, and versionId values. This closes the
traversal path and adds regression coverage for encoded
copy-source inputs.
Reported by 5ud0 / Tarmo Technologies.
quote part etags across all azure multipart surfaces (uploadpart,
listparts, uploadpartcopy) via a shared quoteETag helper, matching the
s3 contract and the posix backend's GenerateEtag convention. populate
the previously-empty UploadPartCopy CopyPartResult with the quoted etag
and last-modified. normalize the client-supplied etag in
CompleteMultipartUpload with backend.TrimEtag so both quoted and raw
etags are accepted. add an azure-full-flow integration test asserting
part etags are quoted and consistent across uploadpart, listparts and
completemultipartupload.
This change adds Windows functional test execution in CI and updates
backend handling so windows filesystem error/path semantics map correctly
to expected S3 outcomes.
The only meta supported on windows right now is sidecar, so the tests
in windows mode also skip sidecar skips.
Future work is to address the skips and/or more clearly document
the unsupported/incompatible behavior on windows.
The windows support will still remain best effort, but these tests
should at least flag when future changes introduce incompatible
behavior on windows.
Fixes#2179
Treat a present but empty `Content-MD5` header as an invalid digest instead of handling it as if the header were absent. This makes PUT operations return `InvalidDigest` for empty `Content-MD5` values while preserving existing behavior for missing headers.
Integrate x-amz-website-redirect-location across object metadata flows so uploads, copies, multipart creation, HEAD, and GET preserve and return redirect locations, and website hosting applies object-level redirects from the stored value.
Enhances the static website hosting implementation with more complete S3-compatible behavior across request handling, backend storage, validation, CORS, and errors.
Adds dedicated website endpoint handling for GET, HEAD, and OPTIONS requests, including index document resolution, error document serving, redirect-all support, pre-fetch and post-error routing rules, query string preservation in redirects, public access checks before object reads, and method-not-allowed responses.
Improves error handling for website responses by returning S3-compatible HTML error bodies with request IDs, host IDs, x-amz-error-code, x-amz-error-message, and specialized error fields. This also fixes website-related validation errors to return more accurate S3-style error codes and messages, including invalid redirect protocols, invalid HTTP redirect/error codes, conflicting routing rule replacements, routing rule limits, and oversized website configuration requests.
Adds website CORS support for GET, HEAD, and OPTIONS preflight requests, including bucket CORS lookup through website host bucket resolution, allowed origin/method/header validation, exposed header handling, ETag exposure, Vary headers, max-age handling, and CORS access-denied responses.
Adds debug logging around website configuration parsing, validation failures, CORS checks, backend lookup failures, and internal website error paths to make failures easier to diagnose.
Adds compressed website configuration storage so larger configs fit backend metadata limits, including gzip storage for POSIX extended attributes and base64-encoded compressed metadata for Azure. Also adds Azure PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite support.
Adds and expands test coverage for website config validation, S3-compatible HTML error bodies, website routing behavior, public access enforcement, HEAD behavior, CORS handling, PutBucketWebsite limits, and end-to-end website hosting through a Docker-based dnsmasq test setup and CI workflow.
niksis02
Ben McClellandandGitHub
DavvyyandGitHub
mu-adnan
Nils LegerandGitHub
HeonJe Lee
m-martin-78
Ed Espino