The S3 gateway and the standalone IAM service exchange authorization decisions over the private endpoints, where a version skew is silently unsafe in both directions: an older service drops a request field it does not know (a `Condition` block, say) and evaluates fail-open, while an older gateway ignores a response field it does not know and misses a deny the service intended. Neither side could previously detect either case.
Both peers now declare a protocol version on every exchange via the `X-Vgw-Private-Protocol` header — the gateway on each request, the service on each response, error responses included — and each refuses a peer it cannot serve safely. The service rejects a gateway below `MinClientProtocol` with a `ProtocolMismatch` code; the gateway rejects a service older than the `ProtocolVersion` it speaks, and rejects a response carrying no version at all, since no build of this protocol omits the header and something else answering on that address should not be interpreted as an IAM decision. `ParseProtocolVersion` is shared by both sides and deliberately strict: an unreadable value is a mismatch, never an assumed default.
A new root-signed `/private/version` endpoint reports the protocol version, the minimum client the service will serve, and the build tag (`WithPrivateServerVersion`). It is exempt from the service's own client-version check so it can still answer a gateway the service refuses — which is how that gateway learns why. Being authenticated like every other private endpoint, it also lets the gateway's startup probe verify its own credential and its mTLS transport in the same round trip.
The gateway probes it once in `NewIAMServiceStandalone` rather than discovering a skew as an opaque per-request 500. An incompatible service is fatal after a 30s window, since a gateway that cannot authorize a single request is more useful refusing to start with the reason in its log; an unreachable one is only a warning, because the two processes legitimately start in parallel and every request checks the version regardless. Only conditions that can resolve on their own are retried — a rejected credential is reported immediately.
Fixes#1327Fixes#1567Closes#2264
Wires the S3 gateway up to the standalone IAM service so identity policies, not just bucket policies and ACLs, are enforced on the S3 data plane. The gateway authenticates SigV4 requests by calling new private derive-signing-key and resolve-identity endpoints on the IAM service instead of holding secrets itself, and evaluates identity policy through the same PolicyEvaluator path added to auth.VerifyAccess, combined with the bucket policy using explicit-deny-wins precedence. The private endpoints are served over their own mTLS listener (new iamapi/private package, genmtlscerts.sh to generate test material, and client-cert support in internal/netutil), separate from the public IAM API. As part of this the vendored aws/signer/v4 package is deleted and replaced by a pure-Go SigV4 implementation in internal/sigv4auth, which now reads canonical request data directly off the fiber.Ctx instead of reconstructing an http.Request, and is shared by both the S3 request-signing verification and the new private-endpoint signing.
DeleteObjects moves from an all-or-nothing authorization check to true partial success: VerifyObjectsAccess evaluates every object in a batch independently against both the identity policy and any object lock, so a denial or a locked object only removes that key from the batch instead of failing the whole request. It also batches the identity-policy round trip and the bucket-policy fetch once per request rather than once per object, and separates plain deletes from versioned ones since a versioned delete needs s3:DeleteObjectVersion rather than s3:DeleteObject. Object lock handling got a few correctness fixes alongside this: a bypass is now modeled as BypassNone/BypassRequested/BypassOverwrite rather than a single bool, because root's blanket ability to override a GOVERNANCE retention should only apply when the client actually asked to bypass it (DeleteObject/DeleteObjects/PutObjectRetention), not when the gateway is silently replacing a locked object via an overwrite, which needs the permission from everyone including root. Retention changes are now correctly classified as an extension (allowed under plain s3:PutObjectRetention) versus a weakening (date or mode change, which needs the bypass permission), and a COMPLIANCE lock can never be weakened by anyone regardless of permissions, matching AWS. Separately, VerifyObjectCopyAccess had a readonly-mode gap: it returned early for root/admin before ever calling VerifyAccess, so the readonly check inside VerifyAccess never ran for them on CopyObject; access checks are now ordered so the readonly gate always applies before any root/admin bypass, for copy as well as every other write path.
Bucket policies also gained Condition block support, via a new shared internal/condition package moved out of the IAM policy package since both bucket and identity policies share the same evaluation semantics. It implements the full AWS operator set — String{Equals,NotEquals,EqualsIgnoreCase,NotEqualsIgnoreCase,Like,NotLike}, Numeric{Equals,NotEquals,LessThan,LessThanEquals,GreaterThan,GreaterThanEquals}, Date{Equals,NotEquals,LessThan,LessThanEquals,GreaterThan,GreaterThanEquals}, Bool, BinaryEquals, Arn{Equals,Like,NotEquals,NotLike}, IpAddress/NotIpAddress, and Null — along with the ForAllValues/ForAnyValue set qualifiers and the IfExists modifier. A new requestConditionContext builds the per-request keys a bucket policy's Condition block can reference — aws:SourceIp, aws:SecureTransport, aws:CurrentTime, aws:EpochTime, aws:UserAgent, aws:Referer, s3:prefix, s3:delimiter, s3:max-keys, s3:x-amz-acl, s3:VersionId — following AWS's own per-action rules for which keys a given S3 operation actually populates. Identity-derived keys such as aws:PrincipalArn and aws:username are deliberately left unwired here, since the gateway has no way to know them; the standalone IAM service fills those in itself when it evaluates an identity policy.
Also added new integration test suites for S3-side IAM: s3_iam_access_control.go and s3_iam_session_access_control.go cover identity-policy enforcement and session-credential requests against real S3 operations, alongside expanded OIDC/web-identity coverage and a new runoidctests.sh runner wired into the OIDC GitHub Actions workflow.
Add IAMAssumeRoleWithWebIdentity_github_oidc_live, the only web-identity test that exercises AssumeRoleWithWebIdentity against a real external OIDC provider end-to-end: GitHub Actions' own issuer, with real discovery-document fetch, JWKS fetch, RS256 signature verification, claims mapping, and session credential issuance. Every other web-identity test in the suite uses a fake token that never reaches real signature verification. The test registers a throwaway OIDC provider and trust role scoped to this repo (via a distinct test audience and repo-scoped sub condition), fetches a real ID token from GitHub's runtime endpoint, assumes the role, and confirms the issued session credentials work with a follow-up GetCallerIdentity call. It cleans up the role and provider unconditionally and skips itself when run outside a GitHub Actions job with id-token: write permission (e.g. local runs or fork PRs, where GitHub downgrades OIDC permissions to read-only).
Add functional-iam-oidc.yml to run this test in CI on push to main and on same-repo pull_request runs, isolated from the full iam suite since it's the only test needing id-token: write.
Add a SKIP counter and skipF() alongside the existing runF/passF/failF, and report it in the final RAN/PASS/FAIL summary, so a test opting out via skipF() (as this one does when OIDC env vars aren't present) is visible instead of silently absent from the count.
Implements the `AssumeRoleWithWebIdentity` and `GetCallerIdentity` STS actions, letting callers exchange an external OIDC token for temporary credentials scoped to an IAM role. Token handling covers JWT claim parsing, issuer/audience resolution (including `azp` override semantics), JWKS fetching and caching with `singleflight`-deduplicated refresh, and rate-limited forced refresh on unrecognized `kid` values. OIDC provider thumbprint fetching now performs a real TLS handshake verified against the system trust store and the provider hostname (previously `InsecureSkipVerify`), since the observed certificate is persisted as a long-lived trust anchor rather than used once and discarded; all discovery-document and JWKS fetches go through an SSRF-safe HTTP client with bounded redirects and response size.
Adds policy `Condition` block evaluation, supporting `String`, `Numeric`, `Date`, `Bool`, `BinaryEquals`, and `IpAddress` operators along with their `IfExists`/`Not` variants and `ForAllValues`/`ForAnyValues` set qualifiers, plus policy variable substitution (e.g. `${aws:username}`) in supported operators. Adds identity-based inline policy evaluation and a new IAM authorization middleware that authorizes each request against action, resource, and condition context together, applying the session-policy-intersects-role-policy semantics for assumed-role sessions.
Adds a new debug logger `--log-level` flag (`silent`/`debug`/`unsafe`), along with a tree-based XML masker that redacts secrets and tokens at the property level in logged request/response bodies instead of skipping the whole body. The old `--debug/VGW_DEBUG` flag is kept as a deprecated alias for `--log-level=debug`, printing a console warning that points users at `--log-level` for finer-grained control.
Fixes a Vault storage bug where CAS (check-and-set) writes always read the current document version as 0 because `kvVersion` asserted metadata as `float64` while the Vault client actually returns `json.Number`, causing every write past the first to be rejected as a concurrent modification. Also adds a constant-time `SecureCompare` for signature/token comparisons in sigv4 auth.
Adds an integration test suite (`iam_access_control.go`) covering IAM access control across user, role, and session identities.
Add support for `CreateOpenIDConnectProvider`, `GetOpenIDConnectProvider`, `ListOpenIDConnectProviders`, `DeleteOpenIDConnectProvider`, `AddClientIDToOpenIDConnectProvider`, `RemoveClientIDFromOpenIDConnectProvider`, and `UpdateOpenIDConnectProviderThumbprint` on both the internal and Vault storage backends, rounding out the standalone IAM service with the same OIDC identity provider management AWS IAM exposes.
CreateOpenIDConnectProvider validates the issuer URL, enforces the client ID and per-provider client ID list limits, and accepts an optional ThumbprintList. When the caller omits ThumbprintList, the provider auto-fetches the thumbprint by opening an outbound TLS connection to the issuer URL and hashing its top-level CA certificate, matching real AWS behavior. This auto-fetch is configurable: it can be turned off with the `--disable-oidc-thumbprint-autofetch` CLI flag (or the `VGW_IAM_DISABLE_OIDC_THUMBPRINT_AUTOFETCH` environment variable) for restricted or air-gapped deployments where the IAM server shouldn't make outbound connections, in which case an omitted ThumbprintList is rejected instead. AddClientIDToOpenIDConnectProvider and RemoveClientIDFromOpenIDConnectProvider manage a provider's client ID list, and UpdateOpenIDConnectProviderThumbprint replaces its thumbprint list, all with the same length and format validation applied at creation time.
Provider ARNs are derived from the issuer URL, and GetOpenIDConnectProvider and DeleteOpenIDConnectProvider resolve providers by ARN, returning NoSuchEntity when a provider doesn't exist. ListOpenIDConnectProviders returns the full set of stored providers. These actions are wired into the IAM API router and given their own XML response types under iamapi/types, with a new iamapi/internal/iamutil package handling URL validation, thumbprint fetching and normalization, and ARN construction shared across the controller methods.
Add support for the `PutRolePolicy`, `GetRolePolicy`, `DeleteRolePolicy`, and `ListRolePolicies` actions in the IAM-compatible gateway service, extending role management with the same inline-policy lifecycle already available for IAM users. `PutRolePolicy` validates the policy name and document, parses the document for AWS-compatible syntax and semantic errors (missing actions/resources, malformed ARNs, disallowed principals, duplicate statement IDs, and so on), and rejects documents once the role's aggregate inline-policy size would exceed `MaxInlinePolicyBytesPerRole` (10240 bytes, distinct from the 2048-byte quota enforced for users). Putting a policy under an existing name overwrites its document in place. `GetRolePolicy` and `DeleteRolePolicy` look up or remove a named inline policy from a role, returning a `NoSuchEntity` error when the role or the policy is not found. `ListRolePolicies` returns a role's inline policy names in sorted order with marker-based pagination.
These actions are implemented for both the internal file-backed store and the Vault-backed store, wired into the IAM API router, and given their own XML response types under `iamapi/types`. A new `NoSuchEntityRolePolicy` error was added to `iamapi/iamerr` to mirror the existing user-policy error.
Adds `CreateRole`, `GetRole`, `ListRoles`, `DeleteRole`, and `UpdateAssumeRolePolicy` to the standalone IAM service, following the same controller/storage patterns established for users. Both the internal filesystem/S3-backed store and the Vault-backed store implement the new `Storer` methods, with role-specific indexing and lookup helpers mirroring the existing user ones.
Role creation requires a trust policy, passed as `AssumeRolePolicyDocument`. A trust policy is a distinct kind of IAM policy document that governs who (or what) is allowed to assume a role, rather than what actions the role itself is permitted to perform. Its grammar is effectively the inverse of an identity policy: `Principal` is required, `Action`/`NotAction` values must carry the `sts:` prefix, and `Resource`/`NotResource` are forbidden. This is implemented in `iamapi/policy/trust.go` as a new validation path alongside the existing identity-policy validation, and is reused by `UpdateAssumeRolePolicy` when replacing a role's trust policy.
Also fixes user name uniqueness enforcement to be case-insensitive, matching AWS IAM behavior, and applies the same case-insensitive handling to role names. The internal store now maintains lowercase name indexes for both users and roles, and the Vault store resolves the canonical stored key via a case-insensitive list-and-compare fallback since Vault's KV paths are case-sensitive.
Add support for AWS-compatible inline identity-based policies on IAM
users, implementing the `PutUserPolicy`, `GetUserPolicy`, `DeleteUserPolicy`, and `ListUserPolicies` actions on both the internal and Vault storage
backends.
- iamapi/policy is a new package that parses and validates policy documents against IAM's parameter-level constraints (max length, allowed charset) and policy grammar (Version, Effect, mutually exclusive Action/NotAction and Resource/NotResource, vendor-prefixed actions, ARN-shaped resources, no Principal/NotPrincipal, unique Sids).
- `PutUserPolicy` creates or replaces a named inline policy on a user, enforcing a 2048-byte aggregate quota across all of a user's inline policies (MaxInlinePolicyBytesPerUser), matching the AWS IAM quota.
- `GetUserPolicy` returns a policy's document RFC 3986 percent-encoded, matching how real IAM encodes the PolicyDocument response element.
- `DeleteUserPolicy` removes a named inline policy from a user.
- `ListUserPolicies` returns a paginated, sorted list of a user's inline policy names, honoring Marker/MaxItems like the other IAM list APIs.
- `DeleteUser` is now rejected with a DeleteConflict error if the user still has inline policies attached, mirroring the existing access-key delete-conflict behavior.
Add `CreateAccessKey`, `UpdateAccessKey`, `DeleteAccessKey`, `ListAccessKeys`, and `GetAccessKeyLastUsed` actions for managing user access keys and retrieving their latest usage details.
Generate AWS-style access key IDs and secrets, validate key identifiers and statuses, enforce per-user key quotas, and prevent deleting users that still own access keys. Persist access keys across internal and Vault storage backends with ownership indexing, pagination, and IAM-compatible errors and XML responses.
Closes#1640
Add a standalone AWS IAM Query API implementation for managing IAM users through standard AWS SDKs and the AWS CLI.
Server usage
Start the IAM server with internal file-backed storage:
mkdir -p /tmp/versitygw-iam
./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --dir /tmp/versitygw-iam
Start the IAM server with Vault KV v2 storage using AppRole:
VGW_IAM_VAULT_ROLE_SECRET=<role-secret> ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --vault-endpoint-url http://127.0.0.1:8200 --vault-auth-method approle --vault-role-id <role-id> --vault-mount-path kv --vault-secret-storage-path iam
Vault authentication also supports root tokens, separate authentication and secret-storage namespaces, custom mount paths, server certificate validation, and mutual TLS client certificates.
Configure the AWS CLI credentials used by the IAM server:
export AWS_ACCESS_KEY_ID=user
export AWS_SECRET_ACCESS_KEY=pass
export AWS_DEFAULT_REGION=us-east-1
Implemented IAM actions
CreateUser creates an IAM user with an AWS-compatible ARN, generated AIDA user ID, creation timestamp, optional path, and tags. It validates usernames, paths, tag limits, reserved tag prefixes, duplicate tag keys, and existing users.
aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob
aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob --path /engineering/ --tags Key=team,Value=storage
GetUser returns a stored user or the root identity when requested without a username through the IAM Query API.
aws --endpoint-url http://127.0.0.1:7070 iam get-user --user-name bob
ListUsers returns users in deterministic username order and supports path filtering, marker-based pagination, and MaxItems limits.
aws --endpoint-url http://127.0.0.1:7070 iam list-users
aws --endpoint-url http://127.0.0.1:7070 iam list-users --path-prefix /engineering/ --max-items 100
UpdateUser updates the username and/or path, recalculates the user ARN, and rejects conflicts with existing users.
aws --endpoint-url http://127.0.0.1:7070 iam update-user --user-name bob --new-user-name robert --new-path /platform/
DeleteUser permanently removes an IAM user and returns AWS-compatible errors for missing users.
aws --endpoint-url http://127.0.0.1:7070 iam delete-user --user-name robert
IAM protocol and authentication
- Support the AWS IAM Query protocol version 2010-05-08 over GET and POST form requests.
- Return AWS-compatible XML responses, error documents, status codes, request IDs, user metadata, and pagination fields.
- Authenticate root credentials with AWS Signature Version 4 for the IAM service in us-east-1.
- Support both Authorization-header and query-string SigV4 authentication.
- Validate credential scope, signed headers, timestamps, clock skew, content length, signatures, and unsupported signature or session-token modes.
- Add IAM-specific validation and error mapping for malformed requests, invalid actions, duplicate entities, missing users, throttling, and internal failures.
Storage implementations
- Add an internal JSON-backed store using iam.json and iam.json.backup with atomic temporary-file replacement, concurrent access protection, stable ordering, pagination, and persistence across restarts.
- Add a Vault KV v2 store with one secret per user, CAS-based duplicate protection, permanent deletion, AppRole reauthentication, namespace support, configurable authentication and KV mounts, root-token authentication, and TLS/mTLS configuration.
- Introduce a common Storer interface and require exactly one storage backend to be configured.
Server and embedding support
- Register the new `versitygw iam` command with environment-variable and CLI configuration for both storage backends.
- Add `embedgw.RunIAMAPI` and `IAMConfig` for embedding the IAM service in Go applications.
Gateway-level internal packages
- Add `internal/iamstore` as a reusable generic file-backed IAM persistence engine and migrate the existing gateway internal IAM service to it.
- Add `internal/sigv4auth` for shared SigV4 header and presigned-query parsing, canonical request generation, signature verification, and structured authentication errors.
- Refactor the S3 authentication paths to use the shared SigV4 implementation while preserving S3-specific error responses.
- Add `internal/httpctx` for shared Fiber context keys and AWS-style request ID handling.
- Add `internal/routekit` for shared query, form, and header route matchers.
- Add `internal/netutil` for reusable certificate storage, hostname-aware listeners, multi-address serving, TLS listeners, and UNIX socket handling.
- Update the custom SigV4 signer to honor an explicitly supplied signed-header list so unrelated headers do not alter IAM signatures.
Testing and CI
- Add AWS IAM SDK-based integration coverage for all supported user actions, header authentication, query authentication, validation, errors, filtering, and pagination.
- Split standalone IAM tests into `versitygw test iam` and retain existing gateway IAM tests under `versitygw test gw-iam`.
- Add unit coverage for controllers, authentication, routing, storage, embedding, listeners, request matching, persistence, and signing behavior.
- Add `runiamtests.sh` to exercise internal storage over HTTP and HTTPS plus Vault storage through AppRole.
- Add a dedicated IAM functional-test workflow with a Vault service and merged runtime coverage reporting.
- Include the IAM test runner in shellcheck and add the AWS IAM SDK dependency.
Add the remaining contract tests for the RC token gate: a 16-hex-char
token with no colon stays on the malformed-token 400 path, and the
legacy descriptor header takes precedence when an RC-shaped combined
token is also present. Note in the malformed non-hex case that the 400
expectation is for the standalone test app; the production fiber error
handler currently maps it to 500, which is pre-existing behavior.
Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
The RDMA token parser accepts only the cuObject combined-token format,
whose first colon-delimited field is a hex-encoded uint64 base address
(at most 16 hex chars). Fixed-width binary RC token schemes such as AMD
hipObject's (a 44-byte payload hex-encoded to 88 chars, optionally
suffixed with ":addr:size") fail that parse and produce a 400 Bad
Request. hipObject clients treat a 400 as a hard failure: they fall
back to the HTTP data path only when the response carries no
x-amz-rdma-reply header, so the 400 blocks the fallback entirely.
Reject structurally incompatible token schemes with 501 Not
Implemented instead. The first field being longer than 16 hex chars
(and entirely hex) cannot be a valid cuObject base address, so this
never reclassifies well-formed cuObject tokens; non-hex fields still
fall through to the existing malformed-token 400 path. The 501 response
is serialized and sent directly from the middleware because the global
error handler collapses non-fiber errors into 500, and the
x-amz-rdma-reply header is deliberately left unset so RDMA-capable
clients recognize the gateway as RDMA-unsupported for this request and
use the HTTP path.
Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
Introduce an opt-in mode that derives ETags from checksums instead of relying on
MD5 assumptions, aligning with AWS-compatible scenarios where ETag should be
treated as an opaque identifier rather than a content-MD5 signal.
This can significantly reduce CPU load on the server to disable MD5 checksums
when we are already validating data with other checksums.
This change adds --data-integrity-etag and VGW_DATA_INTEGRITY_ETAG support for
both posix and scoutfs backends. In this mode, PUT object ETags, multipart part
ETags, and completed multipart object ETags are checksum-derived.
This change introduces an opt-in O_DIRECT mode for POSIX object data paths while
keeping behavior safe and predictable across different filesystems and kernel
constraints. We now use direct I/O when available and beneficial, but preserve
correctness by falling back to buffered I/O when runtime read behavior indicates
alignment or capability mismatches.
The implementation keeps fast paths available for full-object reads and
descriptor-to-descriptor copy operations so kernel-level optimizations can still
be used where possible. At the same time, it avoids global assumptions from
single runtime failures and performs fallback at the stream level so requests
can continue successfully without broad feature disablement.
Multipart partNumber validation was inconsistent because some handlers still
enforced the hardcoded S3 default while others used the instance-level
mpMaxParts setting. This created a split API contract where changing mpMaxParts
did not reliably affect all relevant endpoints. This change routes all
partNumber upper-bound checks through a shared effective limit helper so
deployments that customize mpMaxParts get consistent behavior across GetObject,
HeadObject, UploadPart, and UploadPartCopy, while preserving the default
S3-compatible maximum when mpMaxParts is unset.
Ben McClellandandGitHub
niksis02
dependabot[bot]andGitHub
Jihyeon Gim
m-martin-78
Luke McCrone
buttilda