mirror of
https://github.com/versity/versitygw.git
synced 2026-08-17 20:56:21 +00:00
11e10b45a84e47bdd8fb930601cde2016a1fe566
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 niksis02
|
11e10b45a8 |
feat: integrate standalone IAM service with S3 gateway for identity-based policy enforcement
Fixes #1327 Fixes #1567 Closes #2264 Wires the S3 gateway up to the standalone IAM service so identity policies, not just bucket policies and ACLs, are enforced on the S3 data plane. The gateway authenticates SigV4 requests by calling new private derive-signing-key and resolve-identity endpoints on the IAM service instead of holding secrets itself, and evaluates identity policy through the same PolicyEvaluator path added to auth.VerifyAccess, combined with the bucket policy using explicit-deny-wins precedence. The private endpoints are served over their own mTLS listener (new iamapi/private package, genmtlscerts.sh to generate test material, and client-cert support in internal/netutil), separate from the public IAM API. As part of this the vendored aws/signer/v4 package is deleted and replaced by a pure-Go SigV4 implementation in internal/sigv4auth, which now reads canonical request data directly off the fiber.Ctx instead of reconstructing an http.Request, and is shared by both the S3 request-signing verification and the new private-endpoint signing. DeleteObjects moves from an all-or-nothing authorization check to true partial success: VerifyObjectsAccess evaluates every object in a batch independently against both the identity policy and any object lock, so a denial or a locked object only removes that key from the batch instead of failing the whole request. It also batches the identity-policy round trip and the bucket-policy fetch once per request rather than once per object, and separates plain deletes from versioned ones since a versioned delete needs s3:DeleteObjectVersion rather than s3:DeleteObject. Object lock handling got a few correctness fixes alongside this: a bypass is now modeled as BypassNone/BypassRequested/BypassOverwrite rather than a single bool, because root's blanket ability to override a GOVERNANCE retention should only apply when the client actually asked to bypass it (DeleteObject/DeleteObjects/PutObjectRetention), not when the gateway is silently replacing a locked object via an overwrite, which needs the permission from everyone including root. Retention changes are now correctly classified as an extension (allowed under plain s3:PutObjectRetention) versus a weakening (date or mode change, which needs the bypass permission), and a COMPLIANCE lock can never be weakened by anyone regardless of permissions, matching AWS. Separately, VerifyObjectCopyAccess had a readonly-mode gap: it returned early for root/admin before ever calling VerifyAccess, so the readonly check inside VerifyAccess never ran for them on CopyObject; access checks are now ordered so the readonly gate always applies before any root/admin bypass, for copy as well as every other write path. Bucket policies also gained Condition block support, via a new shared internal/condition package moved out of the IAM policy package since both bucket and identity policies share the same evaluation semantics. It implements the full AWS operator set — String{Equals,NotEquals,EqualsIgnoreCase,NotEqualsIgnoreCase,Like,NotLike}, Numeric{Equals,NotEquals,LessThan,LessThanEquals,GreaterThan,GreaterThanEquals}, Date{Equals,NotEquals,LessThan,LessThanEquals,GreaterThan,GreaterThanEquals}, Bool, BinaryEquals, Arn{Equals,Like,NotEquals,NotLike}, IpAddress/NotIpAddress, and Null — along with the ForAllValues/ForAnyValue set qualifiers and the IfExists modifier. A new requestConditionContext builds the per-request keys a bucket policy's Condition block can reference — aws:SourceIp, aws:SecureTransport, aws:CurrentTime, aws:EpochTime, aws:UserAgent, aws:Referer, s3:prefix, s3:delimiter, s3:max-keys, s3:x-amz-acl, s3:VersionId — following AWS's own per-action rules for which keys a given S3 operation actually populates. Identity-derived keys such as aws:PrincipalArn and aws:username are deliberately left unwired here, since the gateway has no way to know them; the standalone IAM service fills those in itself when it evaluates an identity policy. Also added new integration test suites for S3-side IAM: s3_iam_access_control.go and s3_iam_session_access_control.go cover identity-policy enforcement and session-credential requests against real S3 operations, alongside expanded OIDC/web-identity coverage and a new runoidctests.sh runner wired into the OIDC GitHub Actions workflow. |
||
|
niksis02
|
4756b4d236 |
feat: add STS web identity federation, IAM policy Condition support, and access control enforcement
Implements the `AssumeRoleWithWebIdentity` and `GetCallerIdentity` STS actions, letting callers exchange an external OIDC token for temporary credentials scoped to an IAM role. Token handling covers JWT claim parsing, issuer/audience resolution (including `azp` override semantics), JWKS fetching and caching with `singleflight`-deduplicated refresh, and rate-limited forced refresh on unrecognized `kid` values. OIDC provider thumbprint fetching now performs a real TLS handshake verified against the system trust store and the provider hostname (previously `InsecureSkipVerify`), since the observed certificate is persisted as a long-lived trust anchor rather than used once and discarded; all discovery-document and JWKS fetches go through an SSRF-safe HTTP client with bounded redirects and response size.
Adds policy `Condition` block evaluation, supporting `String`, `Numeric`, `Date`, `Bool`, `BinaryEquals`, and `IpAddress` operators along with their `IfExists`/`Not` variants and `ForAllValues`/`ForAnyValues` set qualifiers, plus policy variable substitution (e.g. `${aws:username}`) in supported operators. Adds identity-based inline policy evaluation and a new IAM authorization middleware that authorizes each request against action, resource, and condition context together, applying the session-policy-intersects-role-policy semantics for assumed-role sessions.
Adds a new debug logger `--log-level` flag (`silent`/`debug`/`unsafe`), along with a tree-based XML masker that redacts secrets and tokens at the property level in logged request/response bodies instead of skipping the whole body. The old `--debug/VGW_DEBUG` flag is kept as a deprecated alias for `--log-level=debug`, printing a console warning that points users at `--log-level` for finer-grained control.
Fixes a Vault storage bug where CAS (check-and-set) writes always read the current document version as 0 because `kvVersion` asserted metadata as `float64` while the Vault client actually returns `json.Number`, causing every write past the first to be rejected as a concurrent modification. Also adds a constant-time `SecureCompare` for signature/token comparisons in sigv4 auth.
Adds an integration test suite (`iam_access_control.go`) covering IAM access control across user, role, and session identities.
|
||
|
niksis02
|
62f9cc1cc9 |
feat: add IAM OIDC provider CRUD
Add support for `CreateOpenIDConnectProvider`, `GetOpenIDConnectProvider`, `ListOpenIDConnectProviders`, `DeleteOpenIDConnectProvider`, `AddClientIDToOpenIDConnectProvider`, `RemoveClientIDFromOpenIDConnectProvider`, and `UpdateOpenIDConnectProviderThumbprint` on both the internal and Vault storage backends, rounding out the standalone IAM service with the same OIDC identity provider management AWS IAM exposes. CreateOpenIDConnectProvider validates the issuer URL, enforces the client ID and per-provider client ID list limits, and accepts an optional ThumbprintList. When the caller omits ThumbprintList, the provider auto-fetches the thumbprint by opening an outbound TLS connection to the issuer URL and hashing its top-level CA certificate, matching real AWS behavior. This auto-fetch is configurable: it can be turned off with the `--disable-oidc-thumbprint-autofetch` CLI flag (or the `VGW_IAM_DISABLE_OIDC_THUMBPRINT_AUTOFETCH` environment variable) for restricted or air-gapped deployments where the IAM server shouldn't make outbound connections, in which case an omitted ThumbprintList is rejected instead. AddClientIDToOpenIDConnectProvider and RemoveClientIDFromOpenIDConnectProvider manage a provider's client ID list, and UpdateOpenIDConnectProviderThumbprint replaces its thumbprint list, all with the same length and format validation applied at creation time. Provider ARNs are derived from the issuer URL, and GetOpenIDConnectProvider and DeleteOpenIDConnectProvider resolve providers by ARN, returning NoSuchEntity when a provider doesn't exist. ListOpenIDConnectProviders returns the full set of stored providers. These actions are wired into the IAM API router and given their own XML response types under iamapi/types, with a new iamapi/internal/iamutil package handling URL validation, thumbprint fetching and normalization, and ARN construction shared across the controller methods. |
||
|
niksis02
|
3227a629dc |
feat: add AWS-compatible standalone IAM service
Closes #1640 Add a standalone AWS IAM Query API implementation for managing IAM users through standard AWS SDKs and the AWS CLI. Server usage Start the IAM server with internal file-backed storage: mkdir -p /tmp/versitygw-iam ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --dir /tmp/versitygw-iam Start the IAM server with Vault KV v2 storage using AppRole: VGW_IAM_VAULT_ROLE_SECRET=<role-secret> ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --vault-endpoint-url http://127.0.0.1:8200 --vault-auth-method approle --vault-role-id <role-id> --vault-mount-path kv --vault-secret-storage-path iam Vault authentication also supports root tokens, separate authentication and secret-storage namespaces, custom mount paths, server certificate validation, and mutual TLS client certificates. Configure the AWS CLI credentials used by the IAM server: export AWS_ACCESS_KEY_ID=user export AWS_SECRET_ACCESS_KEY=pass export AWS_DEFAULT_REGION=us-east-1 Implemented IAM actions CreateUser creates an IAM user with an AWS-compatible ARN, generated AIDA user ID, creation timestamp, optional path, and tags. It validates usernames, paths, tag limits, reserved tag prefixes, duplicate tag keys, and existing users. aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob --path /engineering/ --tags Key=team,Value=storage GetUser returns a stored user or the root identity when requested without a username through the IAM Query API. aws --endpoint-url http://127.0.0.1:7070 iam get-user --user-name bob ListUsers returns users in deterministic username order and supports path filtering, marker-based pagination, and MaxItems limits. aws --endpoint-url http://127.0.0.1:7070 iam list-users aws --endpoint-url http://127.0.0.1:7070 iam list-users --path-prefix /engineering/ --max-items 100 UpdateUser updates the username and/or path, recalculates the user ARN, and rejects conflicts with existing users. aws --endpoint-url http://127.0.0.1:7070 iam update-user --user-name bob --new-user-name robert --new-path /platform/ DeleteUser permanently removes an IAM user and returns AWS-compatible errors for missing users. aws --endpoint-url http://127.0.0.1:7070 iam delete-user --user-name robert IAM protocol and authentication - Support the AWS IAM Query protocol version 2010-05-08 over GET and POST form requests. - Return AWS-compatible XML responses, error documents, status codes, request IDs, user metadata, and pagination fields. - Authenticate root credentials with AWS Signature Version 4 for the IAM service in us-east-1. - Support both Authorization-header and query-string SigV4 authentication. - Validate credential scope, signed headers, timestamps, clock skew, content length, signatures, and unsupported signature or session-token modes. - Add IAM-specific validation and error mapping for malformed requests, invalid actions, duplicate entities, missing users, throttling, and internal failures. Storage implementations - Add an internal JSON-backed store using iam.json and iam.json.backup with atomic temporary-file replacement, concurrent access protection, stable ordering, pagination, and persistence across restarts. - Add a Vault KV v2 store with one secret per user, CAS-based duplicate protection, permanent deletion, AppRole reauthentication, namespace support, configurable authentication and KV mounts, root-token authentication, and TLS/mTLS configuration. - Introduce a common Storer interface and require exactly one storage backend to be configured. Server and embedding support - Register the new `versitygw iam` command with environment-variable and CLI configuration for both storage backends. - Add `embedgw.RunIAMAPI` and `IAMConfig` for embedding the IAM service in Go applications. Gateway-level internal packages - Add `internal/iamstore` as a reusable generic file-backed IAM persistence engine and migrate the existing gateway internal IAM service to it. - Add `internal/sigv4auth` for shared SigV4 header and presigned-query parsing, canonical request generation, signature verification, and structured authentication errors. - Refactor the S3 authentication paths to use the shared SigV4 implementation while preserving S3-specific error responses. - Add `internal/httpctx` for shared Fiber context keys and AWS-style request ID handling. - Add `internal/routekit` for shared query, form, and header route matchers. - Add `internal/netutil` for reusable certificate storage, hostname-aware listeners, multi-address serving, TLS listeners, and UNIX socket handling. - Update the custom SigV4 signer to honor an explicitly supplied signed-header list so unrelated headers do not alter IAM signatures. Testing and CI - Add AWS IAM SDK-based integration coverage for all supported user actions, header authentication, query authentication, validation, errors, filtering, and pagination. - Split standalone IAM tests into `versitygw test iam` and retain existing gateway IAM tests under `versitygw test gw-iam`. - Add unit coverage for controllers, authentication, routing, storage, embedding, listeners, request matching, persistence, and signing behavior. - Add `runiamtests.sh` to exercise internal storage over HTTP and HTTPS plus Vault storage through AppRole. - Add a dedicated IAM functional-test workflow with a Vault service and merged runtime coverage reporting. - Include the IAM test runner in shellcheck and add the AWS IAM SDK dependency. |