The previous revision hard-coded .zfs next to .sgwtmp in the
ListObjects, ListObjectsV2 and ListObjectVersions walks. That hides a
directory named .zfs on filesystems that don't have ZFS snapshots, and
it gives an operator no way to exclude a different control directory.
Add PosixOpts.ExcludeDirs and build the effective skip list once in New
from the gateway's own metadata directory plus the configured entries,
so all three listing operations read the same list. The new
--exclude-dirs flag (VGW_EXCLUDE_DIRS) exposes it, so a dataset mounted
with snapdir=visible is handled with --exclude-dirs .zfs.
.zfs isn't in the default list: on a filesystem without ZFS snapshots a
directory with that name is ordinary bucket content. An operator who
needs the skip sets --exclude-dirs .zfs.
Behind a reverse proxy the request log and the S3, admin, website and WebUI
access logs record the proxy's address instead of the client's. Add a
client-ip-header option that takes the logged address from X-Forwarded-For or
X-Real-Ip. It is off by default, so an existing deployment logs exactly what it
logged before.
The header is only honoured when the connecting peer is listed in the new
trusted-proxies list of IPs and CIDRs. Without that, the value is whatever the
client sent: nginx's standard
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for appends to a
client-supplied header, leaving the client's own address leftmost.
X-Forwarded-For is walked right to left, skipping the listed proxies, so the
first unlisted address is the client and a forged leftmost entry is never
reached; the two-proxy case from #2243 works once both proxies are listed.
Every header line is walked (a proxy may add its own line instead of
appending), ip:port entries are accepted, and a request from an unlisted peer
keeps the socket address. Setting the header with an empty trusted-proxies list
fails startup rather than silently trusting every peer.
X-Real-Ip reads the last header line, so a proxy that adds a line rather than
replacing the client's is the one that speaks. An address carrying an IPv6 zone
is rejected: the zone is free text that the space-delimited access log would
write verbatim, shifting every field after RemoteIP. When every hop in
X-Forwarded-For is a trusted proxy, the leftmost hop is logged, as nginx's
real_ip_recursive does, since the socket peer is then the nearest proxy. The
trusted-proxies list accepts the entry "unix" to trust the peer of a listener
bound to a unix socket path, which fasthttp reports as 0.0.0.0.
The resolver, middleware, log tag and parsers live in internal/clientip, shared
by s3api, iamapi, website and webui; s3api/utils re-exports them for the S3
callers, so the standalone IAM service no longer pulls in the S3 side. On this
branch go list -deps ./iamapi drops from 375 to 299 packages, against 298 on
main.
ctx.IP() is unchanged and still returns the socket peer address, because it
feeds aws:SourceIp in IAM policy conditions and SourceIP in policy evaluation;
only the address written to the logs and S3 event records comes from the
header.
Fixes#2243.
Fixes#2464
`versitygw iam` never set `IAMConfig.SigHup`, so the reload loop in `RunIAMAPI` never ran and the IAM service kept serving the certificates it started with. It is now connected to `gwcli.SigHup`.
The signal forwarder used blocking sends into the one-slot `SigHup` and `SigDone` channels. In modes that never read `SigHup` (`iam` until this fix, `admin`, `utils`), the second SIGHUP left the forwarder stuck, every later SIGINT/SIGTERM was lost, and the process could only be killed. `forwardSignals` now sends without blocking, and a notification that is already pending stands in for the new one.
The private listener loaded its client CA pool (`--private-client-ca`) only once. `netutil.TLSOptions.ClientCAs` is replaced by `GetClientCAs`, which `GetConfigForClient` calls on every handshake and which is backed by the new `netutil.CAPoolStorage`, so a reloaded bundle applies to the next connection. When no pool is loaded the listener refuses the client rather than falling back to the system roots. On SIGHUP the IAM service now also reloads the client CA pool and the certificate of the WebUI it hosts, in addition to its public and private certificates.
The gateway's standalone IAM client loaded its client cert/key and server CA (`--iam-standalone-client-cert[-key]`, `--iam-standalone-server-ca`) only once. With the new `auth.CertReloader` interface, the gateway's SIGHUP handler reloads them through `IAMServiceStandalone.ReloadCerts`. It builds a new `http.Client` from the files, swaps it in atomically and closes the old client's idle connections. Requests already in flight finish on the old client, and a failed load keeps the current one. The transports now set `IdleConnTimeout`, so a connection left on a replaced client cannot stay open indefinitely. The IAM wrapper in `vgwrdma` re-exports `CertReloader` so the capability is not lost there.
Closes#2322
The admin API could not be served under a path prefix. Admin requests are SigV4-signed over the full path, so a reverse proxy or Gateway API route cannot strip a prefix before forwarding. That made it impossible to serve the WebUI and the admin API under one hostname. The new `--admin-path-prefix` (`VGW_ADMIN_PATH_PREFIX`) option mounts the admin routes under a single-segment prefix such as `/admin`, on `--admin-port` or, when that is unset, on the S3 port. Admin clients include the prefix in their endpoint URL, which the admin CLI and the WebUI already support.
The prefix is limited to unreserved characters, because clients and the gateway encode other characters differently when signing. When the admin API shares the S3 port, the prefix must also differ from `--webui-s3-prefix`, otherwise the WebUI mount answers the admin requests. Auto-detected WebUI admin gateway URLs and the startup banner now include the prefix. The admin routes served on the S3 port now reuse `S3AdminRouter` instead of a duplicated route list. This also stops the standalone admin server from recording bucket creation under the `ActionAdminListBuckets` action.
The Helm chart now exposes `webui.pathPrefix`, `webui.s3Prefix` and `admin.pathPrefix`. The WebUI gateway lists are also passed when only `webui.s3Prefix` is set, since the WebUI hosted on the S3 port does not require `webui.enabled`.
Internal IAM data can now live on its own volume. `iam.dir` sets the IAM directory, and `iam.persistence` creates or references a dedicated PVC when `iam.enabled` is true and `iam.type` is `internal`. Otherwise IAM data stays in the `iam` subdirectory of the backend data volume, so existing releases are unaffected.
The chart now rejects overlapping storage directories. `iam.dir` must be an absolute path outside `/mnt/data`, `gateway.backend.sidecarDir` and `gateway.backend.versioningDir`, and those three directories must not overlap each other. Deleting a bucket removes `<dir>/<bucket>` from the sidecar and versioning directories, so with nested directories, deleting a suitably named bucket could wipe IAM accounts, object versions or the entire backend data directory.
`AssumeRoleWithWebIdentity` always fetched a provider's discovery document from `<provider url>/.well-known/openid-configuration`, so an identity provider that issues tokens naming a public issuer while serving its metadata and keys on a cluster-internal path could not be used: reaching it meant relaxing the endpoint checks for every registered provider. `--oidc-discovery-url` moves that one fetch to an operator-named endpoint, which is how keys can be looked up over an optimized private path while the tokens themselves stay verifiable from the public internet against the issuer alone, as the JWT spec requires.
The flag takes `<provider url>=<discovery url>` pairs, can be repeated once per provider, and is also read from `VGW_IAM_OIDC_DISCOVERY_URLS` as a comma-separated list; the Helm chart exposes the same list as `iamServer.oidc.discoveryUrls`. The discovery URL is fetched exactly as written, so it must carry the `/.well-known/openid-configuration` path when the provider serves it there. A malformed pair is rejected at startup rather than at the first assume-role call.
Only the fetch moves. The provider URL is still what a token's `iss` claim is matched against, the fetched document's own `issuer` field must still equal it, and the key set still comes from the `jwks_uri` that document publishes. A configured discovery endpoint is named by the operator at startup rather than by a request, so it and the `jwks_uri` it publishes waive the private-address check for that provider's fetch chain only, without `--oidc-allow-private-endpoints` and its far broader effect on every other provider. Transport rules are unchanged: a plaintext discovery URL still requires `--oidc-allow-insecure-transport`.
Thumbprint auto-fetch follows the override and pins the discovery endpoint's certificate chain, since that is the host every later fetch is verified against.
Closes#2364
`AssumeRoleWithWebIdentity` only ever trusted an `OIDC` provider reachable over verified `https`, at a publicly routable address, on the implicit `:443`. That posture is right for an internet-facing IdP but rejects every address an internal one can have, so a `SPIFFE/SPIRE` OIDC discovery provider in the same cluster — or as a sidecar in the same pod — could never be registered, let alone verified against, and no setting could express "this private address is the IdP".
Two opt-in flags on `versitygw iam`, both off by default:
`--oidc-allow-private-endpoints`
Permit a provider `Url` resolving to a loopback/private/link-local address, and an explicit port. Transport is unchanged: still `https`, still fully verified (a self-signed in-cluster cert is trusted the way AWS documents, through `ThumbprintList`).
`--oidc-allow-insecure-transport`
Additionally permit plaintext `http` provider URLs, discovery/JWKS endpoints and redirects, and drop TLS verification (`thumbprint` pinning included) for `https` ones.
Both apply uniformly to the thumbprint auto-fetch at `CreateOpenIDConnectProvider` time and to the discovery-document plus `JWKS` fetch at `AssumeRoleWithWebIdentity` time. Neither weakens anything past the endpoint: signature verification, issuer matching, audience and trust policy evaluation are untouched, and the DNS-resolve-once/dial-the-resolved-IP shape stays in place so a rebind still cannot redirect a connection.
An `http` provider keeps its scheme in its stored `Url`, `ARN` and `iss` matching, rather than being stripped like an `https` one — otherwise `"http://host"` and `"https://host"` would collapse onto a single ARN and storage key and each could satisfy the other's trust policy. It also stores an empty `ThumbprintList` rather than failing: a plaintext provider presents no certificate to thumbprint.
Helm: `iamServer.oidc.{allowPrivateEndpoints,allowInsecureTransport}`, alongside `disableThumbprintAutofetch` moved into the same block (the flat `iamServer.disableOidcThumbprintAutofetch` stays honored). Chart `0.4.1 -> 0.4.2`.
The WebUI's create-provider form no longer rejects `http` URLs and ports client-side; it cannot see the service's settings, so those two rules are left to the server, whose error surfaces as a toast like any other.
Gates bucket listing behind an identity policy, lets browsers reach the standalone IAM API, and turns the WebUI into a dashboard for S3, IAM, or both.
**Bucket listing.** `ListBuckets` is now gated by the new `s3:ListAllMyBuckets` action, evaluated against `arn:aws:s3:::*`. The request names no bucket, so only identity policies apply — there is no resource policy to combine with, which is the same shape `CreateBucket` already had, so both now share one identity-only evaluation path. Root and admin bypass it, and backends with no identity-policy layer keep listing as before since their listing is already narrowed to the caller's own buckets. The action is IAM-only and is deliberately absent from the bucket-policy action list.
**Fixed bucket ownership.** The standalone IAM client has no per-user ownership to express — accounts are all plain users, cannot be enumerated, and access is decided by policy rather than ACL — so it now implements `auth.FixedBucketOwner` and every bucket is owned by root. Bucket creation stops resolving an owner, `ListBuckets` returns every bucket to every caller (what they may then do with one stays a per-request policy decision), and the admin `ChangeBucketOwner` reports method-not-supported. Other IAM backends are untouched.
**IAM service CORS.** `--cors-allow-origin` now applies to the `iam` command: it answers preflights and stamps the CORS headers, mirroring back the requested method and headers rather than enumerating the SigV4 header set. Without it no browser can reach the IAM API at all, so setting `--webui` without it falls back to `*` with a warning. The chart gets `iamServer.corsAllowOrigin`.
**WebUI.** New IAM pages for users, roles and OIDC providers, signing IAM/STS query-form requests directly from the browser. Navigation is capability-gated rather than role-gated: on sign-in the session probes the S3, admin and IAM endpoints independently and each page shows only what those credentials actually reach, so one build serves an IAM-only dashboard, an S3-only dashboard, and a combined one. The login page takes an optional IAM endpoint, seeded from the new `--webui-iam-gateways` (chart: `webui.iamGateways`) — never auto-detected, since the IAM service is a separate process. The WebUI can also be hosted by `versitygw iam` itself, for deployments with no S3 gateway behind it.
**The admin API is ignored once an IAM endpoint is in play.** The IAM service is then the user directory and bucket ownership is fixed, which leaves the admin API no job: the session is given no admin endpoint at all, its login field is hidden, `users.html` redirects to its IAM counterpart, and every admin-only surface stays off screen. Dashboard and Buckets remain available to any S3 session in such a deployment, running on the S3 and IAM APIs alone and surfacing each denial per action instead of redirecting.
Also fixes two WebUI bugs: embedded assets went out with a zero modification time and no `Cache-Control`, so browsers treated them as fresh for centuries and an upgraded gateway served new HTML against stale JS — they now revalidate against an ETag; and the login page's advanced-options section clipped its last field, since it animated to a height named in the stylesheet rather than the one it measures now.
**Usage**
IAM-only dashboard, served by the IAM service:
versitygw iam --port :7076 --webui :8080 --cors-allow-origin http://localhost:8080/
IAM + S3, dashboard served by the IAM service — point it at the gateway with `--webui-gateways`, and let the gateway accept the dashboard's origin:
versitygw iam --port :7076 --webui :8080 --webui-gateways http://localhost:7070/ --cors-allow-origin http://localhost:8080/
versitygw --port :7070 --cors-allow-origin http://localhost:8080/ posix /data
IAM + S3, dashboard served by the S3 gateway — point it at the IAM service with `--webui-iam-gateways`, and let the IAM service accept the dashboard's origin:
versitygw --port :7070 --webui :8080 --webui-iam-gateways http://localhost:7076/ posix /data
versitygw iam --port :7076 --cors-allow-origin http://localhost:8080/
This adds a new iamServer Deployment, split public and private Services, a PersistentVolumeClaim, and cert-manager Certificate resources so the standalone versitygw iam API server can be deployed directly from this chart, independently scalable from the S3 gateway and backed by either internal file storage or HashiCorp Vault, with Vault auth credentials and Vault TLS material kept in separate Kubernetes secrets. The gateway side gains iam.type=standalone client wiring that reaches the IAM service over its private mTLS endpoint, with certificates supplied either through an existing secret or auto-provisioned via cert-manager using a shared CA-type issuer so both peers can verify each other from their own certificate's ca.crt, and the chart auto-targets the in-chart service when no external endpoint is configured. gateway.logLevel and iamServer.logLevel replace the old boolean debug flag with the silent, debug, and unsafe levels the binary now supports, docker-entrypoint.sh gained iam as a recognized VGW_BACKEND value so the new deployment can start through the existing entrypoint, NetworkPolicy resources were corrected to use proper peer lists and to default to deny instead of allow when no ingress rules are configured, pod and Deployment selector labels were separated between the gateway and the IAM server to prevent them from matching each other's Services, and a battery of template time validation guards was added to fail fast on invalid combinations such as multiple replicas against the internal file store or a missing certificate for a TCP private endpoint, together with expanded helm lint coverage in CI for these new configurations.
Implements the `AssumeRoleWithWebIdentity` and `GetCallerIdentity` STS actions, letting callers exchange an external OIDC token for temporary credentials scoped to an IAM role. Token handling covers JWT claim parsing, issuer/audience resolution (including `azp` override semantics), JWKS fetching and caching with `singleflight`-deduplicated refresh, and rate-limited forced refresh on unrecognized `kid` values. OIDC provider thumbprint fetching now performs a real TLS handshake verified against the system trust store and the provider hostname (previously `InsecureSkipVerify`), since the observed certificate is persisted as a long-lived trust anchor rather than used once and discarded; all discovery-document and JWKS fetches go through an SSRF-safe HTTP client with bounded redirects and response size.
Adds policy `Condition` block evaluation, supporting `String`, `Numeric`, `Date`, `Bool`, `BinaryEquals`, and `IpAddress` operators along with their `IfExists`/`Not` variants and `ForAllValues`/`ForAnyValues` set qualifiers, plus policy variable substitution (e.g. `${aws:username}`) in supported operators. Adds identity-based inline policy evaluation and a new IAM authorization middleware that authorizes each request against action, resource, and condition context together, applying the session-policy-intersects-role-policy semantics for assumed-role sessions.
Adds a new debug logger `--log-level` flag (`silent`/`debug`/`unsafe`), along with a tree-based XML masker that redacts secrets and tokens at the property level in logged request/response bodies instead of skipping the whole body. The old `--debug/VGW_DEBUG` flag is kept as a deprecated alias for `--log-level=debug`, printing a console warning that points users at `--log-level` for finer-grained control.
Fixes a Vault storage bug where CAS (check-and-set) writes always read the current document version as 0 because `kvVersion` asserted metadata as `float64` while the Vault client actually returns `json.Number`, causing every write past the first to be rejected as a concurrent modification. Also adds a constant-time `SecureCompare` for signature/token comparisons in sigv4 auth.
Adds an integration test suite (`iam_access_control.go`) covering IAM access control across user, role, and session identities.
This change is needed to expose the admin API and WebUI through standard
Kubernetes Ingress resources instead of only through service access or Gateway
API routes. It adds configurable ingress blocks in values.yaml, and renders
dedicated ingress objects for admin and webui when enabled.
Also update corsAllowOrigin comment.
Fixes#2251Fixes#2252
The Helm chart now supports dedicated sidecar metadata and versioning
directories alongside the primary data directory. For POSIX deployments,
users can configure a sidecar metadata path without forcing backend flags
through a single args string, and POSIX or ScoutFS deployments can expose
a persistent versioning directory through the chart as well.
This makes the chart usable for filesystem-backed setups that need
sidecar metadata storage or object versioning support, while keeping the
configuration aligned with the existing persistent data and IAM layout.
The chart version is bumped to reflect the new deployment behavior.
Fixes#2204Fixes#2216
Replace PutBucketWebsite, GetBucketWebsite, DeleteBucketWebsite
NotImplemented test stubs with comprehensive integration tests covering:
- non-existing bucket errors
- validation (empty suffix, suffix with slash, invalid protocol, mutual
exclusion of RedirectAllRequestsTo and IndexDocument)
- successful put/get round-trips for both index+error and redirect-all configs
- delete idempotency and verification
Signed-off-by: Marc Singer <marc@singer.gg>
Add error document serving, routing rules, and integration tests
Implement Features 1 and 2 of S3 static website hosting:
- WebsiteErrorDocument controller wrapper intercepts 4xx errors on
website-enabled buckets and serves the configured error document or
evaluates post-request routing rules (error code match redirects)
- ResolveWebsiteIndex middleware now caches parsed WebsiteConfiguration
in context, handles RedirectAllRequestsTo, evaluates pre-request
routing rules (key prefix match redirects), and rewrites directory
keys for index document
- MatchPreRequestRule and MatchPostRequestRule methods on
WebsiteConfiguration for routing rule evaluation
- 14 unit tests for routing rule matching
- 7 integration tests covering error document, routing rules,
redirect-all, and index document behavior
Signed-off-by: Marc Singer <marc@singer.gg>
Add separate website hosting endpoint with virtual-host routing
Signed-off-by: Marc Singer <marc@singer.gg>
Support catch-all mode for website endpoint when --website-domain is omitted
Signed-off-by: Marc Singer <marc@singer.gg>
Ben McClellandandGitHub
Tung Lam
niksis02
kfox1111andGitHub
Mart-Kuc
Peter Dahlberg
Marc Singerandniksis02
shoce
Jack HenschelandBen McClelland
Anthony Clerici