Follow-up to #2514, which added PosixOpts.ExcludeDirs (--exclude-dirs,
VGW_EXCLUDE_DIRS) so the listings can skip filesystem control
directories such as ZFS's .zfs.
The walks matched skipped names against every directory entry, files
included. In WalkVersions, returning fs.SkipDir for a file skips the
rest of its parent directory, so a regular file named like an excluded
directory (for example an object "d/.zfs") made ListObjectVersions drop
every sibling that sorted after it. Walk hid the file itself.
Previously only .sgwtmp was skipped, which clients can't create, but
excluded names are chosen by the operator and any client can PUT such a
key. Walk and WalkVersions now skip only directories, so a file with an
excluded name is listed like any other object.
Excluded names were also not reserved: requests could still read and
write objects under an excluded directory that the listings never show.
On ZFS, .zfs is reachable by name even with snapdir=hidden, so GetObject
could read snapshot contents. That bypasses bucket policies scoped to a
key prefix and exposes data that was deleted or overwritten since the
snapshot was taken. On other filesystems, objects written there were
orphans that no listing reports. validateObjectName now rejects, with
ReservedObjectKey, any key that has an excluded name as a directory at
any depth, ignoring case and after resolving "..", "." and repeated
slashes. Every object operation already calls it. A directory-object
key (trailing slash) counts its last element as a directory; a plain
key with an excluded last element is still a valid file. The new
ErrExcludedObjectKey keeps the ReservedObjectKey code with a description
that fits excluded directories, leaving the .sgwtmp error unchanged.
Nothing changes when ExcludeDirs is empty.
posix.New now also rejects "." and ".." as excluded names, since they
can never match a directory entry and would silently do nothing.
The previous revision hard-coded .zfs next to .sgwtmp in the
ListObjects, ListObjectsV2 and ListObjectVersions walks. That hides a
directory named .zfs on filesystems that don't have ZFS snapshots, and
it gives an operator no way to exclude a different control directory.
Add PosixOpts.ExcludeDirs and build the effective skip list once in New
from the gateway's own metadata directory plus the configured entries,
so all three listing operations read the same list. The new
--exclude-dirs flag (VGW_EXCLUDE_DIRS) exposes it, so a dataset mounted
with snapdir=visible is handled with --exclude-dirs .zfs.
.zfs isn't in the default list: on a filesystem without ZFS snapshots a
directory with that name is ordinary bucket content. An operator who
needs the skip sets --exclude-dirs .zfs.
The posix backend publishes an object by linking or renaming a temp file
into place, but never flushes it. An acknowledged PUT can therefore be
lost, or come back empty, after a crash or power loss, depending on the
filesystem's writeback interval.
With --fsync (VGW_FSYNC), the backend fsyncs the temp file, which covers
its data and xattr metadata, before publishing it. After publishing, it
fsyncs the object's directory and each parent up to the bucket, so that
the new entry and any directories created for it are durable. A failed
flush fails the request. The option is off by default, so existing
deployments are unaffected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fixes#2464
`versitygw iam` never set `IAMConfig.SigHup`, so the reload loop in `RunIAMAPI` never ran and the IAM service kept serving the certificates it started with. It is now connected to `gwcli.SigHup`.
The signal forwarder used blocking sends into the one-slot `SigHup` and `SigDone` channels. In modes that never read `SigHup` (`iam` until this fix, `admin`, `utils`), the second SIGHUP left the forwarder stuck, every later SIGINT/SIGTERM was lost, and the process could only be killed. `forwardSignals` now sends without blocking, and a notification that is already pending stands in for the new one.
The private listener loaded its client CA pool (`--private-client-ca`) only once. `netutil.TLSOptions.ClientCAs` is replaced by `GetClientCAs`, which `GetConfigForClient` calls on every handshake and which is backed by the new `netutil.CAPoolStorage`, so a reloaded bundle applies to the next connection. When no pool is loaded the listener refuses the client rather than falling back to the system roots. On SIGHUP the IAM service now also reloads the client CA pool and the certificate of the WebUI it hosts, in addition to its public and private certificates.
The gateway's standalone IAM client loaded its client cert/key and server CA (`--iam-standalone-client-cert[-key]`, `--iam-standalone-server-ca`) only once. With the new `auth.CertReloader` interface, the gateway's SIGHUP handler reloads them through `IAMServiceStandalone.ReloadCerts`. It builds a new `http.Client` from the files, swaps it in atomically and closes the old client's idle connections. Requests already in flight finish on the old client, and a failed load keeps the current one. The transports now set `IdleConnTimeout`, so a connection left on a replaced client cannot stay open indefinitely. The IAM wrapper in `vgwrdma` re-exports `CertReloader` so the capability is not lost there.
ScoutFS delegates buffered object transfer paths to the
POSIX backend, so expose IOBufferSize in ScoutfsOpts and
pass it through to PosixOpts. This lets operators tune
transfer buffering for ScoutFS instead of being limited
to the default.
Closes#2304
Adds `versitygw utils convert-posix-dataset` (alias `cpd`), which makes a posix dataset not created by the gateway fully compatible with the posix backend. Every top level directory is treated as a bucket and gets a private ACL owned by `--access-key-id` (defaults to the root `--access`) and `BucketOwnerEnforced` object ownership; root level files are ignored. Metadata is stored in xattrs, or in `--sidecar-dir` when set.
`--calculate-etag` and `--checksum-algorithm` optionally compute the MD5 ETag and the `FULL_OBJECT` checksum of every object in a single streamed read. Objects are converted concurrently, tunable with `--concurrency` and `--read-buffer-size`. Existing bucket and object metadata is never overwritten.
`AssumeRoleWithWebIdentity` always fetched a provider's discovery document from `<provider url>/.well-known/openid-configuration`, so an identity provider that issues tokens naming a public issuer while serving its metadata and keys on a cluster-internal path could not be used: reaching it meant relaxing the endpoint checks for every registered provider. `--oidc-discovery-url` moves that one fetch to an operator-named endpoint, which is how keys can be looked up over an optimized private path while the tokens themselves stay verifiable from the public internet against the issuer alone, as the JWT spec requires.
The flag takes `<provider url>=<discovery url>` pairs, can be repeated once per provider, and is also read from `VGW_IAM_OIDC_DISCOVERY_URLS` as a comma-separated list; the Helm chart exposes the same list as `iamServer.oidc.discoveryUrls`. The discovery URL is fetched exactly as written, so it must carry the `/.well-known/openid-configuration` path when the provider serves it there. A malformed pair is rejected at startup rather than at the first assume-role call.
Only the fetch moves. The provider URL is still what a token's `iss` claim is matched against, the fetched document's own `issuer` field must still equal it, and the key set still comes from the `jwks_uri` that document publishes. A configured discovery endpoint is named by the operator at startup rather than by a request, so it and the `jwks_uri` it publishes waive the private-address check for that provider's fetch chain only, without `--oidc-allow-private-endpoints` and its far broader effect on every other provider. Transport rules are unchanged: a plaintext discovery URL still requires `--oidc-allow-insecure-transport`.
Thumbprint auto-fetch follows the override and pins the discovery endpoint's certificate chain, since that is the host every later fetch is verified against.
Conditional PUTs require a lock primitive that excludes competing gateway
processes sharing a backend filesystem. A successful flock call does not prove
that property: some clustered filesystem configurations accept flock but scope
it to one node, silently leaving cross-gateway check-and-publish races open.
Add an object-lock mode that lets operators select flock or fcntl for
filesystems where that primitive is cluster-coherent, local for the existing
per-process behavior, or none to reject conditional writes with NotImplemented.
Keep the legacy disable flag as an alias for local.
Shared lock modes now verify the selected primitive on the root lock filesystem
during startup and fail closed if it cannot be used. Runtime lock failures no
longer silently downgrade to process-local exclusion. The startup check cannot
establish cross-node coherence, so that remains an explicit operator
requirement.
ScoutFS defaults to none since posix locks are not cluster consistent, but
allow setting local for single node deployments.
Fixes#2351
Closes#2364
`AssumeRoleWithWebIdentity` only ever trusted an `OIDC` provider reachable over verified `https`, at a publicly routable address, on the implicit `:443`. That posture is right for an internet-facing IdP but rejects every address an internal one can have, so a `SPIFFE/SPIRE` OIDC discovery provider in the same cluster — or as a sidecar in the same pod — could never be registered, let alone verified against, and no setting could express "this private address is the IdP".
Two opt-in flags on `versitygw iam`, both off by default:
`--oidc-allow-private-endpoints`
Permit a provider `Url` resolving to a loopback/private/link-local address, and an explicit port. Transport is unchanged: still `https`, still fully verified (a self-signed in-cluster cert is trusted the way AWS documents, through `ThumbprintList`).
`--oidc-allow-insecure-transport`
Additionally permit plaintext `http` provider URLs, discovery/JWKS endpoints and redirects, and drop TLS verification (`thumbprint` pinning included) for `https` ones.
Both apply uniformly to the thumbprint auto-fetch at `CreateOpenIDConnectProvider` time and to the discovery-document plus `JWKS` fetch at `AssumeRoleWithWebIdentity` time. Neither weakens anything past the endpoint: signature verification, issuer matching, audience and trust policy evaluation are untouched, and the DNS-resolve-once/dial-the-resolved-IP shape stays in place so a rebind still cannot redirect a connection.
An `http` provider keeps its scheme in its stored `Url`, `ARN` and `iss` matching, rather than being stripped like an `https` one — otherwise `"http://host"` and `"https://host"` would collapse onto a single ARN and storage key and each could satisfy the other's trust policy. It also stores an empty `ThumbprintList` rather than failing: a plaintext provider presents no certificate to thumbprint.
Helm: `iamServer.oidc.{allowPrivateEndpoints,allowInsecureTransport}`, alongside `disableThumbprintAutofetch` moved into the same block (the flat `iamServer.disableOidcThumbprintAutofetch` stays honored). Chart `0.4.1 -> 0.4.2`.
The WebUI's create-provider form no longer rejects `http` URLs and ports client-side; it cannot see the service's settings, so those two rules are left to the server, whose error surfaces as a toast like any other.
Fail closed on unexpected advisory-lock errors instead of silently reducing
cross-process exclusion to a local mutex. Make local publish-slot waits honor
request cancellation. Move version snapshots under the publish lock so
concurrent versioned PUTs preserve publication order. Add regression coverage
for canceled lock waiters.
Also add an option to disable flock files and only rely on in process locking.
Replace the hardcoded 0644 defaultFilePerm with a NewFilePerm option on
the posix and scoutfs backends, exposed as the --file-perms flag and
VGW_FILE_PERMS env var alongside the existing dir-perms option.
The mode passed to open() is masked by the process umask, so the
O_TMPFILE path now chmods explicitly to match the CreateTemp fallback
path and give new objects the configured mode regardless of umask.
* feat(azure): implement server-side copy with fallback
Add server-side object copy for the Azure backend using StartCopyFromURL, with a
fallback to download+reupload when server-side copy is unavailable. The copy
logic lives in backend/azure/copy.go and handles metadata, tagging and object
lock configurations.
The copy-source SAS service version is configurable via the --copy-sas-version
flag (AZ_COPY_SAS_VERSION) and defaults to the SDK version, so production is
unchanged. Endpoints that lag the SDK's SAS version (e.g. Azurite) cannot verify
a SAS signed with the newer version and can set an older one. On a metadata-COPY,
the internal website-redirect key is dropped from the destination to match the
download+reupload fallback.
Testing:
- Add CopyObject_cross_bucket_server_side_copy, which copies an object with data,
user metadata, content-type and tags across two buckets and verifies all are
preserved and an ETag is returned.
- Configure the Azurite functional-test gateway with AZ_COPY_SAS_VERSION and let
Azurite trust its self-signed test certificate (NODE_EXTRA_CA_CERTS) so it can
fetch the copy source from its own HTTPS endpoint, ensuring CI exercises the
real server-side copy path instead of always falling back.
Signed-off-by: Nils Leger <nils.leger@getflip.com>
* docs: update copyright year
Signed-off-by: Nils Leger <nils.leger@getflip.com>
* fix: always fallback to download+upload whenever there is an error building the server-side copy URL
Signed-off-by: Nils Leger <nils.leger@getflip.com>
---------
Signed-off-by: Nils Leger <nils.leger@getflip.com>
Fixes#1327Fixes#1567Closes#2264
Wires the S3 gateway up to the standalone IAM service so identity policies, not just bucket policies and ACLs, are enforced on the S3 data plane. The gateway authenticates SigV4 requests by calling new private derive-signing-key and resolve-identity endpoints on the IAM service instead of holding secrets itself, and evaluates identity policy through the same PolicyEvaluator path added to auth.VerifyAccess, combined with the bucket policy using explicit-deny-wins precedence. The private endpoints are served over their own mTLS listener (new iamapi/private package, genmtlscerts.sh to generate test material, and client-cert support in internal/netutil), separate from the public IAM API. As part of this the vendored aws/signer/v4 package is deleted and replaced by a pure-Go SigV4 implementation in internal/sigv4auth, which now reads canonical request data directly off the fiber.Ctx instead of reconstructing an http.Request, and is shared by both the S3 request-signing verification and the new private-endpoint signing.
DeleteObjects moves from an all-or-nothing authorization check to true partial success: VerifyObjectsAccess evaluates every object in a batch independently against both the identity policy and any object lock, so a denial or a locked object only removes that key from the batch instead of failing the whole request. It also batches the identity-policy round trip and the bucket-policy fetch once per request rather than once per object, and separates plain deletes from versioned ones since a versioned delete needs s3:DeleteObjectVersion rather than s3:DeleteObject. Object lock handling got a few correctness fixes alongside this: a bypass is now modeled as BypassNone/BypassRequested/BypassOverwrite rather than a single bool, because root's blanket ability to override a GOVERNANCE retention should only apply when the client actually asked to bypass it (DeleteObject/DeleteObjects/PutObjectRetention), not when the gateway is silently replacing a locked object via an overwrite, which needs the permission from everyone including root. Retention changes are now correctly classified as an extension (allowed under plain s3:PutObjectRetention) versus a weakening (date or mode change, which needs the bypass permission), and a COMPLIANCE lock can never be weakened by anyone regardless of permissions, matching AWS. Separately, VerifyObjectCopyAccess had a readonly-mode gap: it returned early for root/admin before ever calling VerifyAccess, so the readonly check inside VerifyAccess never ran for them on CopyObject; access checks are now ordered so the readonly gate always applies before any root/admin bypass, for copy as well as every other write path.
Bucket policies also gained Condition block support, via a new shared internal/condition package moved out of the IAM policy package since both bucket and identity policies share the same evaluation semantics. It implements the full AWS operator set — String{Equals,NotEquals,EqualsIgnoreCase,NotEqualsIgnoreCase,Like,NotLike}, Numeric{Equals,NotEquals,LessThan,LessThanEquals,GreaterThan,GreaterThanEquals}, Date{Equals,NotEquals,LessThan,LessThanEquals,GreaterThan,GreaterThanEquals}, Bool, BinaryEquals, Arn{Equals,Like,NotEquals,NotLike}, IpAddress/NotIpAddress, and Null — along with the ForAllValues/ForAnyValue set qualifiers and the IfExists modifier. A new requestConditionContext builds the per-request keys a bucket policy's Condition block can reference — aws:SourceIp, aws:SecureTransport, aws:CurrentTime, aws:EpochTime, aws:UserAgent, aws:Referer, s3:prefix, s3:delimiter, s3:max-keys, s3:x-amz-acl, s3:VersionId — following AWS's own per-action rules for which keys a given S3 operation actually populates. Identity-derived keys such as aws:PrincipalArn and aws:username are deliberately left unwired here, since the gateway has no way to know them; the standalone IAM service fills those in itself when it evaluates an identity policy.
Also added new integration test suites for S3-side IAM: s3_iam_access_control.go and s3_iam_session_access_control.go cover identity-policy enforcement and session-credential requests against real S3 operations, alongside expanded OIDC/web-identity coverage and a new runoidctests.sh runner wired into the OIDC GitHub Actions workflow.
Add support for `CreateOpenIDConnectProvider`, `GetOpenIDConnectProvider`, `ListOpenIDConnectProviders`, `DeleteOpenIDConnectProvider`, `AddClientIDToOpenIDConnectProvider`, `RemoveClientIDFromOpenIDConnectProvider`, and `UpdateOpenIDConnectProviderThumbprint` on both the internal and Vault storage backends, rounding out the standalone IAM service with the same OIDC identity provider management AWS IAM exposes.
CreateOpenIDConnectProvider validates the issuer URL, enforces the client ID and per-provider client ID list limits, and accepts an optional ThumbprintList. When the caller omits ThumbprintList, the provider auto-fetches the thumbprint by opening an outbound TLS connection to the issuer URL and hashing its top-level CA certificate, matching real AWS behavior. This auto-fetch is configurable: it can be turned off with the `--disable-oidc-thumbprint-autofetch` CLI flag (or the `VGW_IAM_DISABLE_OIDC_THUMBPRINT_AUTOFETCH` environment variable) for restricted or air-gapped deployments where the IAM server shouldn't make outbound connections, in which case an omitted ThumbprintList is rejected instead. AddClientIDToOpenIDConnectProvider and RemoveClientIDFromOpenIDConnectProvider manage a provider's client ID list, and UpdateOpenIDConnectProviderThumbprint replaces its thumbprint list, all with the same length and format validation applied at creation time.
Provider ARNs are derived from the issuer URL, and GetOpenIDConnectProvider and DeleteOpenIDConnectProvider resolve providers by ARN, returning NoSuchEntity when a provider doesn't exist. ListOpenIDConnectProviders returns the full set of stored providers. These actions are wired into the IAM API router and given their own XML response types under iamapi/types, with a new iamapi/internal/iamutil package handling URL validation, thumbprint fetching and normalization, and ARN construction shared across the controller methods.
Closes#1640
Add a standalone AWS IAM Query API implementation for managing IAM users through standard AWS SDKs and the AWS CLI.
Server usage
Start the IAM server with internal file-backed storage:
mkdir -p /tmp/versitygw-iam
./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --dir /tmp/versitygw-iam
Start the IAM server with Vault KV v2 storage using AppRole:
VGW_IAM_VAULT_ROLE_SECRET=<role-secret> ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --vault-endpoint-url http://127.0.0.1:8200 --vault-auth-method approle --vault-role-id <role-id> --vault-mount-path kv --vault-secret-storage-path iam
Vault authentication also supports root tokens, separate authentication and secret-storage namespaces, custom mount paths, server certificate validation, and mutual TLS client certificates.
Configure the AWS CLI credentials used by the IAM server:
export AWS_ACCESS_KEY_ID=user
export AWS_SECRET_ACCESS_KEY=pass
export AWS_DEFAULT_REGION=us-east-1
Implemented IAM actions
CreateUser creates an IAM user with an AWS-compatible ARN, generated AIDA user ID, creation timestamp, optional path, and tags. It validates usernames, paths, tag limits, reserved tag prefixes, duplicate tag keys, and existing users.
aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob
aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob --path /engineering/ --tags Key=team,Value=storage
GetUser returns a stored user or the root identity when requested without a username through the IAM Query API.
aws --endpoint-url http://127.0.0.1:7070 iam get-user --user-name bob
ListUsers returns users in deterministic username order and supports path filtering, marker-based pagination, and MaxItems limits.
aws --endpoint-url http://127.0.0.1:7070 iam list-users
aws --endpoint-url http://127.0.0.1:7070 iam list-users --path-prefix /engineering/ --max-items 100
UpdateUser updates the username and/or path, recalculates the user ARN, and rejects conflicts with existing users.
aws --endpoint-url http://127.0.0.1:7070 iam update-user --user-name bob --new-user-name robert --new-path /platform/
DeleteUser permanently removes an IAM user and returns AWS-compatible errors for missing users.
aws --endpoint-url http://127.0.0.1:7070 iam delete-user --user-name robert
IAM protocol and authentication
- Support the AWS IAM Query protocol version 2010-05-08 over GET and POST form requests.
- Return AWS-compatible XML responses, error documents, status codes, request IDs, user metadata, and pagination fields.
- Authenticate root credentials with AWS Signature Version 4 for the IAM service in us-east-1.
- Support both Authorization-header and query-string SigV4 authentication.
- Validate credential scope, signed headers, timestamps, clock skew, content length, signatures, and unsupported signature or session-token modes.
- Add IAM-specific validation and error mapping for malformed requests, invalid actions, duplicate entities, missing users, throttling, and internal failures.
Storage implementations
- Add an internal JSON-backed store using iam.json and iam.json.backup with atomic temporary-file replacement, concurrent access protection, stable ordering, pagination, and persistence across restarts.
- Add a Vault KV v2 store with one secret per user, CAS-based duplicate protection, permanent deletion, AppRole reauthentication, namespace support, configurable authentication and KV mounts, root-token authentication, and TLS/mTLS configuration.
- Introduce a common Storer interface and require exactly one storage backend to be configured.
Server and embedding support
- Register the new `versitygw iam` command with environment-variable and CLI configuration for both storage backends.
- Add `embedgw.RunIAMAPI` and `IAMConfig` for embedding the IAM service in Go applications.
Gateway-level internal packages
- Add `internal/iamstore` as a reusable generic file-backed IAM persistence engine and migrate the existing gateway internal IAM service to it.
- Add `internal/sigv4auth` for shared SigV4 header and presigned-query parsing, canonical request generation, signature verification, and structured authentication errors.
- Refactor the S3 authentication paths to use the shared SigV4 implementation while preserving S3-specific error responses.
- Add `internal/httpctx` for shared Fiber context keys and AWS-style request ID handling.
- Add `internal/routekit` for shared query, form, and header route matchers.
- Add `internal/netutil` for reusable certificate storage, hostname-aware listeners, multi-address serving, TLS listeners, and UNIX socket handling.
- Update the custom SigV4 signer to honor an explicitly supplied signed-header list so unrelated headers do not alter IAM signatures.
Testing and CI
- Add AWS IAM SDK-based integration coverage for all supported user actions, header authentication, query authentication, validation, errors, filtering, and pagination.
- Split standalone IAM tests into `versitygw test iam` and retain existing gateway IAM tests under `versitygw test gw-iam`.
- Add unit coverage for controllers, authentication, routing, storage, embedding, listeners, request matching, persistence, and signing behavior.
- Add `runiamtests.sh` to exercise internal storage over HTTP and HTTPS plus Vault storage through AppRole.
- Add a dedicated IAM functional-test workflow with a Vault service and merged runtime coverage reporting.
- Include the IAM test runner in shellcheck and add the AWS IAM SDK dependency.
Ben McClelland
Tung Lam
Keith KennedyandClaude Opus 5.5
niksis02
Kyd CaoandBen McClelland
Nils LegerandGitHub