Role last-used tracking was missing entirely - `GetRole` returned a `RoleLastUsed` element that nothing ever wrote, rendering the zero time instead of the empty element AWS returns for an unused role - and access key last-used only ever saw the `IAM`/`STS` control plane, so a credential used exclusively against the S3 gateway reported as never used. Roles now record a use whenever a request authenticates with one of their session credentials, through a new `Storer.RecordRoleUsage` mirroring `RecordAccessKeyUsage`, gated on the session's role still being the one it was minted against so a session outliving its role can't attribute its use to a same-named replacement. `LastUsedDate` became a `*time.Time` so an unused role renders as an empty element.
Both records now cover the S3 data plane as well: the gateway sends its configured region and `s3` on evaluate-policy and the IAM service records the caller there, so `GetAccessKeyLastUsed's` `ServiceName` is now iam, sts or s3. That call was chosen over derive-signing-key, which runs before signature verification and takes its region and service from the caller's own `Authorization` header - recording there would let anyone who knows an access key id refresh and poison another identity's audit record. Requests denied by a bucket policy or made against a public bucket are not recorded, since neither reaches identity-policy evaluation. To keep per-request recording affordable, an update is skipped while the stored record has the same service and region and is under a minute old; a change of either is written through immediately.
Assuming a role is not a use, a request denied by an identity policy is, and both successful and denied S3 requests update the record. Also moves the `OIDC-dependent` tests into the `s3-iam-session` group so runoidctests.sh runs a single group.
Bind the RC session server archive through cgo: Init opens the
verbs device with the resource limits, and the RCSvc wrapper
carries the admission gate handlers use around every session
call (TryEnter/Leave), a service-lifetime context that Close
cancels so in-flight handler I/O unblocks during shutdown, and
an idempotent Close that marks every session for reaping, waits
for admitted calls to drain, and destroys the server.
The rest of the surface maps the C ABI one-to-one: prepare,
ready (with the transfer outcome returned atomically in the
reply), staging borrow/finish, the put-view handoff, session
introspection, and cancel. Non-linux or non-cgo builds compile
against a stub so the package is portable.
Nothing imports the package yet; the gateway integration that
links and exercises it follows.
Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
Wrap the v2 session core with the server-side C ABI the gateway
binds to: prepare/ready/cancel session calls, staging leases for
GET side-loading, the put-view handoff for PUT commits, session
introspection, and the server lifecycle.
The ABI owns the parts that must be shared across sessions: the
verbs device handle, global and per-principal resource limits,
session accounting with consume-once handles keyed by epoch and
nonce, the completion reference that pins a session from the
READY claim until its finalizer, and the reaper that tears
transport objects down once every reference drains. A background
thread expires sessions past their prepare or execute deadlines
so abandoned sessions cannot pin the limits, and teardown
failures keep the affected verbs objects and device alive rather
than freeing memory the NIC may still reference.
A peer-busy READY rolls the claim back and re-arms the QP
through RESET so the client can retry the same session. The
vgwrdma target now links the archive built from these sources.
Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
With the posix backend running --chuid/--chgid against the standalone IAM service, CreateBucket failed for every bucket name and left a half-created directory behind. Bucket ownership is fixed to the gateway's root account there, and that account was constructed from the root credentials alone, so its UserID/GroupID stayed at zero and the gateway tried to chown each new bucket to uid/gid 0 - something a process that is not root can never do. Root-account object writes failed the same way, because the identity the S3 request path uses for root also comes from the root credentials and never from the IAM backend. On top of that, the failed chown returned before the acl xattr was written, so the leftover directory made every later request for that name fail with "get bucket acl: no such key" until it was removed by hand.
The standalone IAM client now reports the root account with UserID, GroupID and ProjectID taken from --iam-standalone-default-uid, -gid and -project-id, returning a copy so the stored root account keeps the credentials it is compared against. ResolveDerivedKey copies that same identity onto root when the IAM backend fixes bucket ownership to the root access key, which keeps root's own writes consistent with the buckets root owns. CreateBucket now removes the bucket directory, its sidecar attributes and its versioning directory on any failure after the mkdir, so a failed create leaves nothing behind and the name stays retryable. A chown EPERM reports the target uid/gid, the flags that asked for it and the process euid/egid instead of a bare "operation not permitted", and the posix backend warns at startup when chuid/chgid are set on an unprivileged gateway.
The built-in IAM backends do not fix bucket ownership, so root and every other account reach the storage backend exactly as before.
S3 explicitly documents this action as owner-only ("To retrieve the versioning state of a bucket, you must be the bucket owner." — https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketVersioning.html), and the handler enforced that with an extra auth.IsAdminOrOwner check on top of VerifyAccess. Real S3 behaves differently: verified against AWS that a bucket policy explicitly denying s3:GetBucketVersioning denies the bucket owner itself, and that an Allow grants the action to a principal that doesn't own the bucket. It goes through ordinary bucket policy/ACL evaluation like any other bucket subresource read, which is what the write side, PutBucketVersioning, already did here. Removes the extra check along with auth.IsAdminOrOwner, which had no other call site.
Port the hipObject v2 reliable-connection session core into
cuwrapper/rc: the session table and state machine, the wire
codec for the hipobj-rc-v2 headers, request parsing, the
injectable clock and randomness sources, the transport layer
(QP/CQ lifecycle, RTR/RTS transitions, staging registration),
the data phase (RDMA write with immediate for GET, receive with
immediate for PUT), the RDMA token codec, and the dynamically
loaded ibverbs shim (ibv-core.h plus the dlopen host binding).
The sources are a port of the upstream hipObject v2 core, kept
close to the original so the two trees can be diffed during
review. Nothing links against them yet; a Makefile rule builds
the objects into rdma/librcserver.a for the ABI layer that
follows.
Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
The cached GetUserAccount keeps serving a stale entry for the
cache TTL after the backing IAM service changes, which delays
credential revocation by up to the configured expiry.
Add GetUserAccountFresh to the IAM cache, which reads directly
from the underlying service and refreshes the cached entry with
the result. Callers that need revocations to take effect
immediately can use it instead of the cached path.
Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
Browser-based POST uploads (POST /{bucket}) read Content-Type straight out
of the form fields, so a form without a content-type field stored the object
with an empty Content-Type. On read, fasthttp substitutes its own default,
so the object came back as "text/plain; charset=utf-8" rather than just
missing a type.
Fall back to defaultContentType ("binary/octet-stream") when the form field
is absent or empty, matching PutObject, CopyObject and CreateMultipartUpload,
as well as AWS S3 and Ceph RGW.
Add Config.IAMService (auth.IAMService) so embedders that already have
their own IAM implementation can inject it directly instead of relying
on one of the gateway's built-in backends (local dir, LDAP, Vault,
S3-backed, FreeIPA, or the standalone IAM service). When set, it takes
priority over all other IAM backend configuration and RunVersityGW
skips calling auth.New entirely.
This avoids forcing embedders to duplicate their own account store as
one of the built-in IAM backends just to satisfy the gateway's IAM
interface.
Replace the hardcoded 0644 defaultFilePerm with a NewFilePerm option on
the posix and scoutfs backends, exposed as the --file-perms flag and
VGW_FILE_PERMS env var alongside the existing dir-perms option.
The mode passed to open() is masked by the process umask, so the
O_TMPFILE path now chmods explicitly to match the CreateTemp fallback
path and give new objects the configured mode regardless of umask.
* feat(azure): carry Azure blob marker in continuation token for pagination
Azure blob markers are opaque values only Azure may mint, so an S3 object
key can never be passed back to Azure as a listing marker. Paging the
underlying Azure listing with an S3 key therefore failed or re-scanned the
whole prefix on every page.
Introduce azMarkerToken, an opaque S3 continuation token that carries the
Azure marker alongside the last returned key: the Azure marker resumes the
blob listing where it stopped, and the last key filters out already-returned
entries. Tokens are versioned with a "vgw1." prefix; anything without it is
treated as a plain key, so tokens from older versions and hand-crafted
markers keep working. The shared listBlobs helper now backs both ListObjects
and ListObjectsV2, applying the S3 marker and delimiter client side.
ListObjectsV2 pages efficiently via the token; ListObjects (v1) has no token
to carry state and walks the prefix from the start each page.
Add unit tests (token round-trip, marker-resumed pagination, delimiter and
common-prefix handling, multipart filtering) driven by a fake Azure
container, and integration tests covering full and delimited pagination.
Signed-off-by: Nils Leger <nils.leger@getflip.com>
* fix: token is now bound to delimiter too
Signed-off-by: Nils Leger <nils.leger@getflip.com>
---------
Signed-off-by: Nils Leger <nils.leger@getflip.com>
* feat(azure): implement server-side copy with fallback
Add server-side object copy for the Azure backend using StartCopyFromURL, with a
fallback to download+reupload when server-side copy is unavailable. The copy
logic lives in backend/azure/copy.go and handles metadata, tagging and object
lock configurations.
The copy-source SAS service version is configurable via the --copy-sas-version
flag (AZ_COPY_SAS_VERSION) and defaults to the SDK version, so production is
unchanged. Endpoints that lag the SDK's SAS version (e.g. Azurite) cannot verify
a SAS signed with the newer version and can set an older one. On a metadata-COPY,
the internal website-redirect key is dropped from the destination to match the
download+reupload fallback.
Testing:
- Add CopyObject_cross_bucket_server_side_copy, which copies an object with data,
user metadata, content-type and tags across two buckets and verifies all are
preserved and an ETag is returned.
- Configure the Azurite functional-test gateway with AZ_COPY_SAS_VERSION and let
Azurite trust its self-signed test certificate (NODE_EXTRA_CA_CERTS) so it can
fetch the copy source from its own HTTPS endpoint, ensuring CI exercises the
real server-side copy path instead of always falling back.
Signed-off-by: Nils Leger <nils.leger@getflip.com>
* docs: update copyright year
Signed-off-by: Nils Leger <nils.leger@getflip.com>
* fix: always fallback to download+upload whenever there is an error building the server-side copy URL
Signed-off-by: Nils Leger <nils.leger@getflip.com>
---------
Signed-off-by: Nils Leger <nils.leger@getflip.com>
Adds `TagOpenIDConnectProvider`, `UntagOpenIDConnectProvider` and `ListOpenIDConnectProviderTags` to the standalone IAM service, backed by both the internal and Vault storers. They follow the user and role tagging actions in most respects — the tag action merges into the provider's existing tags and rejects a repeated key, untag removal is idempotent, and the tag listing is sorted by key and paginated, with the per-request member count and the per-provider tag total enforced as separate quotas so replacing a tag on a provider already at the 50-tag cap still succeeds — but differ in the one respect IAM itself draws: OIDC provider tag keys are compared exactly, not case-insensitively. On a provider `env` and `ENV` are two independent tags, both may be supplied in a single request, only a byte-identical repeat is a duplicate (reported without the "Tag keys are case insensitive" note the user and role actions carry), and untagging `env` leaves `ENV` in place.
That distinction is now carried by `iamutil.TagKeyCase`, which `ParseTags` uses for duplicate detection and which `mergeTags`, `removeTags` and the tag listing's marker lookup use for key matching. `CreateOpenIDConnectProvider` moves onto the exact comparison too, so a provider created with case-differing tag keys keeps both.
All three actions are authorized against the target provider's ARN, so `aws:ResourceTag/<key>` reads the provider's own tags, and the tag and untag actions populate `aws:RequestTag/<key>` and `aws:TagKeys` respectively, so a tag-scoped policy Condition governs which tags a caller may set or remove. All three report a missing provider with the wording `DeleteOpenIDConnectProvider` uses rather than the one `GetOpenIDConnectProvider` uses, which is why the Vault provider read now takes the not-found error its calling action reports.
The WebGUI gains a Tags section in the OIDC provider manage view, replacing the read-only tag row, and the shared tag editor gains a case-sensitive mode that changes its duplicate-key check, its diffing of an edited set into an untag and tag pair, and the wording of its guidance.
Adds `TagRole`, `UntagRole` and `ListRoleTags` to the standalone IAM service, backed by both the internal and Vault storers, with the same semantics the user tagging actions already have: tag keys are matched case-insensitively but stored case-preserving, TagRole merges into the role's existing tags and rejects duplicate keys, UntagRole removal is idempotent, and ListRoleTags is sorted by key and paginated. The per-request member count and the per-role tag total are enforced as separate quotas, so replacing a tag on a role already at the 50-tag cap still succeeds. Role tags and the tags of a user sharing the same name are independent sets.
All three actions are authorized against the target role's ARN, so `aws:ResourceTag/<key>` reads the role's own tags, and TagRole and UntagRole populate `aws:RequestTag/<key>` and `aws:TagKeys` respectively, so a tag-scoped policy Condition governs which tags a caller may set or remove.
The tag storage helpers are now shared between users and roles: `MaxTagsPerUser` becomes `MaxTagsPerResource`, `ListUserTagsOutput` becomes `ListTagsOutput`, and `paginateTags` takes the marker and page size directly instead of a user-specific input struct.
The WebGUI gains a Tags section in the IAM role manage view, reusing the tag editor the user view already uses, which applies a whole edited tag set as a single UntagRole and TagRole pair.
Also corrects the `roleName` length bound across every role action: it was validated against the 128-character user-lookup limit, where IAM caps role names at 64.
Adds `TagUser`, `UntagUser` and `ListUserTags` to the standalone IAM service, backed by both the internal and Vault storers. Tag keys are matched case-insensitively but stored case-preserving, TagUser merges into the user's existing tags and rejects duplicate keys, UntagUser removal is idempotent, and ListUserTags is sorted by key and paginated. The per-request member count and the per-user tag total are enforced as separate quotas.
All three actions are authorized against the target user's ARN, and TagUser and UntagUser populate aws:RequestTag/<key> and aws:TagKeys respectively, so a tag-scoped policy Condition governs which tags a caller may set or remove.
The WebGUI gains a Tags section in the IAM user manage view, with an editor that applies a whole edited tag set as a single UntagUser and TagUser pair.
Also corrects two error shapes that never matched AWS: a half-supplied tag member now reports a ValidationError naming the member field instead of MissingParameter, and the maxItems bound check reports separate lower- and upper-bound errors across every IAM list action.
Gates bucket listing behind an identity policy, lets browsers reach the standalone IAM API, and turns the WebUI into a dashboard for S3, IAM, or both.
**Bucket listing.** `ListBuckets` is now gated by the new `s3:ListAllMyBuckets` action, evaluated against `arn:aws:s3:::*`. The request names no bucket, so only identity policies apply — there is no resource policy to combine with, which is the same shape `CreateBucket` already had, so both now share one identity-only evaluation path. Root and admin bypass it, and backends with no identity-policy layer keep listing as before since their listing is already narrowed to the caller's own buckets. The action is IAM-only and is deliberately absent from the bucket-policy action list.
**Fixed bucket ownership.** The standalone IAM client has no per-user ownership to express — accounts are all plain users, cannot be enumerated, and access is decided by policy rather than ACL — so it now implements `auth.FixedBucketOwner` and every bucket is owned by root. Bucket creation stops resolving an owner, `ListBuckets` returns every bucket to every caller (what they may then do with one stays a per-request policy decision), and the admin `ChangeBucketOwner` reports method-not-supported. Other IAM backends are untouched.
**IAM service CORS.** `--cors-allow-origin` now applies to the `iam` command: it answers preflights and stamps the CORS headers, mirroring back the requested method and headers rather than enumerating the SigV4 header set. Without it no browser can reach the IAM API at all, so setting `--webui` without it falls back to `*` with a warning. The chart gets `iamServer.corsAllowOrigin`.
**WebUI.** New IAM pages for users, roles and OIDC providers, signing IAM/STS query-form requests directly from the browser. Navigation is capability-gated rather than role-gated: on sign-in the session probes the S3, admin and IAM endpoints independently and each page shows only what those credentials actually reach, so one build serves an IAM-only dashboard, an S3-only dashboard, and a combined one. The login page takes an optional IAM endpoint, seeded from the new `--webui-iam-gateways` (chart: `webui.iamGateways`) — never auto-detected, since the IAM service is a separate process. The WebUI can also be hosted by `versitygw iam` itself, for deployments with no S3 gateway behind it.
**The admin API is ignored once an IAM endpoint is in play.** The IAM service is then the user directory and bucket ownership is fixed, which leaves the admin API no job: the session is given no admin endpoint at all, its login field is hidden, `users.html` redirects to its IAM counterpart, and every admin-only surface stays off screen. Dashboard and Buckets remain available to any S3 session in such a deployment, running on the S3 and IAM APIs alone and surfacing each denial per action instead of redirecting.
Also fixes two WebUI bugs: embedded assets went out with a zero modification time and no `Cache-Control`, so browsers treated them as fresh for centuries and an upgraded gateway served new HTML against stale JS — they now revalidate against an ETag; and the login page's advanced-options section clipped its last field, since it animated to a height named in the stylesheet rather than the one it measures now.
**Usage**
IAM-only dashboard, served by the IAM service:
versitygw iam --port :7076 --webui :8080 --cors-allow-origin http://localhost:8080/
IAM + S3, dashboard served by the IAM service — point it at the gateway with `--webui-gateways`, and let the gateway accept the dashboard's origin:
versitygw iam --port :7076 --webui :8080 --webui-gateways http://localhost:7070/ --cors-allow-origin http://localhost:8080/
versitygw --port :7070 --cors-allow-origin http://localhost:8080/ posix /data
IAM + S3, dashboard served by the S3 gateway — point it at the IAM service with `--webui-iam-gateways`, and let the IAM service accept the dashboard's origin:
versitygw --port :7070 --webui :8080 --webui-iam-gateways http://localhost:7076/ posix /data
versitygw iam --port :7076 --cors-allow-origin http://localhost:8080/
niksis02
Ben McClellandandGitHub
dependabot[bot]andGitHub
Luke McCrone
Jihyeon GimandBen McClelland
Muhammad AdnanandGitHub
mu-adnan
Kyd CaoandBen McClelland
Nils LegerandGitHub