A teardown callback that arrived while its record was reserved
was dropped: if the READY that held the reservation then rolled
its claim back, the session had lost both publication paths - no
request owner and no callback owner - and the record stayed
forever, unpublished. The record now stashes the terminal event;
releasing the reservation after a rollback consumes the stash and
publishes the expiry, since the native session is gone and no
second callback will arrive.
A duplicate READY that lost the reservation race still proceeded
to claim the native transfer, so the transfer could complete with
no publication owner. The READY handler now refuses to claim when
the reservation is not granted, answering as a duplicate claim.
Publication also validates ownership against the record's emitter,
so a stale reservation cannot publish over or consume the current
owner's record.
The shutdown drain raced its producers: a callback could enqueue
a publication after the drain checked the queue but before the
worker exited, stranding the record behind a stopped worker. The
drain now runs after the RC service close, which quiesces the
native reaper before returning, so no producer can enqueue behind
the drain; ordering replaces locking.
Error paths that published directly bypassed the single-shot
guard, letting the deferred panic safety net attempt a second
publication of the same record. All outcome publications in the
READY completion flow now go through the guarded path.
Overflow publications ran one goroutine per job: a stalled sink
with a full queue grew them without bound (a probe reached a
thousand blocked calls). Overflow now runs inline under a bounded
semaphore, so at most a fixed number of callers wait and every
record still publishes.
The publication worker outlived the operational sinks: gateway
shutdown closed the RC service and then the sinks while
publications were still queued, losing terminal records (reached
"file already closed" with the real file logger). The shutdown
wrapper now drains the publication queue before closing the RC
service, and the route handler exposes the drain; late terminals
after the drain publish inline instead of queueing behind a
stopped worker.
The publication reservation happened after the transfer claim
returned: in the window between the claim and the reservation, a
concurrent READY's re-authorization denial consumed the unreserved
record, so the claimant's successful transfer lost its publication
to the denial. The READY handler now reserves before the claim and
before re-authorization; a rolled-back claim (wire failure, peer
busy) releases the reservation instead of publishing, so the
session keeps its record for the next claimant or the reaper.
The tracker test now joins the worker through the shutdown drain
and asserts per-session outcomes and byte counts instead of an
aggregate count that a pending fifth record could satisfy.
The teardown callback ran audit, metrics, and event sinks inline on
whatever thread fired it, which is the native reaper thread: one
blocking sink (a synchronous file write on a stalled filesystem)
would stall reaping for every other session. Publications now hand
off to a dedicated worker through a bounded queue; a full queue
falls back to a detached goroutine, so the callback never waits on
a sink and no record is dropped.
A denial issued while a completion owner holds the reservation
(concurrent READY re-authorization failure) no longer steals the
publication: reserved records are invisible to the denial path,
which previously produced a denial record plus the owner's success
record for one transfer.
The tracker tests now drive a recording audit sink and assert the
published record count across expiry, reserved, denied-while-
reserved, and consume-or-noop paths: one record per session.
A native completion call fires the teardown callback before it
returns, so an outcome recorded after the call is too late: every
successful transfer published as an expiry, and PUT failures and
panics lost their real outcome to the callback's placeholder.
The READY handler now reserves the session record before invoking
any completion call. A reserved record is invisible to the
callback, and the handler publishes the real outcome exactly once
after the result is known. A deferred safety net publishes on
panic unwinds before the unwind finalizer retires the session.
Successful PUTs forward the backend-assigned ETag and version into
the object-created event, and the audit record carries the
transferred byte count as the object size.
A READY whose re-authorization fails now publishes the denial
itself before canceling, instead of letting the cancel publish an
expiry.
The native side keeps the terminal reason on every failure exit:
a verify failure no longer overwrites the wire-failure outcome,
and QP transition or re-arm failures record the wire failure they
return. The unclaimed-teardown classification follows the wire
mapping, so every transfer-level failure the client would see as
502 publishes as the same 502 instead of a diverging per-cause
code.
The metrics bucket tag is documented as absent on synthesized
publications: route parameters come from route matching, which a
synthesized request never runs; the audit log derives the bucket
from the path and stays accurate.
Round-3 review of the ownership model found that moving publication
between the request path and the callback by hand leaves edges where
a record is published twice or lost. The publication model is now
structural instead: request paths only record outcomes, and the
native teardown callback - which the ABI fires exactly once per
destroyed session, after every completion call - is the single
publisher.
The READY handler records its outcome (success with the reported
byte count, or the mapped failure) and lets the callback publish.
A deferred recorder covers panic unwinds, so every path through
the handler leaves a final outcome behind. A failed PREPARE
finalization publishes through a consume-or-noop helper: when the
finalizing call already reaped the session the entry is gone and
the helper is a no-op, otherwise it publishes the failure itself.
Unclaimed teardowns no longer all read as expiries: the record is
classified from the native outcome, so a transfer that died on the
wire, failed verification, or timed out carries its own code and
status.
The synthesized publication context runs on an immutable app, so
string accessors copy instead of exposing the pooled request buffer
to the asynchronously serializing event senders. Captured account
and region strings are cloned for the same reason. The nil-error
publication no longer asserts on the S3 error interface, and the
panic marker is a proper internal S3 error.
Unit tests cover the table semantics: expiry publication,
first-record-wins, consume-or-noop failure publication, unregister,
unknown sessions, error normalization, status mapping, and the
outcome classification.
Review of the publication path found that ownership could change
hands at the wrong moment and that records could disagree with both
the wire response and the underlying operation.
Successful transfers lost their completion record: the native
completion calls fire the teardown callback synchronously, so the
callback claimed the publication first and logged every completed
transfer as an expiry, and committed PUTs produced no object-created
events. The READY handler now reserves the publication before
invoking any completion call; a reserved record is invisible to the
callback, and the handler publishes the real outcome exactly once.
The same race existed at creation: the PREPARE finalizer can reap an
expired session and fire the callback before the session is
registered, leaving an orphan entry whose only notification already
happened. Registration now runs before the finalizing call, a
notification that arrives first is parked and consumed by the
registration, and a failed finalization drops the entry.
Retained records referenced the request's pooled header buffer, so
a later request could rewrite a tracked session's bucket and key;
captured strings are cloned now. The synthesized publication path
follows the same rule for the event senders, which serialize
asynchronously.
Operational sinks classify plain errors as 500 on their own, so a
resource-limit rejection logged 500 while the client saw 429. The
publication renders non-S3 errors through the route error mapping
before the record reaches the sinks, and the expiry record carries
a dedicated SessionExpired code instead of a generic one. Malformed
PUT headers now preserve the operation in the record.
Review of the operational publication found four gaps where the
records disagreed with the S3 surface or were missing entirely.
Authorization-failure records lost the requester because the
pre-session publisher did not carry the authenticated account; the
account now flows into the record, so the audit trail names who was
denied.
Signature failures and malformed PREPARE headers ended the request
before any publication point. The auth adapter now publishes
authentication failures through the route handler, and header
validation failures publish with whatever object identity the
headers still carried, matching the S3 surface where every denied
request still logs.
RC requests carried no region: the custom routes run before the
middleware that stores the region local, so event records lacked
awsRegion and audit host headers read s3..amazonaws.com. The auth
adapter sets the region for every verified RC request.
Operational records reported generic 500 statuses for protocol
errors that the wire answers with a specific status (a resource
limit rejection logged 500 while the client saw 429). Status
mapping now reuses the route error mapping, so the recorded status
always equals the wire status.
Two-phase RC transfers were invisible to the access log, metrics,
and bucket notifications: every outcome record on the S3 surface is
driven by the fiber request context, and the RC wire requests carry
the transfer session rather than the object.
Publish one record per session by tracking the operational context
from PREPARE through completion. A tracker table registers each
successfully prepared session with its account, region, object, and
start time; whichever path confirms the final outcome first (READY
completion, READY failure, or the reaper teardown callback) claims
the entry and publishes exactly once. Sessions that end before
PREPARE succeeds publish a request record directly.
The record is emitted through a synthetic fiber context carrying
the object path and the captured locals, so the existing logger,
metrics manager, and event sender produce the same schema as the
S3 surface without any interface change. Bucket notifications fire
for completed PUTs.
The gateway creates the operational services inside RunVersityGW,
after the RC routes exist. Add an OnServicesReady callback to the
embedded gateway config and wire it in vgwrdma to hand the services
to the RC routes and install the teardown callback.
RC sessions that never reach a completion (expired, abandoned, or canceled before READY) currently vanish inside the reaper without any trace on the operational surface. Add a terminal notification callback to the C ABI so the server reports the final outcome of every session exactly once, fired from the reaper with no server lock held.
Go 1.26 marks PKCS#1 v1.5 RSA encryption and direct ECDSA public-key coordinate access
as deprecated. The IPA code path uses rsa.EncryptPKCS1v15 to wrap an AES session key for
vault retrieval, and the OIDC JWKS parser rebuilt EC public keys by setting ecdsa.PublicKey.X
and Y directly.
This change replaces the deprecated RSA wrap with rsa.EncryptOAEP using SHA-256, which is the
standard safe replacement for PKCS#1 v1.5 encryption and preserves the same protocol semantics
for the IPA vault exchange. For EC JWKs, it reconstructs the raw uncompressed public point and
parses it through ecdsa.ParseUncompressedPublicKey, which is the supported Go API for EC public
keys and avoids mutating deprecated fields while preserving the exact mathematical key value.
These changes do not alter the wire protocol or trust decisions; they only migrate to the
supported stdlib APIs for equivalent behavior.
A new glaicer mode for the archiving backend needs the storage
class supplied to the backend for put object, create multipart
upload, and browser post object input.
The `posix` and `azure` backends sorted `ListMultipartUploads` by `Key` and `Initiated` only, leaving uploads with identical values dependent on arbitrary directory/blob listing order. Add `UploadID` as a deterministic tertiary sort key in both backends.
Also apply the one-second delay in `ListMultipartUploads_keyMarker_not_from_list` to all backends so same-key uploads get distinct timestamps in CI, and fix an unrelated error message that referenced the wrong slice.
The integration harness went through `config.LoadDefaultConfig` to build
its `aws.Config`, even though it supplies the region, credentials, endpoint,
and HTTP client itself. That made it read the host's shared AWS
configuration, and with `AWS_PROFILE` set in the environment the SDK
insists the named profile exist: on a machine whose shell sets a profile
the SDK cannot find, every test in `cmd/versitygw` died at client setup
with
error: failed to get shared config profile, <name>
Build the `aws.Config` directly from the harness settings instead. Nothing
the shared configuration could supply was used -- credentials and region
were always overridden -- and disabling only the shared files would not
have helped, since the SDK still requires a profile named by `AWS_PROFILE`
to resolve. The default stderr logger `LoadDefaultConfig` installed is kept
so `--debug` output is unchanged.
`WithOnListen` tells an embedder when the S3 server is serving, but not
where: the callback takes no arguments, and the server reports the
addresses it bound nowhere else. An embedder that wants an ephemeral port
therefore cannot ask for port 0; it has to pick a free port itself,
release it, and pass it in, which loses to any other process that binds
the same port in between.
Add `WithOnListenAddrs`, which passes the callback the address of every
listener the server bound, in port-specification order, so an embedder can
serve on `127.0.0.1:0` and learn the port the kernel chose. `WithOnListen`
is unchanged. `MultiListener` gains `Addrs`, the every-listener counterpart
of `Addr`, to supply them.
Closes#2364
`AssumeRoleWithWebIdentity` only ever trusted an `OIDC` provider reachable over verified `https`, at a publicly routable address, on the implicit `:443`. That posture is right for an internet-facing IdP but rejects every address an internal one can have, so a `SPIFFE/SPIRE` OIDC discovery provider in the same cluster — or as a sidecar in the same pod — could never be registered, let alone verified against, and no setting could express "this private address is the IdP".
Two opt-in flags on `versitygw iam`, both off by default:
`--oidc-allow-private-endpoints`
Permit a provider `Url` resolving to a loopback/private/link-local address, and an explicit port. Transport is unchanged: still `https`, still fully verified (a self-signed in-cluster cert is trusted the way AWS documents, through `ThumbprintList`).
`--oidc-allow-insecure-transport`
Additionally permit plaintext `http` provider URLs, discovery/JWKS endpoints and redirects, and drop TLS verification (`thumbprint` pinning included) for `https` ones.
Both apply uniformly to the thumbprint auto-fetch at `CreateOpenIDConnectProvider` time and to the discovery-document plus `JWKS` fetch at `AssumeRoleWithWebIdentity` time. Neither weakens anything past the endpoint: signature verification, issuer matching, audience and trust policy evaluation are untouched, and the DNS-resolve-once/dial-the-resolved-IP shape stays in place so a rebind still cannot redirect a connection.
An `http` provider keeps its scheme in its stored `Url`, `ARN` and `iss` matching, rather than being stripped like an `https` one — otherwise `"http://host"` and `"https://host"` would collapse onto a single ARN and storage key and each could satisfy the other's trust policy. It also stores an empty `ThumbprintList` rather than failing: a plaintext provider presents no certificate to thumbprint.
Helm: `iamServer.oidc.{allowPrivateEndpoints,allowInsecureTransport}`, alongside `disableThumbprintAutofetch` moved into the same block (the flat `iamServer.disableOidcThumbprintAutofetch` stays honored). Chart `0.4.1 -> 0.4.2`.
The WebUI's create-provider form no longer rejects `http` URLs and ports client-side; it cannot see the service's settings, so those two rules are left to the server, whose error surfaces as a toast like any other.
Add a SessionsSnapshot view over the new C ABI entry point and
serve it from the admin server as GET /rc-sessions. The admin server
gains a WithAdminRoute option so an embedding binary can register
extra admin routes that run with the same signature verification and
admin checks as the built-in endpoints; vgwrdma registers the
snapshot there when the RC feature is enabled. The route replies
with the usual XML error surface so unsigned or non-admin requests
get a 403 rather than a generic 500. Stub builds return a not
supported error, keeping the build matrix unchanged.
Expose rc_server_sessions_snapshot, which copies every live session
into fixed rc_session_snapshot records under the map lock and invokes
the callback once per record outside the lock. Each session records a
monotonic creation timestamp, because the prepare/ready deadlines move
as the session progresses and cannot serve as an age reference. The
state byte combines the session state machine value with a
reap-pending marker, so callers can distinguish sessions that are
about to be reaped from healthy ones. Records whose op or target does
not fit the fixed fields are skipped rather than truncated.
The session id only existed as the sessions map key; the session
record itself kept an empty id string, so the terminal reap record
logged an empty id for every expired, cancelled, or destroyed
session. Copy the id into the record at creation time so teardown
logs identify the session they describe.
The Prepare and ReadyTransfer wrappers embed string views built from
Go heap strings inside request structs passed to C by pointer. The
cgo pointer check rejects such requests when the string data is an
unpinned Go heap pointer, so any live PREPARE or READY call with
header-derived strings panicked at the call boundary and the route
returned a 500. Constant strings passed the check because their data
lives in read-only static storage, which is why standalone callers
kept working while the gateway did not.
Pin the string bytes with runtime.Pinner for the duration of the cgo
call and drop the now redundant KeepAlive calls in those two
wrappers. The other string-taking wrappers pass rc_str_in by value
and are unaffected.
Also add a deviceless cgo boundary regression test that calls the
real Prepare wrapper with heap-backed interior-pointer strings and an
invalid opcode, so C returns from argument validation before the
server handle is touched.
ibv_poll_cq, ibv_post_send, and ibv_post_recv stopped being
exported library symbols in modern rdma-core: verbs.h ships them
as static inline wrappers that dispatch through
cq->context->ops. dlsym therefore returned null for them and the
loader rejected perfectly usable libraries, failing RC server
init with a bare RC_E_INTERNAL on hosts with rdma-core 61+.
Open the first device briefly, read the three function pointers
from its context ops table, and close it again. The check now
only requires symbols that actually exist in the library, and
the failure mode for an ops-less provider is explicit.
The verbs loader and device enumeration failures returned
RC_E_INTERNAL without any stderr trace, which made a VM or
container without RDMA indistinguishable from a genuine
library problem. Print the failing step so operators can tell
the two apart at startup.
Wire C-side diagnostics (session reap, READY data phase outcome,
init failures) through a sink callback so the gateway can surface
them next to its own logs instead of losing them in stderr noise.
The sink is a plain C function pointer installed once after init
and valid until destroy: the Go side registers a fixed cgo
trampoline (closures cannot cross the boundary), copies the
message immediately per the lifetime contract, and never runs
under the session map lock. Error-level lines keep the existing
stderr output; --debug enables the level-2 diagnostic stream.
The hipobj-rc-v2 data plane started with its session, queue pair,
staging and timeout limits hardcoded at the rcserver.Init call
site, so operators could not size the RC plane for their hardware
the way they can for the cuObject backend. Add one flag per limit
plus the READY admission slot count, all defaulting to the values
the gateway passes today, and validate them through a new
rdmamode.V2ValidationError consulted only when the RC data plane
is enabled, mirroring the stale-value handling of the v1 settings.
Counts are parsed as uint64 and range-checked against the uint32
narrowing at the DeviceOpts boundary, and the timeouts carry an
upper bound that keeps the nowMs + timeout deadline arithmetic in
the C core from wrapping.
Move the shutdown-once wrapper installation from after the gateway
option validation to the top of runGateway, so every early error
return closes the backend chain exactly once instead of leaking
it.
Add --rdma-rc-enable (VGW_RDMA_RC_ENABLE, default false) so the
RC control routes and data plane start without implying the
cuObject v1 backend. The global CLI hook resolves the mode
first: gateway commands require either --rdma-ip or
--rdma-rc-enable, and neither path implies the other, so a
v2-only deployment boots without a v1 address.
The v1 port, retry, pool, and DCI validations also ran for every
mode, so stale v1 environment values blocked v2-only startup
with unrelated errors. Those validations moved behind the v1
check as a cgo-free helper in internal/rdmamode, exercised
alongside the mode matrix, and the CQ-depth limit keeps its
32-bit boundary check there.
The RC data plane builds its IAM service, starts the session
server, and mounts the three control routes behind SigV4.
Startup and shutdown own the backend chain through idempotent
guards: the gateway wraps the input backend in a once guard and
defers a rollback closure that follows the chain as it grows;
the completed v1 chain gets its own once owner, and the RC
service is closed first through a backend wrapper installed
right after a successful session-server init. Startup failures
close exactly what was built, the RunVersityGW lifecycle
consumes the same guards instead of closing again, and the RC
sessions drain before the backend chain shuts down.
Classify the platform-stub answer before the internal-error
logging decision, so the expected 501 no longer logs as an
internal 500 while debugging production servers.
Serialize the full S3 error XML body instead of the base error
alone: per-type diagnostics such as the access key and the
string-to-sign survive the route boundary. A regression test
wraps a signature failure with both diagnostic fields and
asserts the response keeps the status, the code, and both
fields.
Assert the response body identifiers equal the request-ID
headers, pinning the two views of the same response.
The RC auth adapter returned signature-verification errors to
Fiber, so the production S3 error handler collapsed them into a
generic 500 response; authentication failures lost their real
status and code the same way route failures did before the
terminal serializer. The adapter now sends verification errors
through the shared serializer as well.
Cover the route error boundary with the real S3 server: the
shared serializer keeps status and body for wrapped S3 errors,
raw fiber errors stay 500, and the platform stub answers 501.
The stub-answer classifier moves next to the shared marker type
in the same commit so every build answers 501 at the point the
test first runs, and the general CI workflow builds the
session-server archive before go test, which the Linux link of
this package now requires.
Classify the rejected-argument, short-transfer, and oversized-
value failures of the session server as bad requests at the
route boundary, answering the closest S3-style protocol error
instead of a generic internal failure.
The RC control routes returned fiber.Error values for 400, 404,
409, 429, 502, and 503 outcomes, but the production S3 error
handler converts ordinary fiber errors into a generic 500
response, so clients observed InternalError for every protocol
outcome. The routes now send the final status and S3-style XML
body themselves through a shared terminal serializer.
S3-aware errors from authentication, authorization, and the
object backend keep their status and code. Session-server
failures map to protocol error codes: InvalidRdmaRequest,
NoSuchRdmaSession, RdmaSessionConflict, RdmaResourceLimit,
RdmaTransferFailed, and RdmaServiceUnavailable. Owner mismatch
answers the same 404 as an unknown session so a session id is
never disclosed across principals. The platform stub keeps its
501 answer and uses the same response shape.
Since 9bde1ddb (tagging support for CreateBucket) the api layer always
populates CreateBucketConfiguration, so the aws-sdk serializes an empty
<CreateBucketConfiguration/> element on every CreateBucket the s3proxy
backend issues. Strict backends reject that request — Ceph RGW answers
400 InvalidArgument — which breaks bucket creation through the gateway
entirely for those backends. MinIO and SeaweedFS tolerate the empty
element, which is why this went unnoticed.
Drop the configuration before calling the backend when it carries no
content; a bodyless CreateBucket is accepted by all tested backends for
this no-location case. Configurations that carry tags, a location
constraint, or location/bucket info are still forwarded unchanged.
Adds wire-level unit tests via an injected capturing HTTP client:
CreateBucket without tags must send no body (fails before this fix),
and configurations carrying tags, a location constraint, location info
or bucket info must be forwarded. Verified end-to-end against Ceph RGW
(Quincy 17.2.8 and Squid 19.2.0): CreateBucket through the patched
gateway succeeds (200) where it previously failed with 400
InvalidArgument, and the created bucket is usable and deletable.
Jihyeon Gim
Ben McClellandandGitHub
dependabot[bot]andBen McClelland
niksis02
RaduBerinde
Ali Erdem CerrahandGitHub
Stefan Markmann