The body stream is now wrapped in a `bodyStreamTracker` before the handler touches it, which remembers the stream's first terminal result rather than asking fasthttp a second, unsafe question. `io.EOF` means the body was read out in full and the connection is still in sync; no terminal result means the handler stopped partway, so the leftovers are drained the way a
`Content-Length` body already was; a framing error means nothing decodable is left and the connection cannot carry another request.
`fasthttp.Request.SetBodyStream` cannot install the wrapper, as it releases the current `*requestStream` back to its pool, so `requestBodyStream` is now the accessor every body reader takes the stream from.
Broken framing no longer gives up on draining either. The connection is closed either way, so a bounded read off the raw socket costs nothing and lets the client finish its write and read the S3 error instead of a reset.
A streamed chunked request can leave bytes queued after the handler returns.
Reusing that connection lets fasthttp parse those bytes as the next request,
allowing a shared upstream proxy connection to mix requests across tenants.
Mark chunked requests Connection: close when the middleware cannot safely
drain them, preventing leftover bytes from crossing the request boundary.
Draining is intentionally skipped because fasthttp's request stream reads past
the terminating chunk when probing for EOF and can block waiting for another
chunk header. The connection-reuse sacrifice is therefore required to avoid both
request desynchronization and delaying the response. Content-Length bodies
retain the existing bounded drain behavior.
A new glaicer mode for the archiving backend needs the storage
class supplied to the backend for put object, create multipart
upload, and browser post object input.
The `posix` and `azure` backends sorted `ListMultipartUploads` by `Key` and `Initiated` only, leaving uploads with identical values dependent on arbitrary directory/blob listing order. Add `UploadID` as a deterministic tertiary sort key in both backends.
Also apply the one-second delay in `ListMultipartUploads_keyMarker_not_from_list` to all backends so same-key uploads get distinct timestamps in CI, and fix an unrelated error message that referenced the wrong slice.
`WithOnListen` tells an embedder when the S3 server is serving, but not
where: the callback takes no arguments, and the server reports the
addresses it bound nowhere else. An embedder that wants an ephemeral port
therefore cannot ask for port 0; it has to pick a free port itself,
release it, and pass it in, which loses to any other process that binds
the same port in between.
Add `WithOnListenAddrs`, which passes the callback the address of every
listener the server bound, in port-specification order, so an embedder can
serve on `127.0.0.1:0` and learn the port the kernel chose. `WithOnListen`
is unchanged. `MultiListener` gains `Addrs`, the every-listener counterpart
of `Addr`, to supply them.
Closes#2364
`AssumeRoleWithWebIdentity` only ever trusted an `OIDC` provider reachable over verified `https`, at a publicly routable address, on the implicit `:443`. That posture is right for an internet-facing IdP but rejects every address an internal one can have, so a `SPIFFE/SPIRE` OIDC discovery provider in the same cluster — or as a sidecar in the same pod — could never be registered, let alone verified against, and no setting could express "this private address is the IdP".
Two opt-in flags on `versitygw iam`, both off by default:
`--oidc-allow-private-endpoints`
Permit a provider `Url` resolving to a loopback/private/link-local address, and an explicit port. Transport is unchanged: still `https`, still fully verified (a self-signed in-cluster cert is trusted the way AWS documents, through `ThumbprintList`).
`--oidc-allow-insecure-transport`
Additionally permit plaintext `http` provider URLs, discovery/JWKS endpoints and redirects, and drop TLS verification (`thumbprint` pinning included) for `https` ones.
Both apply uniformly to the thumbprint auto-fetch at `CreateOpenIDConnectProvider` time and to the discovery-document plus `JWKS` fetch at `AssumeRoleWithWebIdentity` time. Neither weakens anything past the endpoint: signature verification, issuer matching, audience and trust policy evaluation are untouched, and the DNS-resolve-once/dial-the-resolved-IP shape stays in place so a rebind still cannot redirect a connection.
An `http` provider keeps its scheme in its stored `Url`, `ARN` and `iss` matching, rather than being stripped like an `https` one — otherwise `"http://host"` and `"https://host"` would collapse onto a single ARN and storage key and each could satisfy the other's trust policy. It also stores an empty `ThumbprintList` rather than failing: a plaintext provider presents no certificate to thumbprint.
Helm: `iamServer.oidc.{allowPrivateEndpoints,allowInsecureTransport}`, alongside `disableThumbprintAutofetch` moved into the same block (the flat `iamServer.disableOidcThumbprintAutofetch` stays honored). Chart `0.4.1 -> 0.4.2`.
The WebUI's create-provider form no longer rejects `http` URLs and ports client-side; it cannot see the service's settings, so those two rules are left to the server, whose error surfaces as a toast like any other.
Add a SessionsSnapshot view over the new C ABI entry point and
serve it from the admin server as GET /rc-sessions. The admin server
gains a WithAdminRoute option so an embedding binary can register
extra admin routes that run with the same signature verification and
admin checks as the built-in endpoints; vgwrdma registers the
snapshot there when the RC feature is enabled. The route replies
with the usual XML error surface so unsigned or non-admin requests
get a 403 rather than a generic 500. Stub builds return a not
supported error, keeping the build matrix unchanged.
Expose rc_server_sessions_snapshot, which copies every live session
into fixed rc_session_snapshot records under the map lock and invokes
the callback once per record outside the lock. Each session records a
monotonic creation timestamp, because the prepare/ready deadlines move
as the session progresses and cannot serve as an age reference. The
state byte combines the session state machine value with a
reap-pending marker, so callers can distinguish sessions that are
about to be reaped from healthy ones. Records whose op or target does
not fit the fixed fields are skipped rather than truncated.
The session id only existed as the sessions map key; the session
record itself kept an empty id string, so the terminal reap record
logged an empty id for every expired, cancelled, or destroyed
session. Copy the id into the record at creation time so teardown
logs identify the session they describe.
The Prepare and ReadyTransfer wrappers embed string views built from
Go heap strings inside request structs passed to C by pointer. The
cgo pointer check rejects such requests when the string data is an
unpinned Go heap pointer, so any live PREPARE or READY call with
header-derived strings panicked at the call boundary and the route
returned a 500. Constant strings passed the check because their data
lives in read-only static storage, which is why standalone callers
kept working while the gateway did not.
Pin the string bytes with runtime.Pinner for the duration of the cgo
call and drop the now redundant KeepAlive calls in those two
wrappers. The other string-taking wrappers pass rc_str_in by value
and are unaffected.
Also add a deviceless cgo boundary regression test that calls the
real Prepare wrapper with heap-backed interior-pointer strings and an
invalid opcode, so C returns from argument validation before the
server handle is touched.
ibv_poll_cq, ibv_post_send, and ibv_post_recv stopped being
exported library symbols in modern rdma-core: verbs.h ships them
as static inline wrappers that dispatch through
cq->context->ops. dlsym therefore returned null for them and the
loader rejected perfectly usable libraries, failing RC server
init with a bare RC_E_INTERNAL on hosts with rdma-core 61+.
Open the first device briefly, read the three function pointers
from its context ops table, and close it again. The check now
only requires symbols that actually exist in the library, and
the failure mode for an ops-less provider is explicit.
The verbs loader and device enumeration failures returned
RC_E_INTERNAL without any stderr trace, which made a VM or
container without RDMA indistinguishable from a genuine
library problem. Print the failing step so operators can tell
the two apart at startup.
Wire C-side diagnostics (session reap, READY data phase outcome,
init failures) through a sink callback so the gateway can surface
them next to its own logs instead of losing them in stderr noise.
The sink is a plain C function pointer installed once after init
and valid until destroy: the Go side registers a fixed cgo
trampoline (closures cannot cross the boundary), copies the
message immediately per the lifetime contract, and never runs
under the session map lock. Error-level lines keep the existing
stderr output; --debug enables the level-2 diagnostic stream.
The hipobj-rc-v2 data plane started with its session, queue pair,
staging and timeout limits hardcoded at the rcserver.Init call
site, so operators could not size the RC plane for their hardware
the way they can for the cuObject backend. Add one flag per limit
plus the READY admission slot count, all defaulting to the values
the gateway passes today, and validate them through a new
rdmamode.V2ValidationError consulted only when the RC data plane
is enabled, mirroring the stale-value handling of the v1 settings.
Counts are parsed as uint64 and range-checked against the uint32
narrowing at the DeviceOpts boundary, and the timeouts carry an
upper bound that keeps the nowMs + timeout deadline arithmetic in
the C core from wrapping.
Move the shutdown-once wrapper installation from after the gateway
option validation to the top of runGateway, so every early error
return closes the backend chain exactly once instead of leaking
it.
Add --rdma-rc-enable (VGW_RDMA_RC_ENABLE, default false) so the
RC control routes and data plane start without implying the
cuObject v1 backend. The global CLI hook resolves the mode
first: gateway commands require either --rdma-ip or
--rdma-rc-enable, and neither path implies the other, so a
v2-only deployment boots without a v1 address.
The v1 port, retry, pool, and DCI validations also ran for every
mode, so stale v1 environment values blocked v2-only startup
with unrelated errors. Those validations moved behind the v1
check as a cgo-free helper in internal/rdmamode, exercised
alongside the mode matrix, and the CQ-depth limit keeps its
32-bit boundary check there.
The RC data plane builds its IAM service, starts the session
server, and mounts the three control routes behind SigV4.
Startup and shutdown own the backend chain through idempotent
guards: the gateway wraps the input backend in a once guard and
defers a rollback closure that follows the chain as it grows;
the completed v1 chain gets its own once owner, and the RC
service is closed first through a backend wrapper installed
right after a successful session-server init. Startup failures
close exactly what was built, the RunVersityGW lifecycle
consumes the same guards instead of closing again, and the RC
sessions drain before the backend chain shuts down.
Classify the platform-stub answer before the internal-error
logging decision, so the expected 501 no longer logs as an
internal 500 while debugging production servers.
Serialize the full S3 error XML body instead of the base error
alone: per-type diagnostics such as the access key and the
string-to-sign survive the route boundary. A regression test
wraps a signature failure with both diagnostic fields and
asserts the response keeps the status, the code, and both
fields.
Assert the response body identifiers equal the request-ID
headers, pinning the two views of the same response.
The RC auth adapter returned signature-verification errors to
Fiber, so the production S3 error handler collapsed them into a
generic 500 response; authentication failures lost their real
status and code the same way route failures did before the
terminal serializer. The adapter now sends verification errors
through the shared serializer as well.
Cover the route error boundary with the real S3 server: the
shared serializer keeps status and body for wrapped S3 errors,
raw fiber errors stay 500, and the platform stub answers 501.
The stub-answer classifier moves next to the shared marker type
in the same commit so every build answers 501 at the point the
test first runs, and the general CI workflow builds the
session-server archive before go test, which the Linux link of
this package now requires.
Classify the rejected-argument, short-transfer, and oversized-
value failures of the session server as bad requests at the
route boundary, answering the closest S3-style protocol error
instead of a generic internal failure.
The RC control routes returned fiber.Error values for 400, 404,
409, 429, 502, and 503 outcomes, but the production S3 error
handler converts ordinary fiber errors into a generic 500
response, so clients observed InternalError for every protocol
outcome. The routes now send the final status and S3-style XML
body themselves through a shared terminal serializer.
S3-aware errors from authentication, authorization, and the
object backend keep their status and code. Session-server
failures map to protocol error codes: InvalidRdmaRequest,
NoSuchRdmaSession, RdmaSessionConflict, RdmaResourceLimit,
RdmaTransferFailed, and RdmaServiceUnavailable. Owner mismatch
answers the same 404 as an unknown session so a session id is
never disclosed across principals. The platform stub keeps its
501 answer and uses the same response shape.
Since 9bde1ddb (tagging support for CreateBucket) the api layer always
populates CreateBucketConfiguration, so the aws-sdk serializes an empty
<CreateBucketConfiguration/> element on every CreateBucket the s3proxy
backend issues. Strict backends reject that request — Ceph RGW answers
400 InvalidArgument — which breaks bucket creation through the gateway
entirely for those backends. MinIO and SeaweedFS tolerate the empty
element, which is why this went unnoticed.
Drop the configuration before calling the backend when it carries no
content; a bodyless CreateBucket is accepted by all tested backends for
this no-location case. Configurations that carry tags, a location
constraint, or location/bucket info are still forwarded unchanged.
Adds wire-level unit tests via an injected capturing HTTP client:
CreateBucket without tags must send no body (fails before this fix),
and configurations carrying tags, a location constraint, location info
or bucket info must be forwarded. Verified end-to-end against Ceph RGW
(Quincy 17.2.8 and Squid 19.2.0): CreateBucket through the patched
gateway succeeds (200) where it previously failed with 400
InvalidArgument, and the created bucket is usable and deletable.
Bucket policy `Principal` named callers by access key id. Under the standalone IAM service it now names them by AWS-style ARN, as real S3 does: a user ARN, a role ARN (covering every session of that role), an assumed-role ARN (covering one session), the account root ARN or bare account id, or `*`. Every other IAM backend has no ARNs to name anything by and keeps access-key principals unchanged, gated on a new `auth.PrincipalResolver` capability interface that only the standalone client implements.
`auth.Account` carries `Arn` and `RoleArn`, filled at authentication time, so a session can be matched against both its own ARN and its role's. Principals are validated at PutBucketPolicy time through a new `/private/resolve-principals` endpoint, which rejects anything that does not name a live identity with `MalformedPolicy: Invalid principal in policy`.
An `Allow` naming the account root ARN or bare account id delegates to the account's own IAM rather than granting on its own, while a `Deny` naming it denies every principal in the account outright. Denial messages now name the caller by ARN wherever one exists.
Also fixes `aws:PrincipalArn` for assumed-role sessions, which reported the session ARN where AWS reports the role's, and stops an unreachable IAM service being reported as a malformed policy.
AWS SigV4 signatures are attacker-controlled inputs compared against
server-computed HMAC values. Ordinary string comparison exits at the first
differing byte, which can expose the length of the matching prefix through
response timing and, in principle, enable signature forgery for a fixed request
after many probes.
Use the shared sigv4auth.SecureCompare helper for browser POST-policy signatures
and streaming chunk and trailer signatures. The helper preserves the existing
accept/reject behavior, including rejecting malformed or different-length
signatures, while using crypto/subtle.ConstantTimeCompare for equal-length
values.
The access-log and admin-access-log options now accept stdout, stderr, or - for
stdout in addition to file paths. Standard stream destinations are kept open
during shutdown and SIGHUP handling, while file destinations continue to support
normal reopen behavior for log rotation. CLI help, embedded config comments, and
the example configuration describe the new destination values.
Fixes#2245
The multipart upload temp directory was created with a hard-coded
0755 mode instead of the configured --dir-perms value, unlike every
other directory creation path in this backend. This caused
inconsistent permissions below bucket directories.
Fixes#2266
* rdma: add the hipobj-rc-v2 control routes to the vgwrdma gateway
Mount the three control routes (prepare, ready, cancel) on the
S3 port behind the standard SigV4 middleware. The routes own
authentication-adjacent policy the C server cannot see: the
middleware wrapper yields to the handler on success, READY and
CANCEL re-read the account through the IAM cache bypass so
mid-flow deletions and credential rotations take effect
immediately, and every object access re-authorizes against the
decoded bucket and key.
The READY handler implements the session ownership contract:
the completion-reference finalizer installs only after the
transfer claim succeeds, the PUT path hands the reference to
the put view exactly at the borrow point, and the FINAL reply
carries the stored object's metadata. Backend I/O runs under a
context merged with the RC service context so shutdown unblocks
in-flight handlers, with a bounded pool for the fresh IAM
lookups.
vgwrdma starts the session server alongside the gateway when an
RDMA interface is configured, tears it down on exit, and shuts
the IAM service down on any startup failure. embedgw learns the
readonly flag for the object access checks the routes share.
Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
* rdma: add the missing stub handlers for non-Linux builds
The non-Linux rcroutes stub exposed only Register while the vgwrdma
gateway registers the prepare/ready/cancel handlers directly, so
cross-compiling cmd/vgwrdma failed with undefined methods. Add the
three stub handlers answering 501 Not Implemented and let Register
reuse them, matching the Linux Handler API surface.
* auth: drop the duplicated GetUserAccountFresh definition
The rebase onto main (which already carries GetUserAccountFresh from
the iam-cache-fresh change) kept both copies of the method, breaking
the build with a redeclaration error. Remove the second copy so the
method is defined once.
* rdma: address the review findings on the control route wiring
Drop the unused Handler.Register from both build variants: the
gateway mounts the three control routes through s3api.WithRoute so
the SigV4 verifier wrapper (rcAuth) runs in front of each handler,
and nothing else calls Register.
Clear iamOwned only when RunVersityGW returns nil. It shuts the IAM
service down itself at the end of its shutdown sequence, but its
early failure paths return before reaching that point, so the
deferred shutdown must keep covering those errors.
Remove the unused rcserver.SessionInfo parameter from sizeOf; the
transferred byte count comes from the READY response alone.
* rdma: keep transient IAM failures retryable in the fresh revalidation
The fresh account revalidation turned every GetUserAccountFresh
error into 403, which reports transient backend failures (LDAP
timeouts, network errors) as a revoked account and leaves the
client no room to retry. Only a confirmed missing account
(auth.ErrNoSuchUser) means that; answer anything else with 503 so
clients can retry the request.
* rdma: make the IAM shutdown exactly-once and keep gateway errors visible
The gateway and RunVersityGW share the IAM service, and which side
shut it down could not be told from the return value: runtime
failures return after RunVersityGW already shut the service down,
while early setup failures return before any shutdown happens. The
iamOwned flag therefore either shut the service down twice or leaked
it depending on the error, and the error itself was dropped.
Wrap the service so Shutdown runs exactly once no matter which side
calls it, keep the deferred shutdown for every early failure path,
and return the gateway error again. The wrapper re-exposes the
optional interfaces (fresh account reads, signing keys, policy
evaluation, fixed bucket ownership) so feature detection through the
IAM service keeps working.
* rdma: reuse the SigV4 account for RC control requests
READY and CANCEL are independently authenticated SigV4 requests.
Use the account resolved by the normal SigV4 path instead of
bypassing the IAM cache a second time. This aligns RC revocation
latency with other signed S3 requests and removes the extra
backend IAM lookup, its concurrency cap, and the RC-specific IAM
error mapping. The session owner check and the READY target and
operation authorization are unchanged.
* rdma: reword the READY reauthorization comment
The comment implied a revocation inside the session window always
takes effect at READY, but the account used here is the one SigV4
resolved, which may be a cached entry. State what the check does
without claiming account-cache freshness.
* rdma: preserve IAM cache behavior and standalone region
---------
Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
Co-authored-by: Ben McClelland <ben.mcclelland@versity.com>
niksis02
Ben McClelland
RaduBerinde
dependabot[bot]andGitHub
Jihyeon Gim
Ali Erdem CerrahandGitHub
Stefan Markmann
Luke McCrone