a7569a7 added credential-less pulls of public images. Three things about it
were wrong, all of them in how the appview handled the decision that belongs
to the hold.
**Scope handling was all-or-nothing.** IsPullOnlyScope required every
requested action to already be "pull", but clients routinely ask for more
than the operation needs — pull,push is common for a plain read, and some
ask for pull,push,delete up front. Those were rejected and challenged,
leaving a credential-less client no way to pull even a public image, which
is the entire feature. NarrowToPullOnly drops the write actions and issues a
token carrying "pull" and nothing else. Granting a subset is what the
distribution token spec expects. The allowlist property is preserved: "pull"
is the only action that survives, and "*" is deliberately not expanded into
it, since a wildcard request is not evidence the caller wants a read.
**The appview-side read gate was inert.** checkReadAccess passed p.ctx.DID,
the DID of the repository *owner*, not the requester. Any non-empty DID
satisfies a private hold's check, and the owner's is never empty, so it asked
"may the owner read their own hold", answered yes, and admitted everyone.
Worse, CheckReadAccessWithCaptain admitted any authenticated DID to a private
hold at all, on an explicitly-MVP assumption that holding a DID was close
enough to being a sailor. Every doc says otherwise (docs/hold.md:109 "Crew
with blob:read", CLAUDE.md:140, docs/BYOS.md:280) and so does the hold
(ValidateBlobReadAccess: owner, or crew carrying blob:read/blob:write). It
now takes isCrew and requires owner-or-crew, and callers only pay for the
crew lookup when it can change the answer — a public hold or an anonymous
caller is decided by the captain record alone. Nothing here loosens access;
it brings the local gate into agreement with the authority.
**Denials could not reach the client.** distribution's blobHandler.GetBlob
maps everything except ErrBlobUnknown to ErrorCodeUnknown, so a 401 raised
in the blob store left as a 500 — misreporting an auth failure as a server
fault, and giving BearerChallenge no 401 to attach WWW-Authenticate to, so
Docker was told "server error" instead of being prompted for credentials.
Clients that retry 5xx looped: 4.1s per case in the matrix, now 0.01s. The
check moves to Repository(), where an errcode.Error is passed through
verbatim by the registry app — the same mechanism a7569a7 used for
NAME_UNKNOWN. It fails open on a lookup error, since the hold is the
authority and a transient failure should not break public pulls.
Removes auth.allow_anonymous_pull. It could only ever withhold — captain.Public
is what grants — so it was a second flag for a decision the hold already owns,
and gating it appview-side was never the intent. Layer bytes 307 straight to
S3, so the appview is not even in the path whose cost might have justified an
operator-side lever.
Tests: TestAuthMatrix only ever ran against a public hold, and its pull cases
never fetched a layer — crane.Pull is lazy and img.Digest() needs only the
manifest, which ATCR serves from the user's PDS where it is world-readable, so
no pull row in the matrix touched blob authorization at all. Pulls now
materialize layer bytes, and testharness.WithPrivateHold plus
TestAuthMatrixPrivateHold cover public:false + allow_all_crew:true — the
production shape, where anyone with an account pulls and pushes and anonymous
gets nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Credential-less pulls of public images. /auth/token issues a pull-only
token with an empty subject when no Basic auth is present; the
destination hold still enforces captain.Public, and push or delete always
challenges.
- token.IsPullOnlyScope and AuthMethodAnonymous;
Handler.issueAnonymousToken skips the authorizer gate and the
service-auth pre-mint, since there is no identity to reconcile and no
AppView-to-hold service token to bind. The token is still stamped
with the resolved registry domain, so anonymous pull works on
secondary front doors whose access controller demands their own
audience.
- auth.allow_anonymous_pull (default true) turns it fully off, restoring
the previous always-challenge behavior. Mirrored into the deploy
template, since the default means existing deploys pick this up.
- RegistryContext.Anonymous is plumbed from the middleware.
- ProxyBlobStore sends no Authorization header when the service token is
empty, and returns 401 rather than 403 for anonymous denials so Docker
prompts for credentials, including when a stale captain cache lets the
request through and the hold says private.
- BearerChallenge wraps the /v2/ subtree so a 401 raised deep in the
stack via errcode.ServeJSON still carries WWW-Authenticate.
Distribution's own scoped challenges are left alone.
IsPullOnlyScope allowlists the pull action instead of denylisting push and
delete. Distribution's actionSet.contains treats "*" as *every* action, so
a scope of `repository:victim/img:*` names neither denied string and would
have handed an unauthenticated caller a token valid for push and delete on
someone else's repository — clearing the authgate entirely, since anonymous
tokens deliberately skip it. Writes would still have failed further down
(no PDS credential), but the gate itself was bypassable. Now every
requested action must be exactly "pull". Covered by new claims tests.
Unresolvable identities return NAME_UNKNOWN instead of a bare error that
distribution renders as 500. This path was previously unreachable without
credentials; anonymous pull opens it to the internet, and a 5xx on
arbitrary input both misreports a bad request as a server fault and sends
clients that retry 5xx into a retry loop. That loop was real: in the auth
matrix, regclient spent 83s on a single case before this fix, and the
suite now runs in 5s.
Stat preserves an authorization verdict from getPresignedURL rather than
flattening it to ErrBlobUnknown. Distribution calls Stat before ServeBlob
on GET and HEAD, so without this an anonymous pull from a private hold
answered 404 and BearerChallenge had no 401 to annotate — the 401 path
above could never actually reach a client.
The auth matrix is updated to match: anonymous pull of the seeded public
repo now succeeds, anonymous push is denied against a real identity's
namespace (rather than an unresolvable one, which was testing name
resolution rather than authorization), and a new case pins the
NAME_UNKNOWN behavior for an unknown identity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A read-only app password authenticates fine via createSession but cannot
call com.atproto.server.getServiceAuth, which is privileged — the PDS
answers 403 InsufficientScope. That fell through to the generic non-200
path and became a 503, which invites the client to retry a request that
can never succeed, with no indication of what is actually wrong.
Classify it with a sentinel error and map it to a 403 at /auth/token,
carrying text that names the fix: use a full-access app password.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
createSession, the app-password getServiceAuth call, and the hold's
/.well-known/atproto-did resolution all used http.DefaultClient, which
has no timeout. All three run on the /auth/token path, so a slow or
unreachable PDS or hold could hold the request open indefinitely, well
past Docker's own token-fetch timeout.
Give each a bounded client: 15s for createSession, 10s for the
app-password getServiceAuth, 10s for the hold DID fetch. All three are
safe to cut off — the two GETs are idempotent, and a timed-out
createSession only orphans an unused server-side session.
The OAuth refresh path deliberately keeps no overall timeout: its POSTs
run through refreshDetachTransport and must not be cancelled mid-rotation,
which would strand a rotated refresh token.
Note holdDIDResolveClient is package-level in pkg/atproto, so the 10s cap
applies to every ResolveHoldDID caller, including the GC, Jetstream
backfill and hold-health background workers, not only the token path.
That is intended (none of them want an unbounded fetch either), but it is
a wider blast radius than the token path alone.
This bounds three hops, not the whole request: the OAuth getServiceAuth
GET and identity resolution are still unbounded, so /auth/token is not
yet fully time-boxed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The quota gate ran on any scope containing "push" and denied the entire
token request, so "quota exceeded ... Delete images to free space" named
a remedy the gate itself blocked: docker and crane both request
pull,push,delete for a manifest delete, and manifest DELETE is
bearer-only, so there was no path left to free space.
When the request also asks for delete, drop push from the repository
entries and issue the reduced token instead of denying. A plain
pull,push is still denied so the quota message reaches the client that
needs to see it; granting a pushless token there would turn a clear
error into an opaque 401 on the first blob upload.
The narrowing happens in place on the access slice the handler hands to
the issuer, so document that on token.Authorizer along with the ordering
the gate goroutine depends on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An expired app-password token could wedge an account permanently. The 401
branch clears the cached token, but some PDSes report the same condition as
400 with an atproto error name in the body, which fell through to the generic
non-200 branch. That clears only the derived service token, so the dead
bearer token stayed in the cache and every subsequent request replayed it.
Observed on one account against at.hexlab.foo: 16,110 of these errors and
4,254 retryable 503s over 33 hours, with no recovery path. The cache is
in-memory, so it only cleared on process restart.
Now the non-200 branch classifies the atproto error name and evicts on the
ones that mean the presented token is unusable, matching what the 401 branch
already does. For app-passwords that is the equivalent of a refresh: the next
authentication re-mints via createSession.
Deliberately not routed through oauth.IsSessionInvalidError, which excludes
ExpiredToken on purpose — there it would delete a recoverable OAuth session
and sign the user out everywhere, whereas here the only thing discarded is a
cache entry that will be repopulated.
Not addressed here: the failure still surfaces as a 503, which is retryable
and so keeps clients looping. Returning 401 with the re-auth hint would be
the better signal, but it spans the token handler and is a separate change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The route was registered GET-only, so containerd and Docker, which try the
OAuth2 POST endpoint first whenever they hold a secret, ate a 405 and retried
on the GET form. Every authenticated pull paid two auth round trips, and in the
production logs the POST share of token traffic grew from 0.5% to 38% over six
weeks as more clients pulled from k8s with basic-auth imagePullSecrets.
Serve both specs on the same path. After credentials and scope are extracted
the two paths are identical, so this is an extraction branch plus a form-shaped
error writer.
Only grant_type=password is supported and no refresh token is issued: the
registry JWT's lifetime is pinned to the AppView<->hold service-auth, so a
refresh token would be a fourth long-lived credential with its own storage and
revocation. Clients handle its absence by continuing to use the credential they
already hold.
The refresh grant is refused with 401 rather than the 400 that RFC 6749 5.2
prescribes. containerd sends that grant only when it has no username, which is
the same condition that disables its 405 fallback, so a 400 would hard-fail
those clients. 401 is on its retry list and routes them to the GET form, where
a device secret authenticates off the password alone. That shape previously had
no working path at all.
resolveService now takes the requested service as an argument, since it arrives
in the query string on GET and in the form body on POST.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An AppView can front several registry domains that all reach the same
backend (seamark.dev serving buoy.cr, seamark.cr, and soon atcr.io).
Distribution's token access controller holds `service` as a single string
and uses it twice: as the value advertised in the WWW-Authenticate
challenge, and as the sole accepted JWT audience. So it announced one
domain's name on every domain, and honoured one domain's tokens
everywhere. A push to seamark.cr was challenged with service="buoy.cr".
Both uses sit inside Authorized, which already has the request, but the
value is fixed at construction and reachable through no hook — autoredirect
only templates the realm. So register an "atcr-token" controller that
builds one upstream controller per domain and dispatches on r.Host. Each
front door now advertises its own name and demands its own audience. All
signature, certificate and claim verification stays in upstream code; this
only routes.
The token handler stops discarding ?service= and stamps the audience with
the front door the client used, allowlist-checked against the configured
domains so the value stays server-determined despite arriving from the
client. It has to come from the query param because the realm lives on the
UI host, where r.Host names no registry domain.
This is token hygiene and spec conformance, not a privilege boundary: every
domain fronts the same backend, so a client can obtain a token for any of
them just by handshaking there. What it buys is a truthful challenge and
the decoupling needed to later split a domain onto its own AppView.
Also unify the domain list. DomainRoutingMiddleware keyed its map on the
raw config while matching a port-stripped host, so a domain configured with
a port could never match its own requests. It now shares the normalized
cfg.Auth.Services, so routing and authorization agree on one set of names.
cfg.Auth.ServiceName was an exact alias for Services[0] and is replaced by
PrimaryService(), which also removes an empty-slice index.
Rollout: the audience for seamark.cr and bouy.cr changes, so a token minted
just before the restart draws one 401 and Docker re-handshakes into a valid
one. buoy.cr is unchanged (it stays primary), and atcr.io keeps the service
name it already has today. The challenge and the accepted audience come
from the same delegate, so the retry converges by construction. Deploy as a
single flip, not a canary: an old instance ignores ?service= and would keep
minting the primary audience while a new one rejects it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to 37bab32. That commit stopped deleting OAuth sessions on transient
errors, which fixed spurious sign-outs but overshot on one path: a genuinely dead
session stopped being evicted at all, turning a forced re-login into a permanent
failure loop.
GetOrFetchServiceToken flattened every non-200 from getServiceAuth into
fmt.Errorf("service auth failed with status %d: %s"). IsSessionInvalidError then
had nothing structured to inspect, and its string fallback could not help: it
looks for the OAuth 2.0 code invalid_token, while atproto emits the XRPC name
InvalidToken. The difference is the underscore, not the case, so lowercasing
never bridged it. A revoked session came back 401 InvalidToken and was classified
transient, so /auth/token returned 503 forever and the user was never prompted to
re-authenticate.
The non-200 branch now wraps an *atclient.APIError carrying the status and the
parsed atproto error name, which is what the existing structured checks in
IsSessionInvalidError already know how to read. Transient shapes stay transient:
atprotoErrorName returns "" for a non-JSON body, so 500s with HTML, 502s, and
429s do not evict.
ExpiredToken is deliberately not treated as a dead session. It means "refresh
me", and deleting on it would sign the user out of every UI session over an
ordinary access-token expiry a refresh would have fixed. isAuthError omits it for
the same reason; the two classifiers have to agree about the same condition.
The comment on the string fallback claimed it was a looser spelling of the
structured check. It is not — it handles a different error family. indigo's
RefreshTokens returns OAuth token-endpoint failures as a bare fmt.Errorf carrying
the auth server's snake_case code verbatim ("token refresh failed (HTTP 400):
invalid_grant"), never a typed error, so a string match is the only thing that
can classify a refresh failure, which is the invalid_grant replay case 37bab32
exists to detect. Both comments now say which family they cover.
Two hardening items on the same theme:
use_dpop_nonce no longer counts as an auth error in the appview's isOAuthError.
It is a routine handshake step indigo retries with the server-supplied nonce, and
treating it as fatal signed users out over ordinary nonce rotation. It can still
escape when a server sends that error with no DPoP-Nonce header, leaving indigo
nothing to retry with; a stuck session there is preferable to signing everyone
out in the common case, and the comment says so rather than claiming it cannot
happen.
Detached session deletes are bounded by SessionDeleteTimeout. They run on
context.WithoutCancel so a canceled request cannot leave the cleanup half-done,
which also stripped the only deadline they had — a wedged database write blocked
the goroutine with no way to shed it. Matches the bound already on the detached
persist callback. The unparseable-token-endpoint warning is now deduped per
endpoint rather than once per process, since that path fails open by returning
the client unwrapped, silently reinstating the refresh burn.
The refreshDetachTimeout comment now notes the cap is per-POST: the DPoP-nonce
retry means one refresh can issue two, holding the per-DID lock for up to twice
the stated value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When a Docker client canceled a slow /auth/token request mid-refresh, the
token-refresh POST was aborted client-side but completed on the PDS, which
rotated the refresh token. The rotated token was never received or persisted,
so the next refresh replayed the consumed token, got invalid_grant, and the
session (OAuth + UI) was deleted, signing the user out everywhere.
- Detach refresh POSTs from the inbound request context via a per-session
RoundTripper (WithoutCancel + 30s cap); once a refresh starts it completes
- Persist session updates (rotated tokens, DPoP nonces) on a detached context
- Gate session deletion on IsSessionInvalidError: cancellation, timeouts, and
transport errors no longer delete sessions; genuine invalid_grant still does
- Add phase timing to /auth/token and per-DID lock wait warnings to attribute
the ~14s pre-refresh stalls that push requests past Docker's deadline
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Evan JarrettandClaude Opus 5