The quota gate ran on any scope containing "push" and denied the entire
token request, so "quota exceeded ... Delete images to free space" named
a remedy the gate itself blocked: docker and crane both request
pull,push,delete for a manifest delete, and manifest DELETE is
bearer-only, so there was no path left to free space.
When the request also asks for delete, drop push from the repository
entries and issue the reduced token instead of denying. A plain
pull,push is still denied so the quota message reaches the client that
needs to see it; granting a pushless token there would turn a clear
error into an opaque 401 on the first blob upload.
The narrowing happens in place on the access slice the handler hands to
the issuer, so document that on token.Authorizer along with the ordering
the gate goroutine depends on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An expired app-password token could wedge an account permanently. The 401
branch clears the cached token, but some PDSes report the same condition as
400 with an atproto error name in the body, which fell through to the generic
non-200 branch. That clears only the derived service token, so the dead
bearer token stayed in the cache and every subsequent request replayed it.
Observed on one account against at.hexlab.foo: 16,110 of these errors and
4,254 retryable 503s over 33 hours, with no recovery path. The cache is
in-memory, so it only cleared on process restart.
Now the non-200 branch classifies the atproto error name and evicts on the
ones that mean the presented token is unusable, matching what the 401 branch
already does. For app-passwords that is the equivalent of a refresh: the next
authentication re-mints via createSession.
Deliberately not routed through oauth.IsSessionInvalidError, which excludes
ExpiredToken on purpose — there it would delete a recoverable OAuth session
and sign the user out everywhere, whereas here the only thing discarded is a
cache entry that will be repopulated.
Not addressed here: the failure still surfaces as a 503, which is retryable
and so keeps clients looping. Returning 401 with the re-auth hint would be
the better signal, but it spans the token handler and is a separate change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The route was registered GET-only, so containerd and Docker, which try the
OAuth2 POST endpoint first whenever they hold a secret, ate a 405 and retried
on the GET form. Every authenticated pull paid two auth round trips, and in the
production logs the POST share of token traffic grew from 0.5% to 38% over six
weeks as more clients pulled from k8s with basic-auth imagePullSecrets.
Serve both specs on the same path. After credentials and scope are extracted
the two paths are identical, so this is an extraction branch plus a form-shaped
error writer.
Only grant_type=password is supported and no refresh token is issued: the
registry JWT's lifetime is pinned to the AppView<->hold service-auth, so a
refresh token would be a fourth long-lived credential with its own storage and
revocation. Clients handle its absence by continuing to use the credential they
already hold.
The refresh grant is refused with 401 rather than the 400 that RFC 6749 5.2
prescribes. containerd sends that grant only when it has no username, which is
the same condition that disables its 405 fallback, so a 400 would hard-fail
those clients. 401 is on its retry list and routes them to the GET form, where
a device secret authenticates off the password alone. That shape previously had
no working path at all.
resolveService now takes the requested service as an argument, since it arrives
in the query string on GET and in the form body on POST.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An AppView can front several registry domains that all reach the same
backend (seamark.dev serving buoy.cr, seamark.cr, and soon atcr.io).
Distribution's token access controller holds `service` as a single string
and uses it twice: as the value advertised in the WWW-Authenticate
challenge, and as the sole accepted JWT audience. So it announced one
domain's name on every domain, and honoured one domain's tokens
everywhere. A push to seamark.cr was challenged with service="buoy.cr".
Both uses sit inside Authorized, which already has the request, but the
value is fixed at construction and reachable through no hook — autoredirect
only templates the realm. So register an "atcr-token" controller that
builds one upstream controller per domain and dispatches on r.Host. Each
front door now advertises its own name and demands its own audience. All
signature, certificate and claim verification stays in upstream code; this
only routes.
The token handler stops discarding ?service= and stamps the audience with
the front door the client used, allowlist-checked against the configured
domains so the value stays server-determined despite arriving from the
client. It has to come from the query param because the realm lives on the
UI host, where r.Host names no registry domain.
This is token hygiene and spec conformance, not a privilege boundary: every
domain fronts the same backend, so a client can obtain a token for any of
them just by handshaking there. What it buys is a truthful challenge and
the decoupling needed to later split a domain onto its own AppView.
Also unify the domain list. DomainRoutingMiddleware keyed its map on the
raw config while matching a port-stripped host, so a domain configured with
a port could never match its own requests. It now shares the normalized
cfg.Auth.Services, so routing and authorization agree on one set of names.
cfg.Auth.ServiceName was an exact alias for Services[0] and is replaced by
PrimaryService(), which also removes an empty-slice index.
Rollout: the audience for seamark.cr and bouy.cr changes, so a token minted
just before the restart draws one 401 and Docker re-handshakes into a valid
one. buoy.cr is unchanged (it stays primary), and atcr.io keeps the service
name it already has today. The challenge and the accepted audience come
from the same delegate, so the retry converges by construction. Deploy as a
single flip, not a canary: an old instance ignores ?service= and would keep
minting the primary audience while a new one rejects it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to 37bab32. That commit stopped deleting OAuth sessions on transient
errors, which fixed spurious sign-outs but overshot on one path: a genuinely dead
session stopped being evicted at all, turning a forced re-login into a permanent
failure loop.
GetOrFetchServiceToken flattened every non-200 from getServiceAuth into
fmt.Errorf("service auth failed with status %d: %s"). IsSessionInvalidError then
had nothing structured to inspect, and its string fallback could not help: it
looks for the OAuth 2.0 code invalid_token, while atproto emits the XRPC name
InvalidToken. The difference is the underscore, not the case, so lowercasing
never bridged it. A revoked session came back 401 InvalidToken and was classified
transient, so /auth/token returned 503 forever and the user was never prompted to
re-authenticate.
The non-200 branch now wraps an *atclient.APIError carrying the status and the
parsed atproto error name, which is what the existing structured checks in
IsSessionInvalidError already know how to read. Transient shapes stay transient:
atprotoErrorName returns "" for a non-JSON body, so 500s with HTML, 502s, and
429s do not evict.
ExpiredToken is deliberately not treated as a dead session. It means "refresh
me", and deleting on it would sign the user out of every UI session over an
ordinary access-token expiry a refresh would have fixed. isAuthError omits it for
the same reason; the two classifiers have to agree about the same condition.
The comment on the string fallback claimed it was a looser spelling of the
structured check. It is not — it handles a different error family. indigo's
RefreshTokens returns OAuth token-endpoint failures as a bare fmt.Errorf carrying
the auth server's snake_case code verbatim ("token refresh failed (HTTP 400):
invalid_grant"), never a typed error, so a string match is the only thing that
can classify a refresh failure, which is the invalid_grant replay case 37bab32
exists to detect. Both comments now say which family they cover.
Two hardening items on the same theme:
use_dpop_nonce no longer counts as an auth error in the appview's isOAuthError.
It is a routine handshake step indigo retries with the server-supplied nonce, and
treating it as fatal signed users out over ordinary nonce rotation. It can still
escape when a server sends that error with no DPoP-Nonce header, leaving indigo
nothing to retry with; a stuck session there is preferable to signing everyone
out in the common case, and the comment says so rather than claiming it cannot
happen.
Detached session deletes are bounded by SessionDeleteTimeout. They run on
context.WithoutCancel so a canceled request cannot leave the cleanup half-done,
which also stripped the only deadline they had — a wedged database write blocked
the goroutine with no way to shed it. Matches the bound already on the detached
persist callback. The unparseable-token-endpoint warning is now deduped per
endpoint rather than once per process, since that path fails open by returning
the client unwrapped, silently reinstating the refresh burn.
The refreshDetachTimeout comment now notes the cap is per-POST: the DPoP-nonce
retry means one refresh can issue two, holding the per-DID lock for up to twice
the stated value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When a Docker client canceled a slow /auth/token request mid-refresh, the
token-refresh POST was aborted client-side but completed on the PDS, which
rotated the refresh token. The rotated token was never received or persisted,
so the next refresh replayed the consumed token, got invalid_grant, and the
session (OAuth + UI) was deleted, signing the user out everywhere.
- Detach refresh POSTs from the inbound request context via a per-session
RoundTripper (WithoutCancel + 30s cap); once a refresh starts it completes
- Persist session updates (rotated tokens, DPoP nonces) on a detached context
- Gate session deletion on IsSessionInvalidError: cancellation, timeouts, and
transport errors no longer delete sessions; genuine invalid_grant still does
- Add phase timing to /auth/token and per-DID lock wait warnings to attribute
the ~14s pre-refresh stalls that push requests past Docker's deadline
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Evan JarrettandClaude Opus 5