Seen in production on 2026-09-11: three cold pulls of a 22-layer image failed
with BLOB_UNKNOWN for layers that exist. The hold had answered 403 "service
token authentication failed: token has expired", and the same blobs served
fine a minute later.
Three things lined up. The registry middleware's validation cache kept a
fetched service token for a flat 45 seconds regardless of its real remaining
life, so a token fetched with 12 seconds left was still handed to the hold
half a minute after it died. The registry JWT is stamped from the auth cache's
expiry, which trailed the real exp by only 10 seconds, while distribution
accepts a JWT for 60 seconds past its exp, so a client could hold an accepted
JWT for most of a minute after the credential behind it was gone. And the
hold's 403 was flattened to BLOB_UNKNOWN, so the client failed instead of
re-authenticating.
Now the validation cache bounds an entry by the token's exp minus a shared
ServiceTokenSafetyMargin of 60 seconds, the same margin the auth cache and the
JWT stamp use, chosen to equal distribution's leeway so the last instant a JWT
is accepted is the service token's real exp. A PDS that grants less than the
margin gets half its remaining life instead of an already-past deadline. When
the hold rejects the service token as expired or missing, the appview drops
both cached copies and returns a 401 challenge so Docker and crane re-run the
token dance and retry; a genuine permission denial stays a 403, and a hold
that is down still maps to blob unknown.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EvFJr4Dwz8p2NDAeXmgmBt
The registry middleware used to GET the user's hold's /.well-known/did.json
on every registry request in testmode builds, and fall back to the appview
default hold if it did not answer. The fallback only ever changed anything
when the user's chosen hold differed from the default AND was down; in local
development and in the integration harness the two are the same hold, so the
probe's answer was discarded every time. In the production-shaped benchmark
it was 94 of 150 hold requests per p90 push and 23 of 90 per pull, hiding
the real hold traffic behind a testmode artifact.
Remove isHoldReachable, the fallbackUnreachable field, and the probe branch.
An empty choice still means the default hold. Testmode and production now
take the same path here.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WTdBxLFU5TpwmqVdVsN1wq
server.test_mode survived the build-tag refactor only to feed five
behavioral branches: the registry's fall-back to the default hold when
the user's hold is unreachable, backfill warning suppression for
external holds, the appview listener close on shutdown, the hold's
relay-crawl skip, and the hold's appview-issuer tolerance. Every one of
them is a "this is a local development build" decision, which is what
the tag already says, and local development has to build with the tag
or nothing resolves. So they read atproto.TestModeBuild now, and the
flag, SetTestMode, IsTestMode, the middleware option, the backfill
constructor parameter, the never-read field on RemoteHoldAuthorizer,
the example and template YAML lines, and the docker-compose env vars
are gone. The registry keeps the fallback as a field seeded from the
constant so the production-path tests can pin it off under the tag.
The 24 SetTestMode calls in tests were dead already: stripping them and
running the affected packages tagged changed nothing.
Tests that resolve a loopback did:web used to t.Fatal naming the tag,
which left a bare `go test ./...` permanently red in five packages.
They now live under `//go:build testmode`: whole-file constraints where
every test needs it, and sibling *_testmode_test.go files holding the
moved tests plus their fixtures where a file mixed. The harness carries
the constraint too, with its package doc in an untagged doc.go so the
package still exists without it. An untagged run compiles those tests
out and passes; make test keeps the tag and runs everything.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UwYzaG3Yy7uA8FbZ5qk3tQ
indigo's identity directory refuses HTTP and IP-hosted did:web, and its
OAuth client is growing an SSRF-guarded transport that refuses loopback
and private addresses. Local development and the test suites need both,
and the workarounds were scattered: two did:web fallbacks in the
resolver, a hand-rolled appview key fetch on the hold, and the OAuth
client left on indigo's defaults so any test driving it against an
httptest server depended on the transport staying permissive.
Move every departure from indigo's defaults into one file pair in
pkg/atproto: indigo_prod.go (!testmode) returns indigo's directory and
OAuth client unchanged; indigo_local.go (testmode) wraps the directory
so a did:web naming an IP, localhost, or a host with a port resolves
over plain HTTP, and gives the OAuth client plain HTTP clients. All six
identity and OAuth constructor call sites go through NewDirectory and
NewOAuthClientApp. The resolver fallbacks, DIDWebToURL, and the hold's
scheme-guessing key fetch are gone; the hold resolves the appview key
through the directory, preferring #appview, and purges and retries once
on a signature failure so a re-keyed appview is not masked by the
24-hour cache.
There is no runtime switch for this: a production binary cannot be
configured to resolve local DIDs. The runtime test_mode flag still
gates the remaining behavioral branches only.
Tests, the harness, make dev, Air, Dockerfile.dev, and docker-compose
build with the tag; fixtures that need loopback did:web fail fast
naming it. Test hold servers now serve a did.json via pkg/testpds so
they resolve as real holds under the tag.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UwYzaG3Yy7uA8FbZ5qk3tQ
Hold discovery in the registry middleware called getRecord on the
repository owner's PDS for every request under /v2/: every HEAD, POST,
PATCH, PUT and GET. A 10-layer push was 40 or more PDS round trips, and
it was the last per-request network call on the push path that had
nothing to do with moving bytes. Only two profile fields are used
there: the default hold and the auto-remove-untagged flag.
The users row already caches the default hold, written by the Jetstream
processor on every profile event and prefilled by the backfill, and the
auth gate already reads it from there. This makes the row a faithful
copy of what the registry needs and switches the middleware to it.
The auto-remove flag gets a nullable users column. NULL means the value
has never been learned; the processor writes 0 or 1 on every profile
event and never NULL. On a request whose row is missing or still NULL,
the middleware does one live fetch, uses it, and writes both fields
back, including a 0 for a user with no profile at all, so the fallback
runs at most once per user. A failed fetch writes nothing and uses the
appview default for that request, so a network error is never cached.
That single mechanism covers the minutes after a deploy while the
startup backfill fills the column, a brand-new user, and a user the
backfill has not reached.
The processor also stops returning early on an empty default hold,
which left a user who removed their custom hold pushing to it forever.
Empty is now written through and means the appview default, matching
what the auth gate already reads.
Tests count PDS requests with a test server: a populated row makes
none, a NULL row makes exactly one and then none, a missing profile is
cached as known, and a failed fetch degrades without writing. The
migration was applied to a fresh database and to one built from the
previous schema.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
A user with an unresolvable defaultHold (did:web:localhost%3A8080) got HTTP
500 with a body of literally {} on every request in their namespace, and
crane retried it three times because 500 is retryable.
The cause is in the distribution library. handlers/app.go:755 switches on the
error type returned by Repository() with cases for ErrRepositoryUnknown,
ErrRepositoryNameInvalid and errcode.Error, and no default. A bare fmt.Errorf
matches none of them, so context.Errors stays empty; ServeJSON then finds no
ErrorCoder, leaves sc == 0, falls through to 500, and Errors.MarshalJSON
renders the nil slice as {} via omitempty.
So every error leaving Repository() must be coded. Four were not:
hold URL unresolvable -> 404 NAME_UNKNOWN when errors.Is
atproto.ErrHoldDIDPermanent, else 503 UNAVAILABLE
no hold DID configured -> 500 UNKNOWN, but with a body and a log line
invalid image name -> 400 NAME_INVALID
name missing an owner -> 400 NAME_INVALID
The permanent/transient split is the point: a DNS blip must stay retryable,
but a did:web that can never resolve must not be retried at all. Stored user
data that cannot resolve is a 4xx condition, not a server fault, and the
NAME_UNKNOWN message now names the hold so the owner can fix their profile.
The hold DID is already world-readable in their sailor profile record.
Tests assert through a helper that replays distribution's exact type switch,
so an uncoded error still surfaces as 500 {} and the assertions bind to the
real behaviour rather than to the constructors.
Does not address the logrus line at app.go:757, which fires unconditionally
before the type switch and cannot be avoided by any returned error type.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9
a7569a7 added credential-less pulls of public images. Three things about it
were wrong, all of them in how the appview handled the decision that belongs
to the hold.
**Scope handling was all-or-nothing.** IsPullOnlyScope required every
requested action to already be "pull", but clients routinely ask for more
than the operation needs — pull,push is common for a plain read, and some
ask for pull,push,delete up front. Those were rejected and challenged,
leaving a credential-less client no way to pull even a public image, which
is the entire feature. NarrowToPullOnly drops the write actions and issues a
token carrying "pull" and nothing else. Granting a subset is what the
distribution token spec expects. The allowlist property is preserved: "pull"
is the only action that survives, and "*" is deliberately not expanded into
it, since a wildcard request is not evidence the caller wants a read.
**The appview-side read gate was inert.** checkReadAccess passed p.ctx.DID,
the DID of the repository *owner*, not the requester. Any non-empty DID
satisfies a private hold's check, and the owner's is never empty, so it asked
"may the owner read their own hold", answered yes, and admitted everyone.
Worse, CheckReadAccessWithCaptain admitted any authenticated DID to a private
hold at all, on an explicitly-MVP assumption that holding a DID was close
enough to being a sailor. Every doc says otherwise (docs/hold.md:109 "Crew
with blob:read", CLAUDE.md:140, docs/BYOS.md:280) and so does the hold
(ValidateBlobReadAccess: owner, or crew carrying blob:read/blob:write). It
now takes isCrew and requires owner-or-crew, and callers only pay for the
crew lookup when it can change the answer — a public hold or an anonymous
caller is decided by the captain record alone. Nothing here loosens access;
it brings the local gate into agreement with the authority.
**Denials could not reach the client.** distribution's blobHandler.GetBlob
maps everything except ErrBlobUnknown to ErrorCodeUnknown, so a 401 raised
in the blob store left as a 500 — misreporting an auth failure as a server
fault, and giving BearerChallenge no 401 to attach WWW-Authenticate to, so
Docker was told "server error" instead of being prompted for credentials.
Clients that retry 5xx looped: 4.1s per case in the matrix, now 0.01s. The
check moves to Repository(), where an errcode.Error is passed through
verbatim by the registry app — the same mechanism a7569a7 used for
NAME_UNKNOWN. It fails open on a lookup error, since the hold is the
authority and a transient failure should not break public pulls.
Removes auth.allow_anonymous_pull. It could only ever withhold — captain.Public
is what grants — so it was a second flag for a decision the hold already owns,
and gating it appview-side was never the intent. Layer bytes 307 straight to
S3, so the appview is not even in the path whose cost might have justified an
operator-side lever.
Tests: TestAuthMatrix only ever ran against a public hold, and its pull cases
never fetched a layer — crane.Pull is lazy and img.Digest() needs only the
manifest, which ATCR serves from the user's PDS where it is world-readable, so
no pull row in the matrix touched blob authorization at all. Pulls now
materialize layer bytes, and testharness.WithPrivateHold plus
TestAuthMatrixPrivateHold cover public:false + allow_all_crew:true — the
production shape, where anyone with an account pulls and pushes and anonymous
gets nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Credential-less pulls of public images. /auth/token issues a pull-only
token with an empty subject when no Basic auth is present; the
destination hold still enforces captain.Public, and push or delete always
challenges.
- token.IsPullOnlyScope and AuthMethodAnonymous;
Handler.issueAnonymousToken skips the authorizer gate and the
service-auth pre-mint, since there is no identity to reconcile and no
AppView-to-hold service token to bind. The token is still stamped
with the resolved registry domain, so anonymous pull works on
secondary front doors whose access controller demands their own
audience.
- auth.allow_anonymous_pull (default true) turns it fully off, restoring
the previous always-challenge behavior. Mirrored into the deploy
template, since the default means existing deploys pick this up.
- RegistryContext.Anonymous is plumbed from the middleware.
- ProxyBlobStore sends no Authorization header when the service token is
empty, and returns 401 rather than 403 for anonymous denials so Docker
prompts for credentials, including when a stale captain cache lets the
request through and the hold says private.
- BearerChallenge wraps the /v2/ subtree so a 401 raised deep in the
stack via errcode.ServeJSON still carries WWW-Authenticate.
Distribution's own scoped challenges are left alone.
IsPullOnlyScope allowlists the pull action instead of denylisting push and
delete. Distribution's actionSet.contains treats "*" as *every* action, so
a scope of `repository:victim/img:*` names neither denied string and would
have handed an unauthenticated caller a token valid for push and delete on
someone else's repository — clearing the authgate entirely, since anonymous
tokens deliberately skip it. Writes would still have failed further down
(no PDS credential), but the gate itself was bypassable. Now every
requested action must be exactly "pull". Covered by new claims tests.
Unresolvable identities return NAME_UNKNOWN instead of a bare error that
distribution renders as 500. This path was previously unreachable without
credentials; anonymous pull opens it to the internet, and a 5xx on
arbitrary input both misreports a bad request as a server fault and sends
clients that retry 5xx into a retry loop. That loop was real: in the auth
matrix, regclient spent 83s on a single case before this fix, and the
suite now runs in 5s.
Stat preserves an authorization verdict from getPresignedURL rather than
flattening it to ErrBlobUnknown. Distribution calls Stat before ServeBlob
on GET and HEAD, so without this an anonymous pull from a private hold
answered 404 and BearerChallenge had no 401 to annotate — the 401 path
above could never actually reach a client.
The auth matrix is updated to match: anonymous pull of the seeded public
repo now succeeds, anonymous push is denied against a real identity's
namespace (rather than an unresolvable one, which was testing name
resolution rather than authorization), and a new case pins the
NAME_UNKNOWN behavior for an unknown identity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Evan JarrettandClaude Fable 5.1