The Jetstream consumer, backfill, labeler subscriber, cleanup sweep and billing
tier refresh all started unconditionally in every process. That is correct for
one instance and wrong for two.
The consumer is the case with teeth. StatsCache is per-process in-memory state,
and the aggregate it produces is written to repository_stats as an absolute
value rather than an increment, so two consumers each hold a partial view of the
holds and each write their partial sum as though it were the whole truth,
overwriting one another indefinitely. The webhook dispatcher hangs off the same
processor, so a second consumer also doubles every delivery.
Each now runs under a named lease, so exactly one instance runs it and a
replacement takes over when that instance goes away. The health worker is
deliberately not leased: it refreshes a cache each instance needs locally, so
running it everywhere is correct.
Two structural changes came with it. The cleanup loop moved out of
InitializeDatabase, where it was a bare goroutine with no way to reach the lease
manager, into RunPeriodicCleanup called from the server. And backfill's startup
run and periodic schedule became one leased worker instead of two goroutines on
context.Background(), so shutdown actually stops a backfill in flight rather
than letting it run on against a closing database. With interval=0 that worker
holds its lease instead of returning, since releasing would let another instance
acquire and run its own startup backfill, turning "once" into "once per
instance".
Verified with two instances against one database: exactly one acquired, the
other contended without starting a worker; SIGTERM handed over in 13ms via the
release, SIGKILL handed over in ~12s via TTL expiry.
That first number only holds because of Manager.Go and Manager.Wait, which this
commit adds. The first cut used `go m.Run(...)` and cancelled the worker context
during shutdown without waiting, so the process exited before the release landed
and the lease survived to its TTL — a rolling deploy would have paused indexing
for a minute rather than a second. Nothing in the unit tests caught it; the
two-instance run did. TestWaitBlocksUntilLeaseReleased covers it now.
leases.enabled defaults to true. A single instance is unaffected, since it
always wins its own leases, while an operator who scales out without reading the
docs still gets correct behavior instead of silent stats corruption.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
a7569a7 added credential-less pulls of public images. Three things about it
were wrong, all of them in how the appview handled the decision that belongs
to the hold.
**Scope handling was all-or-nothing.** IsPullOnlyScope required every
requested action to already be "pull", but clients routinely ask for more
than the operation needs — pull,push is common for a plain read, and some
ask for pull,push,delete up front. Those were rejected and challenged,
leaving a credential-less client no way to pull even a public image, which
is the entire feature. NarrowToPullOnly drops the write actions and issues a
token carrying "pull" and nothing else. Granting a subset is what the
distribution token spec expects. The allowlist property is preserved: "pull"
is the only action that survives, and "*" is deliberately not expanded into
it, since a wildcard request is not evidence the caller wants a read.
**The appview-side read gate was inert.** checkReadAccess passed p.ctx.DID,
the DID of the repository *owner*, not the requester. Any non-empty DID
satisfies a private hold's check, and the owner's is never empty, so it asked
"may the owner read their own hold", answered yes, and admitted everyone.
Worse, CheckReadAccessWithCaptain admitted any authenticated DID to a private
hold at all, on an explicitly-MVP assumption that holding a DID was close
enough to being a sailor. Every doc says otherwise (docs/hold.md:109 "Crew
with blob:read", CLAUDE.md:140, docs/BYOS.md:280) and so does the hold
(ValidateBlobReadAccess: owner, or crew carrying blob:read/blob:write). It
now takes isCrew and requires owner-or-crew, and callers only pay for the
crew lookup when it can change the answer — a public hold or an anonymous
caller is decided by the captain record alone. Nothing here loosens access;
it brings the local gate into agreement with the authority.
**Denials could not reach the client.** distribution's blobHandler.GetBlob
maps everything except ErrBlobUnknown to ErrorCodeUnknown, so a 401 raised
in the blob store left as a 500 — misreporting an auth failure as a server
fault, and giving BearerChallenge no 401 to attach WWW-Authenticate to, so
Docker was told "server error" instead of being prompted for credentials.
Clients that retry 5xx looped: 4.1s per case in the matrix, now 0.01s. The
check moves to Repository(), where an errcode.Error is passed through
verbatim by the registry app — the same mechanism a7569a7 used for
NAME_UNKNOWN. It fails open on a lookup error, since the hold is the
authority and a transient failure should not break public pulls.
Removes auth.allow_anonymous_pull. It could only ever withhold — captain.Public
is what grants — so it was a second flag for a decision the hold already owns,
and gating it appview-side was never the intent. Layer bytes 307 straight to
S3, so the appview is not even in the path whose cost might have justified an
operator-side lever.
Tests: TestAuthMatrix only ever ran against a public hold, and its pull cases
never fetched a layer — crane.Pull is lazy and img.Digest() needs only the
manifest, which ATCR serves from the user's PDS where it is world-readable, so
no pull row in the matrix touched blob authorization at all. Pulls now
materialize layer bytes, and testharness.WithPrivateHold plus
TestAuthMatrixPrivateHold cover public:false + allow_all_crew:true — the
production shape, where anyone with an account pulls and pushes and anonymous
gets nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Credential-less pulls of public images. /auth/token issues a pull-only
token with an empty subject when no Basic auth is present; the
destination hold still enforces captain.Public, and push or delete always
challenges.
- token.IsPullOnlyScope and AuthMethodAnonymous;
Handler.issueAnonymousToken skips the authorizer gate and the
service-auth pre-mint, since there is no identity to reconcile and no
AppView-to-hold service token to bind. The token is still stamped
with the resolved registry domain, so anonymous pull works on
secondary front doors whose access controller demands their own
audience.
- auth.allow_anonymous_pull (default true) turns it fully off, restoring
the previous always-challenge behavior. Mirrored into the deploy
template, since the default means existing deploys pick this up.
- RegistryContext.Anonymous is plumbed from the middleware.
- ProxyBlobStore sends no Authorization header when the service token is
empty, and returns 401 rather than 403 for anonymous denials so Docker
prompts for credentials, including when a stale captain cache lets the
request through and the hold says private.
- BearerChallenge wraps the /v2/ subtree so a 401 raised deep in the
stack via errcode.ServeJSON still carries WWW-Authenticate.
Distribution's own scoped challenges are left alone.
IsPullOnlyScope allowlists the pull action instead of denylisting push and
delete. Distribution's actionSet.contains treats "*" as *every* action, so
a scope of `repository:victim/img:*` names neither denied string and would
have handed an unauthenticated caller a token valid for push and delete on
someone else's repository — clearing the authgate entirely, since anonymous
tokens deliberately skip it. Writes would still have failed further down
(no PDS credential), but the gate itself was bypassable. Now every
requested action must be exactly "pull". Covered by new claims tests.
Unresolvable identities return NAME_UNKNOWN instead of a bare error that
distribution renders as 500. This path was previously unreachable without
credentials; anonymous pull opens it to the internet, and a 5xx on
arbitrary input both misreports a bad request as a server fault and sends
clients that retry 5xx into a retry loop. That loop was real: in the auth
matrix, regclient spent 83s on a single case before this fix, and the
suite now runs in 5s.
Stat preserves an authorization verdict from getPresignedURL rather than
flattening it to ErrBlobUnknown. Distribution calls Stat before ServeBlob
on GET and HEAD, so without this an anonymous pull from a private hold
answered 404 and BearerChallenge had no 401 to annotate — the 401 path
above could never actually reach a client.
The auth matrix is updated to match: anonymous pull of the seeded public
repo now succeeds, anonymous push is denied against a real identity's
namespace (rather than an unresolvable one, which was testing name
resolution rather than authorization), and a new case pins the
NAME_UNKNOWN behavior for an unknown identity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1. Multiple registry domains + per-user domain preference
The biggest feature. The appview can serve several registry domains (e.g. buoy.cr, atcr.io),
and users can now pick which one shows up in their pull/push commands.
- Lexicon/record: adds registryDomain (and documents ociClient)
to the sailor profile (lexicons/.../profile.json, pkg/atproto/lexicon.go).
- DB: new registry_domain column on users (schema.sql + migration 0027),
with GetUserByDID/Handle reads, UpdateUserRegistryDomain writer,
and Jetstream caching it on profile updates (writes unconditionally so clearing propagates).
- UI/handlers: new UpdateRegistryDomainHandler + /api/profile/registry-domain route,
a <select> in the user settings panel (only shown when >1 domain configured), and resolveRegistryURL()
which falls back to the primary domain if the user's pref is stale/removed. Tests added for all of it.
2. default_hold_did removed → first managed_holds entry is the default
Consolidates two overlapping config fields into one. ServerConfig.DefaultHoldDID is gone;
PrimaryHoldDID() now returns managed_holds[0]. managed_holds is now REQUIRED.
Updated in config, validation, server wiring, test harness, example YAML, and the deploy template.
3. Admin long-running operations → generic background-job framework
New pkg/hold/admin/jobs.go introduces a reusable startJob/jobRegistry pattern
(a detached context.Background() job + a /admin/api/jobs/{key}/status polling endpoint).
This replaces the bespoke scan-backfill goroutine state machine, and now also wraps crew tier remap and crew import
all three previously looped synchronously on the request context and got 504'd/cancelled mid-run by the reverse proxy.
Forms switched from POST-redirect to htmx fragments (job_progress.html, job_result.html, crew_import_results.html)
the old crew_import_results.html page and scan_backfill_progress.html partial were deleted.
This is also captured as a new rule in CLAUDE.md.
4. Cascade-delete manifest on last-tag deletion
DeleteTagHandler now, after removing the last tag pointing to a digest, cascade-deletes the manifest itself
(PDS + DB + hold blob purge) — but only if it's not a child of a manifest list (multi-arch parent).
New GetTagDigest and ShouldCascadeDeleteManifest queries back it, plus cascade_delete_test.go.
Also switches tag rkey computation to the atproto.RepositoryTagToRKey helper.
5. Billing simplification
Drops the OwnerBadge config option (hold-owner supporter badge).
The user-profile template no longer special-cases an "owner" badge value (only "Captain").
Example tiers renamed to the nautical scheme (deckhand/bosun/quartermaster).
6. Build/deploy: go generate always runs via Make
make generate is now a phony target that always runs go generate ./... (regenerating cbor_gen, icon sprites, etc.),
and build-trixie depends on it. The deploy tooling (provision.go/update.go)
drops its own runGenerate calls since the Makefile handles it.
7. New cmd/firehose-tap tool (untracked)
A standalone CLI that subscribes to a com.atproto.sync.subscribeRepos endpoint and pretty-prints events,
with emphasis on Sync 1.1 compliance fields (per-op prev CIDs, commit prevData) and a --validate CI mode.
Fits with the recent "more sync1.1 compliant" commit.
Evan JarrettandClaude Opus 5