Each in-flight blob upload buffers up to 16MB, Docker pushes five layers
at once per client, and nothing bounded the total. Writers also lived in
the package-level map forever: a client that died mid-push left its
writer, its buffer, and any hold-side S3 multipart session behind with
no expiry.
A process-wide budget (golang.org/x/sync semaphore, default 512MB,
server.upload_buffer_budget_mb) now caps memory held in upload buffers.
A writer charges its buffer's projected backing capacity before growing,
so a config blob costs kilobytes and a full writer costs exactly one
buffer, and releases once, on Commit, Cancel, or reap. A write that
needs budget waits on the request's context with a five minute cap,
outside the writer's lock so Cancel and the sweeper cannot queue behind
it; that wait is backpressure on the client. The budget is clamped to
at least one buffer so a single upload can never deadlock.
A sweeper started with the other appview workers reaps writers idle
past server.upload_idle_timeout (default 1h), aborting the hold-side
multipart on a detached context and releasing the budget. It measures
inactivity, not age, so a slow push is never reaped, and it skips a
writer whose lock is held so it cannot race a live part upload.
Write also gains a fix the budget made visible. It appended a whole
chunk and checked afterwards, so the last chunk before a flush could
land a few bytes past 16MB, which did not fit the backing array;
bytes.Buffer doubled it to 32MB and Reset kept that for the rest of the
upload. Only chunk sizes that tile 16MB exactly avoided it, and the
network read loop promises no such thing. Every large layer could hold
32MB while the budget charged 16. Write now fills to exactly the
threshold, flushes, and continues with the remainder, so capacity is
pinned at 16MB for any chunk size, every part is exactly one buffer,
and a single oversized Write streams through as parts instead of
buffering whole. The test streams 24KB chunks across the boundary and
fails against the old code with cap 33554432.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
Two corrections to the HTTP/2 change, both found by running it.
UpCloud refuses http2_enabled on a backend that is not TLS:
invalid_params_properties.http2_enabled='Tls must be enabled to enable HTTP2.'
This load balancer reaches both origins over the private network in cleartext,
so the option is not available here at all, and asking for it failed the whole
provision run with a 400 after the frontend had already been changed. Backend
HTTP/2 is removed rather than made conditional: it would need TLS terminated at
the origins, which is a much larger change than it earns. The frontend is the
leg that mattered, and it is enabled and confirmed live — all three domains now
negotiate h2 by ALPN.
The second correction is more interesting: enabling HTTP/2 did not fix the
symptom it was aimed at, and briefly made it look worse. With the browser no
longer rationing itself to ~6 connections, every row of the crew tab now reaches
the server at once, and the 504s went from a flat 10s to a climbing ladder:
[HTTP/2 504 10026ms] ... [HTTP/2 504 15230ms] ... [HTTP/2 504 25976ms]
HTTP/1.1 had been hiding a server-side limit by throttling the client. The queue
moved; it did not disappear. What is actually slow is resolveHandle, which calls
live identity resolution per row with no bound of its own, so a DID whose
resolution hangs — a did:web on a host that stopped answering, a PDS that
accepts and then stalls — holds its request open until the proxy gives up. The
crew tab issues one request per member, so a hold with hundreds of crew gets
hundreds of chances to hit one.
resolveHandle now bounds each lookup at 2s. A handle is decoration on a row
whose DID is already rendered beside it, so waiting seconds for one and failing
the row when it does not arrive trades something load-bearing for something
cosmetic. Two seconds is far above a warm cached lookup and far below the
proxy's 10s cut, so a slow DID costs its own row a handle and costs the page
nothing.
The quota lookup on the same handler was the other suspect and was measured out:
records(collection, did) is indexed and the aggregation returns immediately on
the production database.
This does not remove the N+1 — the tab still issues one request per member, and
a batch endpoint is the real fix. It removes the failure.
Verified: the guard is covered by a test using a directory that hangs until its
context is cancelled, asserting both that the call returns near the bound and
that it did not return suspiciously early. make lint 0 issues, make test green
across 44 packages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AV6Mk2AgghFsNo4HWQEaBV
build-trixie hardcoded `-tags billing`, so the only way to build a non-billing
appview was to bypass the Makefile and drive the trixie container by hand. Both
production deploys of this deployment did exactly that, and the workspace notes
record it as "the deploy tool cannot reproduce this build" — a tool that cannot
produce the artifact you actually ship is a tool nobody can safely use.
Worse, the failure is silent and one-directional. Nothing about a billing binary
looks different: same name, same version stamp, same vcs.revision. A deploy that
forgot the manual path would quietly switch billing on for a deployment whose
operator had chosen to run without it, exposing a live Stripe webhook endpoint
and a paid tier ladder on a service configured for neither. Production has been
running a non-billing appview since May precisely because someone did not use
`make build-trixie`.
Billing is now opt-in and off by default:
make build-trixie # no billing
make build-trixie BILLING=1 # billing
deploy/upcloud update appview # no billing
deploy/upcloud update appview --with-billing
Only the appview is affected; hold, scanner, labeler and the credential helper
never reference pkg/billing.
The flag alone is not enough, so verifyAppviewBilling reads the built binary
before anything is uploaded and refuses to ship a mismatch. It observes rather
than trusts, because the flag and the artifact can disagree for reasons the flag
cannot see: a stale bin/atcr-appview from an earlier build, a Makefile that
hardcodes the tag, a builder image that ignored it. The Stripe SDK links only
under the tag, so its symbols are a direct measurement — verified as a
discriminator here: 2338 stripe-go strings with the tag, 0 without.
A missing binary is an error rather than an absence of symbols, so a deploy
cannot proceed on a file that was never built by reading it as "no billing".
Verified: the default build-trixie output has 0 stripe-go symbols and does carry
the "Billing is not enabled on this deployment" stub. Guard covered both
directions by test, plus the missing-binary case. make lint 0 issues across
root, deploy and credential-helper; make test green across 44 packages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AV6Mk2AgghFsNo4HWQEaBV
`go fix` carries the modernize analyzers now, and the tree had drifted behind
them. This is the mechanical result, reviewed rather than trusted: the tool is
capable of rewriting code into something that no longer tests or does what it
did, so every non-test change was read individually and the concurrency-bearing
packages were re-run under -race.
Production code, four changes, all semantics-preserving:
- leases/manager.go: wg.Add(1) + go + defer wg.Done() becomes wg.Go. The
comment above that function turns on Add happening before the goroutine
starts, so that a Wait cannot return before the worker has run. wg.Go does
the Add synchronously on the calling goroutine, so the invariant it
describes still holds.
- auth/token/handler.go: strings.Fields -> strings.FieldsSeq, same splitting,
iterated rather than allocated.
- hold/gc/gc.go: a hand-written map copy -> maps.Copy.
- hold/pds/scan_broadcaster.go: three-clause loop -> range over int.
The rest are tests. The one worth naming is carstore_contention_test.go, where a
careless rewrite could have quietly stopped exercising contention: go fix
converted the reader and side-table goroutines to loopWG.Go but correctly
declined to touch the writer loop, which passes its index as a parameter. The
writer/reader/side-table shape and the stop channel are unchanged, so the test
still contends over the same carstore transactions.
Verified: go build for hold and appview, `make lint` 0 issues, the deploy and
credential-helper modules 0 issues, `make test` green across all 43 packages,
and -race green on leases, hold/pds, hold/gc and auth/token. The scanner module's
two lint findings are unchanged from HEAD and are in files go fix never touched.
Kept separate from the HTTP/2 commit so that one stays readable, and so this can
be reverted on its own if a modernization turns out to matter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TA9D4DjaLZTvzQ7dJbu4eg
The admin crew tab renders one row per crew member and gives each row its own
hx-get, so opening it on a hold with 551 crew issues 551 requests. Over
HTTP/1.1 a browser runs at most ~6 per origin, so they queue six at a time and
every other request to the same host queues behind them — which is why loading
the relay page stalls while the crew rows are still resolving, and why the rows
that lose the race come back as "Server error" toasts. The 504 behind that toast
is the load balancer's, not the hold's: the hold logs those requests as 200.
HTTP/2 multiplexes them over one connection and the queue disappears. It does
not make the slow rows fast — that is a separate fix to the per-row identity
lookup — but it stops one slow surface from blocking the rest of the panel.
Two halves, because neither works alone.
The load balancer terminates TLS and speaks cleartext to the origin, so ALPN
never runs on the backend leg and net/http can only answer HTTP/1.1 there. Both
servers now wrap their handler in h2c. The wrapper is opt-in per connection: it
upgrades only for a client sending the h2c preface or "Upgrade: h2c", and passes
everything else through untouched, so an HTTP/1.1 WebSocket upgrade is
unaffected. Verified both directions against this wiring — HTTP/1.1 for a plain
client, HTTP/2.0 with --http2-prior-knowledge.
The frontend's http2_enabled was never set, so it sat at the UpCloud default of
off. That is the half the browser actually sees. timeout_client is now stated
explicitly at its current 10s rather than left implicit: it is the boundary that
produces the 504s above, so it belongs somewhere visible. It is deliberately
unchanged — raising it without fixing the slow lookup would only make a stalled
row stall longer.
The hold's *backend* stays on HTTP/1.1. It serves subscribeRepos over WebSocket
to external relays and to the scanner, and WebSocket over HTTP/2 needs the RFC
8441 Extended CONNECT that Go's http2 server does not implement for Upgrade:.
Routing that backend over h2 would break the firehose. The appview accepts no
inbound WebSocket and has no such constraint. Both origins carry h2c regardless,
so enabling it for the hold later is a config change, not a code change.
createLoadBalancer only runs when there is no LB yet, so properties set there
would reach a new deployment and never an existing one. ensureLBHTTP2 reconciles
them onto an LB that already exists, following ensureLBForwardedHeaders: read
what is there, change only what differs, report what it did, and no-op on a
second run. Backend modifies carry the existing health check back, since
Properties replaces the object wholesale.
Also gofmt: provision.go was not gofmt-clean at HEAD, unrelated to this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TA9D4DjaLZTvzQ7dJbu4eg
buildOCILayout already removes its scan dir on every error path, and syft.go
defers the stereoscope generator's Cleanup. What neither can do is clean up
after a process that dies mid-scan: the deferred call never runs, and nothing
afterwards ever looks at what was left. Every restart therefore leaks the
in-flight layout and extraction permanently, and a restart is routine — a
deploy is one.
On seamark-hold that reached 8.8 GB of orphaned scan-*, syft-scan-* and
syft-cataloger-* directories under a 20 GB disk, at which point the disk was
97% full and scans began failing on it:
failed to load OCI image: unable to populate layer cache
dir="/var/lib/seamark/scanner/tmp/syft-scan-1546187834/..."
: no space left on device
failed to download layer 5: failed to write blob:
write /var/lib/seamark/scanner/tmp/scan-4160414849/blobs/sha256/...
: no space left on device
The leaked directories cluster at the scanner's restart timestamps, which is
what identifies the killed process rather than the error paths as the source.
Manual removal reclaimed 8.8 GB and took the disk from 97% to 50%.
Startup is where this belongs: it is the one moment the previous process is
known to be gone, and it is immediately after the event that caused the leak.
The sweep runs in WorkerPool.Start after TMPDIR is set and before any worker
can dequeue, so nothing it removes can be work in progress here.
Three constraints shape what it will touch:
- Only the three per-job prefixes, only as direct children, only
directories. The Grype database lives beside the tmp dir at
<parent>/vulndb and go-getter unpacks into grype-dl underneath it; both
are state the scanner needs and neither matches a prefix. The prefixes now
have one definition each, used by both the creator and the sweeper, so
renaming a directory cannot silently take it out of the sweep's scope.
- An age threshold, vuln.sweep_max_age, default 1h. A second scanner sharing
the directory has an in-flight scan-* dir that is minutes old, and
scanner.job_timeout is 8m, so an hour clears both with room to spare. 0
disables the sweep rather than removing a peer's live work.
- Nothing is fatal. A stat or removal failure is a WARN and the sweep moves
on, so a permission problem in the tmp dir cannot keep the scanner from
starting.
The sweep only runs at startup, so a scanner that is killed twice between
deploys carries the first leak until its next restart. That is the tradeoff
for never racing a live peer; a periodic sweep would be the follow-up if
processes ever live long enough for it to matter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TA9D4DjaLZTvzQ7dJbu4eg
Nothing limited how long one job could take. The worker's context was the
process's, buildOCILayout took none, and blob downloads used a package-level
client whose five-minute timeout is per request with no context, so a 19-layer
image had a hundred-minute worst case on downloads alone and cancellation could
not touch it. At the default single worker, one wedged job stopped that scanner
entirely.
scanner.job_timeout, default 8m, against the hold's 10m scanning timeout. Both
clocks start at the same instant: the worker sends "started" on dequeue and
derives the job context on the next line, so the scanner loses by two minutes,
which is enough for its terminal message to cross the socket and be recorded.
If the hold wins instead it re-dispatches while this scanner is still working,
which is duplicate work recorded under a generic reason. A scanner cannot read
the hold's config, so the relation is a mirrored constant used only for a
boot-time warning, and the same warning fires if the deadline is disabled.
What is actually bounded, since a deadline the code cannot honour is worse than
none: presign, download, stereoscope's Provide, Syft's CreateSBOM, and Grype,
which does have FindMatchesContext even though FindMatches does not.
stereoscope's img.Read takes no context and is 81% of a scan, so it is checked
either side rather than interrupted. Abandoning it on a goroutine would trade a
bounded overrun for one writing gigabytes into a directory the caller has
already deleted. max_image_size remains the real bound on that stage.
A timeout reports error, not skipped. It describes this host at this moment, a
contended CPU or a slow bucket, not the image, and skips are never retried, so
one bad afternoon would retire an image permanently with nothing in the record
to say why. Retry cost is bounded on the other side by max_image_size and by
the stale-scan schedule. The classification asks the job context rather than
the error, because several stages replace the cause and the uninterruptible one
knows nothing about the deadline, and a job that finishes after an overrun
still reports its real result rather than throwing away completed work.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF
The scanner shares a 1 GiB host with the hold, and nothing stopped it taking
the whole box down with it.
max_image_size was absent from the template, so production ran the 2 GiB
default. Peak RSS tracks image size and a 389 MiB compressed image already
reaches 561 MiB, so 512 MiB is the ceiling that fits. Sizing matters in this
order: max_image_size rejects before a byte is downloaded, while MemoryMax
kills mid-scan, and the hold then returns the row to pending and the restarted
scanner is handed the same job again, which is a permanent crash loop on one
image. If the cgroup cap is ever seen firing, lower max_image_size rather than
raising it.
GOMEMLIMIT is soft: the runtime collects harder to respect it and never fails
an allocation to honour it. With no cgroup cap the kernel picks its own OOM
victim, and the other large process on that host is the hold, so the scanner's
overshoot could kill the service it reports to.
CPUWeight and IOWeight, because no unit here set any scheduling directive: a
scan finishing a minute later costs nothing, a hold that cannot answer a pull
costs a user. Weights apply only under contention, so the scanner still uses
the whole box when nothing else wants it.
workers drops to 1. Proactive dispatch was depth-one hold-wide until this
release so the second worker never received anything, and now that it can, two
concurrent scans of a node:22-class image need 687 MiB against a host this
size. Raise it with MemoryMax and the host together.
Note the cgroup ceiling still wants rechecking: it was sized from measurements
taken with Grype disabled, and the vulnerability database's resident pages
count against MemoryMax even though GOMEMLIMIT cannot see them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF
seamark.dev's /install and /settings/devices told users to pipe
seamark.dev/static/install.sh into bash. That file was the unmodified ATCR
script: it announced itself as the "ATCR Credential Helper Installer",
installed docker-credential-atcr, and finished by telling the user to configure
credHelpers for atcr.io, the wrong registry for that deployment. Anyone
following the documented setup ended up pointed at another service. The
templates hardcoded docker-credential-atcr, "atcr" and ~/.atcr/device.json
alongside a correctly themed {{ .RegistryURL }}.
The scripts are now rendered from config by a handler, rather than forked per
brand. A theme overlay was the alternative and was worse: it needed a full copy
of both install.sh and install.ps1 per brand, four scripts to keep in sync, and
the operator asked for these values to come from config.
credential_helper.name is the single knob. Docker resolves a credHelpers value
x by exec'ing docker-credential-x, so the credHelpers value, the binary suffix
and the config directory are genuinely one word, not three that can drift. It
is validated against a strict pattern because it is interpolated into a shell
script.
install.sh renders byte-identical to the deleted static file under the atcr
default, so existing installs are unaffected. install.ps1 differs by one line,
where a stale usage comment named a path the script is not served at.
Two behaviour changes worth noting: these two URLs drop from a one-year
Cache-Control to five minutes, since the body now depends on deployment config;
and credential_helper.tangled_repo becomes a real overridable default. It was
previously assigned over unconditionally and read by nothing, while the shipped
script used a different URL form.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9
Update every direct dependency across all five workspace modules to
latest. Notable jumps: syft v1.43.0 -> v1.51.1, grype v0.111.1 ->
v0.118.0, stereoscope v0.1.23 -> v0.3.1, indigo -> 2026-09-01,
aws-sdk-go-v2/service/s3 v1.99.1 -> v1.110.0, grpc v1.80.0 -> v1.83.2,
x/crypto v0.50.0 -> v0.55.0.
Three deps needed more than a version bump:
go-libipfs could not be updated at all. The repo was renamed to boxo, so
every tag past v0.7.0 declares `module github.com/ipfs/boxo` and cannot
be required under the old path. sqlite_store.go already imported
go-block-format alongside it and used the archived package exactly once,
inside a function already returning blockformat.Block, so it was relying
on structural interface satisfaction. Collapsing to the native type drops
the archived dependency entirely.
go-didplc moved its package from the repo root into a didplc/ subdir in
v0.2.2. Package name is unchanged and every symbol we use (RegularOp,
OpEnum, OpService, Client.DirectoryURL, Submit) is intact, so this is an
import path change only.
The go-diskfs replace in scanner/go.mod had inverted. It pinned v1.7.0
because syft v1.43 passed diskfs entries as os.FileInfo; syft v1.51.1
fixed that upstream and now requires v1.9.4, so the workaround had become
the thing breaking the build. Removed per its own "Remove when syft ships
a fix" note, closing anchore/syft#4796 for us.
The indigo bump needed no code changes: of the 21 packages we import only
5 changed, and the repo/MST/CAR-store core is byte-identical. It does
bring a util/ssrf fix blocking 6to4 addresses (2002::/16), which we
inherit through atproto/auth/oauth.
Go 1.26.7 across go.work, all five go.mod files, the four Dockerfiles,
the three tangled workflows, and the stale references in
docs/DEVELOPMENT.md. Verified golang:1.26.7-trixie resolves on
mirror.gcr.io, which is what the Dockerfiles actually pull from.
Makefile's TRIXIE_BUILDER_IMAGE stays on the floating golang:1-trixie.
make test, make lint, and make test-race all pass, as do the scanner
module's tests and the integration-tagged build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KWoKzpgtBJ33sCyGxJGR7x
The Jetstream consumer, backfill, labeler subscriber, cleanup sweep and billing
tier refresh all started unconditionally in every process. That is correct for
one instance and wrong for two.
The consumer is the case with teeth. StatsCache is per-process in-memory state,
and the aggregate it produces is written to repository_stats as an absolute
value rather than an increment, so two consumers each hold a partial view of the
holds and each write their partial sum as though it were the whole truth,
overwriting one another indefinitely. The webhook dispatcher hangs off the same
processor, so a second consumer also doubles every delivery.
Each now runs under a named lease, so exactly one instance runs it and a
replacement takes over when that instance goes away. The health worker is
deliberately not leased: it refreshes a cache each instance needs locally, so
running it everywhere is correct.
Two structural changes came with it. The cleanup loop moved out of
InitializeDatabase, where it was a bare goroutine with no way to reach the lease
manager, into RunPeriodicCleanup called from the server. And backfill's startup
run and periodic schedule became one leased worker instead of two goroutines on
context.Background(), so shutdown actually stops a backfill in flight rather
than letting it run on against a closing database. With interval=0 that worker
holds its lease instead of returning, since releasing would let another instance
acquire and run its own startup backfill, turning "once" into "once per
instance".
Verified with two instances against one database: exactly one acquired, the
other contended without starting a worker; SIGTERM handed over in 13ms via the
release, SIGKILL handed over in ~12s via TTL expiry.
That first number only holds because of Manager.Go and Manager.Wait, which this
commit adds. The first cut used `go m.Run(...)` and cancelled the worker context
during shutdown without waiting, so the process exited before the release landed
and the lease survived to its TTL — a rolling deploy would have paused indexing
for a minute rather than a second. Nothing in the unit tests caught it; the
two-instance run did. TestWaitBlocksUntilLeaseReleased covers it now.
leases.enabled defaults to true. A single instance is unaffected, since it
always wins its own leases, while an operator who scales out without reading the
docs still gets correct behavior instead of silent stats corruption.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
a7569a7 added credential-less pulls of public images. Three things about it
were wrong, all of them in how the appview handled the decision that belongs
to the hold.
**Scope handling was all-or-nothing.** IsPullOnlyScope required every
requested action to already be "pull", but clients routinely ask for more
than the operation needs — pull,push is common for a plain read, and some
ask for pull,push,delete up front. Those were rejected and challenged,
leaving a credential-less client no way to pull even a public image, which
is the entire feature. NarrowToPullOnly drops the write actions and issues a
token carrying "pull" and nothing else. Granting a subset is what the
distribution token spec expects. The allowlist property is preserved: "pull"
is the only action that survives, and "*" is deliberately not expanded into
it, since a wildcard request is not evidence the caller wants a read.
**The appview-side read gate was inert.** checkReadAccess passed p.ctx.DID,
the DID of the repository *owner*, not the requester. Any non-empty DID
satisfies a private hold's check, and the owner's is never empty, so it asked
"may the owner read their own hold", answered yes, and admitted everyone.
Worse, CheckReadAccessWithCaptain admitted any authenticated DID to a private
hold at all, on an explicitly-MVP assumption that holding a DID was close
enough to being a sailor. Every doc says otherwise (docs/hold.md:109 "Crew
with blob:read", CLAUDE.md:140, docs/BYOS.md:280) and so does the hold
(ValidateBlobReadAccess: owner, or crew carrying blob:read/blob:write). It
now takes isCrew and requires owner-or-crew, and callers only pay for the
crew lookup when it can change the answer — a public hold or an anonymous
caller is decided by the captain record alone. Nothing here loosens access;
it brings the local gate into agreement with the authority.
**Denials could not reach the client.** distribution's blobHandler.GetBlob
maps everything except ErrBlobUnknown to ErrorCodeUnknown, so a 401 raised
in the blob store left as a 500 — misreporting an auth failure as a server
fault, and giving BearerChallenge no 401 to attach WWW-Authenticate to, so
Docker was told "server error" instead of being prompted for credentials.
Clients that retry 5xx looped: 4.1s per case in the matrix, now 0.01s. The
check moves to Repository(), where an errcode.Error is passed through
verbatim by the registry app — the same mechanism a7569a7 used for
NAME_UNKNOWN. It fails open on a lookup error, since the hold is the
authority and a transient failure should not break public pulls.
Removes auth.allow_anonymous_pull. It could only ever withhold — captain.Public
is what grants — so it was a second flag for a decision the hold already owns,
and gating it appview-side was never the intent. Layer bytes 307 straight to
S3, so the appview is not even in the path whose cost might have justified an
operator-side lever.
Tests: TestAuthMatrix only ever ran against a public hold, and its pull cases
never fetched a layer — crane.Pull is lazy and img.Digest() needs only the
manifest, which ATCR serves from the user's PDS where it is world-readable, so
no pull row in the matrix touched blob authorization at all. Pulls now
materialize layer bytes, and testharness.WithPrivateHold plus
TestAuthMatrixPrivateHold cover public:false + allow_all_crew:true — the
production shape, where anyone with an account pulls and pushes and anonymous
gets nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Credential-less pulls of public images. /auth/token issues a pull-only
token with an empty subject when no Basic auth is present; the
destination hold still enforces captain.Public, and push or delete always
challenges.
- token.IsPullOnlyScope and AuthMethodAnonymous;
Handler.issueAnonymousToken skips the authorizer gate and the
service-auth pre-mint, since there is no identity to reconcile and no
AppView-to-hold service token to bind. The token is still stamped
with the resolved registry domain, so anonymous pull works on
secondary front doors whose access controller demands their own
audience.
- auth.allow_anonymous_pull (default true) turns it fully off, restoring
the previous always-challenge behavior. Mirrored into the deploy
template, since the default means existing deploys pick this up.
- RegistryContext.Anonymous is plumbed from the middleware.
- ProxyBlobStore sends no Authorization header when the service token is
empty, and returns 401 rather than 403 for anonymous denials so Docker
prompts for credentials, including when a stale captain cache lets the
request through and the hold says private.
- BearerChallenge wraps the /v2/ subtree so a 401 raised deep in the
stack via errcode.ServeJSON still carries WWW-Authenticate.
Distribution's own scoped challenges are left alone.
IsPullOnlyScope allowlists the pull action instead of denylisting push and
delete. Distribution's actionSet.contains treats "*" as *every* action, so
a scope of `repository:victim/img:*` names neither denied string and would
have handed an unauthenticated caller a token valid for push and delete on
someone else's repository — clearing the authgate entirely, since anonymous
tokens deliberately skip it. Writes would still have failed further down
(no PDS credential), but the gate itself was bypassable. Now every
requested action must be exactly "pull". Covered by new claims tests.
Unresolvable identities return NAME_UNKNOWN instead of a bare error that
distribution renders as 500. This path was previously unreachable without
credentials; anonymous pull opens it to the internet, and a 5xx on
arbitrary input both misreports a bad request as a server fault and sends
clients that retry 5xx into a retry loop. That loop was real: in the auth
matrix, regclient spent 83s on a single case before this fix, and the
suite now runs in 5s.
Stat preserves an authorization verdict from getPresignedURL rather than
flattening it to ErrBlobUnknown. Distribution calls Stat before ServeBlob
on GET and HEAD, so without this an anonymous pull from a private hold
answered 404 and BearerChallenge had no 401 to annotate — the 401 path
above could never actually reach a client.
The auth matrix is updated to match: anonymous pull of the seeded public
repo now succeeds, anonymous push is denied against a real identity's
namespace (rather than an unresolvable one, which was testing name
resolution rather than authorization), and a new case pins the
NAME_UNKNOWN behavior for an unknown identity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1. Multiple registry domains + per-user domain preference
The biggest feature. The appview can serve several registry domains (e.g. buoy.cr, atcr.io),
and users can now pick which one shows up in their pull/push commands.
- Lexicon/record: adds registryDomain (and documents ociClient)
to the sailor profile (lexicons/.../profile.json, pkg/atproto/lexicon.go).
- DB: new registry_domain column on users (schema.sql + migration 0027),
with GetUserByDID/Handle reads, UpdateUserRegistryDomain writer,
and Jetstream caching it on profile updates (writes unconditionally so clearing propagates).
- UI/handlers: new UpdateRegistryDomainHandler + /api/profile/registry-domain route,
a <select> in the user settings panel (only shown when >1 domain configured), and resolveRegistryURL()
which falls back to the primary domain if the user's pref is stale/removed. Tests added for all of it.
2. default_hold_did removed → first managed_holds entry is the default
Consolidates two overlapping config fields into one. ServerConfig.DefaultHoldDID is gone;
PrimaryHoldDID() now returns managed_holds[0]. managed_holds is now REQUIRED.
Updated in config, validation, server wiring, test harness, example YAML, and the deploy template.
3. Admin long-running operations → generic background-job framework
New pkg/hold/admin/jobs.go introduces a reusable startJob/jobRegistry pattern
(a detached context.Background() job + a /admin/api/jobs/{key}/status polling endpoint).
This replaces the bespoke scan-backfill goroutine state machine, and now also wraps crew tier remap and crew import
all three previously looped synchronously on the request context and got 504'd/cancelled mid-run by the reverse proxy.
Forms switched from POST-redirect to htmx fragments (job_progress.html, job_result.html, crew_import_results.html)
the old crew_import_results.html page and scan_backfill_progress.html partial were deleted.
This is also captured as a new rule in CLAUDE.md.
4. Cascade-delete manifest on last-tag deletion
DeleteTagHandler now, after removing the last tag pointing to a digest, cascade-deletes the manifest itself
(PDS + DB + hold blob purge) — but only if it's not a child of a manifest list (multi-arch parent).
New GetTagDigest and ShouldCascadeDeleteManifest queries back it, plus cascade_delete_test.go.
Also switches tag rkey computation to the atproto.RepositoryTagToRKey helper.
5. Billing simplification
Drops the OwnerBadge config option (hold-owner supporter badge).
The user-profile template no longer special-cases an "owner" badge value (only "Captain").
Example tiers renamed to the nautical scheme (deckhand/bosun/quartermaster).
6. Build/deploy: go generate always runs via Make
make generate is now a phony target that always runs go generate ./... (regenerating cbor_gen, icon sprites, etc.),
and build-trixie depends on it. The deploy tooling (provision.go/update.go)
drops its own runGenerate calls since the Makefile handles it.
7. New cmd/firehose-tap tool (untracked)
A standalone CLI that subscribes to a com.atproto.sync.subscribeRepos endpoint and pretty-prints events,
with emphasis on Sync 1.1 compliance fields (per-op prev CIDs, commit prevData) and a --validate CI mode.
Fits with the recent "more sync1.1 compliant" commit.
did:plc Identity Support (pkg/hold/pds/did.go, pkg/hold/config.go, pkg/hold/server.go)
The big feature — holds can now use did:plc identities instead of only did:web. This adds:
- LoadOrCreateDID() — resolves hold DID by priority: config DID > did.txt on disk > create new
- CreatePLCIdentity() — builds a genesis operation, signs with rotation key, submits to PLC directory
- EnsurePLCCurrent() — on boot, compares local signing key + URL against PLC directory and auto-updates if they've drifted (requires rotation key)
- New config fields: did_method (web/plc), did, plc_directory_url, rotation_key_path
- GenerateDIDDocument() now uses the stored DID instead of always deriving did:web from URL
- NewHoldServer wired up to call LoadOrCreateDID instead of GenerateDIDFromURL
CAR Export/Import (pkg/hold/pds/export.go, pkg/hold/pds/import.go, cmd/hold/repo.go)
New CLI subcommands for repo backup/restore:
- atcr-hold repo export — streams the hold's repo as a CAR file to stdout
- atcr-hold repo import <file>... — reads CAR files, upserts all records in a single atomic commit. Uses a bulkImportRecords method that opens a delta session, checks each record for
create vs update, commits once, and fires repo events.
- openHoldPDS() helper to spin up a HoldPDS from config for offline CLI operations
Admin UI Fixes (pkg/hold/admin/)
- Logout changed from GET to POST — nav template now uses a <form method=POST> instead of an <a> link (prevents CSRF on logout)
- Removed return_to parameter from login flow — simplified redirect logic, auth middleware now redirects to /admin/auth/login without query params
Config/Deploy
- config-hold.example.yaml and deploy/upcloud/configs/hold.yaml.tmpl updated with the four new did:plc config fields
- go.mod / go.sum — added github.com/did-method-plc/go-didplc dependency
1. Removing distribution/distribution from the Hold Service (biggest change)
The hold service previously used distribution's StorageDriver interface for all blob operations. This replaces it with direct AWS SDK v2 calls through ATCR's own pkg/s3.S3Service:
- New S3Service methods: Stat(), PutBytes(), Move(), Delete(), WalkBlobs(), ListPrefix() added to pkg/s3/types.go
- Pull zone fix: Presigned URLs are now generated against the real S3 endpoint, then the host is swapped to the CDN URL post-signing (previously the CDN URL was set as the endpoint, which
broke SigV4 signatures)
- All hold subsystems migrated: GC, OCI uploads, XRPC handlers, profile uploads, scan broadcaster, manifest posts — all now use *s3.S3Service instead of storagedriver.StorageDriver
- Config simplified: Removed configuration.Storage type and buildStorageConfigFromFields(); replaced with a simple S3Params() method
- Mock expanded: MockS3Client gains an in-memory object store + 5 new methods, replacing duplicate mockStorageDriver implementations in tests (~160 lines deleted from each test file)
2. Vulnerability Scan UI in AppView (new feature)
Displays scan results from the hold's PDS on the repository page:
- New lexicon: io/atcr/hold/scan.json with vulnReportBlob field for storing full Grype reports
- Two new HTMX endpoints: /api/scan-result (badge) and /api/vuln-details (modal with CVE table)
- New templates: vuln-badge.html (severity count chips) and vuln-details.html (full CVE table with NVD/GHSA links)
- Repository page: Lazy-loads scan badges per manifest via HTMX
- Tests: ~590 lines of test coverage for both handlers
3. S3 Diagnostic Tool
New cmd/s3-test/main.go (418 lines) — tests S3 connectivity with both SDK v1 and v2, including presigned URL generation, pull zone host swapping, and verbose signing debug output.
4. Deployment Tooling
- New syncServiceUnit() for comparing/updating systemd units on servers
- Update command now syncs config keys (adds missing keys from template) and service units with daemon-reload
5. DB Migration
0011_fix_captain_successor_column.yaml — rebuilds hold_captain_records to add the successor column that was missed in a previous migration.
6. Documentation
- APPVIEW-UI-FUTURE.md rewritten as a status-tracked feature inventory
- DISTRIBUTION.md renamed to CREDENTIAL_HELPER.md
- New REMOVING_DISTRIBUTION.md — 480-line analysis of fully removing distribution from the appview side
7. go.mod
aws-sdk-go v1 moved from indirect to direct (needed by cmd/s3-test).
Evan JarrettandClaude Fable 5.1