Two corrections to the HTTP/2 change, both found by running it.
UpCloud refuses http2_enabled on a backend that is not TLS:
invalid_params_properties.http2_enabled='Tls must be enabled to enable HTTP2.'
This load balancer reaches both origins over the private network in cleartext,
so the option is not available here at all, and asking for it failed the whole
provision run with a 400 after the frontend had already been changed. Backend
HTTP/2 is removed rather than made conditional: it would need TLS terminated at
the origins, which is a much larger change than it earns. The frontend is the
leg that mattered, and it is enabled and confirmed live — all three domains now
negotiate h2 by ALPN.
The second correction is more interesting: enabling HTTP/2 did not fix the
symptom it was aimed at, and briefly made it look worse. With the browser no
longer rationing itself to ~6 connections, every row of the crew tab now reaches
the server at once, and the 504s went from a flat 10s to a climbing ladder:
[HTTP/2 504 10026ms] ... [HTTP/2 504 15230ms] ... [HTTP/2 504 25976ms]
HTTP/1.1 had been hiding a server-side limit by throttling the client. The queue
moved; it did not disappear. What is actually slow is resolveHandle, which calls
live identity resolution per row with no bound of its own, so a DID whose
resolution hangs — a did:web on a host that stopped answering, a PDS that
accepts and then stalls — holds its request open until the proxy gives up. The
crew tab issues one request per member, so a hold with hundreds of crew gets
hundreds of chances to hit one.
resolveHandle now bounds each lookup at 2s. A handle is decoration on a row
whose DID is already rendered beside it, so waiting seconds for one and failing
the row when it does not arrive trades something load-bearing for something
cosmetic. Two seconds is far above a warm cached lookup and far below the
proxy's 10s cut, so a slow DID costs its own row a handle and costs the page
nothing.
The quota lookup on the same handler was the other suspect and was measured out:
records(collection, did) is indexed and the aggregation returns immediately on
the production database.
This does not remove the N+1 — the tab still issues one request per member, and
a batch endpoint is the real fix. It removes the failure.
Verified: the guard is covered by a test using a directory that hangs until its
context is cancelled, asserting both that the call returns near the bound and
that it did not return suspiciously early. make lint 0 issues, make test green
across 44 packages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AV6Mk2AgghFsNo4HWQEaBV
build-trixie hardcoded `-tags billing`, so the only way to build a non-billing
appview was to bypass the Makefile and drive the trixie container by hand. Both
production deploys of this deployment did exactly that, and the workspace notes
record it as "the deploy tool cannot reproduce this build" — a tool that cannot
produce the artifact you actually ship is a tool nobody can safely use.
Worse, the failure is silent and one-directional. Nothing about a billing binary
looks different: same name, same version stamp, same vcs.revision. A deploy that
forgot the manual path would quietly switch billing on for a deployment whose
operator had chosen to run without it, exposing a live Stripe webhook endpoint
and a paid tier ladder on a service configured for neither. Production has been
running a non-billing appview since May precisely because someone did not use
`make build-trixie`.
Billing is now opt-in and off by default:
make build-trixie # no billing
make build-trixie BILLING=1 # billing
deploy/upcloud update appview # no billing
deploy/upcloud update appview --with-billing
Only the appview is affected; hold, scanner, labeler and the credential helper
never reference pkg/billing.
The flag alone is not enough, so verifyAppviewBilling reads the built binary
before anything is uploaded and refuses to ship a mismatch. It observes rather
than trusts, because the flag and the artifact can disagree for reasons the flag
cannot see: a stale bin/atcr-appview from an earlier build, a Makefile that
hardcodes the tag, a builder image that ignored it. The Stripe SDK links only
under the tag, so its symbols are a direct measurement — verified as a
discriminator here: 2338 stripe-go strings with the tag, 0 without.
A missing binary is an error rather than an absence of symbols, so a deploy
cannot proceed on a file that was never built by reading it as "no billing".
Verified: the default build-trixie output has 0 stripe-go symbols and does carry
the "Billing is not enabled on this deployment" stub. Guard covered both
directions by test, plus the missing-binary case. make lint 0 issues across
root, deploy and credential-helper; make test green across 44 packages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AV6Mk2AgghFsNo4HWQEaBV
`go fix` carries the modernize analyzers now, and the tree had drifted behind
them. This is the mechanical result, reviewed rather than trusted: the tool is
capable of rewriting code into something that no longer tests or does what it
did, so every non-test change was read individually and the concurrency-bearing
packages were re-run under -race.
Production code, four changes, all semantics-preserving:
- leases/manager.go: wg.Add(1) + go + defer wg.Done() becomes wg.Go. The
comment above that function turns on Add happening before the goroutine
starts, so that a Wait cannot return before the worker has run. wg.Go does
the Add synchronously on the calling goroutine, so the invariant it
describes still holds.
- auth/token/handler.go: strings.Fields -> strings.FieldsSeq, same splitting,
iterated rather than allocated.
- hold/gc/gc.go: a hand-written map copy -> maps.Copy.
- hold/pds/scan_broadcaster.go: three-clause loop -> range over int.
The rest are tests. The one worth naming is carstore_contention_test.go, where a
careless rewrite could have quietly stopped exercising contention: go fix
converted the reader and side-table goroutines to loopWG.Go but correctly
declined to touch the writer loop, which passes its index as a parameter. The
writer/reader/side-table shape and the stop channel are unchanged, so the test
still contends over the same carstore transactions.
Verified: go build for hold and appview, `make lint` 0 issues, the deploy and
credential-helper modules 0 issues, `make test` green across all 43 packages,
and -race green on leases, hold/pds, hold/gc and auth/token. The scanner module's
two lint findings are unchanged from HEAD and are in files go fix never touched.
Kept separate from the HTTP/2 commit so that one stays readable, and so this can
be reverted on its own if a modernization turns out to matter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TA9D4DjaLZTvzQ7dJbu4eg
The admin crew tab renders one row per crew member and gives each row its own
hx-get, so opening it on a hold with 551 crew issues 551 requests. Over
HTTP/1.1 a browser runs at most ~6 per origin, so they queue six at a time and
every other request to the same host queues behind them — which is why loading
the relay page stalls while the crew rows are still resolving, and why the rows
that lose the race come back as "Server error" toasts. The 504 behind that toast
is the load balancer's, not the hold's: the hold logs those requests as 200.
HTTP/2 multiplexes them over one connection and the queue disappears. It does
not make the slow rows fast — that is a separate fix to the per-row identity
lookup — but it stops one slow surface from blocking the rest of the panel.
Two halves, because neither works alone.
The load balancer terminates TLS and speaks cleartext to the origin, so ALPN
never runs on the backend leg and net/http can only answer HTTP/1.1 there. Both
servers now wrap their handler in h2c. The wrapper is opt-in per connection: it
upgrades only for a client sending the h2c preface or "Upgrade: h2c", and passes
everything else through untouched, so an HTTP/1.1 WebSocket upgrade is
unaffected. Verified both directions against this wiring — HTTP/1.1 for a plain
client, HTTP/2.0 with --http2-prior-knowledge.
The frontend's http2_enabled was never set, so it sat at the UpCloud default of
off. That is the half the browser actually sees. timeout_client is now stated
explicitly at its current 10s rather than left implicit: it is the boundary that
produces the 504s above, so it belongs somewhere visible. It is deliberately
unchanged — raising it without fixing the slow lookup would only make a stalled
row stall longer.
The hold's *backend* stays on HTTP/1.1. It serves subscribeRepos over WebSocket
to external relays and to the scanner, and WebSocket over HTTP/2 needs the RFC
8441 Extended CONNECT that Go's http2 server does not implement for Upgrade:.
Routing that backend over h2 would break the firehose. The appview accepts no
inbound WebSocket and has no such constraint. Both origins carry h2c regardless,
so enabling it for the hold later is a config change, not a code change.
createLoadBalancer only runs when there is no LB yet, so properties set there
would reach a new deployment and never an existing one. ensureLBHTTP2 reconciles
them onto an LB that already exists, following ensureLBForwardedHeaders: read
what is there, change only what differs, report what it did, and no-op on a
second run. Backend modifies carry the existing health check back, since
Properties replaces the object wholesale.
Also gofmt: provision.go was not gofmt-clean at HEAD, unrelated to this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TA9D4DjaLZTvzQ7dJbu4eg
1. Multiple registry domains + per-user domain preference
The biggest feature. The appview can serve several registry domains (e.g. buoy.cr, atcr.io),
and users can now pick which one shows up in their pull/push commands.
- Lexicon/record: adds registryDomain (and documents ociClient)
to the sailor profile (lexicons/.../profile.json, pkg/atproto/lexicon.go).
- DB: new registry_domain column on users (schema.sql + migration 0027),
with GetUserByDID/Handle reads, UpdateUserRegistryDomain writer,
and Jetstream caching it on profile updates (writes unconditionally so clearing propagates).
- UI/handlers: new UpdateRegistryDomainHandler + /api/profile/registry-domain route,
a <select> in the user settings panel (only shown when >1 domain configured), and resolveRegistryURL()
which falls back to the primary domain if the user's pref is stale/removed. Tests added for all of it.
2. default_hold_did removed → first managed_holds entry is the default
Consolidates two overlapping config fields into one. ServerConfig.DefaultHoldDID is gone;
PrimaryHoldDID() now returns managed_holds[0]. managed_holds is now REQUIRED.
Updated in config, validation, server wiring, test harness, example YAML, and the deploy template.
3. Admin long-running operations → generic background-job framework
New pkg/hold/admin/jobs.go introduces a reusable startJob/jobRegistry pattern
(a detached context.Background() job + a /admin/api/jobs/{key}/status polling endpoint).
This replaces the bespoke scan-backfill goroutine state machine, and now also wraps crew tier remap and crew import
all three previously looped synchronously on the request context and got 504'd/cancelled mid-run by the reverse proxy.
Forms switched from POST-redirect to htmx fragments (job_progress.html, job_result.html, crew_import_results.html)
the old crew_import_results.html page and scan_backfill_progress.html partial were deleted.
This is also captured as a new rule in CLAUDE.md.
4. Cascade-delete manifest on last-tag deletion
DeleteTagHandler now, after removing the last tag pointing to a digest, cascade-deletes the manifest itself
(PDS + DB + hold blob purge) — but only if it's not a child of a manifest list (multi-arch parent).
New GetTagDigest and ShouldCascadeDeleteManifest queries back it, plus cascade_delete_test.go.
Also switches tag rkey computation to the atproto.RepositoryTagToRKey helper.
5. Billing simplification
Drops the OwnerBadge config option (hold-owner supporter badge).
The user-profile template no longer special-cases an "owner" badge value (only "Captain").
Example tiers renamed to the nautical scheme (deckhand/bosun/quartermaster).
6. Build/deploy: go generate always runs via Make
make generate is now a phony target that always runs go generate ./... (regenerating cbor_gen, icon sprites, etc.),
and build-trixie depends on it. The deploy tooling (provision.go/update.go)
drops its own runGenerate calls since the Makefile handles it.
7. New cmd/firehose-tap tool (untracked)
A standalone CLI that subscribes to a com.atproto.sync.subscribeRepos endpoint and pretty-prints events,
with emphasis on Sync 1.1 compliance fields (per-op prev CIDs, commit prevData) and a --validate CI mode.
Fits with the recent "more sync1.1 compliant" commit.
Evan JarrettandClaude Opus 5