at-container-registry/docs/HORIZONTAL_SCALING.md

Horizontally scaling the AppView

Status of the work to make the AppView safe to run as N instances, and what is left before the database can move to local-write (Turso-style) embedded replicas.

The thing that decides everything else: most tables are derived

The AppView database is mostly a cache of ATProto records. Jetstream and the backfill rebuild it from users' PDSes and from hold services. Losing a derived table costs a re-crawl, not data.

A small set of tables is authoritative: nothing upstream can rebuild them, so staleness or loss is real loss. Almost every remaining scaling concern lives in that set, and the derived tables can mostly be ignored.

Derived — rebuilt by jetstream/backfill

Table	Source record
manifests, layers, manifest_references	io.atcr.manifest
tags	io.atcr.tag
stars	io.atcr.sailor.star
repo_pages	io.atcr.repo.page
repository_annotations	annotations on io.atcr.manifest
repository_stats, repository_stats_daily	io.atcr.hold.stats
hold_captain_records	io.atcr.hold.captain
hold_crew_members	io.atcr.hold.crew
scans	io.atcr.hold.scan
users (all but last_seen)	DID resolution, app.bsky.actor.profile, io.atcr.sailor.profile

Stale reads here are self-correcting. A user who pushes an image and does not see it for a few seconds is a cosmetic problem; the next backfill fixes any divergence permanently.

Authoritative — nothing upstream can rebuild these

Table	Cost if lost or read stale
crypto_keys	Catastrophic. Every registry JWT and OAuth client assertion becomes unverifiable.
oauth_sessions	Every user must re-authenticate. Refresh tokens cannot be recovered.
ui_sessions	Users logged out.
devices	Every registered device must be re-enrolled; the secret is not recoverable.
pending_device_auth	In-flight device logins fail.
webhooks	User-created configuration, silently gone.
stripe_processed_events	Idempotency ledger. Losing it means reprocessing Stripe events.
schema_migrations	Migrations re-run against a database that already has them.
advisor_suggestions	Regenerable, at AI cost.
users.last_seen	The only non-derived column in an otherwise derived table.

Self-healing, so effectively free to lose: instance_leases, hold_crew_approvals, hold_crew_denials, jetstream_cursor (costs a re-crawl), labeler_cursor + taken_down_subjects (replayable from the labeler from cursor 0).

users is fully derived, including preferences

oci_client and registry_domain look local but are not: both are fields on the io.atcr.sailor.profile record. The settings form writes them to the user's PDS and the local columns are a cache, refreshed by ProcessSailorProfile. Same for default_hold_did. So the whole table can be rebuilt, preferences included.

last_seen is the exception, and it is not derived from anything — see below.

What is done

Running N instances against one shared database is safe now.

• instance_leases + pkg/appview/leases. Exactly one instance runs the Jetstream consumer, backfill, labeler subscriber, cleanup sweep and billing tier refresh. The consumer in particular must be a singleton: StatsCache is per-process in-memory state whose aggregate is written to repository_stats as an absolute value, so two consumers overwrite each other with partial sums, and every webhook fires twice.
• OAuth session compare-and-swap. Refresh tokens rotate on use, and the per-DID mutex that serialized refreshes is in-process only. A second instance refreshing the same account got invalid_grant and deleted the session out from under the user. Writes now CAS on oauth_sessions.rev, and the delete path checks whether the revision moved before destroying anything.
• Atomic crew denial counter. Was a read-modify-write; concurrent denials lost increments and the backoff escalated slower than configured.
• crypto_keys first-writer-wins. Two instances booting against a fresh database both generated a key and the loser kept its own in memory.
• Denial cache no longer wiped on every boot. DELETE FROM hold_crew_denials ran unconditionally at startup, so a rolling deploy wiped the shared table once per instance.
• Node-independent keys. tags.id dropped; manifests.id replaced by manifest_key, derived from (did, repository, digest). No rowid is allocated by a node any more.
• Schema drift is checked, both as a test (schema.sql vs the migrations) and as a warning at boot.

What is left: read-after-write under local-write replicas

None of the following is a problem today. With write-forwarding replicas every write goes to one primary, so all instances read a single consistent state. They become problems only if the database moves to local-write replicas, where each node writes locally and reconciles afterwards.

Given the derived/authoritative split, the list is short.

1. Session and device flows break visibly

These are authoritative and read immediately after write, by a different instance than the one that wrote:

• ui_sessions — log in on instance A, the next request is routed to B, B does not have the session yet, user appears logged out.
• pending_device_auth — A creates the pending row, the user approves on B, the CLI polls C. If the poll interval is shorter than the sync interval the CLI reports "still pending" after approval already happened, and may time out.
• devices — enrol on A, first push authenticates against B.

These need read-through-to-primary (or a forced sync) on the specific endpoints, not a general consistency guarantee. The set of endpoints is small: the OAuth callback, the device-code poll, and device authentication.

2. The OAuth CAS stops being a CAS

oauth_sessions.rev compare-and-swap assumes the UPDATE ... WHERE rev = ? either wins or loses against one authoritative row. Under local writes both nodes' updates succeed locally and conflict at reconciliation, where last-writer wins by default — exactly the clobber the CAS exists to prevent.

This is the one place where local-write replication is genuinely incompatible with the current design rather than merely inconvenient. Options: keep OAuth sessions on a single-writer store, or move the per-DID lock to something with a real serialization point.

3. stripe_processed_events needs a real barrier

The whole point of the table is that an event is processed exactly once. Two nodes handling a redelivery concurrently would both find the row absent locally. Billing is behind a build tag and low volume, so pinning webhook handling to one instance (a lease) is likely simpler than making the ledger conflict-free.

4. Write amplification on the hot path

Not correctness, cost. Both are cheap to fix and worth doing before any remote primary carries production traffic:

• UpdateUserLastSeen ran per Jetstream event for cached users. Now throttled to once per five minutes per user.
• DeviceStore.UpdateLastUsed ran per /auth/token call, i.e. per docker push/pull. Now throttled the same way.

last_seen means "this user did something recently"

It is the one column in users that nothing upstream can rebuild, and it was being written by the wrong things.

The backfill walks every historical record in the network. Stamping last_seen there recorded when the backfill ran, for every user at once, on every run, which destroys the only signal the column carries. It is now written on the two paths that represent real activity: an interactive login, and a live commit event observed on the firehose (which does mean the user just wrote a record).

Anyone computing MAU from this column should know it was unreliable for every run before this change.

5. Not a problem, contrary to earlier suspicion

The second ?mode=ro connection in readonly.go is fine in local-only mode: verified that a write through the read-write handle is immediately visible to the read-only one. Under embedded replicas it reads a file the replica connector is syncing beneath it, which has not been verified against a real remote, but the staleness that implies is already the documented expectation for that handle.

The labeler's labels.id is not the same problem

pkg/labeler still uses INTEGER PRIMARY KEY AUTOINCREMENT, and should.

That id is the sequence number of the com.atproto.label.subscribeLabels stream. Consumers use it as a resumption cursor (GetLabelsSince is WHERE id > ? ORDER BY id ASC), and LatestSeq is MAX(id). Label negation ordering also depends on it (l2.id > l1.id decides which label supersedes which). A protocol stream sequence must be monotonic and totally ordered, which by definition requires a single allocator. A derived key would have no ordering at all, so the trick used for manifests does not transfer.

That is not a scaling defect, because a labeler is a single logical publisher. The right shape is one writer with read replicas, not N writers. It is also a separate service with its own database and its own data_dir, so none of the AppView's storage decisions reach it.

The one thing worth knowing: pkg/labeler/config.go exposes LibsqlSyncURL, so the labeler can be run as an embedded replica. If that ever became a local-write replica with two instances creating labels, both would allocate the same sequence number and consumers would silently miss labels — no error, just a gap where a takedown should have been. If the labeler ever needs HA, it needs a leader election like the AppView's, not a cleverer key.

Recommended order

1. Throttle the two hot-path writes (§4). Useful now, independent of everything.
2. Decide the sync model. Under write-forwarding, nothing else here is required.
3. If moving to local-write: fix the session and device flows (§1), then resolve OAuth sessions and the Stripe ledger (§2, §3), which may mean keeping those tables on a single-writer store rather than making them conflict-free.