at-container-registry/docs/HORIZONTAL_SCALING.md

# Horizontally scaling the AppView

Status of the work to make the AppView safe to run as N instances, and what is
left before the database can move to local-write (Turso-style) embedded
replicas.

## The thing that decides everything else: most tables are derived

The AppView database is mostly a **cache of ATProto records**. Jetstream and the
backfill rebuild it from users' PDSes and from hold services. Losing a derived
table costs a re-crawl, not data.

A small set of tables is **authoritative**: nothing upstream can rebuild them,
so staleness or loss is real loss. Almost every remaining scaling concern lives
in that set, and the derived tables can mostly be ignored.

### Derived — rebuilt by jetstream/backfill

| Table | Source record |
|---|---|
| `manifests`, `layers`, `manifest_references` | `io.atcr.manifest` |
| `tags` | `io.atcr.tag` |
| `stars` | `io.atcr.sailor.star` |
| `repo_pages` | `io.atcr.repo.page` |
| `repository_annotations` | annotations on `io.atcr.manifest` |
| `repository_stats`, `repository_stats_daily` | `io.atcr.hold.stats` |
| `hold_captain_records` | `io.atcr.hold.captain` |
| `hold_crew_members` | `io.atcr.hold.crew` |
| `scans` | `io.atcr.hold.scan` |
| `users` (all but `last_seen`) | DID resolution, `app.bsky.actor.profile`, `io.atcr.sailor.profile` |

Stale reads here are self-correcting. A user who pushes an image and does not
see it for a few seconds is a cosmetic problem; the next backfill fixes any
divergence permanently.

### Authoritative — nothing upstream can rebuild these

| Table | Cost if lost or read stale |
|---|---|
| `crypto_keys` | Catastrophic. Every registry JWT and OAuth client assertion becomes unverifiable. |
| `oauth_sessions` | Every user must re-authenticate. Refresh tokens cannot be recovered. |
| `ui_sessions` | Users logged out. |
| `devices` | Every registered device must be re-enrolled; the secret is not recoverable. |
| `pending_device_auth` | In-flight device logins fail. |
| `webhooks` | User-created configuration, silently gone. |
| `stripe_processed_events` | Idempotency ledger. Losing it means reprocessing Stripe events. |
| `schema_migrations` | Migrations re-run against a database that already has them. |
| `advisor_suggestions` | Regenerable, at AI cost. |
| `users.last_seen` | The only non-derived column in an otherwise derived table. |

Self-healing, so effectively free to lose: `instance_leases`,
`hold_crew_approvals`, `hold_crew_denials`, `jetstream_cursor` (costs a
re-crawl), `labeler_cursor` + `taken_down_subjects` (replayable from the labeler
from cursor 0).

### `users` is fully derived, including preferences

`oci_client` and `registry_domain` look local but are not: both are fields on the
`io.atcr.sailor.profile` record. The settings form writes them to the user's PDS
and the local columns are a cache, refreshed by `ProcessSailorProfile`. Same for
`default_hold_did`. So the whole table can be rebuilt, preferences included.

`last_seen` is the exception, and it is not derived from anything — see below.

## What is done

Running N instances against one shared database is safe now.

- **`instance_leases` + `pkg/appview/leases`.** Exactly one instance runs the
  Jetstream consumer, backfill, labeler subscriber, cleanup sweep and billing
  tier refresh. The consumer in particular *must* be a singleton: `StatsCache` is
  per-process in-memory state whose aggregate is written to `repository_stats` as
  an absolute value, so two consumers overwrite each other with partial sums, and
  every webhook fires twice.
- **OAuth session compare-and-swap.** Refresh tokens rotate on use, and the
  per-DID mutex that serialized refreshes is in-process only. A second instance
  refreshing the same account got `invalid_grant` and deleted the session out
  from under the user. Writes now CAS on `oauth_sessions.rev`, and the delete
  path checks whether the revision moved before destroying anything.
- **Atomic crew denial counter.** Was a read-modify-write; concurrent denials
  lost increments and the backoff escalated slower than configured.
- **`crypto_keys` first-writer-wins.** Two instances booting against a fresh
  database both generated a key and the loser kept its own in memory.
- **Denial cache no longer wiped on every boot.** `DELETE FROM
  hold_crew_denials` ran unconditionally at startup, so a rolling deploy wiped
  the shared table once per instance.
- **Node-independent keys.** `tags.id` dropped; `manifests.id` replaced by
  `manifest_key`, derived from `(did, repository, digest)`. No rowid is allocated
  by a node any more.
- **Schema drift is checked**, both as a test (`schema.sql` vs the migrations)
  and as a warning at boot.

## What is left: read-after-write under local-write replicas

None of the following is a problem today. With write-forwarding replicas every
write goes to one primary, so all instances read a single consistent state.
They become problems only if the database moves to **local-write** replicas,
where each node writes locally and reconciles afterwards.

Given the derived/authoritative split, the list is short.

### 1. Session and device flows break visibly

These are authoritative and read immediately after write, by a *different*
instance than the one that wrote:

- **`ui_sessions`** — log in on instance A, the next request is routed to B, B
  does not have the session yet, user appears logged out.
- **`pending_device_auth`** — A creates the pending row, the user approves on B,
  the CLI polls C. If the poll interval is shorter than the sync interval the CLI
  reports "still pending" after approval already happened, and may time out.
- **`devices`** — enrol on A, first push authenticates against B.

These need read-through-to-primary (or a forced sync) on the specific endpoints,
not a general consistency guarantee. The set of endpoints is small: the OAuth
callback, the device-code poll, and device authentication.

### 2. The OAuth CAS stops being a CAS

`oauth_sessions.rev` compare-and-swap assumes the `UPDATE ... WHERE rev = ?`
either wins or loses against one authoritative row. Under local writes both
nodes' updates succeed locally and conflict at reconciliation, where last-writer
wins by default — exactly the clobber the CAS exists to prevent.

This is the one place where local-write replication is genuinely incompatible
with the current design rather than merely inconvenient. Options: keep OAuth
sessions on a single-writer store, or move the per-DID lock to something with a
real serialization point.

### 3. `stripe_processed_events` needs a real barrier

The whole point of the table is that an event is processed exactly once. Two
nodes handling a redelivery concurrently would both find the row absent locally.
Billing is behind a build tag and low volume, so pinning webhook handling to one
instance (a lease) is likely simpler than making the ledger conflict-free.

### 4. Write amplification on the hot path

Not correctness, cost. Both are cheap to fix and worth doing before any remote
primary carries production traffic:

- `UpdateUserLastSeen` ran per Jetstream **event** for cached users. Now
  throttled to once per five minutes per user.
- `DeviceStore.UpdateLastUsed` ran per `/auth/token` call, i.e. per docker
  push/pull. Now throttled the same way.

### `last_seen` means "this user did something recently"

It is the one column in `users` that nothing upstream can rebuild, and it was
being written by the wrong things.

The backfill walks every historical record in the network. Stamping `last_seen`
there recorded *when the backfill ran*, for every user at once, on every run,
which destroys the only signal the column carries. It is now written on the two
paths that represent real activity: an interactive login, and a live commit event
observed on the firehose (which does mean the user just wrote a record).

Anyone computing MAU from this column should know it was unreliable for every
run before this change.

### 5. Not a problem, contrary to earlier suspicion

The second `?mode=ro` connection in `readonly.go` is fine in local-only mode:
verified that a write through the read-write handle is immediately visible to the
read-only one. Under embedded replicas it reads a file the replica connector is
syncing beneath it, which has not been verified against a real remote, but the
staleness that implies is already the documented expectation for that handle.

## The labeler's `labels.id` is not the same problem

`pkg/labeler` still uses `INTEGER PRIMARY KEY AUTOINCREMENT`, and should.

That id is the **sequence number of the `com.atproto.label.subscribeLabels`
stream**. Consumers use it as a resumption cursor (`GetLabelsSince` is
`WHERE id > ? ORDER BY id ASC`), and `LatestSeq` is `MAX(id)`. Label negation
ordering also depends on it (`l2.id > l1.id` decides which label supersedes
which). A protocol stream sequence must be monotonic and totally ordered, which
by definition requires a single allocator. A derived key would have no ordering
at all, so the trick used for manifests does not transfer.

That is not a scaling defect, because a labeler **is** a single logical
publisher. The right shape is one writer with read replicas, not N writers. It
is also a separate service with its own database and its own `data_dir`, so none
of the AppView's storage decisions reach it.

The one thing worth knowing: `pkg/labeler/config.go` exposes `LibsqlSyncURL`, so
the labeler *can* be run as an embedded replica. If that ever became a
local-write replica with two instances creating labels, both would allocate the
same sequence number and consumers would silently miss labels — no error, just a
gap where a takedown should have been. If the labeler ever needs HA, it needs a
leader election like the AppView's, not a cleverer key.

## Recommended order

1. Throttle the two hot-path writes (§4). Useful now, independent of everything.
2. Decide the sync model. Under write-forwarding, nothing else here is required.
3. If moving to local-write: fix the session and device flows (§1), then resolve
   OAuth sessions and the Stripe ledger (§2, §3), which may mean keeping those
   tables on a single-writer store rather than making them conflict-free.