Same defect as the OCI path, two more places. The io.atcr.manifest record is
keyed by digest alone, so one record backs every repository of a DID holding
identical content, and both of these deleted it without asking whether another
repository still wants it — then purged the layers on the hold.
DeleteManifestHandler already removes this repository's tags before deleting
the record, so a tag remaining at that point can only belong to another
repository. It now checks IsManifestTaggedAnyRepo there and keeps the shared
record when one does, reporting sharedRecordKept so the caller can tell the
difference between "deleted" and "deliberately left alone".
DeleteUntaggedManifestsHandler is the subtler one. Its digest list comes from
GetAllUntaggedManifestDigests, whose tag join is scoped to one repository
(m.repository = t.repository), so a digest tagged only in a DIFFERENT
repository is reported as untagged and swept. The query is a reasonable
per-repository view and a dangerous delete list; the guard goes at the delete,
not in the query, matching how DeleteTagHandler already works. Skips are
counted separately from failures, because a skip is the guard working and
folding it into "failed" would make a correct run look broken.
Both fail closed. Leaving a manifest behind is recoverable; deleting one
another repository is still serving is not.
The new db test pins both halves of the interaction: that the query really does
report a cross-repo-tagged digest as untagged, so a change there is noticed,
and that IsManifestTaggedAnyRepo answers DID-wide, which is the thing actually
standing between the sweep and another repo's live image.
DeleteTagHandler needed no change — 2580dcd already routed it through
ShouldCascadeDeleteManifest, which is where the correct policy was written down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeaUS5AFPX9gqCahoLRMRh
TestDeviceStore_ValidateDoesNotScanIndexedRows is documented as "the
regression guard for the O(n) bcrypt scan", but it does not observe whether a
scan happened. It asserts that every row is indexed and that an unknown secret
errors, and both hold with or without the fix. Deleting the
`WHERE secret_lookup IS NULL` filter from the fallback query — which is the
defect 08121f3 removed — leaves it green.
That matters more here than elsewhere in the batch. No backfill of the
production table is possible (plaintext is not recoverable from bcrypt), so
all 244 devices are legacy on day one and migrate lazily on first auth. A
regression on this path locks out every existing user while new devices keep
working, which is the failure mode least likely to show up in a smoke test.
The scan is only observable in time, so the new guard makes one comparison
expensive (bcrypt cost 13, ~300ms here) and asserts a deadline. Six seeded
rows cost ~1.8s to scan and ~0ms to skip. Only one hash is generated: the
others are copies with a mutated final byte, which bcrypt still runs the full
key derivation over before rejecting, so setup stays at a single 300ms hash.
Mutation-verified in a worktree by removing the filter: the old guard reports
ok, the new one fails at 1.84s against a 400ms budget.
Both halves are asserted — an unknown secret (the old code's worst case, where
nothing matches and every row is compared) and a known indexed one — because
only the first catches the missing filter and only the second catches the
index being bypassed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeaUS5AFPX9gqCahoLRMRh
Only one oauth_sessions row is kept per account, so signing in again — on a
second device, or simply a second time — replaces it and leaves every earlier
ui_sessions row pointing at an oauth_session_id that no longer exists. Get
checked only expiry, so those still read back as usable.
Found on a live appview: four ui_sessions rows, three orphaned, and requesting
/settings/user with an orphaned cookie returned 200 with the account's handle
rendered throughout, where an anonymous request gets a 302. The browser looks
signed in while the credential behind it is gone, so every PDS-backed action
fails against a UI insisting the session is fine. It now fails closed and sends
the user back through login.
Get also never checked ownership. oauth_sessions is unique on
(account_did, session_id), so the existence check is scoped by both; matching
session_id alone would let one account's live OAuth session validate another
account's dangling reference. That has its own test.
An empty oauth_session_id stays valid, since Create makes sessions that never
had one, and a test pins that so the check cannot start rejecting them.
TestSessionStore_CreateWithOAuth referenced an OAuth session it never inserted,
which is an orphan by definition, so it now creates the row. Its intent was
that CreateWithOAuth persists the ID; it relied on the orphan behaviour only
incidentally. Its not-found branch used t.Error and then dereferenced the nil
session, so that is now t.Fatal.
Pre-existing at efabb677 rather than introduced by this range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
last_seen means "this user did something recently". The backfill walks every
historical record in the network, so stamping it there recorded when the
backfill ran, not when the user was active — for every user at once, on every
run. That destroys the only signal the column carries, and it is the one column
in users that nothing upstream can rebuild.
It is now written on the two paths that represent real activity: an interactive
login, and a live commit event on the firehose, which does mean the user just
wrote a record. The backfill still corrects handle, PDS endpoint and avatar,
which is why it re-resolves rather than trusting a cache; it just no longer
claims the user was present.
UpsertUser grows an options form rather than a fourth named variant, since the
avatar and last_seen decisions are independent and all four combinations occur.
Anyone computing MAU from this column should know it was unreliable for every
backfill run before this change.
Also corrects docs/HORIZONTAL_SCALING.md, which claimed oci_client and
registry_domain were local-only preferences. They are fields on
io.atcr.sailor.profile: settings writes them to the user's PDS and
ProcessSailorProfile refreshes the local cache. users is fully derived apart
from last_seen.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Neither is a correctness problem; both are round trips on hot paths for
timestamps nothing reads at that resolution.
UpdateUserLastSeen ran per Jetstream event for cached users, so once per indexed
record. DeviceStore.UpdateLastUsed ran per /auth/token call, so once per docker
push and pull including each layer's re-auth. Cheap against a local file, a
network round trip each against a remote primary, and the second sat on the
authentication path.
Both are now throttled to once per five minutes per subject. The MAU queries and
the admin views work in hours or days, so nothing loses meaning. The throttle
state is per-process and lost on restart, costing at most one extra write per
subject per boot; only the lease holder runs the consumer, so exactly one process
is doing the first of these at a time.
UpdateLastUsed stamps the throttle before writing rather than after, so a slow or
failing write cannot let every concurrent layer upload through to pile on more of
them.
Verified by disabling the throttle: 50 back-to-back calls then rewrite the
timestamp every time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The MAX(id) replacement touched four queries that each carry their own copy of
the same CTE: SearchRepositories, GetRepoCards, GetUserRepoCards and
GetStarredRepoCards. Only the third had a test. Fixing one and missing another
would leave the UI disagreeing with itself about which manifest is current,
depending on which page you were looking at.
All four now assert that recency follows created_at rather than insert order,
and that a tie between manifests pushed in the same second resolves the same way
on every surface. Verified by regressing the CTEs back to rowid ordering: each
of the four fails independently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes the swap 0033 set up. layers and manifest_references move onto
manifest_key and manifests.id is gone, which removes the last node-allocated
identifier in the AppView schema.
Statement order in 0034 is load-bearing. With foreign keys on, DROP TABLE
performs an implicit DELETE FROM, so dropping manifests while layers still holds
an ON DELETE CASCADE reference deletes every layer row. Migration 0009 did
exactly that; it went unnoticed because the Jetstream backfill rebuilds layers
from PDS records, so the damage healed itself. PRAGMA foreign_keys is no help:
it is a no-op inside a transaction and migrations run in one. So the new
children are built pointing at manifests_new, the old children are dropped
first, and only then is the old manifests table dropped, by which point nothing
references it. Verified both behaviors before relying on them.
manifest_key is declared NOT NULL as well as PRIMARY KEY, because in SQLite a
PRIMARY KEY column still accepts NULL unless it is INTEGER PRIMARY KEY. That
constraint immediately caught four test helpers inserting manifests without one.
Five queries used MAX(id) as "the newest manifest in this repo", which I had
previously reported as absent after grepping only for ORDER BY. A derived key
has no ordering, so recency now comes from created_at with manifest_key as a
deterministic tiebreak. This is a real behavior change, and a fix: the two
disagree whenever a manifest is indexed out of order, which the backfill does
routinely, and created_at is the push time these queries always wanted. Both
directions are tested, including that ties resolve the same way every run.
InsertManifest and BatchInsertManifests no longer read anything back. The key is
derived from (did, repository, digest), so the writer knows it before the
statement runs: the select-back, its per-DID IN list, and the "manifest missing
id after batch insert" branch all go away, along with the UNIQUE-conflict
fallback that existed only to recover a rowid.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0033 added the column and left the fill to the Jetstream backfill. That is fine
for a running system and wrong for replay: a database several releases behind
runs every pending migration back-to-back at boot, long before any worker
starts. Anything built on top of manifest_key would, on that path, silently
operate on NULLs while working perfectly on a system that had been up for a
while. Establishing a migration's data precondition out-of-band means replay
cannot see it.
The runner now supports a Go step per migration version, running inside the same
transaction as that migration's SQL, after the DDL it depends on and before the
version is recorded. A version is never recorded without its Go half.
0033's step fills manifest_key for every row lacking one. It has to be Go: the
value is a truncated sha256 and SQLite has no hash builtin. It pages through the
table and writes one UPDATE ... CASE per 500 rows, because a statement per row
would be correct and unusably slow against a remote primary.
Verified by unregistering the hook: replay then leaves 3 of 3 seeded manifests
with NULL keys. With it, all three are filled, from a snapshot of the pre-0009
schema forward.
The upserts keep their "OR manifests.manifest_key IS NULL" clause. It is a
self-healing net for rows that somehow arrive without a key rather than the
mechanism anything depends on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First half of replacing manifests.id with a node-independent identity. Nothing
depends on the column yet: id is still the primary key, and layers and
manifest_references still reference it. Getting the column in place and filled
first means the eventual swap operates on data that is already complete and
already proven unique, instead of doing the fill and three table rebuilds in one
step.
The value cannot be computed by the migration. It is a truncated sha256, SQLite
has no hash builtin, and go-libsql exposes no way to register one. So the fill
uses machinery that already exists: the Jetstream backfill re-upserts every
manifest across the protocol on startup, and both upsert paths now write
manifest_key.
That only works because of one extra clause. Both upserts guard their DO UPDATE
with a WHERE that skips rows where nothing changed, which on a backfill re-run is
nearly every row, so they would have skipped the very manifests that need
filling. Adding "OR manifests.manifest_key IS NULL" is what makes an otherwise
no-op pass populate the column. Verified by removing it: the backfill then fills
zero of three manifests instead of three of three.
The index is UNIQUE even though the column is nullable. SQLite treats NULLs as
distinct, so unfilled rows coexist while every filled row is checked. That makes
production data verify the 16-byte truncation rather than us assuming it: if two
manifests ever derived the same key, it fails loudly at insert instead of
silently attaching one manifest's layers to another after the swap.
AppView logs the unfilled count at startup, since there is no single moment at
which this becomes complete and the follow-up migration is only safe at zero.
ManifestKey replaces the old fat "did|repo|digest" map key rather than sitting
beside it; they were always the same question, answered without asking the
database.
The jetstream tests hand-maintained their own CREATE TABLE statements, which is
the drift problem moved into a test: the copy had already fallen behind (it still
had tags.id) and only failed once a query touched the difference. They use
db.InitDB now, with foreign keys switched off to preserve the behavior the
hand-rolled schema had.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing joined on it. It was selected into a struct field no caller read, and
used only by DeleteTagsNotInList, which fetched surrogate ids, filtered them in
Go with a nested loop over the keep list, and issued one DELETE per row. The
natural key was already enforced by UNIQUE(did, repository, tag), so that
becomes the primary key and the column goes.
An AUTOINCREMENT rowid is allocated by whichever node performs the insert. That
is fine while every write funnels through one writer and stops being a stable
identity the moment they do not, so removing an identifier nobody used is the
cheapest way to shrink that surface before local-write replicas.
DeleteTagsNotInList now diffs against a set and deletes in batches. It still
reads the current tags first rather than issuing one NOT IN over the keep list:
that would need two placeholders per kept tag and would break past the driver's
parameter ceiling for a user with enough tags, and it cannot be chunked, because
each chunk would delete the tags every other chunk meant to keep. An explicit
delete list chunks safely.
idx_tags_did_repo is dropped rather than recreated: the new primary key indexes
(did, repository) as a prefix. It existed only because the primary key used to be
the surrogate id.
The rebuild names its columns explicitly. Column order is not guaranteed to
match between a fresh install and a migrated one, so INSERT ... SELECT * here
could write values into the wrong columns. TestMigration0032PreservesTagRows runs
the migration body against a table in the old shape and checks the contents
survive, which the schema drift test cannot: it compares shape, not data.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Refresh tokens rotate on use, and DoWithSession serializes refreshes per DID with
an in-process mutex. That is the right mechanism and it protects nothing once
there are two instances: both can refresh the same account at the same time, the
slower one presents a refresh token the auth server has already superseded, gets
invalid_grant, and isAuthError deletes the session. The user is signed out
mid-push, and the session another instance had just legitimately refreshed is
destroyed along with it.
oauth_sessions gains a rev that increments on every write. A store that has read
a session writes with a compare-and-swap against the revision it read and gets
ErrSessionRevConflict if anyone wrote first, so a stale writer can no longer
replace rotated tokens with invalidated ones. The persist callback treats that
conflict as an ordinary outcome rather than an error, since leaving the newer
state alone is exactly right.
The delete path is now guarded by the same signal. An auth error on a session
whose revision has moved since we read it means "someone else refreshed this",
not "this session is dead", so it retries once against the newer tokens instead
of deleting. Exactly once: a second failure means staleness was not the problem,
and looping would hold the per-DID lock while getting the same answer.
The guard is deliberately conservative. A store without revisions, no recorded
revision, a failed lookup, a session that is simply gone: all answer "not
advanced" and keep the previous delete-on-error behavior. Wrongly claiming a
concurrent refresh would keep a genuinely dead session alive with no way out but
waiting; wrongly missing one costs a re-login.
The sentinel lives in pkg/auth/oauth rather than next to the SQLite store,
because pkg/appview/db already imports pkg/auth/oauth and the other direction
would be an import cycle. The db package re-exports it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two instances booting against a fresh database both find no key, both generate
one, and both write. PutCryptoKey was last-writer-wins, so the loser kept its own
key in memory while the database held the other's. It then signed OAuth client
assertions with a key absent from the published JWKS, and issued registry JWTs
that did not match the certificate written to disk. Every one of them fails
verification, and nothing logs why.
PutCryptoKey now keeps the first write, and both loaders re-read afterwards and
use whatever is stored. Nothing in the codebase rotates a key through this
function, so the update arm only ever fired on the race.
Verified against the old behavior: with last-writer-wins restored, three of six
concurrent loaders returned a key that was not the one in the database.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Jetstream consumer, backfill, labeler subscriber, cleanup sweep and billing
tier refresh all started unconditionally in every process. That is correct for
one instance and wrong for two.
The consumer is the case with teeth. StatsCache is per-process in-memory state,
and the aggregate it produces is written to repository_stats as an absolute
value rather than an increment, so two consumers each hold a partial view of the
holds and each write their partial sum as though it were the whole truth,
overwriting one another indefinitely. The webhook dispatcher hangs off the same
processor, so a second consumer also doubles every delivery.
Each now runs under a named lease, so exactly one instance runs it and a
replacement takes over when that instance goes away. The health worker is
deliberately not leased: it refreshes a cache each instance needs locally, so
running it everywhere is correct.
Two structural changes came with it. The cleanup loop moved out of
InitializeDatabase, where it was a bare goroutine with no way to reach the lease
manager, into RunPeriodicCleanup called from the server. And backfill's startup
run and periodic schedule became one leased worker instead of two goroutines on
context.Background(), so shutdown actually stops a backfill in flight rather
than letting it run on against a closing database. With interval=0 that worker
holds its lease instead of returning, since releasing would let another instance
acquire and run its own startup backfill, turning "once" into "once per
instance".
Verified with two instances against one database: exactly one acquired, the
other contended without starting a worker; SIGTERM handed over in 13ms via the
release, SIGKILL handed over in ~12s via TTL expiry.
That first number only holds because of Manager.Go and Manager.Wait, which this
commit adds. The first cut used `go m.Run(...)` and cancelled the worker context
during shutdown without waiting, so the process exited before the release landed
and the lease survived to its TTL — a rolling deploy would have paused indexing
for a minute rather than a second. Nothing in the unit tests caught it; the
two-instance run did. TestWaitBlocksUntilLeaseReleased covers it now.
leases.enabled defaults to true. A single instance is unaffected, since it
always wins its own leases, while an operator who scales out without reading the
docs still gets correct behavior instead of silent stats corruption.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for running more than one AppView instance. Nothing is wired to this
yet; the next commit moves the background workers onto it.
Several workers must run on exactly one instance. The Jetstream consumer is the
sharpest case: StatsCache is per-process in-memory state, and the aggregate it
produces is written to repository_stats as an absolute value rather than an
increment, so two consumers would each hold a partial view of the holds and each
write its partial sum as the whole truth, overwriting one another indefinitely.
The webhook dispatcher hangs off the same processor, so a second consumer also
means every webhook fires twice.
Instances contend for a named lease; only the holder runs the worker. Acquire is
a single INSERT ... ON CONFLICT ... WHERE, so two instances racing for the same
expired lease cannot both win: the loser's update matches no rows. The fence
token increments on every change of custody, so a process that stalled past its
TTL discovers on its next renewal that it was superseded, rather than continuing
to act as the holder.
A renewal blackout is treated as a loss. If the database has been unreachable
for longer than the TTL, another instance is entitled to steal the lease and we
must assume it has, even though we cannot ask. Continuing to work in that state
is the one outcome the lease exists to prevent.
Clean shutdown expires the lease in place rather than deleting the row, so a
replacement starts in seconds instead of waiting out the TTL, while the fence
token survives to keep a stalled former holder from matching again.
Timestamps are Unix milliseconds, not TIMESTAMP text. libSQL normalizes
date-like TEXT on the way in, and Go's driver and CURRENT_TIMESTAMP disagree on
format, so a stored expiry and a literal would compare as strings that sort
differently. That comparison is the whole safety property, so it does not get to
be subtle. The cost is a dependency on roughly-synced clocks, the same
assumption Kubernetes leases make; keep the TTL well above any plausible skew.
The lease tests are file-backed rather than :memory:. go-libsql gives every
connection to an in-memory DSN its own private database, so with MaxOpenConns of
8 a second goroutine lands on a connection where the schema was never applied
("no such table"). Every existing test in the package is sequential and reuses
one pooled connection, which is why this has stayed invisible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Migrations can be recorded without being executed. That is not hypothetical:
migration 0009 exists to clean up after 0004, which production recorded but
never applied, leaving eleven columns behind that fresh installs never had.
Nothing reported it at the time; it surfaced later as confusing behavior.
InitDB now compares an existing database against schema.sql after migrations run
and logs one warning per difference. Fresh databases skip the check, since they
were just built from schema.sql and agree by construction.
The comparison works by applying schema.sql to a throwaway in-memory database
and introspecting that, rather than parsing the DDL. SQLite's own resolution of
types, defaults and implicit indexes is exactly what we want to compare against,
and a hand-rolled parser would drift from the engine. The introspection is
shared with TestSchemaMatchesMigrations, so the test exercises the same code
that runs at boot.
Warn-only, never fatal. A database merely ahead of or behind schema.sql is
almost always still able to serve traffic, so refusing to boot would turn a diff
that wants a corrective migration into an outage, during a deploy, which is the
worst possible moment to have one.
The README claimed new tables go in schema.sql only. That is wrong in the
direction that hurts: InitDB skips schema.sql entirely once schema_migrations
has rows, so such a table appears on fresh installs, passes every test, and is
silently absent in production. Documented the real rule along with two others
the test cannot enforce: migrations must not return rows (go-libsql rejects them
with "Execute returned rows"), and rebuild migrations must name columns
explicitly, since column order legitimately differs between fresh and upgraded
databases.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
schema.sql and migrations/ are meant to move in lockstep, and nothing checked
that they did. The cost is already on the record: migration 0009 exists only
because 0004 "was supposed to drop these columns but either failed or was only
recorded (not executed) on production", leaving production carrying eleven
columns fresh installs never had.
Build the schema both ways and compare. testdata/base_schema.sql is the shape
immediately before 0009, reconstructed by reversing 0009-0029 out of the current
schema.sql; applying it and then running every migration must land on the same
place as applying schema.sql directly.
Columns compare as a set, ignoring ordinal position. Migrations append with ADD
COLUMN while schema.sql places the same column mid-table, so manifests, users,
devices and repo_pages legitimately differ in order. Reordering them would mean
four rebuild migrations for no functional gain, and nothing in pkg/appview reads
by position (no SELECT *, no column-less INSERT ... VALUES).
Verified non-vacuous three ways: a column only in schema.sql, a column only in a
migration, and a table only in schema.sql are each caught. That last case is the
one with teeth, since InitDB skips schema.sql entirely once schema_migrations has
rows, so a table added only there never reaches an existing database.
The snapshot records versions 1-8 as applied, which is what "before 0009" means.
It also sidesteps a live rake: migration 0001's query is a bare SELECT, and
go-libsql rejects row-returning statements passed to Exec. Every real database
recorded 0001 long ago so it never fires, but a future migration opening with a
SELECT would fail the same way.
This cannot independently verify tables that appear in no migration; those are
copied from schema.sql and compare against themselves. Drift there is the
startup check's job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
io.atcr.manifest rkeys are the digest alone (digestToRKey), so a single
record backs every repository of a user holding identical content. Both
paths that cascade-delete that record checked for remaining tags scoped to
one repository, which asks the wrong question: a tag in another repo keeps
the shared record alive just as much as a tag in this one.
With me/a:v1 and me/b:v1 at the same digest, deleting me/a:v1 saw no
remaining tags in repo a, deleted the shared PDS record, and purged the
layers on the hold. me/b:v1 was left pointing at content that no longer
exists, and the firehose delete handler then cleared the rows for every
repo (DeleteManifest with an empty repository argument).
The collision predates this, but it was reachable only behind the opt-in
AutoRemoveUntagged profile flag. 1b91768 enabled OCI manifest DELETE and
made TagStore.Untag cascade unconditionally, which turned a latent metadata
collision into blob-level data loss on an ordinary skopeo/crane delete.
- cleanupUntaggedManifest no longer filters candidate tags to
rctx.Repository. The surrounding comment already noted that the tag
collection is account-wide; the filter contradicted it.
- ShouldCascadeDeleteManifest takes the tag question DID-wide via a new
IsManifestTaggedAnyRepo, and drops its now-meaningless repository
parameter.
- IsManifestTagged stays repository-scoped and keeps its caller: the
delete-manifest confirmation prompt is genuinely asking about the one
repo whose tags the user is about to remove.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two migration files shipped as version 28: 0028_add_device_secret_lookup
(08121f3) and 0028_create_stripe_processed_events (12c55ed).
runMigrations keys applied migrations by the integer parsed from the
filename and skips any version already present in schema_migrations, so
the first file to load records 28 and the second is skipped in silence —
no error, no log. loadMigrations enumerates via fs.Glob, which sorts
lexically, so add_device_secret_lookup won and stripe_processed_events
never ran.
On an existing database that leaves stripe_processed_events missing, and
StripeEventSeen then fails closed: the error wraps into ErrWebhookProcessing,
the webhook returns 500, and Stripe redelivers into the same missing table
forever. No subscription, tier, or dispute event is ever applied — defeating
the exact idempotency 12c55ed was written to add.
Fresh installs were unaffected, which is why no test caught it: they take
the applySchema path where schema.sql already has the table and both
version-28 rows are merely recorded as applied.
Renumbered to 0029 rather than renumbering the device migration, so a
database that already ran this build (28 recorded, devices.secret_lookup
present, stripe table missing) picks the migration up on next boot instead
of staying broken. Renumbering the other file would have left that database
with the table still missing and re-run its ALTER on a column that exists.
Adds two guards: one asserting migration versions are distinct, and one
exercising the upgrade path where a duplicate manifests as a missing
schema_migrations row. Both fail on a planted duplicate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The webhook limit was only checked at creation, so losing entitlement (a
hold switch or a plan downgrade) left previously-created webhooks firing
paid behavior forever.
- Dispatcher takes a WebhookLimiter, consulted on every dispatch. It
caps the list to the current allowance, keeping the oldest N to match
what the creation gate would have permitted, and masks paid trigger
bits.
- GetWebhooksForUser orders by created_at ASC, id ASC so that cap is
deterministic. ListWebhooks gets the same tiebreak: it feeds the
settings UI, and without it the list a user sees could disagree with
the one the dispatcher truncates.
- webhooks.FreeTriggerMask is shared by the creation gate and the
dispatch backstop so the two cannot drift.
Capping is logged when it actually truncates. The webhooks stay visible in
settings, so from the user's side delivery would otherwise just stop with
no signal — and the same line is the only evidence if the limiter itself
degraded, since a billing lookup failure falls back to free-tier limits
and would quietly demote a paying user mid-dispatch.
Two cost fixes, both because this puts the entitlement lookup on a hot
path it was never on before:
findCustomerByDID now consults the customer cache instead of always
issuing a Stripe customer search. GetWebhookLimits reaches it via
GetSubscriptionInfo on every delivery, so uncached it meant a
rate-limited Search API call for every push and every scan record of
every user with a webhook configured.
DispatchForQuota checks whether the user has any quota webhook at all
before fetching the allowance. The original code filtered first precisely
so the common path (no quota webhooks) did no work; taking the allowance
up front would have spent the expensive lookup on every push. The cap
itself is still computed over the full list, since the count limit spans
all webhook types.
Note DeliverTest is deliberately not capped: it is an explicit,
user-initiated "send test" from the settings page, not automatic delivery.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Entitlements were keyed on the Stripe subscription alone, so a subscriber
who switched to a self-hosted hold kept paying for features the appview
cannot deliver, and could still reach checkout.
- billing.ActiveHoldChecker and Manager.onManagedHold gate every
entitlement. An empty default hold counts as managed: the user has no
explicit preference and falls back to the operator's primary managed
hold.
- The checker reads the primary DB, not the read replica. A hold switch
writes default_hold_did to the primary, and replica lag would keep
paid features alive after a switch away.
- db.GetUserDefaultHoldDID is the clean default-hold signal, unlike
GetUserHoldDID which falls back to a manifest hold_endpoint (a URL,
not a DID).
- Jetstream fails closed: an unresolvable hold reference is cached raw
rather than left empty, since an empty value reads as managed.
- UI: the billing tab is hidden on self-hosted, a cancel/manage banner
appears when a self-hosted user still has an active plan, the image
advisor returns managed_hold_required instead of upgrade_required,
and the checkout route returns 403. The portal stays open so existing
subscribers can still cancel.
Two consistency fixes fall out of wiring this up:
The settings UI reads the resolved default_hold_did rather than the raw
profile.DefaultHold. The profile field is the record value as written and
may be a URL-form reference; jetstream resolves it to a DID on the way
into the DB, and the server-side gate reads that resolved value. Comparing
the raw form against managed DIDs would show the "you are self-hosted"
banner and hide billing from a user whose entitlements say otherwise.
HasAIAdvisor falls back to the free tier's AIAdvisor setting when
off-managed instead of a hard false, matching GetWebhookLimits. Losing a
managed hold should drop a user to free-tier entitlements, not below them.
BEHAVIOR CHANGE for existing paying users on self-hosted holds: they lose
the AI advisor, supporter badge and paid webhook limits as soon as this
deploys, while Stripe keeps charging them. The only notice is the banner
on /settings/storage, which they have to visit to see. Decide on a
migration (notification, or a one-time reconciliation over active
subscriptions) before shipping this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Webhook delivery was neither idempotent nor order-safe, and every failure
returned 400, which Stripe does not retry. A transient DB or hold error
therefore dropped a subscription change silently and permanently.
- New stripe_processed_events table: event_id as primary key dedups
redelivery, and event_created per customer drops stale out-of-order
deliveries.
- HandleWebhook distinguishes ErrWebhookSignature (400, no retry) from
ErrWebhookProcessing (500, Stripe redelivers). The event handlers
return errors instead of swallowing them. ErrBillingDisabled maps to
400: the route is mounted but billing is off, so redelivery can never
succeed and Stripe should stop rather than retry to exhaustion.
- Refuse to boot when billing is enabled with an empty
STRIPE_WEBHOOK_SECRET. Stripe HMACs with the empty key, so an
attacker can reproduce the signature and the endpoint is forgeable.
- UpdateCrewTierOnAllHolds retries each hold (3 attempts, linear
backoff, 5s per request) and returns a joined error so the webhook
can fail and let Stripe redeliver.
The fan-out contacts holds concurrently rather than in sequence. Serially,
one unreachable hold burns the caller's entire 10s budget on its own
retries (3 x 5s plus backoff) and the holds after it are never contacted;
because Stripe redelivers in the same order, a persistently-down first
hold means the rest are never updated at all.
On the hold, the signature-validated sub claim is now the source of truth
for updateCrewTier: a mismatched body userDid is rejected with 403 rather
than retargeting the grant to another DID. "Not crew on this hold" is a
200 no-op, since the appview fans updates out to every managed hold and a
subscriber is not crew everywhere.
That no-op has to be told apart from a storage failure. GetCrewMember
collapsed both into one generic error, so a CAR-store failure read as
"not a member", answered 200, and let the appview record the event as
processed — losing the tier grant permanently, which is exactly the
failure mode this commit exists to prevent. Missing records now carry an
ErrCrewMemberNotFound sentinel, and anything else returns 500 so Stripe
redelivers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ValidateDeviceSecret ran bcrypt.CompareHashAndPassword against every row in
the devices table until one matched — no WHERE clause. At bcrypt cost 10
(~65ms on the single-core production host) and 244 registered devices, a
device near the end of the scan cost ~15.8s of pure CPU per /auth/token,
which is past Docker's client deadline. Measured on production: 15.7-16.0s
steady state with the appview pinned at 100% CPU for the duration, while
anonymous requests on the same box served in 20ms.
The cost grew linearly with every device registered, and the scan ran in
rowid order, so the newest devices — the ones most likely to be in active
use — paid the most. This is the timeout users were reporting.
Devices now carry secret_lookup = hex(sha256(secret)), indexed, and
authentication fetches the single matching row.
SHA-256 is the verifier here, not merely an index. Device secrets are 32
bytes from crypto/rand, so presenting a value that hashes to a stored digest
requires a preimage or a 2^256 search. bcrypt's work factor only helps when
the input space is small enough to enumerate, which does not apply to a
random 256-bit token, and a database leak exposes no more than before.
The plaintext is not recoverable from a bcrypt hash, so existing rows cannot
be backfilled directly. They are migrated lazily on their next successful
authentication, which any push, pull or login triggers, and the legacy scan
is filtered to un-migrated rows so its cost decays as devices migrate. The
backfill runs after the cursor is closed: issuing it inside the rows loop
deadlocks, because the open cursor holds the connection the write needs.
bcrypt now exists solely to carry legacy rows across and can be deleted once
the table is fully migrated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- diff view gains a Packages tab with added/removed/changed/unchanged
package tables and purl-derived type/license/upstream links
- captain records verified against the DID's atcr_hold service before
caching (processor + batch backfill), preventing forged holds
- fix empty-handle updates clobbering cached handles and colliding on
the UNIQUE constraint
- move fillPrevCIDs into repo.go; DirectRepoOperator is now canonical,
repomgr kept as a test oracle
- surface read-only crew status in hold selector
- reconcile docs
1. Multiple registry domains + per-user domain preference
The biggest feature. The appview can serve several registry domains (e.g. buoy.cr, atcr.io),
and users can now pick which one shows up in their pull/push commands.
- Lexicon/record: adds registryDomain (and documents ociClient)
to the sailor profile (lexicons/.../profile.json, pkg/atproto/lexicon.go).
- DB: new registry_domain column on users (schema.sql + migration 0027),
with GetUserByDID/Handle reads, UpdateUserRegistryDomain writer,
and Jetstream caching it on profile updates (writes unconditionally so clearing propagates).
- UI/handlers: new UpdateRegistryDomainHandler + /api/profile/registry-domain route,
a <select> in the user settings panel (only shown when >1 domain configured), and resolveRegistryURL()
which falls back to the primary domain if the user's pref is stale/removed. Tests added for all of it.
2. default_hold_did removed → first managed_holds entry is the default
Consolidates two overlapping config fields into one. ServerConfig.DefaultHoldDID is gone;
PrimaryHoldDID() now returns managed_holds[0]. managed_holds is now REQUIRED.
Updated in config, validation, server wiring, test harness, example YAML, and the deploy template.
3. Admin long-running operations → generic background-job framework
New pkg/hold/admin/jobs.go introduces a reusable startJob/jobRegistry pattern
(a detached context.Background() job + a /admin/api/jobs/{key}/status polling endpoint).
This replaces the bespoke scan-backfill goroutine state machine, and now also wraps crew tier remap and crew import
all three previously looped synchronously on the request context and got 504'd/cancelled mid-run by the reverse proxy.
Forms switched from POST-redirect to htmx fragments (job_progress.html, job_result.html, crew_import_results.html)
the old crew_import_results.html page and scan_backfill_progress.html partial were deleted.
This is also captured as a new rule in CLAUDE.md.
4. Cascade-delete manifest on last-tag deletion
DeleteTagHandler now, after removing the last tag pointing to a digest, cascade-deletes the manifest itself
(PDS + DB + hold blob purge) — but only if it's not a child of a manifest list (multi-arch parent).
New GetTagDigest and ShouldCascadeDeleteManifest queries back it, plus cascade_delete_test.go.
Also switches tag rkey computation to the atproto.RepositoryTagToRKey helper.
5. Billing simplification
Drops the OwnerBadge config option (hold-owner supporter badge).
The user-profile template no longer special-cases an "owner" badge value (only "Captain").
Example tiers renamed to the nautical scheme (deckhand/bosun/quartermaster).
6. Build/deploy: go generate always runs via Make
make generate is now a phony target that always runs go generate ./... (regenerating cbor_gen, icon sprites, etc.),
and build-trixie depends on it. The deploy tooling (provision.go/update.go)
drops its own runGenerate calls since the Makefile handles it.
7. New cmd/firehose-tap tool (untracked)
A standalone CLI that subscribes to a com.atproto.sync.subscribeRepos endpoint and pretty-prints events,
with emphasis on Sync 1.1 compliance fields (per-op prev CIDs, commit prevData) and a --validate CI mode.
Fits with the recent "more sync1.1 compliant" commit.
Evan JarrettandClaude Opus 5