mirror of
https://tangled.org/evan.jarrett.net/at-container-registry
synced 2026-09-02 00:06:58 +00:00
2984331f0c85e04ad07acb01c5321ce1d6498367
470
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Evan JarrettandClaude Opus 5
|
2984331f0c |
hold/gc: test the blob sweep, and give the S3 mock object ages
deleteOrphanedBlobs is the only part of GC that removes bytes and it had no test. It could not have had one: object age decides whether a blob is deletable, the in-process harness stamps every object with time.Now(), and MockS3Client's ListObjectsV2 set no LastModified at all. Neither could express "this blob is nine days old", so both halves of the grace rule went unexercised — including the half that protects a push still in flight. MockS3Client gains ObjectTimes, a per-key LastModified consulted by ListObjectsV2. Keys with no entry list without a timestamp exactly as before, so existing tests are unaffected. It also gains DeleteObjectError, matching the error injection the other operations already had. Four cases, three of which are reasons NOT to delete: an old unreferenced blob goes; a young unreferenced blob stays; a referenced old blob stays; a /link object is never treated as a blob. Plus a failure case pinning that one undeletable object does not abort the walk, and that a blob which never left storage is not counted as deleted or reported as reclaimed space. Verified by mutation. Disabling the grace check deletes the young blob; disabling the referenced check deletes the live one; both fail. Disabling the /data suffix check changes nothing, because extractDigestFromPath anchors on /data$ and rejects everything else — so that check is a redundant early-out rather than a guard, and the test says so rather than implying otherwise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
4542897f08 |
hold/gc: escape the DID in listRecords/getRecord repo parameters
fetchUserManifestsFromEndpoint, fetchUserTags and fetchUserProfile
interpolated the user DID straight into a query string. A did:web carrying a
port spells that port as a literal %3A, so the PDS received the parameter
decoded back to ":" — a different DID, matching no repo. listRecords then
answers 200 with an empty list and GC reads it as "this user has no
manifests": every blob they own drops out of the referenced set and is
deleted once past the seven-day blob grace.
There is no error and no status code to notice, which is the same soft
failure
|
||
|
Evan JarrettandClaude Opus 5
|
891ad01de3 |
hold/gc: require a predecessor's successor to name this hold
|
||
|
Evan JarrettandClaude Opus 5
|
48eee49ef9 |
hold/gc: cover checkPredecessorAt and the unresolved-holds reset
|
||
|
Evan JarrettandClaude Opus 5
|
bb45a80d51 |
hold/pds: cover the scanner-disconnect teardown 05b856b fixed
newTestScanBroadcaster builds the struct with only a database, so nothing in
the suite ever reached handleWriter, handleReader, or the Unsubscribe teardown
they share — which is precisely what
|
||
|
Evan JarrettandClaude Opus 5
|
8cd59a61f1 |
appview: stop a UI session outliving the OAuth session behind it
Only one oauth_sessions row is kept per account, so signing in again — on a
second device, or simply a second time — replaces it and leaves every earlier
ui_sessions row pointing at an oauth_session_id that no longer exists. Get
checked only expiry, so those still read back as usable.
Found on a live appview: four ui_sessions rows, three orphaned, and requesting
/settings/user with an orphaned cookie returned 200 with the account's handle
rendered throughout, where an anonymous request gets a 302. The browser looks
signed in while the credential behind it is gone, so every PDS-backed action
fails against a UI insisting the session is fine. It now fails closed and sends
the user back through login.
Get also never checked ownership. oauth_sessions is unique on
(account_did, session_id), so the existence check is scoped by both; matching
session_id alone would let one account's live OAuth session validate another
account's dangling reference. That has its own test.
An empty oauth_session_id stays valid, since Create makes sessions that never
had one, and a test pins that so the check cannot start rejecting them.
TestSessionStore_CreateWithOAuth referenced an OAuth session it never inserted,
which is an orphan by definition, so it now creates the row. Its intent was
that CreateWithOAuth persists the ID; it relied on the orphan behaviour only
incidentally. Its not-found branch used t.Error and then dereferenced the nil
session, so that is now t.Fatal.
Pre-existing at
|
||
|
Evan JarrettandClaude Opus 5
|
c9f8b4178c |
credhelper: stop dev builds nagging about an update to themselves
isNewerVersion split versions on "." and ran each component through
strconv.Atoi, discarding the error and substituting 0. For a git-describe
build the last component is "4-18-g8f70cce", which does not parse, so it
became 0 and every published release compared as newer. Running
v0.1.4-18-g8f70cce printed "Update available: v0.1.4" on every single
invocation, naming a version the binary was already 18 commits past.
Versions are now parsed properly: the "-<commits>-g<sha>" tail is recognised
and kept as a count of commits past the tag, and a version that cannot be
read in full returns false rather than being silently treated as 0. That
second part is the actual root cause — the comparison could not distinguish
"this component is zero" from "I could not read this component".
Ordering for a git-describe build is deliberately not semver, where a
prerelease sorts below its release. Such a build is commits AHEAD of its tag,
so v0.1.4-18-g8f70cce is newer than v0.1.4 and older than v0.1.4-20-gabc1234.
The function had no tests. Both failing cases are pinned along with the
ordinary release comparisons, so the git-describe handling cannot regress the
normal upgrade path.
Pre-existing at
|
||
|
Evan JarrettandClaude Opus 5
|
f8d9ad7fe9 |
appview: let a registry domain keep its port, and its /v2
DomainRoutingMiddleware normalized the request Host to a bare hostname but
matched server.registry_domains verbatim, so any configured domain carrying a
port could never match. config-appview.example.yaml ships
`registry_domains: [127.0.0.1:5000, atcr.io]`, which means that entry has been
inert since it was written.
It fails closed in the worst way. That same host is also the auto-detected UI
host, and `host == uiHost` was evaluated first, so /v2/* was answered with
"registry API is not available on this domain, use 127.0.0.1:5000" — naming the
exact host the client had just used. The registry API is unreachable on the dev
stack, and any single-host deployment hits the same wall: listing a host in
registry_domains does nothing if it is also the UI host.
Both sides are now normalized through hostWithoutPort, and a registry domain
takes /v2/* even when it doubles as the UI host, which is a legitimate
single-domain deployment. Everything else is unchanged: a UI-only host still
refuses /v2/, registry domains still redirect non-/v2 traffic to the UI, and
/auth/token and /auth/device/* are still served directly so a cross-host 307
cannot strip the Authorization header.
hostWithoutPort uses net.SplitHostPort instead of the previous LastIndex(":")
scan, which mangled bracketed IPv6 literals into "[::1" and could never match
the "::1" that url.URL.Hostname() yields for the UI host.
The middleware had no tests at all. The two failing cases are pinned first, and
the four pre-existing behaviours are pinned alongside them so the reorder
cannot quietly widen what /v2/ is served on.
Pre-existing at
|
||
|
Evan JarrettandClaude Opus 5
|
4c04983e23 |
appview: stop the backfill claiming every user was just active
last_seen means "this user did something recently". The backfill walks every historical record in the network, so stamping it there recorded when the backfill ran, not when the user was active — for every user at once, on every run. That destroys the only signal the column carries, and it is the one column in users that nothing upstream can rebuild. It is now written on the two paths that represent real activity: an interactive login, and a live commit event on the firehose, which does mean the user just wrote a record. The backfill still corrects handle, PDS endpoint and avatar, which is why it re-resolves rather than trusting a cache; it just no longer claims the user was present. UpsertUser grows an options form rather than a fourth named variant, since the avatar and last_seen decisions are independent and all four combinations occur. Anyone computing MAU from this column should know it was unreliable for every backfill run before this change. Also corrects docs/HORIZONTAL_SCALING.md, which claimed oci_client and registry_domain were local-only preferences. They are fields on io.atcr.sailor.profile: settings writes them to the user's PDS and ProcessSailorProfile refreshes the local cache. users is fully derived apart from last_seen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
13edb7184d |
appview: stop writing last_seen and last_used on every event
Neither is a correctness problem; both are round trips on hot paths for timestamps nothing reads at that resolution. UpdateUserLastSeen ran per Jetstream event for cached users, so once per indexed record. DeviceStore.UpdateLastUsed ran per /auth/token call, so once per docker push and pull including each layer's re-auth. Cheap against a local file, a network round trip each against a remote primary, and the second sat on the authentication path. Both are now throttled to once per five minutes per subject. The MAU queries and the admin views work in hours or days, so nothing loses meaning. The throttle state is per-process and lost on restart, costing at most one extra write per subject per boot; only the lease holder runs the consumer, so exactly one process is doing the first of these at a time. UpdateLastUsed stamps the throttle before writing rather than after, so a slow or failing write cannot let every concurrent layer upload through to pile on more of them. Verified by disabling the throttle: 50 back-to-back calls then rewrite the timestamp every time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
e9d43aa767 |
db: cover the recency change on all four surfaces, not one
The MAX(id) replacement touched four queries that each carry their own copy of the same CTE: SearchRepositories, GetRepoCards, GetUserRepoCards and GetStarredRepoCards. Only the third had a test. Fixing one and missing another would leave the UI disagreeing with itself about which manifest is current, depending on which page you were looking at. All four now assert that recency follows created_at rather than insert order, and that a tie between manifests pushed in the same second resolves the same way on every surface. Verified by regressing the CTEs back to rowid ordering: each of the four fails independently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
454a6bad3d |
db: key manifests by manifest_key and drop the rowid
Completes the swap 0033 set up. layers and manifest_references move onto manifest_key and manifests.id is gone, which removes the last node-allocated identifier in the AppView schema. Statement order in 0034 is load-bearing. With foreign keys on, DROP TABLE performs an implicit DELETE FROM, so dropping manifests while layers still holds an ON DELETE CASCADE reference deletes every layer row. Migration 0009 did exactly that; it went unnoticed because the Jetstream backfill rebuilds layers from PDS records, so the damage healed itself. PRAGMA foreign_keys is no help: it is a no-op inside a transaction and migrations run in one. So the new children are built pointing at manifests_new, the old children are dropped first, and only then is the old manifests table dropped, by which point nothing references it. Verified both behaviors before relying on them. manifest_key is declared NOT NULL as well as PRIMARY KEY, because in SQLite a PRIMARY KEY column still accepts NULL unless it is INTEGER PRIMARY KEY. That constraint immediately caught four test helpers inserting manifests without one. Five queries used MAX(id) as "the newest manifest in this repo", which I had previously reported as absent after grepping only for ORDER BY. A derived key has no ordering, so recency now comes from created_at with manifest_key as a deterministic tiebreak. This is a real behavior change, and a fix: the two disagree whenever a manifest is indexed out of order, which the backfill does routinely, and created_at is the push time these queries always wanted. Both directions are tested, including that ties resolve the same way every run. InsertManifest and BatchInsertManifests no longer read anything back. The key is derived from (did, repository, digest), so the writer knows it before the statement runs: the select-back, its per-DID IN list, and the "manifest missing id after batch insert" branch all go away, along with the UNIQUE-conflict fallback that existed only to recover a rowid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
11b85e5102 |
db: fill manifest_key inside its migration, not at runtime
0033 added the column and left the fill to the Jetstream backfill. That is fine for a running system and wrong for replay: a database several releases behind runs every pending migration back-to-back at boot, long before any worker starts. Anything built on top of manifest_key would, on that path, silently operate on NULLs while working perfectly on a system that had been up for a while. Establishing a migration's data precondition out-of-band means replay cannot see it. The runner now supports a Go step per migration version, running inside the same transaction as that migration's SQL, after the DDL it depends on and before the version is recorded. A version is never recorded without its Go half. 0033's step fills manifest_key for every row lacking one. It has to be Go: the value is a truncated sha256 and SQLite has no hash builtin. It pages through the table and writes one UPDATE ... CASE per 500 rows, because a statement per row would be correct and unusably slow against a remote primary. Verified by unregistering the hook: replay then leaves 3 of 3 seeded manifests with NULL keys. With it, all three are filled, from a snapshot of the pre-0009 schema forward. The upserts keep their "OR manifests.manifest_key IS NULL" clause. It is a self-healing net for rows that somehow arrive without a key rather than the mechanism anything depends on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
802cc4ba96 |
db: add manifest_key and let the existing backfill populate it
First half of replacing manifests.id with a node-independent identity. Nothing depends on the column yet: id is still the primary key, and layers and manifest_references still reference it. Getting the column in place and filled first means the eventual swap operates on data that is already complete and already proven unique, instead of doing the fill and three table rebuilds in one step. The value cannot be computed by the migration. It is a truncated sha256, SQLite has no hash builtin, and go-libsql exposes no way to register one. So the fill uses machinery that already exists: the Jetstream backfill re-upserts every manifest across the protocol on startup, and both upsert paths now write manifest_key. That only works because of one extra clause. Both upserts guard their DO UPDATE with a WHERE that skips rows where nothing changed, which on a backfill re-run is nearly every row, so they would have skipped the very manifests that need filling. Adding "OR manifests.manifest_key IS NULL" is what makes an otherwise no-op pass populate the column. Verified by removing it: the backfill then fills zero of three manifests instead of three of three. The index is UNIQUE even though the column is nullable. SQLite treats NULLs as distinct, so unfilled rows coexist while every filled row is checked. That makes production data verify the 16-byte truncation rather than us assuming it: if two manifests ever derived the same key, it fails loudly at insert instead of silently attaching one manifest's layers to another after the swap. AppView logs the unfilled count at startup, since there is no single moment at which this becomes complete and the follow-up migration is only safe at zero. ManifestKey replaces the old fat "did|repo|digest" map key rather than sitting beside it; they were always the same question, answered without asking the database. The jetstream tests hand-maintained their own CREATE TABLE statements, which is the drift problem moved into a test: the copy had already fallen behind (it still had tags.id) and only failed once a query touched the difference. They use db.InitDB now, with foreign keys switched off to preserve the behavior the hand-rolled schema had. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
f186760847 |
db: drop the vestigial tags.id
Nothing joined on it. It was selected into a struct field no caller read, and used only by DeleteTagsNotInList, which fetched surrogate ids, filtered them in Go with a nested loop over the keep list, and issued one DELETE per row. The natural key was already enforced by UNIQUE(did, repository, tag), so that becomes the primary key and the column goes. An AUTOINCREMENT rowid is allocated by whichever node performs the insert. That is fine while every write funnels through one writer and stops being a stable identity the moment they do not, so removing an identifier nobody used is the cheapest way to shrink that surface before local-write replicas. DeleteTagsNotInList now diffs against a set and deletes in batches. It still reads the current tags first rather than issuing one NOT IN over the keep list: that would need two placeholders per kept tag and would break past the driver's parameter ceiling for a user with enough tags, and it cannot be chunked, because each chunk would delete the tags every other chunk meant to keep. An explicit delete list chunks safely. idx_tags_did_repo is dropped rather than recreated: the new primary key indexes (did, repository) as a prefix. It existed only because the primary key used to be the surrogate id. The rebuild names its columns explicitly. Column order is not guaranteed to match between a fresh install and a migrated one, so INSERT ... SELECT * here could write values into the wrong columns. TestMigration0032PreservesTagRows runs the migration body against a table in the old shape and checks the contents survive, which the schema drift test cannot: it compares shape, not data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
e75b2e246b |
oauth: compare-and-swap session writes so a concurrent refresh cannot delete a live session
Refresh tokens rotate on use, and DoWithSession serializes refreshes per DID with an in-process mutex. That is the right mechanism and it protects nothing once there are two instances: both can refresh the same account at the same time, the slower one presents a refresh token the auth server has already superseded, gets invalid_grant, and isAuthError deletes the session. The user is signed out mid-push, and the session another instance had just legitimately refreshed is destroyed along with it. oauth_sessions gains a rev that increments on every write. A store that has read a session writes with a compare-and-swap against the revision it read and gets ErrSessionRevConflict if anyone wrote first, so a stale writer can no longer replace rotated tokens with invalidated ones. The persist callback treats that conflict as an ordinary outcome rather than an error, since leaving the newer state alone is exactly right. The delete path is now guarded by the same signal. An auth error on a session whose revision has moved since we read it means "someone else refreshed this", not "this session is dead", so it retries once against the newer tokens instead of deleting. Exactly once: a second failure means staleness was not the problem, and looping would hold the per-DID lock while getting the same answer. The guard is deliberately conservative. A store without revisions, no recorded revision, a failed lookup, a session that is simply gone: all answer "not advanced" and keep the previous delete-on-error behavior. Wrongly claiming a concurrent refresh would keep a genuinely dead session alive with no way out but waiting; wrongly missing one costs a re-login. The sentinel lives in pkg/auth/oauth rather than next to the SQLite store, because pkg/appview/db already imports pkg/auth/oauth and the other direction would be an import cycle. The db package re-exports it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
934e4a2a59 |
appview: stop the crypto key first-boot race
Two instances booting against a fresh database both find no key, both generate one, and both write. PutCryptoKey was last-writer-wins, so the loser kept its own key in memory while the database held the other's. It then signed OAuth client assertions with a key absent from the published JWKS, and issued registry JWTs that did not match the certificate written to disk. Every one of them fails verification, and nothing logs why. PutCryptoKey now keeps the first write, and both loaders re-read afterwards and use whatever is stored. Nothing in the codebase rotates a key through this function, so the update arm only ever fired on the race. Verified against the old behavior: with last-writer-wins restored, three of six concurrent loaders returned a key that was not the one in the database. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
985ebd3a5f |
auth: make the crew denial counter atomic
cacheDenial read denial_count, incremented it in Go, and wrote the result back.
Two overlapping denials for the same (hold, user) both read the same value and
both wrote the same value, so one increment vanished. The effect is that the
backoff ladder advances more slowly than configured, which means a denied client
keeps hammering the hold's PDS for longer than intended. Already reachable
across goroutines on one instance; routine with several behind a load balancer.
It is now a single INSERT ... ON CONFLICT DO UPDATE that increments in place.
next_retry_at moved into SQL as well, derived from the count the same statement
is producing, rather than computed in Go from a count that may already be stale
by the time the write lands. The CASE ladder is generated from
dbBackoffDurations so configuration still drives the backoff, and no request
data reaches the string.
The measured difference, with 20 concurrent denials: the old code recorded 12
where it should have recorded 21, losing 9. The new code loses none.
The surviving SELECT only picks a branch (first denial goes to memory only), so
a stale answer costs at most one skipped or one extra write, never a count.
Two implementation notes. datetime() truncates to whole seconds and the backoff
ladder is sub-second in tests, so timestamps use
strftime('%Y-%m-%dT%H:%M:%fZ', ...) instead; libSQL normalizes that to RFC 3339
and it scans back into time.Time with the right instant, which was verified
before relying on it. And a one-rung ladder emits a bare number rather than a
CASE, because "CASE ELSE x END" with no WHEN arm is a syntax error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Evan JarrettandClaude Opus 5
|
182a5463d6 |
auth: stop wiping the shared denial cache on every boot
ClearAllDenials ran unconditionally at startup, and its database half is "DELETE FROM hold_crew_denials" with no scoping at all. One instance, that is a clean slate on deploy. Several instances, and a rolling deploy wipes the shared table once per instance while every scale-out event wipes it again, so the backoff that exists to stop a denied client hammering a hold's PDS keeps getting reset out from under it. The intent is worth keeping: a restart usually means a fix shipped, and someone sitting on a backoff of up to an hour should get to retry rather than wait it out. So it moves under the cleanup lease instead of being deleted, and now happens once per deploy rather than once per instance. Worth noting the in-memory half was always a no-op here. recentDenials belongs to the process, and a process that has just started has an empty one, so the table-wide DELETE was the only thing the startup call ever really did. The cleanup worker moved down past the hold authorizer's construction, since it now needs a handle on it. Reading s.HoldAuthorizer from the worker goroutine while the constructor was still assigning it would have been a data race. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
6a7ddb819b |
appview: run background workers under a lease
The Jetstream consumer, backfill, labeler subscriber, cleanup sweep and billing tier refresh all started unconditionally in every process. That is correct for one instance and wrong for two. The consumer is the case with teeth. StatsCache is per-process in-memory state, and the aggregate it produces is written to repository_stats as an absolute value rather than an increment, so two consumers each hold a partial view of the holds and each write their partial sum as though it were the whole truth, overwriting one another indefinitely. The webhook dispatcher hangs off the same processor, so a second consumer also doubles every delivery. Each now runs under a named lease, so exactly one instance runs it and a replacement takes over when that instance goes away. The health worker is deliberately not leased: it refreshes a cache each instance needs locally, so running it everywhere is correct. Two structural changes came with it. The cleanup loop moved out of InitializeDatabase, where it was a bare goroutine with no way to reach the lease manager, into RunPeriodicCleanup called from the server. And backfill's startup run and periodic schedule became one leased worker instead of two goroutines on context.Background(), so shutdown actually stops a backfill in flight rather than letting it run on against a closing database. With interval=0 that worker holds its lease instead of returning, since releasing would let another instance acquire and run its own startup backfill, turning "once" into "once per instance". Verified with two instances against one database: exactly one acquired, the other contended without starting a worker; SIGTERM handed over in 13ms via the release, SIGKILL handed over in ~12s via TTL expiry. That first number only holds because of Manager.Go and Manager.Wait, which this commit adds. The first cut used `go m.Run(...)` and cancelled the worker context during shutdown without waiting, so the process exited before the release landed and the lease survived to its TTL — a rolling deploy would have paused indexing for a minute rather than a second. Nothing in the unit tests caught it; the two-instance run did. TestWaitBlocksUntilLeaseReleased covers it now. leases.enabled defaults to true. A single instance is unaffected, since it always wins its own leases, while an operator who scales out without reading the docs still gets correct behavior instead of silent stats corruption. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
f84e8ffa27 |
db: create instance_leases and add the lease manager
Groundwork for running more than one AppView instance. Nothing is wired to this
yet; the next commit moves the background workers onto it.
Several workers must run on exactly one instance. The Jetstream consumer is the
sharpest case: StatsCache is per-process in-memory state, and the aggregate it
produces is written to repository_stats as an absolute value rather than an
increment, so two consumers would each hold a partial view of the holds and each
write its partial sum as the whole truth, overwriting one another indefinitely.
The webhook dispatcher hangs off the same processor, so a second consumer also
means every webhook fires twice.
Instances contend for a named lease; only the holder runs the worker. Acquire is
a single INSERT ... ON CONFLICT ... WHERE, so two instances racing for the same
expired lease cannot both win: the loser's update matches no rows. The fence
token increments on every change of custody, so a process that stalled past its
TTL discovers on its next renewal that it was superseded, rather than continuing
to act as the holder.
A renewal blackout is treated as a loss. If the database has been unreachable
for longer than the TTL, another instance is entitled to steal the lease and we
must assume it has, even though we cannot ask. Continuing to work in that state
is the one outcome the lease exists to prevent.
Clean shutdown expires the lease in place rather than deleting the row, so a
replacement starts in seconds instead of waiting out the TTL, while the fence
token survives to keep a stalled former holder from matching again.
Timestamps are Unix milliseconds, not TIMESTAMP text. libSQL normalizes
date-like TEXT on the way in, and Go's driver and CURRENT_TIMESTAMP disagree on
format, so a stored expiry and a literal would compare as strings that sort
differently. That comparison is the whole safety property, so it does not get to
be subtle. The cost is a dependency on roughly-synced clocks, the same
assumption Kubernetes leases make; keep the TTL well above any plausible skew.
The lease tests are file-backed rather than :memory:. go-libsql gives every
connection to an in-memory DSN its own private database, so with MaxOpenConns of
8 a second goroutine lands on a connection where the schema was never applied
("no such table"). Every existing test in the package is sequential and reuses
one pooled connection, which is why this has stayed invisible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Evan JarrettandClaude Opus 5
|
2abcae95f7 |
db: warn on schema drift at startup
Migrations can be recorded without being executed. That is not hypothetical: migration 0009 exists to clean up after 0004, which production recorded but never applied, leaving eleven columns behind that fresh installs never had. Nothing reported it at the time; it surfaced later as confusing behavior. InitDB now compares an existing database against schema.sql after migrations run and logs one warning per difference. Fresh databases skip the check, since they were just built from schema.sql and agree by construction. The comparison works by applying schema.sql to a throwaway in-memory database and introspecting that, rather than parsing the DDL. SQLite's own resolution of types, defaults and implicit indexes is exactly what we want to compare against, and a hand-rolled parser would drift from the engine. The introspection is shared with TestSchemaMatchesMigrations, so the test exercises the same code that runs at boot. Warn-only, never fatal. A database merely ahead of or behind schema.sql is almost always still able to serve traffic, so refusing to boot would turn a diff that wants a corrective migration into an outage, during a deploy, which is the worst possible moment to have one. The README claimed new tables go in schema.sql only. That is wrong in the direction that hurts: InitDB skips schema.sql entirely once schema_migrations has rows, so such a table appears on fresh installs, passes every test, and is silently absent in production. Documented the real rule along with two others the test cannot enforce: migrations must not return rows (go-libsql rejects them with "Execute returned rows"), and rebuild migrations must name columns explicitly, since column order legitimately differs between fresh and upgraded databases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
f5ddc229a1 |
db: prove schema.sql and the migrations agree
schema.sql and migrations/ are meant to move in lockstep, and nothing checked that they did. The cost is already on the record: migration 0009 exists only because 0004 "was supposed to drop these columns but either failed or was only recorded (not executed) on production", leaving production carrying eleven columns fresh installs never had. Build the schema both ways and compare. testdata/base_schema.sql is the shape immediately before 0009, reconstructed by reversing 0009-0029 out of the current schema.sql; applying it and then running every migration must land on the same place as applying schema.sql directly. Columns compare as a set, ignoring ordinal position. Migrations append with ADD COLUMN while schema.sql places the same column mid-table, so manifests, users, devices and repo_pages legitimately differ in order. Reordering them would mean four rebuild migrations for no functional gain, and nothing in pkg/appview reads by position (no SELECT *, no column-less INSERT ... VALUES). Verified non-vacuous three ways: a column only in schema.sql, a column only in a migration, and a table only in schema.sql are each caught. That last case is the one with teeth, since InitDB skips schema.sql entirely once schema_migrations has rows, so a table added only there never reaches an existing database. The snapshot records versions 1-8 as applied, which is what "before 0009" means. It also sidesteps a live rake: migration 0001's query is a bare SELECT, and go-libsql rejects row-returning statements passed to Exec. Every real database recorded 0001 long ago so it never fires, but a future migration opening with a SELECT would fail the same way. This cannot independently verify tables that appear in no migration; those are copied from schema.sql and compare against themselves. Drift there is the startup check's job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
5aa13abdc2 |
auth: make anonymous pull work, and let the hold decide it
|
||
|
Evan JarrettandClaude Opus 5
|
2580dcdb0f |
appview: stop a tag delete cascading into another repo's live image
io.atcr.manifest rkeys are the digest alone (digestToRKey), so a single
record backs every repository of a user holding identical content. Both
paths that cascade-delete that record checked for remaining tags scoped to
one repository, which asks the wrong question: a tag in another repo keeps
the shared record alive just as much as a tag in this one.
With me/a:v1 and me/b:v1 at the same digest, deleting me/a:v1 saw no
remaining tags in repo a, deleted the shared PDS record, and purged the
layers on the hold. me/b:v1 was left pointing at content that no longer
exists, and the firehose delete handler then cleared the rows for every
repo (DeleteManifest with an empty repository argument).
The collision predates this, but it was reachable only behind the opt-in
AutoRemoveUntagged profile flag.
|
||
|
Evan JarrettandClaude Opus 5
|
adc6394ebc |
db: renumber the second 0028 migration so it actually runs
Two migration files shipped as version 28: 0028_add_device_secret_lookup ( |
||
|
Evan JarrettandClaude Opus 5
|
a7569a7717 |
registry: allow anonymous pull of public images
Credential-less pulls of public images. /auth/token issues a pull-only
token with an empty subject when no Basic auth is present; the
destination hold still enforces captain.Public, and push or delete always
challenges.
- token.IsPullOnlyScope and AuthMethodAnonymous;
Handler.issueAnonymousToken skips the authorizer gate and the
service-auth pre-mint, since there is no identity to reconcile and no
AppView-to-hold service token to bind. The token is still stamped
with the resolved registry domain, so anonymous pull works on
secondary front doors whose access controller demands their own
audience.
- auth.allow_anonymous_pull (default true) turns it fully off, restoring
the previous always-challenge behavior. Mirrored into the deploy
template, since the default means existing deploys pick this up.
- RegistryContext.Anonymous is plumbed from the middleware.
- ProxyBlobStore sends no Authorization header when the service token is
empty, and returns 401 rather than 403 for anonymous denials so Docker
prompts for credentials, including when a stale captain cache lets the
request through and the hold says private.
- BearerChallenge wraps the /v2/ subtree so a 401 raised deep in the
stack via errcode.ServeJSON still carries WWW-Authenticate.
Distribution's own scoped challenges are left alone.
IsPullOnlyScope allowlists the pull action instead of denylisting push and
delete. Distribution's actionSet.contains treats "*" as *every* action, so
a scope of `repository:victim/img:*` names neither denied string and would
have handed an unauthenticated caller a token valid for push and delete on
someone else's repository — clearing the authgate entirely, since anonymous
tokens deliberately skip it. Writes would still have failed further down
(no PDS credential), but the gate itself was bypassable. Now every
requested action must be exactly "pull". Covered by new claims tests.
Unresolvable identities return NAME_UNKNOWN instead of a bare error that
distribution renders as 500. This path was previously unreachable without
credentials; anonymous pull opens it to the internet, and a 5xx on
arbitrary input both misreports a bad request as a server fault and sends
clients that retry 5xx into a retry loop. That loop was real: in the auth
matrix, regclient spent 83s on a single case before this fix, and the
suite now runs in 5s.
Stat preserves an authorization verdict from getPresignedURL rather than
flattening it to ErrBlobUnknown. Distribution calls Stat before ServeBlob
on GET and HEAD, so without this an anonymous pull from a private hold
answered 404 and BearerChallenge had no 401 to annotate — the 401 path
above could never actually reach a client.
The auth matrix is updated to match: anonymous pull of the seeded public
repo now succeeds, anonymous push is denied against a real identity's
namespace (rather than an unresolvable one, which was testing name
resolution rather than authorization), and a new case pins the
NAME_UNKNOWN behavior for an unknown identity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Evan JarrettandClaude Opus 5
|
6510c16dd4 |
webhooks: enforce the entitlement at dispatch time
The webhook limit was only checked at creation, so losing entitlement (a
hold switch or a plan downgrade) left previously-created webhooks firing
paid behavior forever.
- Dispatcher takes a WebhookLimiter, consulted on every dispatch. It
caps the list to the current allowance, keeping the oldest N to match
what the creation gate would have permitted, and masks paid trigger
bits.
- GetWebhooksForUser orders by created_at ASC, id ASC so that cap is
deterministic. ListWebhooks gets the same tiebreak: it feeds the
settings UI, and without it the list a user sees could disagree with
the one the dispatcher truncates.
- webhooks.FreeTriggerMask is shared by the creation gate and the
dispatch backstop so the two cannot drift.
Capping is logged when it actually truncates. The webhooks stay visible in
settings, so from the user's side delivery would otherwise just stop with
no signal — and the same line is the only evidence if the limiter itself
degraded, since a billing lookup failure falls back to free-tier limits
and would quietly demote a paying user mid-dispatch.
Two cost fixes, both because this puts the entitlement lookup on a hot
path it was never on before:
findCustomerByDID now consults the customer cache instead of always
issuing a Stripe customer search. GetWebhookLimits reaches it via
GetSubscriptionInfo on every delivery, so uncached it meant a
rate-limited Search API call for every push and every scan record of
every user with a webhook configured.
DispatchForQuota checks whether the user has any quota webhook at all
before fetching the allowance. The original code filtered first precisely
so the common path (no quota webhooks) did no work; taking the allowance
up front would have spent the expensive lookup on every push. The cap
itself is still computed over the full list, since the count limit spans
all webhook types.
Note DeliverTest is deliberately not capped: it is an explicit,
user-initiated "send test" from the settings page, not automatic delivery.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Evan JarrettandClaude Opus 5
|
2b71be59f7 |
billing: require a managed hold for paid features
Entitlements were keyed on the Stripe subscription alone, so a subscriber
who switched to a self-hosted hold kept paying for features the appview
cannot deliver, and could still reach checkout.
- billing.ActiveHoldChecker and Manager.onManagedHold gate every
entitlement. An empty default hold counts as managed: the user has no
explicit preference and falls back to the operator's primary managed
hold.
- The checker reads the primary DB, not the read replica. A hold switch
writes default_hold_did to the primary, and replica lag would keep
paid features alive after a switch away.
- db.GetUserDefaultHoldDID is the clean default-hold signal, unlike
GetUserHoldDID which falls back to a manifest hold_endpoint (a URL,
not a DID).
- Jetstream fails closed: an unresolvable hold reference is cached raw
rather than left empty, since an empty value reads as managed.
- UI: the billing tab is hidden on self-hosted, a cancel/manage banner
appears when a self-hosted user still has an active plan, the image
advisor returns managed_hold_required instead of upgrade_required,
and the checkout route returns 403. The portal stays open so existing
subscribers can still cancel.
Two consistency fixes fall out of wiring this up:
The settings UI reads the resolved default_hold_did rather than the raw
profile.DefaultHold. The profile field is the record value as written and
may be a URL-form reference; jetstream resolves it to a DID on the way
into the DB, and the server-side gate reads that resolved value. Comparing
the raw form against managed DIDs would show the "you are self-hosted"
banner and hide billing from a user whose entitlements say otherwise.
HasAIAdvisor falls back to the free tier's AIAdvisor setting when
off-managed instead of a hard false, matching GetWebhookLimits. Losing a
managed hold should drop a user to free-tier entitlements, not below them.
BEHAVIOR CHANGE for existing paying users on self-hosted holds: they lose
the AI advisor, supporter badge and paid webhook limits as soon as this
deploys, while Stripe keeps charging them. The only notice is the banner
on /settings/storage, which they have to visit to see. Decide on a
migration (notification, or a one-time reconciliation over active
subscriptions) before shipping this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Evan JarrettandClaude Opus 5
|
12c55ed560 |
billing: make Stripe webhook delivery idempotent and retryable
Webhook delivery was neither idempotent nor order-safe, and every failure
returned 400, which Stripe does not retry. A transient DB or hold error
therefore dropped a subscription change silently and permanently.
- New stripe_processed_events table: event_id as primary key dedups
redelivery, and event_created per customer drops stale out-of-order
deliveries.
- HandleWebhook distinguishes ErrWebhookSignature (400, no retry) from
ErrWebhookProcessing (500, Stripe redelivers). The event handlers
return errors instead of swallowing them. ErrBillingDisabled maps to
400: the route is mounted but billing is off, so redelivery can never
succeed and Stripe should stop rather than retry to exhaustion.
- Refuse to boot when billing is enabled with an empty
STRIPE_WEBHOOK_SECRET. Stripe HMACs with the empty key, so an
attacker can reproduce the signature and the endpoint is forgeable.
- UpdateCrewTierOnAllHolds retries each hold (3 attempts, linear
backoff, 5s per request) and returns a joined error so the webhook
can fail and let Stripe redeliver.
The fan-out contacts holds concurrently rather than in sequence. Serially,
one unreachable hold burns the caller's entire 10s budget on its own
retries (3 x 5s plus backoff) and the holds after it are never contacted;
because Stripe redelivers in the same order, a persistently-down first
hold means the rest are never updated at all.
On the hold, the signature-validated sub claim is now the source of truth
for updateCrewTier: a mismatched body userDid is rejected with 403 rather
than retargeting the grant to another DID. "Not crew on this hold" is a
200 no-op, since the appview fans updates out to every managed hold and a
subscriber is not crew everywhere.
That no-op has to be told apart from a storage failure. GetCrewMember
collapsed both into one generic error, so a CAR-store failure read as
"not a member", answered 200, and let the appview record the event as
processed — losing the tier grant permanently, which is exactly the
failure mode this commit exists to prevent. Missing records now carry an
ErrCrewMemberNotFound sentinel, and anything else returns 500 so Stripe
redelivers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Evan JarrettandClaude Opus 5
|
1b917686b2 |
appview: support OCI manifest DELETE
DELETE /v2/<name>/manifests/<ref> answered UNSUPPORTED before ever
reaching the ATProto-backed stores: distribution v3.1.1's DeleteManifest
handler short-circuits unless app.deleteEnabled is set, which comes from
storage.delete.enabled. Set it (mirrored in the test harness). The
companion storage.EnableDelete option it appends only affects
distribution's built-in store, which RoutingRepository replaces, so it is
a no-op for us.
With the route reachable, make the stores do the right thing:
- ManifestStore.Delete purges the hold's per-layer, scan and image
config records on a detached context, since the DELETE handler
returns immediately and cancels the request context.
- TagStore.Untag resolves the digest before deleting the tag record,
then deletes the manifest if that was its last tag and it is not a
manifest list child, matching the web UI's delete-tag behavior so
deleting an only-tagged image doesn't orphan the manifest.
- cleanupUntaggedManifest becomes package-level over *RegistryContext
so both stores share one implementation.
- purgeOnHold moves out of handlers into pkg/appview/holdpurge so the
storage layer can call it: handlers already depends on storage via
middleware, so storage to handlers would be an import cycle.
- ProxyBlobStore.Delete returns distribution.ErrUnsupported, so the
always-registered blob DELETE route gives a clean OCI UNSUPPORTED
error instead of a generic 500. Layer bytes are reclaimed by the
hold's refcounted GC.
The cascade's still-tagged re-check pages through the tag records rather
than reading a single capped page. Tags for all of a user's repositories
share one collection, so one page is a per-account budget: past ~100 tags
a live tag fell off the end and the manifest was deleted while still
referenced. Incomplete enumeration now skips the delete, since an
orphaned manifest is recoverable and a deleted live one is not.
This also makes the over-quota delete grant added in
|
||
|
Evan JarrettandClaude Opus 5
|
9d4ad84a3e |
auth: surface read-only app passwords as 403, not 503
A read-only app password authenticates fine via createSession but cannot call com.atproto.server.getServiceAuth, which is privileged — the PDS answers 403 InsufficientScope. That fell through to the generic non-200 path and became a 503, which invites the client to retry a request that can never succeed, with no indication of what is actually wrong. Classify it with a sentinel error and map it to a 403 at /auth/token, carrying text that names the fix: use a full-access app password. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
e6959e6dc6 |
auth: bound the PDS and hold HTTP clients on the token path
createSession, the app-password getServiceAuth call, and the hold's /.well-known/atproto-did resolution all used http.DefaultClient, which has no timeout. All three run on the /auth/token path, so a slow or unreachable PDS or hold could hold the request open indefinitely, well past Docker's own token-fetch timeout. Give each a bounded client: 15s for createSession, 10s for the app-password getServiceAuth, 10s for the hold DID fetch. All three are safe to cut off — the two GETs are idempotent, and a timed-out createSession only orphans an unused server-side session. The OAuth refresh path deliberately keeps no overall timeout: its POSTs run through refreshDetachTransport and must not be cancelled mid-rotation, which would strand a rotated refresh token. Note holdDIDResolveClient is package-level in pkg/atproto, so the 10s cap applies to every ResolveHoldDID caller, including the GC, Jetstream backfill and hold-health background workers, not only the token path. That is intended (none of them want an unbounded fetch either), but it is a wider blast radius than the token path alone. This bounds three hops, not the whole request: the OAuth getServiceAuth GET and identity resolution are still unbounded, so /auth/token is not yet fully time-boxed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
cecb8879de |
billing: escape the DID before interpolating it into a Stripe search
findCustomerByDID built its search query with fmt.Sprintf, so the DID was interpolated into a quoted Stripe search string unescaped. DIDs arriving here are OAuth-validated and the DID grammar forbids quotes, so this is not currently exploitable, but the query's safety depended on a validator several layers away rather than on anything visible at the call site. Escape backslash and single quote the way Stripe's search grammar specifies. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
fa34da0f26 |
appview: point the footer Bluesky link at the DID
Handles can change; the DID cannot. Linking the DID keeps the footer correct if the account's handle is ever reassigned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
7d9de7c090 |
admin: resolve handles after limiting top users
The dashboard's top-users panel resolved a handle for every user with a
quota record, then sorted and truncated to ten. On a hold with ~500 crew
that is ~500 serial identity lookups to render ten rows.
Each lookup goes through the shared identity directory, whose HTTP client
allows 10s per request. One stalled lookup consumed the entire reverse
proxy budget, so the panel returned a partial body and the client hung up
mid-render:
admin/auth.go:161 "Failed to render template"
template=partials/top_users.html
error="write: broken pipe"
"GET /admin/api/top-users?limit=10" - 200 4096B in 10.005s
Sort and truncate first, then resolve only the surviving rows, so the
count is bounded by the limit rather than by hold size. Resolve those
concurrently under a 3s deadline: a slow lookup now degrades to a bare
DID instead of taking the whole request down with it.
The crew tab has the same underlying problem in a different shape, one
lazy-load request per row, and is not addressed here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Evan JarrettandClaude Opus 5
|
6e426dc695 |
auth: let over-quota users delete by granting the non-push subset
The quota gate ran on any scope containing "push" and denied the entire token request, so "quota exceeded ... Delete images to free space" named a remedy the gate itself blocked: docker and crane both request pull,push,delete for a manifest delete, and manifest DELETE is bearer-only, so there was no path left to free space. When the request also asks for delete, drop push from the repository entries and issue the reduced token instead of denying. A plain pull,push is still denied so the quota message reaches the client that needs to see it; granting a pushless token there would turn a clear error into an opaque 401 on the first blob upload. The narrowing happens in place on the access slice the handler hands to the issuer, so document that on token.Authorizer along with the ordering the gate goroutine depends on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
c035f50f69 |
appview: delete tag records with the encoded rkey on manifest delete
DeleteManifestHandler built the tag rkey as "repo:tag" while the write path uses RepositoryTagToRKey, which is "repo_tag" with "/" encoded as "~". For a nested repo like stream/cache the two never match, so the cascade leaves the tag record on the PDS while removing the local cache row, and the tag reappears on the next backfill. Depending on the variant it either no-ops (deleteRecord is idempotent) or fails outright on an rkey containing a slash. Every other io.atcr.tag call site already routes through the helper. This was the last hand-built one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
95d4f7c31b |
hold/gc: stop an unreachable predecessor hold from losing its blobs
checkPredecessor returned a bare false on every failure path (DNS failure, dial error, non-200, 5s timeout, unparseable body), indistinguishable from a hold affirmatively answering "I have no successor". isPredecessorHold then cached that false in predecessorCache, which lives for the life of the process and is never reset, so one blip during a single GC run permanently unreferenced that hold's manifests. Those blobs are long past the 7-day grace period that protects recent content, so the next run deleted them outright with nothing to fall back on. checkPredecessor now reports whether its answer is definitive, and only definitive answers are cached. An inconclusive check keeps the hold's manifests referenced and is recorded in predecessorUnresolved, which bounds the cost to one timeout per run rather than one per manifest and is cleared at the start of every analysis so a hold that was down once is re-checked next time instead of written off. This matches the convention the rest of the package already follows: a user whose PDS cannot be reached has their records treated as referenced, never as garbage. An outage must not be the reason content becomes deletable. Non-200 counts as inconclusive on the same reasoning. A reachable service that cannot produce its own captain record is malfunctioning, not answering, and over-protecting an unrelated hold merely leaves some blobs unreclaimed. Splits the fetch-and-parse half into checkPredecessorAt so it can be tested against a local server without depending on DNS. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
e6d3a122f6 |
auth: evict app-password tokens a PDS reports stale, not just on 401
An expired app-password token could wedge an account permanently. The 401 branch clears the cached token, but some PDSes report the same condition as 400 with an atproto error name in the body, which fell through to the generic non-200 branch. That clears only the derived service token, so the dead bearer token stayed in the cache and every subsequent request replayed it. Observed on one account against at.hexlab.foo: 16,110 of these errors and 4,254 retryable 503s over 33 hours, with no recovery path. The cache is in-memory, so it only cleared on process restart. Now the non-200 branch classifies the atproto error name and evicts on the ones that mean the presented token is unusable, matching what the 401 branch already does. For app-passwords that is the equivalent of a refresh: the next authentication re-mints via createSession. Deliberately not routed through oauth.IsSessionInvalidError, which excludes ExpiredToken on purpose — there it would delete a recoverable OAuth session and sign the user out everywhere, whereas here the only thing discarded is a cache entry that will be repopulated. Not addressed here: the failure still surfaces as a 503, which is retryable and so keeps clients looping. Returning 401 with the re-auth hint would be the better signal, but it spans the token handler and is a separate change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
08121f3cd0 |
appview: fix O(n) bcrypt scan making /auth/token take 15s+
ValidateDeviceSecret ran bcrypt.CompareHashAndPassword against every row in the devices table until one matched — no WHERE clause. At bcrypt cost 10 (~65ms on the single-core production host) and 244 registered devices, a device near the end of the scan cost ~15.8s of pure CPU per /auth/token, which is past Docker's client deadline. Measured on production: 15.7-16.0s steady state with the appview pinned at 100% CPU for the duration, while anonymous requests on the same box served in 20ms. The cost grew linearly with every device registered, and the scan ran in rowid order, so the newest devices — the ones most likely to be in active use — paid the most. This is the timeout users were reporting. Devices now carry secret_lookup = hex(sha256(secret)), indexed, and authentication fetches the single matching row. SHA-256 is the verifier here, not merely an index. Device secrets are 32 bytes from crypto/rand, so presenting a value that hashes to a stored digest requires a preimage or a 2^256 search. bcrypt's work factor only helps when the input space is small enough to enumerate, which does not apply to a random 256-bit token, and a database leak exposes no more than before. The plaintext is not recoverable from a bcrypt hash, so existing rows cannot be backfilled directly. They are migrated lazily on their next successful authentication, which any push, pull or login triggers, and the legacy scan is filtered to un-migrated rows so its cost decays as devices migrate. The backfill runs after the cursor is closed: issuing it inside the rows loop deadlocks, because the open cursor holds the connection the write needs. bcrypt now exists solely to carry legacy rows across and can be deleted once the table is fully migrated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
b25aee336b |
auth: serve the OAuth2 POST form at /auth/token
The route was registered GET-only, so containerd and Docker, which try the OAuth2 POST endpoint first whenever they hold a secret, ate a 405 and retried on the GET form. Every authenticated pull paid two auth round trips, and in the production logs the POST share of token traffic grew from 0.5% to 38% over six weeks as more clients pulled from k8s with basic-auth imagePullSecrets. Serve both specs on the same path. After credentials and scope are extracted the two paths are identical, so this is an extraction branch plus a form-shaped error writer. Only grant_type=password is supported and no refresh token is issued: the registry JWT's lifetime is pinned to the AppView<->hold service-auth, so a refresh token would be a fourth long-lived credential with its own storage and revocation. Clients handle its absence by continuing to use the credential they already hold. The refresh grant is refused with 401 rather than the 400 that RFC 6749 5.2 prescribes. containerd sends that grant only when it has no username, which is the same condition that disables its 405 fallback, so a 400 would hard-fail those clients. 401 is on its retry list and routes them to the GET form, where a device secret authenticates off the password alone. That shape previously had no working path at all. resolveService now takes the requested service as an argument, since it arrives in the query string on GET and in the form body on POST. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
05b856bf4e |
hold/pds: fix two scanner-disconnect panics in the scan broadcaster
Same class of bug as the firehose backfill, plus a second one found alongside it. Both take down the whole hold process. 1. send on closed channel. Subscribe spawns drainPendingJobs in its own goroutine, and it sends to sub.send without holding sb.mu. Unsubscribe closed sub.send under the lock, so a scanner disconnecting during the drain closed the channel out from under an in-flight send. The existing `case <-sub.done` guard did not help: done meant "writer goroutine exited" and was closed by handleWriter, which is a different event from unsubscribing. 2. close of closed channel. Unsubscribe closed sub.send unconditionally, but it is called from two places — handleWriter on write error, and handleReader in its defer. A scanner dropping mid-write hits both, and the slice-removal loop had no guard, so the second call fell straight through to the close. The unassign UPDATE ran twice for the same reason, which could return jobs a replacement scanner had already been handed. sub.send is now never closed. done is repurposed to mean "this subscriber is gone", closed only by Unsubscribe and guarded on whether the subscriber was actually still registered. That makes drainPendingJobs' existing done case correct, and handleWriter selects on done rather than ranging over send. dispatchJob was already safe — it sends under sb.mu, which excludes Unsubscribe. hold01 is unaffected in practice (scanner disabled, no shared secret), but seamark-hold runs the scanner continuously and is exposed on any scanner restart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
ca539b1f9d |
hold/pds: stop firehose backfill panicking on subscriber disconnect
A relay that connected with a stale cursor and then dropped mid-backfill took
the whole hold process down:
panic: send on closed channel
pds.sendBackfillMsg events.go:869
pds.backfillFromDatabase events.go:809
pds.backfillSubscriber events.go:716
Subscribe spawns backfillSubscriber in its own goroutine, and that goroutine
writes to sub.send without holding b.mu. Unsubscribe closed sub.send under the
lock, so a disconnect during backfill closed the channel out from under an
in-flight send. select cannot guard that — a send on a closed channel panics
unconditionally.
sub.send is now never closed. Subscriber gains a done channel that Unsubscribe
closes instead, and every sender that runs unlocked selects on it. The map
check in Unsubscribe keeps the close single-shot, which matters because both
readPump and handleSubscriber call it on the way out. handleSubscriber selects
on done rather than ranging over send, since nothing closes send any more.
Broadcast and BroadcastIdentity were already safe (they send under b.mu, which
excludes Unsubscribe). backfillFromMemory was safe too via b.mu.RLock, but now
routes through the shared helper so a disconnect aborts immediately instead of
stalling up to 5s per event while holding the read lock and blocking every
broadcast.
The bug dates to 2025-10, so every build since is affected. It only fires when
a backfill goroutine exists, which Subscribe skips when cursor == currentSeq —
that is why caught-up relays never triggered it and a hold whose relays are all
behind is exposed on every reconnect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Evan JarrettandClaude Opus 5
|
017755c6d4 |
hold/gc: stop the aux-record sweep deleting live records
Follow-up to
|
||
|
Evan JarrettandClaude Opus 4.8
|
3f21cc98b8 |
hold/gc: sweep orphaned scan and image config records
A user who deletes a manifest record directly on their PDS was invisible
to the hold. purgeManifest only fires on appview-driven deletes, and the
GC orphan sweep only walked io.atcr.hold.layer, so scan and image config
records survived forever and the user's storage number never dropped.
analyzeRecords now also walks io.atcr.hold.scan and
io.atcr.hold.image.config, applying the same orphan test the layer sweep
uses: a record dies only when its owning PDS was reachable and
demonstrably lacks the manifest. Unreachable PDS, unparseable URI,
unparseable timestamp, and immature records all keep.
Splitting the grace period was required to make this useful, not
cosmetic. Records only need to outlast a push (blobs and layer records
are written before the manifest reaches the user's PDS), so they age out
at 24h. Blobs stay on the 7 day window, but since records now age out
faster than the blobs they name, a record can no longer serve as its
blob's clock. Blob age comes from S3 LastModified instead, which
WalkBlobs already received from ListObjectsV2 and was discarding.
Net effect: pruning an old manifest frees the user's storage on the next
nightly run rather than never, while the bytes are still reclaimed on
the same best-effort schedule as before.
DeleteManifestAuxRecord is deliberately narrow, accepting only the scan
and image config collections, so a bug in the sweep can't reach captain,
crew, or layer records.
Not addressed: the SBOM and vuln report blobs those scan records point
at. They share the /repos/{holdDID}/blobs/ prefix with the hold's avatar
and OG images, so collecting them needs a referenced-CID set built from
the hold's own records rather than a prefix walk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Evan JarrettandClaude Opus 5
|
2719428071 |
appview: give each registry domain its own JWT service name
An AppView can front several registry domains that all reach the same backend (seamark.dev serving buoy.cr, seamark.cr, and soon atcr.io). Distribution's token access controller holds `service` as a single string and uses it twice: as the value advertised in the WWW-Authenticate challenge, and as the sole accepted JWT audience. So it announced one domain's name on every domain, and honoured one domain's tokens everywhere. A push to seamark.cr was challenged with service="buoy.cr". Both uses sit inside Authorized, which already has the request, but the value is fixed at construction and reachable through no hook — autoredirect only templates the realm. So register an "atcr-token" controller that builds one upstream controller per domain and dispatches on r.Host. Each front door now advertises its own name and demands its own audience. All signature, certificate and claim verification stays in upstream code; this only routes. The token handler stops discarding ?service= and stamps the audience with the front door the client used, allowlist-checked against the configured domains so the value stays server-determined despite arriving from the client. It has to come from the query param because the realm lives on the UI host, where r.Host names no registry domain. This is token hygiene and spec conformance, not a privilege boundary: every domain fronts the same backend, so a client can obtain a token for any of them just by handshaking there. What it buys is a truthful challenge and the decoupling needed to later split a domain onto its own AppView. Also unify the domain list. DomainRoutingMiddleware keyed its map on the raw config while matching a port-stripped host, so a domain configured with a port could never match its own requests. It now shares the normalized cfg.Auth.Services, so routing and authorization agree on one set of names. cfg.Auth.ServiceName was an exact alias for Services[0] and is replaced by PrimaryService(), which also removes an empty-slice index. Rollout: the audience for seamark.cr and bouy.cr changes, so a token minted just before the restart draws one 401 and Docker re-handshakes into a valid one. buoy.cr is unchanged (it stays primary), and atcr.io keeps the service name it already has today. The challenge and the accepted audience come from the same delegate, so the retry converges by construction. Deploy as a single flip, not a canary: an old instance ignores ?service= and would keep minting the primary audience while a new one rejects it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Evan JarrettandClaude Opus 5
|
3298797603 |
appview: resolve managed hold names off the privacy render path
Follow-up to
|
||
|
Evan JarrettandClaude Opus 5
|
500ee2f8d1 |
auth: classify service-token failures structurally, not by string
Follow-up to |
||
|
Evan JarrettandClaude Opus 5
|
c615d7253b |
credhelper: make config writes non-destructive
Follow-up to
|