io.atcr.manifest rkeys are the digest alone (digestToRKey), so a single
record backs every repository of a user holding identical content. Both
paths that cascade-delete that record checked for remaining tags scoped to
one repository, which asks the wrong question: a tag in another repo keeps
the shared record alive just as much as a tag in this one.
With me/a:v1 and me/b:v1 at the same digest, deleting me/a:v1 saw no
remaining tags in repo a, deleted the shared PDS record, and purged the
layers on the hold. me/b:v1 was left pointing at content that no longer
exists, and the firehose delete handler then cleared the rows for every
repo (DeleteManifest with an empty repository argument).
The collision predates this, but it was reachable only behind the opt-in
AutoRemoveUntagged profile flag. 1b91768 enabled OCI manifest DELETE and
made TagStore.Untag cascade unconditionally, which turned a latent metadata
collision into blob-level data loss on an ordinary skopeo/crane delete.
- cleanupUntaggedManifest no longer filters candidate tags to
rctx.Repository. The surrounding comment already noted that the tag
collection is account-wide; the filter contradicted it.
- ShouldCascadeDeleteManifest takes the tag question DID-wide via a new
IsManifestTaggedAnyRepo, and drops its now-meaningless repository
parameter.
- IsManifestTagged stays repository-scoped and keeps its caller: the
delete-manifest confirmation prompt is genuinely asking about the one
repo whose tags the user is about to remove.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two migration files shipped as version 28: 0028_add_device_secret_lookup
(08121f3) and 0028_create_stripe_processed_events (12c55ed).
runMigrations keys applied migrations by the integer parsed from the
filename and skips any version already present in schema_migrations, so
the first file to load records 28 and the second is skipped in silence —
no error, no log. loadMigrations enumerates via fs.Glob, which sorts
lexically, so add_device_secret_lookup won and stripe_processed_events
never ran.
On an existing database that leaves stripe_processed_events missing, and
StripeEventSeen then fails closed: the error wraps into ErrWebhookProcessing,
the webhook returns 500, and Stripe redelivers into the same missing table
forever. No subscription, tier, or dispute event is ever applied — defeating
the exact idempotency 12c55ed was written to add.
Fresh installs were unaffected, which is why no test caught it: they take
the applySchema path where schema.sql already has the table and both
version-28 rows are merely recorded as applied.
Renumbered to 0029 rather than renumbering the device migration, so a
database that already ran this build (28 recorded, devices.secret_lookup
present, stripe table missing) picks the migration up on next boot instead
of staying broken. Renumbering the other file would have left that database
with the table still missing and re-run its ALTER on a column that exists.
Adds two guards: one asserting migration versions are distinct, and one
exercising the upgrade path where a duplicate manifests as a missing
schema_migrations row. Both fail on a planted duplicate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The webhook limit was only checked at creation, so losing entitlement (a
hold switch or a plan downgrade) left previously-created webhooks firing
paid behavior forever.
- Dispatcher takes a WebhookLimiter, consulted on every dispatch. It
caps the list to the current allowance, keeping the oldest N to match
what the creation gate would have permitted, and masks paid trigger
bits.
- GetWebhooksForUser orders by created_at ASC, id ASC so that cap is
deterministic. ListWebhooks gets the same tiebreak: it feeds the
settings UI, and without it the list a user sees could disagree with
the one the dispatcher truncates.
- webhooks.FreeTriggerMask is shared by the creation gate and the
dispatch backstop so the two cannot drift.
Capping is logged when it actually truncates. The webhooks stay visible in
settings, so from the user's side delivery would otherwise just stop with
no signal — and the same line is the only evidence if the limiter itself
degraded, since a billing lookup failure falls back to free-tier limits
and would quietly demote a paying user mid-dispatch.
Two cost fixes, both because this puts the entitlement lookup on a hot
path it was never on before:
findCustomerByDID now consults the customer cache instead of always
issuing a Stripe customer search. GetWebhookLimits reaches it via
GetSubscriptionInfo on every delivery, so uncached it meant a
rate-limited Search API call for every push and every scan record of
every user with a webhook configured.
DispatchForQuota checks whether the user has any quota webhook at all
before fetching the allowance. The original code filtered first precisely
so the common path (no quota webhooks) did no work; taking the allowance
up front would have spent the expensive lookup on every push. The cap
itself is still computed over the full list, since the count limit spans
all webhook types.
Note DeliverTest is deliberately not capped: it is an explicit,
user-initiated "send test" from the settings page, not automatic delivery.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Entitlements were keyed on the Stripe subscription alone, so a subscriber
who switched to a self-hosted hold kept paying for features the appview
cannot deliver, and could still reach checkout.
- billing.ActiveHoldChecker and Manager.onManagedHold gate every
entitlement. An empty default hold counts as managed: the user has no
explicit preference and falls back to the operator's primary managed
hold.
- The checker reads the primary DB, not the read replica. A hold switch
writes default_hold_did to the primary, and replica lag would keep
paid features alive after a switch away.
- db.GetUserDefaultHoldDID is the clean default-hold signal, unlike
GetUserHoldDID which falls back to a manifest hold_endpoint (a URL,
not a DID).
- Jetstream fails closed: an unresolvable hold reference is cached raw
rather than left empty, since an empty value reads as managed.
- UI: the billing tab is hidden on self-hosted, a cancel/manage banner
appears when a self-hosted user still has an active plan, the image
advisor returns managed_hold_required instead of upgrade_required,
and the checkout route returns 403. The portal stays open so existing
subscribers can still cancel.
Two consistency fixes fall out of wiring this up:
The settings UI reads the resolved default_hold_did rather than the raw
profile.DefaultHold. The profile field is the record value as written and
may be a URL-form reference; jetstream resolves it to a DID on the way
into the DB, and the server-side gate reads that resolved value. Comparing
the raw form against managed DIDs would show the "you are self-hosted"
banner and hide billing from a user whose entitlements say otherwise.
HasAIAdvisor falls back to the free tier's AIAdvisor setting when
off-managed instead of a hard false, matching GetWebhookLimits. Losing a
managed hold should drop a user to free-tier entitlements, not below them.
BEHAVIOR CHANGE for existing paying users on self-hosted holds: they lose
the AI advisor, supporter badge and paid webhook limits as soon as this
deploys, while Stripe keeps charging them. The only notice is the banner
on /settings/storage, which they have to visit to see. Decide on a
migration (notification, or a one-time reconciliation over active
subscriptions) before shipping this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Webhook delivery was neither idempotent nor order-safe, and every failure
returned 400, which Stripe does not retry. A transient DB or hold error
therefore dropped a subscription change silently and permanently.
- New stripe_processed_events table: event_id as primary key dedups
redelivery, and event_created per customer drops stale out-of-order
deliveries.
- HandleWebhook distinguishes ErrWebhookSignature (400, no retry) from
ErrWebhookProcessing (500, Stripe redelivers). The event handlers
return errors instead of swallowing them. ErrBillingDisabled maps to
400: the route is mounted but billing is off, so redelivery can never
succeed and Stripe should stop rather than retry to exhaustion.
- Refuse to boot when billing is enabled with an empty
STRIPE_WEBHOOK_SECRET. Stripe HMACs with the empty key, so an
attacker can reproduce the signature and the endpoint is forgeable.
- UpdateCrewTierOnAllHolds retries each hold (3 attempts, linear
backoff, 5s per request) and returns a joined error so the webhook
can fail and let Stripe redeliver.
The fan-out contacts holds concurrently rather than in sequence. Serially,
one unreachable hold burns the caller's entire 10s budget on its own
retries (3 x 5s plus backoff) and the holds after it are never contacted;
because Stripe redelivers in the same order, a persistently-down first
hold means the rest are never updated at all.
On the hold, the signature-validated sub claim is now the source of truth
for updateCrewTier: a mismatched body userDid is rejected with 403 rather
than retargeting the grant to another DID. "Not crew on this hold" is a
200 no-op, since the appview fans updates out to every managed hold and a
subscriber is not crew everywhere.
That no-op has to be told apart from a storage failure. GetCrewMember
collapsed both into one generic error, so a CAR-store failure read as
"not a member", answered 200, and let the appview record the event as
processed — losing the tier grant permanently, which is exactly the
failure mode this commit exists to prevent. Missing records now carry an
ErrCrewMemberNotFound sentinel, and anything else returns 500 so Stripe
redelivers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ValidateDeviceSecret ran bcrypt.CompareHashAndPassword against every row in
the devices table until one matched — no WHERE clause. At bcrypt cost 10
(~65ms on the single-core production host) and 244 registered devices, a
device near the end of the scan cost ~15.8s of pure CPU per /auth/token,
which is past Docker's client deadline. Measured on production: 15.7-16.0s
steady state with the appview pinned at 100% CPU for the duration, while
anonymous requests on the same box served in 20ms.
The cost grew linearly with every device registered, and the scan ran in
rowid order, so the newest devices — the ones most likely to be in active
use — paid the most. This is the timeout users were reporting.
Devices now carry secret_lookup = hex(sha256(secret)), indexed, and
authentication fetches the single matching row.
SHA-256 is the verifier here, not merely an index. Device secrets are 32
bytes from crypto/rand, so presenting a value that hashes to a stored digest
requires a preimage or a 2^256 search. bcrypt's work factor only helps when
the input space is small enough to enumerate, which does not apply to a
random 256-bit token, and a database leak exposes no more than before.
The plaintext is not recoverable from a bcrypt hash, so existing rows cannot
be backfilled directly. They are migrated lazily on their next successful
authentication, which any push, pull or login triggers, and the legacy scan
is filtered to un-migrated rows so its cost decays as devices migrate. The
backfill runs after the cursor is closed: issuing it inside the rows loop
deadlocks, because the open cursor holds the connection the write needs.
bcrypt now exists solely to carry legacy rows across and can be deleted once
the table is fully migrated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- diff view gains a Packages tab with added/removed/changed/unchanged
package tables and purl-derived type/license/upstream links
- captain records verified against the DID's atcr_hold service before
caching (processor + batch backfill), preventing forged holds
- fix empty-handle updates clobbering cached handles and colliding on
the UNIQUE constraint
- move fillPrevCIDs into repo.go; DirectRepoOperator is now canonical,
repomgr kept as a test oracle
- surface read-only crew status in hold selector
- reconcile docs
1. Multiple registry domains + per-user domain preference
The biggest feature. The appview can serve several registry domains (e.g. buoy.cr, atcr.io),
and users can now pick which one shows up in their pull/push commands.
- Lexicon/record: adds registryDomain (and documents ociClient)
to the sailor profile (lexicons/.../profile.json, pkg/atproto/lexicon.go).
- DB: new registry_domain column on users (schema.sql + migration 0027),
with GetUserByDID/Handle reads, UpdateUserRegistryDomain writer,
and Jetstream caching it on profile updates (writes unconditionally so clearing propagates).
- UI/handlers: new UpdateRegistryDomainHandler + /api/profile/registry-domain route,
a <select> in the user settings panel (only shown when >1 domain configured), and resolveRegistryURL()
which falls back to the primary domain if the user's pref is stale/removed. Tests added for all of it.
2. default_hold_did removed → first managed_holds entry is the default
Consolidates two overlapping config fields into one. ServerConfig.DefaultHoldDID is gone;
PrimaryHoldDID() now returns managed_holds[0]. managed_holds is now REQUIRED.
Updated in config, validation, server wiring, test harness, example YAML, and the deploy template.
3. Admin long-running operations → generic background-job framework
New pkg/hold/admin/jobs.go introduces a reusable startJob/jobRegistry pattern
(a detached context.Background() job + a /admin/api/jobs/{key}/status polling endpoint).
This replaces the bespoke scan-backfill goroutine state machine, and now also wraps crew tier remap and crew import
all three previously looped synchronously on the request context and got 504'd/cancelled mid-run by the reverse proxy.
Forms switched from POST-redirect to htmx fragments (job_progress.html, job_result.html, crew_import_results.html)
the old crew_import_results.html page and scan_backfill_progress.html partial were deleted.
This is also captured as a new rule in CLAUDE.md.
4. Cascade-delete manifest on last-tag deletion
DeleteTagHandler now, after removing the last tag pointing to a digest, cascade-deletes the manifest itself
(PDS + DB + hold blob purge) — but only if it's not a child of a manifest list (multi-arch parent).
New GetTagDigest and ShouldCascadeDeleteManifest queries back it, plus cascade_delete_test.go.
Also switches tag rkey computation to the atproto.RepositoryTagToRKey helper.
5. Billing simplification
Drops the OwnerBadge config option (hold-owner supporter badge).
The user-profile template no longer special-cases an "owner" badge value (only "Captain").
Example tiers renamed to the nautical scheme (deckhand/bosun/quartermaster).
6. Build/deploy: go generate always runs via Make
make generate is now a phony target that always runs go generate ./... (regenerating cbor_gen, icon sprites, etc.),
and build-trixie depends on it. The deploy tooling (provision.go/update.go)
drops its own runGenerate calls since the Makefile handles it.
7. New cmd/firehose-tap tool (untracked)
A standalone CLI that subscribes to a com.atproto.sync.subscribeRepos endpoint and pretty-prints events,
with emphasis on Sync 1.1 compliance fields (per-op prev CIDs, commit prevData) and a --validate CI mode.
Fits with the recent "more sync1.1 compliant" commit.
1. Removing distribution/distribution from the Hold Service (biggest change)
The hold service previously used distribution's StorageDriver interface for all blob operations. This replaces it with direct AWS SDK v2 calls through ATCR's own pkg/s3.S3Service:
- New S3Service methods: Stat(), PutBytes(), Move(), Delete(), WalkBlobs(), ListPrefix() added to pkg/s3/types.go
- Pull zone fix: Presigned URLs are now generated against the real S3 endpoint, then the host is swapped to the CDN URL post-signing (previously the CDN URL was set as the endpoint, which
broke SigV4 signatures)
- All hold subsystems migrated: GC, OCI uploads, XRPC handlers, profile uploads, scan broadcaster, manifest posts — all now use *s3.S3Service instead of storagedriver.StorageDriver
- Config simplified: Removed configuration.Storage type and buildStorageConfigFromFields(); replaced with a simple S3Params() method
- Mock expanded: MockS3Client gains an in-memory object store + 5 new methods, replacing duplicate mockStorageDriver implementations in tests (~160 lines deleted from each test file)
2. Vulnerability Scan UI in AppView (new feature)
Displays scan results from the hold's PDS on the repository page:
- New lexicon: io/atcr/hold/scan.json with vulnReportBlob field for storing full Grype reports
- Two new HTMX endpoints: /api/scan-result (badge) and /api/vuln-details (modal with CVE table)
- New templates: vuln-badge.html (severity count chips) and vuln-details.html (full CVE table with NVD/GHSA links)
- Repository page: Lazy-loads scan badges per manifest via HTMX
- Tests: ~590 lines of test coverage for both handlers
3. S3 Diagnostic Tool
New cmd/s3-test/main.go (418 lines) — tests S3 connectivity with both SDK v1 and v2, including presigned URL generation, pull zone host swapping, and verbose signing debug output.
4. Deployment Tooling
- New syncServiceUnit() for comparing/updating systemd units on servers
- Update command now syncs config keys (adds missing keys from template) and service units with daemon-reload
5. DB Migration
0011_fix_captain_successor_column.yaml — rebuilds hold_captain_records to add the successor column that was missed in a previous migration.
6. Documentation
- APPVIEW-UI-FUTURE.md rewritten as a status-tracked feature inventory
- DISTRIBUTION.md renamed to CREDENTIAL_HELPER.md
- New REMOVING_DISTRIBUTION.md — 480-line analysis of fully removing distribution from the appview side
7. go.mod
aws-sdk-go v1 moved from indirect to direct (needed by cmd/s3-test).
Evan JarrettandClaude Opus 5