113 Commits
Author SHA1 Message Date
Evan JarrettandClaude Opus 5 e6959e6dc6 auth: bound the PDS and hold HTTP clients on the token path
createSession, the app-password getServiceAuth call, and the hold's
/.well-known/atproto-did resolution all used http.DefaultClient, which
has no timeout. All three run on the /auth/token path, so a slow or
unreachable PDS or hold could hold the request open indefinitely, well
past Docker's own token-fetch timeout.

Give each a bounded client: 15s for createSession, 10s for the
app-password getServiceAuth, 10s for the hold DID fetch. All three are
safe to cut off — the two GETs are idempotent, and a timed-out
createSession only orphans an unused server-side session.

The OAuth refresh path deliberately keeps no overall timeout: its POSTs
run through refreshDetachTransport and must not be cancelled mid-rotation,
which would strand a rotated refresh token.

Note holdDIDResolveClient is package-level in pkg/atproto, so the 10s cap
applies to every ResolveHoldDID caller, including the GC, Jetstream
backfill and hold-health background workers, not only the token path.
That is intended (none of them want an unbounded fetch either), but it is
a wider blast radius than the token path alone.

This bounds three hops, not the whole request: the OAuth getServiceAuth
GET and identity resolution are still unbounded, so /auth/token is not
yet fully time-boxed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 20:52:13 -05:00
Evan Jarrett 6758996300 add SBOM package diffing, verify hold-service captain records
- diff view gains a Packages tab with added/removed/changed/unchanged
  package tables and purl-derived type/license/upstream links
- captain records verified against the DID's atcr_hold service before
  caching (processor + batch backfill), preventing forged holds
- fix empty-handle updates clobbering cached handles and colliding on
  the UNIQUE constraint
- move fillPrevCIDs into repo.go; DirectRepoOperator is now canonical,
  repomgr kept as a test oracle
- surface read-only crew status in hold selector
- reconcile docs
2026-06-13 12:49:03 -05:00
Evan Jarrett ab4a4ebf9d admin panel long running imrovements, billing fixes, ui cleanup
1. Multiple registry domains + per-user domain preference

The biggest feature. The appview can serve several registry domains (e.g. buoy.cr, atcr.io),
and users can now pick which one shows up in their pull/push commands.

- Lexicon/record: adds registryDomain (and documents ociClient)
to the sailor profile (lexicons/.../profile.json, pkg/atproto/lexicon.go).
- DB: new registry_domain column on users (schema.sql + migration 0027),
with GetUserByDID/Handle reads, UpdateUserRegistryDomain writer,
and Jetstream caching it on profile updates (writes unconditionally so clearing propagates).
- UI/handlers: new UpdateRegistryDomainHandler + /api/profile/registry-domain route,
a <select> in the user settings panel (only shown when >1 domain configured), and resolveRegistryURL()
which falls back to the primary domain if the user's pref is stale/removed. Tests added for all of it.

2. default_hold_did removed → first managed_holds entry is the default

Consolidates two overlapping config fields into one. ServerConfig.DefaultHoldDID is gone;
PrimaryHoldDID() now returns managed_holds[0]. managed_holds is now REQUIRED.
Updated in config, validation, server wiring, test harness, example YAML, and the deploy template.

3. Admin long-running operations → generic background-job framework

New pkg/hold/admin/jobs.go introduces a reusable startJob/jobRegistry pattern
 (a detached context.Background() job + a /admin/api/jobs/{key}/status polling endpoint).
This replaces the bespoke scan-backfill goroutine state machine, and now also wraps crew tier remap and crew import
all three previously looped synchronously on the request context and got 504'd/cancelled mid-run by the reverse proxy.
 Forms switched from POST-redirect to htmx fragments (job_progress.html, job_result.html, crew_import_results.html)
 the old crew_import_results.html page and scan_backfill_progress.html partial were deleted.
This is also captured as a new rule in CLAUDE.md.

4. Cascade-delete manifest on last-tag deletion

DeleteTagHandler now, after removing the last tag pointing to a digest, cascade-deletes the manifest itself
 (PDS + DB + hold blob purge) — but only if it's not a child of a manifest list (multi-arch parent).
 New GetTagDigest and ShouldCascadeDeleteManifest queries back it, plus cascade_delete_test.go.
 Also switches tag rkey computation to the atproto.RepositoryTagToRKey helper.

5. Billing simplification

Drops the OwnerBadge config option (hold-owner supporter badge).
The user-profile template no longer special-cases an "owner" badge value (only "Captain").
Example tiers renamed to the nautical scheme (deckhand/bosun/quartermaster).

6. Build/deploy: go generate always runs via Make

make generate is now a phony target that always runs go generate ./... (regenerating cbor_gen, icon sprites, etc.),
 and build-trixie depends on it. The deploy tooling (provision.go/update.go)
drops its own runGenerate calls since the Makefile handles it.

7. New cmd/firehose-tap tool (untracked)

A standalone CLI that subscribes to a com.atproto.sync.subscribeRepos endpoint and pretty-prints events,
with emphasis on Sync 1.1 compliance fields (per-op prev CIDs, commit prevData) and a --validate CI mode.
Fits with the recent "more sync1.1 compliant" commit.
2026-06-05 20:57:25 -05:00
Evan Jarrett efabb677e4 clean up some functions to use indigo helpers. make repomgr more sync1.1 compliant 2026-05-26 20:26:29 -05:00
Evan Jarrett 63909aaca0 add webhook for storage quota percentage 2026-05-17 19:57:09 -05:00
Evan Jarrett b5495af2b6 billing bugfixes 2026-05-16 19:39:57 -05:00
Evan Jarrett 902fba4553 convert alert to modal. go fix the codebase 2026-05-16 11:59:47 -05:00
Evan Jarrett a0cc862798 lots of new work for authenticating between appview -> hold. fixed quota handling, improve integration tests for round-trip push/pull/auth/quota checking 2026-05-11 09:20:55 -05:00
Evan Jarrett b2d6842bb7 clean up old migration code. minor bug fixes with appview ui 2026-05-04 21:52:28 -05:00
Evan Jarrett 4328eda814 holds now listen for deletes and labelers for takedowns. GC will defer takedowns for a grace period in case of reversal 2026-05-02 23:31:41 -05:00
Evan Jarrett ab66028151 more labeler improvements. standardize did work between labeler and hold. improve sql race conditions on local-only db 2026-05-02 22:13:53 -05:00
Evan Jarrett 13a793ca90 improve admin tooling 2026-04-29 10:37:36 -05:00
Evan Jarrett 9af6eccc9d improvements to how scanning works, and helmchart ui 2026-04-29 10:12:25 -05:00
Evan Jarrett f9ba8ff62b pass through 429 retry-after from pds 2026-04-29 08:50:54 -05:00
Evan Jarrett 9809c26281 update fonts 2026-04-14 20:56:53 -05:00
Evan Jarrett 564019d1c3 general appview bugfixes 2026-04-09 10:31:19 -05:00
Evan Jarrett fd5bfc3c50 ui fixes for repo page, fix scanner priority, cleanup goreleaser scripts 2026-04-03 16:48:21 -05:00
Evan Jarrett 22cf108a5f fix quirks on repo and digest pages. fix ips not showing in server logs. add basic spam blocking to LB. add setting to configure your oci (docker) client. 2026-03-24 22:24:51 -05:00
Evan Jarrett 23db9be665 add repo page editor. fix deleting all untagged actually deleting all untagged 2026-03-23 21:16:13 -05:00
Evan Jarrett 385f8987fe overhaul repo pages, add tab for 'artifacts' (tags, manifests, helm charts). implement digest page with layer commands and vuln reports 2026-03-22 21:10:47 -05:00
Evan Jarrett 8adbc7505f fix up lexicons and remvoe unused endpoints 2026-03-21 10:51:50 -05:00
Evan Jarrett e886192aeb update seamark theme, add 'delete all untagged' option on record page. add garbage collection flag for untagged 2026-03-16 20:26:56 -05:00
Evan Jarrett 8fb69497e3 add zlay 2026-03-09 21:36:17 -05:00
Evan Jarrett 11a8be1413 upcloud provision fixes and relay tweaks 2026-03-01 20:52:41 -06:00
Evan Jarrett 136c0a0ecc billing refactor, move billing to appview, move webhooks to appview 2026-02-26 22:28:09 -06:00
Evan Jarrett dc31ca2f35 more work on webhook, implement getMetadata endpoint for appview and link holds to a preferred appview 2026-02-22 22:49:33 -06:00
Evan Jarrett 2b9ea997ac fix tier and supporter badge assignments. normalize did:web adresses with ports. various minor fixes 2026-02-22 11:16:55 -06:00
Evan Jarrett 356f9d529a actually check if the requestCrawl endpoint exists via HEAD 2026-02-21 14:24:37 -06:00
Evan Jarrett f90a46e0a4 begin implement supporter badges, clean up lexicons, various other changes 2026-02-20 22:12:18 -06:00
Evan Jarrett 0d00de76c6 implement HandleGetLatestCommit 2026-02-18 21:52:21 -06:00
Evan Jarrett 27cf78158b vuln scanner fixes, major refactor of the credential helper. 2026-02-17 22:38:25 -06:00
Evan Jarrett 2df5377541 more did:plc fixes, more vulnerability scanner fixes 2026-02-15 22:28:36 -06:00
Evan Jarrett abefcfd1ed let appview work with did:plc based storage servers 2026-02-15 14:20:02 -06:00
Evan Jarrett de02e1f046 remove distribution from hold, add vulnerability scanning in appview.
1. Removing distribution/distribution from the Hold Service (biggest change)
  The hold service previously used distribution's StorageDriver interface for all blob operations. This replaces it with direct AWS SDK v2 calls through ATCR's own pkg/s3.S3Service:
  - New S3Service methods: Stat(), PutBytes(), Move(), Delete(), WalkBlobs(), ListPrefix() added to pkg/s3/types.go
  - Pull zone fix: Presigned URLs are now generated against the real S3 endpoint, then the host is swapped to the CDN URL post-signing (previously the CDN URL was set as the endpoint, which
  broke SigV4 signatures)
  - All hold subsystems migrated: GC, OCI uploads, XRPC handlers, profile uploads, scan broadcaster, manifest posts — all now use *s3.S3Service instead of storagedriver.StorageDriver
  - Config simplified: Removed configuration.Storage type and buildStorageConfigFromFields(); replaced with a simple S3Params() method
  - Mock expanded: MockS3Client gains an in-memory object store + 5 new methods, replacing duplicate mockStorageDriver implementations in tests (~160 lines deleted from each test file)
2. Vulnerability Scan UI in AppView (new feature)
  Displays scan results from the hold's PDS on the repository page:
  - New lexicon: io/atcr/hold/scan.json with vulnReportBlob field for storing full Grype reports
  - Two new HTMX endpoints: /api/scan-result (badge) and /api/vuln-details (modal with CVE table)
  - New templates: vuln-badge.html (severity count chips) and vuln-details.html (full CVE table with NVD/GHSA links)
  - Repository page: Lazy-loads scan badges per manifest via HTMX
  - Tests: ~590 lines of test coverage for both handlers
3. S3 Diagnostic Tool
  New cmd/s3-test/main.go (418 lines) — tests S3 connectivity with both SDK v1 and v2, including presigned URL generation, pull zone host swapping, and verbose signing debug output.
4. Deployment Tooling
  - New syncServiceUnit() for comparing/updating systemd units on servers
  - Update command now syncs config keys (adds missing keys from template) and service units with daemon-reload
5. DB Migration
  0011_fix_captain_successor_column.yaml — rebuilds hold_captain_records to add the successor column that was missed in a previous migration.
6. Documentation
  - APPVIEW-UI-FUTURE.md rewritten as a status-tracked feature inventory
  - DISTRIBUTION.md renamed to CREDENTIAL_HELPER.md
  - New REMOVING_DISTRIBUTION.md — 480-line analysis of fully removing distribution from the appview side
7. go.mod
  aws-sdk-go v1 moved from indirect to direct (needed by cmd/s3-test).
2026-02-13 15:26:24 -06:00
Evan Jarrett 92c31835e2 implement the ability to promote a hold as a successor as a way to migrate users to a new storage server 2026-02-12 20:14:19 -06:00
Evan Jarrett aad9ebfc8b fix lint and unit tests 2026-02-09 22:39:38 -06:00
Evan Jarrett 7ba42080c5 more admin panel fixes, allow for fallback relays and jetstreams, improve star lexicon to allow for repo_page backlinks 2026-02-09 21:53:02 -06:00
Evan Jarrett 2c39a78ac2 minor fixes 2026-02-04 20:14:25 -06:00
Evan Jarrett 73109641e8 add scan reports to hold pds 2026-02-04 10:25:09 -06:00
Evan Jarrett d6114cf549 implementation of syft/grype scanner as a separate binary 2026-02-04 09:53:04 -06:00
Evan Jarrett 35f7a47af3 add simple stripe billing implementation for quotas 2026-02-03 21:52:31 -06:00
Evan Jarrett 34f342f637 lots of refactor and cleanup to allow for branding overrides 2026-02-02 22:42:15 -06:00
Evan Jarrett eb3eed5f7a lint, fix repo-card styling 2026-01-14 23:18:35 -06:00
Evan Jarrett 055b34af71 varies fixes for indigo xrpc calls, avatars broken on bsku profile change, opengraph card fixes, other ui improvements 2026-01-14 23:14:43 -06:00
Evan Jarrett 64cdb66957 begin delete my account implementation 2026-01-08 23:17:38 -06:00
Evan Jarrett 3409af6c67 implement hold discovery dropdown in settings. implement a data privacy export feature 2026-01-07 22:41:14 -06:00
Evan Jarrett e0a2dda1af add ability to toggle debug. refactor hold pds logic to allow crew record lookups by rkey rather than a list 2026-01-06 12:48:13 -06:00
Evan Jarrett 487fc8a47e wording 2026-01-04 23:37:31 -06:00
Evan Jarrett af815fbc7d use for range and wg.Go 2026-01-04 22:39:48 -06:00
Evan Jarrett efef46b15a various linting fixes 2026-01-04 22:02:01 -06:00