mirror of https://tangled.org/evan.jarrett.net/at-container-registry synced 2026-09-24 19:24:16 +00:00

Files

T

Evan JarrettandClaude Opus 5 a63f668de0 scanner: fix five crash and halt classes found by a pipeline audit

An audit of the scan pipeline and the hold side of scanning found several
ways scanning stops without saying so. Each fix here was written test-first:
a test expressing the wanted behaviour, confirmed failing for the right
reason, then the change.

A summary-less result crash-looped both processes. worker.go dereferenced
result.Summary unconditionally, but processJob only sets it when Grype runs,
and SendResult puts the nil on the wire before the scanner dies on it, so
handleResult's unguarded log killed the hold too. A nil Summary now means
"not scanned for vulnerabilities", deliberately distinct from "scanned, found
zero" — inventing a zeroed summary would report every image as clean when
Grype never ran. The hold writes a record rather than orphaning the uploaded
SBOM, and the appview renders an "SBOM only" state instead of a green Clean
badge.

The Grype database could wedge with no way back short of a restart. All three
throttles in loadVulnDatabase were guarded by vulnDB != nil, so a scanner
holding no provider retried a full download on every scan under the exclusive
lock. Two earlier attempts at this bug each added one more condition to the
same chain; this replaces the chain with a single decision function over a
state snapshot, consulted by both call sites so they cannot disagree. That
disagreement was itself a bug: the 50-scan reload had never once executed.

Two independent halts. An unparseable frame was dropped in silence, stranding
a row that held the hold's only dispatch slot forever; it is now answered
"skipped" on first delivery. The 10-minute sweep leaked the in-flight digest
and wrote no record, permanently retiring one image per timeout.

A digest went unvalidated into filepath.Join and os.Create, so a layer digest
of sha256:../../../x wrote outside the scan directory, and nothing verified
that downloaded bytes hashed to the digest naming them. Digests come from
records in a user's own PDS. Both are fixed together: verification is what
makes an escaping write self-defeating.

Concurrency did not work on either axis. The proactive capacity gate was
depth-one hold-wide, so neither extra workers nor extra scanner processes
received work. Depth is now the sum of the worker counts scanners advertise on
connect, the gate is scoped to proactive work, and dispatch prefers the
least-loaded scanner. Disconnects no longer hand a running scan to someone
else: a scanner keeps a stable per-process identity and reclaims its own rows
within a grace window, while a process that truly restarted returns with a new
identity and has its work reclaimed, which is correct because the restart did
lose it.

The hold's scanning deadline measured queueing rather than scanning, because
the scanner acks on receipt and handleAck never refreshed assigned_at. A new
"started" message, sent by the worker that dequeues the job, separates the two
budgets. An older scanner never sends it and falls under the queueing budget,
which is more forgiving than the deadline it gets today.

Adds an in-process mock hold and an e2e harness that runs the real client,
queue and worker pool, seeded with 84 real manifest records fetched from a
live PDS. Real image layouts and the Grype database are fetched by scripts and
gitignored; suites needing them skip cleanly, so the default run stays offline
and fast.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF

2026-09-05 15:01:10 -05:00

7.7 KiB

Raw Blame History

Hold Service XRPC Endpoints

This document lists all XRPC endpoints implemented in the Hold service (pkg/hold/).

PDS Endpoints (`pkg/hold/pds/xrpc.go`)

Public (No Auth Required)

Endpoint	Method	Description
`/xrpc/_health`	GET	Health check
`/xrpc/com.atproto.server.describeServer`	GET	Server metadata
`/xrpc/com.atproto.repo.describeRepo`	GET	Repository information
`/xrpc/com.atproto.repo.getRecord`	GET	Retrieve a single record
`/xrpc/com.atproto.repo.listRecords`	GET	List records in a collection (paginated)
`/xrpc/com.atproto.sync.listBlobs`	GET	List blob CIDs for an account
`/xrpc/com.atproto.sync.listRepos`	GET	List all repositories
`/xrpc/com.atproto.sync.getRecord`	GET	Get record as CAR file
`/xrpc/com.atproto.sync.getRepo`	GET	Full repository as CAR file
`/xrpc/com.atproto.sync.getRepoStatus`	GET	Repository hosting status
`/xrpc/com.atproto.sync.getLatestCommit`	GET	Current commit CID and revision
`/xrpc/com.atproto.sync.subscribeRepos`	GET	WebSocket firehose
`/xrpc/com.atproto.identity.resolveHandle`	GET	Resolve handle to DID
`/xrpc/app.bsky.actor.getProfile`	GET	Get actor profile
`/xrpc/app.bsky.actor.getProfiles`	GET	Get multiple profiles
`/xrpc/io.atcr.hold.listTiers`	GET	List hold's available tiers with quotas and features
`/.well-known/did.json`	GET	DID document
`/.well-known/atproto-did`	GET	DID for handle resolution

Conditional Auth (based on captain.public)

Endpoint	Method	Description
`/xrpc/com.atproto.sync.getBlob`	GET/HEAD	Get blob (routes OCI vs ATProto)

Owner/Crew Admin Required

Endpoint	Method	Description
`/xrpc/com.atproto.repo.deleteRecord`	POST	Delete a record
`/xrpc/com.atproto.repo.uploadBlob`	POST	Upload ATProto blob

Inline Auth (per-manifest caller check)

/xrpc/io.atcr.hold.purgeManifest (POST) does not use a router middleware. Auth is validated inline by ValidateManifestPurger, which accepts either a Bearer service token or a DPoP token, then checks the caller's role:

Hold captain (any manifest)
Crew member with crew:admin permission (any manifest)
Crew member whose DID matches the manifest URI's DID (own manifests only)

Idempotent. Does not delete S3 blobs — GC handles those.

Auth Required (Service Token)

The requireAuth middleware validates Bearer service tokens only. requestCrew additionally accepts DPoP tokens when called directly (the handler falls back to ValidateDPoPRequest if no user is in context).

Endpoint	Method	Description
`/xrpc/io.atcr.hold.requestCrew`	POST	Request crew membership (service token or DPoP)
`/xrpc/io.atcr.hold.exportUserData`	GET	GDPR data export (returns user's records; service token only)
`/xrpc/io.atcr.hold.deleteUserData`	DELETE	GDPR data deletion (deletes crew, layer, and stats records; service token only)

Appview Token Required

/xrpc/io.atcr.hold.updateCrewTier (POST) validates the caller inline via ValidateAppviewToken. Returns 503 if the appview DID is not configured on the hold, or 401 on token validation failure.

Endpoint	Method	Description
`/xrpc/io.atcr.hold.updateCrewTier`	POST	Update a crew member's tier (appview JWT, ES256)

Scanner WebSocket

Endpoint	Method	Description
`/xrpc/io.atcr.hold.subscribeScanJobs`	GET (WebSocket)	Scanner job subscription. Auth via `?secret=` query param or `X-Scanner-Secret` header (shared secret). Supports `?cursor=` for backfill, `?workers=` to declare how many scans the process runs at once (default 1), and `?instance=` to declare a stable process identity so a reconnecting scanner resumes its own in-flight jobs.

OCI Multipart Upload Endpoints (`pkg/hold/oci/xrpc.go`)

All require blob:write permission via service token:

Endpoint	Method	Description
`/xrpc/io.atcr.hold.initiateUpload`	POST	Start multipart upload
`/xrpc/io.atcr.hold.getPartUploadUrl`	POST	Get presigned S3 URL for a part; the client PUTs the part bytes directly to S3
`/xrpc/io.atcr.hold.completeUpload`	POST	Finalize multipart upload
`/xrpc/io.atcr.hold.abortUpload`	POST	Cancel multipart upload
`/xrpc/io.atcr.hold.notifyManifest`	POST	Notify manifest push (creates layer records + optional Bluesky post)

ATCR Hold-Specific Endpoints (`io.atcr.hold.*`)

Endpoint	Method	Auth	Description
`/xrpc/io.atcr.hold.initiateUpload`	POST	blob:write	Start multipart upload
`/xrpc/io.atcr.hold.getPartUploadUrl`	POST	blob:write	Get presigned S3 URL for a part; client PUTs bytes directly to S3
`/xrpc/io.atcr.hold.completeUpload`	POST	blob:write	Finalize multipart upload
`/xrpc/io.atcr.hold.abortUpload`	POST	blob:write	Cancel multipart upload
`/xrpc/io.atcr.hold.notifyManifest`	POST	blob:write	Notify manifest push/pull (creates layer records, increments stats, optional Bluesky post)
`/xrpc/io.atcr.hold.requestCrew`	POST	service token or DPoP	Request crew membership
`/xrpc/io.atcr.hold.exportUserData`	GET	service token	GDPR data export
`/xrpc/io.atcr.hold.deleteUserData`	DELETE	service token	GDPR data deletion (crew, layer, stats records)
`/xrpc/io.atcr.hold.getQuota`	GET	none	Get user quota info
`/xrpc/io.atcr.hold.getLayersForManifest`	GET	none	Get layer records for a manifest AT-URI
`/xrpc/io.atcr.hold.image.getConfig`	GET	none	Get OCI image config record for a manifest digest
`/xrpc/io.atcr.hold.purgeManifest`	POST	inline (service token or DPoP; captain, crew:admin, or manifest owner)	Purge layer/scan/image-config records for a single manifest URI. Called by appview on UI delete; called internally on takedown receipt. Does not delete S3 blobs (GC handles those).
`/xrpc/io.atcr.hold.listTiers`	GET	none	List hold's available tiers with quotas and features (scanOnPush)
`/xrpc/io.atcr.hold.updateCrewTier`	POST	appview token (ES256 JWT; 503 if appview DID not configured)	Update crew member's tier
`/xrpc/io.atcr.hold.subscribeScanJobs`	GET (WebSocket)	shared secret (`?secret=` or `X-Scanner-Secret`)	Scanner job subscription; supports `?cursor=` for backfill, `?workers=` for concurrency, `?instance=` for reconnect resumption

Standard ATProto Endpoints (excluding io.atcr.hold.*)

Endpoint
/xrpc/_health
/xrpc/com.atproto.server.describeServer
/xrpc/com.atproto.repo.describeRepo
/xrpc/com.atproto.repo.getRecord
/xrpc/com.atproto.repo.listRecords
/xrpc/com.atproto.repo.deleteRecord
/xrpc/com.atproto.repo.uploadBlob
/xrpc/com.atproto.sync.listBlobs
/xrpc/com.atproto.sync.listRepos
/xrpc/com.atproto.sync.getRecord
/xrpc/com.atproto.sync.getRepo
/xrpc/com.atproto.sync.getRepoStatus
/xrpc/com.atproto.sync.getLatestCommit
/xrpc/com.atproto.sync.getBlob
/xrpc/com.atproto.sync.subscribeRepos
/xrpc/com.atproto.identity.resolveHandle
/xrpc/app.bsky.actor.getProfile
/xrpc/app.bsky.actor.getProfiles
/.well-known/did.json
/.well-known/atproto-did

7.7 KiB Raw Blame History