mirrors/at-container-registry
1
0
Fork 0
mirror of https://tangled.org/evan.jarrett.net/at-container-registry synced 2026-09-26 12:14:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
ba1f5eab406beec36a0f16d33e4694dffff4bcde
at-container-registry/docs/HOLD_XRPC_ENDPOINTS.md
T
Evan JarrettandClaude Fable 5.1 f4343d7956 appview: upload small blobs with one presigned PUT, and verify every digest 
Every blob went through the multipart machinery: an S3 multipart started
on Docker's initial POST, a hold round trip per part, and a complete on
the hold that finished the multipart, HEADed the temp object, copied it
to its final key, and deleted the temp. For a 2KB config blob that was
three hold calls and six S3 operations. On production data 86% of
distinct layers and every config blob fit in a 16MB buffer, and 49% of
image manifests have no layer larger than that.

The writer now buffers up to 16MB (also the multipart part size) and
makes no hold call until it has to. A blob that never overflows the
buffer is written at Commit with a single presigned PUT to its final
key, via the hold's existing method=PUT presign; the multipart only
starts on the first flush. The hold's completeUpload does nothing the
direct path skips: quota, layer records, stats and scan dispatch all
hang off notifyManifest, which is unchanged.

The buffer starts empty and grows on demand, with the doubling capped so
capacity never overshoots 16MB: a config blob costs kilobytes, and only
layers that approach the threshold fill it.

Bytes are hashed as they arrive. Commit compares the computed sha256 to
the digest the client claimed before any network call, and returns
DIGEST_INVALID on mismatch, aborting a multipart if one was started.
Previously nothing verified the content, so a pusher could store wrong
bytes under a digest in the shared content-addressed space.

Tests observe request counts on a fake hold and fake S3 rather than
return values. The growth test streams in 24KB chunks because
power-of-two chunks land on 16MB by luck and hid an earlier weaker guard.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
2026-09-09 09:50:08 -05:00

182 lines
9.5 KiB
Markdown
Raw Blame History

# Hold Service XRPC Endpoints
This document lists all XRPC endpoints implemented in the Hold service (`pkg/hold/`).
## PDS Endpoints (`pkg/hold/pds/xrpc.go`)
### Public (No Auth Required)
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/xrpc/_health` | GET | Health check |
| `/xrpc/com.atproto.server.describeServer` | GET | Server metadata |
| `/xrpc/com.atproto.repo.describeRepo` | GET | Repository information |
| `/xrpc/com.atproto.repo.getRecord` | GET | Retrieve a single record |
| `/xrpc/com.atproto.repo.listRecords` | GET | List records in a collection (paginated) |
| `/xrpc/com.atproto.sync.listBlobs` | GET | List blob CIDs for an account |
| `/xrpc/com.atproto.sync.listRepos` | GET | List all repositories |
| `/xrpc/com.atproto.sync.getRecord` | GET | Get record as CAR file |
| `/xrpc/com.atproto.sync.getRepo` | GET | Full repository as CAR file |
| `/xrpc/com.atproto.sync.getRepoStatus` | GET | Repository hosting status |
| `/xrpc/com.atproto.sync.getLatestCommit` | GET | Current commit CID and revision |
| `/xrpc/com.atproto.sync.subscribeRepos` | GET | WebSocket firehose |
| `/xrpc/com.atproto.identity.resolveHandle` | GET | Resolve handle to DID |
| `/xrpc/app.bsky.actor.getProfile` | GET | Get actor profile |
| `/xrpc/app.bsky.actor.getProfiles` | GET | Get multiple profiles |
| `/xrpc/io.atcr.hold.listTiers` | GET | List hold's available tiers with quotas and features |
| `/.well-known/did.json` | GET | DID document |
| `/.well-known/atproto-did` | GET | DID for handle resolution |
### Conditional Auth (based on captain.public)
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/xrpc/com.atproto.sync.getBlob` | GET/HEAD | Get blob (routes OCI vs ATProto) |
#### getBlob response shape
The endpoint routes on the `cid` parameter and answers differently per branch.
**ATProto blob** (`cid` is a CID): 307 redirect to a presigned URL.
**OCI blob** (`cid` starts with `sha256:`): JSON.
```json
{
"url": "https://s3.example.com/...?X-Amz-Signature=...",
"size": 27129344
}
```
| Field | Type | Notes |
|---|---|---|
| `url` | string | Presigned URL for the requested `method` (GET, HEAD or PUT). Always present on a 200. |
| `size` | int64 | Blob size in bytes. Present on GET and HEAD only, and omitted when the size could not be resolved. |
`size` lets the AppView build an OCI descriptor without a second round trip: `ProxyBlobStore.Stat` used to fetch a presigned HEAD URL here and then HEAD it against S3 purely to read `Content-Length`. It is resolved from the hold's records index (any layer with an `io.atcr.hold.layer` record) and falls back to a `HeadObject` (image config blobs, and layers whose manifest notification has not landed yet).
The field is additive. A client that reads only `url` behaves exactly as before, and an AppView that gets no `size` falls back to HEADing the presigned URL. `size` is never sent for a PUT presign: the object does not exist yet.
A `method=PUT` presign is a write capability (gated on `blob:write`, same as the multipart endpoints) and is how the AppView uploads a blob small enough to fit in its 16MB buffer: one presigned PUT to the blob's final `sha256:` key, instead of initiateUpload plus part URLs plus completeUpload plus the server side copy out of the temp key. The URL is signed with `Content-Type: application/octet-stream`, so the PUT has to carry that header or S3 rejects the signature.
On GET and HEAD, a blob that is not in storage is answered with **404** and a JSON error body instead of a presigned URL:
```json
{
"error": "BlobUnknown",
"message": "blob not found"
}
```
### Owner/Crew Admin Required
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/xrpc/com.atproto.repo.deleteRecord` | POST | Delete a record |
| `/xrpc/com.atproto.repo.uploadBlob` | POST | Upload ATProto blob |
### Inline Auth (per-manifest caller check)
`/xrpc/io.atcr.hold.purgeManifest` (POST) does not use a router middleware. Auth is validated inline by `ValidateManifestPurger`, which accepts either a Bearer service token or a DPoP token, then checks the caller's role:
- Hold captain (any manifest)
- Crew member with `crew:admin` permission (any manifest)
- Crew member whose DID matches the manifest URI's DID (own manifests only)
Idempotent. Does not delete S3 blobs — GC handles those.
### Auth Required (Service Token)
The `requireAuth` middleware validates Bearer service tokens only. `requestCrew` additionally accepts DPoP tokens when called directly (the handler falls back to `ValidateDPoPRequest` if no user is in context).
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/xrpc/io.atcr.hold.requestCrew` | POST | Request crew membership (service token or DPoP) |
| `/xrpc/io.atcr.hold.exportUserData` | GET | GDPR data export (returns user's records; service token only) |
| `/xrpc/io.atcr.hold.deleteUserData` | DELETE | GDPR data deletion (deletes crew, layer, and stats records; service token only) |
### Appview Token Required
`/xrpc/io.atcr.hold.updateCrewTier` (POST) validates the caller inline via `ValidateAppviewToken`. Returns 503 if the appview DID is not configured on the hold, or 401 on token validation failure.
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/xrpc/io.atcr.hold.updateCrewTier` | POST | Update a crew member's tier (appview JWT, ES256) |
### Scanner WebSocket
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/xrpc/io.atcr.hold.subscribeScanJobs` | GET (WebSocket) | Scanner job subscription. Auth via `?secret=` query param or `X-Scanner-Secret` header (shared secret). Supports `?cursor=` for backfill, `?workers=` to declare how many scans the process runs at once (default 1), and `?instance=` to declare a stable process identity so a reconnecting scanner resumes its own in-flight jobs. |
---
## OCI Multipart Upload Endpoints (`pkg/hold/oci/xrpc.go`)
All require `blob:write` permission via service token:
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/xrpc/io.atcr.hold.initiateUpload` | POST | Start multipart upload |
| `/xrpc/io.atcr.hold.getPartUploadUrl` | POST | Get presigned S3 URL for a part; the client PUTs the part bytes directly to S3 |
| `/xrpc/io.atcr.hold.completeUpload` | POST | Finalize multipart upload |
| `/xrpc/io.atcr.hold.abortUpload` | POST | Cancel multipart upload |
| `/xrpc/io.atcr.hold.notifyManifest` | POST | Notify manifest push (creates layer records + optional Bluesky post) |
---
## ATCR Hold-Specific Endpoints (`io.atcr.hold.*`)
| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/xrpc/io.atcr.hold.initiateUpload` | POST | blob:write | Start multipart upload |
| `/xrpc/io.atcr.hold.getPartUploadUrl` | POST | blob:write | Get presigned S3 URL for a part; client PUTs bytes directly to S3 |
| `/xrpc/io.atcr.hold.completeUpload` | POST | blob:write | Finalize multipart upload |
| `/xrpc/io.atcr.hold.abortUpload` | POST | blob:write | Cancel multipart upload |
| `/xrpc/io.atcr.hold.notifyManifest` | POST | blob:write | Notify manifest push/pull (creates layer records, increments stats, optional Bluesky post) |
| `/xrpc/io.atcr.hold.requestCrew` | POST | service token or DPoP | Request crew membership |
| `/xrpc/io.atcr.hold.exportUserData` | GET | service token | GDPR data export |
| `/xrpc/io.atcr.hold.deleteUserData` | DELETE | service token | GDPR data deletion (crew, layer, stats records) |
| `/xrpc/io.atcr.hold.getQuota` | GET | none | Get user quota info |
| `/xrpc/io.atcr.hold.getLayersForManifest` | GET | none | Get layer records for a manifest AT-URI |
| `/xrpc/io.atcr.hold.image.getConfig` | GET | none | Get OCI image config record for a manifest digest |
| `/xrpc/io.atcr.hold.purgeManifest` | POST | inline (service token or DPoP; captain, crew:admin, or manifest owner) | Purge layer/scan/image-config records for a single manifest URI. Called by appview on UI delete; called internally on takedown receipt. Does not delete S3 blobs (GC handles those). |
| `/xrpc/io.atcr.hold.listTiers` | GET | none | List hold's available tiers with quotas and features (scanOnPush) |
| `/xrpc/io.atcr.hold.updateCrewTier` | POST | appview token (ES256 JWT; 503 if appview DID not configured) | Update crew member's tier |
| `/xrpc/io.atcr.hold.subscribeScanJobs` | GET (WebSocket) | shared secret (`?secret=` or `X-Scanner-Secret`) | Scanner job subscription; supports `?cursor=` for backfill, `?workers=` for concurrency, `?instance=` for reconnect resumption |
---
## Standard ATProto Endpoints (excluding io.atcr.hold.*)
| Endpoint |
|----------|
| /xrpc/_health |
| /xrpc/com.atproto.server.describeServer |
| /xrpc/com.atproto.repo.describeRepo |
| /xrpc/com.atproto.repo.getRecord |
| /xrpc/com.atproto.repo.listRecords |
| /xrpc/com.atproto.repo.deleteRecord |
| /xrpc/com.atproto.repo.uploadBlob |
| /xrpc/com.atproto.sync.listBlobs |
| /xrpc/com.atproto.sync.listRepos |
| /xrpc/com.atproto.sync.getRecord |
| /xrpc/com.atproto.sync.getRepo |
| /xrpc/com.atproto.sync.getRepoStatus |
| /xrpc/com.atproto.sync.getLatestCommit |
| /xrpc/com.atproto.sync.getBlob |
| /xrpc/com.atproto.sync.subscribeRepos |
| /xrpc/com.atproto.identity.resolveHandle |
| /xrpc/app.bsky.actor.getProfile |
| /xrpc/app.bsky.actor.getProfiles |
| /.well-known/did.json |
| /.well-known/atproto-did |
---
## See Also
- [DIRECT_HOLD_ACCESS.md](./DIRECT_HOLD_ACCESS.md) - How to call hold endpoints directly without AppView (app passwords, curl examples)
- [BYOS.md](./BYOS.md) - Bring Your Own Storage architecture
- [OAUTH.md](./OAUTH.md) - OAuth + DPoP authentication details
Reference in New Issue View Git Blame Copy Permalink