mirror of
https://tangled.org/evan.jarrett.net/at-container-registry
synced 2026-09-29 13:35:35 +00:00
appview: upload small blobs with one presigned PUT, and verify every digest
Every blob went through the multipart machinery: an S3 multipart started on Docker's initial POST, a hold round trip per part, and a complete on the hold that finished the multipart, HEADed the temp object, copied it to its final key, and deleted the temp. For a 2KB config blob that was three hold calls and six S3 operations. On production data 86% of distinct layers and every config blob fit in a 16MB buffer, and 49% of image manifests have no layer larger than that. The writer now buffers up to 16MB (also the multipart part size) and makes no hold call until it has to. A blob that never overflows the buffer is written at Commit with a single presigned PUT to its final key, via the hold's existing method=PUT presign; the multipart only starts on the first flush. The hold's completeUpload does nothing the direct path skips: quota, layer records, stats and scan dispatch all hang off notifyManifest, which is unchanged. The buffer starts empty and grows on demand, with the doubling capped so capacity never overshoots 16MB: a config blob costs kilobytes, and only layers that approach the threshold fill it. Bytes are hashed as they arrive. Commit compares the computed sha256 to the digest the client claimed before any network call, and returns DIGEST_INVALID on mismatch, aborting a multipart if one was started. Previously nothing verified the content, so a pusher could store wrong bytes under a digest in the shared content-addressed space. Tests observe request counts on a fake hold and fake S3 rather than return values. The growth test streams in 24KB chunks because power-of-two chunks land on 16MB by luck and hid an earlier weaker guard. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
034ea5988b
commit
f4343d7956
+19
-5
@@ -222,23 +222,37 @@ fly secrets set HOLD_REGISTRATION_OWNER_DID=did:plc:your-did-here
|
||||
GET /xrpc/com.atproto.server.getServiceAuth?aud=did:web:alice-storage.fly.dev
|
||||
Response: { "token": "eyJ..." }
|
||||
|
||||
5. AppView initiates multipart upload to hold:
|
||||
5. AppView buffers the blob (16MB limit) and verifies the bytes it received
|
||||
against the digest the client claimed. A mismatch is rejected here, before
|
||||
anything reaches storage.
|
||||
|
||||
5a. Small blob (fits in the buffer, which is every config blob and most layers):
|
||||
AppView asks for one write capability and PUTs the whole blob to its final
|
||||
content-addressed key. No multipart session, no temp object, no copy.
|
||||
GET /xrpc/com.atproto.sync.getBlob?did=...&cid=sha256:abc...&method=PUT
|
||||
Authorization: Bearer {serviceToken}
|
||||
Response: { "url": "https://s3.../presigned" }
|
||||
AppView: PUT that URL with Content-Type: application/octet-stream
|
||||
|
||||
5b. Large blob (outgrew the buffer): multipart, as below.
|
||||
|
||||
6. AppView initiates multipart upload to hold, on the first flush:
|
||||
POST https://alice-storage.fly.dev/xrpc/io.atcr.hold.initiateUpload
|
||||
Authorization: Bearer {serviceToken}
|
||||
Body: { "digest": "sha256:abc..." }
|
||||
Response: { "uploadId": "xyz" }
|
||||
|
||||
6. For each part:
|
||||
7. For each part:
|
||||
- AppView: POST /xrpc/io.atcr.hold.getPartUploadUrl
|
||||
- Hold validates service token, checks crew membership
|
||||
- Hold returns: { "url": "https://s3.../presigned" }
|
||||
- Client uploads directly to S3 presigned URL
|
||||
- AppView uploads the part to the S3 presigned URL
|
||||
|
||||
7. AppView completes upload:
|
||||
8. AppView completes upload:
|
||||
POST /xrpc/io.atcr.hold.completeUpload
|
||||
Body: { "uploadId": "xyz", "digest": "sha256:abc...", "parts": [...] }
|
||||
|
||||
8. Manifest stored in alice's PDS:
|
||||
9. Manifest stored in alice's PDS:
|
||||
- holdDid: "did:web:alice-storage.fly.dev"
|
||||
- holdEndpoint: "https://alice-storage.fly.dev" (backward compat)
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user