at-container-registry/docs/HOLD_PUSH_OFFLOAD.md

# Push Offload: Shifting Blob Uploads to the Hold

**Status:** Proposal / design. Not yet implemented.

This document describes shifting OCI blob *upload* (push) bandwidth off the AppView
and onto the hold, mirroring what the pull path already does for reads. It is the
write-side counterpart to the existing pull `307` redirect.

## Motivation

### The asymmetry

The hold design is deliberately lightweight: it handshakes a few HTTP requests and
hands back presigned S3 URLs. On the **read path** this works perfectly. A pull is a
bodyless `GET` that AppView answers with a `307` to a presigned S3 URL
(`proxy_blob_store.go:267-281` `ServeBlob`). Neither AppView nor the hold ever carries
a blob byte, and reads are 10-100x more frequent than writes, so this is where the
property matters most.

The **write path** inverts this. Today a push streams:

```
client --PATCH/PUT--> AppView (buffers 16MB chunks in RAM) --presigned PUT--> S3
```

AppView ingests every layer and re-uploads it to S3 (`proxy_blob_store.go:586-665`
`Write`/`flushPart`). It carries the full upload bandwidth even though the hold and S3
are the actual storage.

### Why this is economically backwards for BYOS

The entire point of BYOS is that a user runs their own hold and storage so the AppView
operator is **not** on the hook for their data. But on push, a BYOS upload goes:

```
client --bytes--> AppView (operator's bandwidth) --> user's own S3
```

The operator pays ingress + egress to shuttle bytes into a bucket they don't own and
will never bill for. The people most motivated to self-host (to avoid paying the
operator) are exactly the ones costing the operator bandwidth on every push. The BYOS
bargain runs backwards on writes.

### Why pure client -> S3 direct is not possible on push

It is tempting to do for push what pull does: redirect the client straight to a
presigned S3 URL. This cannot work with stock OCI clients (`docker`, `containerd`,
`podman`), for reasons independent of each other:

1. **Method preservation kills PATCH.** A `307`/`308` preserves method and body. The
   chunked path uses `PATCH`; S3 has no `PATCH` (object API is `PUT` or the multipart
   `POST`+`UploadId` flow). A redirected `PATCH` 405s.
2. **Streamed bodies cannot be replayed.** Following a redirect on a write requires
   re-sending the body; Go's `http.Client` only does so when `Request.GetBody` is set
   (rewindable). Docker streams the layer tar, which generally is not rewindable.
3. **The finalize response contract.** The closing `PUT <location>?digest=sha256:...`
   expects `201` + `Docker-Content-Digest` + `Location`. S3 returns 200/XML; the client
   chokes. The client also *appends* `?digest=` to the Location, invalidating any SigV4
   presigned URL used as the Location.
4. **Multipart coordination cannot ride a redirect.** Blobs >5GB need S3 multipart,
   which needs an interactive init -> per-part-URL -> complete loop. A redirect cannot
   express that handshake.

The signature/size question ("you need to know the size before you sign") is a red
herring: S3 lets you presign a single `PUT` with `UNSIGNED-PAYLOAD` (no size needed,
5GB cap), and `POST Object` policies support a `content-length-range` (min/max). The
real blocker is that **the thing terminating an OCI upload must speak OCI and return
OCI responses** -- which S3 cannot do.

### The "pick two" on push

On push you cannot have all three at once:

1. Lightweight hold (never touches bytes)
2. Stock `docker push` (no custom client)
3. AppView free of the byte path

- **(1)+(2)** -> AppView eats the bandwidth. *(Where we are today.)*
- **(2)+(3)** -> the hold terminates the upload and carries the bytes. **<- this proposal**
- **(1)+(3)** -> a custom client streams straight to S3, no `docker push`.

Pull gets all three only because a bodyless GET redirect satisfies them simultaneously.
Since the whole reason for BYOS is "don't pay the operator," the correct trade is
**(2)+(3)**: make pushes terminate on the hold, where the bandwidth belongs.

## Goal and non-goals

**Goal:** On push, the layer bytes go `client -> hold -> S3` and never transit AppView.
The handshake (`POST .../blobs/uploads/`) and the manifest `PUT` stay on AppView (cheap,
no bytes).

**Non-goals:**
- Client -> S3 direct on push (impossible with stock clients, see above).
- Changing the pull path (already optimal).
- A custom push client (out of scope; would be the only way to get (1)+(3)).

The hold "carrying bytes" costs **bandwidth**, not its lightweight character:
- **Monolithic push** (docker sends `Content-Length`): the hold does a streaming
  `PutObject` (`io.Copy` passthrough) -- no disk, trivial RAM.
- **Large/chunked push** (>5GB or chunked): multipart with ~5MB part buffers -- still
  trivial memory.

The hold stays memory-light and ops-simple; it just becomes bandwidth-heavy on writes,
on the BYOS owner's box.

## Current state (for reference)

| Concern | Today |
|---|---|
| Push handshake | AppView `POST .../blobs/uploads/` -> distribution lib calls `ProxyBlobStore.Create`, which makes no hold call. `io.atcr.hold.initiateUpload` is only reached if the blob outgrows the 16MB buffer. |
| Push bytes | Client -> AppView RAM (16MB buffer) -> presigned S3 PUT |
| Push finalize | `ProxyBlobWriter.Commit` verifies the received bytes against the client's digest, then either PUTs the whole buffered blob to its final key (a `com.atproto.sync.getBlob` PUT presign) or, for a blob that went multipart, calls XRPC `io.atcr.hold.completeUpload` |
| Hold upload API | Custom XRPC only: `initiateUpload`, `getPartUploadUrl`, `completeUpload`, `abortUpload`, `notifyManifest` (`pkg/hold/oci/xrpc.go:50-61`). No standard OCI `/v2` upload surface. |
| Upload `Location` | AppView-relative `/v2/<name>/blobs/uploads/<id>`, generated by the distribution library from `BlobWriter.ID()`, not by ATCR code. |
| Hold auth | Service token (Bearer, `aud`=hold DID, signed by user's PDS) or DPoP. Validated on every op (`pkg/hold/pds/auth.go:377-429` `ValidateBlobWriteAccess`, `:507-608` `ValidateServiceToken`). The hold **cannot** validate AppView's registry JWT. |
| Pull (contrast) | `ServeBlob` -> `307` to presigned S3 URL; client fetches direct (`proxy_blob_store.go:267-281`). |

Note: an unused `HoldUploadPart = "/xrpc/io.atcr.hold.uploadPart"` constant already
exists (`pkg/atproto/endpoints.go:29`) for "direct buffered part uploads" -- prior art
for the hold carrying bytes, never registered.

## Proposed design

The flow the operator described:

> AppView looks at what hold it needs to go to, checks if that hold is anonymous-push or
> not, does a serviceAuth request if necessary, then lets the hold take it from there.

```
1. docker:   POST https://<appview>/v2/<identity>/<image>/blobs/uploads/
2. AppView:  resolve hold DID (findHoldDIDAndPrefs + resolveSuccessor)
             check hold push policy (anonymous push allowed?)
             if auth required: mint/fetch service token (aud=hold DID) via user OAuth
             -> 202 Accepted
                Location: https://<hold-public-url>/v2/<identity>/<image>/blobs/uploads/<session>?_t=<serviceToken>
3. docker:   PATCH/PUT the layer bytes to that Location          (client -> hold)
4. hold:     validate service token, bind session to user
             stream bytes to its S3 (passthrough or 5MB multipart)
             finalize -> 201 Docker-Content-Digest: sha256:...   (hold -> S3, OCI response)
5. docker:   PUT https://<appview>/v2/<identity>/<image>/manifests/<ref>
6. AppView:  verify referenced blobs exist on the hold, store manifest in user's PDS
```

AppView is out of the byte path entirely (steps 3-4). It keeps only the cheap
metadata operations (steps 1-2, 5-6).

### Why the handshake stays on AppView

Hold resolution and the serviceAuth request both require AppView's OAuth session for the
user (only AppView holds it). So the `POST .../blobs/uploads/` must be answered by
AppView. Everything after the `202` lives on the hold.

### Cross-host Location is spec-legal and opaque to the client

The OCI distribution spec permits an absolute, cross-host `Location`, and the client
treats the returned session URL as **opaque** -- it `PATCH`/`PUT`s to it verbatim. So:

- The hold's session URL does **not** need to be a literal `/v2/...` path; the hold may
  namespace it however it likes, as long as its responses are OCI-compliant
  (`202` + `Location` for `PATCH` continuation, `201` + `Docker-Content-Digest` for the
  finalize `PUT`).
- Only the finalize `PUT` appends `?digest=` -- the hold parses this normally (it is not
  S3, so the appended query breaks nothing).

## Authentication model

This is the crux, and the operator's proposed flow resolves it cleanly.

### Service token in the Location URL (chosen approach)

AppView mints a **service token** during the handshake (the same machinery used today
for AppView's own XRPC calls: `GetOrFetchServiceToken`, `pkg/auth/servicetoken.go:53`,
`getServiceAuth?aud=<holdDID>&lxm=<method>&exp=<ts>`) and embeds it in the upload
Location.

Why this works and the registry JWT does not:

- The hold **already** validates service tokens (`ValidateServiceToken`,
  `pkg/hold/pds/auth.go:507`): checks `aud` == hold DID, `exp`, and the signature
  against the user's DID document. No new trust infrastructure.
- The docker client only *holds* an AppView-issued registry JWT, which the hold cannot
  validate. By putting the credential in the Location URL, the client does not need to
  know anything about hold auth -- it just uses the URL.
- **Cross-host auth headers:** docker generally does not forward its `Authorization`
  bearer to a *different* host on a redirected upload (credential-leak avoidance). The
  token-in-URL avoids depending on header forwarding entirely.

Trade-off: the token appears in the URL (and thus potentially in hold access logs). It
is short-lived; mitigations below.

### Session-bound authorization (handles token expiry mid-upload)

Service tokens are short-lived (~5 min typical; reference PDSes grant up to 1h). A large
push could outlive the token. To avoid re-validating on every chunk:

- The hold validates the token **once**, when the upload session is created (or first
  byte-bearing request arrives), and **binds the session to the authorized user**.
- Subsequent `PATCH`/`PUT` on that session are authorized by **session ownership**, not
  by re-checking the token. The session ID is the capability.
- Request a longer `exp` for upload tokens where the PDS allows it.

This means token expiry mid-upload does not abort an in-flight push.

### Anonymous / managed-push policy

The handshake checks the hold's push policy before deciding whether to mint a token:

- **Auth required (default):** mint a service token; embed in Location. The hold enforces
  captain/crew `blob:write` exactly as today (`ValidateBlobWriteAccess`).
- **Anonymous push allowed:** return a tokenless Location; the hold accepts the upload
  without a service token.

Note the captain record's existing `Public` flag governs anonymous *reads* only
(`pkg/hold/pds/captain.go:24`, `auth.go:445`); writes always require auth today.
"Anonymous push" is a **new** policy bit (see Open Questions). Most holds will keep
auth-required; anonymous push is opt-in for, e.g., open CI mirrors.

### Rejected alternative: JWKS on AppView

AppView could publish a JWKS so the hold validates AppView's registry JWT directly. This
needs new trust plumbing (JWKS endpoint, hold-side fetch + trust config) and still
requires the client to send the registry JWT cross-host (which it avoids doing). The
service-token approach reuses validation the hold already performs, so JWKS is not
pursued.

## AppView changes

| Change | Where |
|---|---|
| `POST .../blobs/uploads/` returns a cross-host hold Location instead of driving `ProxyBlobStore.Create` | new handler ahead of the distribution `/v2` handler; see "Distribution interaction" |
| Hold push-policy check during handshake | `findHoldDIDAndPrefs` result + captain push policy (Jetstream-fed local table, keep it local-fast) |
| Mint service token for upload, embed in Location | reuse `GetOrFetchServiceToken` (`pkg/auth/servicetoken.go`) |
| `HEAD .../blobs/<digest>` (existence / layer skip) stays on AppView | answer from local layer metadata / hold query; keeps docker's skip-existing fast |
| Manifest `PUT` stays on AppView; verify referenced blobs exist on the hold before storing | `manifest_store.go` + a hold existence check |
| `ProxyBlobWriter` byte path becomes unused for offloaded pushes | `proxy_blob_store.go` (see below) |

### Distribution library interaction

The distribution library currently generates the upload Location itself and owns the
`PATCH`/`PUT` session. Two ways to take over the handshake:

1. **Intercept the upload-init route** ahead of distribution: register
   `POST /v2/<name>/blobs/uploads/` on ATCR's router so it short-circuits distribution,
   returning the cross-host hold Location. The subsequent `PATCH`/`PUT` go to the hold
   (different host), so distribution's blob-upload machinery is never invoked for these
   pushes. Cleanest incremental path.
2. **Do it as part of `docs/REMOVING_DISTRIBUTION.md`.** Push-offload makes
   `proxy_blob_store.go`'s `BlobWriter` -- the doc's single gnarliest distribution impl
   -- obsolete for BYOS. The two efforts reinforce each other: owning the `/v2` HTTP
   layer makes emitting a cross-host upload Location trivial, and push-offload removes
   the hardest reason the `BlobWriter` exists.

Recommendation: do (1) first behind a capability flag, fold into (2) when distribution
is removed. After offload, AppView's push surface is just three cheap operations:
`POST uploads/` (handshake), `HEAD blob` (existence), `PUT manifest`.

## Hold changes

The hold grows a real, client-facing OCI blob-upload surface (it speaks only custom
XRPC today):

| Capability | Notes |
|---|---|
| Accept the upload session created/referenced by the Location | session state on the hold (existing multipart manager is the basis: `pkg/hold/oci/multipart.go`) |
| `PATCH` (chunked) and `PUT` (monolithic + finalize) byte handlers returning OCI-compliant responses | `202`+`Location` for continuation; `201`+`Docker-Content-Digest`+`Location` for finalize |
| Stream to S3: monolithic `PutObject` passthrough; multipart (~5MB parts) for chunked/large | reuse `StartMultipartUploadWithManager` / `CompleteMultipartUploadWithManager` (`multipart.go:134-282`) but feed bytes from the client request instead of presigning back to AppView |
| Validate the service token once, bind session to user | `ValidateServiceToken` (`auth.go:507`) + new session ownership |
| Enforce push policy (auth-required vs anonymous push) | captain record policy |
| Existing post-upload side effects still fire | layer records, quota, Bluesky status, scan dispatch -- today via `notifyManifest` (`pkg/hold/oci/xrpc.go:189-484`); fold into finalize or keep as the AppView->hold call after manifest PUT |

The presigned-URL-to-AppView path (`getPartUploadUrl`) is no longer needed for offloaded
pushes -- the hold writes to its own S3 directly with its own credentials (presigning
buys nothing when you hold the keys).

## Capability detection and rollout

Not every hold will support offload immediately (older holds, third-party holds). AppView
must detect support and fall back to today's proxy path:

1. **Per-hold capability flag** in the captain record (e.g. `supportsPushOffload: true`)
   or a hold `/v2`-probe. AppView reads it during hold discovery.
2. **Offload-capable hold** -> return cross-host Location (this proposal).
3. **Legacy hold** -> keep today's `ProxyBlobStore` path (AppView buffers).

Suggested rollout:
- Phase 0: implement, default **off**; managed holds opt in first (operator's own infra,
  safe to test client compatibility).
- Phase 1: enable for managed holds; validate `docker` / `containerd` / `podman` /
  `buildkit` push compatibility (cross-host upload Location is the main compatibility
  risk).
- Phase 2: enable for BYOS holds that advertise the capability.
- Phase 3: fold into `REMOVING_DISTRIBUTION` and drop the `ProxyBlobWriter` byte path.

## Economics after this change

| Path | Today | After offload |
|---|---|---|
| Pull (any hold) | client <- S3 (307), operator carries 0 | unchanged |
| Push, managed hold (operator storage) | client -> AppView -> operator S3 (operator pays both legs) | client -> hold -> operator S3 (operator still pays, but one leg; or keep proxy) |
| Push, BYOS hold (user storage) | client -> **AppView** -> user S3 (**operator pays**) | client -> **user's hold** -> user S3 (**user pays**) |

The BYOS bargain is restored: the operator pays for reads it serves; self-hosters pay
for writes to their own storage.

## Edge cases and risks

- **Client compatibility with cross-host upload Location.** The primary risk. Some
  clients historically mishandle absolute/cross-host upload locations or auth across
  hosts. Token-in-URL mitigates the auth half. Must validate docker, containerd, podman,
  buildkit before BYOS rollout.
- **Token in URL leaks to hold logs.** Short-lived; bound to a single hold; consider
  redaction in hold access logs and a dedicated short `exp` for upload tokens.
- **Token expiry mid-upload.** Handled by session-bound authorization (validate once,
  bind session) plus a longer `exp` where the PDS grants it.
- **Blobs > 5GB.** Require multipart on the hold (single `PutObject` caps at 5GB). The
  hold's existing multipart manager covers this.
- **Resumable / chunked uploads, `GET` upload status.** The hold must implement the OCI
  upload-status `GET` and `Range` semantics if clients use them.
- **Cross-repo blob mount** (`POST .../uploads/?mount=<digest>&from=<repo>`). Resolve at
  AppView (metadata) or proxy a mount hint to the hold; document which.
- **Abort / cleanup** (`DELETE` upload, client disconnect). The hold owns session
  cleanup and S3 multipart abort (existing `abortUpload` logic).
- **Quota enforcement timing.** Quotas are enforced at the hold; ensure the hold checks
  quota at session creation and/or finalize, not only via the old AppView path.
- **Manifest references unflushed blobs.** Manifest `PUT` on AppView must verify all
  referenced blobs exist on the hold (the upload happened out-of-band on the hold), and
  fail the manifest if any are missing.
- **Two TLS endpoints / hostnames.** The hold needs a valid public TLS cert at
  `server.public_url` (already required: `pkg/hold/config.go:155,371`).

## Implementation checklist

Per repo conventions (`CLAUDE.md`):

1. **Lexicons / endpoints:** add the hold's OCI upload endpoints (or document the `/v2`
   passthrough) in `lexicons/`, `pkg/atproto/endpoints.go`, and
   `docs/HOLD_XRPC_ENDPOINTS.md`; run `make lex-lint`.
2. **Captain record:** add the push-policy / `supportsPushOffload` field; regenerate cbor
   (`go generate ./pkg/atproto/...`) and register in `pkg/hold/pds/server.go` if the
   record type changes.
3. **Config:** any new hold/appview config (e.g. offload toggle) -> regenerate example
   configs and sync `deploy/upcloud/configs/*.yaml.tmpl`.
4. **AppView:** new upload-init handler ahead of distribution; push-policy check; service
   token mint into Location; manifest-time blob existence check; capability detection +
   fallback.
5. **Hold:** OCI `PATCH`/`PUT`/finalize/`DELETE`/status handlers; streaming-to-S3 +
   multipart; service-token validation + session binding; push policy; post-upload side
   effects (layer records, quota, scan dispatch).
6. **Jetstream:** if the captain push-policy field is firehose-fed, update
   `pkg/appview/jetstream/backfill.go` and `processor.go`.
7. **Docs:** update `BYOS.md`, `hold.md`, `HOLD_XRPC_ENDPOINTS.md`; cross-link
   `REMOVING_DISTRIBUTION.md`.
8. **Tests + lint:** `make lint`, `make test`; add client-compat integration coverage.

## Open questions

1. **Push-policy shape.** New `supportsPushOffload` + `anonymousPush` bits on the captain
   record, or reuse/extend `Public`? (`Public` currently = anonymous reads only.)
2. **Upload session URL namespace.** Real `/v2/...` on the hold, or a hold-specific path
   (opaque to the client)? `/v2` is most compatible; opaque is cleaner internally.
3. **Service token `lxm` binding.** `getServiceAuth` binds a token to a lexicon method;
   OCI `/v2` routes are not XRPC. Bind to a representative method (e.g.
   `io.atcr.hold.initiateUpload`) and have the hold accept it for the upload routes, or
   relax `lxm` checking for `blob:write`?
4. **Managed holds:** keep proxying through AppView (operator pays anyway and avoids
   client-compat risk), or also offload for consistency?
5. **Sequencing vs `REMOVING_DISTRIBUTION`.** Ship the intercept-handler version first,
   or wait and build it into the post-distribution `/v2` layer?

## See also

- `docs/BYOS.md` -- BYOS architecture and authorization model
- `docs/REMOVING_DISTRIBUTION.md` -- owning the `/v2` HTTP layer (reinforces this work)
- `docs/HOLD_XRPC_ENDPOINTS.md` -- current hold upload endpoints
- `docs/DIRECT_HOLD_ACCESS.md` -- service token acquisition flow (same credential reused here)
- `docs/HOLD_DISCOVERY.md` -- how AppView resolves the target hold