# ATCR Hold Service

Hold Service is the BYOS (Bring Your Own Storage) blob storage backend for ATCR. It stores container image layers in your own S3-compatible storage (AWS S3, Storj, Minio, UpCloud, etc.) and generates presigned URLs so clients transfer data directly to/from S3. Each hold runs an embedded ATProto PDS with its own DID, repository, and crew-based access control.

Hold Service is one component of the ATCR ecosystem:

1. **[AppView](https://atcr.io/r/evan.jarrett.net/atcr-appview)** — Registry API + web interface
2. **Hold Service** (this component) — Storage backend with embedded PDS
3. **Credential Helper** — Client-side tool for ATProto OAuth authentication

```
Docker Client --> AppView (resolves identity) --> User's PDS (stores manifest)
                      |
                Hold Service (generates presigned URL)
                      |
                S3/Storj/etc. (client uploads/downloads directly)
```

Manifests (small JSON metadata) live in users' ATProto PDS. Blobs (large binary layers) live in hold services. AppView orchestrates the routing.

## When to Run Your Own Hold

Most users can push to the default hold at **https://hold01.atcr.io** — you don't need to run your own.

Run your own hold if you want to:
- Control where your container layer data is stored (own S3 bucket, geographic region)
- Manage access for a team or organization via crew membership
- Run a shared hold for a community or project
- Use a CDN pull zone for faster downloads

**Prerequisites:** S3-compatible storage with a bucket already created, and a domain with TLS for production.

## Quick Start

### 1. Generate Configuration

```bash
# Build the hold binary
go build -o bin/atcr-hold ./cmd/hold

# Generate a fully-commented config file with all defaults
./bin/atcr-hold config init config-hold.yaml
```

Or generate config from Docker without building locally:

```bash
docker run --rm -i $(docker build -q -f Dockerfile.hold .) config init > config-hold.yaml
```

The generated file documents every option with inline comments. Edit only what you need.

### 2. Minimal Configuration

Only three things need to be set — everything else has sensible defaults:

```yaml
storage:
  access_key: "YOUR_S3_ACCESS_KEY"
  secret_key: "YOUR_S3_SECRET_KEY"
  bucket: "your-bucket-name"
  endpoint: "https://gateway.storjshare.io"  # omit for AWS S3

server:
  public_url: "https://hold.example.com"

registration:
  owner_did: "did:plc:your-did-here"
```

- **`server.public_url`** — Your hold's public HTTPS URL. This becomes the hold's `did:web` identity.
- **`storage.bucket`** — S3 bucket name (must already exist).
- **`registration.owner_did`** — Your ATProto DID. Creates you as captain (admin) on first boot. Get yours from: `https://bsky.social/xrpc/com.atproto.identity.resolveHandle?handle=yourhandle.bsky.social`

### 3. Build and Run with Docker

```bash
# Build the image
docker build -f Dockerfile.hold -t atcr-hold:latest .

# Run it
docker run -d \
  --name atcr-hold \
  -p 8080:8080 \
  -v $(pwd)/config-hold.yaml:/config.yaml:ro \
  -v atcr-hold-data:/var/lib/atcr-hold \
  atcr-hold:latest serve --config /config.yaml
```

- **`/var/lib/atcr-hold`** — Persistent volume for the embedded PDS (carstore database + signing keys). Back this up.
- **Port 8080** — Default listen address. Put a reverse proxy (Caddy, nginx) in front for TLS.
- The image is built `FROM scratch` — the binary includes SQLite statically linked.

## Configuration

Config loads in layers: **defaults → YAML file → environment variables**. Later layers override earlier ones.

All YAML fields can be overridden with environment variables using the `HOLD_` prefix and `_` path separators. For example, `server.public_url` becomes `HOLD_SERVER_PUBLIC_URL`.

S3 credentials also accept standard AWS environment variable names: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, `S3_BUCKET`, `S3_ENDPOINT`.

For the complete configuration reference with all options and defaults, see [`config-hold.example.yaml`](../config-hold.example.yaml) or run `atcr-hold config init`.

## Access Control

| Setting | Who can pull | Who can push |
|---|---|---|
| `server.public: true` | Anyone | Captain + crew with `blob:write` |
| `server.public: false` (default) | Crew with `blob:read` | Captain + crew with `blob:write` |
| + `registration.allow_all_crew: true` | (per above) | Any authenticated user |

The captain (set via `registration.owner_did`) has all permissions implicitly. `blob:write` implies `blob:read`.

Authentication uses ATProto service tokens: AppView requests a token from the user's PDS scoped to the hold's DID, then includes it in XRPC requests. The hold validates the token and checks crew membership.

See [BYOS.md](BYOS.md) for the full authorization model.

## Optional Subsystems

| Subsystem | Default | Config key | Notes |
|---|---|---|---|
| Admin panel | Enabled | `admin.enabled` | Web UI for crew, settings, and storage management |
| Quotas | Disabled | `quota.tiers` | Tier-based storage limits (e.g., deckhand=5GB, bosun=50GB) |
| Garbage collection | Disabled | `gc.enabled` | Nightly cleanup of orphaned blobs and records |
| Vulnerability scanner | Disabled | `scanner.secret`, `scanner.rescan_interval` | Requires separate scanner service; see [SBOM_SCANNING.md](SBOM_SCANNING.md) |
| Labeler | Disabled | `labeler.did`, `labeler.subscribe_url` | Consumes takedown labels from an ATProto labeler; purges affected records and gates GC blob cleanup |
| Bluesky posts | Disabled | `registration.enable_bluesky_posts` | Posts push notifications from hold's identity |

## Hold Identity

**did:web (default)** — Derived from `server.public_url` with zero setup. `https://hold.example.com` becomes `did:web:hold.example.com`. The DID document is served at `/.well-known/did.json`. Tied to domain ownership — if you lose the domain, you lose the identity.

**did:plc (portable)** — Set `database.did_method: plc` in config. Registered with plc.directory. Survives domain changes. Requires a rotation key. If `database.rotation_key` is not set, a new K-256 key is generated **in memory only** and logged once via `slog.Warn` — it is never written to disk. You must copy it from the startup logs into `database.rotation_key` in your config immediately, or you will lose the ability to update or recover the DID. Only the signing key (`database.key_path`, default `{database.path}/signing.key`) is persisted to disk automatically. Use `database.did` to adopt an existing DID for recovery or migration.

## Relay Subscription

A hold's embedded PDS is only useful to the wider network if a relay is subscribed to its
firehose. Nothing about a hold's own operation depends on that subscription: pushes, pulls
and the admin panel all work fine with zero relays connected. What breaks is everything
downstream. The AppView learns about `io.atcr.*` records from Jetstream, which is fed by
the Bluesky relays, so a hold no relay is listening to stops contributing live events and
falls back to whatever the AppView's periodic backfill picks up.

This section covers how that subscription is established today, a failure mode that took
the whole ATCR fleet off the network for two weeks in August 2026, how to diagnose and
recover from it, and a proposed automatic fix that is **not implemented**.

### How it works today

**1. The hold announces itself once, at boot.**

`ServeWithListener` fires a single `requestCrawls()` goroutine during startup
(`pkg/hold/server.go:400`), skipped entirely in a `-tags testmode` build because local
dev holds are not reachable by public relays. The implementation
(`pkg/hold/server.go:448-483`) builds a deduplicated target list from
`atproto.KnownRelays` (`pkg/atproto/relays.go:22`, the hardcoded list also documented in
[KNOWN_RELAYS.md](KNOWN_RELAYS.md)) plus any extra entries in `server.relay_endpoints`,
then POSTs `com.atproto.sync.requestCrawl` to all of them concurrently
(`pkg/atproto/relays.go:206-243`). The request body is just `{"hostname": "<host of
server.public_url>"}`. Per-relay failures are logged at warn level and never block
startup.

This is the only automatic crawl request in the codebase. There is no ticker, no retry
after the initial fan-out, and no check anywhere that a relay is actually subscribed
afterwards.

**2. Relays connect back over the firehose WebSocket.**

A relay that accepts the crawl request dials `com.atproto.sync.subscribeRepos` on the hold
(`pkg/hold/pds/xrpc.go:184`, handler at `:1042`), which registers it as a subscriber on
the `EventBroadcaster` (`pkg/hold/pds/events.go:440`). The broadcaster keeps its
subscribers in an unexported map (`events.go:449`) and tracks the current sequence number,
readable via `GetCurrentSeq()` (`events.go:1062-1066`).

**3. Events originate from repo commits.**

Every write to the hold's repo goes through `commitWrite` (`pkg/hold/pds/repo.go:141`),
which invokes the repo event handler registered at startup
(`pkg/hold/server.go:172-173`): the records-index handler wrapping
`broadcaster.SetRepoEventHandler()` (`events.go:1052-1060`). That calls `Broadcast`
(`events.go:547`), which persists the event to the `firehose_events` table
(`events.go:650`) and fans it out to every connected subscriber. The persisted table is
what makes cursor-based backfill possible when a relay reconnects behind.

### Failure mode: silently dropped by the relay

The relay side of this is one-directional and unforgiving. In the indigo version pinned by
this repo (`github.com/bluesky-social/indigo v0.0.0-20260901021441-b1f966883e38`):

- `subscribeWithRedialer` (`cmd/relay/relay/slurper.go:286`) retries a failed dial with
  backoff, incrementing a counter each time. Once the counter passes 15, meaning 16
  consecutive failed dials, it logs "host does not appear to be online, disabling for
  now", persists `HostStatusOffline`, and **returns from the goroutine**
  (`slurper.go:332-339`). Nothing schedules another attempt.
- `ResubscribeAllHosts` runs on relay startup but selects only hosts with
  `status = "active"` (`cmd/relay/relay/crawl.go:59-63`), so an offline host is not picked
  back up even by a relay restart.
- The only path back is a fresh `com.atproto.sync.requestCrawl`
  (`cmd/relay/handlers.go:19-68` → `SubscribeToHost`, `cmd/relay/relay/crawl.go:11-56`),
  which resubscribes the existing host row and puts it back in rotation.

Combine that with the hold sending `requestCrawl` exactly once at boot, and the result is
a trap: **a hold that the relay drops stays invisible until its process restarts.** Any
transient outage long enough to burn 16 dials, a reverse proxy restart, a TLS renewal
hiccup, a host reboot, a network partition, is permanent from the relay's point of view.

Worse, the failure is quiet. The hold logs nothing, because from its side nothing
happened: a subscriber disconnected, which is routine. No metric counts connected
subscribers. Nothing compares the hold's sequence number against any relay's view of it.

#### Worked example: the 2026-08-20 outage

Around 2026-08-20 the Bluesky relays marked every ATCR hold `offline` and stopped dialing
them. It went unnoticed until 2026-09-03, when it surfaced indirectly as a complaint that
push and pull counts on the site were up to 24 hours stale. The stale counters were a
symptom. The actual state was that the entire fleet had been disconnected from the network
for two weeks.

Measured read-only against the public relay, before the manual re-crawl:

```
relay1.us-west  getHostStatus?hostname=us-chi1.cove.seamark.dev
  -> {"accountCount":1,"seq":124596,"status":"offline"}

relay's view of the hold's repo:  rev 3mtji3hmiqi22   (2026-08-20 14:41 UTC)
hold's own view:                  rev 3mulfs3e63z22   (seconds old, still advancing)
```

Every ATCR hold the relay knew about was in the same state: `hold01.atcr.io`,
`hold.cetacean.club`, `hold.styx.mrijke.nl`, `hold.biglargeclarke.com`, `lexicon.store`.

The downstream effect on the AppView is worth spelling out. The AppView consumes
`wss://jetstream{1,2}.{us-west,us-east}.bsky.network/subscribe`
(`pkg/appview/config.go:277-282`), which those same relays feed. With every hold offline
at the relay, no live hold events reached the AppView at all, and the only remaining path
was the periodic backfill (`jetstream.backfill_interval`, default `24h`,
`pkg/appview/config.go:284`). Hence "up to 24 h stale": that was the backfill period
showing through, not a stats bug.

### Diagnosing it

The fastest check is the **Relays tab in the hold admin panel**
(`/admin#relays`, handler `pkg/hold/admin/handlers_relays.go:67-106`). For each known
relay it probes status and capabilities and renders the relay's view of the hold's repo
rev next to the hold's own current rev, so a stalled subscription shows up as a `Behind
(rev: ...)` badge instead of `Known (rev: ...)`. That rev comparison is the single most
reliable tell.

Note one gap: the handler fetches `getHostStatus` into `RelayStatusView.HostStatus`
(`handlers_relays.go:101`), but `partials/relay_status.html` never renders it. The
`Online` badge on that row reflects whether the *relay* answered, not whether the relay
considers *this hold* online. So the admin panel will happily show a green "Online" relay
that has given up on you. Read the rev column, not the badge.

By hand, against any relay:

```bash
# 1. Does the relay still consider this hold a live host?
curl -s "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.getHostStatus?hostname=hold.example.com"
# {"hostname":"hold.example.com","seq":124596,"accountCount":1,"status":"offline"}

# 2. How far along is the relay's copy of the repo?
curl -s "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.getLatestCommit?did=did:web:hold.example.com"

# 3. How far along is the hold itself?
curl -s "https://hold.example.com/xrpc/com.atproto.sync.getLatestCommit?did=did:web:hold.example.com"
```

Reading the results:

- `"status":"active"` means the relay has a live subscription, or intends to. This is the
  healthy state.
- `"status":"offline"` means the redialer gave up after 16 failed dials and will never
  retry on its own. This requires a manual `requestCrawl`.
- `"status":"banned"` is different and not self-inflicted by an outage.
  `SubscribeToHost` refuses banned hosts outright (`crawl.go:51-53`), so a re-crawl will
  not help. Note that indigo bans a host if its `Server` response header contains
  `atproto-relay` (`slurper.go:344-351`), so check your reverse proxy is not adding one.
- The decisive signal is the pair of revs from calls 2 and 3. **If the relay's rev is
  frozen at some point in the past while the hold's own rev keeps moving, the relay is not
  receiving your events.** Since the same rev is queryable from both sides, the timestamp
  encoded in the frozen TID tells you roughly when you were dropped.

A relay reporting a rev equal to the hold's own is fully caught up and healthy. A rev
slightly behind on a busy hold is normal lag, not a fault.

### Recovering manually

Re-request a crawl. From the admin panel, the Relays tab has a per-relay "Request Crawl"
button (`POST /admin/relays/crawl`, `pkg/hold/admin/admin.go:518`) and a crawl-all button
that fans out to every entry in `KnownRelays` concurrently (`POST
/admin/relays/crawl-all`, `admin.go:519`, handler `handlers_relays.go:154-201`). Both are
captain-gated and both log the requesting DID.

Equivalently, by hand:

```bash
curl -s -X POST "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.requestCrawl" \
  -H 'Content-Type: application/json' \
  -d '{"hostname":"hold.example.com"}'
```

Restarting the hold process also works, since that re-runs the boot fan-out
(`server.go:400`), but it is the blunt version of the same action.

Two things to understand about scope:

1. **It is per host.** A relay's offline marking is per-host row. Crawling one hold does
   nothing for any other hold, so during a fleet-wide event every hold's operator has to
   act independently. There is no bulk operation across holds.
2. **It is per relay.** Each relay keeps its own host table, so a hold can be `active` on
   one relay and `offline` on another. The admin crawl-all button covers every relay for
   one hold; it does not cover every hold.

After crawling, re-run the `getHostStatus` check. A successful recovery flips `status` to
`active` and the `seq` starts advancing again.

> ### Warning: check for the replay fix before provoking a re-crawl
>
> A relay coming back resumes from its stored cursor, which after a long outage is far
> behind. That makes the hold replay history out of `firehose_events` to the reconnecting
> subscriber. Before commit `ca539b1` ("hold/pds: stop firehose backfill panicking on
> subscriber disconnect", 2026-08-08) that replay path could **take the entire hold process
> down** with `panic: send on closed channel` if the relay dropped mid-backfill: the
> backfill goroutine wrote to `sub.send` without holding the broadcaster lock while
> `Unsubscribe` closed that channel underneath it.
>
> The bug only fires when a backfill goroutine exists, which `Subscribe` skips when the
> subscriber's cursor already equals the current sequence. That is exactly why it stayed
> hidden: caught-up relays never triggered it, and a hold whose relays are all far behind
> is exposed on every single reconnect. Provoking a re-crawl on an old build is therefore
> close to the worst case for it.
>
> **Verify the deployed build contains the fix before requesting a crawl:**
>
> ```bash
> git merge-base --is-ancestor ca539b1 <deployed-commit> && echo "safe" || echo "NOT SAFE - upgrade first"
> ```
>
> As of this writing the build deployed to seamark (`efabb677`, 2026-05-26) does **not**
> contain it. Upgrade the hold before re-crawling, or expect the process to fall over
> partway through the replay.

### Proposed automatic fix (NOT IMPLEMENTED)

> **Status: deferred by decision. None of the following exists in the codebase.** The
> 2026-08-20 outage was resolved by requesting crawls manually. The design below is
> recorded so that whoever revisits this does not have to rederive it. Do not read this
> section as a description of current behaviour.

The shape of the fix:

1. **Keep the boot `requestCrawl` exactly as it is.** It is correct for the common case of
   a new or restarted hold, and it is what makes a hold discoverable at all.

2. **Add a jittered re-crawl ticker, roughly hourly.** An hour is short relative to the
   two weeks this outage ran and long relative to how quickly a relay recovers, and it
   bounds worst-case invisibility to about an hour. The jitter matters: without it, every
   hold that started from the same deployment fires at the same instant and the fan-out
   arrives at the relays as a synchronized burst.

3. **Guard the ticker on liveness, so it normally does nothing.** The `EventBroadcaster`
   already knows whether anyone is connected: it holds the subscriber set at
   `events.go:449` and the sequence number at `events.go:1062-1066`. If a relay-shaped
   subscriber is connected, skip the fan-out entirely and the ticker costs nothing. Only
   when no such subscriber is present does it re-announce. This makes the mechanism a
   recovery path rather than a periodic broadcast, which is the whole reason it is
   tolerable to run it on every hold.

   Two small pieces of plumbing this needs, neither of which exists today: the broadcaster
   exposes no accessor for the subscriber count, only the unexported map; and `Subscriber`
   (`events.go:42-54`) does not retain the `userAgent` that `Subscribe` receives and logs
   (`events.go:440`, `:453`), so "relay-shaped" cannot currently be evaluated. Either store
   the user agent, or settle for a plain non-zero subscriber count and accept that a
   curious human with `websocat` attached would suppress a re-crawl.

4. **Surface liveness somewhere visible.** This is arguably more important than the
   ticker. The deeper problem was not that recovery required a manual step, it was that
   two weeks passed before anyone knew there was anything to recover. Exposing the
   subscriber count and `GetCurrentSeq()` on the health endpoint or the admin dashboard
   would have made "zero subscribers, seq unchanged for a fortnight" visible at a glance.
   Rendering the already-fetched `HostStatus` in `partials/relay_status.html` is a
   near-free improvement in the same direction.

Risks to weigh:

- **Relay rate limiting.** An unconditional hourly fan-out from every hold to all 15
  entries in `KnownRelays` is a meaningful amount of unsolicited traffic and a plausible
  way to get an IP or a domain throttled. The liveness guard is the primary mitigation,
  since a healthy hold sends nothing at all, and the jitter is the secondary one. A
  disconnected hold that re-crawls hourly is a rounding error; a fleet of healthy holds
  all re-crawling hourly is not.
- **The replay hazard above becomes automatic.** A re-crawl ticker means reconnects with
  stale cursors happen without anyone watching. This is fine on any build containing
  `ca539b1`, and it is a way to crash a hold repeatedly on any build that predates it. Any
  implementation of this should be gated on that fix being present, not merely assumed.
- **`getHostStatus` polling was considered and rejected as the trigger.** Asking each relay
  whether it still likes us is more direct than inferring from the subscriber count, but it
  is N outbound requests per interval per hold whether or not anything is wrong, which is
  precisely the traffic pattern the liveness guard is trying to avoid.

## Verification

After starting your hold, verify it's working:

```bash
# Health check — should return {"version":"..."}
curl https://hold.example.com/xrpc/_health

# DID document — should return valid JSON with service endpoints
curl https://hold.example.com/.well-known/did.json

# Captain record — should show your owner DID
curl "https://hold.example.com/xrpc/com.atproto.repo.listRecords?repo=HOLD_DID&collection=io.atcr.hold.captain"

# Crew records
curl "https://hold.example.com/xrpc/com.atproto.repo.listRecords?repo=HOLD_DID&collection=io.atcr.hold.crew"
```

Replace `HOLD_DID` with your hold's DID (from the `/.well-known/did.json` response).

## Docker Compose

```yaml
services:
  atcr-hold:
    build:
      context: .
      dockerfile: Dockerfile.hold
    command: ["serve", "--config", "/config.yaml"]
    volumes:
      - ./config-hold.yaml:/config.yaml:ro
      - atcr-hold-data:/var/lib/atcr-hold
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "/healthcheck", "http://localhost:8080/xrpc/_health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s

volumes:
  atcr-hold-data:
```

For production with TLS termination, see [`deploy/docker-compose.prod.yml`](../deploy/docker-compose.prod.yml) which includes a Caddy reverse proxy.

## Further Reading

- [`config-hold.example.yaml`](../config-hold.example.yaml) — Complete configuration reference with inline comments
- [BYOS.md](BYOS.md) — Bring Your Own Storage architecture and authorization model
- [KNOWN_RELAYS.md](KNOWN_RELAYS.md) — Relay list and capabilities (see also [Relay Subscription](#relay-subscription) above)
- [HOLD_XRPC_ENDPOINTS.md](HOLD_XRPC_ENDPOINTS.md) — XRPC endpoint reference
- [BILLING.md](BILLING.md) — Stripe billing integration
- [QUOTAS.md](QUOTAS.md) — Quota management
- [SBOM_SCANNING.md](SBOM_SCANNING.md) — Vulnerability scanning