at-container-registry/docs/hold.md

ATCR Hold Service

Hold Service is the BYOS (Bring Your Own Storage) blob storage backend for ATCR. It stores container image layers in your own S3-compatible storage (AWS S3, Storj, Minio, UpCloud, etc.) and generates presigned URLs so clients transfer data directly to/from S3. Each hold runs an embedded ATProto PDS with its own DID, repository, and crew-based access control.

Hold Service is one component of the ATCR ecosystem:

1. AppView — Registry API + web interface
2. Hold Service (this component) — Storage backend with embedded PDS
3. Credential Helper — Client-side tool for ATProto OAuth authentication

Docker Client --> AppView (resolves identity) --> User's PDS (stores manifest)
                      |
                Hold Service (generates presigned URL)
                      |
                S3/Storj/etc. (client uploads/downloads directly)

Manifests (small JSON metadata) live in users' ATProto PDS. Blobs (large binary layers) live in hold services. AppView orchestrates the routing.

When to Run Your Own Hold

Most users can push to the default hold at https://hold01.atcr.io — you don't need to run your own.

Run your own hold if you want to:

• Control where your container layer data is stored (own S3 bucket, geographic region)
• Manage access for a team or organization via crew membership
• Run a shared hold for a community or project
• Use a CDN pull zone for faster downloads

Prerequisites: S3-compatible storage with a bucket already created, and a domain with TLS for production.

Quick Start

1. Generate Configuration

# Build the hold binary
go build -o bin/atcr-hold ./cmd/hold

# Generate a fully-commented config file with all defaults
./bin/atcr-hold config init config-hold.yaml

Or generate config from Docker without building locally:

docker run --rm -i $(docker build -q -f Dockerfile.hold .) config init > config-hold.yaml

The generated file documents every option with inline comments. Edit only what you need.

2. Minimal Configuration

Only three things need to be set — everything else has sensible defaults:

storage:
  access_key: "YOUR_S3_ACCESS_KEY"
  secret_key: "YOUR_S3_SECRET_KEY"
  bucket: "your-bucket-name"
  endpoint: "https://gateway.storjshare.io"  # omit for AWS S3

server:
  public_url: "https://hold.example.com"

registration:
  owner_did: "did:plc:your-did-here"

• server.public_url — Your hold's public HTTPS URL. This becomes the hold's did:web identity.
• storage.bucket — S3 bucket name (must already exist).
• registration.owner_did — Your ATProto DID. Creates you as captain (admin) on first boot. Get yours from: https://bsky.social/xrpc/com.atproto.identity.resolveHandle?handle=yourhandle.bsky.social

3. Build and Run with Docker

# Build the image
docker build -f Dockerfile.hold -t atcr-hold:latest .

# Run it
docker run -d \
  --name atcr-hold \
  -p 8080:8080 \
  -v $(pwd)/config-hold.yaml:/config.yaml:ro \
  -v atcr-hold-data:/var/lib/atcr-hold \
  atcr-hold:latest serve --config /config.yaml

• /var/lib/atcr-hold — Persistent volume for the embedded PDS (carstore database + signing keys). Back this up.
• Port 8080 — Default listen address. Put a reverse proxy (Caddy, nginx) in front for TLS.
• The image is built FROM scratch — the binary includes SQLite statically linked.

Configuration

Config loads in layers: defaults → YAML file → environment variables. Later layers override earlier ones.

All YAML fields can be overridden with environment variables using the HOLD_ prefix and _ path separators. For example, server.public_url becomes HOLD_SERVER_PUBLIC_URL.

S3 credentials also accept standard AWS environment variable names: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, S3_BUCKET, S3_ENDPOINT.

For the complete configuration reference with all options and defaults, see config-hold.example.yaml or run atcr-hold config init.

Access Control

Setting	Who can pull	Who can push
server.public: true	Anyone	Captain + crew with blob:write
server.public: false (default)	Crew with blob:read	Captain + crew with blob:write
+ registration.allow_all_crew: true	(per above)	Any authenticated user

The captain (set via registration.owner_did) has all permissions implicitly. blob:write implies blob:read.

Authentication uses ATProto service tokens: AppView requests a token from the user's PDS scoped to the hold's DID, then includes it in XRPC requests. The hold validates the token and checks crew membership.

See BYOS.md for the full authorization model.

Optional Subsystems

Subsystem	Default	Config key	Notes
Admin panel	Enabled	admin.enabled	Web UI for crew, settings, and storage management
Quotas	Disabled	quota.tiers	Tier-based storage limits (e.g., deckhand=5GB, bosun=50GB)
Garbage collection	Disabled	gc.enabled	Nightly cleanup of orphaned blobs and records
Vulnerability scanner	Disabled	scanner.secret, scanner.rescan_interval	Requires separate scanner service; see SBOM_SCANNING.md
Labeler	Disabled	labeler.did, labeler.subscribe_url	Consumes takedown labels from an ATProto labeler; purges affected records and gates GC blob cleanup
Bluesky posts	Disabled	registration.enable_bluesky_posts	Posts push notifications from hold's identity

Hold Identity

did:web (default) — Derived from server.public_url with zero setup. https://hold.example.com becomes did:web:hold.example.com. The DID document is served at /.well-known/did.json. Tied to domain ownership — if you lose the domain, you lose the identity.

did:plc (portable) — Set database.did_method: plc in config. Registered with plc.directory. Survives domain changes. Requires a rotation key. If database.rotation_key is not set, a new K-256 key is generated in memory only and logged once via slog.Warn — it is never written to disk. You must copy it from the startup logs into database.rotation_key in your config immediately, or you will lose the ability to update or recover the DID. Only the signing key (database.key_path, default {database.path}/signing.key) is persisted to disk automatically. Use database.did to adopt an existing DID for recovery or migration.

Relay Subscription

A hold's embedded PDS is only useful to the wider network if a relay is subscribed to its firehose. Nothing about a hold's own operation depends on that subscription: pushes, pulls and the admin panel all work fine with zero relays connected. What breaks is everything downstream. The AppView learns about io.atcr.* records from Jetstream, which is fed by the Bluesky relays, so a hold no relay is listening to stops contributing live events and falls back to whatever the AppView's periodic backfill picks up.

This section covers how that subscription is established today, a failure mode that took the whole ATCR fleet off the network for two weeks in August 2026, how to diagnose and recover from it, and a proposed automatic fix that is not implemented.

How it works today

1. The hold announces itself once, at boot.

ServeWithListener fires a single requestCrawls() goroutine during startup (pkg/hold/server.go:400), skipped entirely in a -tags testmode build because local dev holds are not reachable by public relays. The implementation (pkg/hold/server.go:448-483) builds a deduplicated target list from atproto.KnownRelays (pkg/atproto/relays.go:22, the hardcoded list also documented in KNOWN_RELAYS.md) plus any extra entries in server.relay_endpoints, then POSTs com.atproto.sync.requestCrawl to all of them concurrently (pkg/atproto/relays.go:206-243). The request body is just {"hostname": "<host of server.public_url>"}. Per-relay failures are logged at warn level and never block startup.

This is the only automatic crawl request in the codebase. There is no ticker, no retry after the initial fan-out, and no check anywhere that a relay is actually subscribed afterwards.

2. Relays connect back over the firehose WebSocket.

A relay that accepts the crawl request dials com.atproto.sync.subscribeRepos on the hold (pkg/hold/pds/xrpc.go:184, handler at :1042), which registers it as a subscriber on the EventBroadcaster (pkg/hold/pds/events.go:440). The broadcaster keeps its subscribers in an unexported map (events.go:449) and tracks the current sequence number, readable via GetCurrentSeq() (events.go:1062-1066).

3. Events originate from repo commits.

Every write to the hold's repo goes through commitWrite (pkg/hold/pds/repo.go:141), which invokes the repo event handler registered at startup (pkg/hold/server.go:172-173): the records-index handler wrapping broadcaster.SetRepoEventHandler() (events.go:1052-1060). That calls Broadcast (events.go:547), which persists the event to the firehose_events table (events.go:650) and fans it out to every connected subscriber. The persisted table is what makes cursor-based backfill possible when a relay reconnects behind.

Failure mode: silently dropped by the relay

The relay side of this is one-directional and unforgiving. In the indigo version pinned by this repo (github.com/bluesky-social/indigo v0.0.0-20260901021441-b1f966883e38):

• subscribeWithRedialer (cmd/relay/relay/slurper.go:286) retries a failed dial with backoff, incrementing a counter each time. Once the counter passes 15, meaning 16 consecutive failed dials, it logs "host does not appear to be online, disabling for now", persists HostStatusOffline, and returns from the goroutine (slurper.go:332-339). Nothing schedules another attempt.
• ResubscribeAllHosts runs on relay startup but selects only hosts with status = "active" (cmd/relay/relay/crawl.go:59-63), so an offline host is not picked back up even by a relay restart.
• The only path back is a fresh com.atproto.sync.requestCrawl (cmd/relay/handlers.go:19-68 → SubscribeToHost, cmd/relay/relay/crawl.go:11-56), which resubscribes the existing host row and puts it back in rotation.

Combine that with the hold sending requestCrawl exactly once at boot, and the result is a trap: a hold that the relay drops stays invisible until its process restarts. Any transient outage long enough to burn 16 dials, a reverse proxy restart, a TLS renewal hiccup, a host reboot, a network partition, is permanent from the relay's point of view.

Worse, the failure is quiet. The hold logs nothing, because from its side nothing happened: a subscriber disconnected, which is routine. No metric counts connected subscribers. Nothing compares the hold's sequence number against any relay's view of it.

Worked example: the 2026-08-20 outage

Around 2026-08-20 the Bluesky relays marked every ATCR hold offline and stopped dialing them. It went unnoticed until 2026-09-03, when it surfaced indirectly as a complaint that push and pull counts on the site were up to 24 hours stale. The stale counters were a symptom. The actual state was that the entire fleet had been disconnected from the network for two weeks.

Measured read-only against the public relay, before the manual re-crawl:

relay1.us-west  getHostStatus?hostname=us-chi1.cove.seamark.dev
  -> {"accountCount":1,"seq":124596,"status":"offline"}

relay's view of the hold's repo:  rev 3mtji3hmiqi22   (2026-08-20 14:41 UTC)
hold's own view:                  rev 3mulfs3e63z22   (seconds old, still advancing)

Every ATCR hold the relay knew about was in the same state: hold01.atcr.io, hold.cetacean.club, hold.styx.mrijke.nl, hold.biglargeclarke.com, lexicon.store.

The downstream effect on the AppView is worth spelling out. The AppView consumes wss://jetstream{1,2}.{us-west,us-east}.bsky.network/subscribe (pkg/appview/config.go:277-282), which those same relays feed. With every hold offline at the relay, no live hold events reached the AppView at all, and the only remaining path was the periodic backfill (jetstream.backfill_interval, default 24h, pkg/appview/config.go:284). Hence "up to 24 h stale": that was the backfill period showing through, not a stats bug.

Diagnosing it

The fastest check is the Relays tab in the hold admin panel (/admin#relays, handler pkg/hold/admin/handlers_relays.go:67-106). For each known relay it probes status and capabilities and renders the relay's view of the hold's repo rev next to the hold's own current rev, so a stalled subscription shows up as a Behind (rev: ...) badge instead of Known (rev: ...). That rev comparison is the single most reliable tell.

Note one gap: the handler fetches getHostStatus into RelayStatusView.HostStatus (handlers_relays.go:101), but partials/relay_status.html never renders it. The Online badge on that row reflects whether the relay answered, not whether the relay considers this hold online. So the admin panel will happily show a green "Online" relay that has given up on you. Read the rev column, not the badge.

By hand, against any relay:

# 1. Does the relay still consider this hold a live host?
curl -s "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.getHostStatus?hostname=hold.example.com"
# {"hostname":"hold.example.com","seq":124596,"accountCount":1,"status":"offline"}

# 2. How far along is the relay's copy of the repo?
curl -s "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.getLatestCommit?did=did:web:hold.example.com"

# 3. How far along is the hold itself?
curl -s "https://hold.example.com/xrpc/com.atproto.sync.getLatestCommit?did=did:web:hold.example.com"

Reading the results:

• "status":"active" means the relay has a live subscription, or intends to. This is the healthy state.
• "status":"offline" means the redialer gave up after 16 failed dials and will never retry on its own. This requires a manual requestCrawl.
• "status":"banned" is different and not self-inflicted by an outage. SubscribeToHost refuses banned hosts outright (crawl.go:51-53), so a re-crawl will not help. Note that indigo bans a host if its Server response header contains atproto-relay (slurper.go:344-351), so check your reverse proxy is not adding one.
• The decisive signal is the pair of revs from calls 2 and 3. If the relay's rev is frozen at some point in the past while the hold's own rev keeps moving, the relay is not receiving your events. Since the same rev is queryable from both sides, the timestamp encoded in the frozen TID tells you roughly when you were dropped.

A relay reporting a rev equal to the hold's own is fully caught up and healthy. A rev slightly behind on a busy hold is normal lag, not a fault.

Recovering manually

Re-request a crawl. From the admin panel, the Relays tab has a per-relay "Request Crawl" button (POST /admin/relays/crawl, pkg/hold/admin/admin.go:518) and a crawl-all button that fans out to every entry in KnownRelays concurrently (POST /admin/relays/crawl-all, admin.go:519, handler handlers_relays.go:154-201). Both are captain-gated and both log the requesting DID.

Equivalently, by hand:

curl -s -X POST "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.requestCrawl" \
  -H 'Content-Type: application/json' \
  -d '{"hostname":"hold.example.com"}'

Restarting the hold process also works, since that re-runs the boot fan-out (server.go:400), but it is the blunt version of the same action.

Two things to understand about scope:

1. It is per host. A relay's offline marking is per-host row. Crawling one hold does nothing for any other hold, so during a fleet-wide event every hold's operator has to act independently. There is no bulk operation across holds.
2. It is per relay. Each relay keeps its own host table, so a hold can be active on one relay and offline on another. The admin crawl-all button covers every relay for one hold; it does not cover every hold.

After crawling, re-run the getHostStatus check. A successful recovery flips status to active and the seq starts advancing again.

Warning: check for the replay fix before provoking a re-crawl

A relay coming back resumes from its stored cursor, which after a long outage is far behind. That makes the hold replay history out of firehose_events to the reconnecting subscriber. Before commit ca539b1 ("hold/pds: stop firehose backfill panicking on subscriber disconnect", 2026-08-08) that replay path could take the entire hold process down with panic: send on closed channel if the relay dropped mid-backfill: the backfill goroutine wrote to sub.send without holding the broadcaster lock while Unsubscribe closed that channel underneath it.

The bug only fires when a backfill goroutine exists, which Subscribe skips when the subscriber's cursor already equals the current sequence. That is exactly why it stayed hidden: caught-up relays never triggered it, and a hold whose relays are all far behind is exposed on every single reconnect. Provoking a re-crawl on an old build is therefore close to the worst case for it.

Verify the deployed build contains the fix before requesting a crawl:

git merge-base --is-ancestor ca539b1 <deployed-commit> && echo "safe" || echo "NOT SAFE - upgrade first"

As of this writing the build deployed to seamark (efabb677, 2026-05-26) does not contain it. Upgrade the hold before re-crawling, or expect the process to fall over partway through the replay.

Proposed automatic fix (NOT IMPLEMENTED)

Status: deferred by decision. None of the following exists in the codebase. The 2026-08-20 outage was resolved by requesting crawls manually. The design below is recorded so that whoever revisits this does not have to rederive it. Do not read this section as a description of current behaviour.

The shape of the fix:

1. Keep the boot requestCrawl exactly as it is. It is correct for the common case of a new or restarted hold, and it is what makes a hold discoverable at all.

2. Add a jittered re-crawl ticker, roughly hourly. An hour is short relative to the two weeks this outage ran and long relative to how quickly a relay recovers, and it bounds worst-case invisibility to about an hour. The jitter matters: without it, every hold that started from the same deployment fires at the same instant and the fan-out arrives at the relays as a synchronized burst.

3. Guard the ticker on liveness, so it normally does nothing. The EventBroadcaster already knows whether anyone is connected: it holds the subscriber set at events.go:449 and the sequence number at events.go:1062-1066. If a relay-shaped subscriber is connected, skip the fan-out entirely and the ticker costs nothing. Only when no such subscriber is present does it re-announce. This makes the mechanism a recovery path rather than a periodic broadcast, which is the whole reason it is tolerable to run it on every hold.

   Two small pieces of plumbing this needs, neither of which exists today: the broadcaster exposes no accessor for the subscriber count, only the unexported map; and Subscriber (events.go:42-54) does not retain the userAgent that Subscribe receives and logs (events.go:440, :453), so "relay-shaped" cannot currently be evaluated. Either store the user agent, or settle for a plain non-zero subscriber count and accept that a curious human with websocat attached would suppress a re-crawl.

4. Surface liveness somewhere visible. This is arguably more important than the ticker. The deeper problem was not that recovery required a manual step, it was that two weeks passed before anyone knew there was anything to recover. Exposing the subscriber count and GetCurrentSeq() on the health endpoint or the admin dashboard would have made "zero subscribers, seq unchanged for a fortnight" visible at a glance. Rendering the already-fetched HostStatus in partials/relay_status.html is a near-free improvement in the same direction.

Risks to weigh:

• Relay rate limiting. An unconditional hourly fan-out from every hold to all 15 entries in KnownRelays is a meaningful amount of unsolicited traffic and a plausible way to get an IP or a domain throttled. The liveness guard is the primary mitigation, since a healthy hold sends nothing at all, and the jitter is the secondary one. A disconnected hold that re-crawls hourly is a rounding error; a fleet of healthy holds all re-crawling hourly is not.
• The replay hazard above becomes automatic. A re-crawl ticker means reconnects with stale cursors happen without anyone watching. This is fine on any build containing ca539b1, and it is a way to crash a hold repeatedly on any build that predates it. Any implementation of this should be gated on that fix being present, not merely assumed.
• getHostStatus polling was considered and rejected as the trigger. Asking each relay whether it still likes us is more direct than inferring from the subscriber count, but it is N outbound requests per interval per hold whether or not anything is wrong, which is precisely the traffic pattern the liveness guard is trying to avoid.

Verification

After starting your hold, verify it's working:

# Health check — should return {"version":"..."}
curl https://hold.example.com/xrpc/_health

# DID document — should return valid JSON with service endpoints
curl https://hold.example.com/.well-known/did.json

# Captain record — should show your owner DID
curl "https://hold.example.com/xrpc/com.atproto.repo.listRecords?repo=HOLD_DID&collection=io.atcr.hold.captain"

# Crew records
curl "https://hold.example.com/xrpc/com.atproto.repo.listRecords?repo=HOLD_DID&collection=io.atcr.hold.crew"

Replace HOLD_DID with your hold's DID (from the /.well-known/did.json response).

Docker Compose

services:
  atcr-hold:
    build:
      context: .
      dockerfile: Dockerfile.hold
    command: ["serve", "--config", "/config.yaml"]
    volumes:
      - ./config-hold.yaml:/config.yaml:ro
      - atcr-hold-data:/var/lib/atcr-hold
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "/healthcheck", "http://localhost:8080/xrpc/_health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s

volumes:
  atcr-hold-data:

For production with TLS termination, see deploy/docker-compose.prod.yml which includes a Caddy reverse proxy.

Further Reading

• config-hold.example.yaml — Complete configuration reference with inline comments
• BYOS.md — Bring Your Own Storage architecture and authorization model
• KNOWN_RELAYS.md — Relay list and capabilities (see also Relay Subscription above)
• HOLD_XRPC_ENDPOINTS.md — XRPC endpoint reference
• BILLING.md — Stripe billing integration
• QUOTAS.md — Quota management
• SBOM_SCANNING.md — Vulnerability scanning