docs: record how a hold falls off the relay and stays off

Around 2026-08-20 the Bluesky relay marked every ATCR hold offline and stopped dialing. Nobody noticed for two weeks, and it surfaced only indirectly as "pull and push counts are up to 24 h stale". The stats were a symptom; the fleet was simply disconnected. It cannot recover on its own. Indigo's relay gives up on a host after 16 consecutive dial failures and returns from the redialer, and only a fresh requestCrawl revives it. The hold sends requestCrawl exactly once, at boot (server.go:400), with no ticker and no check that any relay is subscribed. So a dropped hold is invisible until its process restarts, silently. Documents the current mechanics, the failure mode, how to diagnose it with getHostStatus and a frozen repo rev, and how to recover. The automatic re-crawl is described as a deferred proposal and explicitly NOT implemented, by decision: a jittered ticker guarded on subscriber liveness, plus surfacing the subscriber count, since the deeper problem is that this was silent. Two things found while writing it, both recorded. The proposal needs plumbing that does not exist: EventBroadcaster has no exported subscriber count, and Subscriber does not retain the userAgent, so "is a relay listening" cannot currently be answered. And ResubscribeAllHosts selects only active hosts, so an offline host is not recovered even by a relay restart. Carries a replay warning. ca539b1 fixed a panic on subscriber disconnect during firehose backfill, and the exposure condition is that a backfill goroutine exists at all, which Subscribe skips when the cursor is current. So a caught-up relay never triggered it and a hold whose relays are far behind is exposed on every reconnect. Verify a deployed hold contains ca539b1 before provoking a re-crawl; efabb677 does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9
2026-09-20 01:04:15 +00:00 · 2026-09-02 22:30:37 -05:00
parent 25e2aa0228
commit a95c89aaef
3 changed files with 299 additions and 0 deletions
@@ -39,6 +39,8 @@ Last verified: 2026-02-08

 The hold announces its embedded PDS to relays on startup via `com.atproto.sync.requestCrawl`. On startup, `requestCrawls()` fans out to every relay in `KnownRelays` (all 15 entries hardcoded in `pkg/atproto/relays.go`) plus any additional entries in `server.relay_endpoints` (a list; defaults to `relay1.us-east.bsky.network` and `relay1.us-west.bsky.network`). Per-relay failures are logged but never block startup. All healthy relays above accept `requestCrawl`.

+That boot fan-out is the *only* automatic crawl request the hold makes. A relay that drops a hold (indigo marks a host `offline` after 16 consecutive failed dials and then stops retrying) will not pick it back up until someone requests a crawl again or the hold process restarts. See [hold.md, "Relay Subscription"](hold.md#relay-subscription) for the failure mode, the 2026-08-20 fleet outage it caused, and how to diagnose and recover.
+
 ### Appview backfill (`listReposByCollection`)

 The appview uses `com.atproto.sync.listReposByCollection` to discover DIDs with `io.atcr.*` records during backfill. Only Bluesky's regional relays support this endpoint. The appview's `jetstream.relay_endpoints` defaults to both `relay1.us-east.bsky.network` and `relay1.us-west.bsky.network` with failover between them.
@@ -316,6 +316,35 @@ error: failed to upload blob: connection refused

 ---

+### Hold Dropped by the Relay
+
+**Symptom:**
+
+Push and pull counts on the AppView go stale (up to 24 hours behind), or the hold's
+records stop appearing anywhere on the network, while the hold itself serves pushes and
+pulls normally and logs no errors at all.
+
+**Diagnosis:**
+
+```bash
+curl -s "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.getHostStatus?hostname=hold.example.com"
+```
+
+`"status":"offline"` means the relay gave up dialing your hold and, because the hold only
+sends `requestCrawl` at boot, it will never retry on its own.
+
+**Solution:**
+
+Request a crawl from the hold admin panel's Relays tab, or restart the hold.
+
+**Before doing either**, confirm the deployed build contains commit `ca539b1`. Older
+builds can crash during the firehose replay that a reconnecting relay triggers.
+
+Full write-up, including the 2026-08-20 fleet outage, the diagnosis procedure, and a
+deferred proposal for automatic recovery: [hold.md, "Relay Subscription"](hold.md#relay-subscription).
+
+---
+
 ## Performance Issues

 ### High Database Lock Contention
@@ -132,6 +132,273 @@ See [BYOS.md](BYOS.md) for the full authorization model.

 **did:plc (portable)** — Set `database.did_method: plc` in config. Registered with plc.directory. Survives domain changes. Requires a rotation key. If `database.rotation_key` is not set, a new K-256 key is generated **in memory only** and logged once via `slog.Warn` — it is never written to disk. You must copy it from the startup logs into `database.rotation_key` in your config immediately, or you will lose the ability to update or recover the DID. Only the signing key (`database.key_path`, default `{database.path}/signing.key`) is persisted to disk automatically. Use `database.did` to adopt an existing DID for recovery or migration.

+## Relay Subscription
+
+A hold's embedded PDS is only useful to the wider network if a relay is subscribed to its
+firehose. Nothing about a hold's own operation depends on that subscription: pushes, pulls
+and the admin panel all work fine with zero relays connected. What breaks is everything
+downstream. The AppView learns about `io.atcr.*` records from Jetstream, which is fed by
+the Bluesky relays, so a hold no relay is listening to stops contributing live events and
+falls back to whatever the AppView's periodic backfill picks up.
+
+This section covers how that subscription is established today, a failure mode that took
+the whole ATCR fleet off the network for two weeks in August 2026, how to diagnose and
+recover from it, and a proposed automatic fix that is **not implemented**.
+
+### How it works today
+
+**1. The hold announces itself once, at boot.**
+
+`ServeWithListener` fires a single `requestCrawls()` goroutine during startup
+(`pkg/hold/server.go:400`), skipped entirely when `server.test_mode` is set because local
+dev holds are not reachable by public relays. The implementation
+(`pkg/hold/server.go:448-483`) builds a deduplicated target list from
+`atproto.KnownRelays` (`pkg/atproto/relays.go:22`, the hardcoded list also documented in
+[KNOWN_RELAYS.md](KNOWN_RELAYS.md)) plus any extra entries in `server.relay_endpoints`,
+then POSTs `com.atproto.sync.requestCrawl` to all of them concurrently
+(`pkg/atproto/relays.go:206-243`). The request body is just `{"hostname": "<host of
+server.public_url>"}`. Per-relay failures are logged at warn level and never block
+startup.
+
+This is the only automatic crawl request in the codebase. There is no ticker, no retry
+after the initial fan-out, and no check anywhere that a relay is actually subscribed
+afterwards.
+
+**2. Relays connect back over the firehose WebSocket.**
+
+A relay that accepts the crawl request dials `com.atproto.sync.subscribeRepos` on the hold
+(`pkg/hold/pds/xrpc.go:184`, handler at `:1042`), which registers it as a subscriber on
+the `EventBroadcaster` (`pkg/hold/pds/events.go:440`). The broadcaster keeps its
+subscribers in an unexported map (`events.go:449`) and tracks the current sequence number,
+readable via `GetCurrentSeq()` (`events.go:1062-1066`).
+
+**3. Events originate from repo commits.**
+
+Every write to the hold's repo goes through `commitWrite` (`pkg/hold/pds/repo.go:141`),
+which invokes the repo event handler registered at startup
+(`pkg/hold/server.go:172-173`): the records-index handler wrapping
+`broadcaster.SetRepoEventHandler()` (`events.go:1052-1060`). That calls `Broadcast`
+(`events.go:547`), which persists the event to the `firehose_events` table
+(`events.go:650`) and fans it out to every connected subscriber. The persisted table is
+what makes cursor-based backfill possible when a relay reconnects behind.
+
+### Failure mode: silently dropped by the relay
+
+The relay side of this is one-directional and unforgiving. In the indigo version pinned by
+this repo (`github.com/bluesky-social/indigo v0.0.0-20260901021441-b1f966883e38`):
+
+- `subscribeWithRedialer` (`cmd/relay/relay/slurper.go:286`) retries a failed dial with
+  backoff, incrementing a counter each time. Once the counter passes 15, meaning 16
+  consecutive failed dials, it logs "host does not appear to be online, disabling for
+  now", persists `HostStatusOffline`, and **returns from the goroutine**
+  (`slurper.go:332-339`). Nothing schedules another attempt.
+- `ResubscribeAllHosts` runs on relay startup but selects only hosts with
+  `status = "active"` (`cmd/relay/relay/crawl.go:59-63`), so an offline host is not picked
+  back up even by a relay restart.
+- The only path back is a fresh `com.atproto.sync.requestCrawl`
+  (`cmd/relay/handlers.go:19-68` → `SubscribeToHost`, `cmd/relay/relay/crawl.go:11-56`),
+  which resubscribes the existing host row and puts it back in rotation.
+
+Combine that with the hold sending `requestCrawl` exactly once at boot, and the result is
+a trap: **a hold that the relay drops stays invisible until its process restarts.** Any
+transient outage long enough to burn 16 dials, a reverse proxy restart, a TLS renewal
+hiccup, a host reboot, a network partition, is permanent from the relay's point of view.
+
+Worse, the failure is quiet. The hold logs nothing, because from its side nothing
+happened: a subscriber disconnected, which is routine. No metric counts connected
+subscribers. Nothing compares the hold's sequence number against any relay's view of it.
+
+#### Worked example: the 2026-08-20 outage
+
+Around 2026-08-20 the Bluesky relays marked every ATCR hold `offline` and stopped dialing
+them. It went unnoticed until 2026-09-03, when it surfaced indirectly as a complaint that
+push and pull counts on the site were up to 24 hours stale. The stale counters were a
+symptom. The actual state was that the entire fleet had been disconnected from the network
+for two weeks.
+
+Measured read-only against the public relay, before the manual re-crawl:
+
+```
+relay1.us-west  getHostStatus?hostname=us-chi1.cove.seamark.dev
+  -> {"accountCount":1,"seq":124596,"status":"offline"}
+
+relay's view of the hold's repo:  rev 3mtji3hmiqi22   (2026-08-20 14:41 UTC)
+hold's own view:                  rev 3mulfs3e63z22   (seconds old, still advancing)
+```
+
+Every ATCR hold the relay knew about was in the same state: `hold01.atcr.io`,
+`hold.cetacean.club`, `hold.styx.mrijke.nl`, `hold.biglargeclarke.com`, `lexicon.store`.
+
+The downstream effect on the AppView is worth spelling out. The AppView consumes
+`wss://jetstream{1,2}.{us-west,us-east}.bsky.network/subscribe`
+(`pkg/appview/config.go:277-282`), which those same relays feed. With every hold offline
+at the relay, no live hold events reached the AppView at all, and the only remaining path
+was the periodic backfill (`jetstream.backfill_interval`, default `24h`,
+`pkg/appview/config.go:284`). Hence "up to 24 h stale": that was the backfill period
+showing through, not a stats bug.
+
+### Diagnosing it
+
+The fastest check is the **Relays tab in the hold admin panel**
+(`/admin#relays`, handler `pkg/hold/admin/handlers_relays.go:67-106`). For each known
+relay it probes status and capabilities and renders the relay's view of the hold's repo
+rev next to the hold's own current rev, so a stalled subscription shows up as a `Behind
+(rev: ...)` badge instead of `Known (rev: ...)`. That rev comparison is the single most
+reliable tell.
+
+Note one gap: the handler fetches `getHostStatus` into `RelayStatusView.HostStatus`
+(`handlers_relays.go:101`), but `partials/relay_status.html` never renders it. The
+`Online` badge on that row reflects whether the *relay* answered, not whether the relay
+considers *this hold* online. So the admin panel will happily show a green "Online" relay
+that has given up on you. Read the rev column, not the badge.
+
+By hand, against any relay:
+
+```bash
+# 1. Does the relay still consider this hold a live host?
+curl -s "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.getHostStatus?hostname=hold.example.com"
+# {"hostname":"hold.example.com","seq":124596,"accountCount":1,"status":"offline"}
+
+# 2. How far along is the relay's copy of the repo?
+curl -s "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.getLatestCommit?did=did:web:hold.example.com"
+
+# 3. How far along is the hold itself?
+curl -s "https://hold.example.com/xrpc/com.atproto.sync.getLatestCommit?did=did:web:hold.example.com"
+```
+
+Reading the results:
+
+- `"status":"active"` means the relay has a live subscription, or intends to. This is the
+  healthy state.
+- `"status":"offline"` means the redialer gave up after 16 failed dials and will never
+  retry on its own. This requires a manual `requestCrawl`.
+- `"status":"banned"` is different and not self-inflicted by an outage.
+  `SubscribeToHost` refuses banned hosts outright (`crawl.go:51-53`), so a re-crawl will
+  not help. Note that indigo bans a host if its `Server` response header contains
+  `atproto-relay` (`slurper.go:344-351`), so check your reverse proxy is not adding one.
+- The decisive signal is the pair of revs from calls 2 and 3. **If the relay's rev is
+  frozen at some point in the past while the hold's own rev keeps moving, the relay is not
+  receiving your events.** Since the same rev is queryable from both sides, the timestamp
+  encoded in the frozen TID tells you roughly when you were dropped.
+
+A relay reporting a rev equal to the hold's own is fully caught up and healthy. A rev
+slightly behind on a busy hold is normal lag, not a fault.
+
+### Recovering manually
+
+Re-request a crawl. From the admin panel, the Relays tab has a per-relay "Request Crawl"
+button (`POST /admin/relays/crawl`, `pkg/hold/admin/admin.go:518`) and a crawl-all button
+that fans out to every entry in `KnownRelays` concurrently (`POST
+/admin/relays/crawl-all`, `admin.go:519`, handler `handlers_relays.go:154-201`). Both are
+captain-gated and both log the requesting DID.
+
+Equivalently, by hand:
+
+```bash
+curl -s -X POST "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.requestCrawl" \
+  -H 'Content-Type: application/json' \
+  -d '{"hostname":"hold.example.com"}'
+```
+
+Restarting the hold process also works, since that re-runs the boot fan-out
+(`server.go:400`), but it is the blunt version of the same action.
+
+Two things to understand about scope:
+
+1. **It is per host.** A relay's offline marking is per-host row. Crawling one hold does
+   nothing for any other hold, so during a fleet-wide event every hold's operator has to
+   act independently. There is no bulk operation across holds.
+2. **It is per relay.** Each relay keeps its own host table, so a hold can be `active` on
+   one relay and `offline` on another. The admin crawl-all button covers every relay for
+   one hold; it does not cover every hold.
+
+After crawling, re-run the `getHostStatus` check. A successful recovery flips `status` to
+`active` and the `seq` starts advancing again.
+
+> ### Warning: check for the replay fix before provoking a re-crawl
+>
+> A relay coming back resumes from its stored cursor, which after a long outage is far
+> behind. That makes the hold replay history out of `firehose_events` to the reconnecting
+> subscriber. Before commit `ca539b1` ("hold/pds: stop firehose backfill panicking on
+> subscriber disconnect", 2026-08-08) that replay path could **take the entire hold process
+> down** with `panic: send on closed channel` if the relay dropped mid-backfill: the
+> backfill goroutine wrote to `sub.send` without holding the broadcaster lock while
+> `Unsubscribe` closed that channel underneath it.
+>
+> The bug only fires when a backfill goroutine exists, which `Subscribe` skips when the
+> subscriber's cursor already equals the current sequence. That is exactly why it stayed
+> hidden: caught-up relays never triggered it, and a hold whose relays are all far behind
+> is exposed on every single reconnect. Provoking a re-crawl on an old build is therefore
+> close to the worst case for it.
+>
+> **Verify the deployed build contains the fix before requesting a crawl:**
+>
+> ```bash
+> git merge-base --is-ancestor ca539b1 <deployed-commit> && echo "safe" || echo "NOT SAFE - upgrade first"
+> ```
+>
+> As of this writing the build deployed to seamark (`efabb677`, 2026-05-26) does **not**
+> contain it. Upgrade the hold before re-crawling, or expect the process to fall over
+> partway through the replay.
+
+### Proposed automatic fix (NOT IMPLEMENTED)
+
+> **Status: deferred by decision. None of the following exists in the codebase.** The
+> 2026-08-20 outage was resolved by requesting crawls manually. The design below is
+> recorded so that whoever revisits this does not have to rederive it. Do not read this
+> section as a description of current behaviour.
+
+The shape of the fix:
+
+1. **Keep the boot `requestCrawl` exactly as it is.** It is correct for the common case of
+   a new or restarted hold, and it is what makes a hold discoverable at all.
+
+2. **Add a jittered re-crawl ticker, roughly hourly.** An hour is short relative to the
+   two weeks this outage ran and long relative to how quickly a relay recovers, and it
+   bounds worst-case invisibility to about an hour. The jitter matters: without it, every
+   hold that started from the same deployment fires at the same instant and the fan-out
+   arrives at the relays as a synchronized burst.
+
+3. **Guard the ticker on liveness, so it normally does nothing.** The `EventBroadcaster`
+   already knows whether anyone is connected: it holds the subscriber set at
+   `events.go:449` and the sequence number at `events.go:1062-1066`. If a relay-shaped
+   subscriber is connected, skip the fan-out entirely and the ticker costs nothing. Only
+   when no such subscriber is present does it re-announce. This makes the mechanism a
+   recovery path rather than a periodic broadcast, which is the whole reason it is
+   tolerable to run it on every hold.
+
+   Two small pieces of plumbing this needs, neither of which exists today: the broadcaster
+   exposes no accessor for the subscriber count, only the unexported map; and `Subscriber`
+   (`events.go:42-54`) does not retain the `userAgent` that `Subscribe` receives and logs
+   (`events.go:440`, `:453`), so "relay-shaped" cannot currently be evaluated. Either store
+   the user agent, or settle for a plain non-zero subscriber count and accept that a
+   curious human with `websocat` attached would suppress a re-crawl.
+
+4. **Surface liveness somewhere visible.** This is arguably more important than the
+   ticker. The deeper problem was not that recovery required a manual step, it was that
+   two weeks passed before anyone knew there was anything to recover. Exposing the
+   subscriber count and `GetCurrentSeq()` on the health endpoint or the admin dashboard
+   would have made "zero subscribers, seq unchanged for a fortnight" visible at a glance.
+   Rendering the already-fetched `HostStatus` in `partials/relay_status.html` is a
+   near-free improvement in the same direction.
+
+Risks to weigh:
+
+- **Relay rate limiting.** An unconditional hourly fan-out from every hold to all 15
+  entries in `KnownRelays` is a meaningful amount of unsolicited traffic and a plausible
+  way to get an IP or a domain throttled. The liveness guard is the primary mitigation,
+  since a healthy hold sends nothing at all, and the jitter is the secondary one. A
+  disconnected hold that re-crawls hourly is a rounding error; a fleet of healthy holds
+  all re-crawling hourly is not.
+- **The replay hazard above becomes automatic.** A re-crawl ticker means reconnects with
+  stale cursors happen without anyone watching. This is fine on any build containing
+  `ca539b1`, and it is a way to crash a hold repeatedly on any build that predates it. Any
+  implementation of this should be gated on that fix being present, not merely assumed.
+- **`getHostStatus` polling was considered and rejected as the trigger.** Asking each relay
+  whether it still likes us is more direct than inferring from the subscriber count, but it
+  is N outbound requests per interval per hold whether or not anything is wrong, which is
+  precisely the traffic pattern the liveness guard is trying to avoid.
+
 ## Verification

 After starting your hold, verify it's working:
@@ -183,6 +450,7 @@ For production with TLS termination, see [`deploy/docker-compose.prod.yml`](../d

 - [`config-hold.example.yaml`](../config-hold.example.yaml) — Complete configuration reference with inline comments
 - [BYOS.md](BYOS.md) — Bring Your Own Storage architecture and authorization model
+- [KNOWN_RELAYS.md](KNOWN_RELAYS.md) — Relay list and capabilities (see also [Relay Subscription](#relay-subscription) above)
 - [HOLD_XRPC_ENDPOINTS.md](HOLD_XRPC_ENDPOINTS.md) — XRPC endpoint reference
 - [BILLING.md](BILLING.md) — Stripe billing integration
 - [QUOTAS.md](QUOTAS.md) — Quota management