diff --git a/docs/KNOWN_RELAYS.md b/docs/KNOWN_RELAYS.md index 577d4dc..bbe60d0 100644 --- a/docs/KNOWN_RELAYS.md +++ b/docs/KNOWN_RELAYS.md @@ -39,6 +39,8 @@ Last verified: 2026-02-08 The hold announces its embedded PDS to relays on startup via `com.atproto.sync.requestCrawl`. On startup, `requestCrawls()` fans out to every relay in `KnownRelays` (all 15 entries hardcoded in `pkg/atproto/relays.go`) plus any additional entries in `server.relay_endpoints` (a list; defaults to `relay1.us-east.bsky.network` and `relay1.us-west.bsky.network`). Per-relay failures are logged but never block startup. All healthy relays above accept `requestCrawl`. +That boot fan-out is the *only* automatic crawl request the hold makes. A relay that drops a hold (indigo marks a host `offline` after 16 consecutive failed dials and then stops retrying) will not pick it back up until someone requests a crawl again or the hold process restarts. See [hold.md, "Relay Subscription"](hold.md#relay-subscription) for the failure mode, the 2026-08-20 fleet outage it caused, and how to diagnose and recover. + ### Appview backfill (`listReposByCollection`) The appview uses `com.atproto.sync.listReposByCollection` to discover DIDs with `io.atcr.*` records during backfill. Only Bluesky's regional relays support this endpoint. The appview's `jetstream.relay_endpoints` defaults to both `relay1.us-east.bsky.network` and `relay1.us-west.bsky.network` with failover between them. diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index bf92706..d25ae3c 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -316,6 +316,35 @@ error: failed to upload blob: connection refused --- +### Hold Dropped by the Relay + +**Symptom:** + +Push and pull counts on the AppView go stale (up to 24 hours behind), or the hold's +records stop appearing anywhere on the network, while the hold itself serves pushes and +pulls normally and logs no errors at all. + +**Diagnosis:** + +```bash +curl -s "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.getHostStatus?hostname=hold.example.com" +``` + +`"status":"offline"` means the relay gave up dialing your hold and, because the hold only +sends `requestCrawl` at boot, it will never retry on its own. + +**Solution:** + +Request a crawl from the hold admin panel's Relays tab, or restart the hold. + +**Before doing either**, confirm the deployed build contains commit `ca539b1`. Older +builds can crash during the firehose replay that a reconnecting relay triggers. + +Full write-up, including the 2026-08-20 fleet outage, the diagnosis procedure, and a +deferred proposal for automatic recovery: [hold.md, "Relay Subscription"](hold.md#relay-subscription). + +--- + ## Performance Issues ### High Database Lock Contention diff --git a/docs/hold.md b/docs/hold.md index 00d8ab0..2acac71 100644 --- a/docs/hold.md +++ b/docs/hold.md @@ -132,6 +132,273 @@ See [BYOS.md](BYOS.md) for the full authorization model. **did:plc (portable)** — Set `database.did_method: plc` in config. Registered with plc.directory. Survives domain changes. Requires a rotation key. If `database.rotation_key` is not set, a new K-256 key is generated **in memory only** and logged once via `slog.Warn` — it is never written to disk. You must copy it from the startup logs into `database.rotation_key` in your config immediately, or you will lose the ability to update or recover the DID. Only the signing key (`database.key_path`, default `{database.path}/signing.key`) is persisted to disk automatically. Use `database.did` to adopt an existing DID for recovery or migration. +## Relay Subscription + +A hold's embedded PDS is only useful to the wider network if a relay is subscribed to its +firehose. Nothing about a hold's own operation depends on that subscription: pushes, pulls +and the admin panel all work fine with zero relays connected. What breaks is everything +downstream. The AppView learns about `io.atcr.*` records from Jetstream, which is fed by +the Bluesky relays, so a hold no relay is listening to stops contributing live events and +falls back to whatever the AppView's periodic backfill picks up. + +This section covers how that subscription is established today, a failure mode that took +the whole ATCR fleet off the network for two weeks in August 2026, how to diagnose and +recover from it, and a proposed automatic fix that is **not implemented**. + +### How it works today + +**1. The hold announces itself once, at boot.** + +`ServeWithListener` fires a single `requestCrawls()` goroutine during startup +(`pkg/hold/server.go:400`), skipped entirely when `server.test_mode` is set because local +dev holds are not reachable by public relays. The implementation +(`pkg/hold/server.go:448-483`) builds a deduplicated target list from +`atproto.KnownRelays` (`pkg/atproto/relays.go:22`, the hardcoded list also documented in +[KNOWN_RELAYS.md](KNOWN_RELAYS.md)) plus any extra entries in `server.relay_endpoints`, +then POSTs `com.atproto.sync.requestCrawl` to all of them concurrently +(`pkg/atproto/relays.go:206-243`). The request body is just `{"hostname": ""}`. Per-relay failures are logged at warn level and never block +startup. + +This is the only automatic crawl request in the codebase. There is no ticker, no retry +after the initial fan-out, and no check anywhere that a relay is actually subscribed +afterwards. + +**2. Relays connect back over the firehose WebSocket.** + +A relay that accepts the crawl request dials `com.atproto.sync.subscribeRepos` on the hold +(`pkg/hold/pds/xrpc.go:184`, handler at `:1042`), which registers it as a subscriber on +the `EventBroadcaster` (`pkg/hold/pds/events.go:440`). The broadcaster keeps its +subscribers in an unexported map (`events.go:449`) and tracks the current sequence number, +readable via `GetCurrentSeq()` (`events.go:1062-1066`). + +**3. Events originate from repo commits.** + +Every write to the hold's repo goes through `commitWrite` (`pkg/hold/pds/repo.go:141`), +which invokes the repo event handler registered at startup +(`pkg/hold/server.go:172-173`): the records-index handler wrapping +`broadcaster.SetRepoEventHandler()` (`events.go:1052-1060`). That calls `Broadcast` +(`events.go:547`), which persists the event to the `firehose_events` table +(`events.go:650`) and fans it out to every connected subscriber. The persisted table is +what makes cursor-based backfill possible when a relay reconnects behind. + +### Failure mode: silently dropped by the relay + +The relay side of this is one-directional and unforgiving. In the indigo version pinned by +this repo (`github.com/bluesky-social/indigo v0.0.0-20260901021441-b1f966883e38`): + +- `subscribeWithRedialer` (`cmd/relay/relay/slurper.go:286`) retries a failed dial with + backoff, incrementing a counter each time. Once the counter passes 15, meaning 16 + consecutive failed dials, it logs "host does not appear to be online, disabling for + now", persists `HostStatusOffline`, and **returns from the goroutine** + (`slurper.go:332-339`). Nothing schedules another attempt. +- `ResubscribeAllHosts` runs on relay startup but selects only hosts with + `status = "active"` (`cmd/relay/relay/crawl.go:59-63`), so an offline host is not picked + back up even by a relay restart. +- The only path back is a fresh `com.atproto.sync.requestCrawl` + (`cmd/relay/handlers.go:19-68` → `SubscribeToHost`, `cmd/relay/relay/crawl.go:11-56`), + which resubscribes the existing host row and puts it back in rotation. + +Combine that with the hold sending `requestCrawl` exactly once at boot, and the result is +a trap: **a hold that the relay drops stays invisible until its process restarts.** Any +transient outage long enough to burn 16 dials, a reverse proxy restart, a TLS renewal +hiccup, a host reboot, a network partition, is permanent from the relay's point of view. + +Worse, the failure is quiet. The hold logs nothing, because from its side nothing +happened: a subscriber disconnected, which is routine. No metric counts connected +subscribers. Nothing compares the hold's sequence number against any relay's view of it. + +#### Worked example: the 2026-08-20 outage + +Around 2026-08-20 the Bluesky relays marked every ATCR hold `offline` and stopped dialing +them. It went unnoticed until 2026-09-03, when it surfaced indirectly as a complaint that +push and pull counts on the site were up to 24 hours stale. The stale counters were a +symptom. The actual state was that the entire fleet had been disconnected from the network +for two weeks. + +Measured read-only against the public relay, before the manual re-crawl: + +``` +relay1.us-west getHostStatus?hostname=us-chi1.cove.seamark.dev + -> {"accountCount":1,"seq":124596,"status":"offline"} + +relay's view of the hold's repo: rev 3mtji3hmiqi22 (2026-08-20 14:41 UTC) +hold's own view: rev 3mulfs3e63z22 (seconds old, still advancing) +``` + +Every ATCR hold the relay knew about was in the same state: `hold01.atcr.io`, +`hold.cetacean.club`, `hold.styx.mrijke.nl`, `hold.biglargeclarke.com`, `lexicon.store`. + +The downstream effect on the AppView is worth spelling out. The AppView consumes +`wss://jetstream{1,2}.{us-west,us-east}.bsky.network/subscribe` +(`pkg/appview/config.go:277-282`), which those same relays feed. With every hold offline +at the relay, no live hold events reached the AppView at all, and the only remaining path +was the periodic backfill (`jetstream.backfill_interval`, default `24h`, +`pkg/appview/config.go:284`). Hence "up to 24 h stale": that was the backfill period +showing through, not a stats bug. + +### Diagnosing it + +The fastest check is the **Relays tab in the hold admin panel** +(`/admin#relays`, handler `pkg/hold/admin/handlers_relays.go:67-106`). For each known +relay it probes status and capabilities and renders the relay's view of the hold's repo +rev next to the hold's own current rev, so a stalled subscription shows up as a `Behind +(rev: ...)` badge instead of `Known (rev: ...)`. That rev comparison is the single most +reliable tell. + +Note one gap: the handler fetches `getHostStatus` into `RelayStatusView.HostStatus` +(`handlers_relays.go:101`), but `partials/relay_status.html` never renders it. The +`Online` badge on that row reflects whether the *relay* answered, not whether the relay +considers *this hold* online. So the admin panel will happily show a green "Online" relay +that has given up on you. Read the rev column, not the badge. + +By hand, against any relay: + +```bash +# 1. Does the relay still consider this hold a live host? +curl -s "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.getHostStatus?hostname=hold.example.com" +# {"hostname":"hold.example.com","seq":124596,"accountCount":1,"status":"offline"} + +# 2. How far along is the relay's copy of the repo? +curl -s "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.getLatestCommit?did=did:web:hold.example.com" + +# 3. How far along is the hold itself? +curl -s "https://hold.example.com/xrpc/com.atproto.sync.getLatestCommit?did=did:web:hold.example.com" +``` + +Reading the results: + +- `"status":"active"` means the relay has a live subscription, or intends to. This is the + healthy state. +- `"status":"offline"` means the redialer gave up after 16 failed dials and will never + retry on its own. This requires a manual `requestCrawl`. +- `"status":"banned"` is different and not self-inflicted by an outage. + `SubscribeToHost` refuses banned hosts outright (`crawl.go:51-53`), so a re-crawl will + not help. Note that indigo bans a host if its `Server` response header contains + `atproto-relay` (`slurper.go:344-351`), so check your reverse proxy is not adding one. +- The decisive signal is the pair of revs from calls 2 and 3. **If the relay's rev is + frozen at some point in the past while the hold's own rev keeps moving, the relay is not + receiving your events.** Since the same rev is queryable from both sides, the timestamp + encoded in the frozen TID tells you roughly when you were dropped. + +A relay reporting a rev equal to the hold's own is fully caught up and healthy. A rev +slightly behind on a busy hold is normal lag, not a fault. + +### Recovering manually + +Re-request a crawl. From the admin panel, the Relays tab has a per-relay "Request Crawl" +button (`POST /admin/relays/crawl`, `pkg/hold/admin/admin.go:518`) and a crawl-all button +that fans out to every entry in `KnownRelays` concurrently (`POST +/admin/relays/crawl-all`, `admin.go:519`, handler `handlers_relays.go:154-201`). Both are +captain-gated and both log the requesting DID. + +Equivalently, by hand: + +```bash +curl -s -X POST "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.requestCrawl" \ + -H 'Content-Type: application/json' \ + -d '{"hostname":"hold.example.com"}' +``` + +Restarting the hold process also works, since that re-runs the boot fan-out +(`server.go:400`), but it is the blunt version of the same action. + +Two things to understand about scope: + +1. **It is per host.** A relay's offline marking is per-host row. Crawling one hold does + nothing for any other hold, so during a fleet-wide event every hold's operator has to + act independently. There is no bulk operation across holds. +2. **It is per relay.** Each relay keeps its own host table, so a hold can be `active` on + one relay and `offline` on another. The admin crawl-all button covers every relay for + one hold; it does not cover every hold. + +After crawling, re-run the `getHostStatus` check. A successful recovery flips `status` to +`active` and the `seq` starts advancing again. + +> ### Warning: check for the replay fix before provoking a re-crawl +> +> A relay coming back resumes from its stored cursor, which after a long outage is far +> behind. That makes the hold replay history out of `firehose_events` to the reconnecting +> subscriber. Before commit `ca539b1` ("hold/pds: stop firehose backfill panicking on +> subscriber disconnect", 2026-08-08) that replay path could **take the entire hold process +> down** with `panic: send on closed channel` if the relay dropped mid-backfill: the +> backfill goroutine wrote to `sub.send` without holding the broadcaster lock while +> `Unsubscribe` closed that channel underneath it. +> +> The bug only fires when a backfill goroutine exists, which `Subscribe` skips when the +> subscriber's cursor already equals the current sequence. That is exactly why it stayed +> hidden: caught-up relays never triggered it, and a hold whose relays are all far behind +> is exposed on every single reconnect. Provoking a re-crawl on an old build is therefore +> close to the worst case for it. +> +> **Verify the deployed build contains the fix before requesting a crawl:** +> +> ```bash +> git merge-base --is-ancestor ca539b1 && echo "safe" || echo "NOT SAFE - upgrade first" +> ``` +> +> As of this writing the build deployed to seamark (`efabb677`, 2026-05-26) does **not** +> contain it. Upgrade the hold before re-crawling, or expect the process to fall over +> partway through the replay. + +### Proposed automatic fix (NOT IMPLEMENTED) + +> **Status: deferred by decision. None of the following exists in the codebase.** The +> 2026-08-20 outage was resolved by requesting crawls manually. The design below is +> recorded so that whoever revisits this does not have to rederive it. Do not read this +> section as a description of current behaviour. + +The shape of the fix: + +1. **Keep the boot `requestCrawl` exactly as it is.** It is correct for the common case of + a new or restarted hold, and it is what makes a hold discoverable at all. + +2. **Add a jittered re-crawl ticker, roughly hourly.** An hour is short relative to the + two weeks this outage ran and long relative to how quickly a relay recovers, and it + bounds worst-case invisibility to about an hour. The jitter matters: without it, every + hold that started from the same deployment fires at the same instant and the fan-out + arrives at the relays as a synchronized burst. + +3. **Guard the ticker on liveness, so it normally does nothing.** The `EventBroadcaster` + already knows whether anyone is connected: it holds the subscriber set at + `events.go:449` and the sequence number at `events.go:1062-1066`. If a relay-shaped + subscriber is connected, skip the fan-out entirely and the ticker costs nothing. Only + when no such subscriber is present does it re-announce. This makes the mechanism a + recovery path rather than a periodic broadcast, which is the whole reason it is + tolerable to run it on every hold. + + Two small pieces of plumbing this needs, neither of which exists today: the broadcaster + exposes no accessor for the subscriber count, only the unexported map; and `Subscriber` + (`events.go:42-54`) does not retain the `userAgent` that `Subscribe` receives and logs + (`events.go:440`, `:453`), so "relay-shaped" cannot currently be evaluated. Either store + the user agent, or settle for a plain non-zero subscriber count and accept that a + curious human with `websocat` attached would suppress a re-crawl. + +4. **Surface liveness somewhere visible.** This is arguably more important than the + ticker. The deeper problem was not that recovery required a manual step, it was that + two weeks passed before anyone knew there was anything to recover. Exposing the + subscriber count and `GetCurrentSeq()` on the health endpoint or the admin dashboard + would have made "zero subscribers, seq unchanged for a fortnight" visible at a glance. + Rendering the already-fetched `HostStatus` in `partials/relay_status.html` is a + near-free improvement in the same direction. + +Risks to weigh: + +- **Relay rate limiting.** An unconditional hourly fan-out from every hold to all 15 + entries in `KnownRelays` is a meaningful amount of unsolicited traffic and a plausible + way to get an IP or a domain throttled. The liveness guard is the primary mitigation, + since a healthy hold sends nothing at all, and the jitter is the secondary one. A + disconnected hold that re-crawls hourly is a rounding error; a fleet of healthy holds + all re-crawling hourly is not. +- **The replay hazard above becomes automatic.** A re-crawl ticker means reconnects with + stale cursors happen without anyone watching. This is fine on any build containing + `ca539b1`, and it is a way to crash a hold repeatedly on any build that predates it. Any + implementation of this should be gated on that fix being present, not merely assumed. +- **`getHostStatus` polling was considered and rejected as the trigger.** Asking each relay + whether it still likes us is more direct than inferring from the subscriber count, but it + is N outbound requests per interval per hold whether or not anything is wrong, which is + precisely the traffic pattern the liveness guard is trying to avoid. + ## Verification After starting your hold, verify it's working: @@ -183,6 +450,7 @@ For production with TLS termination, see [`deploy/docker-compose.prod.yml`](../d - [`config-hold.example.yaml`](../config-hold.example.yaml) — Complete configuration reference with inline comments - [BYOS.md](BYOS.md) — Bring Your Own Storage architecture and authorization model +- [KNOWN_RELAYS.md](KNOWN_RELAYS.md) — Relay list and capabilities (see also [Relay Subscription](#relay-subscription) above) - [HOLD_XRPC_ENDPOINTS.md](HOLD_XRPC_ENDPOINTS.md) — XRPC endpoint reference - [BILLING.md](BILLING.md) — Stripe billing integration - [QUOTAS.md](QUOTAS.md) — Quota management