Files
57_Wolve 12bf35caf8 anchorage v1.0 initial tree
Greenfield Go multi-tenant IPFS Pinning Service wire-compatible with the
IPFS Pinning Services API spec. Paired 1:1 with Kubo over localhost RPC,
clustered via embedded NATS JetStream, Postgres source-of-truth with
RLS-enforced tenancy, Fiber + huma v2 for the HTTP surface, Authentik
OIDC for session login with kid-rotated HS256 JWT API tokens.

Feature-complete against the 22-milestone build plan, including the
ship-it v1.0 gap items:

  * admin CLIs: drain/uncordon, maintenance, mint-token, rotate-key,
    prune-denylist, rebalance --dry-run, cache-stats, cluster-presences
  * TTL leader election via NATS KV, fence tokens, JetStream dedup
  * rebalancer (plan/apply split), reconciler, requeue sweeper
  * ristretto caches with NATS-backed cross-node invalidation
    (placements live-nodes + token denylist)
  * maintenance watchdog for stuck cluster-pause flag
  * Prometheus /metrics with CIDR ACL, HTTP/pin/scheduler/cache gauges
  * rate limiting: session (10/min) + anonymous global (120/min)
  * integration tests: rebalance, refcount multi-org, RLS belt
  * goreleaser (tar + deb/rpm/apk + Alpine Docker) targeting Gitea

Stack: Cobra/Viper, Fiber v2 + huma v2, embedded NATS JetStream,
pgx/sqlc/golang-migrate, ristretto, TypeID, prometheus/client_golang,
testcontainers-go.
2026-04-16 18:13:36 -05:00

333 lines
11 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Deploying anchorage
Two supported shapes — both use the same container image + binaries
produced by GoReleaser.
> Before either path, configure your OIDC provider: see
> [../docs/authentik-setup.md](../docs/authentik-setup.md) for the
> Authentik walkthrough. Without a valid `auth.authentik.issuer` /
> `clientID` / `audience` in anchorage.yaml the web UI login won't work
> — though `anchorage admin mint-token` and the API still do.
## Option 1 — Linux packages (deb / rpm)
GoReleaser emits `.deb` and `.rpm` artifacts with a bundled systemd
unit, lifecycle hooks, and directory structure under
`/etc/anchorage`, `/var/lib/anchorage`, `/var/log/anchorage`.
```bash
# Debian / Ubuntu
apt install ./anchorage_${VERSION}_linux_amd64.deb
# RHEL / Fedora / Alma
dnf install ./anchorage_${VERSION}_linux_amd64.rpm
```
Post-install flow:
```bash
cp /etc/anchorage/anchorage.yaml.example /etc/anchorage/anchorage.yaml
# edit anchorage.yaml — postgres DSN, authentik issuer, ipfs.rpc, …
openssl rand -base64 48 > /etc/anchorage/jwt.key
chmod 0400 /etc/anchorage/jwt.key
chown anchorage:anchorage /etc/anchorage/jwt.key
# apply schema (advisory-lock-safe on a cluster)
/usr/bin/anchorage migrate up --config /etc/anchorage/anchorage.yaml
systemctl start anchorage
systemctl status anchorage
journalctl -u anchorage -f
```
## Option 2 — Docker Swarm (three-node stack)
The stack in [docker-compose.yml](docker-compose.yml) runs three
anchorage instances, each paired 1:1 with its own Kubo daemon, against
a single Postgres. An nginx LB fronts HTTP and upgrades `/v1/events`
to WebSocket.
### Prerequisites
- Three Docker Swarm nodes (or one node if you don't care about HA —
just drop the `placement.constraints` lines).
- Each anchorage-hosting node needs `anchorage.anchor-id` as a label
and `anchorage.anchor=true`:
```bash
docker swarm init
docker node update --label-add anchorage.db=true node-1
docker node update --label-add anchorage.anchor=true node-1
docker node update --label-add anchorage.anchor=true node-2
docker node update --label-add anchorage.anchor=true node-3
docker node update --label-add anchorage.anchor-id=anchor-1 node-1
docker node update --label-add anchorage.anchor-id=anchor-2 node-2
docker node update --label-add anchorage.anchor-id=anchor-3 node-3
```
### Secrets
```bash
openssl rand -base64 32 | docker secret create anchorage_postgres_password -
openssl rand -base64 48 | docker secret create anchorage_jwt_key -
```
### Env file
```bash
cat > .env <<'EOF'
ANCHORAGE_IMAGE=git.anomalous.dev/alphacentri/anchorage:latest
ANCHORAGE_DOMAIN=anchor.example.com
ANCHORAGE_AUTHENTIK_URL=https://auth.example.com/application/o/anchorage/
POSTGRES_REPLICAS=0
EOF
```
### Deploy
```bash
docker stack deploy -c docker-compose.yml anchorage
docker stack services anchorage
docker service logs anchorage_anchorage-1
```
### Verify
```bash
curl -fsS https://anchor.example.com/v1/health
curl -fsS https://anchor.example.com/v1/ready
```
### Upgrade
```bash
# Bump the image tag in .env, then:
docker stack deploy -c docker-compose.yml anchorage
# Before a disruptive rolling restart, pause the cluster rebalancer
# so brief node absences don't trigger placement thrash:
anchorage admin maintenance on --reason "upgrade to v1.2" --ttl 30m
# …wait for the stack to converge, then:
anchorage admin maintenance off
```
Drain a single node for hardware work:
```bash
anchorage admin drain nod_anchor_2 # also visible in audit log
anchorage admin uncordon nod_anchor_2
```
## Minting a JWT for IPFS clients (`ipfs pin remote`)
Before any OIDC user exists — or when handing a long-lived token to a
CI pipeline or a headless service — use `anchorage admin mint-token`.
It reads the signing key directly off disk and emits a signed JWT
to stdout; no live anchorage process is required.
```bash
# Sysadmin break-glass token, default 395-day TTL (1 year + 30-day grace)
TOKEN=$(anchorage admin mint-token \
--signing-key /etc/anchorage/jwt.key \
--issuer https://auth.example.com/application/o/anchorage/ \
--audience anchorage)
# Hand it to the IPFS CLI:
ipfs pin remote service add anchor https://anchor.example.com/v1 "$TOKEN"
ipfs pin remote add --service=anchor --name "my-dataset" bafybeig...
```
`--issuer` and `--audience` must match the running anchorage's
`auth.authentik.*` config — when mint-token is run from the same host
as the server it reads these from `anchorage.yaml` automatically.
Shorter-lived tokens (e.g., a developer session):
```bash
anchorage admin mint-token --role member --org org_... --ttl 8h
```
Minted tokens are **standalone** — they don't appear in
`GET /v1/tokens` and can't be revoked individually. To revoke one,
either write its `jti` to the denylist via the `/v1/tokens/{jti}`
DELETE endpoint (if registered) or rotate the signing key to invalidate
every outstanding token at once.
## Rotating the JWT signing key
anchorage supports overlap-style rotation — load the new key alongside
the old, flip which one mints new tokens, then drop the retired key
once outstanding tokens have expired or been re-minted. No mass
re-auth event.
Every token carries a `kid` header naming the key that signed it.
The verifier picks the matching key from the currently-loaded set,
so "verify against either A or B" works unambiguously.
### Config shape
`auth.apiToken.signingKeys` is a list. Exactly one entry has
`primary: true` — the minting key; any additional entries are
verify-only.
Steady state:
```yaml
auth:
apiToken:
signingKeys:
- id: "2026-04"
path: /etc/anchorage/jwt.key
primary: true
```
During a rotation overlap:
```yaml
auth:
apiToken:
signingKeys:
- id: "2026-04"
path: /etc/anchorage/jwt.key
primary: true # still the minting key
- id: "2026-10"
path: /etc/anchorage/jwt.key.2026-10
# verify-only until we flip `primary` below
```
### Procedure
**Step 1 — generate the new key and stage it.**
```bash
anchorage admin rotate-signing-key --id 2026-10 --out /etc/anchorage/jwt.key.2026-10
# prints a YAML snippet to stdout — append it to auth.apiToken.signingKeys
```
Distribute the new key file to every anchorage node (Swarm secret,
k8s Secret, Ansible, whatever you already use). The file must have
identical bytes on every node.
Apply the config change adding the new entry (no `primary: true`) and
roll-restart the fleet. Every anchorage now verifies against both
keys but continues minting with the old primary.
**Step 2 — flip primary.** Edit the config so `primary: true` moves
from the old entry to the new one:
```yaml
signingKeys:
- id: "2026-04"
path: /etc/anchorage/jwt.key
- id: "2026-10"
path: /etc/anchorage/jwt.key.2026-10
primary: true
```
Roll-restart. New mints now use `kid=2026-10`. Tokens already in the
wild with `kid=2026-04` continue to verify.
**Step 3 — drop the retired key.** Wait until outstanding old-key
tokens have expired or been re-minted. `auth.apiToken.maxTTL` is the
upper bound:
- 24h default TTL + sessions only: wait 25h and you're done.
- 395-day IPFS client tokens: either wait the full window, or
mass-revoke via the denylist and ask users to re-mint. Most shops
pick the second path for security-driven rotations and the first
for scheduled ones.
Remove the old entry:
```yaml
signingKeys:
- id: "2026-10"
path: /etc/anchorage/jwt.key.2026-10
primary: true
```
Roll-restart. Any straggler token still signed with the old key is
now rejected with `token: unknown kid "2026-04"`. Delete
`/etc/anchorage/jwt.key` from every node once the restart is complete.
### When to rotate
- **Scheduled** (annual / per-security-policy) — follow the full
three-step procedure. Invisible to users whose tokens renew inside
the overlap window.
- **Suspected compromise** — do steps 1+2 immediately (seconds apart),
then mass-denylist every outstanding old-key token or skip directly
to step 3 and accept the breakage.
- **Algorithm migration** (HS256 → ed25519 / RS256) — not yet
supported; the `token` package is HS256-only today. When it lands,
the same three-step rotation pattern will apply.
## Observability: Prometheus /metrics
anchorage serves a Prometheus scrape endpoint at `/metrics` at the
root (not under `/v1`) so standard service-discovery selectors work.
Gated by a CIDR allowlist on the direct TCP peer IP. Defaults to
loopback + RFC1918, which matches the typical compose / swarm / k8s
intra-cluster scrape path without leaking /metrics through a public
LB. Tighten or disable via `server.metrics.allowCIDRs` in
anchorage.yaml.
Series exposed:
```
anchorage_http_requests_total{method,status_class}
anchorage_pin_ops_total{op,result}
anchorage_scheduler_fetch_total{node,result}
anchorage_scheduler_acks_total{node,status}
anchorage_cache_hits_total{name}
anchorage_cache_misses_total{name}
anchorage_leader_is_elected
anchorage_cluster_nodes_live
anchorage_placements_by_status{status}
```
Scrape with the standard Prometheus job config (scrape each anchorage
pod / container directly — the LB is bypassed). Alerting rules are
left to the operator; a reasonable starter set watches for
`anchorage_leader_is_elected == 0` across every node (nobody is the
leader), `rate(anchorage_pin_ops_total{result="err"}[5m])` spikes, and
`anchorage_cluster_nodes_live` falling below minReplicas.
## Rate limiting
Two layers:
- `POST /v1/auth/session` — capped per IP per minute (`server.rateLimit.sessionPerMinute`, default 10). Brute-force guard.
- All anonymous requests — capped per IP per minute (`server.rateLimit.anonymousPerMinute`, default 120). Authenticated traffic (valid Bearer or session cookie) is exempt. Probe paths (`/v1/health`, `/v1/ready`, `/metrics`) are exempt.
Storage is per-process in-memory. Sticky sessions at the LB make this
effectively global; without sticky sessions an attacker can burst
across N anchorage nodes for N× the throughput. If that matters in
your deployment, deploy behind a proxy that enforces its own global
limits (e.g., nginx `limit_req_zone`, envoy `local_ratelimit`).
## Backing up Postgres
```bash
docker exec -it $(docker ps -q -f name=anchorage_postgres) \
pg_dump -U anchorage -Fc anchorage > anchorage_$(date +%F).pgdump
```
## Backing up NATS state
NATS state under `/var/lib/anchorage/nats` is non-authoritative — it
holds in-flight jobs and the leader / cluster-maintenance KV. Losing
it trips the requeue sweeper once and comes back; Postgres is the
source of truth.
Still, if you want it captured:
```bash
docker run --rm -v anchorage_anchorage_1_data:/data \
busybox tar czf - /data/nats > nats_1_$(date +%F).tar.gz
```