# Deploying anchorage

Two supported shapes — both use the same container image + binaries
produced by GoReleaser.

> Before either path, configure your OIDC provider: see
> [../docs/authentik-setup.md](../docs/authentik-setup.md) for the
> Authentik walkthrough. Without a valid `auth.authentik.issuer` /
> `clientID` / `audience` in anchorage.yaml the web UI login won't work
> — though `anchorage admin mint-token` and the API still do.

## Option 1 — Linux packages (deb / rpm)

GoReleaser emits `.deb` and `.rpm` artifacts with a bundled systemd
unit, lifecycle hooks, and directory structure under
`/etc/anchorage`, `/var/lib/anchorage`, `/var/log/anchorage`.

```bash
# Debian / Ubuntu
apt install ./anchorage_${VERSION}_linux_amd64.deb

# RHEL / Fedora / Alma
dnf install ./anchorage_${VERSION}_linux_amd64.rpm
```

Post-install flow:

```bash
cp /etc/anchorage/anchorage.yaml.example /etc/anchorage/anchorage.yaml
# edit anchorage.yaml — postgres DSN, authentik issuer, ipfs.rpc, …

openssl rand -base64 48 > /etc/anchorage/jwt.key
chmod 0400 /etc/anchorage/jwt.key
chown anchorage:anchorage /etc/anchorage/jwt.key

# apply schema (advisory-lock-safe on a cluster)
/usr/bin/anchorage migrate up --config /etc/anchorage/anchorage.yaml

systemctl start anchorage
systemctl status anchorage
journalctl -u anchorage -f
```

## Option 2 — Docker Swarm (three-node stack)

The stack in [docker-compose.yml](docker-compose.yml) runs three
anchorage instances, each paired 1:1 with its own Kubo daemon, against
a single Postgres. An nginx LB fronts HTTP and upgrades `/v1/events`
to WebSocket.

### Prerequisites

- Three Docker Swarm nodes (or one node if you don't care about HA —
  just drop the `placement.constraints` lines).
- Each anchorage-hosting node needs `anchorage.anchor-id` as a label
  and `anchorage.anchor=true`:

```bash
docker swarm init
docker node update --label-add anchorage.db=true         node-1
docker node update --label-add anchorage.anchor=true     node-1
docker node update --label-add anchorage.anchor=true     node-2
docker node update --label-add anchorage.anchor=true     node-3
docker node update --label-add anchorage.anchor-id=anchor-1 node-1
docker node update --label-add anchorage.anchor-id=anchor-2 node-2
docker node update --label-add anchorage.anchor-id=anchor-3 node-3
```

### Secrets

```bash
openssl rand -base64 32 | docker secret create anchorage_postgres_password -
openssl rand -base64 48 | docker secret create anchorage_jwt_key -
```

### Env file

```bash
cat > .env <<'EOF'
ANCHORAGE_IMAGE=git.anomalous.dev/alphacentri/anchorage:latest
ANCHORAGE_DOMAIN=anchor.example.com
ANCHORAGE_AUTHENTIK_URL=https://auth.example.com/application/o/anchorage/
POSTGRES_REPLICAS=0
EOF
```

### Deploy

```bash
docker stack deploy -c docker-compose.yml anchorage

docker stack services anchorage
docker service logs anchorage_anchorage-1
```

### Verify

```bash
curl -fsS https://anchor.example.com/v1/health
curl -fsS https://anchor.example.com/v1/ready
```

### Upgrade

```bash
# Bump the image tag in .env, then:
docker stack deploy -c docker-compose.yml anchorage

# Before a disruptive rolling restart, pause the cluster rebalancer
# so brief node absences don't trigger placement thrash:
anchorage admin maintenance on --reason "upgrade to v1.2" --ttl 30m

# …wait for the stack to converge, then:
anchorage admin maintenance off
```

Drain a single node for hardware work:

```bash
anchorage admin drain nod_anchor_2       # also visible in audit log
anchorage admin uncordon nod_anchor_2
```

## Minting a JWT for IPFS clients (`ipfs pin remote`)

Before any OIDC user exists — or when handing a long-lived token to a
CI pipeline or a headless service — use `anchorage admin mint-token`.
It reads the signing key directly off disk and emits a signed JWT
to stdout; no live anchorage process is required.

```bash
# Sysadmin break-glass token, default 395-day TTL (1 year + 30-day grace)
TOKEN=$(anchorage admin mint-token \
    --signing-key /etc/anchorage/jwt.key \
    --issuer   https://auth.example.com/application/o/anchorage/ \
    --audience anchorage)

# Hand it to the IPFS CLI:
ipfs pin remote service add anchor https://anchor.example.com/v1 "$TOKEN"
ipfs pin remote add --service=anchor --name "my-dataset" bafybeig...
```

`--issuer` and `--audience` must match the running anchorage's
`auth.authentik.*` config — when mint-token is run from the same host
as the server it reads these from `anchorage.yaml` automatically.

Shorter-lived tokens (e.g., a developer session):

```bash
anchorage admin mint-token --role member --org org_... --ttl 8h
```

Minted tokens are **standalone** — they don't appear in
`GET /v1/tokens` and can't be revoked individually. To revoke one,
either write its `jti` to the denylist via the `/v1/tokens/{jti}`
DELETE endpoint (if registered) or rotate the signing key to invalidate
every outstanding token at once.

## Rotating the JWT signing key

anchorage supports overlap-style rotation — load the new key alongside
the old, flip which one mints new tokens, then drop the retired key
once outstanding tokens have expired or been re-minted. No mass
re-auth event.

Every token carries a `kid` header naming the key that signed it.
The verifier picks the matching key from the currently-loaded set,
so "verify against either A or B" works unambiguously.

### Config shape

`auth.apiToken.signingKeys` is a list. Exactly one entry has
`primary: true` — the minting key; any additional entries are
verify-only.

Steady state:

```yaml
auth:
  apiToken:
    signingKeys:
      - id: "2026-04"
        path: /etc/anchorage/jwt.key
        primary: true
```

During a rotation overlap:

```yaml
auth:
  apiToken:
    signingKeys:
      - id: "2026-04"
        path: /etc/anchorage/jwt.key
        primary: true        # still the minting key
      - id: "2026-10"
        path: /etc/anchorage/jwt.key.2026-10
        # verify-only until we flip `primary` below
```

### Procedure

**Step 1 — generate the new key and stage it.**

```bash
anchorage admin rotate-signing-key --id 2026-10 --out /etc/anchorage/jwt.key.2026-10
# prints a YAML snippet to stdout — append it to auth.apiToken.signingKeys
```

Distribute the new key file to every anchorage node (Swarm secret,
k8s Secret, Ansible, whatever you already use). The file must have
identical bytes on every node.

Apply the config change adding the new entry (no `primary: true`) and
roll-restart the fleet. Every anchorage now verifies against both
keys but continues minting with the old primary.

**Step 2 — flip primary.** Edit the config so `primary: true` moves
from the old entry to the new one:

```yaml
signingKeys:
  - id: "2026-04"
    path: /etc/anchorage/jwt.key
  - id: "2026-10"
    path: /etc/anchorage/jwt.key.2026-10
    primary: true
```

Roll-restart. New mints now use `kid=2026-10`. Tokens already in the
wild with `kid=2026-04` continue to verify.

**Step 3 — drop the retired key.** Wait until outstanding old-key
tokens have expired or been re-minted. `auth.apiToken.maxTTL` is the
upper bound:

- 24h default TTL + sessions only: wait 25h and you're done.
- 395-day IPFS client tokens: either wait the full window, or
  mass-revoke via the denylist and ask users to re-mint. Most shops
  pick the second path for security-driven rotations and the first
  for scheduled ones.

Remove the old entry:

```yaml
signingKeys:
  - id: "2026-10"
    path: /etc/anchorage/jwt.key.2026-10
    primary: true
```

Roll-restart. Any straggler token still signed with the old key is
now rejected with `token: unknown kid "2026-04"`. Delete
`/etc/anchorage/jwt.key` from every node once the restart is complete.

### When to rotate

- **Scheduled** (annual / per-security-policy) — follow the full
  three-step procedure. Invisible to users whose tokens renew inside
  the overlap window.
- **Suspected compromise** — do steps 1+2 immediately (seconds apart),
  then mass-denylist every outstanding old-key token or skip directly
  to step 3 and accept the breakage.
- **Algorithm migration** (HS256 → ed25519 / RS256) — not yet
  supported; the `token` package is HS256-only today. When it lands,
  the same three-step rotation pattern will apply.

## Observability: Prometheus /metrics

anchorage serves a Prometheus scrape endpoint at `/metrics` at the
root (not under `/v1`) so standard service-discovery selectors work.

Gated by a CIDR allowlist on the direct TCP peer IP. Defaults to
loopback + RFC1918, which matches the typical compose / swarm / k8s
intra-cluster scrape path without leaking /metrics through a public
LB. Tighten or disable via `server.metrics.allowCIDRs` in
anchorage.yaml.

Series exposed:

```
anchorage_http_requests_total{method,status_class}
anchorage_pin_ops_total{op,result}
anchorage_scheduler_fetch_total{node,result}
anchorage_scheduler_acks_total{node,status}
anchorage_cache_hits_total{name}
anchorage_cache_misses_total{name}
anchorage_leader_is_elected
anchorage_cluster_nodes_live
anchorage_placements_by_status{status}
```

Scrape with the standard Prometheus job config (scrape each anchorage
pod / container directly — the LB is bypassed). Alerting rules are
left to the operator; a reasonable starter set watches for
`anchorage_leader_is_elected == 0` across every node (nobody is the
leader), `rate(anchorage_pin_ops_total{result="err"}[5m])` spikes, and
`anchorage_cluster_nodes_live` falling below minReplicas.

## Rate limiting

Two layers:

- `POST /v1/auth/session` — capped per IP per minute (`server.rateLimit.sessionPerMinute`, default 10). Brute-force guard.
- All anonymous requests — capped per IP per minute (`server.rateLimit.anonymousPerMinute`, default 120). Authenticated traffic (valid Bearer or session cookie) is exempt. Probe paths (`/v1/health`, `/v1/ready`, `/metrics`) are exempt.

Storage is per-process in-memory. Sticky sessions at the LB make this
effectively global; without sticky sessions an attacker can burst
across N anchorage nodes for N× the throughput. If that matters in
your deployment, deploy behind a proxy that enforces its own global
limits (e.g., nginx `limit_req_zone`, envoy `local_ratelimit`).

## Backing up Postgres

```bash
docker exec -it $(docker ps -q -f name=anchorage_postgres) \
    pg_dump -U anchorage -Fc anchorage > anchorage_$(date +%F).pgdump
```

## Backing up NATS state

NATS state under `/var/lib/anchorage/nats` is non-authoritative — it
holds in-flight jobs and the leader / cluster-maintenance KV. Losing
it trips the requeue sweeper once and comes back; Postgres is the
source of truth.

Still, if you want it captured:

```bash
docker run --rm -v anchorage_anchorage_1_data:/data \
    busybox tar czf - /data/nats > nats_1_$(date +%F).tar.gz
```