automations/deployments/ergo/README.md

# ergo

[Ergo](https://ergo.chat) — a modern IRC server with built-in services
(NickServ/ChanServ/HostServ), message history, always-on "bouncer" clients and
IRCv3 — behind Caddy for automatic Let's Encrypt TLS. Ships a host CLI
(**`ergoctl`**) for day-to-day administration and a health-checked updater
(**`update.sh`**) that also keeps Ergo's TLS certificate in sync with Caddy.

## Docker, not an OpenRC/systemd service

Ergo is a single static Go binary, so a native install is perfectly possible.
This stack still uses Docker Compose because it matches the repo's goals better:

| | Docker (this stack) | Native OpenRC / systemd |
|---|---|---|
| Alpine, Debian, Alma | one compose file, one image | Alpine packages `ergo` in community, but a stable branch keeps the version it shipped with (3.24 → 2.18.0, 3.23/3.22 → 2.16.0) while upstream is at 2.19.1, a *security* release; only edge tracks upstream. Debian/Alma have no package at all → hand-rolled tarball install + a unit per init system |
| Upgrade / rollback | `ERGO_TAG` in `.env`, health-checked, DB snapshot, rollback | replace a binary + hand-restore the DB |
| ACME TLS | stock Caddy (already used by every other stack) | certbot/acme.sh per distro + deploy hook |
| Real client IPs | yes — **host networking** (see below) | yes |

The usual objection to containerised IRC — bans, throttling and IP cloaking
need the client's real address, and Docker's userland proxy hides IPv6 clients
behind the bridge gateway — is solved by running **both** containers with
`network_mode: host`. That also means the host's deny-by-default firewall
genuinely governs the ports (the "Docker bypasses INPUT" caveat from the root
README does not apply here).

## Topology

| Port | Who listens | Purpose |
|---|---|---|
| `6697/tcp` | Ergo | IRC over TLS (Let's Encrypt cert, copied from Caddy) |
| `443/tcp` | Caddy | `wss://DOMAIN/webirc` → Ergo's websocket listener on `127.0.0.1:8097`; text landing page at `/` |
| `80/tcp` | Caddy | ACME HTTP-01 + redirect to HTTPS |
| `127.0.0.1:6667` | Ergo | loopback plaintext: compose healthcheck + `ergoctl` (exempt from ip-limits/bans, treated as secure) |
| `6667/tcp` public | Ergo | only with `PLAINTEXT=1` (STS is then advertised) — **not recommended** |

No HTTP/3 (would bind udp/443), Caddy's admin API is **off** (it would sit on
the host's loopback), and there is no `ports:` section — host mode.

## Required `.env` values

| Variable | Notes |
|---|---|
| `ERGO_DOMAIN` | Public hostname (lower-case). Server name, cert subject, Caddy site and websocket origin. **Immutable** after the first deploy. |
| `ACME_EMAIL` | Let's Encrypt registration email. |
| `NETWORK_NAME` | IRC network name (letters, digits, `. _ -`; no spaces). Defaults to `ERGO_DOMAIN`. |
| `ERGO_TAG` | Pinned image tag **with the `v`** (`v2.19.1`). Set by deploy.sh to the newest release; bumped by update.sh. Never `stable`/`latest` — if a first deploy could not reach GitHub it falls back to `stable`, and the next update run pins it. |
| `PLAINTEXT` | `0` (default) or `1` — public plaintext 6667. **Fixed at the first deploy** (it selects a listener in `ircd.yaml`, which deploy.sh never rewrites); see below to change it. |
| `UPDATE_POLICY` | `latest` (default) / `security` / `off` — see [Updates](#updates). |

See [`.env.example`](.env.example) for the rest (`UPDATE_GRACE`,
`FORCE_UPDATE`, `CADDY_AUTOUPDATE`, `CADDY_TAG`). `.env` must not contain `$`
(Compose interpolates it) — secrets never live there. The four update knobs are
read from `.env` by the scheduled jobs; an environment variable, or a value
pinned in `/etc/ergo-update.conf`, overrides it for that run.

Deploy-time-only options (not stored in `.env`): `ERGO_AUTOUPDATE=0` installs
`update.sh` without scheduling the daily update (cert sync is still scheduled),
and `CERT_WAIT` (default 180s) bounds the wait for the first Let's Encrypt cert.

To switch `PLAINTEXT` after the first deploy: change the plaintext listener with
`ergoctl edit` (`"127.0.0.1:6667":` ↔ `":6667":`, and `server.sts.enabled` with
it), set `PLAINTEXT` in `.env`, run `ergoctl restart`, then re-run `deploy.sh`
so the host firewall matches. Passing a conflicting `PLAINTEXT=` to a re-run
stops with those instructions rather than half-applying the change.

## Deploy

```bash
./automations.sh        # Deploy on this host → deploy: ergo
```

Or build + run the self-contained artifact:

```bash
./build.sh
scp deploy.sh root@host:
ssh root@host 'bash deploy.sh'
# non-interactive:
#   ERGO_DOMAIN=irc.example.com ACME_EMAIL=me@example.com NETWORK_NAME=MyNet SKIP_PROMPTS=1 bash deploy.sh
```

Unattended provisioning: [`cloud-init.yml`](cloud-init.yml). On first run
deploy.sh:

1. installs Docker (+ `openssl`, `jq`, `curl`) and creates the `ergo` system
   user the container runs as;
2. registers `80,443,6697/tcp` with the host firewall
   (`/etc/firewall/ports.d/ergo.rule`, or ufw/firewalld);
3. pins `ERGO_TAG` to the newest release and seeds `.env`;
4. generates **`ircd/ircd.yaml` once**, from the pulled image's *own*
   `default.yaml` (so it always matches the running version): server/network
   name, the listeners above, the websocket origin, the cloak suffix, and an
   `admin` oper whose random password is saved to `secrets/admin.pass` (0600;
   only the bcrypt hash goes into the yaml). Hard post-checks refuse to
   continue if the upstream template layout ever changes under the edits;
5. validates the config with `ergo run --smoke` in a throwaway container
   **before** anything starts, validates the Caddyfile, then starts the stack
   and waits for both healthchecks (Ergo's is IRC-level: a `QUIT` must be
   answered with `ERROR`);
6. waits for Caddy's Let's Encrypt cert, copies it into `ircd/` and rehashes
   Ergo (until then Ergo serves a self-signed cert from its own `mkcerts`);
7. schedules `update.sh certsync` every 15 min and `update.sh run` daily
   (`ERGO_AUTOUPDATE=0` schedules only the cert sync).

Re-runs are idempotent: they never touch `ircd/ircd.yaml`, `secrets/` or
`caddy/etc/conf.d/`, and they adopt the deployed settings from `.env` rather
than this shell's defaults, so a plain `bash deploy.sh` cannot silently close a
`PLAINTEXT=1` firewall port or reset your update policy. A setting you do pass
explicitly is written back to `.env`. A changed `Caddyfile` triggers a Caddy
restart, since Caddy has no admin API here.

## Administer: `ergoctl`

Installed at `/usr/local/bin/ergoctl`. It talks to Ergo over the loopback
listener as the `admin` oper.

| Command | What it does |
|---|---|
| `ergoctl status` / `users` / `logs [-f]` / `version` | health, versions, TLS state, user count; server log with health-probe noise filtered |
| `ergoctl edit` / `motd` | `$EDITOR` on `ircd.yaml` / `ergo.motd`, then **validate** (throwaway container) and **REHASH**; the previous file is restored if either step fails |
| `ergoctl rehash` | reload config, MOTD and TLS certs without disconnecting anyone |
| `ergoctl restart` / `stop` / `start` | restart the container (drops every user; needed for settings Ergo fixes at startup, such as `server.name`, `datastore.path`, `casemapping`, `enforce-utf8`, `max-line-len`, `idle-timeouts`). Changes to `docker-compose.yml` or `.env` need `docker compose up -d` instead |
| `ergoctl oper list\|add\|passwd\|rm` | manage operators in `ircd.yaml`; passwords are generated (24 chars) and printed once |
| `ergoctl passwd [show\|rotate]` | the `admin` oper password |
| `ergoctl announce <text>` | NOTICE to everyone |
| `ergoctl kill <nick> [reason]` · `ban add\|del\|list\|info` · `defcon [1-5]` | moderation (`UBAN` under the hood). `ban add` takes an **IP, CIDR, `nick!user@host` mask, or account name** — a bare name is an *account* to suspend, not a connected nick, so run `ban info <nick>` first to get their IP |
| `ergoctl cmd <raw IRC line>` | anything else as the admin oper, e.g. `ergoctl cmd NS SAREGISTER alice hunter2`, `ergoctl cmd CS PURGE #spam`; replies are printed with control characters stripped |
| `ergoctl cert [show\|sync]` | cert on disk vs the one served on 6697; force a sync |
| `ergoctl backup [--live] [dir]` / `restore <file>` | see [Backups](#backups) |
| `ergoctl update …` | passthrough to `update.sh` |
| `ergoctl debug on\|off` · `caddy-restart` · `shell` | log level; apply Caddyfile/conf.d changes; a shell in the container |

Editing by hand works too: change `ircd/ircd.yaml`, then `ergoctl rehash` — a
rejected config leaves the old one running (Ergo's rehash is transactional), but
the *file* on disk would then be unbootable, which is why `ergoctl edit`
validates first. Ergo's [manual](https://github.com/ergochat/ergo/blob/master/docs/MANUAL.md)
covers every option; the pristine template for your version is in `templates/`.

## TLS

Caddy owns the ACME account and renews the cert on its own schedule (ARI-driven).
Ergo terminates TLS itself on 6697, so the cert has to reach `ircd/fullchain.pem`
+ `privkey.pem`. `update.sh certsync` (every 15 minutes, via busybox crond or a
systemd timer) does that the way upstream's certbot deploy-hook recipe does,
plus safety checks:

- picks the newest `<domain>.crt` under `caddy/data/caddy/certificates/*/` (Let's
  Encrypt or the ZeroSSL fallback), skipping a renewal that is still mid-write;
- verifies the key matches the cert (certmagic writes them as separate files and
  generates a new key on every renewal), stages both in `ircd/` and installs
  them with two atomic `mv`s — Ergo never sees a half pair, which would be
  fatal at its next start;
- `SIGHUP`s Ergo (rehash swaps the TLS config for new connections only) and
  confirms the fingerprint served on 6697 changed; if not, the previous pair is
  restored and you get an ntfy alert.

Silent when nothing changed. The daily run also warns when the served cert is
still self-signed (DNS/80/443 not right yet) or expires within 14 days
(certsync not running). `ergoctl cert` shows both sides.

## Updates

```bash
ergoctl update check      # running vs latest, published security advisories, cert state; changes nothing
ergoctl update update     # update now (TARGET_VERSION=2.19.1 to pin); honours the safety rails
ergoctl update install    # (re)schedule  /  uninstall to stop
```

`UPDATE_POLICY` (in `.env`) drives the daily `run`:

| Policy | Behaviour |
|---|---|
| `latest` *(default)* | Update to the newest release whenever one exists. |
| `security` | Update **only** when a published GitHub security advisory covers the running version, or a release since it has a `### Security` section in its notes (upstream only opened GHSAs from 2.19.1). |
| `off` | Never change the running version (check/notify only). |

Every update, scheduled or manual:

1. pulls the pinned `ghcr.io/ergochat/ergo:vX.Y.Z` and **pre-flights** it against
   a copy of `ircd/` (`ergo run --smoke` — also dry-runs a DB schema upgrade);
2. `NOTICE`s connected users and waits `UPDATE_GRACE` seconds (default 60);
3. stops Ergo and snapshots `ircd.db` to `backups/` (consistent copy; buntdb is
   append-only, so a live copy could miss the last second), keeps the last 5;
4. flips `ERGO_TAG`, starts, waits for the healthcheck **and** a registration
   handshake;
5. on failure: previous tag back; if the old version refuses the upgraded
   database ("Database requires update") the snapshot is restored too. Cause
   and snapshot path go into the ntfy message;
6. on success: saves the new image's `default.yaml` under `templates/` and, if it
   differs from the previous version's, writes `templates/diff-vA-vB.txt` so
   new/renamed options are visible.

Releases whose notes announce **compatibility breaks** are held (one ntfy, not
daily) until you run `ergoctl update update` or set `FORCE_UPDATE=1`. Restarting
Ergo disconnects every user (there is no hot restart), so pick `security` if
that matters more than being current. Caddy is only updated by
`ergoctl update caddy` (or daily with `CADDY_AUTOUPDATE=1`); recreating it drops
web-client websockets. Update results reuse the ntfy config at
`/etc/ssh-notify.conf` when present.

## Backups

`ergoctl backup` writes `backups/ergo-backup-<ts>.tar.gz` (0600) with
`ircd.yaml`, `ircd.db`, `ergo.motd` and a `meta` file (version, date). By default
it stops Ergo for a few seconds for a consistent copy (`--live` skips that).
Put an age public key in `$STACK_DIR/age-recipients.txt` (e.g. from
`globals/age-pubkey.txt`) **and install `age`** (`apk add age`, `apt install age`,
`dnf install age` — deploy.sh does not) and backups are encrypted with it; with
the recipients file present but `age` missing, `ergoctl backup` refuses rather
than writing the account and oper hashes out in the clear. TLS files are not
included — certsync regenerates them. `ergoctl restore <file>` stops Ergo,
warns if the backup came from a newer version (older Ergo cannot read a newer
schema), restores, and waits for health. Ergo's own `datastore.autoupgrade`
leaves `ircd/ircd.db.v<N>.<ts>.bak` files behind on schema upgrades; prune them
once you are happy with a release.

## Web client (optional)

The `wss://DOMAIN/webirc` endpoint is ready; Ergo only accepts browser
connections whose `Origin` is `https://DOMAIN` (`server.websockets.allowed-origins`).
To host [Gamja](https://codeberg.org/emersion/gamja) on the same origin, unpack a
release tarball into `caddy/www/gamja`, add
`{"server": {"url": "/webirc", "autojoin": "#lobby"}}` as `config.json`, mount
`./caddy/www:/www:ro` in the caddy service and drop a `handle { root * /www/gamja
file_server }` block into `caddy/etc/conf.d/` (see `00-readme.caddy`), then
`ergoctl caddy-restart`.

## Security model

- **Host loopback is Ergo's trust boundary.** Loopback peers are exempt from
  bans and ip-limits, are treated as secure, and (because
  `proxy-allowed-from: [localhost]` is what lets Caddy's `X-Forwarded-For`
  through) may assert a client IP. Every local process and every other
  host-network container can reach `127.0.0.1:6667`/`:8097`. This stack is for
  a single-purpose container host without untrusted local users; deploy.sh
  warns about other host-network containers.
- **Ergo runs unprivileged**: as the host's `ergo` system user, read-only root
  filesystem, all capabilities dropped, `no-new-privileges`. `ircd/` (database,
  config with the oper hash, TLS key) is `0700 ergo:ergo`; `secrets/`,
  `backups/`, `caddy/data` are `0700 root`.
- **Caddy runs as root** (it needs `NET_BIND_SERVICE` for 80/443) with every
  other capability dropped, admin API off, HTTP/3 off.
- **Loopback plaintext is the control plane by design**: OPER over loopback is
  not a downgrade (Ergo marks it secure), and it lets `ergoctl` work even when a
  ban or throttle would lock out a remote client. `PLAINTEXT=1` puts real users'
  credentials on the wire in clear — leave it at `0`.
- The admin password is sent to Ergo only over loopback and stored only in
  `secrets/admin.pass`. `ergo genpasswd` hashes at bcrypt cost 4, so oper
  passwords are always generated (24 random chars), never typed.

## Files

| File | Purpose |
|---|---|
| `docker-compose.yml` | caddy + ergo, both `network_mode: host`, hardened; IRC-level healthcheck. |
| `Caddyfile` | ACME, `/webirc` websocket proxy, text landing page, `admin off`, no h3; imports `conf.d/*.caddy`. |
| `conf.d-readme.caddy` | Installed once as `caddy/etc/conf.d/00-readme.caddy` (operator drop-ins; Gamja example). |
| `ergo.motd` | MOTD template (`__NETWORK_NAME__`, `__DOMAIN__`). |
| `ergolib.sh` | Shared helpers: `.env`, compose, health/IRC probes, `--smoke` validator, oper edits, certsync, ntfy. |
| `update.sh` | Updater + certsync + Caddy update + scheduling. |
| `ergoctl` | Host admin CLI. |
| `.env.example` | Stack tunables. |
| `deploy.sh` / `build.sh` | Self-contained installer + archive embedder. |
| `cloud-init.yml` | Fresh-VM bootstrap (harden SSH, then deploy). |

On the host: `/srv/ergo/{ircd,caddy/{etc,data,config},secrets,backups,templates}`
plus `.state/` (notification de-duplication) and `.env.bak.<ts>` copies from each
update (last 5 kept); `/etc/ergo-update.conf` (where the jobs find the stack);
`/usr/local/bin/ergoctl`; `/var/log/ergo-update.log` (scheduled runs log there,
since busybox crond has nowhere else to put their output).

## Notes

- Only one host-networked Caddy fits on a box: don't co-locate this stack with
  another Caddy/80/443 stack (deploy.sh checks the ports).
- DNS for `ERGO_DOMAIN` must resolve to the host and 80/443 be reachable before
  deploy for the cert to issue; Ergo stays up on a self-signed cert meanwhile.
- Ergo's built-in registration is open by default (`accounts.registration.enabled`);
  edit `ircd.yaml` if you want an invite-only network.