automations/deployments/ergo/README.md

# ergo

[Ergo](https://ergo.chat) — a modern IRC server with built-in services
(NickServ/ChanServ/HostServ), message history, always-on "bouncer" clients and
IRCv3 — behind Caddy for automatic Let's Encrypt TLS. Ships a host CLI
(**`ergoctl`**) for day-to-day administration and a health-checked updater
(**`update.sh`**) that also keeps Ergo's TLS certificate in sync with Caddy.

## Docker, not an OpenRC/systemd service

Ergo is a single static Go binary, so a native install is perfectly possible.
This stack still uses Docker Compose because it matches the repo's goals better:

| | Docker (this stack) | Native OpenRC / systemd |
|---|---|---|
| Alpine, Debian, Alma | one compose file, one image | Alpine packages `ergo` in community, but a stable branch keeps the version it shipped with (3.24 → 2.18.0, 3.23/3.22 → 2.16.0) while upstream is at 2.19.1, a *security* release; only edge tracks upstream. Debian/Alma have no package at all → hand-rolled tarball install + a unit per init system |
| Upgrade / rollback | `ERGO_TAG` in `.env`, health-checked, DB snapshot, rollback | replace a binary + hand-restore the DB |
| ACME TLS | stock Caddy (already used by every other stack) | certbot/acme.sh per distro + deploy hook |
| Real client IPs | yes — **host networking** (see below) | yes |

The usual objection to containerised IRC — bans, throttling and IP cloaking
need the client's real address, and Docker's userland proxy hides IPv6 clients
behind the bridge gateway — is solved by running **both** containers with
`network_mode: host`. That also means the host's deny-by-default firewall
genuinely governs the ports (the "Docker bypasses INPUT" caveat from the root
README does not apply here).

## Topology

| Port | Who listens | Purpose |
|---|---|---|
| `6697/tcp` | Ergo | IRC over TLS (Let's Encrypt cert, copied from Caddy) |
| `443/tcp` | Caddy | `wss://DOMAIN/webirc` → Ergo's websocket listener on `127.0.0.1:8097`; text landing page at `/` |
| `80/tcp` | Caddy | ACME HTTP-01 + redirect to HTTPS |
| `127.0.0.1:6667` | Ergo | loopback plaintext: compose healthcheck + `ergoctl` (exempt from ip-limits/bans, treated as secure) |
| `6667/tcp` public | Ergo | only with `PLAINTEXT=1` (STS is then advertised) — **not recommended** |

No HTTP/3 (would bind udp/443), Caddy's admin API is **off** (it would sit on
the host's loopback), and there is no `ports:` section — host mode.

## Required `.env` values

| Variable | Notes |
|---|---|
| `ERGO_DOMAIN` | Public hostname (lower-case). Server name, cert subject, Caddy site and websocket origin. **Immutable** after the first deploy. |
| `ACME_EMAIL` | Let's Encrypt registration email. |
| `NETWORK_NAME` | IRC network name (letters, digits, `. _ -`; no spaces). Defaults to `ERGO_DOMAIN`. |
| `ERGO_TAG` | Pinned image tag **with the `v`** (`v2.19.1`). Set by deploy.sh to the newest release; bumped by update.sh. Never `stable`/`latest` — if a first deploy could not reach GitHub it falls back to `stable`, and the next update run pins it. |
| `PLAINTEXT` | `0` (default) or `1` — public plaintext 6667. **Fixed at the first deploy** (it selects a listener in `ircd.yaml`, which deploy.sh never rewrites); see below to change it. |
| `HISTORY` | `sqlite` (default) / `postgres` / `off` — see [Message history](#message-history). |
| `UPDATE_POLICY` | `latest` (default) / `security` / `off` — see [Updates](#updates). |

See [`.env.example`](.env.example) for the rest (`UPDATE_GRACE`,
`FORCE_UPDATE`, `CADDY_AUTOUPDATE`, `CADDY_TAG`). `.env` must not contain `$`
(Compose interpolates it) — secrets never live there. The four update knobs are
read from `.env` by the scheduled jobs; an environment variable, or a value
pinned in `/etc/ergo-update.conf`, overrides it for that run.

Deploy-time-only options (not stored in `.env`): `ERGO_AUTOUPDATE=0` installs
`update.sh` without scheduling the daily update (cert sync is still scheduled),
and `CERT_WAIT` (default 180s) bounds the wait for the first Let's Encrypt cert.

To switch `PLAINTEXT` after the first deploy: change the plaintext listener with
`ergoctl edit` (`"127.0.0.1:6667":` ↔ `":6667":`, and `server.sts.enabled` with
it), set `PLAINTEXT` in `.env`, run `ergoctl restart`, then re-run `deploy.sh`
so the host firewall matches. Passing a conflicting `PLAINTEXT=` to a re-run
stops with those instructions rather than half-applying the change.

## Deploy

```bash
./automations.sh        # Deploy on this host → deploy: ergo
```

Or build + run the self-contained artifact:

```bash
./build.sh
scp deploy.sh root@host:
ssh root@host 'bash deploy.sh'
# non-interactive:
#   ERGO_DOMAIN=irc.example.com ACME_EMAIL=me@example.com NETWORK_NAME=MyNet SKIP_PROMPTS=1 bash deploy.sh
```

Unattended provisioning: [`cloud-init.yml`](cloud-init.yml). On first run
deploy.sh:

1. installs Docker (+ `openssl`, `jq`, `curl`) and creates the `ergo` system
   user the container runs as;
2. registers `80,443,6697/tcp` with the host firewall
   (`/etc/firewall/ports.d/ergo.rule`, or ufw/firewalld);
3. pins `ERGO_TAG` to the newest release and seeds `.env`;
4. generates **`ircd/ircd.yaml` once**, from the pulled image's *own*
   `default.yaml` (so it always matches the running version): server/network
   name, the listeners above, the websocket origin, the cloak suffix, and an
   `admin` oper whose random password is saved to `secrets/admin.pass` (0600;
   only the bcrypt hash goes into the yaml). Hard post-checks refuse to
   continue if the upstream template layout ever changes under the edits;
5. validates the config with `ergo run --smoke` in a throwaway container
   **before** anything starts, validates the Caddyfile, then starts the stack
   and waits for both healthchecks (Ergo's is IRC-level: a `QUIT` must be
   answered with `ERROR`);
6. waits for Caddy's Let's Encrypt cert, copies it into `ircd/` and rehashes
   Ergo (until then Ergo serves a self-signed cert from its own `mkcerts`);
7. schedules `update.sh certsync` every 15 min and `update.sh run` daily
   (`ERGO_AUTOUPDATE=0` schedules only the cert sync).

Re-runs are idempotent: they never touch `ircd/ircd.yaml`, `secrets/` or
`caddy/etc/conf.d/`, and they adopt the deployed settings from `.env` rather
than this shell's defaults, so a plain `bash deploy.sh` cannot silently close a
`PLAINTEXT=1` firewall port or reset your update policy. A setting you do pass
explicitly is written back to `.env`. A changed `Caddyfile` triggers a Caddy
restart, since Caddy has no admin API here.

## Administer: `ergoctl`

Installed at `/usr/local/bin/ergoctl`. It talks to Ergo over the loopback
listener as the `admin` oper.

| Command | What it does |
|---|---|
| `ergoctl status` / `users` / `logs [-f]` / `version` | health, versions, TLS state, user count; server log with health-probe noise filtered |
| `ergoctl edit` / `motd` | `$EDITOR` on `ircd.yaml` / `ergo.motd`, then **validate** (throwaway container) and **REHASH**; the previous file is restored if either step fails |
| `ergoctl rehash` | reload config, MOTD and TLS certs without disconnecting anyone |
| `ergoctl restart` / `stop` / `start` | restart the container (drops every user; needed for settings Ergo fixes at startup, such as `server.name`, `datastore.path`, `casemapping`, `enforce-utf8`, `max-line-len`, `idle-timeouts`). Changes to `docker-compose.yml` or `.env` need `docker compose up -d` instead |
| `ergoctl oper list\|add\|passwd\|certfp\|rm` | manage operators in `ircd.yaml`; passwords are generated (24 chars) and printed once. See [Operators](#operators) |
| `ergoctl passwd [show\|rotate]` | the `admin` oper password |
| `ergoctl announce <text>` | NOTICE to everyone |
| `ergoctl kill <nick> [reason]` · `ban add\|del\|list\|info` · `defcon [1-5]` | moderation (`UBAN` under the hood). `ban add` takes an **IP, CIDR, `nick!user@host` mask, or account name** — a bare name is an *account* to suspend, not a connected nick, so run `ban info <nick>` first to get their IP |
| `ergoctl cmd <raw IRC line>` | anything else as the admin oper, e.g. `ergoctl cmd NS SAREGISTER alice hunter2`, `ergoctl cmd CS PURGE ADD #spam` (`CS PURGE DEL` to undo); replies are printed with control characters stripped |
| `ergoctl cert [show\|sync]` | cert on disk vs the one served on 6697; force a sync |
| `ergoctl backup [--live] [dir]` / `restore <file>` | see [Backups](#backups) |
| `ergoctl update …` | passthrough to `update.sh` |
| `ergoctl history [backend]` | show or switch the persistent-history backend |
| `ergoctl debug on\|off` · `caddy-restart` · `shell` | log level; apply Caddyfile/conf.d changes; a shell in the container |

Editing by hand works too: change `ircd/ircd.yaml`, then `ergoctl rehash` — a
rejected config leaves the old one running (Ergo's rehash is transactional), but
the *file* on disk would then be unbootable, which is why `ergoctl edit`
validates first. Ergo's [manual](https://github.com/ergochat/ergo/blob/master/docs/MANUAL.md)
covers every option; the pristine template for your version is in `templates/`.

## TLS

Caddy owns the ACME account and renews the cert on its own schedule (ARI-driven).
Ergo terminates TLS itself on 6697, so the cert has to reach `ircd/fullchain.pem`
+ `privkey.pem`. `update.sh certsync` (every 15 minutes, via busybox crond or a
systemd timer) does that the way upstream's certbot deploy-hook recipe does,
plus safety checks:

- picks the newest `<domain>.crt` under `caddy/data/caddy/certificates/*/` (Let's
  Encrypt or the ZeroSSL fallback), skipping a renewal that is still mid-write;
- verifies the key matches the cert (certmagic writes them as separate files and
  generates a new key on every renewal), stages both in `ircd/` and installs
  them with two atomic `mv`s — Ergo never sees a half pair, which would be
  fatal at its next start;
- `SIGHUP`s Ergo (rehash swaps the TLS config for new connections only) and
  confirms the fingerprint served on 6697 changed; if not, the previous pair is
  restored and you get an ntfy alert.

Silent when nothing changed. The daily run also warns when the served cert is
still self-signed (DNS/80/443 not right yet) or expires within 14 days
(certsync not running). `ergoctl cert` shows both sides.

## Operators

An oper name is a **credential, not a nick** — you authenticate as one and keep
your own nickname. Deploy creates `admin` (class `server-admin`, `hidden: true`),
whose password is in `secrets/admin.pass` (`ergoctl passwd show`):

```
/OPER admin <password>
```

Add your own rather than sharing that one. Ergo casefolds oper names, so use
lowercase letters, digits, `_` and `-`, starting with a letter:

```bash
ergoctl oper add alice server-admin
```

The class is `chat-moderator` by default; `server-admin` adds rehash, account and
channel administration, defcon and massmessage.

### Client-certificate auth

Instead of typing a password every session, an oper can be identified by the
SHA-256 fingerprint of their TLS client certificate. Find it on the `276` line of
your own `/WHOIS`, or from the certificate:

```bash
openssl x509 -noout -fingerprint -sha256 -in client.pem
```

Then pass that fingerprint, or let ergoctl read it off a connected user:

```bash
ergoctl oper certfp alice 57_Wolve --auto
```

Without `--auto`, Ergo requires **both** the certificate and the password, which
makes the certificate a second factor. With `--auto` the password is removed and
the certificate alone grants operator status the moment they connect — nothing
typed, nothing replayable, but anyone holding that key is an operator on sight.
`ergoctl oper certfp alice --clear` revokes it.

**This is refused for `admin`, deliberately.** `ergoctl` opers up over the
loopback plaintext listener, which presents no client certificate, so a certfp
there would make Ergo reject it — and every ergoctl command needing oper,
including the scheduled jobs, would stop working. Keep `admin` on its password
and put certificates on personal opers.

## Message history

Ergo keeps channel and DM history for `CHATHISTORY`, `/HISTORY`, autoreplay-on-join
and always-on (bouncer) clients. **Upstream keeps it in RAM**, so it is lost on
every restart — including the ones this stack's updater performs. So persistence
is on by default here:

| `HISTORY` | What it does |
|---|---|
| `sqlite` *(default)* | A file at `ircd/ergo_history.db`, next to the account database. No extra container, backed up with everything else. Right for a single-node server. |
| `postgres` | A pinned PostgreSQL container (`docker-compose.postgres.yml`), reachable only on `127.0.0.1`. Choose it if you want a real database to query, or already run Postgres. |
| `off` | Upstream behaviour: RAM only. |

Both SQL backends need **Ergo 2.18.0+** (deploy.sh refuses older pins). How long
messages are kept is `history.restrictions.expire-time` in `ircd.yaml` — **one
week by default** — not a size limit: enabling persistence removes the in-memory
`channel-length`/`client-length` caps. Raise or lower it with `ergoctl edit`, and
note the privacy point upstream flags: persisted messages are personal data, so
check what your jurisdiction expects of you before extending retention.

```bash
ergoctl history                 # backend, database size / connectivity, retention
ergoctl history postgres        # switch (restarts Ergo; does NOT migrate messages)
```

Switching backends leaves the old store on disk and starts the new one empty —
there is no migration path, so pick one at deploy time if you can.

### PostgreSQL

`HISTORY=postgres` adds `docker-compose.postgres.yml` to `COMPOSE_FILE` in `.env`,
which `docker compose` reads by itself — so every command in this stack sees the
same services with no extra flags. The container publishes **only to
`127.0.0.1:5432`**, Ergo reaches it there (the host-networked containers cannot
use compose service DNS), and `depends_on: service_healthy` keeps Ergo from
starting before the database accepts connections. The password is generated into
`secrets/postgres.pass` (0600) and passed via `POSTGRES_PASSWORD_FILE`, so it
never lands in `.env`.

**The major version is pinned and the updater never touches it.** PostgreSQL
refuses to start on a data directory written by a different major version, so
moving from `17-alpine` to `18-alpine` is a deliberate dump-and-restore:

```bash
ergoctl backup                      # contains a pg_dump of the history database
ergoctl history off && docker compose down -v postgres   # drops the old data volume
# set POSTGRES_TAG=18-alpine in .env, then:
ergoctl history postgres && ergoctl restore backups/ergo-backup-<ts>.tar.gz
```

Note that Postgres 18 also changed the image's default data directory layout; the
compose file pins `PGDATA` explicitly so a future image default cannot move it.

## Updates

```bash
ergoctl update check      # running vs latest, published security advisories, cert state
                          # reports only, with one exception: a floating ERGO_TAG
                          # ('stable') is pinned to the running version in .env
ergoctl update update     # update now (TARGET_VERSION=2.19.1 to pin); honours the safety rails
ergoctl update install    # (re)schedule  /  uninstall to stop
```

`UPDATE_POLICY` (in `.env`) drives the daily `run`:

| Policy | Behaviour |
|---|---|
| `latest` *(default)* | Update to the newest release whenever one exists. |
| `security` | Update **only** when a published GitHub security advisory covers the running version, or a release since it has a `### Security` section in its notes (upstream only opened GHSAs from 2.19.1). |
| `off` | Never change the running version (check/notify only). |

Every update, scheduled or manual:

1. pulls the pinned `ghcr.io/ergochat/ergo:vX.Y.Z` and **pre-flights** it against
   a copy of `ircd/` (`ergo run --smoke` — also dry-runs a DB schema upgrade);
2. `NOTICE`s connected users and waits `UPDATE_GRACE` seconds (default 60);
3. stops Ergo and snapshots `ircd.db` to `backups/` (consistent copy; buntdb is
   append-only, so a live copy could miss the last second), keeps the last 5;
4. flips `ERGO_TAG`, starts, waits for the healthcheck **and** a registration
   handshake;
5. on failure: previous tag back; if the old version refuses the upgraded
   database ("Database requires update") the snapshot is restored too. Cause
   and snapshot path go into the ntfy message;
6. on success: saves the new image's `default.yaml` under `templates/` and, if it
   differs from the previous version's, writes `templates/diff-vA-vB.txt` so
   new/renamed options are visible.

Releases whose notes announce **compatibility breaks** are held (one ntfy, not
daily) until you run `ergoctl update update` or set `FORCE_UPDATE=1`. Restarting
Ergo disconnects every user (there is no hot restart), so pick `security` if
that matters more than being current. Caddy is only updated by
`ergoctl update caddy` (or daily with `CADDY_AUTOUPDATE=1`); recreating it drops
web-client websockets. Update results reuse the ntfy config at
`/etc/ssh-notify.conf` when present.

## Backups

`ergoctl backup` writes `backups/ergo-backup-<ts>.tar.gz` (0600) with
`ircd.yaml`, `ircd.db`, `ergo.motd`, the message history (the SQLite file, or a
`pg_dump` — a file copy of a live PostgreSQL data directory would not be a valid
backup) and a `meta` file (version, history backend, date). By default
it stops Ergo for a few seconds for a consistent copy (`--live` skips that).
Put an age public key in `$STACK_DIR/age-recipients.txt` (e.g. from
`globals/age-pubkey.txt`) **and install `age`** (`apk add age`, `apt install age`,
`dnf install age` — deploy.sh does not) and backups are encrypted with it; with
the recipients file present but `age` missing, `ergoctl backup` refuses rather
than writing the account and oper hashes out in the clear. TLS files are not
included — certsync regenerates them. `ergoctl restore <file>` stops Ergo, keeps the current config and database as
`backups/{ircd.yaml,ircd.db}.pre-restore.<ts>`, warns if the backup came from a
newer version (older Ergo cannot read a newer schema), restores, validates the
result before starting, and waits for health. Because the archive carries the
*original* host's credentials, restore re-points `datastore.postgresql.password`
at this host's `secrets/postgres.pass` and re-hashes this host's
`secrets/admin.pass` into the config — otherwise a rebuild would come up healthy
but leave you unable to `/OPER`. An encrypted backup needs the private key:

```bash
AGE_IDENTITY=/root/age.key ergoctl restore backups/ergo-backup-<ts>.tar.gz.age
``` Ergo's own `datastore.autoupgrade`
leaves `ircd/ircd.db.v<N>.<ts>.bak` files behind on schema upgrades; prune them
once you are happy with a release.

## Web client (optional)

The `wss://DOMAIN/webirc` endpoint is ready; Ergo only accepts browser
connections whose `Origin` is `https://DOMAIN` (`server.websockets.allowed-origins`).
To host [Gamja](https://codeberg.org/emersion/gamja) on the same origin, unpack a
release tarball into `caddy/www/gamja`, add
`{"server": {"url": "/webirc", "autojoin": "#lobby"}}` as `config.json`, mount
`./caddy/www:/www:ro` in the caddy service and drop a `handle { root * /www/gamja
file_server }` block into `caddy/etc/conf.d/` (see `00-readme.caddy`), then
`ergoctl caddy-restart`.

## Security model

- **Host loopback is Ergo's trust boundary.** Loopback peers are exempt from
  bans and ip-limits, are treated as secure, and (because
  `proxy-allowed-from: [localhost]` is what lets Caddy's `X-Forwarded-For`
  through) may assert a client IP. Every local process and every other
  host-network container can reach `127.0.0.1:6667`/`:8097`. This stack is for
  a single-purpose container host without untrusted local users; deploy.sh
  warns about other host-network containers.
- **Ergo runs unprivileged**: as the host's `ergo` system user, read-only root
  filesystem, all capabilities dropped, `no-new-privileges`. `ircd/` (database,
  config with the oper hash, TLS key) is `0700 ergo:ergo`; `secrets/`,
  `backups/`, `caddy/data` are `0700 root`.
- **Root never follows a symlink into `ircd/`.** That directory is writable by
  the container uid while the cert sync and every `ergoctl` config edit run as
  root, so a plain `cp`/`install`/`>` there would let code execution inside Ergo
  redirect a root write onto any file on the host. Every such write stages under
  `$STACK_DIR` (0700 root) and lands with `mv` — `rename(2)` replaces a symlink
  instead of following it — and root reads use `cp -P` or refuse outright.
- **Caddy runs as root** (it needs `NET_BIND_SERVICE` for 80/443) with every
  other capability dropped, admin API off, HTTP/3 off.
- **Loopback plaintext is the control plane by design**: OPER over loopback is
  not a downgrade (Ergo marks it secure), and it lets `ergoctl` work even when a
  ban or throttle would lock out a remote client. `PLAINTEXT=1` puts real users'
  credentials on the wire in clear — leave it at `0`.
- The admin password is sent to Ergo only over loopback and stored only in
  `secrets/admin.pass`. `ergo genpasswd` hashes at bcrypt cost 4, so oper
  passwords are always generated (24 random chars), never typed.

## Files

| File | Purpose |
|---|---|
| `docker-compose.yml` | caddy + ergo, both `network_mode: host`, hardened; IRC-level healthcheck. |
| `docker-compose.postgres.yml` | Optional PostgreSQL overlay for `HISTORY=postgres` (loopback-only, pinned major). |
| `Caddyfile` | ACME, `/webirc` websocket proxy, text landing page, `admin off`, no h3; imports `conf.d/*.caddy`. |
| `conf.d-readme.caddy` | Installed once as `caddy/etc/conf.d/00-readme.caddy` (operator drop-ins; Gamja example). |
| `ergo.motd` | MOTD template (`__NETWORK_NAME__`, `__DOMAIN__`). |
| `ergolib.sh` | Shared helpers: `.env`, compose, health/IRC probes, `--smoke` validator, oper edits, certsync, ntfy. |
| `update.sh` | Updater + certsync + Caddy update + scheduling. |
| `ergoctl` | Host admin CLI. |
| `.env.example` | Stack tunables. |
| `deploy.sh` / `build.sh` | Self-contained installer + archive embedder. |
| `cloud-init.yml` | Fresh-VM bootstrap (harden SSH, then deploy). |

On the host: `/srv/ergo/{ircd,caddy/{etc,data,config},secrets,backups,templates}`
plus `.state/` (notification de-duplication) and `.env.bak.<ts>` copies from each
update (last 5 kept); `/etc/ergo-update.conf` (where the jobs find the stack);
`/usr/local/bin/ergoctl`; `/var/log/ergo-update.log` (the **daily update run**
logs there, since busybox crond has nowhere else to put its output — cert-sync
failures instead surface through ntfy and as `certsync: LAST FAILURE` in
`ergoctl status`).

## Notes

- Only one host-networked Caddy fits on a box: don't co-locate this stack with
  another Caddy/80/443 stack (deploy.sh checks the ports).
- DNS for `ERGO_DOMAIN` must resolve to the host and 80/443 be reachable before
  deploy for the cert to issue; Ergo stays up on a self-signed cert meanwhile.
- Ergo's built-in registration is open by default, and **`allow-before-connect`
  is also on**, so accounts can be created by a client that has not finished
  connecting, throttled only globally (30 attempts per 10 minutes ≈ 4,300/day).
  Each account is a durable row in `ircd.db`. For a private network set
  `accounts.registration.enabled: false`, or keep registration but set
  `allow-before-connect: false`, with `ergoctl edit`.
- The admin oper password is printed by `deploy.sh` only on the first run and
  only to a terminal. Unattended runs (cloud-init) print the path instead, so the
  credential does not end up in the provider's serial-console log; read it with
  `ergoctl passwd show`.