automations/deployments/ergo

ergo

Ergo — a modern IRC server with built-in services (NickServ/ChanServ/HostServ), message history, always-on "bouncer" clients and IRCv3 — behind Caddy for automatic Let's Encrypt TLS. Ships a host CLI (ergoctl) for day-to-day administration and a health-checked updater (update.sh) that also keeps Ergo's TLS certificate in sync with Caddy.

Docker, not an OpenRC/systemd service

Ergo is a single static Go binary, so a native install is perfectly possible. This stack still uses Docker Compose because it matches the repo's goals better:

	Docker (this stack)	Native OpenRC / systemd
Alpine, Debian, Alma	one compose file, one image	Alpine packages ergo in community, but a stable branch keeps the version it shipped with (3.24 → 2.18.0, 3.23/3.22 → 2.16.0) while upstream is at 2.19.1, a security release; only edge tracks upstream. Debian/Alma have no package at all → hand-rolled tarball install + a unit per init system
Upgrade / rollback	ERGO_TAG in .env, health-checked, DB snapshot, rollback	replace a binary + hand-restore the DB
ACME TLS	stock Caddy (already used by every other stack)	certbot/acme.sh per distro + deploy hook
Real client IPs	yes — host networking (see below)	yes

The usual objection to containerised IRC — bans, throttling and IP cloaking need the client's real address, and Docker's userland proxy hides IPv6 clients behind the bridge gateway — is solved by running both containers with network_mode: host. That also means the host's deny-by-default firewall genuinely governs the ports (the "Docker bypasses INPUT" caveat from the root README does not apply here).

Topology

Port	Who listens	Purpose
6697/tcp	Ergo	IRC over TLS (Let's Encrypt cert, copied from Caddy)
443/tcp	Caddy	wss://DOMAIN/webirc → Ergo's websocket listener on 127.0.0.1:8097; text landing page at /
80/tcp	Caddy	ACME HTTP-01 + redirect to HTTPS
127.0.0.1:6667	Ergo	loopback plaintext: compose healthcheck + ergoctl (exempt from ip-limits/bans, treated as secure)
6667/tcp public	Ergo	only with PLAINTEXT=1 (STS is then advertised) — not recommended

No HTTP/3 (would bind udp/443), Caddy's admin API is off (it would sit on the host's loopback), and there is no ports: section — host mode.

Required .env values

Variable	Notes
ERGO_DOMAIN	Public hostname (lower-case). Server name, cert subject, Caddy site and websocket origin. Immutable after the first deploy.
ACME_EMAIL	Let's Encrypt registration email.
NETWORK_NAME	IRC network name (letters, digits, . _ -; no spaces). Defaults to ERGO_DOMAIN.
ERGO_TAG	Pinned image tag with the v (v2.19.1). Set by deploy.sh to the newest release; bumped by update.sh. Never stable/latest — if a first deploy could not reach GitHub it falls back to stable, and the next update run pins it.
PLAINTEXT	0 (default) or 1 — public plaintext 6667. Fixed at the first deploy (it selects a listener in ircd.yaml, which deploy.sh never rewrites); see below to change it.
HISTORY	sqlite (default) / postgres / off — see Message history.
UPDATE_POLICY	latest (default) / security / off — see Updates.

See .env.example for the rest (UPDATE_GRACE, FORCE_UPDATE, CADDY_AUTOUPDATE, CADDY_TAG). .env must not contain $ (Compose interpolates it) — secrets never live there. The four update knobs are read from .env by the scheduled jobs; an environment variable, or a value pinned in /etc/ergo-update.conf, overrides it for that run.

Deploy-time-only options (not stored in .env): ERGO_AUTOUPDATE=0 installs update.sh without scheduling the daily update (cert sync is still scheduled), and CERT_WAIT (default 180s) bounds the wait for the first Let's Encrypt cert.

To switch PLAINTEXT after the first deploy: change the plaintext listener with ergoctl edit ("127.0.0.1:6667": ↔ ":6667":, and server.sts.enabled with it), set PLAINTEXT in .env, run ergoctl restart, then re-run deploy.sh so the host firewall matches. Passing a conflicting PLAINTEXT= to a re-run stops with those instructions rather than half-applying the change.

Deploy

./automations.sh        # Deploy on this host → deploy: ergo

Or build + run the self-contained artifact:

./build.sh
scp deploy.sh root@host:
ssh root@host 'bash deploy.sh'
# non-interactive:
#   ERGO_DOMAIN=irc.example.com ACME_EMAIL=me@example.com NETWORK_NAME=MyNet SKIP_PROMPTS=1 bash deploy.sh

Unattended provisioning: cloud-init.yml. On first run deploy.sh:

1. installs Docker (+ openssl, jq, curl) and creates the ergo system user the container runs as;
2. registers 80,443,6697/tcp with the host firewall (/etc/firewall/ports.d/ergo.rule, or ufw/firewalld);
3. pins ERGO_TAG to the newest release and seeds .env;
4. generates ircd/ircd.yaml once, from the pulled image's own default.yaml (so it always matches the running version): server/network name, the listeners above, the websocket origin, the cloak suffix, and an admin oper whose random password is saved to secrets/admin.pass (0600; only the bcrypt hash goes into the yaml). Hard post-checks refuse to continue if the upstream template layout ever changes under the edits;
5. validates the config with ergo run --smoke in a throwaway container before anything starts, validates the Caddyfile, then starts the stack and waits for both healthchecks (Ergo's is IRC-level: a QUIT must be answered with ERROR);
6. waits for Caddy's Let's Encrypt cert, copies it into ircd/ and rehashes Ergo (until then Ergo serves a self-signed cert from its own mkcerts);
7. schedules update.sh certsync every 15 min and update.sh run daily (ERGO_AUTOUPDATE=0 schedules only the cert sync).

Re-runs are idempotent: they never touch ircd/ircd.yaml, secrets/ or caddy/etc/conf.d/, and they adopt the deployed settings from .env rather than this shell's defaults, so a plain bash deploy.sh cannot silently close a PLAINTEXT=1 firewall port or reset your update policy. A setting you do pass explicitly is written back to .env. A changed Caddyfile triggers a Caddy restart, since Caddy has no admin API here.

Administer: ergoctl

Installed at /usr/local/bin/ergoctl. It talks to Ergo over the loopback listener as the admin oper.

Command	What it does
ergoctl status / users / logs [-f] / version	health, versions, TLS state, user count; server log with health-probe noise filtered
ergoctl edit / motd	$EDITOR on ircd.yaml / ergo.motd, then validate (throwaway container) and REHASH; the previous file is restored if either step fails
ergoctl rehash	reload config, MOTD and TLS certs without disconnecting anyone
ergoctl restart / stop / start	restart the container (drops every user; needed for settings Ergo fixes at startup, such as server.name, datastore.path, casemapping, enforce-utf8, max-line-len, idle-timeouts). Changes to docker-compose.yml or .env need docker compose up -d instead
ergoctl oper list|add|passwd|rm	manage operators in ircd.yaml; passwords are generated (24 chars) and printed once
ergoctl passwd [show|rotate]	the admin oper password
ergoctl announce <text>	NOTICE to everyone
ergoctl kill <nick> [reason] · ban add|del|list|info · defcon [1-5]	moderation (UBAN under the hood). ban add takes an IP, CIDR, nick!user@host mask, or account name — a bare name is an account to suspend, not a connected nick, so run ban info <nick> first to get their IP
ergoctl cmd <raw IRC line>	anything else as the admin oper, e.g. ergoctl cmd NS SAREGISTER alice hunter2, ergoctl cmd CS PURGE ADD #spam (CS PURGE DEL to undo); replies are printed with control characters stripped
ergoctl cert [show|sync]	cert on disk vs the one served on 6697; force a sync
ergoctl backup [--live] [dir] / restore <file>	see Backups
ergoctl update …	passthrough to update.sh
ergoctl history [backend]	show or switch the persistent-history backend
ergoctl debug on|off · caddy-restart · shell	log level; apply Caddyfile/conf.d changes; a shell in the container

Editing by hand works too: change ircd/ircd.yaml, then ergoctl rehash — a rejected config leaves the old one running (Ergo's rehash is transactional), but the file on disk would then be unbootable, which is why ergoctl edit validates first. Ergo's manual covers every option; the pristine template for your version is in templates/.

TLS

Caddy owns the ACME account and renews the cert on its own schedule (ARI-driven). Ergo terminates TLS itself on 6697, so the cert has to reach ircd/fullchain.pem

• privkey.pem. update.sh certsync (every 15 minutes, via busybox crond or a systemd timer) does that the way upstream's certbot deploy-hook recipe does, plus safety checks:

• picks the newest <domain>.crt under caddy/data/caddy/certificates/*/ (Let's Encrypt or the ZeroSSL fallback), skipping a renewal that is still mid-write;
• verifies the key matches the cert (certmagic writes them as separate files and generates a new key on every renewal), stages both in ircd/ and installs them with two atomic mvs — Ergo never sees a half pair, which would be fatal at its next start;
• SIGHUPs Ergo (rehash swaps the TLS config for new connections only) and confirms the fingerprint served on 6697 changed; if not, the previous pair is restored and you get an ntfy alert.

Silent when nothing changed. The daily run also warns when the served cert is still self-signed (DNS/80/443 not right yet) or expires within 14 days (certsync not running). ergoctl cert shows both sides.

Message history

Ergo keeps channel and DM history for CHATHISTORY, /HISTORY, autoreplay-on-join and always-on (bouncer) clients. Upstream keeps it in RAM, so it is lost on every restart — including the ones this stack's updater performs. So persistence is on by default here:

HISTORY	What it does
sqlite (default)	A file at ircd/ergo_history.db, next to the account database. No extra container, backed up with everything else. Right for a single-node server.
postgres	A pinned PostgreSQL container (docker-compose.postgres.yml), reachable only on 127.0.0.1. Choose it if you want a real database to query, or already run Postgres.
off	Upstream behaviour: RAM only.

Both SQL backends need Ergo 2.18.0+ (deploy.sh refuses older pins). How long messages are kept is history.restrictions.expire-time in ircd.yaml — one week by default — not a size limit: enabling persistence removes the in-memory channel-length/client-length caps. Raise or lower it with ergoctl edit, and note the privacy point upstream flags: persisted messages are personal data, so check what your jurisdiction expects of you before extending retention.

ergoctl history                 # backend, database size / connectivity, retention
ergoctl history postgres        # switch (restarts Ergo; does NOT migrate messages)

Switching backends leaves the old store on disk and starts the new one empty — there is no migration path, so pick one at deploy time if you can.

PostgreSQL

HISTORY=postgres adds docker-compose.postgres.yml to COMPOSE_FILE in .env, which docker compose reads by itself — so every command in this stack sees the same services with no extra flags. The container publishes only to 127.0.0.1:5432, Ergo reaches it there (the host-networked containers cannot use compose service DNS), and depends_on: service_healthy keeps Ergo from starting before the database accepts connections. The password is generated into secrets/postgres.pass (0600) and passed via POSTGRES_PASSWORD_FILE, so it never lands in .env.

The major version is pinned and the updater never touches it. PostgreSQL refuses to start on a data directory written by a different major version, so moving from 17-alpine to 18-alpine is a deliberate dump-and-restore:

ergoctl backup                      # contains a pg_dump of the history database
ergoctl history off && docker compose down -v postgres   # drops the old data volume
# set POSTGRES_TAG=18-alpine in .env, then:
ergoctl history postgres && ergoctl restore backups/ergo-backup-<ts>.tar.gz

Note that Postgres 18 also changed the image's default data directory layout; the compose file pins PGDATA explicitly so a future image default cannot move it.

Updates

ergoctl update check      # running vs latest, published security advisories, cert state
                          # reports only, with one exception: a floating ERGO_TAG
                          # ('stable') is pinned to the running version in .env
ergoctl update update     # update now (TARGET_VERSION=2.19.1 to pin); honours the safety rails
ergoctl update install    # (re)schedule  /  uninstall to stop

UPDATE_POLICY (in .env) drives the daily run:

Policy	Behaviour
latest (default)	Update to the newest release whenever one exists.
security	Update only when a published GitHub security advisory covers the running version, or a release since it has a ### Security section in its notes (upstream only opened GHSAs from 2.19.1).
off	Never change the running version (check/notify only).

Every update, scheduled or manual:

1. pulls the pinned ghcr.io/ergochat/ergo:vX.Y.Z and pre-flights it against a copy of ircd/ (ergo run --smoke — also dry-runs a DB schema upgrade);
2. NOTICEs connected users and waits UPDATE_GRACE seconds (default 60);
3. stops Ergo and snapshots ircd.db to backups/ (consistent copy; buntdb is append-only, so a live copy could miss the last second), keeps the last 5;
4. flips ERGO_TAG, starts, waits for the healthcheck and a registration handshake;
5. on failure: previous tag back; if the old version refuses the upgraded database ("Database requires update") the snapshot is restored too. Cause and snapshot path go into the ntfy message;
6. on success: saves the new image's default.yaml under templates/ and, if it differs from the previous version's, writes templates/diff-vA-vB.txt so new/renamed options are visible.

Releases whose notes announce compatibility breaks are held (one ntfy, not daily) until you run ergoctl update update or set FORCE_UPDATE=1. Restarting Ergo disconnects every user (there is no hot restart), so pick security if that matters more than being current. Caddy is only updated by ergoctl update caddy (or daily with CADDY_AUTOUPDATE=1); recreating it drops web-client websockets. Update results reuse the ntfy config at /etc/ssh-notify.conf when present.

Backups

ergoctl backup writes backups/ergo-backup-<ts>.tar.gz (0600) with ircd.yaml, ircd.db, ergo.motd, the message history (the SQLite file, or a pg_dump — a file copy of a live PostgreSQL data directory would not be a valid backup) and a meta file (version, history backend, date). By default it stops Ergo for a few seconds for a consistent copy (--live skips that). Put an age public key in $STACK_DIR/age-recipients.txt (e.g. from globals/age-pubkey.txt) and install age (apk add age, apt install age, dnf install age — deploy.sh does not) and backups are encrypted with it; with the recipients file present but age missing, ergoctl backup refuses rather than writing the account and oper hashes out in the clear. TLS files are not included — certsync regenerates them. ergoctl restore <file> stops Ergo, keeps the current config and database as backups/{ircd.yaml,ircd.db}.pre-restore.<ts>, warns if the backup came from a newer version (older Ergo cannot read a newer schema), restores, validates the result before starting, and waits for health. Because the archive carries the original host's credentials, restore re-points datastore.postgresql.password at this host's secrets/postgres.pass and re-hashes this host's secrets/admin.pass into the config — otherwise a rebuild would come up healthy but leave you unable to /OPER. An encrypted backup needs the private key:

AGE_IDENTITY=/root/age.key ergoctl restore backups/ergo-backup-<ts>.tar.gz.age
``` Ergo's own `datastore.autoupgrade`
leaves `ircd/ircd.db.v<N>.<ts>.bak` files behind on schema upgrades; prune them
once you are happy with a release.

## Web client (optional)

The `wss://DOMAIN/webirc` endpoint is ready; Ergo only accepts browser
connections whose `Origin` is `https://DOMAIN` (`server.websockets.allowed-origins`).
To host [Gamja](https://codeberg.org/emersion/gamja) on the same origin, unpack a
release tarball into `caddy/www/gamja`, add
`{"server": {"url": "/webirc", "autojoin": "#lobby"}}` as `config.json`, mount
`./caddy/www:/www:ro` in the caddy service and drop a `handle { root * /www/gamja
file_server }` block into `caddy/etc/conf.d/` (see `00-readme.caddy`), then
`ergoctl caddy-restart`.

## Security model

- **Host loopback is Ergo's trust boundary.** Loopback peers are exempt from
  bans and ip-limits, are treated as secure, and (because
  `proxy-allowed-from: [localhost]` is what lets Caddy's `X-Forwarded-For`
  through) may assert a client IP. Every local process and every other
  host-network container can reach `127.0.0.1:6667`/`:8097`. This stack is for
  a single-purpose container host without untrusted local users; deploy.sh
  warns about other host-network containers.
- **Ergo runs unprivileged**: as the host's `ergo` system user, read-only root
  filesystem, all capabilities dropped, `no-new-privileges`. `ircd/` (database,
  config with the oper hash, TLS key) is `0700 ergo:ergo`; `secrets/`,
  `backups/`, `caddy/data` are `0700 root`.
- **Root never follows a symlink into `ircd/`.** That directory is writable by
  the container uid while the cert sync and every `ergoctl` config edit run as
  root, so a plain `cp`/`install`/`>` there would let code execution inside Ergo
  redirect a root write onto any file on the host. Every such write stages under
  `$STACK_DIR` (0700 root) and lands with `mv` — `rename(2)` replaces a symlink
  instead of following it — and root reads use `cp -P` or refuse outright.
- **Caddy runs as root** (it needs `NET_BIND_SERVICE` for 80/443) with every
  other capability dropped, admin API off, HTTP/3 off.
- **Loopback plaintext is the control plane by design**: OPER over loopback is
  not a downgrade (Ergo marks it secure), and it lets `ergoctl` work even when a
  ban or throttle would lock out a remote client. `PLAINTEXT=1` puts real users'
  credentials on the wire in clear — leave it at `0`.
- The admin password is sent to Ergo only over loopback and stored only in
  `secrets/admin.pass`. `ergo genpasswd` hashes at bcrypt cost 4, so oper
  passwords are always generated (24 random chars), never typed.

## Files

| File | Purpose |
|---|---|
| `docker-compose.yml` | caddy + ergo, both `network_mode: host`, hardened; IRC-level healthcheck. |
| `docker-compose.postgres.yml` | Optional PostgreSQL overlay for `HISTORY=postgres` (loopback-only, pinned major). |
| `Caddyfile` | ACME, `/webirc` websocket proxy, text landing page, `admin off`, no h3; imports `conf.d/*.caddy`. |
| `conf.d-readme.caddy` | Installed once as `caddy/etc/conf.d/00-readme.caddy` (operator drop-ins; Gamja example). |
| `ergo.motd` | MOTD template (`__NETWORK_NAME__`, `__DOMAIN__`). |
| `ergolib.sh` | Shared helpers: `.env`, compose, health/IRC probes, `--smoke` validator, oper edits, certsync, ntfy. |
| `update.sh` | Updater + certsync + Caddy update + scheduling. |
| `ergoctl` | Host admin CLI. |
| `.env.example` | Stack tunables. |
| `deploy.sh` / `build.sh` | Self-contained installer + archive embedder. |
| `cloud-init.yml` | Fresh-VM bootstrap (harden SSH, then deploy). |

On the host: `/srv/ergo/{ircd,caddy/{etc,data,config},secrets,backups,templates}`
plus `.state/` (notification de-duplication) and `.env.bak.<ts>` copies from each
update (last 5 kept); `/etc/ergo-update.conf` (where the jobs find the stack);
`/usr/local/bin/ergoctl`; `/var/log/ergo-update.log` (the **daily update run**
logs there, since busybox crond has nowhere else to put its output — cert-sync
failures instead surface through ntfy and as `certsync: LAST FAILURE` in
`ergoctl status`).

## Notes

- Only one host-networked Caddy fits on a box: don't co-locate this stack with
  another Caddy/80/443 stack (deploy.sh checks the ports).
- DNS for `ERGO_DOMAIN` must resolve to the host and 80/443 be reachable before
  deploy for the cert to issue; Ergo stays up on a self-signed cert meanwhile.
- Ergo's built-in registration is open by default, and **`allow-before-connect`
  is also on**, so accounts can be created by a client that has not finished
  connecting, throttled only globally (30 attempts per 10 minutes ≈ 4,300/day).
  Each account is a durable row in `ircd.db`. For a private network set
  `accounts.registration.enabled: false`, or keep registration but set
  `allow-before-connect: false`, with `ergoctl edit`.
- The admin oper password is printed by `deploy.sh` only on the first run and
  only to a terminal. Unattended runs (cloud-init) print the path instead, so the
  credential does not end up in the provider's serial-console log; read it with
  `ergoctl passwd show`.