automations/deployments/ergo

ergo

Ergo — a modern IRC server with built-in services (NickServ/ChanServ/HostServ), message history, always-on "bouncer" clients and IRCv3 — behind Caddy for automatic Let's Encrypt TLS. Ships a host CLI (ergoctl) for day-to-day administration and a health-checked updater (update.sh) that also keeps Ergo's TLS certificate in sync with Caddy.

Docker, not an OpenRC/systemd service

Ergo is a single static Go binary, so a native install is perfectly possible. This stack still uses Docker Compose because it matches the repo's goals better:

	Docker (this stack)	Native OpenRC / systemd
Alpine, Debian, Alma	one compose file, one image	Alpine packages ergo in community, but a stable branch keeps the version it shipped with (3.24 → 2.18.0, 3.23/3.22 → 2.16.0) while upstream is at 2.19.1, a security release; only edge tracks upstream. Debian/Alma have no package at all → hand-rolled tarball install + a unit per init system
Upgrade / rollback	ERGO_TAG in .env, health-checked, DB snapshot, rollback	replace a binary + hand-restore the DB
ACME TLS	stock Caddy (already used by every other stack)	certbot/acme.sh per distro + deploy hook
Real client IPs	yes — host networking (see below)	yes

The usual objection to containerised IRC — bans, throttling and IP cloaking need the client's real address, and Docker's userland proxy hides IPv6 clients behind the bridge gateway — is solved by running both containers with network_mode: host. That also means the host's deny-by-default firewall genuinely governs the ports (the "Docker bypasses INPUT" caveat from the root README does not apply here).

Topology

Port	Who listens	Purpose
6697/tcp	Ergo	IRC over TLS (Let's Encrypt cert, copied from Caddy)
443/tcp	Caddy	wss://DOMAIN/webirc → Ergo's websocket listener on 127.0.0.1:8097; text landing page at /
80/tcp	Caddy	ACME HTTP-01 + redirect to HTTPS
127.0.0.1:6667	Ergo	loopback plaintext: compose healthcheck + ergoctl (exempt from ip-limits/bans, treated as secure)
6667/tcp public	Ergo	only with PLAINTEXT=1 (STS is then advertised) — not recommended

No HTTP/3 (would bind udp/443), Caddy's admin API is off (it would sit on the host's loopback), and there is no ports: section — host mode.

Required .env values

Variable	Notes
ERGO_DOMAIN	Public hostname (lower-case). Server name, cert subject, Caddy site and websocket origin. Immutable after the first deploy.
ACME_EMAIL	Let's Encrypt registration email.
NETWORK_NAME	IRC network name (letters, digits, . _ -; no spaces). Defaults to ERGO_DOMAIN.
ERGO_TAG	Pinned image tag with the v (v2.19.1). Set by deploy.sh to the newest release; bumped by update.sh. Never stable/latest — if a first deploy could not reach GitHub it falls back to stable, and the next update run pins it.
PLAINTEXT	0 (default) or 1 — public plaintext 6667. Fixed at the first deploy (it selects a listener in ircd.yaml, which deploy.sh never rewrites); see below to change it.
UPDATE_POLICY	latest (default) / security / off — see Updates.

See .env.example for the rest (UPDATE_GRACE, FORCE_UPDATE, CADDY_AUTOUPDATE, CADDY_TAG). .env must not contain $ (Compose interpolates it) — secrets never live there. The four update knobs are read from .env by the scheduled jobs; an environment variable, or a value pinned in /etc/ergo-update.conf, overrides it for that run.

Deploy-time-only options (not stored in .env): ERGO_AUTOUPDATE=0 installs update.sh without scheduling the daily update (cert sync is still scheduled), and CERT_WAIT (default 180s) bounds the wait for the first Let's Encrypt cert.

To switch PLAINTEXT after the first deploy: change the plaintext listener with ergoctl edit ("127.0.0.1:6667": ↔ ":6667":, and server.sts.enabled with it), set PLAINTEXT in .env, run ergoctl restart, then re-run deploy.sh so the host firewall matches. Passing a conflicting PLAINTEXT= to a re-run stops with those instructions rather than half-applying the change.

Deploy

./automations.sh        # Deploy on this host → deploy: ergo

Or build + run the self-contained artifact:

./build.sh
scp deploy.sh root@host:
ssh root@host 'bash deploy.sh'
# non-interactive:
#   ERGO_DOMAIN=irc.example.com ACME_EMAIL=me@example.com NETWORK_NAME=MyNet SKIP_PROMPTS=1 bash deploy.sh

Unattended provisioning: cloud-init.yml. On first run deploy.sh:

1. installs Docker (+ openssl, jq, curl) and creates the ergo system user the container runs as;
2. registers 80,443,6697/tcp with the host firewall (/etc/firewall/ports.d/ergo.rule, or ufw/firewalld);
3. pins ERGO_TAG to the newest release and seeds .env;
4. generates ircd/ircd.yaml once, from the pulled image's own default.yaml (so it always matches the running version): server/network name, the listeners above, the websocket origin, the cloak suffix, and an admin oper whose random password is saved to secrets/admin.pass (0600; only the bcrypt hash goes into the yaml). Hard post-checks refuse to continue if the upstream template layout ever changes under the edits;
5. validates the config with ergo run --smoke in a throwaway container before anything starts, validates the Caddyfile, then starts the stack and waits for both healthchecks (Ergo's is IRC-level: a QUIT must be answered with ERROR);
6. waits for Caddy's Let's Encrypt cert, copies it into ircd/ and rehashes Ergo (until then Ergo serves a self-signed cert from its own mkcerts);
7. schedules update.sh certsync every 15 min and update.sh run daily (ERGO_AUTOUPDATE=0 schedules only the cert sync).

Re-runs are idempotent: they never touch ircd/ircd.yaml, secrets/ or caddy/etc/conf.d/, and they adopt the deployed settings from .env rather than this shell's defaults, so a plain bash deploy.sh cannot silently close a PLAINTEXT=1 firewall port or reset your update policy. A setting you do pass explicitly is written back to .env. A changed Caddyfile triggers a Caddy restart, since Caddy has no admin API here.

Administer: ergoctl

Installed at /usr/local/bin/ergoctl. It talks to Ergo over the loopback listener as the admin oper.

Command	What it does
ergoctl status / users / logs [-f] / version	health, versions, TLS state, user count; server log with health-probe noise filtered
ergoctl edit / motd	$EDITOR on ircd.yaml / ergo.motd, then validate (throwaway container) and REHASH; the previous file is restored if either step fails
ergoctl rehash	reload config, MOTD and TLS certs without disconnecting anyone
ergoctl restart / stop / start	restart the container (drops every user; needed for settings Ergo fixes at startup, such as server.name, datastore.path, casemapping, enforce-utf8, max-line-len, idle-timeouts). Changes to docker-compose.yml or .env need docker compose up -d instead
ergoctl oper list|add|passwd|rm	manage operators in ircd.yaml; passwords are generated (24 chars) and printed once
ergoctl passwd [show|rotate]	the admin oper password
ergoctl announce <text>	NOTICE to everyone
ergoctl kill <nick> [reason] · ban add|del|list|info · defcon [1-5]	moderation (UBAN under the hood). ban add takes an IP, CIDR, nick!user@host mask, or account name — a bare name is an account to suspend, not a connected nick, so run ban info <nick> first to get their IP
ergoctl cmd <raw IRC line>	anything else as the admin oper, e.g. ergoctl cmd NS SAREGISTER alice hunter2, ergoctl cmd CS PURGE #spam; replies are printed with control characters stripped
ergoctl cert [show|sync]	cert on disk vs the one served on 6697; force a sync
ergoctl backup [--live] [dir] / restore <file>	see Backups
ergoctl update …	passthrough to update.sh
ergoctl debug on|off · caddy-restart · shell	log level; apply Caddyfile/conf.d changes; a shell in the container

Editing by hand works too: change ircd/ircd.yaml, then ergoctl rehash — a rejected config leaves the old one running (Ergo's rehash is transactional), but the file on disk would then be unbootable, which is why ergoctl edit validates first. Ergo's manual covers every option; the pristine template for your version is in templates/.

TLS

Caddy owns the ACME account and renews the cert on its own schedule (ARI-driven). Ergo terminates TLS itself on 6697, so the cert has to reach ircd/fullchain.pem

• privkey.pem. update.sh certsync (every 15 minutes, via busybox crond or a systemd timer) does that the way upstream's certbot deploy-hook recipe does, plus safety checks:

• picks the newest <domain>.crt under caddy/data/caddy/certificates/*/ (Let's Encrypt or the ZeroSSL fallback), skipping a renewal that is still mid-write;
• verifies the key matches the cert (certmagic writes them as separate files and generates a new key on every renewal), stages both in ircd/ and installs them with two atomic mvs — Ergo never sees a half pair, which would be fatal at its next start;
• SIGHUPs Ergo (rehash swaps the TLS config for new connections only) and confirms the fingerprint served on 6697 changed; if not, the previous pair is restored and you get an ntfy alert.

Silent when nothing changed. The daily run also warns when the served cert is still self-signed (DNS/80/443 not right yet) or expires within 14 days (certsync not running). ergoctl cert shows both sides.

Updates

ergoctl update check      # running vs latest, published security advisories, cert state; changes nothing
ergoctl update update     # update now (TARGET_VERSION=2.19.1 to pin); honours the safety rails
ergoctl update install    # (re)schedule  /  uninstall to stop

UPDATE_POLICY (in .env) drives the daily run:

Policy	Behaviour
latest (default)	Update to the newest release whenever one exists.
security	Update only when a published GitHub security advisory covers the running version, or a release since it has a ### Security section in its notes (upstream only opened GHSAs from 2.19.1).
off	Never change the running version (check/notify only).

Every update, scheduled or manual:

1. pulls the pinned ghcr.io/ergochat/ergo:vX.Y.Z and pre-flights it against a copy of ircd/ (ergo run --smoke — also dry-runs a DB schema upgrade);
2. NOTICEs connected users and waits UPDATE_GRACE seconds (default 60);
3. stops Ergo and snapshots ircd.db to backups/ (consistent copy; buntdb is append-only, so a live copy could miss the last second), keeps the last 5;
4. flips ERGO_TAG, starts, waits for the healthcheck and a registration handshake;
5. on failure: previous tag back; if the old version refuses the upgraded database ("Database requires update") the snapshot is restored too. Cause and snapshot path go into the ntfy message;
6. on success: saves the new image's default.yaml under templates/ and, if it differs from the previous version's, writes templates/diff-vA-vB.txt so new/renamed options are visible.

Releases whose notes announce compatibility breaks are held (one ntfy, not daily) until you run ergoctl update update or set FORCE_UPDATE=1. Restarting Ergo disconnects every user (there is no hot restart), so pick security if that matters more than being current. Caddy is only updated by ergoctl update caddy (or daily with CADDY_AUTOUPDATE=1); recreating it drops web-client websockets. Update results reuse the ntfy config at /etc/ssh-notify.conf when present.

Backups

ergoctl backup writes backups/ergo-backup-<ts>.tar.gz (0600) with ircd.yaml, ircd.db, ergo.motd and a meta file (version, date). By default it stops Ergo for a few seconds for a consistent copy (--live skips that). Put an age public key in $STACK_DIR/age-recipients.txt (e.g. from globals/age-pubkey.txt) and install age (apk add age, apt install age, dnf install age — deploy.sh does not) and backups are encrypted with it; with the recipients file present but age missing, ergoctl backup refuses rather than writing the account and oper hashes out in the clear. TLS files are not included — certsync regenerates them. ergoctl restore <file> stops Ergo, warns if the backup came from a newer version (older Ergo cannot read a newer schema), restores, and waits for health. Ergo's own datastore.autoupgrade leaves ircd/ircd.db.v<N>.<ts>.bak files behind on schema upgrades; prune them once you are happy with a release.

Web client (optional)

The wss://DOMAIN/webirc endpoint is ready; Ergo only accepts browser connections whose Origin is https://DOMAIN (server.websockets.allowed-origins). To host Gamja on the same origin, unpack a release tarball into caddy/www/gamja, add {"server": {"url": "/webirc", "autojoin": "#lobby"}} as config.json, mount ./caddy/www:/www:ro in the caddy service and drop a handle { root * /www/gamja file_server } block into caddy/etc/conf.d/ (see 00-readme.caddy), then ergoctl caddy-restart.

Security model

• Host loopback is Ergo's trust boundary. Loopback peers are exempt from bans and ip-limits, are treated as secure, and (because proxy-allowed-from: [localhost] is what lets Caddy's X-Forwarded-For through) may assert a client IP. Every local process and every other host-network container can reach 127.0.0.1:6667/:8097. This stack is for a single-purpose container host without untrusted local users; deploy.sh warns about other host-network containers.
• Ergo runs unprivileged: as the host's ergo system user, read-only root filesystem, all capabilities dropped, no-new-privileges. ircd/ (database, config with the oper hash, TLS key) is 0700 ergo:ergo; secrets/, backups/, caddy/data are 0700 root.
• Caddy runs as root (it needs NET_BIND_SERVICE for 80/443) with every other capability dropped, admin API off, HTTP/3 off.
• Loopback plaintext is the control plane by design: OPER over loopback is not a downgrade (Ergo marks it secure), and it lets ergoctl work even when a ban or throttle would lock out a remote client. PLAINTEXT=1 puts real users' credentials on the wire in clear — leave it at 0.
• The admin password is sent to Ergo only over loopback and stored only in secrets/admin.pass. ergo genpasswd hashes at bcrypt cost 4, so oper passwords are always generated (24 random chars), never typed.

Files

File	Purpose
docker-compose.yml	caddy + ergo, both network_mode: host, hardened; IRC-level healthcheck.
Caddyfile	ACME, /webirc websocket proxy, text landing page, admin off, no h3; imports conf.d/*.caddy.
conf.d-readme.caddy	Installed once as caddy/etc/conf.d/00-readme.caddy (operator drop-ins; Gamja example).
ergo.motd	MOTD template (__NETWORK_NAME__, __DOMAIN__).
ergolib.sh	Shared helpers: .env, compose, health/IRC probes, --smoke validator, oper edits, certsync, ntfy.
update.sh	Updater + certsync + Caddy update + scheduling.
ergoctl	Host admin CLI.
.env.example	Stack tunables.
deploy.sh / build.sh	Self-contained installer + archive embedder.
cloud-init.yml	Fresh-VM bootstrap (harden SSH, then deploy).

On the host: /srv/ergo/{ircd,caddy/{etc,data,config},secrets,backups,templates} plus .state/ (notification de-duplication) and .env.bak.<ts> copies from each update (last 5 kept); /etc/ergo-update.conf (where the jobs find the stack); /usr/local/bin/ergoctl; /var/log/ergo-update.log (scheduled runs log there, since busybox crond has nowhere else to put their output).

Notes

• Only one host-networked Caddy fits on a box: don't co-locate this stack with another Caddy/80/443 stack (deploy.sh checks the ports).
• DNS for ERGO_DOMAIN must resolve to the host and 80/443 be reachable before deploy for the cert to issue; Ergo stays up on a self-signed cert meanwhile.
• Ergo's built-in registration is open by default (accounts.registration.enabled); edit ircd.yaml if you want an invite-only network.