feat(ergo): persistent message history, plus deployment-readiness audit fixes
Ergo keeps history in RAM by default, so it is lost on every restart -- including the ones this stack's updater performs. HISTORY now selects a backend at first deploy: sqlite (default, a file beside ircd.db, no extra container), postgres (a pinned container via docker-compose.postgres.yml, loopback-only, POSTGRES_PASSWORD_FILE so no secret lands in .env), or off. Both SQL backends need Ergo 2.18.0+, which deploy.sh enforces. HISTORY_EXPIRE (default 30d) sets retention, because upstream's 1w expire-time DELETES from persistent storage -- persistence with the shipped default would buy only a week. Ergo opens the history backend only at startup and, unlike MySQL, has no "after launching the server" guard for sqlite/postgresql: a rehash that enables one reports success and then silently discards every message. So `ergoctl history` restarts, and `ergoctl edit` detects a backend change and refuses to apply it by rehash. The config validator runs with no network by design, so it neutralises postgresql in its copy and reachability is checked separately. Audit fixes (six lenses over first-deploy, runtime, lifecycle, security, docs): - SECURITY (blocker): ircd/ is writable by the container uid while the 15-minute cert sync and every ergoctl config edit run as root. cp/install/> follow a symlink, so code execution in Ergo could redirect a root write onto any host file. All such writes now stage under $STACK_DIR (0700 root) and land via mv (rename(2) replaces a symlink instead of following it); root reads use cp -P or refuse. Regression-tested with real symlinks. - deploy.sh no longer swaps the image when ERGO_TAG changes on a deployed stack (that bypassed pre-flight, snapshot, user warning and rollback); it points at `TARGET_VERSION=... ergoctl update update`. - restore was lossy and could lock you out: it now snapshots the current config as well as the database, restores message history, re-points the postgres password and re-hashes the local admin password into the restored config, and validates before starting. - `install -d -m` re-modes existing directories: `ergoctl backup /var/backups` no longer chmods it 0700 root, and the updater no longer re-modes /var/log (0775 root:syslog on Debian, which rsyslog needs). - The admin oper password is printed only on a first run at a TTY, so it stays out of cloud-init serial-console logs. - A failed update is remembered, so the daily job stops repeating a disruptive warn/stop/swap/roll-back cycle every night; postgres readiness gates an update that would otherwise stop a healthy server it cannot restart. - certsync no longer sends "TLS recovered" for outcomes that synced nothing. - ergoctl history writes .env only after the restart is healthy, and returns 0. - CR/LF is stripped before IRC framing, so a multi-line argument cannot inject a second command; the ntfy token moves out of curl's argv. - ufw/firewalld are additive, so 6667 is now explicitly revoked when PLAINTEXT=0. - build.sh refuses to build a deploy.sh whose archive is missing a file the script reads -- the failure mode that would have shipped a stack aborting on every host. Docs corrected against the code throughout, including retention, the pre-connect account-registration default, encrypted-restore (AGE_IDENTITY), what a re-run really does to .env, and what the update log does and does not contain. Verified locally: all six suites pass (config render for each backend against the real 2.19.1 template, yaml/oper/version/env helpers, the IRC client against a fake server, and the audit fixes including the symlink escalation). Still not exercised on a Docker host: the containers, ACME issuance, cert sync and PostgreSQL itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -120,7 +120,7 @@ deployments/<name>/ # one folder per stack
|
||||
| [`copyparty`](deployments/copyparty/) | Portable file server — web UI/WebDAV behind Caddy, plus direct **SFTP** + **FTPS**. Ships a security-notices-aware updater. | — |
|
||||
| [`simplex`](deployments/simplex/) | SimpleX SMP + XFTP relay with Tor hidden services + encrypted backups. | globals/age-pubkey.txt |
|
||||
| [`openbao`](deployments/openbao/) | Hardened tape-encryption key store for **Kanrisha** (the LTO tape-archive system — separate repo/host). **Exception: native TLS on the LAN, no Caddy/LE.** raft storage, mlock, self-signed or Smallstep-ACME cert, manual/HSM unseal. | globals/age-pubkey.txt (backups) |
|
||||
| [`ergo`](deployments/ergo/) | IRC server ([Ergo](https://ergo.chat)) with Caddy for Let's Encrypt + the websocket endpoint. **Both containers use host networking** (real client IPs for bans/cloaking; the host firewall applies). Ships `ergoctl` (admin CLI) and a health-checked updater that also syncs Caddy's cert into Ergo. | — |
|
||||
| [`ergo`](deployments/ergo/) | IRC server ([Ergo](https://ergo.chat)) with Caddy for Let's Encrypt + the websocket endpoint. **Both containers use host networking** (real client IPs for bans/cloaking; the host firewall applies). Persistent message history via SQLite (default) or an optional PostgreSQL container. Ships `ergoctl` (admin CLI) and a health-checked updater that also syncs Caddy's cert into Ergo. | — |
|
||||
|
||||
## Conventions
|
||||
|
||||
|
||||
@@ -188,6 +188,7 @@ ask_deployment_vars() {
|
||||
ask ERGO_DOMAIN "IRC server hostname (e.g. irc.example.com)"
|
||||
ask ACME_EMAIL "Let's Encrypt email"
|
||||
ask NETWORK_NAME "IRC network name, no spaces (blank = the hostname)" optional
|
||||
ask HISTORY "Persistent message history: sqlite | postgres | off (blank = sqlite)" optional
|
||||
ask PLAINTEXT "Also serve PUBLIC plaintext IRC on 6667? (1 = yes, blank = no)" optional
|
||||
ask UPDATE_POLICY "Auto-update policy: latest | security | off (blank = latest)" optional ;;
|
||||
esac
|
||||
|
||||
@@ -40,6 +40,33 @@ ERGO_GID=
|
||||
# so the host firewall matches.
|
||||
PLAINTEXT=0
|
||||
|
||||
# --- Message history ----------------------------------------------------------
|
||||
# Ergo keeps history in RAM by default, so it is lost on every restart --
|
||||
# including the ones the updater performs. Persist it with:
|
||||
# sqlite (default) a file beside ircd.db; no extra container
|
||||
# postgres a pinned PostgreSQL container (docker-compose.postgres.yml)
|
||||
# off RAM only, upstream behaviour
|
||||
# Needs Ergo >= 2.18.0. How long messages are kept is governed by
|
||||
# history.restrictions.expire-time in ircd.yaml (1 week by default), NOT here.
|
||||
# Chosen at the first deploy; switch later with 'ergoctl history <backend>'
|
||||
# (which does not migrate existing messages).
|
||||
HISTORY=sqlite
|
||||
|
||||
# Set by deploy.sh so every `docker compose` command in this stack sees the same
|
||||
# services. It gains ':docker-compose.postgres.yml' when HISTORY=postgres.
|
||||
COMPOSE_FILE=docker-compose.yml
|
||||
|
||||
# --- PostgreSQL (only when HISTORY=postgres) ----------------------------------
|
||||
# The MAJOR version is pinned on purpose: PostgreSQL refuses to start on a data
|
||||
# directory written by a different major, so update.sh never touches this image.
|
||||
# Changing it is a deliberate dump-and-restore -- see the README.
|
||||
POSTGRES_TAG=17-alpine
|
||||
POSTGRES_USER=ergo
|
||||
POSTGRES_DB=ergo_history
|
||||
# Published on 127.0.0.1 only. The password lives in secrets/postgres.pass
|
||||
# (0600), never here -- compose interpolates .env and a dollar sign would break it.
|
||||
POSTGRES_PORT=5432
|
||||
|
||||
# --- Updates (update.sh) ------------------------------------------------------
|
||||
# This file is where the scheduled jobs read these four knobs from; edit them
|
||||
# here. (An explicit environment variable, or a value pinned in
|
||||
|
||||
+95
-10
@@ -47,6 +47,7 @@ the host's loopback), and there is no `ports:` section — host mode.
|
||||
| `NETWORK_NAME` | IRC network name (letters, digits, `. _ -`; no spaces). Defaults to `ERGO_DOMAIN`. |
|
||||
| `ERGO_TAG` | Pinned image tag **with the `v`** (`v2.19.1`). Set by deploy.sh to the newest release; bumped by update.sh. Never `stable`/`latest` — if a first deploy could not reach GitHub it falls back to `stable`, and the next update run pins it. |
|
||||
| `PLAINTEXT` | `0` (default) or `1` — public plaintext 6667. **Fixed at the first deploy** (it selects a listener in `ircd.yaml`, which deploy.sh never rewrites); see below to change it. |
|
||||
| `HISTORY` | `sqlite` (default) / `postgres` / `off` — see [Message history](#message-history). |
|
||||
| `UPDATE_POLICY` | `latest` (default) / `security` / `off` — see [Updates](#updates). |
|
||||
|
||||
See [`.env.example`](.env.example) for the rest (`UPDATE_GRACE`,
|
||||
@@ -126,10 +127,11 @@ listener as the `admin` oper.
|
||||
| `ergoctl passwd [show\|rotate]` | the `admin` oper password |
|
||||
| `ergoctl announce <text>` | NOTICE to everyone |
|
||||
| `ergoctl kill <nick> [reason]` · `ban add\|del\|list\|info` · `defcon [1-5]` | moderation (`UBAN` under the hood). `ban add` takes an **IP, CIDR, `nick!user@host` mask, or account name** — a bare name is an *account* to suspend, not a connected nick, so run `ban info <nick>` first to get their IP |
|
||||
| `ergoctl cmd <raw IRC line>` | anything else as the admin oper, e.g. `ergoctl cmd NS SAREGISTER alice hunter2`, `ergoctl cmd CS PURGE #spam`; replies are printed with control characters stripped |
|
||||
| `ergoctl cmd <raw IRC line>` | anything else as the admin oper, e.g. `ergoctl cmd NS SAREGISTER alice hunter2`, `ergoctl cmd CS PURGE ADD #spam` (`CS PURGE DEL` to undo); replies are printed with control characters stripped |
|
||||
| `ergoctl cert [show\|sync]` | cert on disk vs the one served on 6697; force a sync |
|
||||
| `ergoctl backup [--live] [dir]` / `restore <file>` | see [Backups](#backups) |
|
||||
| `ergoctl update …` | passthrough to `update.sh` |
|
||||
| `ergoctl history [backend]` | show or switch the persistent-history backend |
|
||||
| `ergoctl debug on\|off` · `caddy-restart` · `shell` | log level; apply Caddyfile/conf.d changes; a shell in the container |
|
||||
|
||||
Editing by hand works too: change `ircd/ircd.yaml`, then `ergoctl rehash` — a
|
||||
@@ -160,10 +162,65 @@ Silent when nothing changed. The daily run also warns when the served cert is
|
||||
still self-signed (DNS/80/443 not right yet) or expires within 14 days
|
||||
(certsync not running). `ergoctl cert` shows both sides.
|
||||
|
||||
## Message history
|
||||
|
||||
Ergo keeps channel and DM history for `CHATHISTORY`, `/HISTORY`, autoreplay-on-join
|
||||
and always-on (bouncer) clients. **Upstream keeps it in RAM**, so it is lost on
|
||||
every restart — including the ones this stack's updater performs. So persistence
|
||||
is on by default here:
|
||||
|
||||
| `HISTORY` | What it does |
|
||||
|---|---|
|
||||
| `sqlite` *(default)* | A file at `ircd/ergo_history.db`, next to the account database. No extra container, backed up with everything else. Right for a single-node server. |
|
||||
| `postgres` | A pinned PostgreSQL container (`docker-compose.postgres.yml`), reachable only on `127.0.0.1`. Choose it if you want a real database to query, or already run Postgres. |
|
||||
| `off` | Upstream behaviour: RAM only. |
|
||||
|
||||
Both SQL backends need **Ergo 2.18.0+** (deploy.sh refuses older pins). How long
|
||||
messages are kept is `history.restrictions.expire-time` in `ircd.yaml` — **one
|
||||
week by default** — not a size limit: enabling persistence removes the in-memory
|
||||
`channel-length`/`client-length` caps. Raise or lower it with `ergoctl edit`, and
|
||||
note the privacy point upstream flags: persisted messages are personal data, so
|
||||
check what your jurisdiction expects of you before extending retention.
|
||||
|
||||
```bash
|
||||
ergoctl history # backend, database size / connectivity, retention
|
||||
ergoctl history postgres # switch (restarts Ergo; does NOT migrate messages)
|
||||
```
|
||||
|
||||
Switching backends leaves the old store on disk and starts the new one empty —
|
||||
there is no migration path, so pick one at deploy time if you can.
|
||||
|
||||
### PostgreSQL
|
||||
|
||||
`HISTORY=postgres` adds `docker-compose.postgres.yml` to `COMPOSE_FILE` in `.env`,
|
||||
which `docker compose` reads by itself — so every command in this stack sees the
|
||||
same services with no extra flags. The container publishes **only to
|
||||
`127.0.0.1:5432`**, Ergo reaches it there (the host-networked containers cannot
|
||||
use compose service DNS), and `depends_on: service_healthy` keeps Ergo from
|
||||
starting before the database accepts connections. The password is generated into
|
||||
`secrets/postgres.pass` (0600) and passed via `POSTGRES_PASSWORD_FILE`, so it
|
||||
never lands in `.env`.
|
||||
|
||||
**The major version is pinned and the updater never touches it.** PostgreSQL
|
||||
refuses to start on a data directory written by a different major version, so
|
||||
moving from `17-alpine` to `18-alpine` is a deliberate dump-and-restore:
|
||||
|
||||
```bash
|
||||
ergoctl backup # contains a pg_dump of the history database
|
||||
ergoctl history off && docker compose down -v postgres # drops the old data volume
|
||||
# set POSTGRES_TAG=18-alpine in .env, then:
|
||||
ergoctl history postgres && ergoctl restore backups/ergo-backup-<ts>.tar.gz
|
||||
```
|
||||
|
||||
Note that Postgres 18 also changed the image's default data directory layout; the
|
||||
compose file pins `PGDATA` explicitly so a future image default cannot move it.
|
||||
|
||||
## Updates
|
||||
|
||||
```bash
|
||||
ergoctl update check # running vs latest, published security advisories, cert state; changes nothing
|
||||
ergoctl update check # running vs latest, published security advisories, cert state
|
||||
# reports only, with one exception: a floating ERGO_TAG
|
||||
# ('stable') is pinned to the running version in .env
|
||||
ergoctl update update # update now (TARGET_VERSION=2.19.1 to pin); honours the safety rails
|
||||
ergoctl update install # (re)schedule / uninstall to stop
|
||||
```
|
||||
@@ -203,16 +260,27 @@ web-client websockets. Update results reuse the ntfy config at
|
||||
## Backups
|
||||
|
||||
`ergoctl backup` writes `backups/ergo-backup-<ts>.tar.gz` (0600) with
|
||||
`ircd.yaml`, `ircd.db`, `ergo.motd` and a `meta` file (version, date). By default
|
||||
`ircd.yaml`, `ircd.db`, `ergo.motd`, the message history (the SQLite file, or a
|
||||
`pg_dump` — a file copy of a live PostgreSQL data directory would not be a valid
|
||||
backup) and a `meta` file (version, history backend, date). By default
|
||||
it stops Ergo for a few seconds for a consistent copy (`--live` skips that).
|
||||
Put an age public key in `$STACK_DIR/age-recipients.txt` (e.g. from
|
||||
`globals/age-pubkey.txt`) **and install `age`** (`apk add age`, `apt install age`,
|
||||
`dnf install age` — deploy.sh does not) and backups are encrypted with it; with
|
||||
the recipients file present but `age` missing, `ergoctl backup` refuses rather
|
||||
than writing the account and oper hashes out in the clear. TLS files are not
|
||||
included — certsync regenerates them. `ergoctl restore <file>` stops Ergo,
|
||||
warns if the backup came from a newer version (older Ergo cannot read a newer
|
||||
schema), restores, and waits for health. Ergo's own `datastore.autoupgrade`
|
||||
included — certsync regenerates them. `ergoctl restore <file>` stops Ergo, keeps the current config and database as
|
||||
`backups/{ircd.yaml,ircd.db}.pre-restore.<ts>`, warns if the backup came from a
|
||||
newer version (older Ergo cannot read a newer schema), restores, validates the
|
||||
result before starting, and waits for health. Because the archive carries the
|
||||
*original* host's credentials, restore re-points `datastore.postgresql.password`
|
||||
at this host's `secrets/postgres.pass` and re-hashes this host's
|
||||
`secrets/admin.pass` into the config — otherwise a rebuild would come up healthy
|
||||
but leave you unable to `/OPER`. An encrypted backup needs the private key:
|
||||
|
||||
```bash
|
||||
AGE_IDENTITY=/root/age.key ergoctl restore backups/ergo-backup-<ts>.tar.gz.age
|
||||
``` Ergo's own `datastore.autoupgrade`
|
||||
leaves `ircd/ircd.db.v<N>.<ts>.bak` files behind on schema upgrades; prune them
|
||||
once you are happy with a release.
|
||||
|
||||
@@ -240,6 +308,12 @@ file_server }` block into `caddy/etc/conf.d/` (see `00-readme.caddy`), then
|
||||
filesystem, all capabilities dropped, `no-new-privileges`. `ircd/` (database,
|
||||
config with the oper hash, TLS key) is `0700 ergo:ergo`; `secrets/`,
|
||||
`backups/`, `caddy/data` are `0700 root`.
|
||||
- **Root never follows a symlink into `ircd/`.** That directory is writable by
|
||||
the container uid while the cert sync and every `ergoctl` config edit run as
|
||||
root, so a plain `cp`/`install`/`>` there would let code execution inside Ergo
|
||||
redirect a root write onto any file on the host. Every such write stages under
|
||||
`$STACK_DIR` (0700 root) and lands with `mv` — `rename(2)` replaces a symlink
|
||||
instead of following it — and root reads use `cp -P` or refuse outright.
|
||||
- **Caddy runs as root** (it needs `NET_BIND_SERVICE` for 80/443) with every
|
||||
other capability dropped, admin API off, HTTP/3 off.
|
||||
- **Loopback plaintext is the control plane by design**: OPER over loopback is
|
||||
@@ -255,6 +329,7 @@ file_server }` block into `caddy/etc/conf.d/` (see `00-readme.caddy`), then
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `docker-compose.yml` | caddy + ergo, both `network_mode: host`, hardened; IRC-level healthcheck. |
|
||||
| `docker-compose.postgres.yml` | Optional PostgreSQL overlay for `HISTORY=postgres` (loopback-only, pinned major). |
|
||||
| `Caddyfile` | ACME, `/webirc` websocket proxy, text landing page, `admin off`, no h3; imports `conf.d/*.caddy`. |
|
||||
| `conf.d-readme.caddy` | Installed once as `caddy/etc/conf.d/00-readme.caddy` (operator drop-ins; Gamja example). |
|
||||
| `ergo.motd` | MOTD template (`__NETWORK_NAME__`, `__DOMAIN__`). |
|
||||
@@ -268,8 +343,10 @@ file_server }` block into `caddy/etc/conf.d/` (see `00-readme.caddy`), then
|
||||
On the host: `/srv/ergo/{ircd,caddy/{etc,data,config},secrets,backups,templates}`
|
||||
plus `.state/` (notification de-duplication) and `.env.bak.<ts>` copies from each
|
||||
update (last 5 kept); `/etc/ergo-update.conf` (where the jobs find the stack);
|
||||
`/usr/local/bin/ergoctl`; `/var/log/ergo-update.log` (scheduled runs log there,
|
||||
since busybox crond has nowhere else to put their output).
|
||||
`/usr/local/bin/ergoctl`; `/var/log/ergo-update.log` (the **daily update run**
|
||||
logs there, since busybox crond has nowhere else to put its output — cert-sync
|
||||
failures instead surface through ntfy and as `certsync: LAST FAILURE` in
|
||||
`ergoctl status`).
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -277,5 +354,13 @@ since busybox crond has nowhere else to put their output).
|
||||
another Caddy/80/443 stack (deploy.sh checks the ports).
|
||||
- DNS for `ERGO_DOMAIN` must resolve to the host and 80/443 be reachable before
|
||||
deploy for the cert to issue; Ergo stays up on a self-signed cert meanwhile.
|
||||
- Ergo's built-in registration is open by default (`accounts.registration.enabled`);
|
||||
edit `ircd.yaml` if you want an invite-only network.
|
||||
- Ergo's built-in registration is open by default, and **`allow-before-connect`
|
||||
is also on**, so accounts can be created by a client that has not finished
|
||||
connecting, throttled only globally (30 attempts per 10 minutes ≈ 4,300/day).
|
||||
Each account is a durable row in `ircd.db`. For a private network set
|
||||
`accounts.registration.enabled: false`, or keep registration but set
|
||||
`allow-before-connect: false`, with `ergoctl edit`.
|
||||
- The admin oper password is printed by `deploy.sh` only on the first run and
|
||||
only to a terminal. Unattended runs (cloud-init) print the path instead, so the
|
||||
credential does not end up in the provider's serial-console log; read it with
|
||||
`ergoctl passwd show`.
|
||||
|
||||
@@ -17,6 +17,7 @@ MARKER="__ARCHIVE_BELOW__"
|
||||
# generated on the host from the pulled image's own default config.
|
||||
FILES=(
|
||||
docker-compose.yml
|
||||
docker-compose.postgres.yml
|
||||
Caddyfile
|
||||
conf.d-readme.caddy
|
||||
ergo.motd
|
||||
@@ -31,6 +32,18 @@ for f in "${FILES[@]}"; do
|
||||
[[ -f "$DIR/$f" ]] || { echo "Missing $DIR/$f" >&2; exit 1; }
|
||||
done
|
||||
|
||||
# Guard: every file deploy.sh reads out of the extracted archive must be in
|
||||
# FILES, and must be in deploy.sh's own EMBEDDED manifest. Forgetting either is
|
||||
# a deploy-time abort on every host, so catch it here instead.
|
||||
missing=0
|
||||
while IFS= read -r ref; do
|
||||
case " ${FILES[*]} " in *" $ref "*) ;; *) echo "deploy.sh reads \$SCRIPT_DIR/$ref but build.sh does not embed it" >&2; missing=1 ;; esac
|
||||
done < <(grep -oE '\$SCRIPT_DIR/[A-Za-z0-9._-]+' "$SCRIPT" | sed 's|^\$SCRIPT_DIR/||' | sort -u)
|
||||
for f in "${FILES[@]}"; do
|
||||
grep -q "EMBEDDED=(.*$f" "$SCRIPT" || { echo "$f is embedded but missing from deploy.sh's EMBEDDED manifest" >&2; missing=1; }
|
||||
done
|
||||
(( missing == 0 )) || { echo "Refusing to build a deploy.sh that would abort on the target host." >&2; exit 1; }
|
||||
|
||||
# Syntax-check the bash payload before embedding it.
|
||||
for f in ergolib.sh update.sh ergoctl deploy.sh; do
|
||||
bash -n "$DIR/$f" || { echo "bash -n failed: $f" >&2; exit 1; }
|
||||
|
||||
@@ -36,6 +36,7 @@ runcmd:
|
||||
ERGO_DOMAIN=irc.example.com \
|
||||
ACME_EMAIL=admin@example.com \
|
||||
NETWORK_NAME=ExampleNet \
|
||||
HISTORY=sqlite \
|
||||
PLAINTEXT=0 \
|
||||
UPDATE_POLICY=latest \
|
||||
SKIP_PROMPTS=1 \
|
||||
|
||||
+766
-489
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,64 @@
|
||||
# PostgreSQL overlay -- used ONLY when HISTORY=postgres.
|
||||
#
|
||||
# deploy.sh activates it by writing
|
||||
# COMPOSE_FILE=docker-compose.yml:docker-compose.postgres.yml
|
||||
# into .env, which docker compose reads by itself. So every `docker compose`
|
||||
# command in this stack (up, ps, stop, logs, exec) sees the same set of services
|
||||
# with no extra flags -- unlike compose profiles, where a service can be silently
|
||||
# absent from one command and present in another.
|
||||
#
|
||||
# Networking: ergo and caddy run in the HOST namespace, so they cannot use
|
||||
# compose's service DNS. PostgreSQL therefore stays on a normal bridge network
|
||||
# and publishes only to the host's loopback; Ergo reaches it at 127.0.0.1:5432
|
||||
# (datastore.postgresql.host in ircd.yaml). Nothing is exposed off-box.
|
||||
#
|
||||
# The major version is PINNED. PostgreSQL will not start on a data directory
|
||||
# written by a different major version, so `update.sh` never touches this image.
|
||||
# Upgrading it is a deliberate dump-and-restore -- see the README ("PostgreSQL").
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:${POSTGRES_TAG:-17-alpine}
|
||||
container_name: ergo-postgres
|
||||
restart: unless-stopped
|
||||
# Loopback only. A Docker-published port bypasses the host INPUT firewall,
|
||||
# so the bind address is the real restriction here.
|
||||
ports:
|
||||
- "127.0.0.1:${POSTGRES_PORT:-5432}:5432"
|
||||
environment:
|
||||
POSTGRES_USER: "${POSTGRES_USER:-ergo}"
|
||||
POSTGRES_DB: "${POSTGRES_DB:-ergo_history}"
|
||||
# The password is read from a file so it never has to live in .env
|
||||
# (compose interpolates .env, and a '$' in a password would break it).
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres-password
|
||||
# 17 and below keep the classic layout; set it explicitly so a future
|
||||
# image default cannot move the data directory under us.
|
||||
PGDATA: /var/lib/postgresql/data/pgdata
|
||||
volumes:
|
||||
- postgres-data:/var/lib/postgresql/data
|
||||
- ./secrets/postgres.pass:/run/secrets/postgres-password:ro
|
||||
cap_drop: [ALL]
|
||||
cap_add:
|
||||
- CHOWN # initdb/entrypoint fix ownership of PGDATA
|
||||
- DAC_READ_SEARCH
|
||||
- FOWNER
|
||||
- SETGID # the entrypoint drops from root to the postgres user
|
||||
- SETUID
|
||||
security_opt: [no-new-privileges:true]
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U \"${POSTGRES_USER:-ergo}\" -d \"${POSTGRES_DB:-ergo_history}\" -q"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
|
||||
# Ergo must not come up before the database is accepting connections: with
|
||||
# persistent history enabled it fails to start if the backend is unreachable.
|
||||
# depends_on is orchestration only, so it works across the host/bridge split.
|
||||
ergo:
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
|
||||
volumes:
|
||||
postgres-data:
|
||||
+216
-12
@@ -44,6 +44,10 @@ Configuration (validated, then REHASH; reverted on failure)
|
||||
restart | stop | start the ergo container (restart drops every user)
|
||||
caddy-restart apply Caddyfile / conf.d changes
|
||||
debug on|off switch the log level (info <-> debug)
|
||||
history [sqlite|postgres|off]
|
||||
show, or switch, the persistent-history backend.
|
||||
Switching does NOT migrate existing messages and
|
||||
needs a restart (Ergo fixes the backend at startup).
|
||||
|
||||
Operators
|
||||
oper list
|
||||
@@ -64,10 +68,14 @@ Moderation (as the admin oper)
|
||||
(replies are printed; control characters stripped)
|
||||
|
||||
Data
|
||||
backup [--live] [dir] tar.gz of ircd.yaml, ircd.db, ergo.motd (default: stop Ergo
|
||||
briefly for a consistent copy; --live = crash-consistent).
|
||||
backup [--live] [dir] tar.gz of ircd.yaml, ircd.db, ergo.motd and the message
|
||||
history (SQLite file, or a pg_dump). Stops Ergo briefly
|
||||
for a consistent copy; --live = crash-consistent.
|
||||
Encrypted with age when $STACK_DIR/age-recipients.txt exists.
|
||||
restore <file> restore a backup (stops Ergo, restarts, waits for health)
|
||||
restore <file> restore a backup: stops Ergo, keeps the current config and
|
||||
database as backups/*.pre-restore.<ts>, reconciles this
|
||||
host's credentials, validates, then restarts.
|
||||
Encrypted backup: AGE_IDENTITY=/path/to/age.key ergoctl restore <file>.age
|
||||
|
||||
Updates (update.sh)
|
||||
update [check|update|run|certsync|caddy|install|uninstall]
|
||||
@@ -102,6 +110,30 @@ apply_change() {
|
||||
for f in "$@"; do rm -f "$f.last-good"; done
|
||||
return 0
|
||||
fi
|
||||
# Ergo opens the history backend ONLY at startup. Unlike MySQL, sqlite and
|
||||
# postgresql have no "after launching the server" guard, so a rehash that
|
||||
# turns one on reports success and then silently discards every message.
|
||||
# Catch that here rather than letting the operator believe it worked.
|
||||
for f in "$@"; do
|
||||
[[ "$f" == "${CONF:-}" && -f "$f.last-good" ]] || continue
|
||||
if [[ "$(history_backend "$f")" != "$(history_backend "$f.last-good")" ]]; then
|
||||
warn "This change switches the message-history backend, which Ergo only reads at startup."
|
||||
warn "A rehash would report success and silently do nothing, so a restart is required."
|
||||
if [[ "${ERGOCTL_ASSUME_YES:-0}" != 1 ]]; then
|
||||
read -r -p "Restart Ergo now (disconnects every user)? [y/N] " ans
|
||||
[[ "${ans,,}" == y* ]] || { cat "$f.last-good" > "$f"; die "Reverted; nothing changed. Use 'ergoctl history <backend>' to switch it properly."; }
|
||||
fi
|
||||
if ! validate_or_fail "$what"; then cat "$f.last-good" > "$f"; die "$what rejected; previous version restored."; fi
|
||||
irc_raw --oper --quiet 1 -- 'NOTICE $$* :Server restarting to change message-history storage.' >/dev/null 2>&1 || true
|
||||
dc up -d --no-deps ergo >/dev/null 2>&1 || dc restart ergo >/dev/null 2>&1 || true
|
||||
if wait_healthy ergo 120 && irc_probe; then
|
||||
rm -f "$f.last-good"; log "$what applied (Ergo restarted)."; return 0
|
||||
fi
|
||||
dc logs --tail 30 ergo >&2 || true
|
||||
cat "$f.last-good" > "$f"
|
||||
die "Ergo is not healthy after the restart; config reverted. Run 'ergoctl restart'."
|
||||
fi
|
||||
done
|
||||
# Take the status from the call itself: `$?` after an `if` whose condition
|
||||
# failed and that has no else branch is 0, which would make the SIGHUP
|
||||
# fallback below unreachable.
|
||||
@@ -156,6 +188,12 @@ cmd_status() {
|
||||
latest="$(normver "$(ergo_latest_tag || true)")"
|
||||
printf 'Ergo: %s | running %s | pinned %s | latest %s\n' "${st:-not created}" "${cur:-?}" "$(ergo_tag)" "${latest:-?}"
|
||||
if [[ "$(env_get PLAINTEXT)" == "1" ]]; then echo " PUBLIC PLAINTEXT LISTENER on :6667 (PLAINTEXT=1)"; fi
|
||||
local hb; hb="$(history_backend)"
|
||||
case "$hb" in
|
||||
postgres) pg_check || true; printf 'History: persistent, PostgreSQL -- %s\n' "$PG_CHECK_MSG" ;;
|
||||
sqlite) printf 'History: persistent, SQLite (%s)\n' "$IRCD_DIR/$(yaml_get_nested "$CONF" datastore sqlite database-path)" ;;
|
||||
*) printf 'History: RAM only -- lost on every restart\n' ;;
|
||||
esac
|
||||
cmd_cert show
|
||||
if ergo_running; then
|
||||
echo; cmd_users || true
|
||||
@@ -278,7 +316,11 @@ cmd_backup() {
|
||||
require_root; require_stack
|
||||
local live=0 dest="$BACKUP_DIR" ts stage out was_running=0 users=""
|
||||
while [[ $# -gt 0 ]]; do case "$1" in --live) live=1 ;; *) dest="$1" ;; esac; shift; done
|
||||
install -d -m 0700 "$dest"
|
||||
# `install -d -m` also re-modes an EXISTING directory, so only create-and-mode
|
||||
# our own default; never change the mode of a directory the operator named
|
||||
# (`ergoctl backup /var/backups` must not make it 0700 root).
|
||||
if [[ "$dest" == "$BACKUP_DIR" ]]; then install -d -m 0700 "$dest"
|
||||
else [[ -d "$dest" ]] || install -d -m 0700 "$dest"; fi
|
||||
# A recipients file means "encrypt these" -- refuse rather than silently
|
||||
# writing account hashes and the oper hash out in the clear.
|
||||
if [[ -f "$STACK_DIR/age-recipients.txt" ]] && ! command -v age >/dev/null 2>&1; then
|
||||
@@ -297,12 +339,37 @@ cmd_backup() {
|
||||
warn "Live backup: ircd.db is copied while Ergo writes to it (crash-consistent; may miss the last second)."
|
||||
fi
|
||||
fi
|
||||
cp -p "$CONF" "$stage/ircd.yaml"
|
||||
[[ -f "$IRCD_DIR/ircd.db" ]] && cp -p "$IRCD_DIR/ircd.db" "$stage/ircd.db"
|
||||
[[ -f "$MOTD" ]] && cp -p "$MOTD" "$stage/ergo.motd"
|
||||
# -P everywhere: ircd/ is writable by the container uid, so a symlink there
|
||||
# must be copied as a link, never followed by this root process.
|
||||
refuse_symlink "$CONF" "ircd/ircd.yaml" || die "Refusing to back up through a symlink."
|
||||
cp -Pp "$CONF" "$stage/ircd.yaml"
|
||||
[[ -f "$IRCD_DIR/ircd.db" ]] && cp -Pp "$IRCD_DIR/ircd.db" "$stage/ircd.db"
|
||||
[[ -f "$MOTD" ]] && cp -Pp "$MOTD" "$stage/ergo.motd"
|
||||
# Message history, whichever backend holds it.
|
||||
local backend hf
|
||||
backend="$(history_backend)"
|
||||
case "$backend" in
|
||||
sqlite)
|
||||
hf="$IRCD_DIR/$(yaml_get_nested "$CONF" datastore sqlite database-path)"
|
||||
# SQLite writes -wal/-shm siblings; with Ergo stopped they are already
|
||||
# checkpointed, and on a --live backup we take them along.
|
||||
for f in "$hf" "$hf-wal" "$hf-shm"; do
|
||||
[[ -f "$f" ]] && cp -Pp "$f" "$stage/$(basename "$f")"
|
||||
done ;;
|
||||
postgres)
|
||||
if pg_check; then
|
||||
log "Dumping the PostgreSQL history database..."
|
||||
# A file copy of a live PGDATA is not a valid backup; pg_dump is.
|
||||
dc exec -T postgres pg_dump -U "$(env_get POSTGRES_USER)" -d "$(env_get POSTGRES_DB)" \
|
||||
> "$stage/history.sql" 2>/dev/null || { rm -f "$stage/history.sql"; warn "pg_dump failed; the backup will NOT contain message history."; }
|
||||
else
|
||||
warn "PostgreSQL is not reachable ($PG_CHECK_MSG); the backup will NOT contain message history."
|
||||
fi ;;
|
||||
esac
|
||||
cat > "$stage/meta" <<EOF
|
||||
ERGO_TAG=$(ergo_tag)
|
||||
ERGO_DOMAIN=$(ergo_domain)
|
||||
HISTORY=$backend
|
||||
DATE=$(date -u +%FT%TZ)
|
||||
HOST=$(hostname -f 2>/dev/null || hostname)
|
||||
CONSISTENT=$(( live == 0 ))
|
||||
@@ -340,13 +407,149 @@ cmd_restore() {
|
||||
log "Stopping Ergo and restoring from $file..."
|
||||
dc stop -t 15 ergo >/dev/null 2>&1 || true
|
||||
install -d -m 0700 "$BACKUP_DIR"
|
||||
[[ -f "$IRCD_DIR/ircd.db" ]] && cp -p "$IRCD_DIR/ircd.db" "$BACKUP_DIR/ircd.db.pre-restore.$(date -u +%Y%m%d%H%M%S)"
|
||||
install -m 0600 -o "$(ergo_uid)" -g "$(ergo_gid)" "$stage/ircd.yaml" "$CONF"
|
||||
install -m 0600 -o "$(ergo_uid)" -g "$(ergo_gid)" "$stage/ircd.db" "$IRCD_DIR/ircd.db"
|
||||
[[ -f "$stage/ergo.motd" ]] && install -m 0644 -o "$(ergo_uid)" -g "$(ergo_gid)" "$stage/ergo.motd" "$MOTD"
|
||||
# Keep the CURRENT config and database, not just the database: if the restore
|
||||
# turns out to be unloadable there has to be a way back.
|
||||
local ts uid gid
|
||||
ts="$(date -u +%Y%m%d%H%M%S)"; uid="$(ergo_uid)"; gid="$(ergo_gid)"
|
||||
[[ -f "$CONF" ]] && cp -Pp "$CONF" "$BACKUP_DIR/ircd.yaml.pre-restore.$ts"
|
||||
[[ -f "$IRCD_DIR/ircd.db" ]] && cp -Pp "$IRCD_DIR/ircd.db" "$BACKUP_DIR/ircd.db.pre-restore.$ts"
|
||||
safe_replace "$stage/ircd.yaml" "$CONF" 0600 "$uid" "$gid"
|
||||
safe_replace "$stage/ircd.db" "$IRCD_DIR/ircd.db" 0600 "$uid" "$gid"
|
||||
[[ -f "$stage/ergo.motd" ]] && safe_replace "$stage/ergo.motd" "$MOTD" 0644 "$uid" "$gid"
|
||||
|
||||
# The backup carries the ORIGINAL host's credentials. Reconcile them with
|
||||
# this host's, or the restored server is unreachable and unadministrable.
|
||||
if [[ "$(history_backend)" == "postgres" && -f "$SECRETS_DIR/postgres.pass" ]]; then
|
||||
log "Re-pointing datastore.postgresql at this host's database password..."
|
||||
yaml_set_nested "$CONF" datastore postgresql password "\"$(cat "$SECRETS_DIR/postgres.pass")\"" \
|
||||
|| warn "Could not update the postgresql password; Ergo may fail to reach its history database."
|
||||
fi
|
||||
if [[ -f "$ADMIN_PASS_FILE" ]]; then
|
||||
local h
|
||||
if h="$(ergo_genpasswd "$(ergo_image)" "$(head -n1 "$ADMIN_PASS_FILE")")"; then
|
||||
yaml_oper_set_password "$CONF" admin "$h" \
|
||||
&& log "Re-hashed this host's admin password into the restored config (ergoctl keeps working)." \
|
||||
|| warn "Could not update the admin oper hash; 'ergoctl passwd rotate' will fix it."
|
||||
else
|
||||
warn "Could not hash the local admin password; the restored config keeps the backup's admin hash. If OPER fails, run 'ergoctl passwd rotate'."
|
||||
fi
|
||||
fi
|
||||
# Message history
|
||||
local f
|
||||
for f in "$stage"/ergo_history.db*; do
|
||||
[[ -f "$f" ]] || continue
|
||||
install -m 0600 -o "$(ergo_uid)" -g "$(ergo_gid)" "$f" "$IRCD_DIR/$(basename "$f")"
|
||||
done
|
||||
if [[ -f "$stage/history.sql" ]]; then
|
||||
log "Starting PostgreSQL and restoring the history dump..."
|
||||
dc up -d --no-deps postgres >/dev/null 2>&1 || true
|
||||
if wait_healthy postgres 120; then
|
||||
dc exec -T postgres psql -U "$(env_get POSTGRES_USER)" -d "$(env_get POSTGRES_DB)" < "$stage/history.sql" >/dev/null 2>&1 \
|
||||
|| warn "Restoring the history dump failed; Ergo will start with an empty history."
|
||||
else
|
||||
warn "PostgreSQL did not become healthy; message history was NOT restored."
|
||||
fi
|
||||
fi
|
||||
rm -rf "$stage"
|
||||
# Validate before starting, so a bad backup does not cost an outage.
|
||||
if ! validate_or_fail "the restored config"; then
|
||||
[[ -f "$BACKUP_DIR/ircd.yaml.pre-restore.$ts" ]] && cp -p "$BACKUP_DIR/ircd.yaml.pre-restore.$ts" "$CONF"
|
||||
[[ -f "$BACKUP_DIR/ircd.db.pre-restore.$ts" ]] && cp -p "$BACKUP_DIR/ircd.db.pre-restore.$ts" "$IRCD_DIR/ircd.db"
|
||||
dc up -d --no-deps ergo >/dev/null 2>&1 || true
|
||||
die "The backup's config does not load; rolled back to what was here before (copies kept as $BACKUP_DIR/*.pre-restore.$ts)."
|
||||
fi
|
||||
dc up -d --no-deps ergo >/dev/null 2>&1 || true
|
||||
if wait_healthy ergo 120 && irc_probe; then log "Restored; Ergo is healthy."; else dc logs --tail 20 ergo >&2 || true; die "Ergo is not healthy after the restore."; fi
|
||||
if wait_healthy ergo 120 && irc_probe; then
|
||||
log "Restored; Ergo is healthy."
|
||||
log "Previous state kept at $BACKUP_DIR/{ircd.yaml,ircd.db}.pre-restore.$ts"
|
||||
else
|
||||
dc logs --tail 20 ergo >&2 || true
|
||||
die "Ergo is not healthy after the restore. Roll back with: cp $BACKUP_DIR/ircd.yaml.pre-restore.$ts $CONF && cp $BACKUP_DIR/ircd.db.pre-restore.$ts $IRCD_DIR/ircd.db && ergoctl start"
|
||||
fi
|
||||
}
|
||||
|
||||
cmd_history() {
|
||||
require_stack
|
||||
local want="${1:-}" cur pw
|
||||
cur="$(history_backend)"
|
||||
if [[ -z "$want" ]]; then
|
||||
echo "History backend: $cur"
|
||||
case "$cur" in
|
||||
sqlite) local f="$IRCD_DIR/$(yaml_get_nested "$CONF" datastore sqlite database-path)"
|
||||
[[ -f "$f" ]] && echo " $f ($(wc -c < "$f") bytes)" || echo " (database not created yet)" ;;
|
||||
postgres) pg_check && echo " $PG_CHECK_MSG" || echo " $PG_CHECK_MSG" ;;
|
||||
off) echo " messages are kept in RAM only and lost on every restart" ;;
|
||||
esac
|
||||
echo " retention: history.restrictions.expire-time = $(yaml_get_nested "$CONF" history restrictions expire-time)"
|
||||
return 0
|
||||
fi
|
||||
require_root
|
||||
case "$want" in sqlite|postgres|off) ;; *) die "Usage: ergoctl history [sqlite|postgres|off]" ;; esac
|
||||
[[ "$want" != "$cur" ]] || { log "History is already using '$cur'."; return 0; }
|
||||
# .env and ircd.yaml must never disagree: deploy.sh reconciles them from .env
|
||||
# on its next run, so a half-applied switch would let it tear down the wrong
|
||||
# thing. Remember both and put them back on every failure path.
|
||||
local prev_hist prev_compose
|
||||
prev_hist="$(env_get HISTORY)"; prev_compose="$(env_get COMPOSE_FILE)"
|
||||
_hist_revert() {
|
||||
[[ -f "$CONF.last-good" ]] && cat "$CONF.last-good" > "$CONF"
|
||||
[[ -n "$prev_hist" ]] && env_set HISTORY "$prev_hist"
|
||||
[[ -n "$prev_compose" ]] && env_set COMPOSE_FILE "$prev_compose"
|
||||
return 0
|
||||
}
|
||||
if [[ "$want" == "postgres" ]]; then
|
||||
[[ -f "$STACK_DIR/docker-compose.postgres.yml" ]] || die "docker-compose.postgres.yml is missing; re-run deploy.sh to install it."
|
||||
if [[ ! -f "$SECRETS_DIR/postgres.pass" ]]; then
|
||||
log "Generating the PostgreSQL password..."
|
||||
( umask 077; ergo_random_password 32 > "$SECRETS_DIR/postgres.pass" )
|
||||
fi
|
||||
chmod 0600 "$SECRETS_DIR/postgres.pass"; pw="$(cat "$SECRETS_DIR/postgres.pass")"
|
||||
[[ -n "$(env_get POSTGRES_TAG)" ]] || env_set POSTGRES_TAG 17-alpine
|
||||
[[ -n "$(env_get POSTGRES_USER)" ]] || env_set POSTGRES_USER ergo
|
||||
[[ -n "$(env_get POSTGRES_DB)" ]] || env_set POSTGRES_DB ergo_history
|
||||
[[ -n "$(env_get POSTGRES_PORT)" ]] || env_set POSTGRES_PORT 5432
|
||||
env_set COMPOSE_FILE "docker-compose.yml:docker-compose.postgres.yml"
|
||||
fi
|
||||
warn "Switching the history backend does NOT migrate existing messages; the old store is left in place."
|
||||
snapshot_file "$CONF"
|
||||
# Turn every backend off first, then enable the requested one.
|
||||
yaml_set_nested "$CONF" datastore sqlite enabled false || true
|
||||
yaml_set_nested "$CONF" datastore postgresql enabled false || true
|
||||
case "$want" in
|
||||
sqlite) yaml_set_nested "$CONF" datastore sqlite enabled true || { cat "$CONF.last-good" > "$CONF"; die "Could not enable datastore.sqlite."; }
|
||||
yaml_set_nested "$CONF" history persistent enabled true || true ;;
|
||||
postgres) yaml_set_nested "$CONF" datastore postgresql enabled true || { cat "$CONF.last-good" > "$CONF"; die "Could not enable datastore.postgresql."; }
|
||||
yaml_set_nested "$CONF" datastore postgresql host "\"127.0.0.1\"" || true
|
||||
yaml_set_nested "$CONF" datastore postgresql port "$(env_get POSTGRES_PORT)" || true
|
||||
yaml_set_nested "$CONF" datastore postgresql user "\"$(env_get POSTGRES_USER)\"" || true
|
||||
yaml_set_nested "$CONF" datastore postgresql password "\"${pw}\"" || true
|
||||
yaml_set_nested "$CONF" datastore postgresql history-database "\"$(env_get POSTGRES_DB)\"" || true
|
||||
yaml_set_nested "$CONF" history persistent enabled true || true
|
||||
log "Starting PostgreSQL..."
|
||||
dc up -d --no-deps postgres >/dev/null 2>&1 || true
|
||||
wait_healthy postgres 120 || { _hist_revert; die "PostgreSQL did not become healthy; config reverted."; } ;;
|
||||
off) yaml_set_nested "$CONF" history persistent enabled false || true ;;
|
||||
esac
|
||||
if ! validate_or_fail "history backend $want"; then _hist_revert; die "Config rejected; reverted."; fi
|
||||
log "Restarting Ergo (the history backend is fixed at startup; every user is disconnected)..."
|
||||
irc_raw --oper --quiet 1 -- 'NOTICE $$* :Server restarting to change message-history storage.' >/dev/null 2>&1 || true
|
||||
dc up -d --no-deps ergo >/dev/null 2>&1 || dc restart ergo >/dev/null 2>&1 || true
|
||||
if wait_healthy ergo 120 && irc_probe; then
|
||||
# Only now is the switch real, so only now does .env change.
|
||||
env_set HISTORY "$want"
|
||||
[[ "$want" == "postgres" ]] || env_set COMPOSE_FILE "docker-compose.yml"
|
||||
rm -f "$CONF.last-good"
|
||||
log "History backend is now '$want'."
|
||||
if [[ "$want" == "off" ]]; then
|
||||
warn "Existing persisted messages are still on disk; remove them yourself if that is the point."
|
||||
fi
|
||||
else
|
||||
dc logs --tail 30 ergo >&2 || true
|
||||
_hist_revert
|
||||
dc up -d --no-deps ergo >/dev/null 2>&1 || true
|
||||
die "Ergo is not healthy after the switch; config and .env reverted. Check 'ergoctl logs'."
|
||||
fi
|
||||
return 0 # never let a trailing false test become this command's exit status
|
||||
}
|
||||
|
||||
cmd_debug() {
|
||||
@@ -379,6 +582,7 @@ case "${1:-help}" in
|
||||
dc run --rm --no-deps -T caddy caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile >/dev/null || die "Caddyfile invalid (see caddy/etc/Caddyfile, conf.d/)."
|
||||
dc restart caddy; wait_healthy caddy 60 && log "Caddy is healthy." || die "Caddy did not become healthy." ;;
|
||||
debug) shift; cmd_debug "$@" ;;
|
||||
history) shift; cmd_history "$@" ;;
|
||||
oper) shift; cmd_oper "$@" ;;
|
||||
passwd) shift; cmd_passwd "$@" ;;
|
||||
cmd) shift; cmd_cmd "$@" ;;
|
||||
|
||||
+134
-13
@@ -79,6 +79,36 @@ fetch() {
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Writing into ircd/ safely
|
||||
#
|
||||
# ircd/ is owned by the unprivileged container uid, but several things that
|
||||
# write there run as ROOT (the 15-minute cert sync, every ergoctl config edit).
|
||||
# A plain `cp`, `install` or `>` follows a symlink at the destination, so code
|
||||
# execution inside Ergo could point one of those writes at any root-owned file
|
||||
# and win the host. rename(2) is the one operation that REPLACES a symlink
|
||||
# instead of following it, so every root write into ircd/ goes: create a temp
|
||||
# under $STACK_DIR (root-owned 0700, same filesystem) -> set mode/owner -> mv.
|
||||
# ---------------------------------------------------------------------------
|
||||
# A temp file that an unprivileged uid cannot reach or pre-empt.
|
||||
stage_tmp() { mktemp "$STACK_DIR/.stage.XXXXXX"; }
|
||||
|
||||
# safe_replace <tmpfile> <target> [mode] [uid] [gid]
|
||||
safe_replace() {
|
||||
local tmp="$1" target="$2" mode="${3:-0600}" uid="${4:-}" gid="${5:-}"
|
||||
chmod "$mode" "$tmp" 2>/dev/null || true
|
||||
[[ -n "$uid" ]] && chown "${uid}:${gid:-$uid}" "$tmp" 2>/dev/null
|
||||
mv -f "$tmp" "$target"
|
||||
}
|
||||
|
||||
# Refuse to read a path under ircd/ that the container uid has turned into a
|
||||
# symlink (a root reader would otherwise copy out whatever it points at).
|
||||
refuse_symlink() { # <path> [what]
|
||||
[[ -L "$1" ]] || return 0
|
||||
warn "${2:-$1} is a symlink; refusing to touch it as root (see README: Security model)."
|
||||
return 1
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# .env access
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -95,6 +125,26 @@ env_set() { # <KEY> <value>: update KEY in .env, or append if absent
|
||||
|
||||
ergo_domain() { env_get ERGO_DOMAIN | tr 'A-Z' 'a-z' || true; }
|
||||
|
||||
# The compose invocation must include the postgres overlay whenever history runs
|
||||
# on PostgreSQL. deploy.sh writes COMPOSE_FILE into .env, which docker compose
|
||||
# reads by itself -- these helpers are for code that needs to know.
|
||||
pg_enabled() { [[ -f "$STACK_DIR/docker-compose.postgres.yml" ]] && grep -q '^COMPOSE_FILE=.*postgres' "$ENV_FILE" 2>/dev/null; }
|
||||
|
||||
# Is PostgreSQL up and accepting our credentials? rc 0 ok, 1 not.
|
||||
PG_CHECK_MSG=""
|
||||
pg_check() {
|
||||
PG_CHECK_MSG=""
|
||||
if ! pg_enabled; then PG_CHECK_MSG="postgres not in use"; return 0; fi
|
||||
if [[ "$(svc_state postgres)" != running* ]]; then PG_CHECK_MSG="the postgres container is not running"; return 1; fi
|
||||
local db user
|
||||
db="$(env_get POSTGRES_DB)"; user="$(env_get POSTGRES_USER)"
|
||||
if dc exec -T postgres pg_isready -U "${user:-ergo}" -d "${db:-ergo_history}" >/dev/null 2>&1; then
|
||||
PG_CHECK_MSG="postgres ready (${db:-ergo_history})"; return 0
|
||||
fi
|
||||
PG_CHECK_MSG="postgres is running but not accepting connections yet"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Fill the named variables from .env when they are not already set in the
|
||||
# environment. Used by update.sh/ergoctl so the knobs .env documents really are
|
||||
# the ones the scheduled jobs obey (environment > /etc/ergo-update.conf > .env).
|
||||
@@ -199,7 +249,13 @@ wait_healthy() { # <service> [timeout-seconds]
|
||||
# NICK+USER+OPER is 3 of the 5-line burst, so keep pre-oper payloads short.
|
||||
# ---------------------------------------------------------------------------
|
||||
IRC_LAST=""
|
||||
_irc_send() { printf '%s\r\n' "$1" >&3 2>/dev/null; }
|
||||
# Strip CR/LF/NUL before framing: an argument containing a newline (e.g.
|
||||
# `ergoctl announce "$(cat file)"`) would otherwise terminate this line and make
|
||||
# the rest a second command, executed with the admin oper's full privileges.
|
||||
_irc_send() {
|
||||
local l="${1//$'\r'/}"; l="${l//$'\n'/ }"; l="${l//$'\0'/}"
|
||||
printf '%s\r\n' "$l" >&3 2>/dev/null
|
||||
}
|
||||
_irc_cmd() { # echo the command/numeric token of a server line
|
||||
local l="$1"
|
||||
if [[ "$l" == :* ]]; then l="${l#* }"; fi
|
||||
@@ -349,6 +405,15 @@ ergo_validate_config() { # [image]
|
||||
VALIDATE_LOG="$(mktemp)"
|
||||
if ! cp -a "$IRCD_DIR"/. "$tmp"/ 2>>"$VALIDATE_LOG"; then rm -rf "$tmp"; return 1; fi
|
||||
rm -f "$tmp/ircd.lock"
|
||||
# The validator has no network (so it cannot fight the live server for ports),
|
||||
# which means it cannot reach a PostgreSQL history backend either. Neutralise
|
||||
# that one setting in the COPY so everything else -- listeners, opers, MOTD,
|
||||
# certs, and any pending DB schema upgrade -- is still really validated.
|
||||
# PostgreSQL reachability is checked separately (see pg_check in update.sh).
|
||||
if [[ "$(yaml_get_nested "$tmp/ircd.yaml" datastore postgresql enabled)" == "true" ]]; then
|
||||
yaml_set_nested "$tmp/ircd.yaml" datastore postgresql enabled false || true
|
||||
yaml_set_nested "$tmp/ircd.yaml" history persistent enabled false || true
|
||||
fi
|
||||
chown -R "$(ergo_uid):$(ergo_gid)" "$tmp"
|
||||
ergo_helper -v "$tmp:/ircd" -w /ircd --entrypoint /bin/sh -- "$image" \
|
||||
-c '/ircd-bin/ergo mkcerts >/dev/null 2>&1; exec /ircd-bin/ergo run --smoke' >>"$VALIDATE_LOG" 2>&1
|
||||
@@ -387,7 +452,7 @@ yaml_oper_set_password() { # <file> <name> <hash>
|
||||
top=="opers:" && cur==ENVIRON["NAME"] && /^ password:/ { print " password: \"" ENVIRON["HASH"] "\""; done=1; next }
|
||||
{ print }
|
||||
END { exit done ? 0 : 1 }' "$1" > "$tmp" || { rm -f "$tmp"; return 1; }
|
||||
cat "$tmp" > "$1"; rm -f "$tmp"
|
||||
safe_replace "$tmp" "$1" 0600 "$(ergo_uid)" "$(ergo_gid)"
|
||||
}
|
||||
|
||||
yaml_oper_add() { # <file> <name> <class> <hash> [whois-line]
|
||||
@@ -403,7 +468,7 @@ yaml_oper_add() { # <file> <name> <class> <hash> [whois-line]
|
||||
/^[a-z]/ { if (inopers && !done) { printf "%s\n", ENVIRON["BLOCK"]; done=1 } inopers=($1=="opers:") }
|
||||
{ print }
|
||||
END { if (inopers && !done) printf "%s", ENVIRON["BLOCK"] }' "$1" > "$tmp" || { rm -f "$tmp"; return 1; }
|
||||
cat "$tmp" > "$1"; rm -f "$tmp"
|
||||
safe_replace "$tmp" "$1" 0600 "$(ergo_uid)" "$(ergo_gid)"
|
||||
yaml_oper_has "$1" "$2"
|
||||
}
|
||||
|
||||
@@ -415,10 +480,49 @@ yaml_oper_rm() { # <file> <name>
|
||||
top=="opers:" && skip && (/^ / || /^[ \t]*$/) { next }
|
||||
top=="opers:" && skip { skip=0 }
|
||||
{ print }' "$1" > "$tmp" || { rm -f "$tmp"; return 1; }
|
||||
cat "$tmp" > "$1"; rm -f "$tmp"
|
||||
safe_replace "$tmp" "$1" 0600 "$(ergo_uid)" "$(ergo_gid)"
|
||||
! yaml_oper_has "$1" "$2"
|
||||
}
|
||||
|
||||
# Set a key nested as <top>:/<4-space sub>:/<8-space key>. Only the key inside
|
||||
# THAT block is rewritten -- datastore's mysql/postgresql/sqlite sub-blocks all
|
||||
# have an `enabled` key, so a plain sed would hit the wrong one. A key that is
|
||||
# not present is an error (rc 1), never a silent no-op.
|
||||
yaml_set_nested() { # <file> <top> <sub> <key> <value>
|
||||
local tmp; tmp="$(mktemp)"
|
||||
TOP="$2" SUB="$3" KEY="$4" VAL="$5" awk '
|
||||
/^[a-z][a-z-]*:/ { t=$0; sub(/:.*/,"",t); cur_top=t; cur_sub="" }
|
||||
/^ [a-z][a-z0-9-]*:/ { s=$0; sub(/^ /,"",s); sub(/:.*/,"",s); cur_sub=s }
|
||||
cur_top==ENVIRON["TOP"] && cur_sub==ENVIRON["SUB"] && $0 ~ "^ " ENVIRON["KEY"] ":" {
|
||||
print " " ENVIRON["KEY"] ": " ENVIRON["VAL"]; done=1; next
|
||||
}
|
||||
{ print }
|
||||
END { exit done ? 0 : 1 }' "$1" > "$tmp" || { rm -f "$tmp"; return 1; }
|
||||
safe_replace "$tmp" "$1" 0600 "$(ergo_uid)" "$(ergo_gid)"
|
||||
}
|
||||
|
||||
# Read a nested key back (for status/asserts); empty when absent.
|
||||
yaml_get_nested() { # <file> <top> <sub> <key>
|
||||
TOP="$2" SUB="$3" KEY="$4" awk '
|
||||
/^[a-z][a-z-]*:/ { t=$0; sub(/:.*/,"",t); cur_top=t; cur_sub="" }
|
||||
/^ [a-z][a-z0-9-]*:/ { s=$0; sub(/^ /,"",s); sub(/:.*/,"",s); cur_sub=s }
|
||||
cur_top==ENVIRON["TOP"] && cur_sub==ENVIRON["SUB"] && $0 ~ "^ " ENVIRON["KEY"] ":" {
|
||||
sub(/^ [a-z0-9-]+:[ \t]*/,""); gsub(/"/,""); print; exit
|
||||
}' "$1"
|
||||
}
|
||||
|
||||
# Which persistent-history backend the deployed ircd.yaml actually uses:
|
||||
# prints sqlite | postgres | off. Reads the config, not .env, so it reports what
|
||||
# Ergo is really doing after a hand edit.
|
||||
history_backend() { # [ircd.yaml]
|
||||
local f="${1:-$IRCD_DIR/ircd.yaml}"
|
||||
[[ -f "$f" ]] || { printf 'off'; return 0; }
|
||||
if [[ "$(yaml_get_nested "$f" history persistent enabled)" != "true" ]]; then printf 'off'; return 0; fi
|
||||
if [[ "$(yaml_get_nested "$f" datastore postgresql enabled)" == "true" ]]; then printf 'postgres'; return 0; fi
|
||||
if [[ "$(yaml_get_nested "$f" datastore sqlite enabled)" == "true" ]]; then printf 'sqlite'; return 0; fi
|
||||
printf 'off'
|
||||
}
|
||||
|
||||
# The first logging block's level (info <-> debug), for `ergoctl debug on|off`.
|
||||
yaml_set_log_level() { # <file> <level>
|
||||
local tmp; tmp="$(mktemp)"
|
||||
@@ -427,7 +531,7 @@ yaml_set_log_level() { # <file> <level>
|
||||
top=="logging:" && !done && /^ level:/ { print " level: " ENVIRON["LEVEL"]; done=1; next }
|
||||
{ print }
|
||||
END { exit done ? 0 : 1 }' "$1" > "$tmp" || { rm -f "$tmp"; return 1; }
|
||||
cat "$tmp" > "$1"; rm -f "$tmp"
|
||||
safe_replace "$tmp" "$1" 0600 "$(ergo_uid)" "$(ergo_gid)"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -486,16 +590,28 @@ ergo_certsync() {
|
||||
CERTSYNC_MSG="cert/key mismatch in Caddy storage (renewal in progress?); retrying later"; return 0
|
||||
fi
|
||||
uid="$(ergo_uid)"; gid="$(ergo_gid)"
|
||||
tmpc="$IRCD_DIR/.fullchain.pem.new"; tmpk="$IRCD_DIR/.privkey.pem.new"
|
||||
# Stage and keep the rollback copies OUTSIDE ircd/: that directory is
|
||||
# writable by the container uid, and `install`/`cp` would follow a symlink
|
||||
# planted there and write as root wherever it points. Only `mv` (rename(2),
|
||||
# which replaces a symlink rather than following it) touches ircd/.
|
||||
local prevdir="$STACK_DIR/.certsync"
|
||||
install -d -m 0700 "$prevdir" || { CERTSYNC_MSG="could not create $prevdir"; return 1; }
|
||||
tmpc="$prevdir/fullchain.pem.new"; tmpk="$prevdir/privkey.pem.new"
|
||||
if ! install -m 0600 -o "$uid" -g "$gid" "$src_crt" "$tmpc" || ! install -m 0600 -o "$uid" -g "$gid" "$src_key" "$tmpk"; then
|
||||
rm -f "$tmpc" "$tmpk"; CERTSYNC_MSG="could not stage the new pair in $IRCD_DIR"; return 1
|
||||
rm -f "$tmpc" "$tmpk"; CERTSYNC_MSG="could not stage the new pair in $prevdir"; return 1
|
||||
fi
|
||||
# source must not have moved under us mid-copy
|
||||
if ! cmp -s "$src_crt" "$tmpc" || ! cmp -s "$src_key" "$tmpk"; then
|
||||
rm -f "$tmpc" "$tmpk"; CERTSYNC_MSG="source changed during copy; retrying later"; return 0
|
||||
fi
|
||||
[[ -f "$dst_crt" ]] && cp -p "$dst_crt" "$dst_crt.prev"
|
||||
[[ -f "$dst_key" ]] && cp -p "$dst_key" "$dst_key.prev"
|
||||
# Roll-back copies, also outside ircd/. Read the live pair only if it is a
|
||||
# real file -- a symlink there is the container trying to exfiltrate.
|
||||
rm -f "$prevdir/fullchain.pem.prev" "$prevdir/privkey.pem.prev"
|
||||
if ! refuse_symlink "$dst_crt" "ircd/fullchain.pem" || ! refuse_symlink "$dst_key" "ircd/privkey.pem"; then
|
||||
rm -f "$tmpc" "$tmpk"; CERTSYNC_MSG="ircd/ TLS path is a symlink; refusing to sync"; return 1
|
||||
fi
|
||||
[[ -f "$dst_crt" ]] && cp -p "$dst_crt" "$prevdir/fullchain.pem.prev"
|
||||
[[ -f "$dst_key" ]] && cp -p "$dst_key" "$prevdir/privkey.pem.prev"
|
||||
mv -f "$tmpc" "$dst_crt"; mv -f "$tmpk" "$dst_key"
|
||||
if ergo_running; then
|
||||
dc kill -s HUP ergo >/dev/null 2>&1 || true
|
||||
@@ -508,15 +624,15 @@ ergo_certsync() {
|
||||
if [[ "$got" != "$want" ]]; then
|
||||
# Ergo kept the previous tls config (a failed rehash leaves the old one
|
||||
# active). Put the old pair back so the next start is not fatal.
|
||||
if [[ -f "$dst_crt.prev" && -f "$dst_key.prev" ]]; then
|
||||
mv -f "$dst_crt.prev" "$dst_crt"; mv -f "$dst_key.prev" "$dst_key"
|
||||
if [[ -f "$prevdir/fullchain.pem.prev" && -f "$prevdir/privkey.pem.prev" ]]; then
|
||||
mv -f "$prevdir/fullchain.pem.prev" "$dst_crt"; mv -f "$prevdir/privkey.pem.prev" "$dst_key"
|
||||
dc kill -s HUP ergo >/dev/null 2>&1 || true
|
||||
fi
|
||||
CERTSYNC_MSG="Ergo did not present the new cert after SIGHUP (see: docker compose logs ergo); previous pair restored"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
rm -f "$dst_crt.prev" "$dst_key.prev"
|
||||
rm -f "$prevdir/fullchain.pem.prev" "$prevdir/privkey.pem.prev"
|
||||
CERTSYNC_MSG="installed cert for $(cert_subject_cn "$dst_crt") (expires $(cert_enddate "$dst_crt")) from ${src_crt#"$CADDY_DIR"/data/caddy/certificates/}"
|
||||
return 0
|
||||
}
|
||||
@@ -541,12 +657,17 @@ send_notice() { # <title> <priority> <body>
|
||||
local title="$1" prio="$2" body="$3" host
|
||||
host="$(hostname -f 2>/dev/null || hostname 2>/dev/null || echo unknown)"
|
||||
set -- -fsS -m 5 -H "X-Title: ${title}" -H "X-Priority: ${prio}"
|
||||
[[ -n "${NTFY_TOKEN:-}" ]] && set -- "$@" -H "Authorization: Bearer ${NTFY_TOKEN}"
|
||||
[[ -n "${NTFY_EMAIL:-}" ]] && set -- "$@" -H "X-Email: ${NTFY_EMAIL}"
|
||||
local t="ergo"; [[ -n "${NTFY_REGION:-}" ]] && t="${t},${NTFY_REGION}"
|
||||
set -- "$@" -H "X-Tags: ${t}"
|
||||
if [[ "${DRY_RUN:-0}" == "1" ]]; then
|
||||
echo "DRY: curl ntfy ($prio): $body"
|
||||
elif [[ -n "${NTFY_TOKEN:-}" ]]; then
|
||||
# The token goes in on stdin, never in argv: /proc/<pid>/cmdline is
|
||||
# world-readable, and every local process is inside this stack's trust
|
||||
# boundary already (see README: Security model).
|
||||
printf 'header = "Authorization: Bearer %s"\n' "$NTFY_TOKEN" \
|
||||
| curl "$@" --config - -d "${body} [${host}]" "$NTFY_URL" >/dev/null 2>&1 || true
|
||||
else
|
||||
curl "$@" -d "${body} [${host}]" "$NTFY_URL" >/dev/null 2>&1 || true
|
||||
fi
|
||||
|
||||
+56
-10
@@ -224,16 +224,29 @@ apply_update() { # <from x.y.z> <to x.y.z> <reason>
|
||||
ts="$(date -u +%Y%m%d%H%M%S)"
|
||||
install -d -m 0700 "$BACKUP_DIR"
|
||||
snap="$BACKUP_DIR/ircd.db.${from:-unknown}.${ts}"
|
||||
if [[ -f "$IRCD_DIR/ircd.db" ]]; then
|
||||
cp -p "$IRCD_DIR/ircd.db" "$snap" && chmod 0600 "$snap"
|
||||
# -P: never follow a symlink in the uid-owned ircd/ (see ergolib's note).
|
||||
if [[ -f "$IRCD_DIR/ircd.db" ]] && refuse_symlink "$IRCD_DIR/ircd.db" "ircd/ircd.db"; then
|
||||
cp -Pp "$IRCD_DIR/ircd.db" "$snap" && chmod 0600 "$snap"
|
||||
else
|
||||
snap=""
|
||||
fi
|
||||
# The message history is a separate store; snapshot it too so a rollback is
|
||||
# not silently lossy.
|
||||
case "$(history_backend)" in
|
||||
sqlite) [[ -f "$IRCD_DIR/ergo_history.db" ]] && cp -Pp "$IRCD_DIR/ergo_history.db" "$BACKUP_DIR/ergo_history.db.${from:-unknown}.${ts}" 2>/dev/null || true ;;
|
||||
postgres) dc up -d --no-deps postgres >/dev/null 2>&1 || true
|
||||
if pg_check; then
|
||||
dc exec -T postgres pg_dump -U "$(env_get POSTGRES_USER)" -d "$(env_get POSTGRES_DB)" \
|
||||
> "$BACKUP_DIR/history.sql.${from:-unknown}.${ts}" 2>/dev/null \
|
||||
|| { rm -f "$BACKUP_DIR/history.sql.${from:-unknown}.${ts}"; warn "pg_dump failed; the rollback would not restore message history."; }
|
||||
fi ;;
|
||||
esac
|
||||
|
||||
env_set ERGO_TAG "v${to}"
|
||||
dc up -d --no-deps ergo >/dev/null 2>&1 || true
|
||||
if wait_healthy ergo 120 && irc_probe; then
|
||||
log "Ergo ${to} is healthy."
|
||||
state_rm failed
|
||||
record_template "$new_image" "$from" "$to"
|
||||
prune_snapshots
|
||||
local b="updated ${from:-?} -> ${to} (${reason})"
|
||||
@@ -259,8 +272,13 @@ apply_update() { # <from x.y.z> <to x.y.z> <reason>
|
||||
wait_healthy ergo 90 || true
|
||||
fi
|
||||
fi
|
||||
prune_snapshots
|
||||
# Remember the target that failed so the daily run does not repeat this
|
||||
# whole disruptive cycle (warn users, stop, swap, roll back) every night.
|
||||
# A manual `ergoctl update update` still forces a retry.
|
||||
state_set failed "$to"
|
||||
if [[ "$(svc_state ergo)" == "running healthy" ]]; then
|
||||
send_notice "Ergo update FAILED" "high" "${to} unhealthy (${cause:-see docker compose logs ergo}); rolled back to ${prev_tag}. Snapshot: ${snap:-none}"
|
||||
send_notice "Ergo update FAILED" "high" "${to} unhealthy (${cause:-see docker compose logs ergo}); rolled back to ${prev_tag}. Snapshot: ${snap:-none}. Will not retry automatically; run 'ergoctl update update' after investigating."
|
||||
else
|
||||
send_notice "Ergo DOWN after failed update" "urgent" "${to} failed (${cause:-?}) and ${prev_tag} did not come back healthy. Snapshot: ${snap:-none}. Run 'ergoctl status' / 'docker compose logs ergo'."
|
||||
fi
|
||||
@@ -287,10 +305,13 @@ record_template() { # <image> <from> <to>
|
||||
log "$TEMPLATE_NOTE"
|
||||
}
|
||||
|
||||
prune_snapshots() { # keep the last 5 DB snapshots and .env backups
|
||||
local f n=0
|
||||
for f in $(ls -1t "$BACKUP_DIR"/ircd.db.* 2>/dev/null); do
|
||||
n=$((n + 1)); (( n > 5 )) && rm -f "$f"
|
||||
prune_snapshots() { # keep the last 5 of each snapshot kind, and 5 .env backups
|
||||
local f n=0 pat
|
||||
for pat in 'ircd.db.*' 'ergo_history.db.*' 'history.sql.*'; do
|
||||
n=0
|
||||
for f in $(ls -1t "$BACKUP_DIR"/$pat 2>/dev/null); do
|
||||
n=$((n + 1)); (( n > 5 )) && rm -f "$f"
|
||||
done
|
||||
done
|
||||
n=0
|
||||
for f in $(ls -1t "$STACK_DIR"/.env.bak.* 2>/dev/null); do
|
||||
@@ -321,7 +342,12 @@ do_certsync() {
|
||||
prev="$(state_get certsync)"
|
||||
case "$rc" in
|
||||
0) [[ "$CERTSYNC_MSG" == "up to date" ]] || log "certsync: $CERTSYNC_MSG"
|
||||
if [[ -n "$prev" ]]; then state_rm certsync; send_notice "Ergo TLS recovered" "default" "cert sync ok: ${CERTSYNC_MSG}"; fi ;;
|
||||
# rc 0 also covers the "renewal in progress, retrying later" cases,
|
||||
# which synced nothing -- clearing a stored failure on those would
|
||||
# send a false all-clear while the real problem persists.
|
||||
if [[ -n "$prev" && ( "$CERTSYNC_MSG" == "up to date" || "$CERTSYNC_MSG" == installed\ cert* ) ]]; then
|
||||
state_rm certsync; send_notice "Ergo TLS recovered" "default" "cert sync ok: ${CERTSYNC_MSG}"
|
||||
fi ;;
|
||||
2) [[ -n "${VERBOSE:-}" ]] && log "certsync: $CERTSYNC_MSG"
|
||||
return 0 ;;
|
||||
*) warn "certsync: $CERTSYNC_MSG"
|
||||
@@ -380,6 +406,17 @@ preflight() {
|
||||
command -v jq >/dev/null 2>&1 || die "jq is required (release metadata); install it."
|
||||
}
|
||||
|
||||
# With HISTORY=postgres, Ergo will not start if its history backend is down --
|
||||
# so an update must not stop a healthy server until the database is confirmed up.
|
||||
# The config validator cannot check this: it runs with no network on purpose.
|
||||
pg_gate() { # <what we are about to do>
|
||||
pg_enabled || return 0
|
||||
pg_check && return 0
|
||||
warn "PostgreSQL is not ready ($PG_CHECK_MSG); skipping $1 -- Ergo would not restart."
|
||||
send_notice "Ergo update skipped" "high" "PostgreSQL not ready (${PG_CHECK_MSG}); ${1} was not attempted."
|
||||
return 1
|
||||
}
|
||||
|
||||
do_check() {
|
||||
preflight
|
||||
local cur latest ghsa held="" prio="min" note
|
||||
@@ -412,7 +449,11 @@ do_check() {
|
||||
do_run() {
|
||||
preflight
|
||||
if [[ "$DRY_RUN" != "1" ]]; then
|
||||
install -d -m 0755 "$(dirname "$LOG")" 2>/dev/null || true
|
||||
# Create the log directory only when it is missing. `install -d -m` also
|
||||
# re-modes an EXISTING directory, and this resolves to /var/log, which on
|
||||
# Debian/Ubuntu ships 0775 root:syslog so rsyslog (running as `syslog`)
|
||||
# can create files there -- re-moding it to 0755 silently breaks logging.
|
||||
[[ -d "$(dirname "$LOG")" ]] || install -d -m 0755 "$(dirname "$LOG")" 2>/dev/null || true
|
||||
echo "=== ergo-update $(date -u +%FT%TZ) ===" >> "$LOG"
|
||||
# Scheduled runs have nowhere to send stdout (busybox crond would try to
|
||||
# mail it and these hosts have no MTA), so keep the whole run in the log.
|
||||
@@ -426,7 +467,11 @@ do_run() {
|
||||
resolve_target "$cur"
|
||||
log "current=${cur:-?} | ${REASON}"
|
||||
if [[ -n "$TARGET" && -n "$cur" ]] && ver_gt "$TARGET" "$cur"; then
|
||||
if [[ "$HOLD" == 1 && "$FORCE_UPDATE" != "1" ]]; then
|
||||
if [[ "$(state_get failed)" == "$TARGET" && "$FORCE_UPDATE" != "1" ]]; then
|
||||
log "Skipping ${TARGET}: it already failed here and was rolled back. Investigate, then run 'ergoctl update update' (or set FORCE_UPDATE=1)."
|
||||
elif ! pg_gate "the update to ${TARGET}"; then
|
||||
: # postgres is down; pg_gate already warned and notified
|
||||
elif [[ "$HOLD" == 1 && "$FORCE_UPDATE" != "1" ]]; then
|
||||
warn "HELD: ${TARGET} announces compatibility breaks. Review the release notes, then 'ergoctl update update' (or FORCE_UPDATE=1)."
|
||||
if [[ "$(state_get held)" != "$TARGET" ]]; then
|
||||
state_set held "$TARGET"
|
||||
@@ -459,6 +504,7 @@ do_update() {
|
||||
warn "Release notes between ${cur} and ${TARGET} announce compatibility breaks: ${NOTES}"
|
||||
warn "Proceeding because you asked explicitly (this is 'update update'). Set FORCE_UPDATE=1 to also auto-apply such releases."
|
||||
fi
|
||||
pg_gate "the update to ${TARGET}" || die "Start PostgreSQL first (docker compose up -d postgres), then retry."
|
||||
apply_update "$cur" "$TARGET" "${REASON} (manual)"
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user