57_WolveandClaude Opus 5
85d945ccde
Ergo keeps history in RAM by default, so it is lost on every restart -- including the ones this stack's updater performs. HISTORY now selects a backend at first deploy: sqlite (default, a file beside ircd.db, no extra container), postgres (a pinned container via docker-compose.postgres.yml, loopback-only, POSTGRES_PASSWORD_FILE so no secret lands in .env), or off. Both SQL backends need Ergo 2.18.0+, which deploy.sh enforces. HISTORY_EXPIRE (default 30d) sets retention, because upstream's 1w expire-time DELETES from persistent storage -- persistence with the shipped default would buy only a week. Ergo opens the history backend only at startup and, unlike MySQL, has no "after launching the server" guard for sqlite/postgresql: a rehash that enables one reports success and then silently discards every message. So `ergoctl history` restarts, and `ergoctl edit` detects a backend change and refuses to apply it by rehash. The config validator runs with no network by design, so it neutralises postgresql in its copy and reachability is checked separately. Audit fixes (six lenses over first-deploy, runtime, lifecycle, security, docs): - SECURITY (blocker): ircd/ is writable by the container uid while the 15-minute cert sync and every ergoctl config edit run as root. cp/install/> follow a symlink, so code execution in Ergo could redirect a root write onto any host file. All such writes now stage under $STACK_DIR (0700 root) and land via mv (rename(2) replaces a symlink instead of following it); root reads use cp -P or refuse. Regression-tested with real symlinks. - deploy.sh no longer swaps the image when ERGO_TAG changes on a deployed stack (that bypassed pre-flight, snapshot, user warning and rollback); it points at `TARGET_VERSION=... ergoctl update update`. - restore was lossy and could lock you out: it now snapshots the current config as well as the database, restores message history, re-points the postgres password and re-hashes the local admin password into the restored config, and validates before starting. - `install -d -m` re-modes existing directories: `ergoctl backup /var/backups` no longer chmods it 0700 root, and the updater no longer re-modes /var/log (0775 root:syslog on Debian, which rsyslog needs). - The admin oper password is printed only on a first run at a TTY, so it stays out of cloud-init serial-console logs. - A failed update is remembered, so the daily job stops repeating a disruptive warn/stop/swap/roll-back cycle every night; postgres readiness gates an update that would otherwise stop a healthy server it cannot restart. - certsync no longer sends "TLS recovered" for outcomes that synced nothing. - ergoctl history writes .env only after the restart is healthy, and returns 0. - CR/LF is stripped before IRC framing, so a multi-line argument cannot inject a second command; the ntfy token moves out of curl's argv. - ufw/firewalld are additive, so 6667 is now explicitly revoked when PLAINTEXT=0. - build.sh refuses to build a deploy.sh whose archive is missing a file the script reads -- the failure mode that would have shipped a stack aborting on every host. Docs corrected against the code throughout, including retention, the pre-connect account-registration default, encrypted-restore (AGE_IDENTITY), what a re-run really does to .env, and what the update log does and does not contain. Verified locally: all six suites pass (config render for each backend against the real 2.19.1 template, yaml/oper/version/env helpers, the IRC client against a fake server, and the audit fixes including the symlink escalation). Still not exercised on a Docker host: the containers, ACME issuance, cert sync and PostgreSQL itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
617 lines
30 KiB
Bash
617 lines
30 KiB
Bash
#!/usr/bin/env bash
|
|
#
|
|
# update.sh -- keep the Ergo stack current: container updates with a DB snapshot
|
|
# and health-checked rollback, plus the Caddy -> Ergo TLS certificate sync.
|
|
# Companion to deploy.sh; installed in $STACK_DIR next to ergolib.sh.
|
|
#
|
|
# Subcommands:
|
|
# check (default) current vs latest, advisories, cert state; changes nothing
|
|
# run what the daily schedule runs: apply UPDATE_POLICY, cert sync, expiry check
|
|
# update update now (to latest, or TARGET_VERSION=x.y.z); honours FORCE_UPDATE
|
|
# certsync copy Caddy's cert into ircd/ and rehash if it changed (15-min schedule)
|
|
# caddy pull a newer Caddy image and recreate it (health-checked)
|
|
# install write /etc/ergo-update.conf + schedule the jobs
|
|
# uninstall remove the schedule
|
|
#
|
|
# Policy (UPDATE_POLICY):
|
|
# latest update to the newest release whenever one exists (default)
|
|
# security update ONLY when a published GitHub security advisory covers the
|
|
# running version, or a release between current and latest has a
|
|
# "### Security" section in its notes
|
|
# off never change the running version (check/notify only)
|
|
#
|
|
# Safety rails (all policies):
|
|
# * releases whose notes announce "Compatibility breaks" are HELD (notify
|
|
# only) unless FORCE_UPDATE=1 -- review the notes, then `update update`
|
|
# * the new image must load the current ircd/ (ergo run --smoke on a copy,
|
|
# which also dry-runs any DB schema upgrade) before the live server is touched
|
|
# * users get a NOTICE and UPDATE_GRACE seconds before the restart
|
|
# * ircd.db is snapshotted with Ergo stopped (buntdb is append-only; a live
|
|
# copy can miss the last second of writes)
|
|
# * health = compose healthcheck (IRC-level) + a registration handshake; on
|
|
# failure the previous tag is restored and, if the new version bumped the
|
|
# DB schema, the snapshot is put back
|
|
#
|
|
# Env, in precedence order: environment > /etc/ergo-update.conf > the stack's
|
|
# .env > the built-in default.
|
|
# STACK_DIR=/srv/ergo UPDATE_POLICY=latest FORCE_UPDATE=0 UPDATE_GRACE=60
|
|
# CADDY_AUTOUPDATE=0 NOTIFY=1 SSH_NOTIFY_CONF=/etc/ssh-notify.conf DRY_RUN=0
|
|
# TARGET_VERSION= GH_REPO=ergochat/ergo LOG=/var/log/ergo-update.log
|
|
# SCHEDULE_UPDATES=1 (install: 0 schedules certsync only, no daily update)
|
|
# VERBOSE= (certsync: also log "nothing to do" outcomes)
|
|
|
|
set -euo pipefail
|
|
|
|
SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
|
|
SELF_DIR="$(dirname "$SELF")"
|
|
|
|
log() { printf '\033[1;32m[+]\033[0m %s\n' "$*"; }
|
|
warn() { printf '\033[1;33m[!]\033[0m %s\n' "$*" >&2; }
|
|
die() { printf '\033[1;31m[x]\033[0m %s\n' "$*" >&2; exit 1; }
|
|
|
|
# shellcheck source=ergolib.sh
|
|
. "$SELF_DIR/ergolib.sh"
|
|
load_conf
|
|
: "${STACK_DIR:=$SELF_DIR}"
|
|
ergo_set_paths
|
|
# .env is the stack's own settings file and documents these knobs, so honour it
|
|
# for anything the environment and /etc/ergo-update.conf did not already set.
|
|
env_defaults UPDATE_POLICY FORCE_UPDATE UPDATE_GRACE CADDY_AUTOUPDATE
|
|
: "${UPDATE_POLICY:=latest}"
|
|
: "${FORCE_UPDATE:=0}"
|
|
: "${UPDATE_GRACE:=60}"
|
|
: "${CADDY_AUTOUPDATE:=0}"
|
|
: "${TARGET_VERSION:=}"
|
|
: "${SCHEDULE_UPDATES:=1}"
|
|
: "${LOG:=/var/log/ergo-update.log}"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Release metadata (GitHub). Unauthenticated: 60 requests/hour -- a run uses 2-3.
|
|
# ---------------------------------------------------------------------------
|
|
RELEASES_JSON=""
|
|
# Fetch the release list ONCE per process. This must be called as a plain
|
|
# statement, never inside a pipeline or $(): the assignment would land in a
|
|
# subshell and every use would re-download the list (and burn the 60/hour
|
|
# anonymous API budget).
|
|
load_releases() {
|
|
[[ -n "$RELEASES_JSON" ]] && return 0
|
|
RELEASES_JSON="$(fetch "https://api.github.com/repos/${GH_REPO}/releases?per_page=50" || true)"
|
|
printf '%s' "$RELEASES_JSON" | jq -e 'type=="array"' >/dev/null 2>&1 || RELEASES_JSON="[]"
|
|
return 0
|
|
}
|
|
|
|
# Bodies of stable releases with cur < version <= upto, newest first:
|
|
# "=== <tag>" header lines followed by the body.
|
|
release_notes_between() { # <cur> <upto>
|
|
local cur="$1" upto="$2" tag v
|
|
load_releases
|
|
while IFS= read -r tag; do
|
|
[[ -n "$tag" ]] || continue
|
|
v="$(normver "$tag")"; [[ -n "$v" ]] || continue
|
|
ver_gt "$v" "$cur" || continue
|
|
ver_le "$v" "$upto" || continue
|
|
printf '=== %s\n' "$tag"
|
|
printf '%s' "$RELEASES_JSON" | jq -r --arg t "$tag" '.[] | select(.tag_name==$t) | .body // ""'
|
|
printf '\n'
|
|
done < <(printf '%s' "$RELEASES_JSON" | jq -r '.[] | select(.prerelease==false and .draft==false) | .tag_name' 2>/dev/null)
|
|
}
|
|
|
|
# Published GitHub security advisories covering <cur>. Prints "GHSA-... <patched>"
|
|
# lines. Ranges look like "<= v2.19.0" or ">= v2.15.0, < v2.19.1".
|
|
ghsa_covering() { # <cur>
|
|
local cur="$1" json id rng patched part op ver ok
|
|
json="$(fetch "https://api.github.com/repos/${GH_REPO}/security-advisories" || true)"
|
|
printf '%s' "$json" | jq -e 'type=="array"' >/dev/null 2>&1 || return 0
|
|
while IFS=$'\t' read -r id rng patched; do
|
|
[[ -n "$rng" ]] || continue
|
|
ok=1
|
|
IFS=',' read -r -a parts <<< "$rng"
|
|
for part in "${parts[@]}"; do
|
|
part="$(printf '%s' "$part" | tr -d ' ')"
|
|
op="$(printf '%s' "$part" | grep -oE '^(<=|>=|==|<|>|=)' || echo '=')"
|
|
ver="$(normver "$part")"; [[ -n "$ver" ]] || { ok=0; break; }
|
|
case "$op" in
|
|
'<') ver_gt "$ver" "$cur" || ok=0 ;;
|
|
'<=') ver_ge "$ver" "$cur" || ok=0 ;;
|
|
'>') ver_gt "$cur" "$ver" || ok=0 ;;
|
|
'>=') ver_ge "$cur" "$ver" || ok=0 ;;
|
|
*) [[ "$(ver_cmp "$ver" "$cur")" == "0" ]] || ok=0 ;;
|
|
esac
|
|
(( ok )) || break
|
|
done
|
|
(( ok )) && printf '%s %s\n' "$id" "$(normver "$patched")"
|
|
done < <(printf '%s' "$json" | jq -r '.[] | select(.state=="published") | .ghsa_id as $id | (.vulnerabilities // [])[] | [$id, (.vulnerable_version_range // ""), ((.patched_versions // "") | tostring)] | @tsv' 2>/dev/null)
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Decide what to do. Sets globals (not via stdout -- a $() would lose them):
|
|
# TARGET (x.y.z or ""), REASON, HOLD (1 = compat break, notify only),
|
|
# NOTES (one-line summary for the notification), DBCHANGE (0/1)
|
|
# ---------------------------------------------------------------------------
|
|
TARGET=""; REASON=""; HOLD=0; NOTES=""; DBCHANGE=0
|
|
resolve_target() { # <current x.y.z>
|
|
local cur="$1" latest notes ghsa ids
|
|
TARGET=""; REASON=""; HOLD=0; NOTES=""; DBCHANGE=0
|
|
latest="$(normver "$(ergo_latest_tag || true)")"
|
|
if [[ -n "$TARGET_VERSION" ]]; then
|
|
TARGET="$(normver "$TARGET_VERSION")"; REASON="target override ${TARGET}"
|
|
else
|
|
case "$UPDATE_POLICY" in
|
|
off) REASON="policy=off (no changes)"; return 0 ;;
|
|
security)
|
|
ghsa="$(ghsa_covering "$cur")"
|
|
if [[ -n "$ghsa" ]]; then
|
|
ids="$(printf '%s\n' "$ghsa" | awk '{print $1}' | paste -sd, - || true)"
|
|
# NF>=2: an advisory with no patched version yields only an id.
|
|
TARGET="$(printf '%s\n' "$ghsa" | awk 'NF>=2 {print $2}' | tail -n1 || true)"
|
|
[[ -n "$TARGET" ]] || TARGET="$latest"
|
|
REASON="security advisory covers ${cur}: ${ids}"
|
|
elif [[ -n "$latest" ]] && ver_gt "$latest" "$cur" \
|
|
&& grep -qiE '^#+ *security|security vulnerabilit' <<< "$(release_notes_between "$cur" "$latest")"; then
|
|
TARGET="$latest"; REASON="a release since ${cur} carries a Security section"
|
|
else
|
|
REASON="no security advisory or security release covers ${cur}"; return 0
|
|
fi ;;
|
|
latest|*)
|
|
[[ -n "$latest" ]] || { REASON="could not resolve the latest release"; return 0; }
|
|
TARGET="$latest"; REASON="policy=latest" ;;
|
|
esac
|
|
fi
|
|
[[ -n "$TARGET" ]] || return 0
|
|
notes="$(release_notes_between "$cur" "$TARGET")"
|
|
# Feed grep from a here-string, not a pipe: `printf | grep -q` on notes
|
|
# larger than the pipe buffer makes printf die of SIGPIPE, and pipefail
|
|
# then reports "no match" for what was in fact a match.
|
|
if grep -qiE 'compatibility break|not backwards[- ]compatible|backwards[- ]incompatible' <<< "$notes"; then
|
|
HOLD=1
|
|
fi
|
|
if grep -qiE 'database change|database file format changes|schema' <<< "$notes" \
|
|
&& ! grep -qiE 'no changes to the config file format or database file format' <<< "$notes"; then
|
|
DBCHANGE=1
|
|
fi
|
|
NOTES="$(grep -iE 'config file format|database (file )?format|database change|compatibility break' <<< "$notes" \
|
|
| head -n 3 | sed 's/^[ *-]*//' | tr '\n' ' ' | cut -c1-300 || true)"
|
|
return 0
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Apply an update: pre-flight on a copy, warn users, stop, snapshot, up, verify;
|
|
# roll back (tag + DB snapshot when needed) on failure.
|
|
# ---------------------------------------------------------------------------
|
|
notify_users() { # <text> -- best effort, only when Ergo is up and we can oper
|
|
ergo_running || return 0
|
|
irc_raw --oper --quiet 1 -- "NOTICE \$\$* :$1" >/dev/null 2>&1 || true
|
|
}
|
|
|
|
apply_update() { # <from x.y.z> <to x.y.z> <reason>
|
|
local from="$1" to="$2" reason="$3" image_base new_image snap ts cause prev_tag
|
|
image_base="$(env_get ERGO_IMAGE)"; image_base="${image_base:-ghcr.io/ergochat/ergo}"
|
|
new_image="${image_base}:v${to}"
|
|
# Roll back to the tag that is actually pinned right now. Reconstructing
|
|
# "v${from}" would write a bare "v" when the running version is unknown
|
|
# (e.g. .env still holds the 'stable' fallback and the container is down).
|
|
prev_tag="$(ergo_tag)"
|
|
log "Updating Ergo ${from:-?} -> ${to} (${reason})..."
|
|
if [[ "$DRY_RUN" == "1" ]]; then
|
|
echo "DRY: pull ${new_image}; validate on a copy; NOTICE users; stop; snapshot ircd.db; ERGO_TAG=v${to}; up -d --no-deps ergo"
|
|
return 0
|
|
fi
|
|
cp -a "$ENV_FILE" "${ENV_FILE}.bak.$(date -u +%Y%m%d%H%M%S)" 2>/dev/null || true
|
|
|
|
log "Pulling ${new_image}..."
|
|
if ! docker pull -q "$new_image" >/dev/null 2>&1; then
|
|
send_notice "Ergo update FAILED" "high" "pull of ${new_image} failed; staying on ${from}"
|
|
return 1
|
|
fi
|
|
|
|
log "Pre-flight: loading the current ircd/ with ${new_image} (throwaway container)..."
|
|
if ! ergo_validate_config "$new_image"; then
|
|
cause="$(tail -n 5 "$VALIDATE_LOG" 2>/dev/null | tr '\n' ' ' | cut -c1-300)"
|
|
warn "Pre-flight failed: $cause"
|
|
send_notice "Ergo update HELD" "high" "${to} cannot load the current config/DB: ${cause}. Staying on ${from}; run 'ergoctl update check'."
|
|
return 1
|
|
fi
|
|
rm -f "$VALIDATE_LOG"
|
|
|
|
if (( UPDATE_GRACE > 0 )) && ergo_running; then
|
|
log "Warning connected users (${UPDATE_GRACE}s grace)..."
|
|
notify_users "Server restarting for an upgrade to Ergo ${to} in ${UPDATE_GRACE} seconds -- you will be disconnected briefly."
|
|
sleep "$UPDATE_GRACE"
|
|
fi
|
|
|
|
log "Stopping Ergo and snapshotting ircd.db..."
|
|
dc stop -t 15 ergo >/dev/null 2>&1 || true
|
|
ts="$(date -u +%Y%m%d%H%M%S)"
|
|
install -d -m 0700 "$BACKUP_DIR"
|
|
snap="$BACKUP_DIR/ircd.db.${from:-unknown}.${ts}"
|
|
# -P: never follow a symlink in the uid-owned ircd/ (see ergolib's note).
|
|
if [[ -f "$IRCD_DIR/ircd.db" ]] && refuse_symlink "$IRCD_DIR/ircd.db" "ircd/ircd.db"; then
|
|
cp -Pp "$IRCD_DIR/ircd.db" "$snap" && chmod 0600 "$snap"
|
|
else
|
|
snap=""
|
|
fi
|
|
# The message history is a separate store; snapshot it too so a rollback is
|
|
# not silently lossy.
|
|
case "$(history_backend)" in
|
|
sqlite) [[ -f "$IRCD_DIR/ergo_history.db" ]] && cp -Pp "$IRCD_DIR/ergo_history.db" "$BACKUP_DIR/ergo_history.db.${from:-unknown}.${ts}" 2>/dev/null || true ;;
|
|
postgres) dc up -d --no-deps postgres >/dev/null 2>&1 || true
|
|
if pg_check; then
|
|
dc exec -T postgres pg_dump -U "$(env_get POSTGRES_USER)" -d "$(env_get POSTGRES_DB)" \
|
|
> "$BACKUP_DIR/history.sql.${from:-unknown}.${ts}" 2>/dev/null \
|
|
|| { rm -f "$BACKUP_DIR/history.sql.${from:-unknown}.${ts}"; warn "pg_dump failed; the rollback would not restore message history."; }
|
|
fi ;;
|
|
esac
|
|
|
|
env_set ERGO_TAG "v${to}"
|
|
dc up -d --no-deps ergo >/dev/null 2>&1 || true
|
|
if wait_healthy ergo 120 && irc_probe; then
|
|
log "Ergo ${to} is healthy."
|
|
state_rm failed
|
|
record_template "$new_image" "$from" "$to"
|
|
prune_snapshots
|
|
local b="updated ${from:-?} -> ${to} (${reason})"
|
|
[[ -n "$NOTES" ]] && b="${b}. Notes: ${NOTES}"
|
|
[[ -n "$TEMPLATE_NOTE" ]] && b="${b}. ${TEMPLATE_NOTE}"
|
|
send_notice "Ergo updated" "default" "$b"
|
|
return 0
|
|
fi
|
|
|
|
cause="$(dc logs --tail 60 ergo 2>/dev/null | grep -E 'Config file did not load|Database requires update|Could not load server|failed to|fatal|panic' | tail -n 2 | cut -c1-200 | tr '\n' ' ' || true)"
|
|
warn "Ergo ${to} did not become healthy (${cause:-no diagnostic in the log}); rolling back to ${prev_tag}."
|
|
dc stop -t 10 ergo >/dev/null 2>&1 || true
|
|
env_set ERGO_TAG "$prev_tag"
|
|
dc up -d --no-deps ergo >/dev/null 2>&1 || true
|
|
if ! wait_healthy ergo 90; then
|
|
# The old binary refuses a newer schema ("Database requires update") --
|
|
# only then does the snapshot go back (otherwise keep the live DB).
|
|
if [[ -n "$snap" ]] && { [[ "$DBCHANGE" == 1 ]] || dc logs --tail 40 ergo 2>/dev/null | grep -q 'Database requires update'; }; then
|
|
warn "Old version rejects the upgraded database; restoring the snapshot ${snap}."
|
|
dc stop -t 10 ergo >/dev/null 2>&1 || true
|
|
install -m 0600 -o "$(ergo_uid)" -g "$(ergo_gid)" "$snap" "$IRCD_DIR/ircd.db"
|
|
dc up -d --no-deps ergo >/dev/null 2>&1 || true
|
|
wait_healthy ergo 90 || true
|
|
fi
|
|
fi
|
|
prune_snapshots
|
|
# Remember the target that failed so the daily run does not repeat this
|
|
# whole disruptive cycle (warn users, stop, swap, roll back) every night.
|
|
# A manual `ergoctl update update` still forces a retry.
|
|
state_set failed "$to"
|
|
if [[ "$(svc_state ergo)" == "running healthy" ]]; then
|
|
send_notice "Ergo update FAILED" "high" "${to} unhealthy (${cause:-see docker compose logs ergo}); rolled back to ${prev_tag}. Snapshot: ${snap:-none}. Will not retry automatically; run 'ergoctl update update' after investigating."
|
|
else
|
|
send_notice "Ergo DOWN after failed update" "urgent" "${to} failed (${cause:-?}) and ${prev_tag} did not come back healthy. Snapshot: ${snap:-none}. Run 'ergoctl status' / 'docker compose logs ergo'."
|
|
fi
|
|
return 1
|
|
}
|
|
|
|
# Keep the image's default.yaml per version and diff it against the previous one
|
|
# so template drift (new options, renamed keys) is visible after an upgrade.
|
|
TEMPLATE_NOTE=""
|
|
record_template() { # <image> <from> <to>
|
|
local image="$1" from="$2" to="$3" new old diff added removed
|
|
TEMPLATE_NOTE=""
|
|
install -d -m 0700 "$TEMPLATES_DIR"
|
|
new="$TEMPLATES_DIR/default.yaml.v${to}"
|
|
ergo_extract_template "$image" "$new" 2>/dev/null || { rm -f "$new"; return 0; }
|
|
chmod 0600 "$new"
|
|
old="$TEMPLATES_DIR/default.yaml.v${from}"
|
|
[[ -f "$old" ]] || return 0
|
|
diff="$TEMPLATES_DIR/diff-v${from}-v${to}.txt"
|
|
if diff -u "$old" "$new" > "$diff" 2>/dev/null; then rm -f "$diff"; return 0; fi
|
|
chmod 0600 "$diff" 2>/dev/null || true
|
|
added="$(grep -cE '^\+[^+]' "$diff" || true)"; removed="$(grep -cE '^-[^-]' "$diff" || true)"
|
|
TEMPLATE_NOTE="default config changed upstream (+${added}/-${removed} lines): review ${diff} against ircd/ircd.yaml"
|
|
log "$TEMPLATE_NOTE"
|
|
}
|
|
|
|
prune_snapshots() { # keep the last 5 of each snapshot kind, and 5 .env backups
|
|
local f n=0 pat
|
|
for pat in 'ircd.db.*' 'ergo_history.db.*' 'history.sql.*'; do
|
|
n=0
|
|
for f in $(ls -1t "$BACKUP_DIR"/$pat 2>/dev/null); do
|
|
n=$((n + 1)); (( n > 5 )) && rm -f "$f"
|
|
done
|
|
done
|
|
n=0
|
|
for f in $(ls -1t "$STACK_DIR"/.env.bak.* 2>/dev/null); do
|
|
n=$((n + 1)); (( n > 5 )) && rm -f "$f"
|
|
done
|
|
return 0
|
|
}
|
|
|
|
# The pinned tag must be an immutable vX.Y.Z: deploy.sh falls back to 'stable'
|
|
# when GitHub is unreachable, and a floating tag silently changes the running
|
|
# version on any pull, with no pre-flight, snapshot or rollback. Pin it to
|
|
# whatever is running as soon as we can see a version (no restart needed).
|
|
pin_floating_tag() { # <current x.y.z>
|
|
local cur="$1" tag
|
|
tag="$(ergo_tag)"
|
|
[[ -n "$cur" ]] || return 0
|
|
case "$tag" in v[0-9]*) return 0 ;; esac
|
|
env_set ERGO_TAG "v${cur}"
|
|
log "Pinned ERGO_TAG=v${cur} (was '${tag}', which floats)."
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Cert sync + expiry watch
|
|
# ---------------------------------------------------------------------------
|
|
do_certsync() {
|
|
local rc=0 prev
|
|
ergo_certsync || rc=$?
|
|
prev="$(state_get certsync)"
|
|
case "$rc" in
|
|
0) [[ "$CERTSYNC_MSG" == "up to date" ]] || log "certsync: $CERTSYNC_MSG"
|
|
# rc 0 also covers the "renewal in progress, retrying later" cases,
|
|
# which synced nothing -- clearing a stored failure on those would
|
|
# send a false all-clear while the real problem persists.
|
|
if [[ -n "$prev" && ( "$CERTSYNC_MSG" == "up to date" || "$CERTSYNC_MSG" == installed\ cert* ) ]]; then
|
|
state_rm certsync; send_notice "Ergo TLS recovered" "default" "cert sync ok: ${CERTSYNC_MSG}"
|
|
fi ;;
|
|
2) [[ -n "${VERBOSE:-}" ]] && log "certsync: $CERTSYNC_MSG"
|
|
return 0 ;;
|
|
*) warn "certsync: $CERTSYNC_MSG"
|
|
if [[ "$prev" != "$CERTSYNC_MSG" ]]; then state_set certsync "$CERTSYNC_MSG"; send_notice "Ergo TLS sync FAILED" "high" "$CERTSYNC_MSG"; fi
|
|
return 1 ;;
|
|
esac
|
|
return 0
|
|
}
|
|
|
|
cert_expiry_check() { # daily: warn once per day if the served cert is self-signed or expiring
|
|
local f="$IRCD_DIR/fullchain.pem" today msg=""
|
|
[[ -f "$f" ]] || return 0
|
|
today="$(date -u +%F)"
|
|
if cert_is_selfsigned "$f"; then
|
|
msg="Ergo is serving a SELF-SIGNED certificate on 6697 (Caddy has not issued one yet: check DNS for $(ergo_domain) and ports 80/443)."
|
|
elif cert_expires_within "$f" $((14 * 86400)); then
|
|
msg="Ergo's TLS cert expires soon ($(cert_enddate "$f")) and certsync has not replaced it -- is the 15-minute job running? ($SELF certsync)"
|
|
fi
|
|
[[ -n "$msg" ]] || { state_rm certexpiry; return 0; }
|
|
warn "$msg"
|
|
[[ "$(state_get certexpiry)" == "$today" ]] && return 0
|
|
state_set certexpiry "$today"
|
|
send_notice "Ergo TLS attention" "high" "$msg"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Caddy: pull + recreate when the image moved (drops web-client websockets)
|
|
# ---------------------------------------------------------------------------
|
|
do_caddy() {
|
|
preflight
|
|
local tag before after
|
|
tag="$(env_get CADDY_TAG)"; tag="${tag:-2-alpine}"
|
|
before="$(docker inspect --format '{{.Image}}' ergo-caddy 2>/dev/null | head -n1 || true)"
|
|
log "Pulling caddy:${tag}..."
|
|
docker pull -q "caddy:${tag}" >/dev/null 2>&1 || { warn "pull failed"; return 1; }
|
|
after="$(docker image inspect --format '{{.Id}}' "caddy:${tag}" 2>/dev/null | head -n1 || true)"
|
|
if [[ -n "$before" && "$before" == "$after" ]]; then log "Caddy is current (${tag})."; return 0; fi
|
|
[[ "$DRY_RUN" == "1" ]] && { echo "DRY: docker compose up -d --no-deps caddy"; return 0; }
|
|
log "Recreating Caddy (web-client websockets will reconnect)..."
|
|
dc up -d --no-deps caddy >/dev/null 2>&1 || true
|
|
if wait_healthy caddy 60; then
|
|
send_notice "Caddy updated" "min" "caddy:${tag} recreated (image $(printf '%s' "$after" | cut -c8-19))"
|
|
return 0
|
|
fi
|
|
send_notice "Caddy update FAILED" "high" "caddy:${tag} is not healthy after recreate; check docker compose logs caddy"
|
|
return 1
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Subcommands
|
|
# ---------------------------------------------------------------------------
|
|
preflight() {
|
|
[[ $EUID -eq 0 ]] || die "Run as root."
|
|
[[ -f "$STACK_DIR/docker-compose.yml" ]] || die "No stack at $STACK_DIR (set STACK_DIR)."
|
|
command -v docker >/dev/null 2>&1 || die "docker not found."
|
|
command -v jq >/dev/null 2>&1 || die "jq is required (release metadata); install it."
|
|
}
|
|
|
|
# With HISTORY=postgres, Ergo will not start if its history backend is down --
|
|
# so an update must not stop a healthy server until the database is confirmed up.
|
|
# The config validator cannot check this: it runs with no network on purpose.
|
|
pg_gate() { # <what we are about to do>
|
|
pg_enabled || return 0
|
|
pg_check && return 0
|
|
warn "PostgreSQL is not ready ($PG_CHECK_MSG); skipping $1 -- Ergo would not restart."
|
|
send_notice "Ergo update skipped" "high" "PostgreSQL not ready (${PG_CHECK_MSG}); ${1} was not attempted."
|
|
return 1
|
|
}
|
|
|
|
do_check() {
|
|
preflight
|
|
local cur latest ghsa held="" prio="min" note
|
|
cur="$(ergo_version_running)"
|
|
pin_floating_tag "$cur"
|
|
latest="$(normver "$(ergo_latest_tag || true)")"
|
|
log "Running: ${cur:-unknown} (pinned $(ergo_tag)) | latest release: ${latest:-unknown} | policy: ${UPDATE_POLICY}"
|
|
ghsa="$(ghsa_covering "$cur" || true)"
|
|
if [[ -n "$ghsa" ]]; then
|
|
note="VULNERABLE: $(printf '%s\n' "$ghsa" | awk '{print $1}' | paste -sd, -) covers ${cur}"; prio="high"; warn "$note"
|
|
else
|
|
note="no published advisory covers ${cur}"; log "$note"
|
|
fi
|
|
if [[ -n "$cur" && -n "$latest" ]] && ver_gt "$latest" "$cur"; then
|
|
resolve_target "$cur"
|
|
log "A newer release is available: ${cur} -> ${latest}"
|
|
[[ "$prio" == "min" ]] && prio="default"
|
|
note="${note}; newer release ${latest} available"
|
|
if [[ "$HOLD" == 1 ]]; then held=" -- HELD: release notes announce compatibility breaks (review, then FORCE_UPDATE=1 or 'ergoctl update update')"; warn "${held# -- }"; fi
|
|
[[ -n "$NOTES" ]] && log "Release notes: $NOTES"
|
|
send_notice "Ergo check" "$prio" "${note}${held}"
|
|
fi
|
|
local f="$IRCD_DIR/fullchain.pem"
|
|
if [[ -f "$f" ]]; then
|
|
if cert_is_selfsigned "$f"; then warn "TLS: serving a SELF-SIGNED cert (no Let's Encrypt cert from Caddy yet)."
|
|
else log "TLS: $(cert_subject_cn "$f"), expires $(cert_enddate "$f") (served: $( [[ "$(served_fingerprint)" == "$(cert_fingerprint "$f")" ]] && echo matches || echo DIFFERS -- rehash needed ))"; fi
|
|
fi
|
|
}
|
|
|
|
do_run() {
|
|
preflight
|
|
if [[ "$DRY_RUN" != "1" ]]; then
|
|
# Create the log directory only when it is missing. `install -d -m` also
|
|
# re-modes an EXISTING directory, and this resolves to /var/log, which on
|
|
# Debian/Ubuntu ships 0775 root:syslog so rsyslog (running as `syslog`)
|
|
# can create files there -- re-moding it to 0755 silently breaks logging.
|
|
[[ -d "$(dirname "$LOG")" ]] || install -d -m 0755 "$(dirname "$LOG")" 2>/dev/null || true
|
|
echo "=== ergo-update $(date -u +%FT%TZ) ===" >> "$LOG"
|
|
# Scheduled runs have nowhere to send stdout (busybox crond would try to
|
|
# mail it and these hosts have no MTA), so keep the whole run in the log.
|
|
[[ -t 1 ]] || exec >>"$LOG" 2>&1
|
|
fi
|
|
do_certsync || true
|
|
cert_expiry_check || true
|
|
local cur
|
|
cur="$(ergo_version_running)"
|
|
pin_floating_tag "$cur"
|
|
resolve_target "$cur"
|
|
log "current=${cur:-?} | ${REASON}"
|
|
if [[ -n "$TARGET" && -n "$cur" ]] && ver_gt "$TARGET" "$cur"; then
|
|
if [[ "$(state_get failed)" == "$TARGET" && "$FORCE_UPDATE" != "1" ]]; then
|
|
log "Skipping ${TARGET}: it already failed here and was rolled back. Investigate, then run 'ergoctl update update' (or set FORCE_UPDATE=1)."
|
|
elif ! pg_gate "the update to ${TARGET}"; then
|
|
: # postgres is down; pg_gate already warned and notified
|
|
elif [[ "$HOLD" == 1 && "$FORCE_UPDATE" != "1" ]]; then
|
|
warn "HELD: ${TARGET} announces compatibility breaks. Review the release notes, then 'ergoctl update update' (or FORCE_UPDATE=1)."
|
|
if [[ "$(state_get held)" != "$TARGET" ]]; then
|
|
state_set held "$TARGET"
|
|
send_notice "Ergo update HELD" "default" "${cur} -> ${TARGET} announces compatibility breaks; not applied automatically. Notes: ${NOTES:-see GitHub release}. Apply with 'ergoctl update update'."
|
|
fi
|
|
else
|
|
state_rm held
|
|
apply_update "$cur" "$TARGET" "$REASON" || true
|
|
fi
|
|
else
|
|
log "No update to apply."
|
|
fi
|
|
if [[ "$CADDY_AUTOUPDATE" == "1" ]]; then do_caddy || true; fi
|
|
}
|
|
|
|
do_update() {
|
|
preflight
|
|
local cur
|
|
cur="$(ergo_version_running)"
|
|
[[ -n "$cur" ]] || die "Cannot tell which version is running (container down and ERGO_TAG='$(ergo_tag)' is not a version). Start Ergo, or pin ERGO_TAG=vX.Y.Z in $ENV_FILE first."
|
|
pin_floating_tag "$cur"
|
|
if [[ -z "$TARGET_VERSION" && "$UPDATE_POLICY" == "off" ]]; then UPDATE_POLICY=latest; fi
|
|
resolve_target "$cur"
|
|
[[ -n "$TARGET" ]] || die "Could not determine a target version (${REASON})."
|
|
if [[ -n "$cur" ]] && ! ver_gt "$TARGET" "$cur" && [[ -z "$TARGET_VERSION" ]]; then
|
|
log "Already on ${cur} (latest ${TARGET}); nothing to do."
|
|
return 0
|
|
fi
|
|
if [[ "$HOLD" == 1 && "$FORCE_UPDATE" != "1" ]]; then
|
|
warn "Release notes between ${cur} and ${TARGET} announce compatibility breaks: ${NOTES}"
|
|
warn "Proceeding because you asked explicitly (this is 'update update'). Set FORCE_UPDATE=1 to also auto-apply such releases."
|
|
fi
|
|
pg_gate "the update to ${TARGET}" || die "Start PostgreSQL first (docker compose up -d postgres), then retry."
|
|
apply_update "$cur" "$TARGET" "${REASON} (manual)"
|
|
}
|
|
|
|
# Only what the scheduled job needs to FIND the stack lives here. The update
|
|
# knobs (UPDATE_POLICY, FORCE_UPDATE, UPDATE_GRACE, CADDY_AUTOUPDATE) are read
|
|
# from the stack's .env so there is a single place to edit them; setting one
|
|
# here (or in the environment) still overrides .env for that run.
|
|
write_conf() {
|
|
cat > "$ERGO_UPDATE_CONF" <<CONF
|
|
# Where the scheduled Ergo jobs find the stack (deployments/ergo/update.sh).
|
|
# The update knobs live in \${STACK_DIR}/.env -- edit them there. Anything set
|
|
# here overrides .env, so leave it alone unless you mean to pin a value.
|
|
STACK_DIR="${STACK_DIR}"
|
|
NOTIFY="${NOTIFY}"
|
|
CONF
|
|
chmod 644 "$ERGO_UPDATE_CONF"
|
|
log "Wrote $ERGO_UPDATE_CONF (update knobs come from ${STACK_DIR}/.env)"
|
|
}
|
|
|
|
do_install() {
|
|
[[ $EUID -eq 0 ]] || die "Run as root."
|
|
write_conf
|
|
case "$(osfam)" in
|
|
alpine)
|
|
install -d -m 0755 /etc/periodic/15min /etc/periodic/daily
|
|
printf '#!/bin/sh\nexec bash "%s" certsync\n' "$SELF" > /etc/periodic/15min/ergo-certsync
|
|
chmod +x /etc/periodic/15min/ergo-certsync
|
|
if [[ "$SCHEDULE_UPDATES" == "1" ]]; then
|
|
printf '#!/bin/sh\nexec bash "%s" run\n' "$SELF" > /etc/periodic/daily/ergo-update
|
|
chmod +x /etc/periodic/daily/ergo-update
|
|
else
|
|
rm -f /etc/periodic/daily/ergo-update
|
|
fi
|
|
if command -v rc-update >/dev/null 2>&1; then
|
|
rc-update add crond default >/dev/null 2>&1 || true
|
|
rc-service crond status >/dev/null 2>&1 || rc-service crond start >/dev/null 2>&1 || true
|
|
fi
|
|
log "Scheduled: /etc/periodic/15min/ergo-certsync$( [[ "$SCHEDULE_UPDATES" == "1" ]] && printf ', /etc/periodic/daily/ergo-update (policy=%s)' "$UPDATE_POLICY" )." ;;
|
|
*)
|
|
cat > /etc/systemd/system/ergo-certsync.service <<EOF
|
|
[Unit]
|
|
Description=Ergo: sync Caddy's TLS cert into the IRC server
|
|
After=docker.service
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
ExecStart=/usr/bin/env bash $SELF certsync
|
|
EOF
|
|
cat > /etc/systemd/system/ergo-certsync.timer <<EOF
|
|
[Unit]
|
|
Description=Ergo cert sync every 15 minutes
|
|
|
|
[Timer]
|
|
OnCalendar=*:0/15
|
|
Persistent=true
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|
|
EOF
|
|
cat > /etc/systemd/system/ergo-update.service <<EOF
|
|
[Unit]
|
|
Description=Ergo container updater
|
|
After=docker.service
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
ExecStart=/usr/bin/env bash $SELF run
|
|
EOF
|
|
cat > /etc/systemd/system/ergo-update.timer <<EOF
|
|
[Unit]
|
|
Description=Daily Ergo update check
|
|
|
|
[Timer]
|
|
OnCalendar=daily
|
|
Persistent=true
|
|
RandomizedDelaySec=1h
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|
|
EOF
|
|
systemctl daemon-reload
|
|
systemctl enable --now ergo-certsync.timer >/dev/null 2>&1 || true
|
|
if [[ "$SCHEDULE_UPDATES" == "1" ]]; then systemctl enable --now ergo-update.timer >/dev/null 2>&1 || true
|
|
else systemctl disable --now ergo-update.timer >/dev/null 2>&1 || true; fi
|
|
log "Scheduled: ergo-certsync.timer (15 min)$( [[ "$SCHEDULE_UPDATES" == "1" ]] && printf ', ergo-update.timer (daily, policy=%s)' "$UPDATE_POLICY" )." ;;
|
|
esac
|
|
}
|
|
|
|
do_uninstall() {
|
|
[[ $EUID -eq 0 ]] || die "Run as root."
|
|
rm -f /etc/periodic/15min/ergo-certsync /etc/periodic/daily/ergo-update
|
|
if command -v systemctl >/dev/null 2>&1; then
|
|
systemctl disable --now ergo-certsync.timer ergo-update.timer >/dev/null 2>&1 || true
|
|
rm -f /etc/systemd/system/ergo-certsync.{timer,service} /etc/systemd/system/ergo-update.{timer,service}
|
|
systemctl daemon-reload >/dev/null 2>&1 || true
|
|
fi
|
|
log "Removed the scheduled Ergo jobs (config kept at $ERGO_UPDATE_CONF)."
|
|
}
|
|
|
|
case "${1:-check}" in
|
|
check) do_check ;;
|
|
run) do_run ;;
|
|
update) do_update ;;
|
|
certsync) preflight; do_certsync ;;
|
|
caddy) do_caddy ;;
|
|
install) do_install ;;
|
|
uninstall) do_uninstall ;;
|
|
*) die "Usage: update.sh [check|run|update|certsync|caddy|install|uninstall]" ;;
|
|
esac
|