Files
automations/deployments/openbao/update.sh
T
57_WolveandClaude Opus 5 2d3d322b0e feat(openbao): add update.sh, an updater that refuses to strand a sealed vault
Follows the copyparty/ergo updater idiom -- check/update/run/install/uninstall,
a conf file the environment overrides, the version pinned into .env so the
running release is explicit, DRY_RUN -- but inverts its central assumption.

copyparty and ergo come back by themselves after a recreate. OpenBao comes back
SEALED: with the default Shamir seal a restart needs three unseal keys typed in
by a human. A scheduled `latest` update would therefore take the vault offline
at 03:00 and leave it there. So the scheduled path defaults to UPDATE_POLICY=
notify and never changes the running version; `install` schedules a daily CHECK
and says so. UPDATE_POLICY=auto opts in, and is STILL refused unless an
uncommented `seal` stanza is present in config.hcl -- only auto-unseal makes an
unattended update defensible.

`update` preflights before touching anything, because every one of these fails
worse halfway through than up front:
- the container must be running;
- the vault must be UNSEALED, since a sealed vault cannot produce a snapshot and
  there would be no rollback plan;
- BAO_TOKEN must be present, because the snapshot is token-gated on
  sys/storage/raft/snapshot;
- the target must not cross into 2.7.x while a built-in seal "pkcs11" stanza is
  active. That stanza is REMOVED in 2.7.0, not deprecated, so the vault would
  start with no way to unseal at all.

The snapshot is the rollback plan, not a formality: OpenBao's upgrade guide
states that reverting the image alone does not roll back the data store. It is
streamed out with `exec -T ... cat` rather than `compose cp`, which emits a TAR
wrapper that will not restore; written 0600 to /var/backups/openbao; and checked
for being a non-empty valid gzip archive, with a failure treated as fatal.
SKIP_SNAPSHOT=1 exists and warns exactly what it costs.

A failed pull or start rolls the OPENBAO_TAG pin back and restarts the previous
version. BAO_TOKEN is deliberately never written to the conf file: a long-lived
root token sitting next to the vault it opens defeats the vault.

Verified: the 2.7-with-active-pkcs11 refusal and its three negative cases
(2.6.2 active, 3.x active, 2.7 commented); auto-unseal detection distinguishing
a commented stanza from a live one; set_env/env_get; the release-tag parse with
and without a leading v; and all four `run` policy branches, including that
auto refuses without auto-unseal and that an unknown policy dies.

Not verified without a live host: the snapshot, pull and recreate themselves.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 09:21:11 -05:00

448 lines
17 KiB
Bash

#!/usr/bin/env bash
#
# update.sh -- update the OpenBao container. Companion to deploy.sh, installed
# alongside the stack.
#
# THIS UPDATER IS DELIBERATELY UNLIKE THE OTHERS IN THIS REPO. copyparty and
# ergo come back by themselves after a recreate. OpenBao comes back SEALED: with
# the default Shamir seal, a restart needs three unseal keys typed in by a human
# before the vault serves anything again. A scheduled update at 03:00 would
# therefore take the vault offline until someone turns up with the keys.
#
# So the scheduled path NEVER changes the running version by default. It reports.
# UPDATE_POLICY=auto opts in, and even then it refuses unless auto-unseal is
# configured -- because only then does the vault come back on its own.
#
# It also takes a raft snapshot before touching anything, which neither of the
# other updaters needs to do. OpenBao's upgrade guide is explicit that reverting
# the image alone does NOT roll back the data store, so that snapshot is the
# rollback plan, not a formality. The snapshot is token-gated and cannot be taken
# from a sealed vault, both of which this script checks up front rather than
# failing halfway through.
#
# Subcommands:
# check (default) declared vs running vs latest, seal state, and whether
# an unattended update would be safe here. Changes nothing.
# update do it now: preflight -> snapshot -> down -> pin -> pull -> up
# run what the schedule invokes; obeys UPDATE_POLICY
# install schedule the daily `run`
# uninstall remove the schedule
#
# Policy (UPDATE_POLICY):
# notify (default) never change the running version; report only
# auto update when a newer release exists -- but ONLY if auto-unseal is
# configured. Without it, `run` reports and does nothing.
#
# Env (also read from /etc/openbao-update.conf; environment wins):
# STACK_DIR=/srv/openbao UPDATE_POLICY=notify
# BAO_TOKEN= required by `update`: the snapshot is token-gated
# SNAPSHOT_DIR=/var/backups/openbao
# TARGET_VERSION= pin a specific version instead of latest
# DRY_RUN=0 print what would happen, change nothing
# SKIP_SNAPSHOT=0 DANGEROUS: upgrade with no rollback plan
# GH_REPO=openbao/openbao
#
# Usage:
# bash update.sh check
# BAO_TOKEN=<root> bash update.sh update
# BAO_TOKEN=<root> TARGET_VERSION=2.6.2 bash update.sh update
set -euo pipefail
SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
: "${OPENBAO_UPDATE_CONF:=/etc/openbao-update.conf}"
if [[ -r "$OPENBAO_UPDATE_CONF" ]]; then
# Environment wins over the conf: only set what is not already set.
while IFS= read -r _line || [[ -n "$_line" ]]; do
[[ "$_line" =~ ^[[:space:]]*# || -z "${_line//[[:space:]]/}" ]] && continue
_k="${_line%%=*}"; _v="${_line#*=}"; _k="${_k//[[:space:]]/}"
[[ -n "$_k" ]] || continue
[[ -n "${!_k+x}" ]] || printf -v "$_k" '%s' "$_v"
done < "$OPENBAO_UPDATE_CONF"
fi
: "${STACK_DIR:=/srv/openbao}"
: "${UPDATE_POLICY:=notify}"
: "${BAO_TOKEN:=}"
: "${SNAPSHOT_DIR:=/var/backups/openbao}"
: "${TARGET_VERSION:=}"
: "${DRY_RUN:=0}"
: "${SKIP_SNAPSHOT:=0}"
: "${GH_REPO:=openbao/openbao}"
ENV_FILE="$STACK_DIR/.env"
CONFIG_HCL="$STACK_DIR/config.hcl"
log() { printf '\033[1;32m[+]\033[0m %s\n' "$*"; }
warn() { printf '\033[1;33m[!]\033[0m %s\n' "$*" >&2; }
die() { printf '\033[1;31m[x]\033[0m %s\n' "$*" >&2; exit 1; }
# docker compose scoped to the stack dir, so ./config.hcl, ./tls and .env resolve.
dc() { ( cd "$STACK_DIR" && docker compose "$@" ); }
set_env() { # <KEY> <value>: update KEY in .env, or append if absent
# Value goes through the ENVIRONMENT, never interpolated into a sed script --
# see 947c899 / 185f404 for why.
local key="$1" val="$2" tmp
if [[ ! -f "$ENV_FILE" ]]; then
printf '%s=%s\n' "$key" "$val" >> "$ENV_FILE"
return 0
fi
tmp="$(mktemp)"
_SE_KEY="$key" _SE_VAL="$val" awk '
BEGIN { k = ENVIRON["_SE_KEY"]; v = ENVIRON["_SE_VAL"]; seen = 0 }
!seen && index($0, k "=") == 1 { print k "=" v; seen = 1; next }
{ print }
END { if (!seen) print k "=" v }
' "$ENV_FILE" > "$tmp"
cat "$tmp" > "$ENV_FILE"
rm -f "$tmp"
}
env_get() { # <KEY> -> value from .env ('' if absent)
[[ -f "$ENV_FILE" ]] || { printf ''; return 0; }
sed -n "s/^$1=//p" "$ENV_FILE" | tail -n1
}
# ---------------------------------------------------------------------------
# State
# ---------------------------------------------------------------------------
declared_version() { env_get OPENBAO_TAG; }
running_version() {
# `bao status` prints "Version 2.6.2". Works sealed or unsealed.
# `|| true` inside the substitution: status exits 2 when sealed, and this is
# a bare assignment at every call site.
dc exec -T openbao bao status -address=https://127.0.0.1:8200 -tls-skip-verify 2>/dev/null \
| awk '/^Version/ { print $2; exit }' || true
}
# 0 = unsealed, 2 = sealed, anything else = could not tell.
seal_state() {
local rc=0
dc exec -T openbao bao status -address=https://127.0.0.1:8200 -tls-skip-verify \
>/dev/null 2>&1 || rc=$?
printf '%s' "$rc"
}
container_running() {
local id
id="$(dc ps -q openbao 2>/dev/null || true)"
[[ -n "$id" ]] || return 1
[[ "$(docker inspect -f '{{.State.Status}}' "$id" 2>/dev/null || echo unknown)" == running ]]
}
# An uncommented `seal "..." {` stanza means the vault unseals itself, which is
# the only condition under which an unattended update is defensible.
auto_unseal_configured() {
[[ -f "$CONFIG_HCL" ]] || return 1
grep -qE '^[[:space:]]*seal[[:space:]]+"' "$CONFIG_HCL"
}
latest_version() {
# No jq on a stock Alpine host; parse the tag out with sed.
curl -fsSL --max-time 20 "https://api.github.com/repos/${GH_REPO}/releases/latest" 2>/dev/null \
| sed -n 's/.*"tag_name"[[:space:]]*:[[:space:]]*"v\{0,1\}\([^"]*\)".*/\1/p' \
| head -n1 || true
}
# ---------------------------------------------------------------------------
# Preflight + snapshot
# ---------------------------------------------------------------------------
# The built-in pkcs11 seal is REMOVED in 2.7.0 (not merely deprecated), and the
# HSM distribution is discontinued. Crossing that line with the stanza active
# gives a vault that cannot unseal itself and cannot be unsealed by hand either.
check_pkcs11_cliff() { # <target-version>
local target="$1" major_minor
major_minor="${target%.*}"
case "$major_minor" in
2.7|2.8|2.9|3.*) ;;
*) return 0 ;;
esac
if [[ -f "$CONFIG_HCL" ]] && grep -qE '^[[:space:]]*seal[[:space:]]+"pkcs11"' "$CONFIG_HCL"; then
die "config.hcl has an active built-in seal \"pkcs11\" stanza and the target is ${target}. That stanza is REMOVED in 2.7.0 -- the vault would start with no way to unseal. Migrate to the external 'plugin \"kms\" \"pkcs11\"' first."
fi
}
take_snapshot() { # <target-version> -> echoes the snapshot path
local target="$1" stamp dest
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
dest="${SNAPSHOT_DIR}/openbao-pre-${target}-${stamp}.snap"
if [[ "$SKIP_SNAPSHOT" == "1" ]]; then
warn "SKIP_SNAPSHOT=1 -- upgrading with NO rollback plan. Reverting the image alone does not roll back the data store."
printf ''
return 0
fi
[[ -n "$BAO_TOKEN" ]] || die "BAO_TOKEN is required: the raft snapshot is token-gated (sys/storage/raft/snapshot). The root token works."
if [[ "$DRY_RUN" == "1" ]]; then
echo "DRY: snapshot -> ${dest}" >&2
printf '%s' "$dest"
return 0
fi
install -d -m 0700 "$SNAPSHOT_DIR"
log "Taking a raft snapshot to ${dest}..." >&2
# Write inside the container, then stream the RAW file out. `compose cp` is
# NOT usable here: it emits a TAR wrapper that will not restore.
dc exec -T -e BAO_TOKEN="$BAO_TOKEN" openbao \
bao operator raft snapshot save -address=https://127.0.0.1:8200 -tls-skip-verify /tmp/pre-upgrade.snap \
>/dev/null || die "Snapshot failed -- not upgrading. Check the token has sys/storage/raft/snapshot."
dc exec -T openbao cat /tmp/pre-upgrade.snap > "$dest" || die "Could not stream the snapshot out -- not upgrading."
dc exec -T openbao rm -f /tmp/pre-upgrade.snap >/dev/null 2>&1 || true
chmod 0600 "$dest"
# A snapshot is a gzip-wrapped tar. An empty or truncated file here means the
# rollback plan does not exist, so treat it as fatal rather than cosmetic.
[[ -s "$dest" ]] || die "Snapshot at ${dest} is empty -- not upgrading."
if command -v gzip >/dev/null 2>&1; then
gzip -t "$dest" 2>/dev/null || die "Snapshot at ${dest} is not a valid gzip archive -- not upgrading."
fi
log "Snapshot OK ($(wc -c < "$dest") bytes). Copy it OFF this host." >&2
printf '%s' "$dest"
}
# ---------------------------------------------------------------------------
# Subcommands
# ---------------------------------------------------------------------------
do_check() {
[[ -d "$STACK_DIR" ]] || die "No stack at $STACK_DIR (set STACK_DIR)."
local declared running latest state auto
declared="$(declared_version)"
latest="$(latest_version)"
if container_running; then
running="$(running_version)"
state="$(seal_state)"
else
running=""
state="down"
fi
auto_unseal_configured && auto=yes || auto=no
printf ' declared (.env OPENBAO_TAG): %s\n' "${declared:-<unset>}"
printf ' running (bao status): %s\n' "${running:-<not running>}"
printf ' latest (%s): %s\n' "$GH_REPO" "${latest:-<lookup failed>}"
case "$state" in
0) printf ' seal state: UNSEALED\n' ;;
2) printf ' seal state: SEALED (needs 3 keys)\n' ;;
down) printf ' seal state: container not running\n' ;;
*) printf ' seal state: unknown (bao status rc=%s)\n' "$state" ;;
esac
printf ' auto-unseal configured: %s\n' "$auto"
printf ' policy: %s\n' "$UPDATE_POLICY"
if [[ -z "$latest" ]]; then
warn "Could not reach the GitHub releases API; cannot say whether an update exists."
return 0
fi
if [[ "$declared" == "$latest" ]]; then
log "Up to date."
return 0
fi
log "Update available: ${declared:-?} -> ${latest}"
if [[ "$auto" == no ]]; then
warn "A restart will leave the vault SEALED until someone enters three unseal keys."
warn "Run it when you can do that: BAO_TOKEN=<token> bash ${SELF} update"
fi
}
do_update() {
[[ $EUID -eq 0 ]] || die "Run as root."
[[ -d "$STACK_DIR" ]] || die "No stack at $STACK_DIR (set STACK_DIR)."
local from to snap
from="$(declared_version)"
to="${TARGET_VERSION:-$(latest_version)}"
[[ -n "$to" ]] || die "Could not determine a target version (GitHub lookup failed). Set TARGET_VERSION=x.y.z."
if [[ "$from" == "$to" ]]; then
log "Already pinned to ${to}; nothing to do."
return 0
fi
check_pkcs11_cliff "$to"
container_running || die "The openbao container is not running. Start it first: cd ${STACK_DIR} && docker compose up -d"
# A sealed vault cannot produce a snapshot, so there would be no rollback
# plan. Catch it here rather than after the container is already down.
local state; state="$(seal_state)"
case "$state" in
0) ;;
2) die "The vault is SEALED. A snapshot cannot be taken from a sealed vault, so there would be no rollback plan. Unseal first, then re-run." ;;
*) die "Could not read the seal state (bao status rc=${state}). Refusing to upgrade blind." ;;
esac
log "Updating OpenBao: ${from:-?} -> ${to}"
snap="$(take_snapshot "$to")"
if [[ "$DRY_RUN" == "1" ]]; then
echo "DRY: docker compose down; set OPENBAO_TAG=${to}; docker compose pull; docker compose up -d"
return 0
fi
# Clean shutdown before swapping the image, as the upgrade guide asks.
log "Stopping the stack (the vault will be sealed when it returns)..."
dc down
set_env OPENBAO_TAG "$to"
if ! dc pull; then
warn "Pull of ${to} failed; rolling the pin back to ${from:-<unset>}."
[[ -n "$from" ]] && set_env OPENBAO_TAG "$from"
dc up -d || true
die "Update aborted. The previous version is starting again; unseal it."
fi
if ! dc up -d --remove-orphans; then
warn "Starting ${to} failed; rolling the pin back to ${from:-<unset>}."
[[ -n "$from" ]] && set_env OPENBAO_TAG "$from"
dc up -d || true
die "Update aborted. If the data store is at fault, restore ${snap:-the snapshot} per the README's DR runbook."
fi
log "Started ${to}."
cat <<EOF
================================================================
UPDATED ${from:-?} -> ${to}
The vault is SEALED. Nothing works until you unseal it:
cd ${STACK_DIR}
docker compose exec -e BAO_ADDR=https://127.0.0.1:8200 openbao \\
bao operator unseal -tls-skip-verify # x3, three different keys
Then verify:
docker compose exec -e BAO_ADDR=https://127.0.0.1:8200 openbao \\
bao status -tls-skip-verify # Version ${to}, Sealed false
Rollback, if ${to} misbehaves: reverting the image alone does NOT roll back the
data store. Restore the pre-upgrade snapshot -- see the README's DR runbook.
snapshot: ${snap:-<none taken>}
================================================================
EOF
}
do_run() {
# What the schedule invokes. The whole point of this branch is that it is
# conservative: an unattended update of a manually-unsealed vault takes it
# offline until a human arrives, so that is never the default.
case "$UPDATE_POLICY" in
notify)
do_check
return 0 ;;
auto)
if ! auto_unseal_configured; then
do_check
warn "UPDATE_POLICY=auto but no seal stanza is configured, so the vault would stay SEALED after a restart with nobody present. Not updating."
return 0
fi
local from to
from="$(declared_version)"
to="${TARGET_VERSION:-$(latest_version)}"
if [[ -z "$to" || "$from" == "$to" ]]; then
do_check
return 0
fi
log "UPDATE_POLICY=auto and auto-unseal is configured; updating ${from:-?} -> ${to}."
do_update
return 0 ;;
*)
die "UPDATE_POLICY must be 'notify' or 'auto' (got '${UPDATE_POLICY}')." ;;
esac
}
write_conf() {
install -d -m 0755 "$(dirname "$OPENBAO_UPDATE_CONF")"
# BAO_TOKEN is deliberately NOT written here: a long-lived root token sitting
# in a conf file next to the vault it opens defeats the point of the vault.
cat > "$OPENBAO_UPDATE_CONF" <<EOF
# openbao-update.conf -- read by update.sh; the environment wins over this file.
STACK_DIR=${STACK_DIR}
UPDATE_POLICY=${UPDATE_POLICY}
SNAPSHOT_DIR=${SNAPSHOT_DIR}
GH_REPO=${GH_REPO}
# BAO_TOKEN is intentionally absent. \`update\` needs one for the snapshot; pass
# it in the environment for that run. UPDATE_POLICY=auto therefore only works
# with auto-unseal AND a token supplied some other way.
EOF
chmod 0600 "$OPENBAO_UPDATE_CONF"
}
do_install() {
[[ $EUID -eq 0 ]] || die "Run as root."
write_conf
if [[ "$UPDATE_POLICY" == notify ]]; then
log "Scheduling a daily CHECK (policy=notify: it will never change the running version)."
else
log "Scheduling a daily run (policy=${UPDATE_POLICY})."
fi
if [[ -r /etc/os-release ]] && grep -q '^ID=alpine' /etc/os-release; then
install -d -m 0755 /etc/periodic/daily
cat > /etc/periodic/daily/openbao-update <<EOF
#!/bin/sh
exec bash "$SELF" run
EOF
chmod +x /etc/periodic/daily/openbao-update
if command -v rc-update >/dev/null 2>&1; then
rc-update add crond default >/dev/null 2>&1 || true
rc-service crond start >/dev/null 2>&1 || true
fi
log "Installed /etc/periodic/daily/openbao-update."
else
cat > /etc/systemd/system/openbao-update.service <<EOF
[Unit]
Description=OpenBao container updater
After=docker.service
[Service]
Type=oneshot
ExecStart=/usr/bin/env bash $SELF run
EOF
cat > /etc/systemd/system/openbao-update.timer <<EOF
[Unit]
Description=Daily OpenBao update check
[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=1h
[Install]
WantedBy=timers.target
EOF
systemctl daemon-reload
systemctl enable --now openbao-update.timer >/dev/null 2>&1 || true
log "Installed the openbao-update.timer systemd timer."
fi
}
do_uninstall() {
[[ $EUID -eq 0 ]] || die "Run as root."
rm -f /etc/periodic/daily/openbao-update
if command -v systemctl >/dev/null 2>&1; then
systemctl disable --now openbao-update.timer >/dev/null 2>&1 || true
rm -f /etc/systemd/system/openbao-update.timer /etc/systemd/system/openbao-update.service
systemctl daemon-reload >/dev/null 2>&1 || true
fi
log "Schedule removed. ${OPENBAO_UPDATE_CONF} left in place."
}
case "${1:-check}" in
check) do_check ;;
update) do_update ;;
run) do_run ;;
install) do_install ;;
uninstall) do_uninstall ;;
*) die "Usage: $(basename "$0") {check|update|run|install|uninstall}" ;;
esac