Two things that cost real debugging time on a live deploy.
A Docker-published port on a Proxmox guest is filtered by the per-VM firewall
on the PVE host, upstream of everything in the guest. The failure is
maximally misleading: inside the VM the DNAT rule, the FORWARD jumps, the
docker-proxy socket and a local curl to the bind address are all correct and
the listener answers 503, while a LAN client times out. SSH working proves
nothing -- it only proves a rule exists for 22. The decisive test is
`tcpdump -ni eth0 'tcp port 8200'` capturing zero packets during a failed
connection, which says the guest never saw them.
And: the publish is pinned to OPENBAO_BIND, so a browser on another subnet
cannot reach the UI. Document the SSH tunnel rather than widening the publish
-- the generated cert already carries DNS:localhost + IP:127.0.0.1, so
https://localhost:8200 validates against it unchanged. That is also the
better way to run the first `operator init`, since the unseal keys are then
shown in a browser instead of a root shell's scrollback.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
947c899 removed this idiom from copyparty, headscale, pocket-id and squid;
openbao was missed. Its first-run seed still interpolated values into
`sed -i -e "s|^KEY=.*|KEY=${VAL}|"`, which silently corrupts any value
containing & (sed expands it to the whole match) and aborts the run under
set -e on one containing the s||| delimiter.
That was survivable while .env held only an address, a bind and an image tag.
It stops being survivable the moment an OIDC client secret or an issuer URL
with a query string goes in there, which is the next commit -- so this lands
first, on its own, with no behaviour change.
Adds the same awk/ENVIRON helper the other four use, replaces the seed's sed
block with set_env calls, and collapses the inlined copy of that awk (added
in dc9761a for the UI write-through) into a call to it.
Verified against plain, sec&ret, pipe|val, back\slash, an Authentik-shaped
discovery URL with an & query string, p@ss&w|rd and the empty string; plus
non-target lines left intact and the 0600 mode preserved across an in-place
update.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
config.hcl shipped `ui = false`, so there was no UI to reach even with working
connectivity. That forces init and unseal through `docker compose exec`, which
puts the unseal keys and root token into a server shell's history and
scrollback -- the one place they should never be.
Make it a substitution point driven by OPENBAO_UI, default on. This adds no
exposure: the UI is served on the same listener, and anything that can reach
:8200 can already do everything through the API. What it changes is where the
unseal material is displayed -- the operator's browser instead of the host.
Rendering config.hcl rather than copying it needed three things to be right:
- Precedence. OPENBAO_UI passed to a run wins; otherwise the value .env already
deploys applies; otherwise the default. Without that, `OPENBAO_UI=0 bash
deploy.sh` against an existing node would report success and change nothing.
Env-presence is captured before the ":=" default, so an explicit 0 is
distinguishable from "not mentioned" and a re-run cannot silently re-enable
the UI. A value passed this run is written back to .env.
- Reload. A bind-mounted file's CONTENTS are not part of the compose config
hash, so `up -d` leaves a changed config.hcl unloaded -- the same trap fixed
in copyparty (76d2a09) and simplex (0f22735). Restart only on an actual
change, and say loudly that a restart RE-SEALS the vault, because that is not
a free action on a running one.
- `cmp -s` stays inside an `if` condition; as `cmp -s A B && CONFIG_CHANGED=1`
it would trip set -e whenever the files matched.
Verified: default renders `ui = true` and seeds .env; an identical re-run
reports no change; .env's 0 is honoured when nothing is passed; a passed 1 beats
.env's 0 and is written through; a non-boolean value dies.
Note for LAN-isolated hosts: the listener publishes on OPENBAO_BIND only, so a
browser on another subnet still cannot reach it. The generated cert already
carries DNS:localhost + IP:127.0.0.1 in its SANs, so
`ssh -L 8200:<bind-addr>:8200` and https://localhost:8200 works against the
existing cert without widening the publish.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The image's entrypoint starts as root and then runs `su-exec openbao "$@"`
before exec'ing the server. deploy.sh probed with
`--entrypoint id -u`, which BYPASSES the entrypoint, so it reported UID 0 and
chowned config.hcl, tls/ and the raft volume to root -- while the server ran
as uid 100 and could write none of it:
error initializing storage of type raft: failed to create fsm:
failed to open bolt file: open /openbao/data/vault.db: permission denied
The container crash-looped on that, and tls.key (0600 root:root) would have
failed the listener straight afterwards. Confirmed on the affected host:
the probe reports 0, `id -u openbao` reports 100, and the entrypoint's line
92 is `set -- su-exec openbao "$@"`.
Ask for the account the entrypoint switches to, falling back to the old probe
when the image has no such account (then the server really does run as
whatever the entrypoint started as). Preferring the service account is also
the safe direction to be wrong in: root ignores file permissions, so chowning
to the unprivileged uid still works if the server turns out to run as root,
whereas the reverse is fatal.
Gate the raft chown on the volume's ACTUAL ownership rather than on a
first-run flag. The flag was false forever after the first deploy, so a volume
left root-owned by an earlier run -- exactly the state this bug created --
could never be repaired by re-running; the operator had to chown it by hand.
Reading the owner costs one container start and still keeps the recursive
chown off a healthy live raft dir. FIRST_RUN is now unused, so it is gone.
Also from the same run, two reporting failures:
- `docker compose ps` printed "Restarting (1)" and the script went on to print
an unqualified DEPLOYED banner. It now inspects the container state and, when
it is not `running`, says plainly that this is a crash loop rather than the
expected sealed-and-unhealthy state -- before and after the banner, with the
logs command to run.
- Every `docker compose` command in that banner assumed the project directory.
deploy.sh runs them from $STACK_DIR itself, so the omission only bit the
operator afterwards, with "no configuration file provided: not found". The
banner now says to cd there first and quotes that error.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
host_addrs() ends in a pipeline, so under `set -o pipefail` a probe that
exits non-zero -- even after printing perfectly usable addresses, or
because awk is missing -- made `addrs="$(host_addrs)"` non-zero, and
`set -e` killed deploy.sh at that line.
Nothing was printed when it did: the 2>/dev/null had already swallowed the
tool's own error and the fail-open guard on the next lines was never
reached, so the operator got a bare exit 1 mid-deploy with nothing to
diagnose. Worst case it aborted a deploy whose bind address was CORRECT --
reproduced with an `ip` stub that prints the matching address, then exits 1.
Capture with `|| true` so the emptiness test actually drives the fail-open
the comment beside it already promised. Neutralising inside host_addrs
instead would not cover a missing awk, since pipefail takes the rightmost
non-zero status.
Re-verified the check is not weakened: a healthy probe with the address
genuinely absent still lists the host's addresses and dies with the full
message, and all seven .env/environment precedence cases are unchanged.
Found by adversarial review of 920edc5.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
deploy.sh narrows the Docker publish bind to OPENBAO_ADDR when that is an
IP, but never checked that the host actually holds that address. A typo'd
IP therefore failed late, inside `docker compose up`:
failed to bind host port 192.160.100.50:8200/tcp:
cannot assign requested address
...by which point .env had been seeded and the TLS cert generated with the
bad address in its SAN. Neither is rewritten on a re-run (.env is never
overwritten, gen-tls.sh never regenerates over an existing pair), so
re-running with a corrected OPENBAO_ADDR silently changed nothing.
Add host_addrs() + check_bind_addr(), run before anything is written:
- lists the host's addresses from plain `ip addr show` -- no -o/scope
filters, since busybox ip supports neither -- falling back to ifconfig,
and skipping the check when neither exists rather than blocking;
- skips 0.0.0.0 / :: / *, and unwraps an [IPv6] publish literal;
- SKIP_BIND_CHECK=1 overrides for an address that only comes up later.
Resolve the bind compose will really interpolate, which follows compose's
own precedence -- shell environment before .env:
- exported (automations.sh passes answers via `env VAR=...`, or a
standalone OPENBAO_BIND=... run): the environment wins, so warn when
.env disagrees, because a later bare `docker compose up` would not;
- derived here: that assignment is not exported, so .env wins;
- in neither: compose falls back to 0.0.0.0 and publishes the API on
every interface -- warn, since that is a silent exposure.
Also warn when .env's OPENBAO_ADDR differs from this run's, naming the
cert that has to be deleted for the SAN to be regenerated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An adversarial re-verification of 0812f34 found a couple of the fixes were
incomplete or regressed; addressed here.
- DR was an INCOMPLETE fix: `raft snapshot save`/`restore` are token-gated
(sys/storage/raft/snapshot is sudo-capable) and the container carries no
ambient token, so the previously-"fixed" backup returned "missing client
token" and streamed a zero-byte snapshot. Both the deploy.sh runbook and the
README DR flow now pass `-e BAO_TOKEN=<token>` on save and restore.
- IPv6 bind REGRESSION (introduced by the bind-narrowing): a bare IPv6 literal
in the compose port map (`fd00::10:8200:8200`) is invalid and aborts at
`docker compose pull`. Now IPv4 vs IPv6 are classified separately and IPv6 is
bracketed (`[fd00::10]:8200:8200`).
- Docker readiness race (now reachable since install_docker is actually
called): openrc backgrounds dockerd and returns before the socket is up, so
the next `docker compose pull` raced it under set -e. install_docker now polls
`docker info` for up to 30s.
- UID detection hardened: added `-T` to the one-off `docker compose run`, and an
empty result is now a loud warning (with the manual-chown remedy) instead of a
silent fall-through to root that would re-create the crash-loop on a non-root
image.
- The raft-volume chown is gated to first run (captured before any compose-run
instantiates the volume), so idempotent re-deploys don't recursively re-chown
a live raft dir.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A multi-agent sanity audit of the freshly-merged deployment found four
end-to-end blockers (and several smaller issues); all fixed here.
HIGH (were blocking):
- deploy.sh never called install_docker(), so `docker compose pull` hit
command-not-found on any host without Docker. Now called before the
compose steps.
- The container's server process runs as the image's own (often non-root)
user but the mounted config/TLS were root-owned 0640/0600 and the raft
volume root-owned -> vault crash-looped, never binding :8200. deploy.sh
now detects the image UID after pull and aligns ownership of config.hcl,
./tls and the data volume (a no-op when the image runs as root);
config.hcl is installed 0644 (holds no secrets).
- Docs told operators to set the daemon key `openbao_ca_cert`, but the
Kanrisha daemon's key is `ca_cert` (config.go, mapstructure:"ca_cert").
The wrong key is fatal on strict unmarshal / leaves TLS unverified.
Renamed in all 5 places (config.hcl, gen-tls.sh, deploy.sh x2, README).
- DR backup used `docker compose cp openbao:… -`, which emits a TAR stream,
so the age-encrypted snapshot was tar-wrapped and would not restore.
Switched to `docker compose exec -T openbao cat` for the raw bytes, wrote
the snapshot to a scratch path (not the live raft dir), and documented the
matching restore.
MEDIUM:
- Swap detection used `swapon --show` (absent on BusyBox) and `\s` (GNU-only)
-> silently no-op on Alpine, leaving swap on. Now uses /proc/swaps and
[[:space:]] so mlock hardening actually holds on musl.
- A Docker-published port bypasses the host INPUT firewall, so the source
rule was illusory. deploy.sh now narrows OPENBAO_BIND to OPENBAO_ADDR when
it is an IP, the compose/README/.env comments state the reality, and a new
Exposure section + an init-immediately warning were added.
- Fixed broken ../kanrisha/ and deployments/kanrisha/ links (separate repo).
LOW:
- OPENBAO_TLS_SANS is now honored (folded into the SAN list from the env).
- .gitignore excludes *.snap / *.snap.age.
- Bootstrap note clarifies bootstrap.sh needs the `bao` CLI (run it from the
Kanrisha host/workstation, not this Docker-only vault host).
- README multi-OS count corrected (eight stacks) + automations.sh header
lists openbao.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A dedicated OpenBao deployment, kept OFF the Kanrisha tape host so a compromise
of the tape node can't reach the vault.
- Native TLS on the listener (self-signed by default via gen-tls.sh, or a
CA-signed cert from a Smallstep CA over ACME) — no Caddy/Let's Encrypt;
reached over the LAN, not the public internet.
- Integrated raft storage (clean snapshot-based DR).
- mlock on (cap_add IPC_LOCK + memlock unlimited + host swapoff in deploy.sh).
- Manual unseal by default; optional PKCS#11 HSM auto-unseal.
- deploy.sh: Docker install (Alpine/Debian/Alma), self-signed cert, .env seed,
swapoff, firewall 8200/tcp, compose up; then prints init/unseal + the KV-v2 +
AppRole bootstrap for Kanrisha + the raft-snapshot DR flow. Self-contained
(config payload embedded by build.sh).
- Registered in automations.sh + the README deployment table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
57_WolveandClaude Opus 5