The set_env/env_set helper escaped its value with
`esc=${val//\/\\}; esc=${esc//|/\|}; esc=${esc//&/\&}` and then
interpolated it into `sed -i -e "s|^KEY=.*|KEY=${esc}|"`. The escaping does
not do what it looks like. Tested on bash 5.2:
set_env K 'a&b' -> K=aK=seedb (sed expanded & to the whole match)
set_env K 'a|b' -> sed: unknown option to `s' (rc!=0, aborts under set -e)
So any value containing & is silently corrupted and any value containing the
s||| delimiter kills the run. That is reachable: headscale writes
OIDC_CLIENT_SECRET through this, pocket-id writes REDIRECT_URL, copyparty
writes DATA_DIR. A generated secret or a URL query string hits both cases.
The copies in headscale and pocket-id were additionally mangled when they
were introduced -- `${val//\/\}` (pattern `\/`, a literal SLASH) and a raw
newline inside `printf '%s=%s\n'`. The mangled form is a no-op rather than a
corrupter, so the practical failure mode was the same as the original.
Replace all of them with an awk rewrite that passes the key and value through
the ENVIRONMENT, so the value is never parsed as part of a script and needs no
escaping at all. ENVIRON and index() are POSIX, so busybox awk handles them.
Output goes to a temp file and is copied back with `cat >`, which preserves the
original mode and owner -- a .env holding secrets stays 0600. If awk fails,
set -e aborts before .env is touched, which `sed -i` could not promise.
Verified against plain, a&b, a|b, a\b, p@ss&w|rd\x, R&D, a URL with a query
string, s/foo/bar/, a trailing space and the empty string; plus the
append-when-key-absent path, the file-does-not-exist path, non-target lines
left intact, no line-count drift, and mode preservation.
copyparty/update.sh and the rebuilt copyparty payload are included because
update.sh is embedded; regenerated with build.sh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The caddy-init service's `command:` used a YAML FOLDED scalar (`>`). Base
indent is set by `sh -c '` at 6; the closing `}` and the heredoc's `EOF`
were BOTH at 6, so YAML folded them into `} EOF`. The heredoc opened with
`<<EOF` therefore never saw a terminator line.
Confirmed by parsing the generated compose file: the command contains
`} EOF` and no bare EOF line. Feeding the resulting script to sh:
here-document at line 3 delimited by end-of-file (wanted `EOF')
syntax error: unexpected end of file (rc=2)
So caddy-init exited 2 before its `>` redirect ever ran -- no Caddyfile was
ever written, on any deploy. The `if [ ! -f /etc/caddy/Caddyfile ]` guard
never executed at all. caddy gates on
`caddy-init: condition: service_completed_successfully`, and smp-server and
xftp-server gate on caddy being healthy, so nothing in the stack started.
`docker compose up -d` returned non-zero and `set -euo pipefail` aborted the
script before the final report. The same failure hit every entry point:
automations.sh, cloud-init, the OpenRC unit's start() (so every boot), and
restore.sh -- which restores the same broken compose file backup.sh saved.
Generate the Caddyfile from the deploy shell instead and drop caddy-init
entirely, which removes the whole class of problem. Unlike the old
first-run-only guard this also applies a changed DOMAIN, ACME_EMAIL or
KEY_TYPE on a re-run rather than freezing them at the first deploy.
The heredoc here is deliberately unquoted so ${DOMAIN}/${ACME_EMAIL}/
${KEY_TYPE} expand; Caddy's {uri} has no `$` and survives. `cmp -s` stays
inside an `if` condition -- as `cmp -s A B && CADDY_CHANGED=1` it would trip
set -e whenever the files matched. Caddy is restarted only when the file
actually changed AND compose did not already replace the container, since a
bind-mounted file's contents are not part of the compose config hash and
restarting seconds after a first start would interrupt initial ACME issuance.
The Caddyfile is now real on-disk state rather than something a container
regenerates, so it is added to backup.sh's targets and restore.sh's file
list; without that, a restore would start caddy against an empty
./caddy_conf and the servers would never pass their health gate.
Verified: first deploy writes it and reports no change; an identical re-run
reports no change; a corrected DOMAIN rewrites it and flags the restart;
{uri} survives and no .new file is left behind.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Caddyfile is reassembled every run, but the WebFinger decision reads
BASE_DOMAIN/REDIRECT_URL from .env, and .env was written only on the first
run. So enabling WebFinger on a re-run was a silent no-op: the operator
answers the apex prompt, and the script prints the green
"[+] No BASE_DOMAIN -- pocket-id only" -- contradicting what they just
typed -- then pulls, restarts and prints DEPLOYED with exit 0.
The failure is invisible at the far end too. Caddy has no site block for the
apex, so it never gets a cert for it and discovery fails with a TLS or
connection error rather than a 404. Nothing warns; the one contradicting
line is a green [+] among docker pull output.
Moving the apex had the squid shape: compose reads the shell environment
first, so the new value served for that run only, and the documented later
`docker compose up -d` fell back to stale .env and reverted it.
Record which of the two keys arrived in the environment BEFORE the ":="
defaults, then write those through with a set_env upsert. Only keys actually
passed are touched: automations.sh drops a blank optional answer, so "blank"
cannot be told apart from "not supplied" and must not be read as "disable".
Since that means the prompt cannot retire WebFinger, the enabled path now
says so and points at the .env edit that can.
Verified: enabling on a re-run lands in .env and appends the block; changing
the apex lands and is logged; passing nothing touches nothing; BASE_DOMAIN
without REDIRECT_URL still dies rather than half-enabling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The summary printed at the end of every run says passive FTPS behind NAT is
fixed by re-running with FTP_NAT=... That could never work: the ftp-nat
substitution sat inside `if [[ ! -f "$CONF" ]]`, so on any re-run the conf
existed, the substitution was skipped, and ftp-nat stayed commented out.
cfg/copyparty.conf is the ONLY route FTP_NAT has to the service -- nothing
in docker-compose.yml interpolates it and the copyparty service is given no
environment, so .env's copy is a record, not the live setting. The remedial
run logged "exists; leaving it alone" as a green [+], reported healthy, and
reprinted the same advice. Passive FTPS failed exactly as before, with
nothing to distinguish "you did it wrong" from "it didn't take" -- and
because the template's commented example carries a literal IP, grepping the
conf shows an ftp-nat line with an address in it.
Lift the application out of the creation guard so it runs against an
existing conf too, and match `#?` so an already-set value (ISP change, typo)
is corrected rather than only the commented template line. Warn instead of
guessing if no ftp-nat line exists at all -- appending at EOF would land in
[accounts]. Kept above the chown, since `sed -i` rewrites as root.
`|| true` on the current-value read is load-bearing: on a still-commented
conf the grep matches nothing, and under `set -o pipefail` that would abort
the script inside the assignment -- the exact trap fixed in bf52426.
Also restart copyparty when the conf changed: compose will not recreate a
service whose image and config are unchanged, so `up -d` alone would leave
the edit on disk and the old value in the running process. And record the
value in .env so the two files do not disagree about what is deployed.
Verified: commented template applies; identical value is a no-op; a
different value is corrected; a conf with no ftp-nat line warns without
corruption; empty FTP_NAT skips the block entirely.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
deploy.sh deliberately re-reads .env before rendering config.yaml and
headplane.yaml ("so config.yaml substitution sees what's actually
deployed"). That design is right, but .env was seeded only when absent, so
"what's actually deployed" could never change: `set -a; . "$ENV_FILE"`
overwrote every value passed to the run, and `set -a` re-exported the stale
ones into compose as well.
Worse, the prompts at :192-197 run 56 lines BEFORE ENV_FILE is even defined,
so an interactive re-run asked for all six required values and then threw
every answer away. Via automations.sh the same six arrive exported and meet
the same fate. The validation only greps .env for non-empty values, which
the stale ones satisfy, so the run printed DEPLOYED and exited 0.
The case that matters is a rotated OIDC_CLIENT_SECRET: headscale keeps
presenting the retired secret at pocket-id's token endpoint, and
only_start_if_oidc_is_available probes issuer discovery, not the secret, so
nothing fails at deploy time. Either the leaked credential is still live and
the rotation is fiction, or tailnet OIDC login is broken and surfaces later
at some user's `tailscale up`. A corrected HEADSCALE_DOMAIN leaves the LE
cert hostname and the OIDC redirect URI on the typo; newly-supplied
headplane OIDC creds silently leave /admin on API-key login, so the
IdP-group gating the README recommends is never in force.
Record which keys arrived in the environment BEFORE the ":=" defaults erase
the distinction -- writing a blank over a live OIDC secret is exactly the
wrong move. Pre-load .env into keys NOT passed, before prompting, so the
prompts stop asking questions they will discard (and SKIP_PROMPTS=1 no
longer dies demanding values .env already has). Then write only the passed
keys through with a set_env upsert. Secret values are never echoed; a
changed HEADSCALE_DOMAIN warns about the new cert and the redirect URI.
The `. "$ENV_FILE"` stays: once .env carries this run's values it reads back
what was passed, and .env, config.yaml, headplane.yaml and compose agree.
Verified: rotated secret lands and is not echoed; changed domain lands and
warns; new headplane creds land; a re-run passing nothing touches nothing;
SKIP_PROMPTS=1 with a configured .env and no env vars no longer dies.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
.env was seeded only when absent, so a re-run with a corrected value logged
".env exists; leaving it alone" and dropped it. That looked harmless because
compose reads the shell environment before .env: an exported BIND_ADDR did
narrow the bind for that run, `ss -ltn` confirmed it, and the deploy reported
success -- while .env still said 0.0.0.0.
The drift surfaces later. The documented update path is a plain
`docker compose up -d`, which has no such environment, falls back to .env, and
republishes an SSL-bumping intercepting proxy on every interface. Nothing
warned. TRUSTED_CIDR has the same shape: a tightened allow-list silently
reverts to whatever .env kept.
Record which runtime keys actually arrived in the environment BEFORE the ":="
defaults run -- PROXY_PORT especially, whose default is a non-empty 3128, so
afterwards an unset variable is indistinguishable from a supplied one. On a
re-run, write just those keys through with the existing set_env() (a targeted
per-key rewrite, not a file overwrite) and log each change. Keys not passed
that run are untouched, so hand-edits to .env survive.
Verified: narrowing BIND_ADDR updates .env and logs it; a re-run with nothing
exported leaves .env alone; re-passing identical values is a silent no-op; and
a hand-edited PROXY_PORT=8080 survives all three, which is what the
capture-before-defaults ordering exists for.
Found by an adversarial sweep for the openbao bug class (920edc5), then
confirmed by hand.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
host_addrs() ends in a pipeline, so under `set -o pipefail` a probe that
exits non-zero -- even after printing perfectly usable addresses, or
because awk is missing -- made `addrs="$(host_addrs)"` non-zero, and
`set -e` killed deploy.sh at that line.
Nothing was printed when it did: the 2>/dev/null had already swallowed the
tool's own error and the fail-open guard on the next lines was never
reached, so the operator got a bare exit 1 mid-deploy with nothing to
diagnose. Worst case it aborted a deploy whose bind address was CORRECT --
reproduced with an `ip` stub that prints the matching address, then exits 1.
Capture with `|| true` so the emptiness test actually drives the fail-open
the comment beside it already promised. Neutralising inside host_addrs
instead would not cover a missing awk, since pipefail takes the rightmost
non-zero status.
Re-verified the check is not weakened: a healthy probe with the address
genuinely absent still lists the host's addresses and dies with the full
message, and all seven .env/environment precedence cases are unchanged.
Found by adversarial review of 920edc5.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
deploy.sh narrows the Docker publish bind to OPENBAO_ADDR when that is an
IP, but never checked that the host actually holds that address. A typo'd
IP therefore failed late, inside `docker compose up`:
failed to bind host port 192.160.100.50:8200/tcp:
cannot assign requested address
...by which point .env had been seeded and the TLS cert generated with the
bad address in its SAN. Neither is rewritten on a re-run (.env is never
overwritten, gen-tls.sh never regenerates over an existing pair), so
re-running with a corrected OPENBAO_ADDR silently changed nothing.
Add host_addrs() + check_bind_addr(), run before anything is written:
- lists the host's addresses from plain `ip addr show` -- no -o/scope
filters, since busybox ip supports neither -- falling back to ifconfig,
and skipping the check when neither exists rather than blocking;
- skips 0.0.0.0 / :: / *, and unwraps an [IPv6] publish literal;
- SKIP_BIND_CHECK=1 overrides for an address that only comes up later.
Resolve the bind compose will really interpolate, which follows compose's
own precedence -- shell environment before .env:
- exported (automations.sh passes answers via `env VAR=...`, or a
standalone OPENBAO_BIND=... run): the environment wins, so warn when
.env disagrees, because a later bare `docker compose up` would not;
- derived here: that assignment is not exported, so .env wins;
- in neither: compose falls back to 0.0.0.0 and publishes the API on
every interface -- warn, since that is a silent exposure.
Also warn when .env's OPENBAO_ADDR differs from this run's, naming the
cert that has to be deleted for the SAN to be regenerated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Native Alpine deployment for the anycast DNS estate -- no Docker, no Caddy,
alongside squid and openbao as an exception to the repo norm. Knot binds :53
directly, needs real client addresses for RRL and DNS cookies, and its DNSSEC
key store must live on the host filesystem.
Deploys a NODE. Zone data lives in the separate dns repo and arrives from its
pipeline. The split is /etc/knot/knot.conf: written here once as a skeleton of
include: lines covering only what belongs to a box (identity, NSID, storage
paths, listen, logging, control socket); everything that is DNS policy --
templates, dnssec policy, remotes/ACLs, modules, the domain inventory and the
zone files -- is delivered by the dns repo.
knsctl replaces adddns.pl and adddnssec.pl, fixing four defects:
- the duplicate check searched for the domain in BIND named.conf double-quote
syntax (/"$domain"/) against unquoted YAML, so it could never match; only
the -f zone-file test ever caught anything
- neither script consulted the other class's manifest, so a domain already in
public.conf could be appended to dnssec.conf and fail the reload AFTER both
files had been written
- nothing validated before reloading
- the reload was non-blocking, so a rejected config reported success
Its manifest matching is anchored on the YAML key and escapes the dot, so
barsrvno.de and srvnoXde no longer false-positive against srvno.de.
Aliases preserve the existing muscle memory with three corrections: -b on
every triggering knotc command (without it knotc returns OK when the command
was SENT, not when it succeeded); knzr (zone-reload) added alongside knrl
(reload), since reloading one zone's data is the right verb for a record
change and a full reload is only needed when a zone is added or removed; and
serial/NSID helpers that query unicast addresses, because asking the anycast
service address reaches whichever node is nearest and says nothing about
which node is stale.
Break-glass writes (add/remove/edit) warn and audit-log: they are overwritten
by the next pipeline deploy unless the change also lands in git. Removal
refuses to purge DNSSEC keys -- zone-purge +keys is irreversible on Knot
3.5.x, the key trash bin having arrived in 3.6.0 -- and prints the ordering
requirement, since removing a signed zone before the parent DS is withdrawn
is an outage for validating resolvers rather than a graceful shutdown.
deploy.sh, build.sh and cloud-init.yml are deliberately not included yet;
they are blocked on the Knot version decision, which sets the apk pin and
feature availability. See the Status section in the README.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two quick-reference documents, written to be scanned rather than read:
- USER-GUIDE.md for people new to IRC -- connecting, claiming a nickname,
the dozen commands that matter, scrollback, and a plain-language privacy
section (cloaked IP, channels logged for the configured retention, how to
turn off DM storage).
- ADMIN-CHEATSHEET.md split by where you work: from IRC as an operator
(UBAN, KILL, DEFCON, ChanServ, NickServ) and on the host via ergoctl, plus
mode tables and a "when things go wrong" section.
Both use the same placeholder convention as ergo.motd and are rendered by
deploy.sh into $STACK_DIR/docs/ with the network's real name, domain and
retention, so they can be handed straight to users and moderators. Refreshed
on every run, like the other installed files.
Command and mode references were checked against the v2.19.1 sources rather
than written from memory: irc/modes/modes.go for every mode letter, and
irc/chanserv.go and irc/nickserv.go for the service subcommands and which
require an oper capability.
build.sh's embed guard caught the docs being added to FILES without being
added to deploy.sh's EMBEDDED manifest -- the exact failure it was added to
prevent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the placeholder MOTD with a usable one: a coloured header, five house
rules, the NickServ/ChanServ/SASL commands people actually need, and a short
"worth knowing" section.
The retention line is generated rather than hard-coded, so it matches the
deployment instead of drifting from it: HISTORY=off says messages are not
stored, otherwise it names the real HISTORY_EXPIRE. Users are told plainly that
channels are logged and for how long, which is the honest counterpart to
enabling persistent history by default.
Rendered and checked: valid UTF-8 (an invalid byte makes the whole config fail
to load), ASCII-only for old clients, no stray $-escapes, only documented colour
names, and 72 columns at the widest.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seven confirmed findings from an observed-vs-expected review of a real deploy
transcript against the code (four other proposed findings were refuted and no
change was made for them).
- Caddyfile reformatted so `caddy fmt` is clean, removing the warning Caddy
printed on every validate and every start. Two causes, not one: `caddy fmt`
indents with TABS, and it deletes a blank line whose following line begins
with `{` -- which is why the warning pointed at line 17, the blank before the
global options block. The check is a whole-file byte comparison, so the line
number was only the first difference and the entire file had to be
reformatted. Verified whitespace-only outside the heredoc, whose body is left
byte-identical: Caddy strips padding derived from the closing marker's
indentation, so re-indenting it would change what the page serves.
- deploy.sh no longer upgrades Caddy behind the operator's back. A bare
`docker compose pull` refreshed the floating `caddy:2-alpine` on every re-run
and `up -d` then recreated it, doing exactly what CADDY_AUTOUPDATE=0 promises
not to, with none of update.sh's health check or rollback. Pulls are now
per service: the pinned Ergo tag always, Caddy only when absent or opted in.
- deploy.sh seeds the ACME_EMAIL and NETWORK_NAME prompts from .env, so pressing
Enter through a re-run no longer renames the network to the hostname in the
summary while .env keeps the real one.
- The Caddy restart guard compares the container's identity across `up -d`. A
container compose created or recreated has already read the new Caddyfile;
only one left running still holds the old config, and `svc_state` cannot tell
those apart.
- Container logs are now rotated (json-file, 10m x 3) on every service. Alpine's
docker package ships no daemon.json, so the default is unbounded, and the
60-second health probe alone writes a log line per run.
- ergoctl notes that LUSERS counts its own probe connection, which is why a
server with nobody on it reports one invisible user.
Verified: seven local suites pass, the embedded archive round-trips, line
endings are LF, and the reformatted Caddyfile is a verified fixed point of the
formatter (semantically identical token-for-token to the previous one).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds client-certificate authentication for operators, so an oper can be
identified by the SHA-256 fingerprint of their TLS client certificate instead of
a typed password:
ergoctl oper certfp <name> <fingerprint|nick> [--auto] [--clear]
The fingerprint can be given directly or read off a connected user over IRC
(276 RPL_WHOISCERTFP). Without --auto Ergo requires BOTH the certificate and the
password -- a second factor; with --auto the password is removed and the
certificate alone grants oper on connect.
It refuses to do this to the 'admin' oper, which matters: ergoctl authenticates
as admin over the loopback PLAINTEXT listener, which presents no client
certificate, and Ergo requires a configured certfp to match. Setting one there
would lock ergoctl -- and the scheduled jobs that use it -- out of the server
permanently.
Polish from the first real deploy on irc-1.srvno.de:
- deploy.sh no longer restarts Caddy on a first deploy. The Caddyfile compare
treated "destination does not exist" as a change, so Caddy was recreated
seconds after starting, while the initial ACME order was in flight.
- ergoctl status no longer prints the raw leading parameters of the 265/266
LUSERS numerics ("1 1 Current local users 1, max 1"), which duplicate the
counts already in the text. 252/254 keep theirs, where it is the only count.
- ergoctl oper list now shows which opers have a certfp and which are auto.
Verified: seven local suites pass, including new coverage for fingerprint
normalisation (colons and uppercase), insertion into the correct oper block
without touching siblings, replace-not-duplicate, the --auto and --clear paths,
276 parsing, and the admin guard.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ergo keeps history in RAM by default, so it is lost on every restart --
including the ones this stack's updater performs. HISTORY now selects a backend
at first deploy: sqlite (default, a file beside ircd.db, no extra container),
postgres (a pinned container via docker-compose.postgres.yml, loopback-only,
POSTGRES_PASSWORD_FILE so no secret lands in .env), or off. Both SQL backends
need Ergo 2.18.0+, which deploy.sh enforces. HISTORY_EXPIRE (default 30d) sets
retention, because upstream's 1w expire-time DELETES from persistent storage --
persistence with the shipped default would buy only a week.
Ergo opens the history backend only at startup and, unlike MySQL, has no
"after launching the server" guard for sqlite/postgresql: a rehash that enables
one reports success and then silently discards every message. So `ergoctl
history` restarts, and `ergoctl edit` detects a backend change and refuses to
apply it by rehash. The config validator runs with no network by design, so it
neutralises postgresql in its copy and reachability is checked separately.
Audit fixes (six lenses over first-deploy, runtime, lifecycle, security, docs):
- SECURITY (blocker): ircd/ is writable by the container uid while the 15-minute
cert sync and every ergoctl config edit run as root. cp/install/> follow a
symlink, so code execution in Ergo could redirect a root write onto any host
file. All such writes now stage under $STACK_DIR (0700 root) and land via mv
(rename(2) replaces a symlink instead of following it); root reads use cp -P
or refuse. Regression-tested with real symlinks.
- deploy.sh no longer swaps the image when ERGO_TAG changes on a deployed stack
(that bypassed pre-flight, snapshot, user warning and rollback); it points at
`TARGET_VERSION=... ergoctl update update`.
- restore was lossy and could lock you out: it now snapshots the current config
as well as the database, restores message history, re-points the postgres
password and re-hashes the local admin password into the restored config, and
validates before starting.
- `install -d -m` re-modes existing directories: `ergoctl backup /var/backups`
no longer chmods it 0700 root, and the updater no longer re-modes /var/log
(0775 root:syslog on Debian, which rsyslog needs).
- The admin oper password is printed only on a first run at a TTY, so it stays
out of cloud-init serial-console logs.
- A failed update is remembered, so the daily job stops repeating a disruptive
warn/stop/swap/roll-back cycle every night; postgres readiness gates an update
that would otherwise stop a healthy server it cannot restart.
- certsync no longer sends "TLS recovered" for outcomes that synced nothing.
- ergoctl history writes .env only after the restart is healthy, and returns 0.
- CR/LF is stripped before IRC framing, so a multi-line argument cannot inject a
second command; the ntfy token moves out of curl's argv.
- ufw/firewalld are additive, so 6667 is now explicitly revoked when PLAINTEXT=0.
- build.sh refuses to build a deploy.sh whose archive is missing a file the
script reads -- the failure mode that would have shipped a stack aborting on
every host.
Docs corrected against the code throughout, including retention, the pre-connect
account-registration default, encrypted-restore (AGE_IDENTITY), what a re-run
really does to .env, and what the update log does and does not contain.
Verified locally: all six suites pass (config render for each backend against the
real 2.19.1 template, yaml/oper/version/env helpers, the IRC client against a
fake server, and the audit fixes including the symlink escalation). Still not
exercised on a Docker host: the containers, ACME issuance, cert sync and
PostgreSQL itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
New deployments/ergo/: the Ergo IRC server behind Caddy for Let's Encrypt TLS
and the IRC-over-WebSocket endpoint. Docker rather than a native OpenRC/systemd
service, because Alpine's apk ergo trails upstream (3.24 ships 2.18.0 against a
2.19.1 security release) and Debian/Alma package it at all -- so native would
mean three install paths plus a per-distro ACME client.
Both containers run with network_mode: host. IRC bans, throttling and cloaking
key on the client's address, and Docker's userland proxy would hide every IPv6
client behind the bridge gateway; host mode also makes the repo's INPUT firewall
genuinely govern 80/443/6697. Caddy reaches Ergo over loopback, which is what
lets Ergo honour X-Forwarded-For (proxy-allowed-from defaults to localhost) and
mark web sessions secure.
- deploy.sh generates ircd.yaml ONCE from the pulled image's own default.yaml
(version-matched), rewriting the listeners/websockets blocks wholesale rather
than patching lines, then asserts hard post-conditions and validates with
`ergo run --smoke` in a throwaway container before anything starts.
- update.sh: pinned vX.Y.Z tags, GHSA + "### Security" release-note policies,
pre-flight against the new image, user NOTICE + grace, stop-consistent DB
snapshot, health check (IRC-level, not a bare TCP connect) and rollback that
restores the DB only when the schema actually moved. Compatibility-break
releases are held for review. certsync copies Caddy's cert pairwise-atomically
and verifies the fingerprint served on 6697 after the rehash.
- ergoctl: status/users/logs, validated edit+rehash, oper add/passwd/rm,
moderation, backup/restore, cert and update passthrough. Talks IRC to the
loopback listener over bash /dev/tcp and strips control characters from
replies.
- Ergo runs as a non-root system user, read-only rootfs, all caps dropped;
Caddy keeps only NET_BIND_SERVICE, with admin API and HTTP/3 off.
Reviewed adversarially across six lenses; 20 confirmed findings fixed, notably
a dead SIGHUP fallback (`rc=$?` after an `if` is always 0), several `set -e`
aborts from non-total pipelines, a release-list cache that only ever populated
in a subshell, and re-runs that used shell defaults instead of the deployed
.env. Verified locally: bash -n, LF endings, the ircd.yaml render against the
real 2.19.1 template in both PLAINTEXT modes, the yaml/oper/version/env helpers,
and the IRC client against a fake server (registration, oper, rehash success and
400-failure, control-character stripping, server-down paths). Not yet exercised
on a Docker host: the containers themselves, ACME issuance and cert sync.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An adversarial re-verification of 0812f34 found a couple of the fixes were
incomplete or regressed; addressed here.
- DR was an INCOMPLETE fix: `raft snapshot save`/`restore` are token-gated
(sys/storage/raft/snapshot is sudo-capable) and the container carries no
ambient token, so the previously-"fixed" backup returned "missing client
token" and streamed a zero-byte snapshot. Both the deploy.sh runbook and the
README DR flow now pass `-e BAO_TOKEN=<token>` on save and restore.
- IPv6 bind REGRESSION (introduced by the bind-narrowing): a bare IPv6 literal
in the compose port map (`fd00::10:8200:8200`) is invalid and aborts at
`docker compose pull`. Now IPv4 vs IPv6 are classified separately and IPv6 is
bracketed (`[fd00::10]:8200:8200`).
- Docker readiness race (now reachable since install_docker is actually
called): openrc backgrounds dockerd and returns before the socket is up, so
the next `docker compose pull` raced it under set -e. install_docker now polls
`docker info` for up to 30s.
- UID detection hardened: added `-T` to the one-off `docker compose run`, and an
empty result is now a loud warning (with the manual-chown remedy) instead of a
silent fall-through to root that would re-create the crash-loop on a non-root
image.
- The raft-volume chown is gated to first run (captured before any compose-run
instantiates the volume), so idempotent re-deploys don't recursively re-chown
a live raft dir.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A multi-agent sanity audit of the freshly-merged deployment found four
end-to-end blockers (and several smaller issues); all fixed here.
HIGH (were blocking):
- deploy.sh never called install_docker(), so `docker compose pull` hit
command-not-found on any host without Docker. Now called before the
compose steps.
- The container's server process runs as the image's own (often non-root)
user but the mounted config/TLS were root-owned 0640/0600 and the raft
volume root-owned -> vault crash-looped, never binding :8200. deploy.sh
now detects the image UID after pull and aligns ownership of config.hcl,
./tls and the data volume (a no-op when the image runs as root);
config.hcl is installed 0644 (holds no secrets).
- Docs told operators to set the daemon key `openbao_ca_cert`, but the
Kanrisha daemon's key is `ca_cert` (config.go, mapstructure:"ca_cert").
The wrong key is fatal on strict unmarshal / leaves TLS unverified.
Renamed in all 5 places (config.hcl, gen-tls.sh, deploy.sh x2, README).
- DR backup used `docker compose cp openbao:… -`, which emits a TAR stream,
so the age-encrypted snapshot was tar-wrapped and would not restore.
Switched to `docker compose exec -T openbao cat` for the raw bytes, wrote
the snapshot to a scratch path (not the live raft dir), and documented the
matching restore.
MEDIUM:
- Swap detection used `swapon --show` (absent on BusyBox) and `\s` (GNU-only)
-> silently no-op on Alpine, leaving swap on. Now uses /proc/swaps and
[[:space:]] so mlock hardening actually holds on musl.
- A Docker-published port bypasses the host INPUT firewall, so the source
rule was illusory. deploy.sh now narrows OPENBAO_BIND to OPENBAO_ADDR when
it is an IP, the compose/README/.env comments state the reality, and a new
Exposure section + an init-immediately warning were added.
- Fixed broken ../kanrisha/ and deployments/kanrisha/ links (separate repo).
LOW:
- OPENBAO_TLS_SANS is now honored (folded into the SAN list from the env).
- .gitignore excludes *.snap / *.snap.age.
- Bootstrap note clarifies bootstrap.sh needs the `bao` CLI (run it from the
Kanrisha host/workstation, not this Docker-only vault host).
- README multi-OS count corrected (eight stacks) + automations.sh header
lists openbao.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A dedicated OpenBao deployment, kept OFF the Kanrisha tape host so a compromise
of the tape node can't reach the vault.
- Native TLS on the listener (self-signed by default via gen-tls.sh, or a
CA-signed cert from a Smallstep CA over ACME) — no Caddy/Let's Encrypt;
reached over the LAN, not the public internet.
- Integrated raft storage (clean snapshot-based DR).
- mlock on (cap_add IPC_LOCK + memlock unlimited + host swapoff in deploy.sh).
- Manual unseal by default; optional PKCS#11 HSM auto-unseal.
- deploy.sh: Docker install (Alpine/Debian/Alma), self-signed cert, .env seed,
swapoff, firewall 8200/tcp, compose up; then prints init/unseal + the KV-v2 +
AppRole bootstrap for Kanrisha + the raft-snapshot DR flow. Self-contained
(config payload embedded by build.sh).
- Registered in automations.sh + the README deployment table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New deployments/copyparty/: copyparty (copyparty/ac) behind Caddy/LE for the
web UI/WebDAV, plus its own SFTP (password auth) and FTPS listeners published
directly. Ships update.sh, which drives container updates off copyparty's
security-advisories API (api.copyparty.eu/advisories) -- policies latest|security|off.
- Real client IP end-to-end: Caddy XFF/X-Real-IP + copyparty xff-src: lan.
- SFTP host key + self-signed FTPS cert generated/persisted in /cfg; admin
password generated on first deploy; conf auto-included via the image's % /cfg.
- Firewall opens 80/443 + SFTP/FTPS + passive range (colon form for ports.d).
- Wired into automations.sh, README, .gitignore; cloud-init for fresh VMs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New deployments/squid/: an explicit forward proxy with SSL-bump TLS
interception (local CA, generated on first deploy) and hostname-targeted
static-content caching. Unlike the other stacks it is a forward proxy, not
a Caddy/Let's-Encrypt inbound site.
- Self-built minimal Alpine image (apk squid ships ssl-bump); entrypoint
renders squid.conf and generates the cache policy from the domain lists.
- Wildcard hostname caching (cache-domains.txt leading-dot + optional
cache-domains.regex); boost vs strict-allowlist toggle (CACHE_ONLY_LISTED).
- Storage gate never caches HTML or dynamic content; query strings exempt on
boosted domains so versioned static assets still cache.
- splice-domains.txt passthrough for pinned/banking domains.
- Deny-by-default http_access (TRUSTED_CIDR) + BIND_ADDR pinning; CA key 0600
on host, never embedded, git-ignored.
- Wired into automations.sh, README, .gitignore; cloud-init for fresh VMs.
Add a reusable iptables baseline that hardens hosts with ICMP + SSH
defaults and lets deployments register the ports they need. INPUT is
deny-by-default (loopback, established, ICMP, SSH on the configured port,
plus registered ports); OUTPUT stays open and FORWARD is left untouched so
Docker container networking is unaffected.
Persistence is native -- no boot hook. Rules are saved and restored by the
distro's own package (iptables/ip6tables on Alpine, iptables-persistent on
Debian, iptables-services on Alma) via the new oslib helpers
install_iptables / fw_save_cmd / fw_enable_restore. The saved ruleset
carries the INPUT->sshguard jump, so brute-force protection survives reboot
without the old sshguard-iptables hook.
A self-contained /usr/local/sbin/firewall-apply rebuilds INPUT from
declarative drop-ins under /etc/firewall/ports.d and runs the native save,
so deployments add a port without needing the repo present:
printf '80/tcp\n443/tcp\n' > /etc/firewall/ports.d/mystack.rule
/usr/local/sbin/firewall-apply
- SSH port read live from sshd_config (custom bastion ports just work);
FW_SSH_SOURCE restricts the source CIDR; FW_ALLOW_PING gates echo
- harden-ssh.sh / harden-jumphost.sh install it when ENABLE_FIREWALL=1
(default) and skip the sshguard-only hook; ENABLE_FIREWALL=0 keeps it
- cloud-init base.yml / jumphost.yml forward the toggle
- the four stack deploy.sh open_web_ports() register 80/443 via the
firewall (ufw/firewalld kept as fallback); Docker-published ports bypass
INPUT, so this is belt-and-braces and self-documenting
- README + cloud-init/README document the mechanism, Docker caveat, and the
`disable` recovery path
Ship allow-everything as the active default (first acl rule), with the
Tailscale default policy translated to headscale's acls format included but
commented for when you lock down. Pre-fix the headscale gotchas: tag:shared
owned by group:admins (not an autogroup), autogroup:self/ssh-check flagged
experimental. Rebuild embedded archive.
headscale rejects autogroups as tagOwners (only user/group:/tag:), which made
the shipped policy fatal. Replace with a valid allow-all default plus correct
commented examples for tightening. Document gating /admin to a pocket-id
superuser group via the headplane client's Allowed User Groups. Rebuild archive.
Integrate headplane (ghcr.io/tale/headplane) into the headscale stack, served
by Caddy at /admin. API-only (no Docker socket); deploy.sh mints a headscale
API key on first run, generates headplane.yaml, and wires optional OIDC login
via pocket-id (second client, /admin/oidc/callback). Adds HEADPLANE_* env,
compose service, Caddy routing; rebuild embedded archive.
Ship policy.hujson (mounted + installed on first deploy, edits preserved) and
wire policy.mode=file / policy.path in config.yaml. Translate the Tailscale
"grants" default into headscale's legacy "acls" format (self-access, tag:shared,
Tailscale SSH), since headscale 0.28 doesn't support grants. Embed in deploy.sh
and document `headscale policy check`.
Make pkce.method: S256 explicit in config.yaml (alongside pkce.enabled: true)
and note the pocket-id client must have PKCE enabled too. Rebuild embedded
archive.
Add /usr/local/bin/headscale (generated by deploy.sh) that runs the headscale
CLI inside the container with the compose file path baked in, so `headscale
users list` etc. work from any directory instead of erroring with
"no configuration file provided". Update the post-deploy hints accordingly.
Restructure around a single entry point (automations.sh) with a Gum wizard and
a self-extracting bundle for repo-less installs. Add scripts/oslib.sh so the
provisioning scripts (setup-host, harden-ssh, harden-jumphost, sshuser) run on
Alpine/Debian/Alma; seed root keys from globals/.
- ntfy SSH-login alerts (user, source IP, key, region, jump target) via pam_exec
- daily auto-updates: AUTO_REBOOT=idle reboots only when no SSH active; opt-in
Alpine stable-branch upgrades (ALLOW_RELEASE_UPGRADE)
- cloud-init: generic base/jumphost + per-deployment, which harden SSH by
default on fresh VMs
- pocket-id: optional WebFinger block (BASE_DOMAIN), tag v2.8.0
- headscale: fix oidc.expiry schema for 0.28 so the container starts
- Gitea release workflow on tag (TOKEN_GITEA); repo URLs -> Gitea
- README/LICENSE/.gitignore/.gitattributes (force LF)
Restructure around a single entry point (automations.sh) with a Gum wizard and
a self-extracting bundle for repo-less installs. Add scripts/oslib.sh so the
provisioning scripts (setup-host, harden-ssh, harden-jumphost, sshuser) run on
Alpine/Debian/Alma; seed root keys from globals/.
- ntfy SSH-login alerts (user, source IP, key, region, jump target) via pam_exec
- daily auto-updates with AUTO_REBOOT=idle (reboots only when no SSH active) and
opt-in Alpine stable-branch upgrades
- generic + per-deployment cloud-init; Gitea release workflow on tag
- README/LICENSE/.gitignore/.gitattributes (force LF); repo URLs -> Gitea
57_WolveandClaude Opus 5