automations

Deployment and automation scripts for self-hosted, security-hardened infrastructure. The host-provisioning scripts run on Alpine, Debian, and Alma Linux (every distro difference is gated in scripts/oslib.sh); each Docker stack runs behind Caddy with automatic Let's Encrypt TLS, orchestrated with Docker Compose.

About this repo. This started as years of personal notes and one-off scripts I'd accumulated running my own infrastructure. I worked with Claude to clean them up, make them consistent and multi-distro, and add some quality-of-life options — so friends can use them too, and build on them. I'm still working through my collection and adding more as I go, so expect this to keep growing. Treat it as a starting point: fork it, wire in your own domains/keys, and send improvements back. PRs and ideas welcome.

One command to run anything

curl -fsSL https://git.anomalous.dev/57_Wolve/automations/raw/branch/main/automations.sh \
  | REPO_URL=https://git.anomalous.dev/57_Wolve/automations.git sh

Or, from a clone:

./automations.sh

Pipe it to sh, not bash — a stock Alpine box has busybox ash and no bash at all. The launcher's prologue is POSIX sh: it installs git/bash for the distro it finds itself on (apk / apt-get / dnf / yum), then re-execs itself under bash. ./automations.sh from a clone does the same, so a bare Alpine install needs nothing prepared by hand.

automations.sh opens a Gum wizard (auto-installed) that lets you:

• Deploy on this host — pick any deployment or generic script and run it, prompting for the values it needs.
• Build artifacts locally — regenerate a deployment's self-contained deploy.sh and optionally scp it to a target host.

Shared defaults (email, repo URL, SSH key source, backup recipient) come from globals/ so you set them once.

Self-contained bundle (no repo access)

For hosts that shouldn't have repo/git access, package the whole repo into one self-extracting script and serve it from a webserver or a public Gitea release (the repo itself can stay private):

./build-bundle.sh              # -> dist/automations-bundle.sh (HEAD)
./build-bundle.sh v1.2.0       # a specific tag
./build-bundle.sh worktree     # current working tree

On the target host, download then run it (it extracts itself — it can't read a pipe, so download first):

curl -fsSLO https://your-host/automations-bundle.sh
sh automations-bundle.sh                                 # launcher wizard
sh automations-bundle.sh bash scripts/setup-host.sh      # run one script
SSH_PORT=2222 sh automations-bundle.sh bash scripts/harden-jumphost.sh

Like the launcher, the bundle's stub is POSIX sh and installs bash if the host lacks it, so it runs on a bare Alpine box.

It extracts to INSTALL_DIR (default /opt/automations) and runs the launcher or the command you pass. The payload excludes ignored files, so no secrets are embedded.

Releases via Gitea Actions: pushing a vX.Y.Z tag runs .gitea/workflows/release.yml, which builds the bundle and attaches it to a release. Create a secret named TOKEN_GITEA (a Gitea access token with write:repository) — note Gitea reserves the GITEA_ prefix, so the secret can't be called GITEA_TOKEN. Adjust runs-on to match your runner. Mark the release public and automations-bundle.sh is fetchable without repo access.

Layout

The rule: generic, run-anywhere scripts live in scripts/; deployment-specific files live under deployments/<name>/. Shared values live in globals/.

automations.sh        # the launcher (one-liner entry point)
cloud-init/           # generic base + jumphost cloud-init (any of the 3 distros)
globals/              # shared assets: age-pubkey, authorized_keys, motd, globals.env
scripts/              # generic scripts (Alpine/Debian/Alma)
deployments/<name>/   # one folder per stack

scripts/ — generic (Alpine / Debian / Alma)

Script	What it does
setup-host.sh	Set hostname per the naming schema (derives FQDN + Node ID) and render the shared MOTD with auto-computed border spacing.
harden-ssh.sh	SSH hardening: post-quantum hybrid KEX (every spelling the host's OpenSSH supports; SSH_ALLOW_CLASSIC_KEX=1 adds a curve25519 fallback for clients with no PQ method), fresh Ed25519 host keys, key-only auth, external SFTP subsystem, sshguard.
harden-jumphost.sh	Bastion hardening on top of harden-ssh: ssh-admins (shell) vs ssh-jumpers (ProxyJump-only) with a PermitOpen allow-list.
harden-firewall.sh	Deny-by-default host firewall: iptables on Alpine/Debian, firewalld on Alma/RHEL (set FW_BACKEND to override), skipped on Proxmox (pve-firewall owns the ruleset). Loopback, established, ICMP, SSH (configurable port) + registered ports; persisted natively (no boot hook). Same allow/deny/list/disable sub-commands on both.
sshuser.sh	Add/edit/remove SSH users on a hardened jump host (Gum TUI or CLI flags). Installed standalone as sshuser.
ntfy-ssh-login.sh	pam_exec hook that posts SSH logins to ntfy (user, source IP, key used, best-effort jump target), gated by group. Config: ssh-notify.conf.example.
auto-update.sh	Daily unattended package updates; reports (doesn't auto-jump) a new Alpine branch; reboot detection; ntfy summary. install/run/uninstall.
oslib.sh	OS-abstraction layer (detection, package manager, init system, SFTP path, hostname, boot hooks, native firewall persistence, sshguard, login notifier). Sourced, not run. One file holds every distro difference.
lib.sh	Launcher helpers (ensure_gum, load_globals, resolve_ssh_keys); sources oslib.sh. Not run directly.

deployments/ — per stack

Deployment	What it is	Depends on
pocket-id	OIDC provider (Caddy + Anubis PoW gate + Pocket-ID).	—
beszel	Server monitoring hub. OIDC via pocket-id (post-deploy).	pocket-id (OIDC, optional)
headscale	Self-hosted Tailscale control server, OIDC login.	pocket-id (OIDC)
webfinger	Serves /.well-known/webfinger for OIDC discovery; redirects the rest.	pocket-id (issuer)
squid	SSL-bump caching forward proxy — static-content cache + TLS interception via a local CA. The exception: a forward proxy, not a Caddy/LE site.	—
copyparty	Portable file server — web UI/WebDAV behind Caddy, plus direct SFTP + FTPS. Ships a security-notices-aware updater.	—
simplex	SimpleX SMP + XFTP relay with Tor hidden services + encrypted backups.	globals/age-pubkey.txt
openbao	Hardened tape-encryption key store for Kanrisha (the LTO tape-archive system — separate repo/host). Exception: native TLS on the LAN, no Caddy/LE. raft storage, mlock, self-signed or Smallstep-ACME cert, manual/HSM unseal.	globals/age-pubkey.txt (backups)
ergo	IRC server (Ergo) with Caddy for Let's Encrypt + the websocket endpoint. Both containers use host networking (real client IPs for bans/cloaking; the host firewall applies). Persistent message history via SQLite (default) or an optional PostgreSQL container. Ships ergoctl (admin CLI) and a health-checked updater that also syncs Caddy's cert into Ergo.	—

Conventions

• Alpine + Docker Compose + Caddy/Let's Encrypt across every stack, with two exceptions: squid (a forward proxy with a local TLS-interception CA) and openbao (a same-LAN secrets store with native TLS) — neither uses Caddy/LE. ergo keeps Caddy/LE but runs both containers in the host network namespace (IRC needs real client IPs), so it has no published ports.
• build.sh → deploy.sh: each stack's build.sh embeds its docker-compose.yml / Caddyfile / .env.example into a single self-contained deploy.sh (base64 tar.gz). That one file can be scp'd to a host and run on its own. Rebuild after editing the loose files. (simplex is the exception — it deploys via install-simplex.sh.)
• Secrets live in .env (generated on first deploy) and globals/globals.env, both git-ignored. Only *.example templates and public keys are committed.
• Non-interactive: every deploy.sh honors SKIP_PROMPTS=1 with values supplied via the environment — which is how the per-deployment cloud-init.yml templates stand a stack up unattended.

Cloud-init

• Provision a host: cloud-init/base.yml (hostname + MOTD + SSH hardening) or cloud-init/jumphost.yml (bastion). Distro-agnostic — they install prerequisites for whatever the image is, then run the scripts.
• Stand up a stack: each deployment ships its own cloud-init.yml (e.g. deployments/pocket-id/cloud-init.yml). These assume a fresh VM, so they harden SSH first (harden-ssh.sh, HARDEN_SSH=1 by default) and then deploy the stack.

Fill in REPO_URL and the values at the top of the runcmd block, paste as instance user-data, and the host configures itself on first boot.

Multi-OS notes

The host-provisioning scripts (setup-host, harden-ssh, harden-jumphost, sshuser) and the nine Docker stacks (pocket-id, beszel, headscale, webfinger, squid, copyparty, simplex, openbao, ergo) run on Alpine, Debian, and Alma. Distro differences live in scripts/oslib.sh — package manager (apk/apt/dnf), init system (OpenRC/systemd), sshd service name, the per-distro sftp-server path, hostname, boot hooks, and the sshguard log source/backend.

simplex remains Alpine-targeted — it depends on awall and Tor hidden services with Alpine-specific wiring, so it isn't part of the tri-distro set.

Key exchange and old clients

The KEX list is built from what the host's OpenSSH actually supports (ssh -Q kex), not from its version number, and it offers every spelling of each algorithm. That matters: OpenSSH called the same hybrid sntrup761x25519-sha512@openssh.com before the method was standardised and sntrup761x25519-sha512 after, and SSH matches algorithm names as exact strings — so a server offering only the new spelling rejects a client implementing the identical algorithm under the old one, with Unable to negotiate ... no matching key exchange method found.

Client floor for a PQ-only host, from the OpenSSH release notes:

Client	Result
9.9+	works — knows both spellings, and ML-KEM
8.9 – 9.8	works — its default proposal carries sntrup761x25519-sha512@openssh.com
8.5 – 8.8	has the algorithm but does not offer it by default; needs KexAlgorithms +sntrup761x25519-sha512@openssh.com client-side
< 8.5	no PQ hybrid exists — locked out by design
Windows in-box ssh.exe	locked out at every version, including a fully-patched 9.5p2

The Windows case is not an age problem, which makes it easy to misdiagnose: Microsoft's fork compiles sntrup761 out, because it needs C99 variable-length arrays that MSVC does not support (Win32-OpenSSH #2140, #2391). So a fully-patched Windows 11 reports OpenSSH_for_Windows_9.5p2 — new enough on paper — and still lists zero PQ methods in ssh -Q kex. Two ways out, in order of preference:

1. Use a client that has PQ. Check what is already on the box before installing anything — Git for Windows bundles genuine upstream OpenSSH:

   & "$env:ProgramFiles\Git\usr\bin\ssh.exe" -Q kex | Select-String 'mlkem|sntrup'
   

   If that prints anything, use that binary and install nothing. Otherwise winget install --id Microsoft.OpenSSH.Preview (10.0.0.0p2 added both algorithms). Add-WindowsCapability does not help — it installs the same in-box 9.5p2. Mind the PATH order: the MSI appends C:\Program Files\OpenSSH, but C:\Windows\System32\OpenSSH is already ahead of it, so a bare ssh still resolves to the old binary. Confirm with (Get-Command ssh).Source.

2. SSH_ALLOW_CLASSIC_KEX=1 — also offer curve25519-sha256 (and its @libssh.org spelling). Off by default. A session that negotiates it is secure against a classical attacker but has no store-now-decrypt-later protection. Modern clients still pick a PQ method, because the client's preference order decides. Set it per host rather than fleet-wide, and drop it once the old client is gone.

harden-ssh.sh rewrites /etc/ssh/sshd_config wholesale on every run, so hand edits to that file do not survive a re-run. Use the knob, not sed.

Host firewall

scripts/harden-firewall.sh installs a deny-by-default baseline, with the backend chosen per family (override with FW_BACKEND=iptables|firewalld):

• Alpine / Debian → iptables. INPUT drops everything except loopback, established/related, ICMP, and SSH on the configured port — plus any ports a deployment registers.
• Alma / RHEL → firewalld (its native firewall). The default zone is already deny-by-default; we strip the stock ssh/cockpit services, open SSH + registered ports, and let sshguard block via the sshguard-firewalld backend (no INPUT → sshguard jump needed).
• Proxmox → nothing. The host is skipped — and left with no host firewall until you enable Proxmox's own. See below.

OUTPUT/egress stays open and FORWARD is left untouched, so Docker container networking is unaffected. The harden scripts and cloud-init/base.yml / jumphost.yml install it automatically (ENABLE_FIREWALL=1 by default).

• Configurable SSH port — read live from sshd_config, so a bastion on 2222 is firewalled correctly with no extra flags. Restrict the source with FW_SSH_SOURCE=<cidr>; drop ping with FW_ALLOW_PING=0.
• Native persistence, no boot hook — on iptables hosts the ruleset is saved and restored by the distro's own package: iptables + ip6tables (Alpine/OpenRC) or iptables-persistent (Debian); the saved ruleset carries the INPUT → sshguard jump. On firewalld hosts every change is --permanent, so it persists across reboot natively and sshguard manages its own blocks.
• Scripted additions — deployments drop a rule file and re-apply:

  printf '80/tcp\n443/tcp\n' > /etc/firewall/ports.d/mystack.rule
  /usr/local/sbin/firewall-apply
  

  or interactively: harden-firewall.sh allow 443/tcp 51820/udp / allow web / deny 51820/udp / list.
• Docker caveat — containers published with -p (e.g. Caddy's 80/443) reach the host through nat/FORWARD and bypass INPUT, so the firewall neither blocks nor needs to open them; the per-stack rule files are belt-and-braces for any host-bound bind and self-documentation. The one stack that runs with network_mode: host (ergo) binds on the host directly, so there the registered ports are the real gate.
• Recovery — harden-firewall.sh disable un-locks you: on iptables it flushes the rules and sets INPUT back to ACCEPT (persisted); on firewalld it re-opens SSH (the ssh service + the configured port) and leaves firewalld running. A re-apply never drops the live SSH session — on iptables the established-connection accept is added before the policy flips to DROP, and firewalld reloads preserve established connections.

Proxmox hosts are skipped

Proxmox VE and Proxmox Mail Gateway are Debian underneath, so everything else in this repo treats them as Debian — but they already ship a firewall, and pve-firewall owns the host ruleset. The conflict is not the one you'd expect: pve-firewall does not delete third-party rules. It restores with iptables-restore -n (--noflush), only ever flushes chains matching its own patterns (PVEFW-*, tapNiM-*, vethNiM-*, fwbrN-*, GROUP-*), appends -A INPUT -j PVEFW-INPUT only when that hook is missing, and never sets a built-in chain's policy. Our rules would survive it fine. The damage runs the other way — we break Proxmox:

• -P INPUT DROP is ours alone, and PVEFW-HOST-IN returns on accept rather than accepting (it still has to check the tap rules), so traffic Proxmox explicitly allowed falls out of its chain and lands on our DROP. We silently override the platform's own accepts;
• the persistence layer is worse: netfilter-persistent restores at boot with a full iptables-restore (no --noflush), wiping PVE's -j PVEFW-INPUT hook along with everything else until the daemon re-appends it on its next ~10s pass;
• a deny-by-default chain has to enumerate the whole platform to stay usable: 8006/tcp web UI, 5405-5412/udp corosync (5405 + knet link number, up to 8 links — the bare 5405 is the pre-6.x multicast-era number), 60000-60050/tcp migration, 5900-5999/tcp VNC, 3128/tcp SPICE, 22/tcp SSH, 111/udp rpcbind with NFS storage, plus Ceph's 6789/3300/6800-7300 when hyperconverged. Miss one and you lose the GUI or the cluster; miss -i lo and pveproxy can't reach pvedaemon on 127.0.0.1:85, breaking the API locally;
• under the opt-in nftables backend (PVE 8.2+), proxmox-firewall registers its own nft input hook, and an nft DROP beats an iptables ACCEPT — our rules wouldn't even be authoritative.

So harden-firewall.sh detects Proxmox (is_proxmox in oslib.sh: it looks for pve-firewall/pveversion//etc/pve/nodes) and does nothing — apply explains and exits cleanly, allow/deny refuse loudly rather than pretend, and list shows pve-firewall status. harden-ssh.sh / harden-jumphost.sh skip the firewall too and install only the INPUT → sshguard boot hook, so brute-force protection still works while the Proxmox firewall is off. Manage the host firewall where Proxmox expects it — Datacenter → Firewall and Node → Firewall, or the .fw files directly:

[RULES]
IN ACCEPT -p tcp -dport 443

FW_IGNORE_PVE=1 forces our firewall on anyway. It will fight pve-firewall and can lock you out of the GUI and the cluster; Proxmox Backup Server ships no firewall of its own and is not detected, so it hardens as a normal Debian host.

Skipped is not the same as protected. Proxmox's firewall is off by default — the cluster-wide enable in cluster.fw defaults to 0, and while it is, the pve-firewall daemon actively tears its chains down every ~10 seconds, leaving INPUT at policy ACCEPT with no rules. Until you enable it at Datacenter → Firewall → Options, a Proxmox host has no host firewall at all and 8006, 22, 3128 and 111 are open on every interface. Don't be reassured by the node panel reading Firewall: Yes — that setting is ignored while the datacenter one reads No. On these hosts, do your filtering in Proxmox or upstream of it.

The INPUT → sshguard jump the harden scripts install is safe alongside pve-firewall: it is inserted with -I, so it sits ahead of the appended PVEFW-INPUT hook and keeps getting first look at new connections either way.

Already hardened a Proxmox host? Detection only helps hosts set up from now on, so disable cleans up one that already has our firewall:

bash scripts/harden-firewall.sh disable

On Proxmox that sub-command checks for leftovers (/etc/firewall, the engine, or a DROP policy) and, if it finds them, sets INPUT back to ACCEPT and flushes it — policy first, so it never drops the SSH session you are running it over — deletes /etc/firewall and /usr/local/sbin/firewall-apply, disables the boot-restore service and renames the saved rulesets to *.bak-harden-firewall (saving the open state, as the normal disable does, would snapshot pve-firewall's own PVEFW-* chains and restore that stale copy at the next boot), re-inserts the INPUT → sshguard jump, and restarts pve-firewall. On a host that was correctly skipped it finds nothing and says so. apply points you at it when it spots leftovers.

SSH login notifications

The harden scripts can install a pam_exec hook (scripts/ntfy-ssh-login.sh) that posts every SSH login to an ntfy topic. Enable it by passing NTFY_URL (and optionally NTFY_TOKEN) when running harden-ssh.sh / harden-jumphost.sh, or via the launcher / jumphost cloud-init. Each alert reports:

• the user and source IP,
• the SSH key they authenticated with (fingerprint, via ExposeAuthInfo),
• a region tag so you know which bastion fired it (derived from the host's FQDN, e.g. us-evi-1),
• best-effort the next hop of a ProxyJump (see caveat below).

Filtering is by group: NOTIFY_GROUPS limits alerts to certain groups/security levels, and NOTIFY_PRIORITY_MAP sets a per-group ntfy priority. Config lives at /etc/ssh-notify.conf (mode 0600; scripts/ssh-notify.conf.example documents every key). A publish token is optional — leave it empty for a read-gated topic.

Jump-target caveat: a ProxyJump opens a direct-tcpip channel (no session), so the destination never reaches pam_exec. The bastion only logs it at LogLevel VERBOSE/DEBUG; harden-jumphost.sh sets VERBOSE and the notifier parses the log best-effort. If the target isn't in the log it is simply omitted.

Daily updates

scripts/auto-update.sh keeps a host patched unattended — ideal for an SSH-only bastion, where a routine upgrade can barely break anything. harden-jumphost.sh schedules it by default (set AUTO_UPDATE=0 to skip); harden-ssh.sh takes AUTO_UPDATE=1. It runs daily via busybox crond (/etc/periodic/daily) on Alpine or a systemd timer on Debian/Alma.

Each run:

• applies all in-branch package upgrades (apk/apt/dnf);
• reports a new Alpine branch (e.g. 3.21 → 3.22) by default — that rewrites the repo branch, so it's opt-in. Set ALLOW_RELEASE_UPGRADE=1 to also apply it: it repoints /etc/apk/repositories to the newest stable vX.Y (never edge), runs apk upgrade --available, and forces a reboot flag. Debian/Alma stay report-only;
• detects when a reboot is needed (kernel/libc/openssl). AUTO_REBOOT controls it: 0 = never (just flag), 1 = always, idle = only when no SSH connections are active — so a bastion reboots itself once the admins and ProxyJump tunnels have cleared, never mid-session. harden-jumphost.sh defaults the bastion to idle. A deferred reboot is tracked in /run and retried each day until it happens;
• sends an ntfy summary (reusing /etc/ssh-notify.conf).

Schedule it standalone with auto-update.sh install (or via the launcher), run a pass now with auto-update.sh run, and preview safely with DRY_RUN=1 auto-update.sh run.

License

MIT.