automations/README.md

automations

Deployment and automation scripts for self-hosted, security-hardened infrastructure. The host-provisioning scripts run on Alpine, Debian, and Alma Linux (every distro difference is gated in scripts/oslib.sh); each Docker stack runs behind Caddy with automatic Let's Encrypt TLS, orchestrated with Docker Compose.

About this repo. This started as years of personal notes and one-off scripts I'd accumulated running my own infrastructure. I worked with Claude to clean them up, make them consistent and multi-distro, and add some quality-of-life options — so friends can use them too, and build on them. I'm still working through my collection and adding more as I go, so expect this to keep growing. Treat it as a starting point: fork it, wire in your own domains/keys, and send improvements back. PRs and ideas welcome.

One command to run anything

curl -fsSL https://git.anomalous.dev/57_Wolve/automations/raw/branch/main/automations.sh \
  | REPO_URL=https://git.anomalous.dev/57_Wolve/automations.git bash

Or, from a clone:

./automations.sh

automations.sh opens a Gum wizard (auto-installed) that lets you:

• Deploy on this host — pick any deployment or generic script and run it, prompting for the values it needs.
• Build artifacts locally — regenerate a deployment's self-contained deploy.sh and optionally scp it to a target host.

Shared defaults (email, repo URL, SSH key source, backup recipient) come from globals/ so you set them once.

Self-contained bundle (no repo access)

For hosts that shouldn't have repo/git access, package the whole repo into one self-extracting script and serve it from a webserver or a public Gitea release (the repo itself can stay private):

./build-bundle.sh              # -> dist/automations-bundle.sh (HEAD)
./build-bundle.sh v1.2.0       # a specific tag
./build-bundle.sh worktree     # current working tree

On the target host, download then run it (it extracts itself — it can't read a pipe, so download first):

curl -fsSLO https://your-host/automations-bundle.sh
bash automations-bundle.sh                                 # launcher wizard
bash automations-bundle.sh bash scripts/setup-host.sh      # run one script
SSH_PORT=2222 bash automations-bundle.sh bash scripts/harden-jumphost.sh

It extracts to INSTALL_DIR (default /opt/automations) and runs the launcher or the command you pass. The payload excludes ignored files, so no secrets are embedded.

Releases via Gitea Actions: pushing a vX.Y.Z tag runs .gitea/workflows/release.yml, which builds the bundle and attaches it to a release. Create a secret named TOKEN_GITEA (a Gitea access token with write:repository) — note Gitea reserves the GITEA_ prefix, so the secret can't be called GITEA_TOKEN. Adjust runs-on to match your runner. Mark the release public and automations-bundle.sh is fetchable without repo access.

Layout

The rule: generic, run-anywhere scripts live in scripts/; deployment-specific files live under deployments/<name>/. Shared values live in globals/.

automations.sh        # the launcher (one-liner entry point)
cloud-init/           # generic base + jumphost cloud-init (any of the 3 distros)
globals/              # shared assets: age-pubkey, authorized_keys, motd, globals.env
scripts/              # generic scripts (Alpine/Debian/Alma)
deployments/<name>/   # one folder per stack

scripts/ — generic (Alpine / Debian / Alma)

Script	What it does
setup-host.sh	Set hostname per the naming schema (derives FQDN + Node ID) and render the shared MOTD with auto-computed border spacing.
harden-ssh.sh	SSH hardening: post-quantum hybrid KEX, fresh Ed25519 host keys, key-only auth, external SFTP subsystem, sshguard.
harden-jumphost.sh	Bastion hardening on top of harden-ssh: ssh-admins (shell) vs ssh-jumpers (ProxyJump-only) with a PermitOpen allow-list.
harden-firewall.sh	Deny-by-default host firewall: iptables on Alpine/Debian, firewalld on Alma/RHEL (set FW_BACKEND to override). Loopback, established, ICMP, SSH (configurable port) + registered ports; persisted natively (no boot hook). Same allow/deny/list/disable sub-commands on both.
sshuser.sh	Add/edit/remove SSH users on a hardened jump host (Gum TUI or CLI flags). Installed standalone as sshuser.
ntfy-ssh-login.sh	pam_exec hook that posts SSH logins to ntfy (user, source IP, key used, best-effort jump target), gated by group. Config: ssh-notify.conf.example.
auto-update.sh	Daily unattended package updates; reports (doesn't auto-jump) a new Alpine branch; reboot detection; ntfy summary. install/run/uninstall.
oslib.sh	OS-abstraction layer (detection, package manager, init system, SFTP path, hostname, boot hooks, native firewall persistence, sshguard, login notifier). Sourced, not run. One file holds every distro difference.
lib.sh	Launcher helpers (ensure_gum, load_globals, resolve_ssh_keys); sources oslib.sh. Not run directly.

deployments/ — per stack

Deployment	What it is	Depends on
pocket-id	OIDC provider (Caddy + Anubis PoW gate + Pocket-ID).	—
beszel	Server monitoring hub. OIDC via pocket-id (post-deploy).	pocket-id (OIDC, optional)
headscale	Self-hosted Tailscale control server, OIDC login.	pocket-id (OIDC)
webfinger	Serves /.well-known/webfinger for OIDC discovery; redirects the rest.	pocket-id (issuer)
squid	SSL-bump caching forward proxy — static-content cache + TLS interception via a local CA. The exception: a forward proxy, not a Caddy/LE site.	—
copyparty	Portable file server — web UI/WebDAV behind Caddy, plus direct SFTP + FTPS. Ships a security-notices-aware updater.	—
simplex	SimpleX SMP + XFTP relay with Tor hidden services + encrypted backups.	globals/age-pubkey.txt

Conventions

• Alpine + Docker Compose + Caddy/Let's Encrypt across every stack (except squid, which is a forward proxy with a local TLS-interception CA — no Caddy/LE).
• build.sh → deploy.sh: each stack's build.sh embeds its docker-compose.yml / Caddyfile / .env.example into a single self-contained deploy.sh (base64 tar.gz). That one file can be scp'd to a host and run on its own. Rebuild after editing the loose files. (simplex is the exception — it deploys via install-simplex.sh.)
• Secrets live in .env (generated on first deploy) and globals/globals.env, both git-ignored. Only *.example templates and public keys are committed.
• Non-interactive: every deploy.sh honors SKIP_PROMPTS=1 with values supplied via the environment — which is how the per-deployment cloud-init.yml templates stand a stack up unattended.

Cloud-init

• Provision a host: cloud-init/base.yml (hostname + MOTD + SSH hardening) or cloud-init/jumphost.yml (bastion). Distro-agnostic — they install prerequisites for whatever the image is, then run the scripts.
• Stand up a stack: each deployment ships its own cloud-init.yml (e.g. deployments/pocket-id/cloud-init.yml). These assume a fresh VM, so they harden SSH first (harden-ssh.sh, HARDEN_SSH=1 by default) and then deploy the stack.

Fill in REPO_URL and the values at the top of the runcmd block, paste as instance user-data, and the host configures itself on first boot.

Multi-OS notes

The host-provisioning scripts (setup-host, harden-ssh, harden-jumphost, sshuser) and the six Docker stacks (pocket-id, beszel, headscale, webfinger, squid, copyparty) run on Alpine, Debian, and Alma. Distro differences live in scripts/oslib.sh — package manager (apk/apt/dnf), init system (OpenRC/systemd), sshd service name, the per-distro sftp-server path, hostname, boot hooks, and the sshguard log source/backend.

simplex remains Alpine-targeted — it depends on awall and Tor hidden services with Alpine-specific wiring, so it isn't part of the tri-distro set.

Host firewall

scripts/harden-firewall.sh installs a deny-by-default baseline, with the backend chosen per family (override with FW_BACKEND=iptables|firewalld):

• Alpine / Debian → iptables. INPUT drops everything except loopback, established/related, ICMP, and SSH on the configured port — plus any ports a deployment registers.
• Alma / RHEL → firewalld (its native firewall). The default zone is already deny-by-default; we strip the stock ssh/cockpit services, open SSH + registered ports, and let sshguard block via the sshguard-firewalld backend (no INPUT → sshguard jump needed).

OUTPUT/egress stays open and FORWARD is left untouched, so Docker container networking is unaffected. The harden scripts and cloud-init/base.yml / jumphost.yml install it automatically (ENABLE_FIREWALL=1 by default).

• Configurable SSH port — read live from sshd_config, so a bastion on 2222 is firewalled correctly with no extra flags. Restrict the source with FW_SSH_SOURCE=<cidr>; drop ping with FW_ALLOW_PING=0.
• Native persistence, no boot hook — on iptables hosts the ruleset is saved and restored by the distro's own package: iptables + ip6tables (Alpine/OpenRC) or iptables-persistent (Debian); the saved ruleset carries the INPUT → sshguard jump. On firewalld hosts every change is --permanent, so it persists across reboot natively and sshguard manages its own blocks.
• Scripted additions — deployments drop a rule file and re-apply:

  printf '80/tcp\n443/tcp\n' > /etc/firewall/ports.d/mystack.rule
  /usr/local/sbin/firewall-apply
  

  or interactively: harden-firewall.sh allow 443/tcp 51820/udp / allow web / deny 51820/udp / list.
• Docker caveat — containers published with -p (e.g. Caddy's 80/443) reach the host through nat/FORWARD and bypass INPUT, so the firewall neither blocks nor needs to open them; the per-stack rule files are belt-and-braces for any host-bound bind and self-documentation.
• Recovery — harden-firewall.sh disable un-locks you: on iptables it flushes the rules and sets INPUT back to ACCEPT (persisted); on firewalld it re-opens SSH (the ssh service + the configured port) and leaves firewalld running. A re-apply never drops the live SSH session — on iptables the established-connection accept is added before the policy flips to DROP, and firewalld reloads preserve established connections.

SSH login notifications

The harden scripts can install a pam_exec hook (scripts/ntfy-ssh-login.sh) that posts every SSH login to an ntfy topic. Enable it by passing NTFY_URL (and optionally NTFY_TOKEN) when running harden-ssh.sh / harden-jumphost.sh, or via the launcher / jumphost cloud-init. Each alert reports:

• the user and source IP,
• the SSH key they authenticated with (fingerprint, via ExposeAuthInfo),
• a region tag so you know which bastion fired it (derived from the host's FQDN, e.g. us-evi-1),
• best-effort the next hop of a ProxyJump (see caveat below).

Filtering is by group: NOTIFY_GROUPS limits alerts to certain groups/security levels, and NOTIFY_PRIORITY_MAP sets a per-group ntfy priority. Config lives at /etc/ssh-notify.conf (mode 0600; scripts/ssh-notify.conf.example documents every key). A publish token is optional — leave it empty for a read-gated topic.

Jump-target caveat: a ProxyJump opens a direct-tcpip channel (no session), so the destination never reaches pam_exec. The bastion only logs it at LogLevel VERBOSE/DEBUG; harden-jumphost.sh sets VERBOSE and the notifier parses the log best-effort. If the target isn't in the log it is simply omitted.

Daily updates

scripts/auto-update.sh keeps a host patched unattended — ideal for an SSH-only bastion, where a routine upgrade can barely break anything. harden-jumphost.sh schedules it by default (set AUTO_UPDATE=0 to skip); harden-ssh.sh takes AUTO_UPDATE=1. It runs daily via busybox crond (/etc/periodic/daily) on Alpine or a systemd timer on Debian/Alma.

Each run:

• applies all in-branch package upgrades (apk/apt/dnf);
• reports a new Alpine branch (e.g. 3.21 → 3.22) by default — that rewrites the repo branch, so it's opt-in. Set ALLOW_RELEASE_UPGRADE=1 to also apply it: it repoints /etc/apk/repositories to the newest stable vX.Y (never edge), runs apk upgrade --available, and forces a reboot flag. Debian/Alma stay report-only;
• detects when a reboot is needed (kernel/libc/openssl). AUTO_REBOOT controls it: 0 = never (just flag), 1 = always, idle = only when no SSH connections are active — so a bastion reboots itself once the admins and ProxyJump tunnels have cleared, never mid-session. harden-jumphost.sh defaults the bastion to idle. A deferred reboot is tracked in /run and retried each day until it happens;
• sends an ntfy summary (reusing /etc/ssh-notify.conf).

Schedule it standalone with auto-update.sh install (or via the launcher), run a pass now with auto-update.sh run, and preview safely with DRY_RUN=1 auto-update.sh run.

License

MIT.