Hardened hosts rejected clients that implement the very same key exchange. The
KEX list was assembled from version arithmetic and emitted only the
standardised spellings:
KexAlgorithms mlkem768x25519-sha256,sntrup761x25519-sha512
OpenSSH called that hybrid sntrup761x25519-sha512@openssh.com before the method
was standardised (8.5, in the default proposal from 8.9) and
sntrup761x25519-sha512 after (9.9), and KEXINIT matches names byte-exactly with
no alias resolution -- so every client older than the rename was refused with
"no matching key exchange method found" despite implementing the algorithm. The
same arithmetic was a latent server-side bug: on OpenSSH 9.0-9.8 it wrote the
post-standardisation name into sshd_config, which those builds do not know, and
sshd fatals on an unknown KexAlgorithms token rather than starting.
Ask the binary instead of guessing. oslib gains kex_supported(),
ssh_kex_pq_list(), ssh_kex_classic_list(), ssh_kex_list() and ssh_kex_has_pq(),
which filter candidates through `ssh -Q kex` and offer every spelling the host
actually has. Version thresholds are gone, and with them both failure modes --
including on distros whose backports make the version string meaningless.
SSH_ALLOW_CLASSIC_KEX=1 (off by default) additionally offers curve25519-sha256
and its @libssh.org spelling. Some clients have no PQ method at all: notably
Windows' in-box ssh.exe, which is not merely old -- Microsoft's fork compiles
sntrup761 out because it needs C99 VLAs that MSVC lacks, so even a fully patched
9.5p2 reports zero PQ methods. The knob is a real trade and says so in the
warning, the generated sshd_config comment, and the README: such a session is
safe against a classical attacker but has no store-now-decrypt-later protection.
Modern clients still negotiate PQ, since the client's preference order decides.
Three defects found reviewing the above, fixed here:
- the printed pre-reload verification command pinned the server's full list via
`-o KexAlgorithms=`, which ssh rejects at option-parse time when the client
lacks any one name. That made the one safety gate before a wholesale
sshd_config swap a false negative for exactly the clients this commit admits.
Dropped, matching harden-jumphost.sh.
- the no-PQ branch was unreachable: without the opt-in the classical names are
never collected, so a host with no PQ hybrid died reporting "no usable key
exchange method" instead of the actionable message written for it. The branch
now keys off a separate PQ probe, and the empty-list die is narrowed to a
genuinely empty `ssh -Q kex`.
- SSH_VER is cosmetic but its grep could abort the whole run under pipefail on
any banner that does not match (vendor forks, OpenSSH_for_Windows_9.5p2) --
silently, with no message. Guarded.
Wired through cloud-init/base.yml and jumphost.yml, since harden-ssh.sh rewrites
sshd_config wholesale on every run and a hand edit there does not survive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Proxmox VE and Proxmox Mail Gateway are Debian, so os_detect classified them as
debian and harden-firewall.sh installed the iptables backend on top of
pve-firewall.
The conflict is not the obvious one. pve-firewall does NOT delete third-party
rules: it restores with `iptables-restore -n` (--noflush), only ever flushes
chains matching its own patterns (PVEFW-*, tapNiM-*, vethNiM-*, fwbrN-*,
GROUP-*), appends `-A INPUT -j PVEFW-INPUT` only when that hook is missing, and
never sets a built-in chain's policy. Our rules survive it. We are the ones
doing damage:
- `-P INPUT DROP` is ours alone, and PVEFW-HOST-IN RETURNs on accept rather than
ACCEPTing (it still has to check tap rules), so traffic Proxmox explicitly
allowed falls out of its chain onto our DROP -- we silently override the
platform's own accepts.
- netfilter-persistent restores at boot with a full iptables-restore (no
--noflush), wiping PVE's hook along with everything else until the daemon
re-appends it ~10s later.
- a deny-by-default chain has to enumerate the whole platform to stay usable:
8006, 5405-5412/udp corosync, 60000-60050, 5900-5999, 3128, 22, 111/udp, plus
Ceph when hyperconverged -- and `-i lo`, or pveproxy loses pvedaemon on :85.
- under the nftables backend (PVE 8.2+) an nft DROP beats an iptables ACCEPT, so
our rules would not even be authoritative.
So don't manage a firewall there at all:
- oslib: is_proxmox() -- matches hosts shipping pve-firewall (VE/PMG), not PBS.
- harden-firewall.sh: a third backend, "pve", that deliberately does nothing.
apply explains and exits 0, allow/deny refuse loudly rather than fake success
for a rule they didn't add, list shows pve-firewall status. It overrides an
explicit FW_BACKEND; FW_IGNORE_PVE=1 is the one escape hatch.
- harden-ssh.sh / harden-jumphost.sh: skip the firewall and install the
standalone INPUT -> sshguard boot hook instead. That jump is safe alongside
pve-firewall -- inserted with -I, it sits ahead of the appended PVEFW-INPUT
hook and keeps first look at NEW connections.
Detection only helps hosts built from here on, so `disable` cleans up one that
was hardened earlier: it detects leftovers (and says which signal fired), sets
INPUT ACCEPT *before* flushing so it can't drop the SSH session it runs over,
deletes /etc/firewall and the engine, disables boot restore and renames the
saved rulesets aside rather than persisting the open state, re-adds the sshguard
jump, and restarts pve-firewall. `apply` points at it when it spots leftovers.
Documented plainly that skipped is NOT protected: Proxmox's firewall is off by
default (cluster-wide enable defaults to 0, and the daemon tears its chains down
every ~10s while it is), so these hosts have no host firewall until someone
enables it -- and the node panel's "Firewall: Yes" is ignored while the
datacenter one says No.
Also: svc_disable + fw_restore_services/fw_saved_files in oslib
(fw_enable_restore now derives from the former), and usage() prints the whole
header block instead of a hardcoded line range.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two Alpine-only failures from an irc-1 run.
1. harden-firewall.sh died with
line 161: /usr/local/sbin/firewall-apply: No such file or directory
and left INPUT unfiltered. Alpine's baselayout ships /usr/local/{bin,lib,
share} but no sbin, and nothing created it. Create it before writing the
engine; same guard in oslib's install_boot_hook / install_daily_job (which
`install` into /usr/local/sbin, and `install` does not make parent dirs) and
for /usr/local/bin in harden-jumphost.
2. The ntfy login notifier never fired despite reporting ACTIVE. Alpine keeps
PAM support in a SEPARATE binary: openssh-server gives /usr/sbin/sshd (no
PAM), openssh-server-pam gives /usr/sbin/sshd.pam. The OpenRC init only
picks the PAM one in start_pre (checkconfig -> update_command), and its
reload/stop match the process with `start-stop-daemon --exec "$command"`.
So a host already running /usr/sbin/sshd never swaps: reload signals a
non-matching process, stop matches nothing, a later start hits "address
already in use". sshd keeps serving without PAM, so /etc/pam.d/sshd -- and
the pam_exec hook in it -- is never consulted.
Add oslib sshd_wanted_binary / sshd_running_binary / sshd_apply_config, and
end the harden scripts with sshd_apply_config instead of svc_reload: reload
as before, but on Alpine stop by pidfile and start again when the running
binary isn't the one the config calls for. Established sessions are separate
processes and survive; only the listener blinks.
install_login_notifier now checks that PAM can actually be reached on Alpine
(sshd.pam present, UsePAM yes) and warns with the fix instead of reporting a
hook that can never run as ACTIVE -- the same "silent notifier" trap the
Alma run hit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A fresh Alma box has firewalld active, and the iptables-based harden-firewall.sh
refused to run there (caught by harden-ssh's '|| warn', so the host firewall was
silently skipped). Use firewalld natively on the rhel family instead of fighting it.
- harden-firewall.sh: family-aware backend. On rhel, apply/allow/deny/list/disable
drive firewall-cmd (deny-by-default zone, SSH + registered ports, ping policy,
source-restricted rich rules); Alpine/Debian keep the iptables engine unchanged.
FW_BACKEND=iptables|firewalld overrides.
- oslib: install_firewalld(); sshguard_backend() prefers sshg-fw-firewalld on rhel
so brute-force blocks land in firewalld (no INPUT->sshguard jump needed).
- Deployments already fall through to a firewall-cmd branch when the iptables
engine is absent, so they need no changes.
- README + script header document the per-family backend.
harden-ssh / harden-jumphost are unchanged -- they call harden-firewall.sh apply
and read sshguard_backend(), so the switch happens underneath them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
install_openssh ended with '[[ "$OS_FAMILY" == alpine ]] && pkg_install ...'.
As the function's LAST statement, that trailing test returns 1 on every
non-Alpine OS (a false '[[ ]]' exits 1), so the function reported failure even
when the packages installed fine. Harmless while the call was bare under set -e
(a short-circuited && is exempt), but the new 'install_openssh || die' guard
read it as a real failure and aborted harden-ssh on Alma right after
'Installing OpenSSH server...'.
Fix: convert the Alpine-only linux-pam step to an if-block, and add '|| return 1'
to the main install so a genuine package failure still propagates honestly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a fresh AlmaLinux 9.8 box, install_bruteforce_protection ran unguarded under
'set -euo pipefail'. When sshguard (from EPEL) wasn't installable at that moment,
the single failed dnf aborted the ENTIRE harden run before it wrote sshd_config
or installed the pam_exec login notifier -- leaving a stock, unhardened box and a
silently-missing ntfy hook.
- oslib: install the iptables backend best-effort first, then sshguard, and
return sshguard's status so callers can treat it as non-fatal.
- harden-ssh/harden-jumphost: install_openssh now dies with a clear message on
failure; sshguard is '|| warn' so sshd hardening and the notifier still apply.
- install_login_notifier verifies the script + pam hook landed and logs
'Login notifier ACTIVE' (or a loud warning) instead of failing silently.
- ntfy-ssh-login.sh: NTFY_DEBUG=1 logs delivery attempts + curl errors to
/var/log/ssh-notify.log so the next silent failure leaves a trace.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a reusable iptables baseline that hardens hosts with ICMP + SSH
defaults and lets deployments register the ports they need. INPUT is
deny-by-default (loopback, established, ICMP, SSH on the configured port,
plus registered ports); OUTPUT stays open and FORWARD is left untouched so
Docker container networking is unaffected.
Persistence is native -- no boot hook. Rules are saved and restored by the
distro's own package (iptables/ip6tables on Alpine, iptables-persistent on
Debian, iptables-services on Alma) via the new oslib helpers
install_iptables / fw_save_cmd / fw_enable_restore. The saved ruleset
carries the INPUT->sshguard jump, so brute-force protection survives reboot
without the old sshguard-iptables hook.
A self-contained /usr/local/sbin/firewall-apply rebuilds INPUT from
declarative drop-ins under /etc/firewall/ports.d and runs the native save,
so deployments add a port without needing the repo present:
printf '80/tcp\n443/tcp\n' > /etc/firewall/ports.d/mystack.rule
/usr/local/sbin/firewall-apply
- SSH port read live from sshd_config (custom bastion ports just work);
FW_SSH_SOURCE restricts the source CIDR; FW_ALLOW_PING gates echo
- harden-ssh.sh / harden-jumphost.sh install it when ENABLE_FIREWALL=1
(default) and skip the sshguard-only hook; ENABLE_FIREWALL=0 keeps it
- cloud-init base.yml / jumphost.yml forward the toggle
- the four stack deploy.sh open_web_ports() register 80/443 via the
firewall (ufw/firewalld kept as fallback); Docker-published ports bypass
INPUT, so this is belt-and-braces and self-documenting
- README + cloud-init/README document the mechanism, Docker caveat, and the
`disable` recovery path
Restructure around a single entry point (automations.sh) with a Gum wizard and
a self-extracting bundle for repo-less installs. Add scripts/oslib.sh so the
provisioning scripts (setup-host, harden-ssh, harden-jumphost, sshuser) run on
Alpine/Debian/Alma; seed root keys from globals/.
- ntfy SSH-login alerts (user, source IP, key, region, jump target) via pam_exec
- daily auto-updates with AUTO_REBOOT=idle (reboots only when no SSH active) and
opt-in Alpine stable-branch upgrades
- generic + per-deployment cloud-init; Gitea release workflow on tag
- README/LICENSE/.gitignore/.gitattributes (force LF); repo URLs -> Gitea
57_WolveandClaude Opus 5