feat(knot-dns): authoritative Knot DNS node deployment

Native Alpine deployment for the anycast DNS estate -- no Docker, no Caddy,
alongside squid and openbao as an exception to the repo norm. Knot binds :53
directly, needs real client addresses for RRL and DNS cookies, and its DNSSEC
key store must live on the host filesystem.

Deploys a NODE. Zone data lives in the separate dns repo and arrives from its
pipeline. The split is /etc/knot/knot.conf: written here once as a skeleton of
include: lines covering only what belongs to a box (identity, NSID, storage
paths, listen, logging, control socket); everything that is DNS policy --
templates, dnssec policy, remotes/ACLs, modules, the domain inventory and the
zone files -- is delivered by the dns repo.

knsctl replaces adddns.pl and adddnssec.pl, fixing four defects:
- the duplicate check searched for the domain in BIND named.conf double-quote
  syntax (/"$domain"/) against unquoted YAML, so it could never match; only
  the -f zone-file test ever caught anything
- neither script consulted the other class's manifest, so a domain already in
  public.conf could be appended to dnssec.conf and fail the reload AFTER both
  files had been written
- nothing validated before reloading
- the reload was non-blocking, so a rejected config reported success

Its manifest matching is anchored on the YAML key and escapes the dot, so
barsrvno.de and srvnoXde no longer false-positive against srvno.de.

Aliases preserve the existing muscle memory with three corrections: -b on
every triggering knotc command (without it knotc returns OK when the command
was SENT, not when it succeeded); knzr (zone-reload) added alongside knrl
(reload), since reloading one zone's data is the right verb for a record
change and a full reload is only needed when a zone is added or removed; and
serial/NSID helpers that query unicast addresses, because asking the anycast
service address reaches whichever node is nearest and says nothing about
which node is stale.

Break-glass writes (add/remove/edit) warn and audit-log: they are overwritten
by the next pipeline deploy unless the change also lands in git. Removal
refuses to purge DNSSEC keys -- zone-purge +keys is irreversible on Knot
3.5.x, the key trash bin having arrived in 3.6.0 -- and prints the ordering
requirement, since removing a signed zone before the parent DS is withdrawn
is an outage for validating resolvers rather than a graceful shutdown.

deploy.sh, build.sh and cloud-init.yml are deliberately not included yet;
they are blocked on the Knot version decision, which sets the apk pin and
feature availability. See the Status section in the README.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.gitattributes

@@ -16,5 +16,7 @@ deployments/*/deploy.sh text eol=lf
 
 # Extension-less / unusual-extension scripts and configs that must stay LF.
 deployments/ergo/ergoctl text eol=lf
+deployments/knot-dns/knsctl text eol=lf
+*.tmpl  text eol=lf
 *.caddy text eol=lf
 *.motd  text eol=lf

deployments/knot-dns/README.md

@@ -0,0 +1,101 @@
+# knot-dns
+
+Authoritative DNS node — [Knot DNS](https://www.knot-dns.cz/) on Alpine, native
+(no container), for a global anycast estate with automatic DNSSEC.
+
+**Exception to the repo norm**, alongside `squid` and `openbao`: no Docker, no
+Caddy, no Let's Encrypt. Knot binds :53 directly, needs real client addresses
+for RRL and DNS cookies, and its DNSSEC key store must live on the host
+filesystem. Containerising it buys nothing and costs the host firewall.
+
+## What this deploys, and what it does not
+
+This stands up **a node**. It does not manage zones — that is the
+[`dns`](../../../dns) repo, and its pipeline delivers zone data here.
+
+The split point is `/etc/knot/knot.conf`. This deployment writes it once as a
+skeleton of `include:` lines covering only what belongs to a *box*: identity,
+NSID, storage paths, listen addresses, logging, control socket. Everything that
+belongs to *DNS policy* — templates, DNSSEC policy, remotes/ACLs, modules, the
+domain inventory, the zone files — arrives from the `dns` repo.
+
+| Deployed here, once | Delivered by the `dns` repo, continuously |
+|---|---|
+| `knot.conf` skeleton | `templates.conf`, `policy.conf`, `remotes.conf`, `modules.conf` |
+| `secrets.conf` (TSIG, rendered locally, never in git) | `dnssec.conf`, `public.conf`, `arpa.conf` |
+| `knsctl`, aliases, MOTD, `zone.tmpl` | `zones/**/*.zone` |
+| packages, users, directories, firewall | |
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `knot.conf` | Bootstrap skeleton. `@NODE_ID@` / `@LISTEN@` substituted from `.env`. |
+| `secrets.conf.example` | TSIG `key:` block. Rendered to `/etc/knot/secrets.conf`, `0640 root:knot`. Never committed. |
+| `knsctl` | Admin CLI. Replaces `adddns.pl` / `adddnssec.pl` and fixes four defects in them. |
+| `knotdns-aliases.sh` | `/etc/profile.d/` — the existing muscle memory, with `-b` added everywhere. |
+| `knot-dns.motd` | The cheat-sheet MOTD. |
+| `zone.tmpl` | New-zone skeleton (mirrors `dns/skeleton/zone.tmpl`). |
+
+## Roles
+
+`ROLE` in `.env` selects what the node is:
+
+- **`primary`** — holds zone files, signs with automatic DNSSEC, owns the KASP
+  database, notifies the secondaries. Exactly one node. Not necessarily
+  public-facing: a hidden (stealth) primary is the recommended shape.
+- **`secondary`** — receives zones by AXFR/IXFR with TSIG, serves queries,
+  signs nothing. `dnssec-signing` must be **off**; a secondary serves data that
+  is already signed.
+
+Adding a public node is a `secondary` deploy plus one address in the primary's
+`remotes.conf`. With catalog zones configured, the new node self-populates.
+
+## What survives from the previous setup
+
+Everything an operator types. `knrl`, `knsc`, `knzc`, `knpbz`, `knsec`,
+`knarpa`, `ozf` all still work, from the same paths, on the same box. Three
+corrections were folded in:
+
+1. **`-b` on every triggering `knotc` command.** Without it `knotc` returns OK
+   when the command was *sent*, not when it succeeded, so a rejected zone file
+   reports green. `knrl` was a bare `knotc reload`.
+2. **`knzr` (`zone-reload`) added** next to `knrl` (`reload`). Reloading one
+   zone's data is the right verb for a record change — smaller blast radius,
+   and a parse error in one zone cannot disturb the others. `knrl` reloads
+   configuration and is only needed when a zone is added or removed.
+3. **`knsctl` replaces the Perl scripts**, whose duplicate check searched for
+   the domain in BIND `named.conf` double-quote syntax against unquoted YAML
+   and therefore never matched; which never consulted the other class's
+   manifest, so a domain in `public.conf` could be appended to `dnssec.conf`
+   and fail the reload after both files were already written; and which
+   reloaded without validating.
+
+## Key material
+
+`/var/lib/knot/keys` (the KASP LMDB) is node-local to the primary and
+replicates nowhere. Losing it is the one unrecoverable failure in this system:
+Knot will happily generate fresh KSKs, every published DS will point at keys
+that no longer exist, and every signed domain goes bogus until each registrar
+is updated by hand.
+
+```sh
+knotc -b zone-backup +backupdir /var/backups/knot/$(date -u +%FT%H) +journal
+```
+
+**`+journal` is not the default** — the documented default filter set excludes
+it, and in this configuration the journal holds the only durable copy of the
+signed zone and the last real SOA serial. A default `zone-backup` captures your
+keys and loses your zone data. Restore order is config → KASP + journal → *then*
+start `knotd`, never the reverse.
+
+## Usage
+
+```sh
+# interactive
+bash deploy.sh
+
+# non-interactive (this is what cloud-init.yml does)
+ROLE=secondary NODE_ID=ANYCAST-DNS-3 PRIMARY_ADDR=10.1.24.64 \
+    SKIP_PROMPTS=1 bash deploy.sh
+```

deployments/knot-dns/knot-dns.motd

@@ -0,0 +1,48 @@
+
+  ┌────────────────────────────────────────────────────────────────────────┐
+  │                    !! DANGER - CRITICAL INFRASTRUCTURE !!              │
+  │                                                                        │
+  │            YOU are connected to a CRITICAL SYSTEM                      │
+  │      CHECK YOUR WORK BEFORE RELOAD / RESTART OF THE SERVICE            │
+  ├────────────────────────────────────────────────────────────────────────┤
+  │   Data Center: @DATACENTER@                                            │
+  │   Hostname:    @FQDN@                                                  │
+  │   Node ID:     @NODE_ID@                                               │
+  │   Role:        @ROLE@                                                  │
+  └────────────────────────────────────────────────────────────────────────┘
+
+  Zones are managed in GIT. Changes belong in a signed PR to the dns repo.
+  Anything you edit here is overwritten by the next deploy unless you also
+  land it in git. Use knsctl for anything that writes -- it audit-logs.
+
+  Will's Cheat Sheet  \(^-^)/
+
+  Reload / check
+    knrl              Reload config (use when a zone was ADDED or REMOVED)
+    knzr domain.tld   Reload ONE zone's data (prefer this for record changes)
+    knsc              Check server config is valid
+    knzc domain.tld   Check a zone via the running server (no output = OK)
+    knst              Zone status for everything
+
+  Navigate / edit
+    knpbz             Go to public zones      /var/lib/knot/zones/public
+    knsec             Go to DNSSEC zones      /var/lib/knot/zones/dnssec
+    knarpa            Go to ARPA zones        /var/lib/knot/zones/arpa
+    ozf domain.tld    Open zone file here in $EDITOR
+    ozfa domain.tld   Find + open it anywhere (warns: break-glass)
+
+  Inspect
+    knser domain.tld  This node's serial for a zone
+    knsers            Serial comparison across PEERS
+    knsid domain.tld  Which node answered (EDNS NSID)
+    knstale           Look for silently-ignored zone files
+
+  Domains
+    knsctl add domain.tld --class dnssec|public     BREAK-GLASS
+    knsctl remove domain.tld                        BREAK-GLASS - read the
+                                                    ordering warning first
+    knsctl help
+
+  Every knotc command here blocks (-b). Without it knotc reports OK when the
+  command was SENT, not when it succeeded -- a rejected zone looks green.
+

deployments/knot-dns/knot.conf

@@ -0,0 +1,51 @@
+# knot.conf -- BOOTSTRAP SKELETON. Deployed once by deploy.sh; rarely changes.
+#
+# This file owns only what belongs to the BOX: identity, storage paths, listen
+# addresses, logging, control socket. Everything that belongs to DNS POLICY --
+# templates, dnssec policy, remotes/ACLs, modules, and the domain inventory --
+# is delivered from the `dns` repo by its CD pipeline. Do not add zone: or
+# template: sections here.
+#
+# Values in @UPPER@ are substituted by deploy.sh from .env.
+#
+# Reload:  knotc -b conf-check && knotc -b reload
+# The -b is not optional. Without it knotc returns OK when the command was
+# *sent*, not when it succeeded -- a rejected zone file reports green.
+
+server:
+    identity: "@NODE_ID@"
+    nsid: "@NODE_ID@"
+    rundir: "/run/knot"
+    user: knot:knot
+    automatic-acl: on
+    listen: [ @LISTEN@ ]
+
+control:
+    listen: "/run/knot/knot.sock"
+    timeout: 0
+
+log:
+  - target: syslog
+    server: warning
+    control: warning
+    zone: info          # zone-load rejections and KSK-submission results log
+                        # at info; at warning they are invisible
+    quic: warning
+    any: error
+
+database:
+    storage: /var/lib/knot/database
+    journal-db: /var/lib/knot/journal
+    kasp-db: /var/lib/knot/keys
+    timer-db: /var/lib/knot/timer
+    catalog-db: /var/lib/knot/catalog
+
+# --- delivered by the `dns` repo pipeline; see that repo's knot/ directory ---
+include: /etc/knot/secrets.conf     # rendered by deploy.sh, 0640 root:knot, NOT in git
+include: /etc/knot/remotes.conf
+include: /etc/knot/policy.conf
+include: /etc/knot/modules.conf
+include: /etc/knot/templates.conf
+include: /etc/knot/arpa.conf
+include: /etc/knot/dnssec.conf
+include: /etc/knot/public.conf

deployments/knot-dns/knotdns-aliases.sh

@@ -0,0 +1,62 @@
+# knotdns-aliases.sh -> /etc/profile.d/knotdns-aliases.sh
+#
+# The operator's existing muscle memory, preserved verbatim where it was
+# already right and corrected only where it was silently wrong. Sourced by
+# login shells on every Knot node.
+
+# -----------------------------------------------------------------------------
+# Reload / check
+# -----------------------------------------------------------------------------
+# -b is NOT cosmetic. Without it knotc returns OK as soon as the command has
+# been *sent* to the server, not when it succeeded -- so a zone file the server
+# rejected reports success. Every triggering command here blocks.
+alias knrl='knotc -b reload'          # config reload: picks up new/removed zones
+alias knsc='knotc -b conf-check'      # validate configuration
+alias knst='knotc zone-status'
+
+# Reload ONE zone's data. Prefer this over knrl for record changes: smaller
+# blast radius, and a parse failure in one zone cannot disturb the others.
+# knrl (full reload) is only needed when a zone was ADDED or REMOVED.
+knzr() { knotc -b zone-reload "$1"; }
+
+# Check a zone using the server's own load path with that zone's configured
+# semantic-checks applied. knotc(8) marks zone-check "(*)" -- a LOCAL operation
+# requiring only a configuration, so it works with knotd stopped, and CI can
+# run it against a checked-out config without standing a server up.
+# Stricter and more config-aware than standalone kzonecheck, which knows
+# nothing about which template a zone uses.
+knzc() { knotc -b zone-check "$1"; }
+
+# -----------------------------------------------------------------------------
+# Navigation
+# -----------------------------------------------------------------------------
+alias knpbz='cd /var/lib/knot/zones/public'
+alias knsec='cd /var/lib/knot/zones/dnssec'
+alias knarpa='cd /var/lib/knot/zones/arpa'
+
+# Open a zone file in the editor, relative to the current directory -- composes
+# with the cd aliases above (knsec; ozf srvno.de).
+ozf() { "${EDITOR:-vi}" "$1".zone; }
+
+# Same, but finds the file wherever it lives, and warns that a hand edit is a
+# break-glass action that the next deploy will overwrite.
+alias ozfa='knsctl edit'
+
+# -----------------------------------------------------------------------------
+# Inspection
+# -----------------------------------------------------------------------------
+# Serial for one zone from this node specifically. Always query the UNICAST
+# address: asking the anycast service address reaches whichever node is
+# nearest, which tells you nothing about which node is stale.
+knser() { kdig +short @127.0.0.1 SOA "$1" | awk '{print $3}'; }
+
+# Which node actually answered? NSID is set per node in knot.conf.
+knsid() { kdig +nsid "@${2:-127.0.0.1}" SOA "$1" | grep -i nsid; }
+
+# Serial comparison across the estate. Set PEERS in /etc/profile.d or the env.
+alias knsers='knsctl serials'
+
+# Catch a silently-ignored zone file: on a cold start Knot can log this and
+# serve the journal copy instead, leaving git and the served zone diverged with
+# a clean exit code everywhere.
+alias knstale='logread 2>/dev/null | grep -i "ignoring zone file" || dmesg | grep -i "ignoring zone file"'

deployments/knot-dns/knsctl

@@ -0,0 +1,256 @@
+#!/usr/bin/env bash
+#
+# knsctl -- Knot DNS admin CLI. Replaces adddns.pl / adddnssec.pl.
+#
+# Fixes four real defects in the Perl scripts it replaces:
+#   1. Their duplicate check searched for the domain wrapped in double quotes
+#      (/"$domain"/) -- BIND named.conf syntax. What they wrote is unquoted
+#      YAML (`  - domain: example.com`), so the check could never match. Only
+#      the -f zone-file test ever caught a duplicate.
+#   2. Neither script consulted the OTHER class's manifest, so a domain already
+#      in public.conf could be appended to dnssec.conf; the following
+#      `knotc reload` then failed on a duplicate zone -- after both files had
+#      already been written, leaving the server misconfigured.
+#   3. Nothing validated before reloading.
+#   4. The reload was non-blocking, so a rejected config reported success.
+#
+# NORMAL CHANGES GO THROUGH GIT. add/remove/edit here are BREAK-GLASS: they
+# write directly to this server and the next pipeline deploy will overwrite
+# them unless the change is also made in the `dns` repo. They warn and they
+# audit-log.
+set -euo pipefail
+
+ZONES_ROOT=${ZONES_ROOT:-/var/lib/knot/zones}
+CONF_DIR=${CONF_DIR:-/etc/knot}
+SKELETON=${SKELETON:-$CONF_DIR/zone.tmpl}
+AUDIT=${AUDIT:-/var/log/knsctl-audit.log}
+KNOT_OWNER=${KNOT_OWNER:-knot:knot}
+
+die()  { printf 'knsctl: %s\n' "$*" >&2; exit 1; }
+warn() { printf 'knsctl: %s\n' "$*" >&2; }
+audit() {
+    printf '%s %s %s\n' "$(date -u +%FT%TZ)" "${SUDO_USER:-${USER:-root}}" "$*" \
+        >>"$AUDIT" 2>/dev/null || true
+}
+
+usage() {
+    cat <<'USAGE'
+knsctl <command> [args]
+
+Inspection (always safe):
+  list                    list configured zones
+  status [domain]         zone status; all zones if omitted
+  check <domain>          knotc zone-check -- the running server's own load path
+  serials                 compare this node's serials against PEERS
+  path <domain>           print the zone file path
+  conf-check              validate the configuration
+
+Apply:
+  reload [domain]         zone-reload <domain>; full config reload if omitted
+
+Break-glass (writes to this server; must be reconciled into git):
+  add <domain> --class dnssec|public
+  remove <domain>
+  edit <domain>           open the zone file in $EDITOR
+
+Environment:
+  PEERS="10.1.24.68 10.1.24.69"   peers for `serials`
+USAGE
+}
+
+# Escape a domain for use as a literal in a POSIX ERE. valid_domain() already
+# restricts input to [a-z0-9.-], so the dot is the only metacharacter that can
+# appear. (Do NOT reach for a general bracket expression here: one starting
+# "[." opens a POSIX collating symbol and silently breaks the pattern.)
+ere_quote() { printf '%s' "$1" | sed 's/\./\\./g'; }
+
+# Which manifest, if any, already lists this domain? Checks ALL of them --
+# looking at only one was the Perl scripts' second bug.
+find_in_manifests() {
+    local d rx f
+    d=$1; rx=$(ere_quote "$d")
+    for f in "$CONF_DIR"/dnssec.conf "$CONF_DIR"/public.conf "$CONF_DIR"/arpa.conf; do
+        [[ -f $f ]] || continue
+        # Anchored on the YAML key so `foo.net` cannot match `barfoo.net`.
+        if grep -qE "^[[:space:]]*-[[:space:]]*domain:[[:space:]]*${rx}[[:space:]]*\$" "$f"; then
+            printf '%s\n' "${f##*/}"
+            return 0
+        fi
+    done
+    return 1
+}
+
+find_zone_file() {
+    local d c
+    d=$1
+    for c in dnssec public arpa; do
+        if [[ -f "$ZONES_ROOT/$c/$d.zone" ]]; then
+            printf '%s\n' "$ZONES_ROOT/$c/$d.zone"
+            return 0
+        fi
+    done
+    return 1
+}
+
+valid_domain() {
+    [[ $1 =~ ^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$ ]]
+}
+
+zone_names() {
+    knotc zone-status 2>/dev/null | awk '/^\[/{gsub(/[][]/, "", $1); print $1}'
+}
+
+cmd_add() {
+    local domain class serial zf rx
+    domain=${1:-}; class=""
+    shift || true
+    while [[ $# -gt 0 ]]; do
+        case $1 in
+            --class) class=${2:-}; shift 2 ;;
+            *) die "unknown option: $1" ;;
+        esac
+    done
+    [[ -n $domain ]] || die "usage: knsctl add <domain> --class dnssec|public"
+    valid_domain "$domain" || die "invalid domain: $domain"
+    case $class in
+        dnssec|public) ;;
+        *) die "--class must be dnssec or public" ;;
+    esac
+
+    local existing
+    if existing=$(find_in_manifests "$domain"); then
+        die "$domain is already configured in $existing"
+    fi
+    zf="$ZONES_ROOT/$class/$domain.zone"
+    [[ -e $zf ]] && die "$zf already exists -- refusing to overwrite"
+    [[ -f $SKELETON ]] || die "skeleton not found at $SKELETON"
+
+    warn "BREAK-GLASS: adding $domain directly on this server."
+    warn "  The next pipeline deploy will remove it unless you also add it to"
+    warn "  the dns repo (zones/$class/$domain.zone + ci/gen-manifest.sh)."
+
+    # The serial is only ever a cold-start seed: Knot assigns the live serial
+    # under zonefile-load: difference-no-serial. It must still be large and
+    # RFC 1982-sane, so seed with today's dateserial rather than 1.
+    serial="$(date -u +%Y%m%d)01"
+    sed -e "s/@DOMAIN@/$domain/g" -e "s/@SERIAL@/$serial/g" "$SKELETON" >"$zf.tmp"
+    mv "$zf.tmp" "$zf"
+    chown "$KNOT_OWNER" "$zf"
+    chmod 0644 "$zf"
+
+    # Append to the manifest only after the zone file is in place, so a failure
+    # never leaves config referencing a file that does not exist.
+    printf '  - domain: %s\n    template: %s-records\n' "$domain" "$class" \
+        >>"$CONF_DIR/$class.conf"
+    chown "$KNOT_OWNER" "$CONF_DIR/$class.conf"
+
+    # Validate BEFORE reloading -- the Perl scripts reloaded blind.
+    if ! knotc -b conf-check; then
+        warn "conf-check failed; rolling back"
+        rx=$(ere_quote "$domain")
+        sed -i "/^[[:space:]]*-[[:space:]]*domain:[[:space:]]*${rx}[[:space:]]*\$/,+1d" \
+            "$CONF_DIR/$class.conf"
+        rm -f "$zf"
+        die "configuration invalid; no changes applied"
+    fi
+
+    # A NEW zone needs a full config reload. zone-reload only reloads data for
+    # an already-configured zone and would not see this one.
+    knotc -b reload || die "reload failed"
+    knotc -b zone-check "$domain" || warn "zone-check reported problems for $domain"
+
+    audit "add $domain class=$class serial=$serial"
+    printf 'added %s (%s), cold-start seed %s\n' "$domain" "$class" "$serial"
+    if [[ $class == dnssec ]]; then
+        printf 'DS submission to the registrar is manual: keymgr %s ds\n' "$domain"
+    fi
+}
+
+cmd_remove() {
+    local domain manifest zf rx confirm
+    domain=${1:-}
+    [[ -n $domain ]] || die "usage: knsctl remove <domain>"
+    manifest=$(find_in_manifests "$domain") || die "$domain is not configured"
+    zf=$(find_zone_file "$domain") || warn "no zone file found for $domain"
+
+    cat >&2 <<EOF
+knsctl: BREAK-GLASS removal of $domain
+
+  ORDER MATTERS. For a signed zone, removing it here before the parent's DS
+  record is withdrawn causes SERVFAIL for every validating resolver -- that is
+  an outage, not a graceful shutdown. The correct sequence is:
+
+    1. Publish the RFC 8078 delete signal, or remove the DS at the registrar.
+    2. Wait out the parent DS TTL (commonly 86400 at gTLDs).
+    3. Remove the NS delegation; wait out the NS TTL.
+    4. Only then remove the zone here.
+
+  This command performs step 4 only.
+
+  DNSSEC keys are deliberately NOT purged: 'zone-purge +keys' is irreversible
+  on Knot 3.5.x (the key trash bin arrived in 3.6.0). Orphaned keys are left
+  in place for a later, deliberate cleanup.
+EOF
+    read -r -p "Type the domain to confirm: " confirm
+    [[ $confirm == "$domain" ]] || die "aborted"
+
+    rx=$(ere_quote "$domain")
+    sed -i "/^[[:space:]]*-[[:space:]]*domain:[[:space:]]*${rx}[[:space:]]*\$/,+1d" \
+        "$CONF_DIR/$manifest"
+    knotc -b conf-check || die "conf-check failed after edit -- inspect $CONF_DIR/$manifest"
+    knotc -b reload || die "reload failed"
+    if [[ -n ${zf:-} ]]; then
+        mv "$zf" "$zf.removed-$(date -u +%Y%m%d)"
+        warn "zone file retained as $zf.removed-$(date -u +%Y%m%d)"
+    fi
+    audit "remove $domain manifest=$manifest"
+    printf 'removed %s from %s. DNSSEC keys retained.\n' "$domain" "$manifest"
+}
+
+cmd_serials() {
+    local peers d local_serial p
+    peers=${PEERS:-}
+    [[ -n $peers ]] || warn "set PEERS='10.1.24.68 ...' to compare across nodes"
+    # Query each node's UNICAST address. Querying the anycast service address
+    # reaches whichever node is nearest, which tells you nothing about which
+    # node is stale.
+    while read -r d; do
+        [[ -n $d ]] || continue
+        local_serial=$(kdig +short @127.0.0.1 SOA "$d" 2>/dev/null | awk '{print $3}')
+        printf '%-34s local=%-12s' "$d" "${local_serial:-?}"
+        for p in $peers; do
+            printf ' %s=%-12s' "$p" \
+                "$(kdig +short "@$p" SOA "$d" 2>/dev/null | awk '{print $3}')"
+        done
+        printf '\n'
+    done < <(zone_names)
+}
+
+case ${1:-} in
+    add)        shift; cmd_add "$@" ;;
+    remove)     shift; cmd_remove "$@" ;;
+    list)       zone_names ;;
+    status)     shift; knotc zone-status "$@" ;;
+    check)      shift; [[ -n ${1:-} ]] || die "usage: knsctl check <domain>"
+                knotc -b zone-check "$1" ;;
+    conf-check) knotc -b conf-check ;;
+    serials)    cmd_serials ;;
+    path)       shift; [[ -n ${1:-} ]] || die "usage: knsctl path <domain>"
+                find_zone_file "$1" || die "no zone file for $1" ;;
+    edit)       shift; [[ -n ${1:-} ]] || die "usage: knsctl edit <domain>"
+                f=$(find_zone_file "$1") || die "no zone file for $1"
+                warn "BREAK-GLASS: edits here are overwritten by the next deploy"
+                warn "  unless the same change is made in the dns repo."
+                audit "edit $1"
+                "${EDITOR:-vi}" "$f" ;;
+    reload)     shift
+                if [[ -n ${1:-} ]]; then
+                    # zone-reload for content; full reload only for config
+                    # changes (a new or removed zone).
+                    knotc -b zone-reload "$1"
+                else
+                    knotc -b reload
+                fi ;;
+    ''|-h|--help|help) usage ;;
+    *) die "unknown command: $1 (try: knsctl help)" ;;
+esac

deployments/knot-dns/secrets.conf.example

@@ -0,0 +1,15 @@
+# secrets.conf.example -> /etc/knot/secrets.conf   (0640 root:knot)
+#
+# NEVER COMMIT THE RENDERED FILE. It is in .gitignore.
+# Generate a secret with:  keymgr -t <keyname> hmac-sha256
+#
+# knot.conf includes this before every file that references these key ids.
+
+key:
+  - id: authortive-tsig            # DNS-1 <-> DNS-2 replication (AXFR/IXFR + NOTIFY)
+    algorithm: hmac-sha256
+    secret: REPLACE_ME
+
+  - id: admin-tsig                 # read-only AXFR for CI drift-checking and admins
+    algorithm: hmac-sha256
+    secret: REPLACE_ME

deployments/knot-dns/zone.tmpl

@@ -0,0 +1,26 @@
+;
+; BIND zone file for domain: @DOMAIN@
+;
+
+$ORIGIN @DOMAIN@.
+
+$TTL	3600
+
+@DOMAIN@.	IN	SOA	dns-1.datacenter.gg. dns.alphacentri.com. (
+	@SERIAL@	; Serial -- COLD-START SEED ONLY. Knot owns the live serial
+			;   (zonefile-load: difference-no-serial). Never edit this
+			;   value: CI fails on any change. It is only read when a
+			;   node starts with no journal entry for this zone.
+	3600		; Refresh -- fallback if a NOTIFY is missed
+	900		; Retry
+	1209600		; Expire -- 14d. MUST stay below rrsig-lifetime minus
+			;   rrsig-refresh (23d), or a secondary serves records
+			;   whose signatures have already expired.
+	300		; Negative Cache TTL
+)
+
+@DOMAIN@.	IN	NS	dns-1.datacenter.gg.
+@DOMAIN@.	IN	NS	dns-2.datacenter.gg.
+
+;; Edit Below This Line.
+