deploy: bound the scanner's memory and yield the box to the hold

The scanner shares a 1 GiB host with the hold, and nothing stopped it taking
the whole box down with it.

max_image_size was absent from the template, so production ran the 2 GiB
default. Peak RSS tracks image size and a 389 MiB compressed image already
reaches 561 MiB, so 512 MiB is the ceiling that fits. Sizing matters in this
order: max_image_size rejects before a byte is downloaded, while MemoryMax
kills mid-scan, and the hold then returns the row to pending and the restarted
scanner is handed the same job again, which is a permanent crash loop on one
image. If the cgroup cap is ever seen firing, lower max_image_size rather than
raising it.

GOMEMLIMIT is soft: the runtime collects harder to respect it and never fails
an allocation to honour it. With no cgroup cap the kernel picks its own OOM
victim, and the other large process on that host is the hold, so the scanner's
overshoot could kill the service it reports to.

CPUWeight and IOWeight, because no unit here set any scheduling directive: a
scan finishing a minute later costs nothing, a hold that cannot answer a pull
costs a user. Weights apply only under contention, so the scanner still uses
the whole box when nothing else wants it.

workers drops to 1. Proactive dispatch was depth-one hold-wide until this
release so the second worker never received anything, and now that it can, two
concurrent scans of a node:22-class image need 687 MiB against a host this
size. Raise it with MemoryMax and the host together.

Note the cgroup ceiling still wants rechecking: it was sized from measurements
taken with Grype disabled, and the vulnerability database's resident pages
count against MemoryMax even though GOMEMLIMIT cannot see them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF

deploy/upcloud/configs/scanner.yaml.tmpl

@@ -13,9 +13,27 @@ hold:
     url: "ws://localhost:8080"
     secret: "{{.ScannerSecret}}"
 scanner:
-    workers: 2
+    # One worker, deliberately. Two reasons, and the first one may go away:
+    #  - The hold's proactive dispatch loop gates on waitForCapacity() and
+    #    hands out one job at a time hold-wide, so a second worker only ever
+    #    receives work when two pushes coincide.
+    #  - Peak RSS is per concurrent scan. Two concurrent scans of a
+    #    node:22-class image measured 687 MiB with the 512 MiB GOMEMLIMIT in
+    #    force and 1357 MiB without it. On a host this size that is the
+    #    difference between working and OOM-killing the hold.
+    # Raise this only together with MemoryMax in the unit file and the memory
+    # available on the host.
+    workers: 1
     queue_size: 100
 vuln:
     enabled: true
     db_path: "{{.BasePath}}/scanner/vulndb"
+    # Must be on disk, not tmpfs. Layers are extracted here and amplify
+    # roughly 3.8x over their compressed size (measured: node:22, 389 MiB ->
+    # 1493 MiB), so a tmpfs would spend host memory to hold them.
     tmp_dir: "{{.BasePath}}/scanner/tmp"
+    # 512 MiB compressed. The shipped default is 2 GiB, which no small host
+    # can survive: peak RSS tracks image size, and 389 MiB compressed already
+    # reached 561 MiB RSS with the memory limit applied. Images above this are
+    # rejected before any blob is downloaded.
+    max_image_size: 536870912

deploy/upcloud/systemd/scanner.service.tmpl

@@ -11,6 +11,39 @@ ExecStart={{.BinaryPath}} serve --config {{.ConfigPath}}
 Restart=on-failure
 RestartSec=10
 
+# Memory containment. The scanner sets a 512 MiB Go soft limit itself
+# (GOMEMLIMIT, see cmd/scanner/main.go), but that is soft: the runtime
+# collects harder to respect it and never fails an allocation to honour it, so
+# a large enough image walks straight through it. Without a cgroup cap the
+# kernel OOM killer chooses its own victim, and on a shared host the other
+# large process is the hold, meaning the scanner's overshoot kills the service
+# it reports to. These make the scanner the one that dies.
+#
+# MemoryHigh throttles and reclaims; MemoryMax kills. Sized from measurement:
+# one scan of a node:22-class image peaks at 561 MiB RSS with the Go limit in
+# force, so MemoryHigh sits above that and MemoryMax leaves headroom for the
+# hold and the OS on a 1 GiB host. Raise both together with scanner.workers.
+MemoryHigh=640M
+MemoryMax=768M
+
+# Scheduling priority. The scanner is the lowest-value process on a shared
+# host: a scan finishing a minute later costs nothing, a hold that cannot
+# answer a pull costs a user. Syft's extraction saturates CPU and writes
+# several times the compressed image size to disk, so left at the default
+# weight it competes evenly with the services that matter.
+#
+# Weights apply only under contention: the scanner still uses the whole box
+# when nothing else wants it, which is what makes this better than the fixed
+# inter-job sleep in worker.go (that yields on a timer whether or not anyone
+# needs the CPU, and yields nothing while a scan is actually running).
+# Everything else runs at the default weight of 100.
+#
+# IOWeight needs the io controller with a scheduler that honours it (BFQ, or
+# io.cost configured); where it is unsupported systemd ignores it silently and
+# CPUWeight still applies.
+CPUWeight=20
+IOWeight=20
+
 ReadWritePaths={{.DataDir}}
 ProtectSystem=strict
 ProtectHome=yes