scanner: fix five crash and halt classes found by a pipeline audit

An audit of the scan pipeline and the hold side of scanning found several
ways scanning stops without saying so. Each fix here was written test-first:
a test expressing the wanted behaviour, confirmed failing for the right
reason, then the change.

A summary-less result crash-looped both processes. worker.go dereferenced
result.Summary unconditionally, but processJob only sets it when Grype runs,
and SendResult puts the nil on the wire before the scanner dies on it, so
handleResult's unguarded log killed the hold too. A nil Summary now means
"not scanned for vulnerabilities", deliberately distinct from "scanned, found
zero" — inventing a zeroed summary would report every image as clean when
Grype never ran. The hold writes a record rather than orphaning the uploaded
SBOM, and the appview renders an "SBOM only" state instead of a green Clean
badge.

The Grype database could wedge with no way back short of a restart. All three
throttles in loadVulnDatabase were guarded by vulnDB != nil, so a scanner
holding no provider retried a full download on every scan under the exclusive
lock. Two earlier attempts at this bug each added one more condition to the
same chain; this replaces the chain with a single decision function over a
state snapshot, consulted by both call sites so they cannot disagree. That
disagreement was itself a bug: the 50-scan reload had never once executed.

Two independent halts. An unparseable frame was dropped in silence, stranding
a row that held the hold's only dispatch slot forever; it is now answered
"skipped" on first delivery. The 10-minute sweep leaked the in-flight digest
and wrote no record, permanently retiring one image per timeout.

A digest went unvalidated into filepath.Join and os.Create, so a layer digest
of sha256:../../../x wrote outside the scan directory, and nothing verified
that downloaded bytes hashed to the digest naming them. Digests come from
records in a user's own PDS. Both are fixed together: verification is what
makes an escaping write self-defeating.

Concurrency did not work on either axis. The proactive capacity gate was
depth-one hold-wide, so neither extra workers nor extra scanner processes
received work. Depth is now the sum of the worker counts scanners advertise on
connect, the gate is scoped to proactive work, and dispatch prefers the
least-loaded scanner. Disconnects no longer hand a running scan to someone
else: a scanner keeps a stable per-process identity and reclaims its own rows
within a grace window, while a process that truly restarted returns with a new
identity and has its work reclaimed, which is correct because the restart did
lose it.

The hold's scanning deadline measured queueing rather than scanning, because
the scanner acks on receipt and handleAck never refreshed assigned_at. A new
"started" message, sent by the worker that dequeues the job, separates the two
budgets. An older scanner never sends it and falls under the queueing budget,
which is more forgiving than the deadline it gets today.

Adds an in-process mock hold and an e2e harness that runs the real client,
queue and worker pool, seeded with 84 real manifest records fetched from a
live PDS. Real image layouts and the Grype database are fetched by scripts and
gitignored; suites needing them skip cleanly, so the default run stays offline
and fast.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF

docs/HOLD_XRPC_ENDPOINTS.md

@@ -72,7 +72,7 @@ The `requireAuth` middleware validates Bearer service tokens only. `requestCrew`
 
 | Endpoint | Method | Description |
 |----------|--------|-------------|
-| `/xrpc/io.atcr.hold.subscribeScanJobs` | GET (WebSocket) | Scanner job subscription. Auth via `?secret=` query param or `X-Scanner-Secret` header (shared secret). Supports `?cursor=` for backfill. |
+| `/xrpc/io.atcr.hold.subscribeScanJobs` | GET (WebSocket) | Scanner job subscription. Auth via `?secret=` query param or `X-Scanner-Secret` header (shared secret). Supports `?cursor=` for backfill, `?workers=` to declare how many scans the process runs at once (default 1), and `?instance=` to declare a stable process identity so a reconnecting scanner resumes its own in-flight jobs. |
 
 ---
 
@@ -108,7 +108,7 @@ All require `blob:write` permission via service token:
 | `/xrpc/io.atcr.hold.purgeManifest` | POST | inline (service token or DPoP; captain, crew:admin, or manifest owner) | Purge layer/scan/image-config records for a single manifest URI. Called by appview on UI delete; called internally on takedown receipt. Does not delete S3 blobs (GC handles those). |
 | `/xrpc/io.atcr.hold.listTiers` | GET | none | List hold's available tiers with quotas and features (scanOnPush) |
 | `/xrpc/io.atcr.hold.updateCrewTier` | POST | appview token (ES256 JWT; 503 if appview DID not configured) | Update crew member's tier |
-| `/xrpc/io.atcr.hold.subscribeScanJobs` | GET (WebSocket) | shared secret (`?secret=` or `X-Scanner-Secret`) | Scanner job subscription; supports `?cursor=` for backfill |
+| `/xrpc/io.atcr.hold.subscribeScanJobs` | GET (WebSocket) | shared secret (`?secret=` or `X-Scanner-Secret`) | Scanner job subscription; supports `?cursor=` for backfill, `?workers=` for concurrency, `?instance=` for reconnect resumption |
 
 ---
 

docs/SBOM_SCANNING.md

@@ -104,7 +104,7 @@ a YAML file or pure env vars with the `SCANNER_` prefix. Run with
 |---------------------|------------------------------|----------------------------------|---------|
 | `hold.url`          | `SCANNER_HOLD_URL`           | — (**required**)                 | WebSocket URL of the hold, e.g. `ws://localhost:8080` or `wss://hold01.atcr.io`. `http(s)` is auto-converted to `ws(s)`. |
 | `hold.secret`       | `SCANNER_HOLD_SECRET`        | — (**required**)                 | Must match the hold's `scanner.secret`. Sent as `?secret=`. |
-| `scanner.workers`   | `SCANNER_SCANNER_WORKERS`    | `1`                              | Number of concurrent scan workers. |
+| `scanner.workers`   | `SCANNER_SCANNER_WORKERS`    | `1`                              | Number of concurrent scan workers. Declared to the hold on connect, which sizes the hold's dispatch budget for this process; raise it only alongside `vuln.max_image_size` and a cgroup memory cap. |
 | `scanner.queue_size`| `SCANNER_SCANNER_QUEUE_SIZE` | `100`                            | Max depth of the local priority queue. |
 | `vuln.enabled`      | `SCANNER_VULN_ENABLED`       | `true`                           | Run Grype after Syft. When false, only the SBOM is produced (no counts). |
 | `vuln.db_path`      | `SCANNER_VULN_DB_PATH`       | `/var/lib/atcr-scanner/vulndb`   | Directory for the Grype vulnerability database. |
@@ -143,11 +143,34 @@ scanned on push — it gets picked up later by the proactive discovery loop.
 ### 2. Dispatch
 
 The `ScanBroadcaster.Enqueue` inserts the job into the `scan_jobs` SQLite table
-(status `pending`) and immediately tries to dispatch it round-robin to one of the
-connected scanners. Jobs survive hold restarts. If no scanner is connected, the job
-waits; newly connected scanners drain pending jobs. Assigned-but-unacked jobs time out
-after 5 minutes and are re-dispatched; jobs stuck in `processing` for 10 minutes are
-marked failed (scanner likely crashed).
+(status `pending`) and immediately tries to dispatch it to a connected scanner. Jobs
+survive hold restarts. If no scanner is connected, the job waits.
+
+**Which scanner gets it.** Selection is by spare capacity, not position: each
+connection declares how many scans it runs at once (`?workers=`, default 1) and the
+hold prefers the scanner with the smallest fraction of its capacity committed, with
+ties resolved round-robin. A job that no connected scanner has room for stays
+`pending` rather than being pushed into a scanner's own queue, and is offered again
+the moment any scanner finishes something. Keeping the queue on the hold is what
+makes the job re-routable to whichever process frees up first, and what makes the
+deadlines below mean anything.
+
+**Deadlines.** Assigned-but-unacked jobs time out after 5 minutes and are
+re-dispatched. Once a job is acked the scanner has it, but it may be queued behind
+that scanner's workers, so there are two further budgets: 10 minutes from the
+`started` message a worker sends when it actually begins the scan, and 60 minutes
+from dispatch for a job that was acked but never reported as started (which is also
+what a scanner too old to send `started` gets). Both write a failed scan record and
+release the manifest for re-scanning.
+
+**Disconnects.** A dropped WebSocket does not return a scanner's in-flight jobs to
+the pool: its worker pool never learns the socket went away and keeps scanning, so
+handing that work to another process would have two scanners scanning the same image.
+The rows are marked instead. A scanner sends a stable per-process identity
+(`?instance=`) on every connect and resumes its own jobs on reconnect; a scanner that
+does not come back within 2 minutes has them reclaimed and re-offered. A scanner that
+actually restarted comes back with a new identity, so its old work is reclaimed
+rather than resumed — which is right, since a restart really did lose it.
 
 ### 3. Scan pipeline (scanner)
 
@@ -163,8 +186,16 @@ For each job (`scanner/internal/scan/worker.go`):
 5. **Grype** (if `vuln.enabled`) — scans the SBOM, producing the full JSON report and
    a severity summary (critical/high/medium/low/total).
 
-The scanner then sends one of three messages back over the WebSocket: `result`
-(SBOM + optional vuln report + summary), `error`, or `skipped` (with a reason).
+A worker sends `started` when it dequeues a job, before step 1. This is distinct
+from the `ack`, which the WebSocket reader sends the instant a job frame arrives:
+the gap between them is however long the job waits in this scanner's own queue, and
+the hold measures its scanning deadline from `started` so that queueing does not
+count against it.
+
+The scanner then sends one of three terminal messages back over the WebSocket:
+`result` (SBOM + optional vuln report + summary), `error`, or `skipped` (with a
+reason). All four messages are ignored by the hold unless the job is currently
+assigned to the scanner sending them.
 
 ### 4. Result storage (hold)
 
@@ -292,8 +323,18 @@ When `scanner.rescan_interval > 0`, the hold runs three background loops:
 - **Stale-scan loop**: walks the local scan records and re-queues any `ok`/`failed`
   record older than `rescan_interval`. Skipped records are left alone.
 - **Dispatch loop**: drains the unscanned queue (higher priority) before the stale
-  queue, throttled to one proactive job at a time so push-triggered scans aren't
-  starved.
+  queue, throttled to one proactive job per connected scanner worker — the sum of
+  every connected scanner's declared `workers`. The throttle counts only proactive
+  jobs: push-triggered scans bypass it entirely, so counting them meant a hold with
+  steady pushes never dispatched a proactive scan at all. With no scanner connected
+  the budget is zero and nothing is dispatched.
+
+Scaling this out is therefore a matter of running more scanner processes against the
+same hold, raising `scanner.workers`, or both: the dispatch budget, the drain on
+connect and the choice of scanner all follow the declared capacity. Do this only
+after bounding scanner memory — concurrency is what holds peak RSS down on a small
+host, and two concurrent scans of a `node:22`-class image measured 687 MiB with a
+512 MiB `GOMEMLIMIT` in force and 1357 MiB without.
 
 ## Accessing Results