scanner: fix five crash and halt classes found by a pipeline audit

An audit of the scan pipeline and the hold side of scanning found several
ways scanning stops without saying so. Each fix here was written test-first:
a test expressing the wanted behaviour, confirmed failing for the right
reason, then the change.

A summary-less result crash-looped both processes. worker.go dereferenced
result.Summary unconditionally, but processJob only sets it when Grype runs,
and SendResult puts the nil on the wire before the scanner dies on it, so
handleResult's unguarded log killed the hold too. A nil Summary now means
"not scanned for vulnerabilities", deliberately distinct from "scanned, found
zero" — inventing a zeroed summary would report every image as clean when
Grype never ran. The hold writes a record rather than orphaning the uploaded
SBOM, and the appview renders an "SBOM only" state instead of a green Clean
badge.

The Grype database could wedge with no way back short of a restart. All three
throttles in loadVulnDatabase were guarded by vulnDB != nil, so a scanner
holding no provider retried a full download on every scan under the exclusive
lock. Two earlier attempts at this bug each added one more condition to the
same chain; this replaces the chain with a single decision function over a
state snapshot, consulted by both call sites so they cannot disagree. That
disagreement was itself a bug: the 50-scan reload had never once executed.

Two independent halts. An unparseable frame was dropped in silence, stranding
a row that held the hold's only dispatch slot forever; it is now answered
"skipped" on first delivery. The 10-minute sweep leaked the in-flight digest
and wrote no record, permanently retiring one image per timeout.

A digest went unvalidated into filepath.Join and os.Create, so a layer digest
of sha256:../../../x wrote outside the scan directory, and nothing verified
that downloaded bytes hashed to the digest naming them. Digests come from
records in a user's own PDS. Both are fixed together: verification is what
makes an escaping write self-defeating.

Concurrency did not work on either axis. The proactive capacity gate was
depth-one hold-wide, so neither extra workers nor extra scanner processes
received work. Depth is now the sum of the worker counts scanners advertise on
connect, the gate is scoped to proactive work, and dispatch prefers the
least-loaded scanner. Disconnects no longer hand a running scan to someone
else: a scanner keeps a stable per-process identity and reclaims its own rows
within a grace window, while a process that truly restarted returns with a new
identity and has its work reclaimed, which is correct because the restart did
lose it.

The hold's scanning deadline measured queueing rather than scanning, because
the scanner acks on receipt and handleAck never refreshed assigned_at. A new
"started" message, sent by the worker that dequeues the job, separates the two
budgets. An older scanner never sends it and falls under the queueing budget,
which is more forgiving than the deadline it gets today.

Adds an in-process mock hold and an e2e harness that runs the real client,
queue and worker pool, seeded with 84 real manifest records fetched from a
live PDS. Real image layouts and the Grype database are fetched by scripts and
gitignored; suites needing them skip cleanly, so the default run stays offline
and fast.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF

docs/HOLD_XRPC_ENDPOINTS.md

@@ -72,7 +72,7 @@ The `requireAuth` middleware validates Bearer service tokens only. `requestCrew`
 
 | Endpoint | Method | Description |
 |----------|--------|-------------|
-| `/xrpc/io.atcr.hold.subscribeScanJobs` | GET (WebSocket) | Scanner job subscription. Auth via `?secret=` query param or `X-Scanner-Secret` header (shared secret). Supports `?cursor=` for backfill. |
+| `/xrpc/io.atcr.hold.subscribeScanJobs` | GET (WebSocket) | Scanner job subscription. Auth via `?secret=` query param or `X-Scanner-Secret` header (shared secret). Supports `?cursor=` for backfill, `?workers=` to declare how many scans the process runs at once (default 1), and `?instance=` to declare a stable process identity so a reconnecting scanner resumes its own in-flight jobs. |
 
 ---
 
@@ -108,7 +108,7 @@ All require `blob:write` permission via service token:
 | `/xrpc/io.atcr.hold.purgeManifest` | POST | inline (service token or DPoP; captain, crew:admin, or manifest owner) | Purge layer/scan/image-config records for a single manifest URI. Called by appview on UI delete; called internally on takedown receipt. Does not delete S3 blobs (GC handles those). |
 | `/xrpc/io.atcr.hold.listTiers` | GET | none | List hold's available tiers with quotas and features (scanOnPush) |
 | `/xrpc/io.atcr.hold.updateCrewTier` | POST | appview token (ES256 JWT; 503 if appview DID not configured) | Update crew member's tier |
-| `/xrpc/io.atcr.hold.subscribeScanJobs` | GET (WebSocket) | shared secret (`?secret=` or `X-Scanner-Secret`) | Scanner job subscription; supports `?cursor=` for backfill |
+| `/xrpc/io.atcr.hold.subscribeScanJobs` | GET (WebSocket) | shared secret (`?secret=` or `X-Scanner-Secret`) | Scanner job subscription; supports `?cursor=` for backfill, `?workers=` for concurrency, `?instance=` for reconnect resumption |
 
 ---
 

docs/SBOM_SCANNING.md

@@ -104,7 +104,7 @@ a YAML file or pure env vars with the `SCANNER_` prefix. Run with
 |---------------------|------------------------------|----------------------------------|---------|
 | `hold.url`          | `SCANNER_HOLD_URL`           | — (**required**)                 | WebSocket URL of the hold, e.g. `ws://localhost:8080` or `wss://hold01.atcr.io`. `http(s)` is auto-converted to `ws(s)`. |
 | `hold.secret`       | `SCANNER_HOLD_SECRET`        | — (**required**)                 | Must match the hold's `scanner.secret`. Sent as `?secret=`. |
-| `scanner.workers`   | `SCANNER_SCANNER_WORKERS`    | `1`                              | Number of concurrent scan workers. |
+| `scanner.workers`   | `SCANNER_SCANNER_WORKERS`    | `1`                              | Number of concurrent scan workers. Declared to the hold on connect, which sizes the hold's dispatch budget for this process; raise it only alongside `vuln.max_image_size` and a cgroup memory cap. |
 | `scanner.queue_size`| `SCANNER_SCANNER_QUEUE_SIZE` | `100`                            | Max depth of the local priority queue. |
 | `vuln.enabled`      | `SCANNER_VULN_ENABLED`       | `true`                           | Run Grype after Syft. When false, only the SBOM is produced (no counts). |
 | `vuln.db_path`      | `SCANNER_VULN_DB_PATH`       | `/var/lib/atcr-scanner/vulndb`   | Directory for the Grype vulnerability database. |
@@ -143,11 +143,34 @@ scanned on push — it gets picked up later by the proactive discovery loop.
 ### 2. Dispatch
 
 The `ScanBroadcaster.Enqueue` inserts the job into the `scan_jobs` SQLite table
-(status `pending`) and immediately tries to dispatch it round-robin to one of the
-connected scanners. Jobs survive hold restarts. If no scanner is connected, the job
-waits; newly connected scanners drain pending jobs. Assigned-but-unacked jobs time out
-after 5 minutes and are re-dispatched; jobs stuck in `processing` for 10 minutes are
-marked failed (scanner likely crashed).
+(status `pending`) and immediately tries to dispatch it to a connected scanner. Jobs
+survive hold restarts. If no scanner is connected, the job waits.
+
+**Which scanner gets it.** Selection is by spare capacity, not position: each
+connection declares how many scans it runs at once (`?workers=`, default 1) and the
+hold prefers the scanner with the smallest fraction of its capacity committed, with
+ties resolved round-robin. A job that no connected scanner has room for stays
+`pending` rather than being pushed into a scanner's own queue, and is offered again
+the moment any scanner finishes something. Keeping the queue on the hold is what
+makes the job re-routable to whichever process frees up first, and what makes the
+deadlines below mean anything.
+
+**Deadlines.** Assigned-but-unacked jobs time out after 5 minutes and are
+re-dispatched. Once a job is acked the scanner has it, but it may be queued behind
+that scanner's workers, so there are two further budgets: 10 minutes from the
+`started` message a worker sends when it actually begins the scan, and 60 minutes
+from dispatch for a job that was acked but never reported as started (which is also
+what a scanner too old to send `started` gets). Both write a failed scan record and
+release the manifest for re-scanning.
+
+**Disconnects.** A dropped WebSocket does not return a scanner's in-flight jobs to
+the pool: its worker pool never learns the socket went away and keeps scanning, so
+handing that work to another process would have two scanners scanning the same image.
+The rows are marked instead. A scanner sends a stable per-process identity
+(`?instance=`) on every connect and resumes its own jobs on reconnect; a scanner that
+does not come back within 2 minutes has them reclaimed and re-offered. A scanner that
+actually restarted comes back with a new identity, so its old work is reclaimed
+rather than resumed — which is right, since a restart really did lose it.
 
 ### 3. Scan pipeline (scanner)
 
@@ -163,8 +186,16 @@ For each job (`scanner/internal/scan/worker.go`):
 5. **Grype** (if `vuln.enabled`) — scans the SBOM, producing the full JSON report and
    a severity summary (critical/high/medium/low/total).
 
-The scanner then sends one of three messages back over the WebSocket: `result`
-(SBOM + optional vuln report + summary), `error`, or `skipped` (with a reason).
+A worker sends `started` when it dequeues a job, before step 1. This is distinct
+from the `ack`, which the WebSocket reader sends the instant a job frame arrives:
+the gap between them is however long the job waits in this scanner's own queue, and
+the hold measures its scanning deadline from `started` so that queueing does not
+count against it.
+
+The scanner then sends one of three terminal messages back over the WebSocket:
+`result` (SBOM + optional vuln report + summary), `error`, or `skipped` (with a
+reason). All four messages are ignored by the hold unless the job is currently
+assigned to the scanner sending them.
 
 ### 4. Result storage (hold)
 
@@ -292,8 +323,18 @@ When `scanner.rescan_interval > 0`, the hold runs three background loops:
 - **Stale-scan loop**: walks the local scan records and re-queues any `ok`/`failed`
   record older than `rescan_interval`. Skipped records are left alone.
 - **Dispatch loop**: drains the unscanned queue (higher priority) before the stale
-  queue, throttled to one proactive job at a time so push-triggered scans aren't
-  starved.
+  queue, throttled to one proactive job per connected scanner worker — the sum of
+  every connected scanner's declared `workers`. The throttle counts only proactive
+  jobs: push-triggered scans bypass it entirely, so counting them meant a hold with
+  steady pushes never dispatched a proactive scan at all. With no scanner connected
+  the budget is zero and nothing is dispatched.
+
+Scaling this out is therefore a matter of running more scanner processes against the
+same hold, raising `scanner.workers`, or both: the dispatch budget, the drain on
+connect and the choice of scanner all follow the declared capacity. Do this only
+after bounding scanner memory — concurrency is what holds peak RSS down on a small
+host, and two concurrent scans of a `node:22`-class image measured 687 MiB with a
+512 MiB `GOMEMLIMIT` in force and 1357 MiB without.
 
 ## Accessing Results
 

lexicons/io/atcr/hold/subscribeScanJobs.json

@@ -11,6 +11,17 @@
           "cursor": {
             "type": "integer",
             "description": "Sequence number to resume from. If omitted, starts from latest. Use -1 to receive only new jobs."
+          },
+          "workers": {
+            "type": "integer",
+            "minimum": 1,
+            "maximum": 32,
+            "description": "How many scans this scanner runs concurrently. The hold keeps this many jobs in flight for the connection. Omitted or unusable means one."
+          },
+          "instance": {
+            "type": "string",
+            "maxLength": 64,
+            "description": "Stable identity of the scanner process, sent on every connect. A scanner that reconnects with the same value resumes the jobs it was holding when the connection dropped, instead of having them offered to another scanner. Omitted means the hold assigns a per-connection identity and the scanner's in-flight work is reclaimed rather than resumed."
           }
         }
       },

pkg/appview/handlers/image_advisor.go

@@ -469,8 +469,15 @@ func generateAdvisorPrompt(w io.Writer, r *advisorReportData) {
 	// Vulnerability summary
 	if r.ScanRecord != nil {
 		sr := r.ScanRecord
-		fmt.Fprintf(w, "vulns: {critical: %d, high: %d, medium: %d, low: %d, total: %d}\n",
-			sr.Critical, sr.High, sr.Medium, sr.Low, sr.Total)
+		if vulnScanDidNotRun(sr) {
+			// Counts of zero here would be read as "no vulnerabilities", but
+			// this record was written by a scan that never ran a vulnerability
+			// database against the image.
+			fmt.Fprintf(w, "vulns: not scanned (SBOM only, no vulnerability data)\n")
+		} else {
+			fmt.Fprintf(w, "vulns: {critical: %d, high: %d, medium: %d, low: %d, total: %d}\n",
+				sr.Critical, sr.High, sr.Medium, sr.Low, sr.Total)
+		}
 	}
 
 	// Fixable critical/high vulns

pkg/appview/handlers/scan_result.go

@@ -25,50 +25,74 @@ type ScanResultHandler struct {
 }
 
 // vulnBadgeData is the template data for the vuln-badge partial.
-// The badge renders one of five states, in priority order:
-//  1. Error       — we couldn't reach the hold at all (network/5xx)
-//  2. NotScanned  — hold reachable, no scan record for this digest (404)
-//  3. Skipped     — scan record explicitly marks this artifact as not-scannable
-//  4. ScanFailed  — scan record exists but the scanner errored
-//  5. Found       — scan succeeded; render tier counts (or "Clean" when zero)
+// The badge renders one of six states, in priority order:
+//  1. Error           — we couldn't reach the hold at all (network/5xx)
+//  2. NotScanned      — hold reachable, no scan record for this digest (404)
+//  3. Skipped         — scan record explicitly marks this artifact as not-scannable
+//  4. ScanFailed      — scan record exists but the scanner errored
+//  5. VulnsNotScanned — scan succeeded but no vulnerability data was produced
+//  6. Found           — scan succeeded; render tier counts (or "Clean" when zero)
 //
 // These states must stay distinct so users can tell "hold is down" from
 // "this hasn't been scanned yet" from "scanner errored on this image" from
-// "this artifact type is intentionally not scanned".
+// "this artifact type is intentionally not scanned" from "we catalogued the
+// image but never matched it against a vulnerability database".
 type vulnBadgeData struct {
-	Critical     int64
-	High         int64
-	Medium       int64
-	Low          int64
-	Total        int64
-	ScannedAt    string
-	Found        bool   // true if scan record exists and succeeded
-	Error        bool   // true if hold unreachable (network/5xx)
-	NotScanned   bool   // true if hold is up but no scan record (404)
-	ScanFailed   bool   // true if scan record exists but scan failed
-	Skipped      bool   // true if scan record marks the artifact as intentionally not scanned (helm, in-toto, etc.)
-	Digest       string // for the detail modal link
-	HoldEndpoint string // for the detail modal link
+	Critical   int64
+	High       int64
+	Medium     int64
+	Low        int64
+	Total      int64
+	ScannedAt  string
+	Found      bool // true if scan record exists and succeeded
+	Error      bool // true if hold unreachable (network/5xx)
+	NotScanned bool // true if hold is up but no scan record (404)
+	ScanFailed bool // true if scan record exists but scan failed
+	Skipped    bool // true if scan record marks the artifact as intentionally not scanned (helm, in-toto, etc.)
+	// VulnsNotScanned means the scan produced an SBOM but no vulnerability
+	// data, which is what a scanner running with vulnerability scanning off
+	// reports. Its zero counts are the absence of a measurement, not a finding,
+	// so the badge must not render them as "Clean".
+	VulnsNotScanned bool
+	Digest          string // for the detail modal link
+	HoldEndpoint    string // for the detail modal link
+}
+
+// vulnScanDidNotRun reports whether a successful scan record carries no
+// vulnerability data at all: no report blob and no counts. That is what the
+// hold writes when the scanner ran with vulnerability scanning disabled, and
+// its zeros must never be presented as "no vulnerabilities found".
+//
+// It deliberately requires an explicit "ok" status. Records written before the
+// vulnReportBlob field existed carry counts and an SBOM but no status, and
+// those really were scanned, so the legacy shape stays out of this branch.
+func vulnScanDidNotRun(scanRecord *atproto.ScanRecord) bool {
+	return scanRecord.Status == atproto.ScanStatusOK &&
+		scanRecord.VulnReportBlob == nil &&
+		scanRecord.Total == 0
 }
 
 // classifyScanRecord maps a scan record's Status field to badge data flags.
 // An empty Status is treated as a legacy record from before the status field
 // existed: nil-blob + zero-counts = treat as failed (preserves the prior badge
 // for un-backfilled holds); otherwise treat as success.
-func classifyScanRecord(scanRecord *atproto.ScanRecord) (found, skipped, failed bool) {
+func classifyScanRecord(scanRecord *atproto.ScanRecord) (found, skipped, failed, vulnsNotScanned bool) {
 	switch scanRecord.Status {
 	case atproto.ScanStatusSkipped:
-		return false, true, false
+		return false, true, false, false
 	case atproto.ScanStatusFailed:
-		return false, false, true
+		return false, false, true, false
 	case atproto.ScanStatusOK:
-		return true, false, false
+		if vulnScanDidNotRun(scanRecord) {
+			return false, false, false, true
+		}
+		return true, false, false, false
 	default:
 		// Legacy record (status field didn't exist when this was written).
 		if scanRecord.SbomBlob == nil && scanRecord.Total == 0 {
-			return false, false, true
+			return false, false, true, false
 		}
-		return true, false, false
+		return true, false, false, false
 	}
 }
 
@@ -146,19 +170,21 @@ func (h *ScanResultHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
-	found, skipped, failed := classifyScanRecord(&scanRecord)
+	found, skipped, failed, vulnsNotScanned := classifyScanRecord(&scanRecord)
 	h.renderBadge(w, vulnBadgeData{
-		Critical:     scanRecord.Critical,
-		High:         scanRecord.High,
-		Medium:       scanRecord.Medium,
-		Low:          scanRecord.Low,
-		Total:        scanRecord.Total,
-		ScannedAt:    scanRecord.ScannedAt,
-		Found:        found,
-		Skipped:      skipped,
-		ScanFailed:   failed,
-		Digest:       digest,
-		HoldEndpoint: holdDID,
+		Critical:   scanRecord.Critical,
+		High:       scanRecord.High,
+		Medium:     scanRecord.Medium,
+		Low:        scanRecord.Low,
+		Total:      scanRecord.Total,
+		ScannedAt:  scanRecord.ScannedAt,
+		Found:      found,
+		Skipped:    skipped,
+		ScanFailed: failed,
+
+		VulnsNotScanned: vulnsNotScanned,
+		Digest:          digest,
+		HoldEndpoint:    holdDID,
 	})
 }
 
@@ -211,19 +237,21 @@ func fetchScanRecord(ctx context.Context, holdEndpoint, holdDID, hexDigest strin
 		return vulnBadgeData{Error: true}
 	}
 
-	found, skipped, failed := classifyScanRecord(&scanRecord)
+	found, skipped, failed, vulnsNotScanned := classifyScanRecord(&scanRecord)
 	return vulnBadgeData{
-		Critical:     scanRecord.Critical,
-		High:         scanRecord.High,
-		Medium:       scanRecord.Medium,
-		Low:          scanRecord.Low,
-		Total:        scanRecord.Total,
-		ScannedAt:    scanRecord.ScannedAt,
-		Found:        found,
-		Skipped:      skipped,
-		ScanFailed:   failed,
-		Digest:       fullDigest,
-		HoldEndpoint: holdDID,
+		Critical:   scanRecord.Critical,
+		High:       scanRecord.High,
+		Medium:     scanRecord.Medium,
+		Low:        scanRecord.Low,
+		Total:      scanRecord.Total,
+		ScannedAt:  scanRecord.ScannedAt,
+		Found:      found,
+		Skipped:    skipped,
+		ScanFailed: failed,
+
+		VulnsNotScanned: vulnsNotScanned,
+		Digest:          fullDigest,
+		HoldEndpoint:    holdDID,
 	}
 }
 

pkg/appview/handlers/scan_result_test.go

@@ -503,3 +503,95 @@ func TestBatchScanResult_SingleDigest(t *testing.T) {
 		t.Error("Expected critical count of 1")
 	}
 }
+
+// mockSBOMOnlyScanRecord is what the hold writes when the scanner finished a
+// scan with vulnerability scanning turned off: status "ok", an SBOM blob, no
+// vulnerability report, and zero counts because Grype never ran.
+//
+// The zero counts are not a finding. Rendering them as "Clean" would tell every
+// user of a vuln-disabled scanner that their images have no vulnerabilities,
+// which is the misreading this record shape has to avoid.
+func mockSBOMOnlyScanRecord() string {
+	record := map[string]any{
+		"$type":      "io.atcr.hold.scan",
+		"manifest":   "at://did:plc:test/io.atcr.manifest/abc123",
+		"repository": "myapp",
+		"userDid":    "did:plc:test",
+		"status":     "ok",
+		"critical":   0,
+		"high":       0,
+		"medium":     0,
+		"low":        0,
+		"total":      0,
+		"sbomBlob": map[string]any{
+			"$type":    "blob",
+			"ref":      map[string]any{"$link": "bafkreigv3xw47pk7cbeahkmttetf4smxyluwlu3jmteo2nzke2oa7dbhhm"},
+			"mimeType": "application/spdx+json",
+			"size":     1234,
+		},
+		"scannerVersion": "atcr-scanner-v1.0.0",
+		"scannedAt":      "2025-01-15T10:30:00Z",
+	}
+	envelope := map[string]any{
+		"uri":   "at://did:web:hold.example.com/io.atcr.hold.scan/abc123",
+		"cid":   "bafyreiabc123",
+		"value": record,
+	}
+	b, _ := json.Marshal(envelope)
+	return string(b)
+}
+
+func TestScanResult_SBOMWithoutVulnScanIsNotClean(t *testing.T) {
+	hold := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if handleMockDID(w, r) {
+			return
+		}
+		w.Header().Set("Content-Type", "application/json")
+		w.Write([]byte(mockSBOMOnlyScanRecord()))
+	}))
+	defer hold.Close()
+
+	handler := setupScanResultHandler(t, hold.URL)
+
+	req := httptest.NewRequest("GET", "/api/scan-result?digest=sha256:abc123&holdEndpoint="+hold.URL, nil)
+	rr := httptest.NewRecorder()
+	handler.ServeHTTP(rr, req)
+
+	body := rr.Body.String()
+
+	if strings.Contains(body, "Clean") || strings.Contains(body, "badge-success") {
+		t.Errorf("a record with no vulnerability data was rendered as clean: %s", body)
+	}
+	if strings.Contains(body, "vuln-strip") {
+		t.Errorf("a record with no vulnerability data rendered severity counts: %s", body)
+	}
+	if !strings.Contains(body, "SBOM only") {
+		t.Errorf("expected the SBOM-only badge, got: %s", body)
+	}
+}
+
+// TestScanResult_LegacyCleanRecordStillReadsClean guards the discriminator from
+// the other side. Records written before the vulnReportBlob field existed carry
+// an SBOM, zero counts and no status, and they really were scanned clean, so
+// the "no vulnerability data" rule must key on the explicit status "ok" and not
+// swallow them.
+func TestScanResult_LegacyCleanRecordStillReadsClean(t *testing.T) {
+	hold := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if handleMockDID(w, r) {
+			return
+		}
+		w.Header().Set("Content-Type", "application/json")
+		w.Write([]byte(mockScanRecord(0, 0, 0, 0, 0)))
+	}))
+	defer hold.Close()
+
+	handler := setupScanResultHandler(t, hold.URL)
+
+	req := httptest.NewRequest("GET", "/api/scan-result?digest=sha256:abc123&holdEndpoint="+hold.URL, nil)
+	rr := httptest.NewRecorder()
+	handler.ServeHTTP(rr, req)
+
+	if body := rr.Body.String(); !strings.Contains(body, "Clean") {
+		t.Errorf("legacy zero-count record no longer reads clean: %s", body)
+	}
+}

pkg/appview/handlers/vuln_details.go

@@ -60,12 +60,17 @@ type vulnDetailsData struct {
 	// NotScanned means the hold answered but holds no scan record for this
 	// manifest. Distinct from Error: nothing failed, the image was simply
 	// never scanned, and the UI must not present it as a failure.
-	NotScanned   bool
-	Status       string // scan record's status field (ok | failed | skipped); empty for legacy records
-	Reason       string // scan record's reason field (only meaningful when Status != ok)
-	ScannedAt    string
-	Digest       string // image digest (for download URLs)
-	HoldEndpoint string // hold DID (for download URLs)
+	NotScanned bool
+	// VulnsNotScanned means the scan succeeded and produced an SBOM, but no
+	// vulnerability data: the scanner ran with vulnerability scanning off. The
+	// zero counts are an absence of measurement, so the panel must say that
+	// rather than report zero findings or blame a failed fetch.
+	VulnsNotScanned bool
+	Status          string // scan record's status field (ok | failed | skipped); empty for legacy records
+	Reason          string // scan record's reason field (only meaningful when Status != ok)
+	ScannedAt       string
+	Digest          string // image digest (for download URLs)
+	HoldEndpoint    string // hold DID (for download URLs)
 }
 
 type vulnMatch struct {
@@ -171,6 +176,18 @@ func (h *VulnDetailsHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		Total:    scanRecord.Total,
 	}
 
+	// A successful record with no report blob and no counts is a scan that
+	// never ran Grype. Distinct from a missing report: there is nothing to
+	// fetch and nothing went wrong.
+	if vulnScanDidNotRun(&scanRecord) {
+		h.renderDetails(w, vulnDetailsData{
+			ScannedAt:       scanRecord.ScannedAt,
+			Status:          scanRecord.Status,
+			VulnsNotScanned: true,
+		})
+		return
+	}
+
 	// Step 2: Fetch the vulnerability report blob
 	if scanRecord.VulnReportBlob == nil || scanRecord.VulnReportBlob.Ref.String() == "" {
 		h.renderDetails(w, vulnDetailsData{
@@ -343,6 +360,17 @@ func FetchVulnDetails(ctx context.Context, holdEndpoint, digest string) vulnDeta
 		}
 	}
 
+	// A successful record with no report blob and no counts is a scan that
+	// never ran Grype. Distinct from a missing report: there is nothing to
+	// fetch and nothing went wrong.
+	if vulnScanDidNotRun(&scanRecord) {
+		return vulnDetailsData{
+			ScannedAt:       scanRecord.ScannedAt,
+			Status:          scanRecord.Status,
+			VulnsNotScanned: true,
+		}
+	}
+
 	// Fetch the vulnerability report blob
 	if scanRecord.VulnReportBlob == nil || scanRecord.VulnReportBlob.Ref.String() == "" {
 		return vulnDetailsData{

pkg/appview/handlers/vuln_details_test.go

@@ -356,3 +356,63 @@ func TestVulnDetails_MissingParams(t *testing.T) {
 		t.Error("Expected error message for missing parameters")
 	}
 }
+
+// mockSBOMOnlyRecordEnvelope is the detail-modal counterpart of the badge
+// fixture in scan_result_test.go: status "ok", an SBOM, no vulnerability report
+// and zero counts, which is what a scanner running with vuln.enabled=false
+// produces.
+func mockSBOMOnlyRecordEnvelope() string {
+	record := map[string]any{
+		"$type":          "io.atcr.hold.scan",
+		"manifest":       "at://did:plc:test/io.atcr.manifest/abc123",
+		"repository":     "myapp",
+		"userDid":        "did:plc:test",
+		"status":         "ok",
+		"critical":       0,
+		"high":           0,
+		"medium":         0,
+		"low":            0,
+		"total":          0,
+		"scannerVersion": "atcr-scanner-v1.0.0",
+		"scannedAt":      "2025-01-15T10:30:00Z",
+	}
+	envelope := map[string]any{
+		"uri":   "at://did:web:hold.example.com/io.atcr.hold.scan/abc123",
+		"cid":   "bafyreiabc123",
+		"value": record,
+	}
+	b, _ := json.Marshal(envelope)
+	return string(b)
+}
+
+// TestVulnDetails_VulnScanDidNotRun is the modal's half of the same rule: a
+// record with no vulnerability data must say so, not report zero findings and
+// not blame the hold for a fetch that never happened.
+func TestVulnDetails_VulnScanDidNotRun(t *testing.T) {
+	hold := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if handleMockDID(w, r) {
+			return
+		}
+		w.Header().Set("Content-Type", "application/json")
+		w.Write([]byte(mockSBOMOnlyRecordEnvelope()))
+	}))
+	defer hold.Close()
+
+	handler := setupVulnDetailsHandler(t)
+
+	req := httptest.NewRequest("GET", "/api/vuln-details?digest=sha256:abc123&holdEndpoint="+hold.URL, nil)
+	rr := httptest.NewRecorder()
+	handler.ServeHTTP(rr, req)
+
+	body := rr.Body.String()
+
+	if !strings.Contains(body, "Vulnerability scanning did not run") {
+		t.Errorf("expected copy saying vulnerability scanning did not run, got: %s", body)
+	}
+	if strings.Contains(body, "0 vulnerabilities") {
+		t.Errorf("a record with no vulnerability data claimed zero vulnerabilities: %s", body)
+	}
+	if strings.Contains(body, "No detailed vulnerability report") {
+		t.Errorf("a scan that never ran Grype was reported as a missing report: %s", body)
+	}
+}

pkg/appview/templates/partials/vuln-badge.html

@@ -14,6 +14,11 @@
 <span></span>
 {{ else if .ScanFailed }}
 <span class="badge badge-sm badge-warning" title="Scanner ran but produced no SBOM">{{ icon "alert-triangle" "size-3" }} Scan failed</span>
+{{ else if .VulnsNotScanned }}
+{{/* The image was catalogued but never matched against a vulnerability
+     database, so its zero counts mean "unmeasured", not "clean". Ghost, like
+     "Not scanned": this is an absence of data, not a good result. */}}
+<span class="badge badge-sm badge-ghost" title="An SBOM was generated, but this image was not checked for vulnerabilities">{{ icon "file-text" "size-3" }} SBOM only</span>
 {{ else if eq .Total 0 }}
 <span class="badge badge-sm badge-success" title="No vulnerabilities found (scanned {{ .ScannedAt }})">{{ icon "shield-check" "size-3" }} Clean</span>
 {{ else }}

pkg/appview/templates/partials/vuln-details.html

@@ -4,6 +4,12 @@
         <p class="font-medium text-base-content">No vulnerability scan available yet</p>
         <p class="mt-1">Scans run automatically shortly after a push. Check back in a few minutes, or push a new tag to trigger a scan.</p>
     </div>
+{{ else if .VulnsNotScanned }}
+    <div class="py-8 text-sm text-base-content/70 max-w-prose">
+        <p class="font-medium text-base-content">Vulnerability scanning did not run for this image</p>
+        <p class="mt-1">The scanner catalogued the image contents, so an SBOM is available, but it was never matched against a vulnerability database. No result here means no data, not a clean bill of health.</p>
+        {{ if .ScannedAt }}<p class="mt-2 text-xs text-base-content/60">Scanned: {{ .ScannedAt }}</p>{{ end }}
+    </div>
 {{ else if .Error }}
     {{ if gt .Summary.Total 0 }}
     <!-- Summary available but no detailed report -->

pkg/hold/pds/scan_broadcaster_concurrency_test.go

@@ -0,0 +1,962 @@
+package pds
+
+import (
+	"context"
+	"database/sql"
+	"testing"
+	"time"
+
+	"atcr.io/pkg/atproto"
+)
+
+// These tests describe a hold that can keep more than one scan running at a
+// time — across the workers of a single scanner process (vertical) and across
+// several scanner processes (horizontal).
+//
+// Both were defeated by the same thing: dispatchLoop called waitForCapacity()
+// before every dispatch, that gate blocked until no row anywhere was
+// 'assigned' or 'processing', and then exactly one candidate was dispatched.
+// Proactive scanning was therefore depth one hold-wide, so a second worker
+// could never be given proactive work and neither could a second scanner
+// process, even though dispatchJob already round-robins across subscribers.
+//
+// Two separate defects live in that one gate. It counted every row rather than
+// only the proactive ones, so a push-triggered scan starved proactive dispatch
+// and vice versa; and the depth was hardcoded at one rather than derived from
+// how much scanning capacity is actually connected.
+
+// newConcurrencyBroadcaster is newRecordingScanBroadcaster plus the channels
+// the dispatch gate selects on. The bare helpers leave stopCh nil, which makes
+// every select in waitForProactiveCapacity block forever.
+func newConcurrencyBroadcaster(t *testing.T) *ScanBroadcaster {
+	t.Helper()
+
+	sb := newRecordingScanBroadcaster(t)
+	sb.stopCh = make(chan struct{})
+	sb.completionSignal = make(chan struct{}, 1)
+	t.Cleanup(func() {
+		select {
+		case <-sb.stopCh:
+		default:
+			close(sb.stopCh)
+		}
+	})
+	return sb
+}
+
+// seedJob inserts one pending job with an explicit origin, which is what the
+// proactive capacity gate keys on.
+func seedJob(t *testing.T, sb *ScanBroadcaster, digest, origin string) int64 {
+	t.Helper()
+
+	res, err := sb.db.Exec(`
+		INSERT INTO scan_jobs
+			(manifest_digest, repository, tag, user_did, user_handle,
+			 hold_did, hold_endpoint, tier, config_json, layers_json, status, origin)
+		VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending', ?)
+	`, digest, "repo", "latest", "did:plc:user", "user.example.com",
+		sb.holdDID, sb.holdEndpoint, "deckhand", "{}", "[]", origin)
+	if err != nil {
+		t.Fatalf("seed %s job: %v", origin, err)
+	}
+	seq, err := res.LastInsertId()
+	if err != nil {
+		t.Fatalf("seq: %v", err)
+	}
+	return seq
+}
+
+func setStatus(t *testing.T, sb *ScanBroadcaster, seq int64, status string) {
+	t.Helper()
+
+	if _, err := sb.db.Exec(`UPDATE scan_jobs SET status = ? WHERE seq = ?`, status, seq); err != nil {
+		t.Fatalf("set status %q on %d: %v", status, seq, err)
+	}
+}
+
+func assignedTo(t *testing.T, sb *ScanBroadcaster, seq int64) string {
+	t.Helper()
+
+	var to sql.NullString
+	if err := sb.db.QueryRow(`SELECT assigned_to FROM scan_jobs WHERE seq = ?`, seq).Scan(&to); err != nil {
+		t.Fatalf("query assigned_to for %d: %v", seq, err)
+	}
+	return to.String
+}
+
+// markStarted sets started_at the given number of minutes in the past, as a
+// 'started' message from a worker would have done at that time.
+func markStarted(t *testing.T, sb *ScanBroadcaster, seq int64, agoMinutes int) {
+	t.Helper()
+
+	at := time.Now().Add(-time.Duration(agoMinutes) * time.Minute)
+	if _, err := sb.db.Exec(`UPDATE scan_jobs SET started_at = ? WHERE seq = ?`, at, seq); err != nil {
+		t.Fatalf("mark started %d: %v", seq, err)
+	}
+}
+
+// jobIsDisconnected reports whether the row is marked as belonging to a
+// scanner that has dropped its connection but may still be running it.
+func jobIsDisconnected(t *testing.T, sb *ScanBroadcaster, seq int64) bool {
+	t.Helper()
+
+	var at sql.NullTime
+	if err := sb.db.QueryRow(`SELECT disconnected_at FROM scan_jobs WHERE seq = ?`, seq).Scan(&at); err != nil {
+		t.Fatalf("query disconnected_at for %d: %v", seq, err)
+	}
+	return at.Valid
+}
+
+func startedAt(t *testing.T, sb *ScanBroadcaster, seq int64) sql.NullTime {
+	t.Helper()
+
+	var at sql.NullTime
+	if err := sb.db.QueryRow(`SELECT started_at FROM scan_jobs WHERE seq = ?`, seq).Scan(&at); err != nil {
+		t.Fatalf("query started_at for %d: %v", seq, err)
+	}
+	return at
+}
+
+// ---------------------------------------------------------------------------
+// Dispatch depth
+// ---------------------------------------------------------------------------
+
+// TestScanProactiveDispatchLimit_TracksConnectedScannerWorkers pins what the
+// proactive dispatch depth is derived from: the scanning capacity actually
+// connected, summed over subscribers.
+//
+// A scanner declares its worker count when it subscribes. One that declares
+// nothing — every scanner built before the parameter existed — counts as one
+// worker, which is exactly the depth the hold had before, so an old scanner
+// against a new hold behaves as it always did.
+func TestScanProactiveDispatchLimit_TracksConnectedScannerWorkers(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+
+	if got := sb.proactiveDispatchLimit(); got != 0 {
+		t.Errorf("limit with no scanner connected = %d, want 0: there is "+
+			"nothing to dispatch to", got)
+	}
+
+	old := newTestScanSubscriber(t, sb, 4)
+	old.id = "legacy"
+	if got := sb.proactiveDispatchLimit(); got != 1 {
+		t.Errorf("limit for a scanner that declares no worker count = %d, want 1", got)
+	}
+
+	old.capacity = 2 // the same process reconnecting with workers: 2
+	if got := sb.proactiveDispatchLimit(); got != 2 {
+		t.Errorf("limit for one two-worker scanner = %d, want 2: workers above "+
+			"one must buy proactive throughput", got)
+	}
+
+	second := newTestScanSubscriber(t, sb, 4)
+	second.id = "second-process"
+	second.capacity = 3
+	if got := sb.proactiveDispatchLimit(); got != 5 {
+		t.Errorf("limit across two scanner processes = %d, want 5", got)
+	}
+
+	sb.Unsubscribe(second)
+	if got := sb.proactiveDispatchLimit(); got != 2 {
+		t.Errorf("limit after a scanner disconnected = %d, want 2", got)
+	}
+}
+
+// TestScanProactiveCapacity_AllowsOneJobPerWorker is the vertical scaling case:
+// `scanner.workers: 2` must genuinely produce two concurrent proactive scans.
+//
+// Depth is one proactive job per connected worker. Not more: the scanner acks
+// on receipt and queues internally, so anything beyond one per worker is a
+// backlog the hold cannot see into, which is what made the ten-minute deadline
+// fire under a healthy scanner. Not fewer: one worker would sit idle.
+func TestScanProactiveCapacity_AllowsOneJobPerWorker(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+	sub.capacity = 2
+
+	first := seedJob(t, sb, "sha256:first", originProactive)
+	sb.dispatchJob(&ScanJobEvent{Type: "job", Seq: first, Repository: "repo"})
+
+	if n, ok := sb.activeProactiveJobs(); !ok || n != 1 {
+		t.Fatalf("active proactive jobs = %d (ok=%v), want 1", n, ok)
+	}
+	if !sb.hasProactiveCapacity() {
+		t.Fatal("a two-worker scanner with one job running has capacity for a " +
+			"second; depth-one dispatch is what defeats scanner.workers")
+	}
+
+	second := seedJob(t, sb, "sha256:second", originProactive)
+	sb.dispatchJob(&ScanJobEvent{Type: "job", Seq: second, Repository: "repo"})
+
+	if n, _ := sb.activeProactiveJobs(); n != 2 {
+		t.Fatalf("active proactive jobs = %d, want 2", n)
+	}
+	if sb.hasProactiveCapacity() {
+		t.Error("both workers are busy; the hold must stop dispatching rather " +
+			"than pile a backlog into the scanner's own queue")
+	}
+
+	// Both jobs really are out with the scanner, not merely counted.
+	if len(sub.send) != 2 {
+		t.Errorf("scanner received %d jobs, want 2", len(sub.send))
+	}
+	if got := assignedTo(t, sb, first); got != sub.id {
+		t.Errorf("job %d assigned_to = %q, want %q", first, got, sub.id)
+	}
+	if got := assignedTo(t, sb, second); got != sub.id {
+		t.Errorf("job %d assigned_to = %q, want %q", second, got, sub.id)
+	}
+}
+
+// TestScanProactiveCapacity_ScalesAcrossScannerProcesses is the horizontal
+// case. Two single-worker scanners are two units of capacity, and dispatchJob
+// already round-robins, so the two jobs must land on different scanners.
+func TestScanProactiveCapacity_ScalesAcrossScannerProcesses(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	a := newTestScanSubscriber(t, sb, 4)
+	a.id = "scanner-a"
+	b := newTestScanSubscriber(t, sb, 4)
+	b.id = "scanner-b"
+
+	first := seedJob(t, sb, "sha256:first", originProactive)
+	sb.dispatchJob(&ScanJobEvent{Type: "job", Seq: first, Repository: "repo"})
+
+	if !sb.hasProactiveCapacity() {
+		t.Fatal("a second scanner process is a second unit of capacity and " +
+			"could never receive proactive work under a depth-one gate")
+	}
+
+	second := seedJob(t, sb, "sha256:second", originProactive)
+	sb.dispatchJob(&ScanJobEvent{Type: "job", Seq: second, Repository: "repo"})
+
+	if len(a.send) != 1 || len(b.send) != 1 {
+		t.Errorf("jobs per scanner = a:%d b:%d, want 1 each", len(a.send), len(b.send))
+	}
+	if sb.hasProactiveCapacity() {
+		t.Error("both scanners are busy; dispatch must stop")
+	}
+}
+
+// TestScanProactiveCapacity_IgnoresPushTriggeredJobs covers the second defect
+// in the gate. Push-triggered scans bypass it entirely — oci/xrpc.go calls
+// Enqueue directly — but they were counted by it, so on a hold with steady
+// pushes "one proactive job at a time" was really "none while anyone pushes".
+func TestScanProactiveCapacity_IgnoresPushTriggeredJobs(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	newTestScanSubscriber(t, sb, 4) // one worker: depth one
+
+	push := seedJob(t, sb, "sha256:pushed", originPush)
+	setStatus(t, sb, push, "processing")
+
+	if n, ok := sb.activeProactiveJobs(); !ok || n != 0 {
+		t.Fatalf("active proactive jobs = %d (ok=%v), want 0: a push-triggered "+
+			"scan is not proactive work", n, ok)
+	}
+	if !sb.hasProactiveCapacity() {
+		t.Error("a push-triggered scan is throttling proactive dispatch; the " +
+			"two paths must not consume each other's budget")
+	}
+
+	// And the mirror: a proactive job in flight fills the proactive budget.
+	proactive := seedJob(t, sb, "sha256:proactive", originProactive)
+	setStatus(t, sb, proactive, "assigned")
+
+	if n, _ := sb.activeProactiveJobs(); n != 1 {
+		t.Errorf("active proactive jobs = %d, want 1", n)
+	}
+	if sb.hasProactiveCapacity() {
+		t.Error("the one worker is busy with a proactive job already")
+	}
+}
+
+// TestScanWaitForProactiveCapacity_BlocksAtTheLimitAndReleasesOnCompletion
+// checks the gate the dispatch loop actually calls: it must park while the
+// budget is full, and wake on the completion signal rather than on a timer.
+func TestScanWaitForProactiveCapacity_BlocksAtTheLimitAndReleasesOnCompletion(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	newTestScanSubscriber(t, sb, 4) // one worker
+
+	seq := seedJob(t, sb, "sha256:running", originProactive)
+	setStatus(t, sb, seq, "processing")
+
+	got := make(chan bool, 1)
+	go func() { got <- sb.waitForProactiveCapacity() }()
+
+	select {
+	case <-got:
+		t.Fatal("the gate returned while the only worker was busy")
+	case <-time.After(250 * time.Millisecond):
+	}
+
+	setStatus(t, sb, seq, "completed")
+	sb.signalCompletion()
+
+	select {
+	case ok := <-got:
+		if !ok {
+			t.Error("the gate reported no capacity after the job completed")
+		}
+	case <-time.After(5 * time.Second):
+		t.Fatal("the gate did not wake on the completion signal")
+	}
+}
+
+// TestScanWaitForProactiveCapacity_YieldsWhenNoScannerIsConnected keeps the
+// gate safe at zero capacity. Blocking inside it forever would be correct-ish
+// today but leaves the dispatch loop unable to notice a scanner arriving, and
+// unable to re-check anything else; it returns instead so the loop can wait on
+// the connection.
+func TestScanWaitForProactiveCapacity_YieldsWhenNoScannerIsConnected(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+
+	done := make(chan bool, 1)
+	go func() { done <- sb.waitForProactiveCapacity() }()
+
+	select {
+	case ok := <-done:
+		if ok {
+			t.Error("the gate granted capacity with no scanner connected")
+		}
+	case <-time.After(5 * time.Second):
+		t.Fatal("the gate blocked with no scanner connected; the dispatch loop " +
+			"cannot re-check the subscriber list from in there")
+	}
+}
+
+// TestScanWaitForProactiveCapacity_YieldsWhenTheLastScannerDisconnects is the
+// mid-flight version: capacity that vanishes while the gate is parked must
+// release it, not strand the loop.
+func TestScanWaitForProactiveCapacity_YieldsWhenTheLastScannerDisconnects(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	seq := seedJob(t, sb, "sha256:running", originProactive)
+	setStatus(t, sb, seq, "processing")
+
+	got := make(chan bool, 1)
+	go func() { got <- sb.waitForProactiveCapacity() }()
+
+	select {
+	case <-got:
+		t.Fatal("the gate returned while the only worker was busy")
+	case <-time.After(250 * time.Millisecond):
+	}
+
+	sb.Unsubscribe(sub)
+
+	select {
+	case ok := <-got:
+		if ok {
+			t.Error("the gate granted capacity after the last scanner left")
+		}
+	case <-time.After(10 * time.Second):
+		t.Fatal("the gate stayed parked after the last scanner disconnected")
+	}
+}
+
+// ---------------------------------------------------------------------------
+// F4: the deadline must measure scanning, not queueing
+// ---------------------------------------------------------------------------
+
+// TestScanStarted_StartsTheScanningClock covers the new signal.
+//
+// The ack means "I have it": the scanner sends it off the WebSocket reader the
+// moment a frame arrives, before the job is even queued. The job then waits in
+// the scanner's own 100-deep queue behind its workers. Measuring the scanning
+// deadline from dispatch therefore budgets queueing, and a 100-deep queue of
+// no-op jobs crosses ten minutes at position 59.
+//
+// 'started' is sent by the worker that dequeues the job, so it marks the one
+// moment the hold could not otherwise observe.
+func TestScanStarted_StartsTheScanningClock(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	seq := seedJob(t, sb, "sha256:queued", originProactive)
+	assignJob(t, sb, seq, sub, 0)
+	sb.handleAck(sub, seq)
+
+	if at := startedAt(t, sb, seq); at.Valid {
+		t.Fatal("the ack started the scanning clock; it means \"I have it\", " +
+			"not \"a worker is on it\"")
+	}
+
+	sb.handleStarted(sub, seq)
+
+	at := startedAt(t, sb, seq)
+	if !at.Valid {
+		t.Fatal("'started' did not stamp started_at, so the deadline still " +
+			"measures queueing rather than scanning")
+	}
+	if d := time.Since(at.Time); d > time.Minute || d < -time.Minute {
+		t.Errorf("started_at is %s away from now, want ~0", d)
+	}
+	if got := jobStatus(t, sb, seq); got != "processing" {
+		t.Errorf("status = %q, want processing", got)
+	}
+}
+
+// TestScanStarted_KeepsAQueuedJobInsideItsScanningDeadline is the behaviour
+// the whole change is for: a job that sat in a scanner's queue for longer than
+// the scanning deadline, and has only just started, must not be cancelled out
+// from under the worker now running it.
+func TestScanStarted_KeepsAQueuedJobInsideItsScanningDeadline(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	const digest = "sha256:longqueue"
+	seq := seedJob(t, sb, digest, originProactive)
+	assignJob(t, sb, seq, sub, 20) // dispatched twenty minutes ago
+	sb.handleAck(sub, seq)
+	sb.addInflight(digest)
+
+	sb.handleStarted(sub, seq) // a worker picked it up just now
+
+	sb.reDispatchTimedOut()
+
+	if got := jobStatus(t, sb, seq); got != "processing" {
+		t.Fatalf("status = %q, want processing: a scan that started seconds "+
+			"ago was cancelled because it had queued for twenty minutes", got)
+	}
+	if _, _, err := sb.pds.GetScanRecord(context.Background(), digest); err == nil {
+		t.Error("a failure record was written for a scan that is actively running")
+	}
+}
+
+// TestScanProcessingTimeout_FailsAJobStuckSinceItStarted keeps the deadline
+// real. Once a worker has said it started, the scanning budget applies from
+// that moment and a wedged scan is still retired.
+func TestScanProcessingTimeout_FailsAJobStuckSinceItStarted(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	const digest = "sha256:wedgedscan"
+	seq := seedJob(t, sb, digest, originProactive)
+	assignJob(t, sb, seq, sub, 30)
+	sb.handleAck(sub, seq)
+	sb.addInflight(digest)
+	markStarted(t, sb, seq, 11) // started eleven minutes ago, still nothing
+
+	sb.reDispatchTimedOut()
+
+	if got := jobStatus(t, sb, seq); got != "failed" {
+		t.Fatalf("status = %q, want failed", got)
+	}
+	_, record, err := sb.pds.GetScanRecord(context.Background(), digest)
+	if err != nil {
+		t.Fatalf("no scan record for a job the hold gave up on: %v", err)
+	}
+	if record.Status != atproto.ScanStatusFailed {
+		t.Errorf("record status = %q, want %q", record.Status, atproto.ScanStatusFailed)
+	}
+	if !sb.addInflight(digest) {
+		t.Error("the timed-out digest is still in flight")
+	}
+}
+
+// TestScanProcessingTimeout_ToleratesAScannerThatNeverReportsStarts is the
+// compatibility half. A scanner built before 'started' existed never sends it,
+// so started_at stays NULL and the hold can only observe dispatch. The budget
+// for that case is the one the hold can actually justify: long enough for a
+// full scanner queue to drain, so a healthy backlogged scanner is not killed,
+// and still bounded so a wedged-but-connected scanner does not hold capacity
+// for the life of the process.
+func TestScanProcessingTimeout_ToleratesAScannerThatNeverReportsStarts(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	const digest = "sha256:oldscanner"
+	seq := seedJob(t, sb, digest, originProactive)
+	assignJob(t, sb, seq, sub, 15) // well past the ten-minute scanning deadline
+	sb.handleAck(sub, seq)
+	sb.addInflight(digest)
+
+	sb.reDispatchTimedOut()
+
+	if got := jobStatus(t, sb, seq); got != "processing" {
+		t.Fatalf("status = %q, want processing: an acked job queued fifteen "+
+			"minutes behind a busy scanner is not evidence of anything wrong", got)
+	}
+
+	// Past the queue budget it is retired like any other stuck job.
+	if _, err := sb.db.Exec(
+		`UPDATE scan_jobs SET assigned_at = ? WHERE seq = ?`,
+		time.Now().Add(-queuedTimeout-time.Minute), seq,
+	); err != nil {
+		t.Fatalf("age the row: %v", err)
+	}
+
+	sb.reDispatchTimedOut()
+
+	if got := jobStatus(t, sb, seq); got != "failed" {
+		t.Errorf("status = %q, want failed: an unstarted job must still have a "+
+			"bound, or a connected-but-wedged scanner holds capacity forever", got)
+	}
+}
+
+// ---------------------------------------------------------------------------
+// Ownership of terminal transitions
+// ---------------------------------------------------------------------------
+
+// TestScanTerminalHandlers_IgnoreAnotherScannersJob closes the hole that opens
+// the moment horizontal scaling works. handleAck guards on assigned_to; the
+// handlers that write records and retire the row did not, so any scanner could
+// complete, fail or skip a job belonging to another one — writing a scan
+// record for an image it never looked at and releasing a digest a different
+// scanner is still working on.
+func TestScanTerminalHandlers_IgnoreAnotherScannersJob(t *testing.T) {
+	cases := []struct {
+		name string
+		send func(sb *ScanBroadcaster, sub *ScanSubscriber, seq int64)
+	}{
+		{"result", func(sb *ScanBroadcaster, sub *ScanSubscriber, seq int64) {
+			sb.handleResult(sub, ScannerMessage{Type: "result", Seq: seq, SBOM: testSBOM})
+		}},
+		{"error", func(sb *ScanBroadcaster, sub *ScanSubscriber, seq int64) {
+			sb.handleError(sub, ScannerMessage{Type: "error", Seq: seq, Error: "boom"})
+		}},
+		{"skipped", func(sb *ScanBroadcaster, sub *ScanSubscriber, seq int64) {
+			sb.handleSkipped(sub, ScannerMessage{Type: "skipped", Seq: seq, Reason: "nope"})
+		}},
+		{"started", func(sb *ScanBroadcaster, sub *ScanSubscriber, seq int64) {
+			sb.handleStarted(sub, seq)
+		}},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			sb := newConcurrencyBroadcaster(t)
+			owner := newTestScanSubscriber(t, sb, 4)
+			owner.id = "owner"
+			intruder := newTestScanSubscriber(t, sb, 4)
+			intruder.id = "intruder"
+
+			digest := "sha256:owned" + tc.name
+			seq := seedJob(t, sb, digest, originProactive)
+			assignJob(t, sb, seq, owner, 0)
+			sb.handleAck(owner, seq)
+			sb.addInflight(digest)
+
+			tc.send(sb, intruder, seq)
+
+			if got := jobStatus(t, sb, seq); got != "processing" {
+				t.Errorf("status = %q, want processing: a scanner that does not "+
+					"own the job retired it", got)
+			}
+			if sb.addInflight(digest) {
+				t.Error("another scanner's message released the in-flight digest")
+			}
+			if _, _, err := sb.pds.GetScanRecord(context.Background(), digest); err == nil {
+				t.Error("a scan record was written by a scanner that never had the job")
+			}
+			if tc.name == "started" {
+				if startedAt(t, sb, seq).Valid {
+					t.Error("another scanner restarted the scanning clock")
+				}
+			}
+		})
+	}
+}
+
+// TestScanTerminalHandlers_AcceptTheOwningScanner is the other half: the guard
+// must not reject the scanner that legitimately holds the job.
+func TestScanTerminalHandlers_AcceptTheOwningScanner(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	owner := newTestScanSubscriber(t, sb, 4)
+	owner.id = "owner"
+
+	const digest = "sha256:legitimate"
+	seq := seedJob(t, sb, digest, originProactive)
+	assignJob(t, sb, seq, owner, 0)
+	sb.handleAck(owner, seq)
+	sb.handleStarted(owner, seq)
+	sb.addInflight(digest)
+
+	sb.handleResult(owner, ScannerMessage{Type: "result", Seq: seq, SBOM: testSBOM})
+
+	if got := jobStatus(t, sb, seq); got != "completed" {
+		t.Fatalf("status = %q, want completed", got)
+	}
+	if !sb.addInflight(digest) {
+		t.Error("the owning scanner's result did not release the digest")
+	}
+}
+
+// ---------------------------------------------------------------------------
+// The buffer-full reset
+// ---------------------------------------------------------------------------
+
+// TestScanDispatchJob_BufferFullResetLeavesAnotherScannersClaimAlone is the
+// mirror of the double-dispatch window closed in drainPendingJobs.
+//
+// dispatchJob assigns the row, finds the scanner's send buffer full, and puts
+// the row back to 'pending' — with no assigned_to or status guard. If another
+// dispatcher claimed the row in between, that reset hands a job a second
+// scanner is already holding back to the pool, and a third scanner gets it.
+//
+// The interleaving is built with a SQLite trigger rather than a race: the
+// trigger fires on dispatchJob's own assign UPDATE and reassigns the row to
+// another scanner, so by the time the buffer-full branch runs, the row is
+// provably not ours. SQLite does not recurse triggers by default, so the reset
+// UPDATE does not re-fire it.
+func TestScanDispatchJob_BufferFullResetLeavesAnotherScannersClaimAlone(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 0) // unbuffered, nobody receiving: always full
+	sub.id = "loser"
+
+	seq := seedJob(t, sb, "sha256:contended", originProactive)
+
+	if _, err := sb.db.Exec(`
+		CREATE TRIGGER steal_assignment AFTER UPDATE OF status ON scan_jobs
+		WHEN NEW.status = 'assigned' AND NEW.assigned_to = 'loser'
+		BEGIN
+			UPDATE scan_jobs SET assigned_to = 'winner' WHERE seq = NEW.seq;
+		END
+	`); err != nil {
+		t.Fatalf("create trigger: %v", err)
+	}
+
+	sb.dispatchJob(&ScanJobEvent{Type: "job", Seq: seq, Repository: "repo"})
+
+	if got := jobStatus(t, sb, seq); got != "assigned" {
+		t.Errorf("status = %q, want assigned: the buffer-full reset returned a "+
+			"row another dispatcher holds to the pool, so a third scanner will "+
+			"be handed a job that is already out", got)
+	}
+	if got := assignedTo(t, sb, seq); got != "winner" {
+		t.Errorf("assigned_to = %q, want \"winner\": the claim was stolen back", got)
+	}
+}
+
+// TestScanDispatchJob_BufferFullReturnsOurOwnRowToPending is the guard's other
+// side: when the row really is ours, a full buffer must still release it so
+// the re-dispatch loop can offer it again rather than leaving it assigned to a
+// scanner that was never sent it.
+func TestScanDispatchJob_BufferFullReturnsOurOwnRowToPending(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 0) // unbuffered, nobody receiving
+	sub.id = "solo"
+
+	seq := seedJob(t, sb, "sha256:nobuffer", originProactive)
+	sb.dispatchJob(&ScanJobEvent{Type: "job", Seq: seq, Repository: "repo"})
+
+	if got := jobStatus(t, sb, seq); got != "pending" {
+		t.Errorf("status = %q, want pending", got)
+	}
+	if got := assignedTo(t, sb, seq); got != "" {
+		t.Errorf("assigned_to = %q, want empty", got)
+	}
+}
+
+// ---------------------------------------------------------------------------
+// Fairness and admission control across N scanner processes
+// ---------------------------------------------------------------------------
+
+// TestScanDispatch_PrefersTheLeastLoadedScanner covers plain round-robin's
+// blind spot. nextIdx hands the next job to whichever subscriber is next in
+// the slice, saturated or idle, which with heterogeneous processes (different
+// worker counts, different hosts, one mid-scan and one just connected) piles
+// work onto a scanner that cannot take it while another sits idle.
+func TestScanDispatch_PrefersTheLeastLoadedScanner(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	busy := newTestScanSubscriber(t, sb, 4)
+	busy.id = "busy"
+	busy.capacity = 2
+	idle := newTestScanSubscriber(t, sb, 4)
+	idle.id = "idle"
+	idle.capacity = 2
+
+	// "busy" is already running one scan.
+	running := seedJob(t, sb, "sha256:running", originPush)
+	if _, err := sb.db.Exec(
+		`UPDATE scan_jobs SET status='processing', assigned_to=?, assigned_at=? WHERE seq=?`,
+		busy.id, time.Now(), running,
+	); err != nil {
+		t.Fatalf("prime load: %v", err)
+	}
+
+	seq := seedJob(t, sb, "sha256:next", originProactive)
+	sb.dispatchJob(&ScanJobEvent{Type: "job", Seq: seq, Repository: "repo"})
+
+	if got := assignedTo(t, sb, seq); got != idle.id {
+		t.Errorf("job went to %q, want %q: dispatch must weigh what each "+
+			"scanner is already holding", got, idle.id)
+	}
+	if len(idle.send) != 1 || len(busy.send) != 0 {
+		t.Errorf("sends: idle=%d busy=%d, want 1/0", len(idle.send), len(busy.send))
+	}
+}
+
+// TestScanDispatch_LeavesTheRowPendingWhenEveryScannerIsSaturated is the
+// admission control the deadline depends on.
+//
+// The hold cannot see into a scanner's own queue: the scanner acks on receipt
+// and the job then waits behind its workers, which is exactly why a deadline
+// measured from dispatch cancels healthy work. Keeping the queue on the hold
+// instead of pushing it into the scanner keeps the row observable, keeps it
+// available to whichever process frees up first, and bounds what any one
+// process is holding.
+func TestScanDispatch_LeavesTheRowPendingWhenEveryScannerIsSaturated(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 20)
+	sub.capacity = 1
+
+	first := seedJob(t, sb, "sha256:one", originPush)
+	sb.dispatchJob(&ScanJobEvent{Type: "job", Seq: first, Repository: "repo"})
+	if got := jobStatus(t, sb, first); got != "assigned" {
+		t.Fatalf("first job status = %q, want assigned", got)
+	}
+
+	second := seedJob(t, sb, "sha256:two", originPush)
+	sb.dispatchJob(&ScanJobEvent{Type: "job", Seq: second, Repository: "repo"})
+
+	if got := jobStatus(t, sb, second); got != "pending" {
+		t.Errorf("second job status = %q, want pending: the only worker is "+
+			"busy, so the row belongs on the hold's queue, not the scanner's", got)
+	}
+	if len(sub.send) != 1 {
+		t.Errorf("scanner received %d jobs, want 1", len(sub.send))
+	}
+}
+
+// TestScanDispatch_OffersPendingWorkAsSoonAsCapacityFrees is the other side of
+// admission control: a row held back must not wait for the thirty-second
+// re-dispatch tick, or holding it back would cost more throughput than the
+// scanner-side queue ever did.
+func TestScanDispatch_OffersPendingWorkAsSoonAsCapacityFrees(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 20)
+	sub.capacity = 1
+
+	first := seedJob(t, sb, "sha256:one", originPush)
+	sb.dispatchJob(&ScanJobEvent{Type: "job", Seq: first, Repository: "repo"})
+	second := seedJob(t, sb, "sha256:two", originPush)
+	sb.dispatchJob(&ScanJobEvent{Type: "job", Seq: second, Repository: "repo"})
+	<-sub.send // drop the first job so the assertion below is unambiguous
+
+	setStatus(t, sb, first, "completed")
+	sb.offerPendingJobs()
+
+	if got := jobStatus(t, sb, second); got != "assigned" {
+		t.Errorf("held job status = %q, want assigned once the worker freed up", got)
+	}
+	select {
+	case job := <-sub.send:
+		if job.Seq != second {
+			t.Errorf("scanner received seq %d, want %d", job.Seq, second)
+		}
+	default:
+		t.Error("the freed worker was given nothing")
+	}
+}
+
+// TestScanDrain_StopsAtTheSubscriberCapacity stops the first scanner to
+// connect from swallowing a whole backlog. drainPendingJobs walked every
+// pending row and pushed it at the new subscriber, so with a backlog and two
+// scanner processes the first one took all of it and the second stayed idle —
+// horizontal scaling defeated at the connect path rather than at the gate.
+func TestScanDrain_StopsAtTheSubscriberCapacity(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 20)
+	sub.capacity = 2
+
+	var seqs []int64
+	for i := 0; i < 5; i++ {
+		seqs = append(seqs, seedJob(t, sb, "sha256:backlog", originProactive))
+	}
+
+	sb.drainPendingJobs(sub, 0)
+
+	var assigned, pending int
+	for _, seq := range seqs {
+		switch got := jobStatus(t, sb, seq); got {
+		case "assigned":
+			assigned++
+		case "pending":
+			pending++
+		default:
+			t.Errorf("job %d in unexpected status %q", seq, got)
+		}
+	}
+	if assigned != 2 || pending != 3 {
+		t.Errorf("assigned=%d pending=%d, want 2/3: a two-worker scanner takes "+
+			"two jobs, and the rest stay available to other processes",
+			assigned, pending)
+	}
+	if len(sub.send) != 2 {
+		t.Errorf("scanner received %d jobs, want 2", len(sub.send))
+	}
+}
+
+// ---------------------------------------------------------------------------
+// Disconnect, reconnect, and who owns the work in between
+// ---------------------------------------------------------------------------
+
+// TestScanUnsubscribe_DoesNotImmediatelyHandOffWorkStillRunning is the N-process
+// version of the duplicate-scan problem.
+//
+// Unsubscribe flipped every assigned and processing row belonging to the
+// dropped subscriber straight back to 'pending'. The scanner's worker pool
+// never learns the socket dropped, so it keeps scanning. With one scanner that
+// produced a duplicate against itself. With several it is worse and cannot be
+// deduped anywhere: scanner A blips, B claims A's rows out of drainPendingJobs,
+// and both processes scan the same images and both report a verdict.
+//
+// A disconnect is not evidence that a scanner is gone. It is marked as
+// disconnected and its work is left alone for a grace period instead.
+func TestScanUnsubscribe_DoesNotImmediatelyHandOffWorkStillRunning(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	first := newTestScanSubscriber(t, sb, 4)
+	first.id = "scanner-a"
+
+	seq := seedJob(t, sb, "sha256:midscan", originProactive)
+	assignJob(t, sb, seq, first, 0)
+	sb.handleAck(first, seq)
+	sb.handleStarted(first, seq)
+
+	sb.Unsubscribe(first)
+
+	if got := jobStatus(t, sb, seq); got != "processing" {
+		t.Fatalf("status = %q, want processing: a five-second blip must not "+
+			"put a running scan back in the pool", got)
+	}
+	if got := assignedTo(t, sb, seq); got != first.id {
+		t.Errorf("assigned_to = %q, want %q", got, first.id)
+	}
+
+	// A second scanner process connecting must not be handed it.
+	second := newTestScanSubscriber(t, sb, 4)
+	second.id = "scanner-b"
+	sb.drainPendingJobs(second, 0)
+
+	select {
+	case job := <-second.send:
+		t.Fatalf("seq %d was handed to a second scanner process while the first "+
+			"is still scanning it", job.Seq)
+	default:
+	}
+}
+
+// TestScanReconnect_LetsAScannerResumeItsOwnWork is why the disconnect is only
+// marked rather than acted on. A scanner keeps one identity for the life of the
+// process and sends it on every connect, so a reconnection inside the grace
+// window reclaims the jobs its workers never stopped running.
+//
+// A scanner that actually restarted comes back with a new identity, so its old
+// rows are not resumed and are reclaimed by the grace timeout instead — which
+// is right, because a restarted process really did lose that work.
+func TestScanReconnect_LetsAScannerResumeItsOwnWork(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+	sub.id = "instance-1"
+
+	seq := seedJob(t, sb, "sha256:resumed", originProactive)
+	assignJob(t, sb, seq, sub, 0)
+	sb.handleAck(sub, seq)
+	sb.handleStarted(sub, seq)
+	sb.Unsubscribe(sub)
+
+	if !jobIsDisconnected(t, sb, seq) {
+		t.Fatal("the row was not marked as belonging to a disconnected scanner")
+	}
+
+	sb.resumeInstance("instance-1")
+
+	if jobIsDisconnected(t, sb, seq) {
+		t.Error("reconnecting did not clear the disconnect mark, so the job " +
+			"will be reclaimed from a scanner that never stopped running it")
+	}
+	if got := jobStatus(t, sb, seq); got != "processing" {
+		t.Errorf("status = %q, want processing", got)
+	}
+
+	// A different process must not adopt it.
+	sb.resumeInstance("instance-2")
+	if got := assignedTo(t, sb, seq); got != "instance-1" {
+		t.Errorf("assigned_to = %q, want instance-1", got)
+	}
+}
+
+// TestScanDisconnect_ReclaimsWorkOnceTheGraceExpires bounds the wait. A scanner
+// that does not come back has its work returned to the pool, so a genuinely
+// dead process costs one grace period rather than a permanent hole.
+func TestScanDisconnect_ReclaimsWorkOnceTheGraceExpires(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+	gone := newTestScanSubscriber(t, sb, 4)
+	gone.id = "departed"
+
+	seq := seedJob(t, sb, "sha256:abandoned", originProactive)
+	assignJob(t, sb, seq, gone, 0)
+	sb.handleAck(gone, seq)
+	sb.handleStarted(gone, seq)
+	sb.Unsubscribe(gone)
+
+	// Inside the grace: still theirs.
+	sb.reDispatchTimedOut()
+	if got := jobStatus(t, sb, seq); got != "processing" {
+		t.Fatalf("status = %q, want processing inside the grace window", got)
+	}
+
+	if _, err := sb.db.Exec(
+		`UPDATE scan_jobs SET disconnected_at = ? WHERE seq = ?`,
+		time.Now().Add(-reconnectGrace-time.Minute), seq,
+	); err != nil {
+		t.Fatalf("age the disconnect: %v", err)
+	}
+
+	replacement := newTestScanSubscriber(t, sb, 4)
+	replacement.id = "replacement"
+
+	sb.reDispatchTimedOut()
+
+	if got := assignedTo(t, sb, seq); got != replacement.id {
+		t.Errorf("assigned_to = %q, want %q: a scanner that never came back "+
+			"must not hold its work forever", got, replacement.id)
+	}
+	if len(replacement.send) != 1 {
+		t.Errorf("replacement received %d jobs, want 1", len(replacement.send))
+	}
+}
+
+// TestScanReconcileOnBoot_TreatsInFlightJobsAsDisconnected covers the restart
+// case, which nothing reconciled before.
+//
+// Every connection the previous process held died with it, so a row still
+// 'assigned' or 'processing' at boot belongs to a scanner the hold has no
+// connection to — whether or not that scanner is still running it. Without the
+// mark those rows sat holding dispatch capacity until their own deadlines
+// fired, which is up to an hour for a job that was queued inside a scanner.
+// With it they fall under the same grace as a live disconnect: resumed by the
+// scanner that redials with the same identity, reclaimed otherwise.
+func TestScanReconcileOnBoot_TreatsInFlightJobsAsDisconnected(t *testing.T) {
+	sb := newConcurrencyBroadcaster(t)
+
+	assigned := seedJob(t, sb, "sha256:assigned", originProactive)
+	setStatus(t, sb, assigned, "assigned")
+	processing := seedJob(t, sb, "sha256:processing", originProactive)
+	setStatus(t, sb, processing, "processing")
+	done := seedJob(t, sb, "sha256:done", originProactive)
+	setStatus(t, sb, done, "completed")
+
+	sb.reconcileOnBoot()
+
+	for _, seq := range []int64{assigned, processing} {
+		if !jobIsDisconnected(t, sb, seq) {
+			t.Errorf("job %d was left holding dispatch capacity across a restart", seq)
+		}
+	}
+	if jobIsDisconnected(t, sb, done) {
+		t.Error("a completed job was marked as in flight")
+	}
+
+	// And the mark is what the grace acts on, so a scanner that comes back
+	// with the same identity still keeps its work.
+	if _, err := sb.db.Exec(`UPDATE scan_jobs SET assigned_to = ? WHERE seq = ?`,
+		"instance-1", processing); err != nil {
+		t.Fatalf("set owner: %v", err)
+	}
+	sb.resumeInstance("instance-1")
+	if jobIsDisconnected(t, sb, processing) {
+		t.Error("a scanner that redialed did not get its own job back")
+	}
+}

pkg/hold/pds/scan_broadcaster_stall_test.go

@@ -1,6 +1,7 @@
 package pds
 
 import (
+	"database/sql"
 	"testing"
 	"time"
 )
@@ -28,62 +29,85 @@ func jobStatus(t *testing.T, sb *ScanBroadcaster, seq int64) string {
 	return status
 }
 
-// TestScanHasActiveJobs_IgnoresLongPendingJob is the regression test for the
-// nine-day deployment-wide scanning outage. A single job sat in 'pending' with
-// nothing left to dispatch it, hasActiveJobs() counted it forever, and
-// waitForCapacity() therefore never let the proactive dispatch loop enqueue
-// another job — so discovery kept finding unscanned images and creating none.
-func TestScanHasActiveJobs_IgnoresLongPendingJob(t *testing.T) {
-	sb := newTestScanBroadcaster(t)
-	seedPendingJobs(t, sb, 1)
+// activeProactive is the count half of activeProactiveJobs, failing the test if
+// the answer is not usable.
+func activeProactive(t *testing.T, sb *ScanBroadcaster) int {
+	t.Helper()
 
-	if !sb.hasActiveJobs() {
+	n, ok := sb.activeProactiveJobs()
+	if !ok {
+		t.Fatal("activeProactiveJobs reported an unusable answer")
+	}
+	return n
+}
+
+// TestScanActiveProactiveJobs_IgnoresLongPendingJob is the regression test for
+// the nine-day deployment-wide scanning outage. A single job sat in 'pending'
+// with nothing left to dispatch it, the capacity check counted it forever, and
+// the dispatch gate therefore never let the proactive loop enqueue another job
+// — so discovery kept finding unscanned images and creating none.
+func TestScanActiveProactiveJobs_IgnoresLongPendingJob(t *testing.T) {
+	sb := newTestScanBroadcaster(t)
+	seq := seedJob(t, sb, "sha256:pendingforever", originProactive)
+
+	if activeProactive(t, sb) != 1 {
 		t.Fatal("a freshly enqueued pending job must count as active")
 	}
 
-	backdateJob(t, sb, 1, 60)
+	backdateJob(t, sb, seq, 60)
 
-	if sb.hasActiveJobs() {
-		t.Error("a job pending for an hour must not block dispatch capacity")
+	if n := activeProactive(t, sb); n != 0 {
+		t.Errorf("active = %d, want 0: a job pending for an hour must not block "+
+			"dispatch capacity", n)
 	}
 }
 
-// TestScanHasActiveJobs_CountsAssignedAndProcessing guards the other half:
-// assigned and processing jobs have their own reclaim timeouts, so they must
-// still hold capacity no matter how old the row is.
-func TestScanHasActiveJobs_CountsAssignedAndProcessing(t *testing.T) {
+// TestScanActiveProactiveJobs_CountsAssignedAndProcessing guards the other
+// half: assigned and processing jobs have their own reclaim timeouts, so they
+// must still hold capacity no matter how old the row is.
+func TestScanActiveProactiveJobs_CountsAssignedAndProcessing(t *testing.T) {
 	for _, status := range []string{"assigned", "processing"} {
 		t.Run(status, func(t *testing.T) {
 			sb := newTestScanBroadcaster(t)
-			seedPendingJobs(t, sb, 1)
-			backdateJob(t, sb, 1, 60)
+			seq := seedJob(t, sb, "sha256:inflight", originProactive)
+			backdateJob(t, sb, seq, 60)
+			setStatus(t, sb, seq, status)
 
-			if _, err := sb.db.Exec(`UPDATE scan_jobs SET status = ? WHERE seq = 1`, status); err != nil {
-				t.Fatalf("set status: %v", err)
-			}
-
-			if !sb.hasActiveJobs() {
+			if activeProactive(t, sb) != 1 {
 				t.Errorf("%s job must hold dispatch capacity", status)
 			}
 		})
 	}
 }
 
-// TestScanHasActiveJobs_IgnoresTerminalJobs keeps completed and failed rows out
-// of the capacity check — the table holds tens of thousands of them.
-func TestScanHasActiveJobs_IgnoresTerminalJobs(t *testing.T) {
+// TestScanActiveProactiveJobs_IgnoresTerminalJobs keeps completed and failed
+// rows out of the capacity check — the table holds tens of thousands of them.
+func TestScanActiveProactiveJobs_IgnoresTerminalJobs(t *testing.T) {
 	sb := newTestScanBroadcaster(t)
-	seedPendingJobs(t, sb, 2)
+	setStatus(t, sb, seedJob(t, sb, "sha256:done", originProactive), "completed")
+	setStatus(t, sb, seedJob(t, sb, "sha256:dead", originProactive), "failed")
 
-	if _, err := sb.db.Exec(`UPDATE scan_jobs SET status = 'completed' WHERE seq = 1`); err != nil {
-		t.Fatalf("complete: %v", err)
-	}
-	if _, err := sb.db.Exec(`UPDATE scan_jobs SET status = 'failed' WHERE seq = 2`); err != nil {
-		t.Fatalf("fail: %v", err)
+	if n := activeProactive(t, sb); n != 0 {
+		t.Errorf("active = %d, want 0: terminal jobs must not hold capacity", n)
 	}
+}
 
-	if sb.hasActiveJobs() {
-		t.Error("terminal jobs must not hold dispatch capacity")
+// TestScanActiveProactiveJobs_IgnoresPushTriggeredWork is the second half of
+// the gate's original defect, alongside its hardcoded depth of one. Push scans
+// never pass through this gate — a user who just pushed is waiting for the
+// answer, so oci/xrpc.go enqueues directly — but they were counted by it, so
+// on a hold with steady pushes the proactive loop had capacity approximately
+// never.
+func TestScanActiveProactiveJobs_IgnoresPushTriggeredWork(t *testing.T) {
+	sb := newTestScanBroadcaster(t)
+	setStatus(t, sb, seedJob(t, sb, "sha256:pushed", originPush), "processing")
+	// Rows written before the column existed read as push for the same reason:
+	// unknown provenance must not throttle proactive dispatch.
+	seedPendingJobs(t, sb, 1)
+
+	if n := activeProactive(t, sb); n != 0 {
+		t.Errorf("active = %d, want 0: push-triggered work must not spend the "+
+			"proactive budget", n)
 	}
 }
 
@@ -165,3 +189,140 @@ func TestScanDispatchJob_SkipsClaimedJob(t *testing.T) {
 		t.Errorf("assignment stolen from the first dispatcher, assigned_to=%q", assignedTo)
 	}
 }
+
+// TestScanDrainPendingJobs_SkipsClaimedJob is the sibling of
+// TestScanDispatchJob_SkipsClaimedJob for the other dispatcher.
+//
+// drainPendingJobs collects every pending row up front, then assigns and sends
+// them one at a time, so a row can be claimed by dispatchJob or the re-dispatch
+// loop in between. Its UPDATE carries the same `AND status = 'pending'` guard,
+// but the result was never read: the job was pushed onto this scanner's queue
+// whether or not this scanner had won it, and both scanners then scanned the
+// same image. The losing scanner's ack is silently dropped by handleAck (which
+// does guard on assigned_to), so nothing downstream notices either.
+//
+// The interleaving is built deterministically: an unbuffered send channel
+// parks the drain inside the send for job 1, which is after job 1's UPDATE and
+// before job 2's, and job 2 is claimed from under it there.
+func TestScanDrainPendingJobs_SkipsClaimedJob(t *testing.T) {
+	sb := newTestScanBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 0) // unbuffered: the drain parks in the send
+	sub.capacity = 2                       // enough room that the drain would reach job 2
+	seedPendingJobs(t, sb, 2)
+
+	done := make(chan struct{})
+	go func() {
+		defer close(done)
+		sb.drainPendingJobs(sub, 0)
+	}()
+
+	// Wait until job 1 is assigned, which means the drain is now parked in the
+	// send for it and has not yet looked at job 2.
+	deadline := time.Now().Add(10 * time.Second)
+	for jobStatus(t, sb, 1) != "assigned" {
+		if time.Now().After(deadline) {
+			t.Fatal("drain never assigned the first job")
+		}
+		time.Sleep(time.Millisecond)
+	}
+
+	// Another dispatcher claims job 2 while the drain is stuck on job 1.
+	if _, err := sb.db.Exec(
+		`UPDATE scan_jobs SET status='assigned', assigned_to='other' WHERE seq = 2 AND status = 'pending'`,
+	); err != nil {
+		t.Fatalf("claim: %v", err)
+	}
+
+	// Release the drain. Job 1 is legitimately ours.
+	select {
+	case job := <-sub.send:
+		if job.Seq != 1 {
+			t.Fatalf("first send was seq %d, want 1", job.Seq)
+		}
+	case <-time.After(10 * time.Second):
+		t.Fatal("drain never sent the first job")
+	}
+
+	// Job 2 must not follow it. The receive is what makes this decisive: a
+	// drain that sends anyway is parked in that send right now, and would
+	// otherwise give up after its own timeout and look identical to a drain
+	// that correctly skipped it.
+	select {
+	case job := <-sub.send:
+		t.Fatalf("job %d was sent to a scanner that did not claim it; both "+
+			"scanners now scan the same image", job.Seq)
+	case <-time.After(time.Second):
+	}
+
+	<-done
+
+	var assignedTo string
+	if err := sb.db.QueryRow(`SELECT assigned_to FROM scan_jobs WHERE seq = 2`).Scan(&assignedTo); err != nil {
+		t.Fatalf("query assigned_to: %v", err)
+	}
+	if assignedTo != "other" {
+		t.Errorf("assignment stolen from the first dispatcher, assigned_to=%q", assignedTo)
+	}
+}
+
+// TestScanHasActiveJobs_FailsOpenAfterPersistentDBErrors covers the last way
+// proactive scanning halts for the life of the process with no recovery.
+//
+// hasActiveJobs returned true on any query error ("assume busy") and
+// waitForCapacity spins on it, so a persistently failing query — a locked
+// database, a handle closed under a shared connection — stopped proactive
+// dispatch entirely. logStalledCapacity could not report it either: it runs
+// the same database and takes its own error branch.
+//
+// A transient error should still be treated as busy, since guessing "idle"
+// piles work onto a scanner that may already have some. A persistent one must
+// not: after a small budget of consecutive failures the check fails open, and
+// says so distinctly in the log.
+func TestScanHasActiveJobs_FailsOpenAfterPersistentDBErrors(t *testing.T) {
+	dbPath := "file:" + t.TempDir() + "/scan.db"
+	sb := newTestScanBroadcaster(t)
+	seedPendingJobs(t, sb, 1)
+
+	// Closing the handle is the cheapest persistent query failure there is.
+	if err := sb.db.Close(); err != nil {
+		t.Fatalf("close db: %v", err)
+	}
+
+	if _, ok := sb.activeProactiveJobs(); ok {
+		t.Error("the first query failure must be treated as busy; a blip is " +
+			"no reason to pile another job on a scanner")
+	}
+	for i := 1; i < activeJobsErrorBudget; i++ {
+		if _, ok := sb.activeProactiveJobs(); ok {
+			t.Errorf("failure %d is still within the budget and must read as busy", i+1)
+		}
+	}
+	if _, ok := sb.activeProactiveJobs(); !ok {
+		t.Fatalf("the capacity check still reports busy after %d consecutive "+
+			"database errors: proactive dispatch is halted for the life of the "+
+			"process with no recovery", activeJobsErrorBudget+1)
+	}
+
+	// A working database restores the budget, so a later blip is absorbed
+	// rather than landing on an already-exhausted counter.
+	db, err := sql.Open("libsql", dbPath)
+	if err != nil {
+		t.Fatalf("reopen db: %v", err)
+	}
+	t.Cleanup(func() { _ = db.Close() })
+	sb.db = db
+	if err := sb.initSchema(); err != nil {
+		t.Fatalf("initSchema: %v", err)
+	}
+	seedJob(t, sb, "sha256:fresh", originProactive)
+
+	if activeProactive(t, sb) != 1 {
+		t.Fatal("a fresh pending job must count as active")
+	}
+	if err := sb.db.Close(); err != nil {
+		t.Fatalf("close db again: %v", err)
+	}
+	if _, ok := sb.activeProactiveJobs(); ok {
+		t.Error("the error budget was not reset by a successful query")
+	}
+}

pkg/hold/pds/scan_broadcaster_stuck_test.go

@@ -0,0 +1,616 @@
+package pds
+
+import (
+	"context"
+	"testing"
+	"time"
+
+	"atcr.io/pkg/atproto"
+)
+
+// These tests cover the ways a scan job wedges on the hold side: the clock the
+// processing deadline is measured from, what happens to a job that blows it,
+// and what the hold does with a job whose scanner vanished mid-scan.
+//
+// They are characterization tests. Where the behaviour they pin is wrong, the
+// comment says so and the assertion still describes what the code does today,
+// so the suite stays green until a fix lands. A test that can only pass after a
+// fix is marked with t.Skip and names the bug.
+//
+// The scanner-side half of the disconnect story lives in
+// scanner/internal/e2e/stuck_test.go; where a scenario here encodes an
+// assumption about what the scanner does with a re-offered job, that file pins
+// it against the real scanner.
+
+// newStuckBroadcaster is the broadcaster these scenarios need: the in-flight
+// digest set and the ack timeout the constructor fills in but the bare helper
+// leaves zeroed, plus the PDS and S3 stand-in behind every terminal
+// transition. The processing timeout writes a scan record now, so a
+// broadcaster with a nil pds is no longer a usable stand-in for one.
+func newStuckBroadcaster(t *testing.T) *ScanBroadcaster {
+	t.Helper()
+
+	sb := newRecordingScanBroadcaster(t)
+	// A buffered signal so a test can assert dispatch capacity was actually
+	// released; production wires the same channel to the dispatch loop.
+	sb.completionSignal = make(chan struct{}, 1)
+	return sb
+}
+
+// seedJobWithDigest inserts one pending job carrying a specific manifest
+// digest. seedPendingJobs gives every row the same digest, which is fine for
+// status bookkeeping but useless for anything that keys on the digest.
+func seedJobWithDigest(t *testing.T, sb *ScanBroadcaster, digest string) int64 {
+	t.Helper()
+
+	res, err := sb.db.Exec(`
+		INSERT INTO scan_jobs
+			(manifest_digest, repository, tag, user_did, user_handle,
+			 hold_did, hold_endpoint, tier, config_json, layers_json, status)
+		VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending')
+	`, digest, "repo", "latest", "did:plc:user", "user.example.com",
+		sb.holdDID, sb.holdEndpoint, "deckhand", "{}", "[]")
+	if err != nil {
+		t.Fatalf("seed job: %v", err)
+	}
+	seq, err := res.LastInsertId()
+	if err != nil {
+		t.Fatalf("seq: %v", err)
+	}
+	return seq
+}
+
+// assignJob puts a row in 'assigned' as dispatchJob would, with assigned_at set
+// the given number of minutes in the past.
+func assignJob(t *testing.T, sb *ScanBroadcaster, seq int64, sub *ScanSubscriber, agoMinutes int) {
+	t.Helper()
+
+	at := time.Now().Add(-time.Duration(agoMinutes) * time.Minute)
+	_, err := sb.db.Exec(`
+		UPDATE scan_jobs SET status = 'assigned', assigned_to = ?, assigned_at = ?
+		WHERE seq = ?
+	`, sub.id, at, seq)
+	if err != nil {
+		t.Fatalf("assign job %d: %v", seq, err)
+	}
+}
+
+func assignedAt(t *testing.T, sb *ScanBroadcaster, seq int64) time.Time {
+	t.Helper()
+
+	var at time.Time
+	if err := sb.db.QueryRow(`SELECT assigned_at FROM scan_jobs WHERE seq = ?`, seq).Scan(&at); err != nil {
+		t.Fatalf("query assigned_at for %d: %v", seq, err)
+	}
+	return at
+}
+
+// TestScanAck_DoesNotStartTheScanningClock is the hold-side half of the ack
+// timing mismatch.
+//
+// The scanner acks the moment a job comes off the WebSocket, before it is even
+// queued (client/hold.go handleFrame sends the ack, then Enqueue). handleAck
+// moves the row 'assigned' → 'processing' and touches nothing else, which is
+// correct — the ack means "I have it", not "a worker is on it".
+//
+// What was wrong was the deadline. The ten-minute processing budget was
+// measured from assigned_at, so it covered however long the job spent queued
+// inside the scanner: a 100-deep queue of no-op jobs drains in 16m40s and
+// crosses the deadline at position 59, and with 16-second scans at position
+// 23. Every job past that point was failed underneath a scanner that was
+// working perfectly, and since the timeout now writes a scan record, each one
+// is a "scan failed" the user can see.
+//
+// The scanning deadline is measured from 'started' instead, and a job that has
+// only been acked falls under the much larger queue budget.
+func TestScanAck_DoesNotStartTheScanningClock(t *testing.T) {
+	sb := newStuckBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	seq := seedJobWithDigest(t, sb, "sha256:aaaa")
+	assignJob(t, sb, seq, sub, 11)
+	before := assignedAt(t, sb, seq)
+
+	sb.handleAck(sub, seq)
+
+	if got := jobStatus(t, sb, seq); got != "processing" {
+		t.Fatalf("after ack: status = %q, want processing", got)
+	}
+	if after := assignedAt(t, sb, seq); !after.Equal(before) {
+		t.Errorf("ack moved assigned_at from %s to %s; it is the dispatch "+
+			"timestamp and the queue budget is measured from it", before, after)
+	}
+	if startedAt(t, sb, seq).Valid {
+		t.Error("the ack stamped started_at; only a worker picking the job up does that")
+	}
+
+	// The scanner acked eleven minutes after dispatch and is still holding the
+	// job. Nothing here is evidence of a problem.
+	sb.reDispatchTimedOut()
+
+	if got := jobStatus(t, sb, seq); got != "processing" {
+		t.Fatalf("status = %q, want processing: a job acked and queued for "+
+			"eleven minutes was cancelled underneath a working scanner", got)
+	}
+}
+
+// TestScanProcessingTimeout_RetiresTheJobProperly is the compounding half of
+// the stuck-scan story, and the reason a single hung scan used to remove an
+// image from scanning until the hold process restarted.
+//
+// Every proactive enqueue path adds the manifest digest to sb.inflight and
+// relies on a terminal transition to take it out again. The ten-minute
+// processing sweep in reDispatchTimedOut was the one terminal transition that
+// did neither of the two things all the others do: it wrote no scan record
+// and never called removeInflight. So the digest stayed in the set —
+// discoverUnscannedForUser and runStalePass both skip any manifest whose
+// addInflight returns false — and nothing in the system recorded that the
+// manifest had ever been attempted, which the appview renders as a grey "Not
+// scanned" indefinitely.
+//
+// The sweep is now a terminal transition like handleError: a failed scan
+// record with a reason, the digest released, and dispatch capacity signalled.
+// "Failed" rather than "skipped" is deliberate — a hung scanner is a
+// transient condition, so the stale loop should retry it on the rescan
+// interval.
+func TestScanProcessingTimeout_RetiresTheJobProperly(t *testing.T) {
+	sb := newStuckBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	const digest = "sha256:hungscan"
+	seq := seedJobWithDigest(t, sb, digest)
+	assignJob(t, sb, seq, sub, 11)
+	sb.handleAck(sub, seq)
+	markStarted(t, sb, seq, 11) // a worker took it and then went quiet
+
+	if !sb.addInflight(digest) {
+		// The enqueue paths do this; do it here so the state matches.
+		t.Fatal("digest was already in flight before the test started")
+	}
+
+	sb.reDispatchTimedOut()
+
+	if got := jobStatus(t, sb, seq); got != "failed" {
+		t.Fatalf("status = %q, want failed", got)
+	}
+
+	if !sb.addInflight(digest) {
+		t.Error("the timed-out digest is still in flight: discovery and the " +
+			"stale loop will skip this manifest for the life of the process")
+	}
+	sb.removeInflight(digest)
+
+	_, record, err := sb.pds.GetScanRecord(context.Background(), digest)
+	if err != nil {
+		t.Fatalf("no scan record was written for a job the hold gave up on, so "+
+			"the appview cannot tell it apart from one enqueued a minute ago: %v", err)
+	}
+	if record.Status != atproto.ScanStatusFailed {
+		t.Errorf("record status = %q, want %q", record.Status, atproto.ScanStatusFailed)
+	}
+	if record.Reason == "" {
+		t.Error("failed record carries no reason; it is the only thing a user sees")
+	}
+
+	select {
+	case <-sb.completionSignal:
+	default:
+		t.Error("no completion signal: the dispatch loop sleeps up to 5s longer than it needs to")
+	}
+
+	// And nothing re-offers the row to the scanner that is still connected.
+	select {
+	case job := <-sub.send:
+		t.Fatalf("timed-out processing job %d was re-dispatched", job.Seq)
+	default:
+	}
+}
+
+// TestScanProcessingTimeout_LeavesAJobInsideTheDeadlineAlone is the guard on
+// the other side of the sweep: it must only touch rows that have actually
+// blown the ten minutes, and it must not write records for jobs a scanner is
+// still legitimately working on.
+func TestScanProcessingTimeout_LeavesAJobInsideTheDeadlineAlone(t *testing.T) {
+	sb := newStuckBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	const digest = "sha256:stillworking"
+	seq := seedJobWithDigest(t, sb, digest)
+	assignJob(t, sb, seq, sub, 2)
+	sb.handleAck(sub, seq)
+	markStarted(t, sb, seq, 2)
+	sb.addInflight(digest)
+
+	sb.reDispatchTimedOut()
+
+	if got := jobStatus(t, sb, seq); got != "processing" {
+		t.Errorf("status = %q, want processing", got)
+	}
+	if sb.addInflight(digest) {
+		t.Error("digest was released while the scan is still inside its deadline")
+	}
+	if _, _, err := sb.pds.GetScanRecord(context.Background(), digest); err == nil {
+		t.Error("a failure record was written for a scan that is still running")
+	}
+}
+
+// TestScanProcessingTimeout_ReleasesCapacityWhileTheScannerIsStillWedged is
+// what turns one hung scan into a slow leak rather than a single lost job.
+//
+// hasActiveJobs counts 'processing' rows with no age bound, so a wedged scan
+// holds the proactive dispatch loop still — until the ten-minute timeout marks
+// it 'failed', at which point capacity is free again and dispatchLoop picks the
+// next candidate and hands it to the same scanner, whose only worker is still
+// stuck on the first one. Repeat every ten minutes: each new job is acked,
+// queued behind the wedge, failed by the timeout, and leaks its digest out of
+// the in-flight set for good.
+//
+// Before dfd604b the same wedge froze dispatch outright, which is the outage
+// that commit was written for. It bounded 'pending' but left 'processing'
+// alone, so the freeze became this drip instead.
+func TestScanProcessingTimeout_ReleasesCapacityWhileTheScannerIsStillWedged(t *testing.T) {
+	sb := newStuckBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	seq := seedJob(t, sb, "sha256:wedged", originProactive)
+	assignJob(t, sb, seq, sub, 11)
+	sb.handleAck(sub, seq)
+	markStarted(t, sb, seq, 11)
+
+	if n := activeProactive(t, sb); n != 1 {
+		t.Fatalf("active proactive jobs = %d, want 1: a processing job must "+
+			"hold dispatch capacity", n)
+	}
+
+	sb.reDispatchTimedOut()
+
+	if activeProactive(t, sb) != 0 {
+		t.Fatal("capacity is still held after the processing timeout; the " +
+			"drip this test describes cannot happen, update it")
+	}
+	t.Logf("job %d is failed and capacity is free, but the scanner that never "+
+		"answered for it is unchanged: the next candidate goes to the same "+
+		"wedged worker. The digest is released and a failed record written, so "+
+		"the manifest is at least retried on the rescan interval", seq)
+}
+
+// TestScanResult_ArrivingAfterTheTimeoutHealsTheRow bounds the previous test:
+// the leak is permanent only when the scanner never speaks for that seq again.
+// handleResult has no status guard, so a result that arrives after the deadline
+// flips 'failed' back to 'completed' and releases the digest.
+//
+// That is what makes the leak hard to see in production. It bites exactly in
+// the cases where the scanner is wedged (an unbounded Syft extraction) or where
+// its terminal message was written to a dead socket and dropped
+// (client/hold.go sendJSON) — the same cases where scanning was already stuck.
+func TestScanResult_ArrivingAfterTheTimeoutHealsTheRow(t *testing.T) {
+	sb := newRecordingScanBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	const digest = "sha256:lateresult"
+	seq := seedJobWithDigest(t, sb, digest)
+	assignJob(t, sb, seq, sub, 11)
+	sb.handleAck(sub, seq)
+	markStarted(t, sb, seq, 11)
+	sb.addInflight(digest)
+
+	sb.reDispatchTimedOut()
+	if got := jobStatus(t, sb, seq); got != "failed" {
+		t.Fatalf("status = %q, want failed", got)
+	}
+
+	// handleResult writes a scan record, so this needs the broadcaster that
+	// has a PDS behind it. The row bookkeeping under test happens after that
+	// write, which is exactly why the write must not be able to panic.
+	sb.handleResult(sub, ScannerMessage{Type: "result", Seq: seq, SBOM: testSBOM})
+
+	if got := jobStatus(t, sb, seq); got != "completed" {
+		t.Errorf("status = %q, want completed: a late result overwrites the "+
+			"failure with no status guard", got)
+	}
+	if !sb.addInflight(digest) {
+		t.Error("late result did not release the in-flight digest")
+	}
+}
+
+// TestScanHandleResult_SurvivesAResultWithoutSummary covers what used to be a
+// hold crash reachable from any scanner running with vuln.enabled=false.
+//
+// The scanner only fills ScanResult.Summary when Grype ran (worker.go
+// processJob, step 3), and SendResult copies it straight through, so with
+// vulnerability scanning disabled every successful scan sends a result with no
+// summary. handleResult guarded the record-writing branch with
+// `if msg.Summary != nil` and then dereferenced msg.Summary unguarded in its
+// final log line. That panic was not in an HTTP handler; it was in the
+// subscriber's reader goroutine, so it took the whole hold process down, and
+// the job was re-dispatched on restart into the same crash.
+//
+// Two things have to hold now. The obvious one is that nothing panics. The
+// less obvious one is that the SBOM still lands in a scan record: the blob was
+// uploaded to S3 before the record write, so skipping the record (as the old
+// `if msg.Summary != nil` guard did) leaves that blob orphaned with nothing
+// referencing it.
+//
+// A nil summary means "not scanned for vulnerabilities", which is NOT the same
+// as "scanned, found zero". The record therefore carries no vulnerability
+// report blob, and the appview keys off that to avoid claiming the image is
+// clean (see classifyScanRecord in pkg/appview/handlers/scan_result.go).
+func TestScanHandleResult_SurvivesAResultWithoutSummary(t *testing.T) {
+	sb := newRecordingScanBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	const digest = "sha256:nosummary"
+	seq := seedJobWithDigest(t, sb, digest)
+	assignJob(t, sb, seq, sub, 0)
+	sb.handleAck(sub, seq)
+
+	func() {
+		defer func() {
+			if r := recover(); r != nil {
+				t.Fatalf("handleResult panicked on a summary-less result: %v", r)
+			}
+		}()
+		// Exactly the message a vuln-disabled scanner sends: an SBOM, no
+		// summary, no vulnerability report.
+		sb.handleResult(sub, ScannerMessage{Type: "result", Seq: seq, SBOM: testSBOM})
+	}()
+
+	if got := jobStatus(t, sb, seq); got != "completed" {
+		t.Errorf("status = %q, want completed", got)
+	}
+
+	_, record, err := sb.pds.GetScanRecord(context.Background(), digest)
+	if err != nil {
+		t.Fatalf("no scan record was written for a summary-less result, so the "+
+			"SBOM blob already in S3 is orphaned: %v", err)
+	}
+	if record.SbomBlob == nil {
+		t.Error("scan record carries no SBOM blob; the uploaded blob is orphaned")
+	}
+	if record.VulnReportBlob != nil {
+		t.Error("scan record carries a vulnerability report that was never produced")
+	}
+	if record.Status != atproto.ScanStatusOK {
+		t.Errorf("status = %q, want %q: the scan itself succeeded", record.Status, atproto.ScanStatusOK)
+	}
+	if record.Total != 0 || record.Critical != 0 || record.High != 0 || record.Medium != 0 || record.Low != 0 {
+		t.Errorf("counts = %d/%d/%d/%d total %d, want all zero: Grype never ran",
+			record.Critical, record.High, record.Medium, record.Low, record.Total)
+	}
+}
+
+// TestScanHandleResult_RecordsSummaryCountsWhenGrypeRan is the other half of
+// the pair: with a summary present the counts must reach the record, so the
+// nil-tolerant path above cannot be satisfied by dropping them everywhere.
+func TestScanHandleResult_RecordsSummaryCountsWhenGrypeRan(t *testing.T) {
+	sb := newRecordingScanBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	const digest = "sha256:withsummary"
+	seq := seedJobWithDigest(t, sb, digest)
+	assignJob(t, sb, seq, sub, 0)
+	sb.handleAck(sub, seq)
+
+	sb.handleResult(sub, ScannerMessage{
+		Type:       "result",
+		Seq:        seq,
+		SBOM:       testSBOM,
+		VulnReport: `{"matches":[]}`,
+		Summary:    &VulnerabilitySummary{Critical: 1, High: 2, Medium: 3, Low: 4, Total: 10},
+	})
+
+	_, record, err := sb.pds.GetScanRecord(context.Background(), digest)
+	if err != nil {
+		t.Fatalf("get scan record: %v", err)
+	}
+	if record.Critical != 1 || record.High != 2 || record.Medium != 3 || record.Low != 4 || record.Total != 10 {
+		t.Errorf("counts = %d/%d/%d/%d total %d, want 1/2/3/4 total 10",
+			record.Critical, record.High, record.Medium, record.Low, record.Total)
+	}
+	if record.SbomBlob == nil || record.VulnReportBlob == nil {
+		t.Errorf("blobs = sbom:%v vuln:%v, want both", record.SbomBlob != nil, record.VulnReportBlob != nil)
+	}
+}
+
+// testSBOM is a stand-in for the SPDX document the scanner sends. Only its
+// bytes matter here: the hold hashes them into a blob CID and never parses it.
+const testSBOM = `{"spdxVersion":"SPDX-2.3","packages":[]}`
+
+// TestScanUnsubscribe_HoldsAJobForAReconnectingScanner is the hold-side
+// assumption behind the duplicate-scan scenario in
+// scanner/internal/e2e/stuck_test.go, updated for a hold that expects several
+// scanner processes.
+//
+// Unsubscribe used to flip every assigned and processing row belonging to the
+// dropped subscriber back to 'pending'. Nothing tells the scanner: its worker
+// pool still has the job queued, or a worker is halfway through downloading
+// blobs for it. drainPendingJobs then handed the same seq straight to whoever
+// connected next. Against a single scanner that was a duplicate against
+// itself, which a local dedupe could in principle catch. Against N processes
+// nothing can catch it, because the two copies are in different processes.
+//
+// A disconnect is now recorded rather than acted on, and a scanner keeps one
+// identity for the life of its process, so the reconnecting scanner gets its
+// own work back and a different process is never offered it.
+func TestScanUnsubscribe_HoldsAJobForAReconnectingScanner(t *testing.T) {
+	sb := newStuckBroadcaster(t)
+	first := newTestScanSubscriber(t, sb, 4)
+	first.id = "instance-1"
+
+	seq := seedJobWithDigest(t, sb, "sha256:duplicated")
+	assignJob(t, sb, seq, first, 0)
+	sb.handleAck(first, seq)
+	sb.handleStarted(first, seq)
+	if got := jobStatus(t, sb, seq); got != "processing" {
+		t.Fatalf("status = %q, want processing", got)
+	}
+
+	sb.Unsubscribe(first)
+
+	if got := jobStatus(t, sb, seq); got != "processing" {
+		t.Fatalf("after disconnect: status = %q, want processing", got)
+	}
+
+	// A different process connecting must not be offered it.
+	other := newTestScanSubscriber(t, sb, 4)
+	other.id = "instance-2"
+	sb.drainPendingJobs(other, 0)
+
+	select {
+	case job := <-other.send:
+		t.Fatalf("seq %d was handed to a second scanner process while the first "+
+			"is still scanning it", job.Seq)
+	default:
+	}
+
+	// The original process redialing resumes it. Nothing is re-sent: the
+	// scanner never lost the job, only the socket.
+	sb.resumeInstance("instance-1")
+	if jobIsDisconnected(t, sb, seq) {
+		t.Error("the reconnecting scanner did not get its own job back")
+	}
+	if got := jobStatus(t, sb, seq); got != "processing" {
+		t.Errorf("status = %q, want processing", got)
+	}
+}
+
+// TestScanUnsubscribe_DoesNotReleaseTheInFlightDigest is a smaller sibling of
+// the timeout leak: a disconnect leaves the digest in flight. That is harmless
+// — the job is still the disconnected scanner's, and will either be resumed or
+// reclaimed — but it means the set cannot be read as "jobs a scanner is
+// working on".
+func TestScanUnsubscribe_DoesNotReleaseTheInFlightDigest(t *testing.T) {
+	sb := newStuckBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	const digest = "sha256:disconnected"
+	seq := seedJobWithDigest(t, sb, digest)
+	assignJob(t, sb, seq, sub, 0)
+	sb.addInflight(digest)
+
+	sb.Unsubscribe(sub)
+
+	if got := jobStatus(t, sb, seq); got != "assigned" {
+		t.Fatalf("status = %q, want assigned: the row stays the disconnected "+
+			"scanner's until it resumes or the grace expires", got)
+	}
+	if !jobIsDisconnected(t, sb, seq) {
+		t.Error("the row was not marked as belonging to a disconnected scanner")
+	}
+	if sb.addInflight(digest) {
+		t.Error("Unsubscribe released the in-flight digest: behaviour changed")
+	}
+}
+
+// TestScanDispatchQueue_ReturnsUndeliverableJobsToPending covers what the
+// drain does when it cannot hand a scanner everything it claimed.
+//
+// sub.send is 20 deep in production. dispatchJob's default branch resets the
+// row to 'pending' when the buffer is full; drainPendingJobs used to block for
+// up to five seconds and then simply return, leaving every row it had already
+// marked 'assigned' owned by a subscriber that was never sent them. Those rows
+// sat in 'assigned' — counted as active dispatch capacity — until the
+// five-minute ack timeout reclaimed them.
+//
+// The drain now puts the row it could not deliver back to 'pending', so the
+// re-dispatch loop can offer it again on its next tick.
+func TestScanDispatchQueue_ReturnsUndeliverableJobsToPending(t *testing.T) {
+	sb := newStuckBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 2) // tiny buffer, nobody draining it
+	sub.capacity = 4                       // the send buffer, not capacity, is the constraint here
+
+	var seqs []int64
+	for i := 0; i < 4; i++ {
+		seqs = append(seqs, seedJobWithDigest(t, sb, "sha256:burst"))
+	}
+
+	done := make(chan struct{})
+	go func() {
+		defer close(done)
+		sb.drainPendingJobs(sub, 0)
+	}()
+
+	select {
+	case <-done:
+	case <-time.After(30 * time.Second):
+		t.Fatal("drainPendingJobs never returned")
+	}
+
+	var assigned, pending int
+	for _, seq := range seqs {
+		switch got := jobStatus(t, sb, seq); got {
+		case "assigned":
+			assigned++
+		case "pending":
+			pending++
+		default:
+			t.Errorf("job %d in unexpected status %q", seq, got)
+		}
+	}
+
+	// Two made it into the buffer and are genuinely assigned. The third blocked
+	// out the five-second window and must have been handed back, and the fourth
+	// was never claimed at all.
+	if assigned != 2 || pending != 2 {
+		t.Errorf("assigned=%d pending=%d, want 2 assigned / 2 pending: a row the "+
+			"scanner was never sent must not stay assigned to it", assigned, pending)
+	}
+}
+
+// TestScanSkipped_RetiresAnUndecodableFrame is the hold-side half of the
+// scanner's answer to a frame it cannot parse
+// (TestUnparseableFramesAreAnsweredWithSkipped in
+// scanner/internal/e2e/protocol_test.go).
+//
+// The scanner now replies "skipped" for any job frame carrying a usable seq,
+// which is only worth doing if it actually retires the row. It does: the job
+// goes to 'completed', a skipped scan record lands in the PDS so the stale
+// loop leaves it alone, and the in-flight digest is released. Before that
+// reply existed, the row stayed 'assigned', timed out after five minutes, was
+// re-offered to the same scanner within thirty seconds, and was dropped again
+// — while hasActiveJobs counted it and no proactive scan was dispatched
+// anywhere in the deployment.
+func TestScanSkipped_RetiresAnUndecodableFrame(t *testing.T) {
+	sb := newStuckBroadcaster(t)
+	sub := newTestScanSubscriber(t, sb, 4)
+
+	const digest = "sha256:undecodable"
+	seq := seedJobWithDigest(t, sb, digest)
+	assignJob(t, sb, seq, sub, 0)
+	sb.addInflight(digest)
+
+	// Exactly what connectOnce sends when the config sub-document does not
+	// decode: no ack ever arrives, so the row is still 'assigned'.
+	sb.handleSkipped(sub, ScannerMessage{
+		Type:   "skipped",
+		Seq:    seq,
+		Reason: "malformed job config: json: cannot unmarshal string into Go value of type scanner.BlobDescriptor",
+	})
+
+	if got := jobStatus(t, sb, seq); got != "completed" {
+		t.Errorf("status = %q, want completed: a skip must be terminal", got)
+	}
+	if !sb.addInflight(digest) {
+		t.Error("skip did not release the in-flight digest")
+	}
+	sb.removeInflight(digest)
+
+	_, record, err := sb.pds.GetScanRecord(context.Background(), digest)
+	if err != nil {
+		t.Fatalf("no scan record for a skipped job: %v", err)
+	}
+	if record.Status != atproto.ScanStatusSkipped {
+		t.Errorf("record status = %q, want %q", record.Status, atproto.ScanStatusSkipped)
+	}
+
+	// Terminal means terminal: the re-dispatch loop must not pick it back up.
+	sb.reDispatchTimedOut()
+	if got := jobStatus(t, sb, seq); got != "completed" {
+		t.Errorf("after a re-dispatch tick: status = %q, want completed", got)
+	}
+	select {
+	case job := <-sub.send:
+		t.Fatalf("a skipped job (%d) was re-offered to the scanner", job.Seq)
+	default:
+	}
+}

pkg/hold/pds/scan_broadcaster_test.go

@@ -4,6 +4,8 @@ import (
 	"database/sql"
 	"testing"
 	"time"
+
+	"atcr.io/pkg/s3"
 )
 
 // newTestScanBroadcaster builds a ScanBroadcaster with just a database, no
@@ -30,6 +32,26 @@ func newTestScanBroadcaster(t *testing.T) *ScanBroadcaster {
 	return sb
 }
 
+// newRecordingScanBroadcaster is newTestScanBroadcaster plus the two
+// dependencies handleResult needs to finish its work: an embedded PDS to write
+// the scan record into, and an S3 stand-in to take the SBOM blob. The bare
+// helper leaves both nil, which is fine for row bookkeeping and fatal for
+// anything that asserts on what was stored.
+func newRecordingScanBroadcaster(t *testing.T) *ScanBroadcaster {
+	t.Helper()
+
+	sb := newTestScanBroadcaster(t)
+	sb.inflight = make(map[string]struct{})
+	sb.ackTimeout = 5 * time.Minute
+
+	pds, _ := setupTestPDS(t)
+	sb.pds = pds
+	sb.holdDID = pds.did
+	sb.s3 = &s3.S3Service{Client: s3.NewMockS3Client(""), Bucket: "test-bucket"}
+
+	return sb
+}
+
 // newTestScanSubscriber mirrors what Subscribe builds, registered with the
 // broadcaster so Unsubscribe finds it.
 func newTestScanSubscriber(t *testing.T, sb *ScanBroadcaster, bufSize int) *ScanSubscriber {
@@ -40,6 +62,9 @@ func newTestScanSubscriber(t *testing.T, sb *ScanBroadcaster, bufSize int) *Scan
 		send: make(chan *ScanJobEvent, bufSize),
 		id:   "test-subscriber",
 		done: make(chan struct{}),
+		// One worker unless a test says otherwise, which is what a scanner
+		// that declares nothing is treated as.
+		capacity: 1,
 	}
 
 	sb.mu.Lock()
@@ -93,41 +118,66 @@ func TestScanUnsubscribe_IsIdempotent(t *testing.T) {
 	}
 }
 
-// TestScanUnsubscribe_UnassignsJobsOnce verifies the idempotency guard protects
-// the job-reassignment UPDATE too. Re-running it would unassign jobs that a
-// replacement scanner had already been given.
-func TestScanUnsubscribe_UnassignsJobsOnce(t *testing.T) {
+// TestScanUnsubscribe_MarksItsOwnJobsOnce verifies the idempotency guard
+// protects the disconnect bookkeeping too, and that the bookkeeping is scoped
+// to this subscriber's rows.
+//
+// Unsubscribe used to flip every assigned and processing row straight back to
+// 'pending', which handed a running scan to whichever process connected next
+// (see TestScanUnsubscribe_DoesNotImmediatelyHandOffWorkStillRunning). It now
+// marks the disconnect and leaves the work where it is. Either way the guard
+// matters for the same reason: a dropped scanner unwinds both handleWriter and
+// handleReader, and each calls Unsubscribe.
+func TestScanUnsubscribe_MarksItsOwnJobsOnce(t *testing.T) {
 	sb := newTestScanBroadcaster(t)
 	sub := newTestScanSubscriber(t, sb, 4)
 	seedPendingJobs(t, sb, 1)
 
-	if _, err := sb.db.Exec(`UPDATE scan_jobs SET status='assigned', assigned_to=?`, sub.id); err != nil {
+	if _, err := sb.db.Exec(`UPDATE scan_jobs SET status='processing', assigned_to=?`, sub.id); err != nil {
 		t.Fatalf("assign: %v", err)
 	}
 
 	sb.Unsubscribe(sub)
 
-	var status string
-	if err := sb.db.QueryRow(`SELECT status FROM scan_jobs LIMIT 1`).Scan(&status); err != nil {
-		t.Fatalf("query: %v", err)
-	}
-	if status != "pending" {
-		t.Errorf("expected job returned to pending, got %q", status)
+	var (
+		status         string
+		assignedTo     sql.NullString
+		disconnectedAt sql.NullTime
+	)
+	row := func() {
+		t.Helper()
+		if err := sb.db.QueryRow(
+			`SELECT status, assigned_to, disconnected_at FROM scan_jobs LIMIT 1`,
+		).Scan(&status, &assignedTo, &disconnectedAt); err != nil {
+			t.Fatalf("query: %v", err)
+		}
 	}
 
+	row()
+	if status != "processing" || assignedTo.String != sub.id {
+		t.Errorf("job = %q/%q, want processing/%q: a disconnect is not evidence "+
+			"the scanner stopped scanning", status, assignedTo.String, sub.id)
+	}
+	if !disconnectedAt.Valid {
+		t.Error("the row was not marked as belonging to a disconnected scanner")
+	}
+	firstMark := disconnectedAt.Time
+
 	// Hand the job to a "replacement" scanner, then unsubscribe the dead one
-	// again. The guard must stop it from stealing the job back.
-	if _, err := sb.db.Exec(`UPDATE scan_jobs SET status='assigned', assigned_to=?`, "replacement"); err != nil {
+	// again. The guard must stop it from touching a row that has moved on.
+	if _, err := sb.db.Exec(
+		`UPDATE scan_jobs SET status='assigned', assigned_to=?, disconnected_at=NULL`, "replacement",
+	); err != nil {
 		t.Fatalf("reassign: %v", err)
 	}
 	sb.Unsubscribe(sub)
 
-	if err := sb.db.QueryRow(`SELECT status FROM scan_jobs LIMIT 1`).Scan(&status); err != nil {
-		t.Fatalf("query: %v", err)
-	}
-	if status != "assigned" {
-		t.Errorf("second Unsubscribe stole the replacement's job, status=%q", status)
+	row()
+	if status != "assigned" || assignedTo.String != "replacement" || disconnectedAt.Valid {
+		t.Errorf("second Unsubscribe touched the replacement's job: %q/%q "+
+			"disconnected=%v", status, assignedTo.String, disconnectedAt.Valid)
 	}
+	_ = firstMark
 }
 
 // TestScanDrainPendingJobs_ConcurrentUnsubscribe is the regression test for
@@ -143,6 +193,7 @@ func TestScanDrainPendingJobs_ConcurrentUnsubscribe(t *testing.T) {
 		func() {
 			sb := newTestScanBroadcaster(t)
 			sub := newTestScanSubscriber(t, sb, 1)
+			sub.capacity = 50 // the drain must walk rows, not stop at capacity
 			seedPendingJobs(t, sb, 50)
 
 			done := make(chan struct{})

pkg/hold/pds/scan_broadcaster_ws_test.go

@@ -67,7 +67,7 @@ func TestScanBroadcaster_UnsubscribeIsIdempotent(t *testing.T) {
 	sb := newTestScanBroadcaster(t)
 	conn, _ := wsPair(t)
 
-	sub := sb.Subscribe(conn, 0)
+	sub := sb.Subscribe(conn, 0, "", 0)
 	if got := subscriberCount(sb); got != 1 {
 		t.Fatalf("subscriber count after Subscribe = %d, want 1", got)
 	}
@@ -88,7 +88,7 @@ func TestScanBroadcaster_WriterExitsOnUnsubscribe(t *testing.T) {
 	sb := newTestScanBroadcaster(t)
 	conn, _ := wsPair(t)
 
-	sub := sb.Subscribe(conn, 0)
+	sub := sb.Subscribe(conn, 0, "", 0)
 	sb.Unsubscribe(sub)
 
 	deadline := time.Now().Add(5 * time.Second)
@@ -108,7 +108,7 @@ func TestScanBroadcaster_DroppedScannerUnwindsOnce(t *testing.T) {
 	sb := newTestScanBroadcaster(t)
 	conn, closeClient := wsPair(t)
 
-	sub := sb.Subscribe(conn, 0)
+	sub := sb.Subscribe(conn, 0, "", 0)
 	closeClient()
 
 	deadline := time.Now().Add(5 * time.Second)

pkg/hold/pds/xrpc.go

@@ -1101,6 +1101,24 @@ func (h *XRPCHandler) HandleSubscribeScanJobs(w http.ResponseWriter, r *http.Req
 		}
 	}
 
+	// How much scanning this process can run at once, and who it is.
+	//
+	// Both are optional: a scanner built before they existed sends neither,
+	// and the broadcaster then treats it as one worker with a
+	// per-connection identity — exactly the behaviour it had before. A
+	// malformed workers value is ignored rather than rejected, since the
+	// connection is still perfectly usable at the default.
+	workers := 0
+	if raw := r.URL.Query().Get("workers"); raw != "" {
+		if n, err := strconv.Atoi(raw); err == nil && n > 0 {
+			workers = n
+		} else {
+			slog.Warn("Scanner declared an unusable worker count, using the default",
+				"workers", raw)
+		}
+	}
+	instanceID := r.URL.Query().Get("instance")
+
 	// Upgrade to WebSocket
 	conn, err := upgrader.Upgrade(w, r, nil)
 	if err != nil {
@@ -1108,7 +1126,7 @@ func (h *XRPCHandler) HandleSubscribeScanJobs(w http.ResponseWriter, r *http.Req
 		return
 	}
 
-	h.scanBroadcaster.Subscribe(conn, cursor)
+	h.scanBroadcaster.Subscribe(conn, cursor, instanceID, workers)
 }
 
 // ScanBroadcasterRef returns the scan broadcaster (used by OCI handler to enqueue jobs)

scanner/cmd/scanner/main.go

@@ -77,8 +77,11 @@ Environment variables always override file values (SCANNER_ prefix).`,
 		// Create priority queue
 		q := queue.NewJobQueue(cfg.Scanner.QueueSize)
 
-		// Create hold WebSocket client
+		// Create hold WebSocket client. Declaring the worker count is what
+		// lets the hold keep that many scans in flight for this process
+		// instead of one.
 		holdClient := client.NewHoldClient(cfg.Hold.URL, cfg.Hold.Secret, q)
+		holdClient.SetWorkers(cfg.Scanner.Workers)
 
 		// Start WebSocket connection (feeds queue)
 		go holdClient.Connect()

scanner/digest.go

@@ -0,0 +1,63 @@
+package scanner
+
+import (
+	"fmt"
+	"strings"
+)
+
+// Digest is a content digest that has been validated, and is therefore safe to
+// use both as a blob name on the wire and as a single path element on disk.
+//
+// The validation is deliberately narrow. A digest reaches the scanner from an
+// io.atcr.manifest record in a user's own PDS, which the user can write
+// directly, and the hold's dispatch guards check the hold DID, the layer count,
+// the subject and the config but never the digest format. Downstream the string
+// is joined onto the blobs directory and handed to os.Create, so anything that
+// is not exactly an algorithm and a hex string is a filesystem primitive
+// wearing a digest's clothes.
+//
+// Only sha256 is accepted. It is the only algorithm the OCI layout the scanner
+// builds uses (blobs/sha256/<hex>) and the only one stereoscope can read, so
+// anything else is unscannable however well formed it is; refusing it here
+// turns a late, retried failure into an early, permanent one.
+type Digest struct {
+	Algorithm string // always "sha256" today
+	Hex       string // lowercase hex, exactly HexLen characters
+}
+
+// SHA256 is the only digest algorithm the scanner accepts.
+const SHA256 = "sha256"
+
+// HexLen is the number of hex characters in a sha256 digest.
+const HexLen = 64
+
+// String renders the digest back into its "algorithm:hex" form.
+func (d Digest) String() string { return d.Algorithm + ":" + d.Hex }
+
+// ParseDigest validates a digest string and returns its parts.
+//
+// It accepts exactly "sha256:" followed by 64 lowercase hex characters, and
+// nothing else: no other algorithm, no uppercase, no other length, no leading
+// or trailing anything. Because the result is constrained to [0-9a-f], the Hex
+// field cannot contain a separator, a dot, or a NUL, and so cannot escape the
+// directory it is joined onto.
+func ParseDigest(digest string) (Digest, error) {
+	algorithm, hex, ok := strings.Cut(digest, ":")
+	if !ok {
+		return Digest{}, fmt.Errorf("digest %q has no algorithm prefix", digest)
+	}
+	if algorithm != SHA256 {
+		return Digest{}, fmt.Errorf("digest %q uses unsupported algorithm %q, want %s", digest, algorithm, SHA256)
+	}
+	if len(hex) != HexLen {
+		return Digest{}, fmt.Errorf("digest %q has %d hex characters, want %d", digest, len(hex), HexLen)
+	}
+	for i := 0; i < len(hex); i++ {
+		c := hex[i]
+		if (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') {
+			continue
+		}
+		return Digest{}, fmt.Errorf("digest %q is not lowercase hex", digest)
+	}
+	return Digest{Algorithm: algorithm, Hex: hex}, nil
+}

scanner/digest_test.go

@@ -0,0 +1,61 @@
+package scanner
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestParseDigestAcceptsOnlyAlgorithmAndHex(t *testing.T) {
+	valid := "sha256:" + strings.Repeat("ab", 32)
+
+	d, err := ParseDigest(valid)
+	if err != nil {
+		t.Fatalf("ParseDigest(%q) = %v, want it accepted", valid, err)
+	}
+	if d.Algorithm != "sha256" {
+		t.Errorf("Algorithm = %q, want sha256", d.Algorithm)
+	}
+	if d.Hex != strings.Repeat("ab", 32) {
+		t.Errorf("Hex = %q, want the 64 hex characters", d.Hex)
+	}
+	if got := d.String(); got != valid {
+		t.Errorf("String() = %q, want %q", got, valid)
+	}
+}
+
+// TestParseDigestRejectsEverythingElse is the security boundary: anything that
+// is not exactly "sha256:<64 lowercase hex>" must be refused before it can be
+// joined onto a filesystem path or sent to the hold as a blob name.
+func TestParseDigestRejectsEverythingElse(t *testing.T) {
+	hex64 := strings.Repeat("ab", 32)
+
+	cases := []struct {
+		name   string
+		digest string
+	}{
+		{"empty", ""},
+		{"no algorithm prefix", hex64},
+		{"path traversal in the hex", "sha256:../../../escaped-marker"},
+		{"traversal with no algorithm", "../../../escaped-marker"},
+		{"absolute path", "sha256:/etc/passwd"},
+		{"separator in the hex", "sha256:ab/cd"},
+		{"null byte", "sha256:" + hex64 + "\x00"},
+		{"unsupported algorithm", "sha512:" + strings.Repeat("cd", 64)},
+		{"uppercase hex", "sha256:" + strings.ToUpper(hex64)},
+		{"short hex", "sha256:abcd"},
+		{"long hex", "sha256:" + hex64 + "ab"},
+		{"non-hex characters", "sha256:" + strings.Repeat("zz", 32)},
+		{"empty hex", "sha256:"},
+		{"empty algorithm", ":" + hex64},
+		{"double colon", "sha256:sha256:" + hex64},
+		{"leading space", " sha256:" + hex64},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			if d, err := ParseDigest(tc.digest); err == nil {
+				t.Errorf("ParseDigest(%q) accepted it as %+v, want an error", tc.digest, d)
+			}
+		})
+	}
+}

scanner/internal/client/hold.go

@@ -3,7 +3,11 @@
 package client
 
 import (
+	"crypto/rand"
+	"crypto/sha256"
+	"encoding/hex"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"io"
 	"log/slog"
@@ -30,18 +34,54 @@ type HoldClient struct {
 	conn    *websocket.Conn
 	mu      sync.Mutex // protects conn writes
 	done    chan struct{}
+
+	// instanceID identifies this scanner process to the hold, and is sent on
+	// every connect. It is deliberately per-process and not persisted: a hold
+	// hands a reconnecting instance back the jobs its workers never stopped
+	// running, and a process that actually restarted has lost that work and
+	// must not claim it.
+	instanceID string
+
+	// workers is how many scans this process runs at once. The hold uses it as
+	// the dispatch budget for this connection; a hold that does not know the
+	// parameter ignores it and assumes one.
+	workers int
 }
 
 // NewHoldClient creates a new hold client
 func NewHoldClient(holdURL, secret string, q *queue.JobQueue) *HoldClient {
 	return &HoldClient{
-		holdURL: holdURL,
-		secret:  secret,
-		queue:   q,
-		done:    make(chan struct{}),
+		holdURL:    holdURL,
+		secret:     secret,
+		queue:      q,
+		done:       make(chan struct{}),
+		instanceID: newInstanceID(),
+		workers:    1,
 	}
 }
 
+// SetWorkers tells the hold how many scans this process runs concurrently, so
+// it can keep that many jobs in flight instead of one.
+//
+// This is a setter rather than a constructor argument on purpose: the
+// connection is entirely usable without it, and a caller that never calls it
+// gets the single-worker default the hold assumes anyway.
+func (c *HoldClient) SetWorkers(n int) {
+	if n > 0 {
+		c.workers = n
+	}
+}
+
+func newInstanceID() string {
+	b := make([]byte, 8)
+	if _, err := rand.Read(b); err != nil {
+		// The hold falls back to a per-connection identity when none is sent,
+		// which costs resumption on reconnect and nothing else.
+		return ""
+	}
+	return hex.EncodeToString(b)
+}
+
 // Connect establishes the WebSocket connection with auto-reconnect
 func (c *HoldClient) Connect() {
 	var cursor int64 = -1
@@ -91,6 +131,12 @@ func (c *HoldClient) connectOnce(cursor int64) error {
 	if cursor >= 0 {
 		q.Set("cursor", fmt.Sprintf("%d", cursor))
 	}
+	if c.instanceID != "" {
+		q.Set("instance", c.instanceID)
+	}
+	if c.workers > 0 {
+		q.Set("workers", fmt.Sprintf("%d", c.workers))
+	}
 	u.RawQuery = q.Encode()
 
 	slog.Info("Connecting to hold service", "url", u.Host)
@@ -117,62 +163,118 @@ func (c *HoldClient) connectOnce(cursor int64) error {
 			return err
 		}
 
-		var raw scanner.ScanJobRaw
-		if err := json.Unmarshal(data, &raw); err != nil {
-			slog.Error("Failed to unmarshal message", "error", err)
-			continue
-		}
-
-		if raw.Type != "job" {
-			slog.Warn("Unknown message type from hold", "type", raw.Type)
-			continue
-		}
-
-		// Parse config and layers from raw JSON
-		var config scanner.BlobDescriptor
-		if err := json.Unmarshal(raw.Config, &config); err != nil {
-			slog.Error("Failed to unmarshal config", "seq", raw.Seq, "error", err)
-			continue
-		}
-
-		var layers []scanner.BlobDescriptor
-		if err := json.Unmarshal(raw.Layers, &layers); err != nil {
-			slog.Error("Failed to unmarshal layers", "seq", raw.Seq, "error", err)
-			continue
-		}
-
-		job := &scanner.ScanJob{
-			Seq:            raw.Seq,
-			ManifestDigest: raw.ManifestDigest,
-			Repository:     raw.Repository,
-			Tag:            raw.Tag,
-			UserDID:        raw.UserDID,
-			UserHandle:     raw.UserHandle,
-			HoldDID:        raw.HoldDID,
-			HoldEndpoint:   raw.HoldEndpoint,
-			Tier:           raw.Tier,
-			Config:         config,
-			Layers:         layers,
-		}
-
-		// Send ack immediately
-		c.SendAck(job.Seq)
-
-		// Enqueue into priority queue
-		if !c.queue.Enqueue(job) {
-			slog.Warn("Queue full, sending error",
-				"seq", job.Seq,
-				"repository", job.Repository)
-			c.SendError(job.Seq, "scanner queue full")
-		}
+		c.handleFrame(data)
 	}
 }
 
+// handleFrame decodes one frame from the hold and either enqueues the job or
+// answers for it.
+//
+// The invariant, which every branch below keeps: a frame carrying a usable seq
+// is never dropped in silence. The hold writes status='assigned' for that seq
+// before it writes the frame, and its only escape from that row is an
+// acknowledgement or a verdict from us. A frame we drop instead sits assigned
+// until the five-minute ack timeout, is re-offered within thirty seconds — to
+// this same scanner, which cannot decode it this time either — and holds the
+// hold's single proactive dispatch slot the entire time. One such frame stops
+// proactive scanning for every user of that hold, permanently.
+func (c *HoldClient) handleFrame(data []byte) {
+	var raw scanner.ScanJobRaw
+	if err := json.Unmarshal(data, &raw); err != nil {
+		// encoding/json records the first type error and keeps decoding the
+		// rest of the object, so the seq that addresses the row is often still
+		// usable even when the frame as a whole was rejected.
+		c.rejectJob(raw.Seq, "malformed job frame", err)
+		return
+	}
+
+	if raw.Type != "job" {
+		// Not a job, so no row of ours is waiting on an answer. The hold sends
+		// nothing but "job" today; a future message type is better ignored by
+		// an old scanner than answered with a verdict about a job.
+		slog.Warn("Unknown message type from hold", "type", raw.Type, "seq", raw.Seq)
+		return
+	}
+
+	// Parse config and layers from raw JSON
+	var config scanner.BlobDescriptor
+	if err := json.Unmarshal(raw.Config, &config); err != nil {
+		c.rejectJob(raw.Seq, "malformed job config", err)
+		return
+	}
+
+	var layers []scanner.BlobDescriptor
+	if err := json.Unmarshal(raw.Layers, &layers); err != nil {
+		c.rejectJob(raw.Seq, "malformed job layers", err)
+		return
+	}
+
+	job := &scanner.ScanJob{
+		Seq:            raw.Seq,
+		ManifestDigest: raw.ManifestDigest,
+		Repository:     raw.Repository,
+		Tag:            raw.Tag,
+		UserDID:        raw.UserDID,
+		UserHandle:     raw.UserHandle,
+		HoldDID:        raw.HoldDID,
+		HoldEndpoint:   raw.HoldEndpoint,
+		Tier:           raw.Tier,
+		Config:         config,
+		Layers:         layers,
+	}
+
+	// Send ack immediately
+	c.SendAck(job.Seq)
+
+	// Enqueue into priority queue
+	if !c.queue.Enqueue(job) {
+		slog.Warn("Queue full, sending error",
+			"seq", job.Seq,
+			"repository", job.Repository)
+		c.SendError(job.Seq, "scanner queue full")
+	}
+}
+
+// rejectJob answers a job frame this scanner cannot decode.
+//
+// "skipped" and not "error": the hold treats a failure as transient and
+// re-queues it on the rescan interval, but a frame that does not decode will
+// not decode on the next attempt either. A skip is terminal — the hold marks
+// the row completed, writes a scan record carrying the reason, and releases
+// the manifest from its in-flight set — so the job leaves the rotation and the
+// reason reaches the user instead of vanishing into this process's log.
+//
+// A seq of zero is the one frame that cannot be answered: the hold's rows
+// start at 1, so there is no job to address. Nothing is stranded by staying
+// quiet there either, because the hold assigns the row by seq before it sends,
+// and a seq that never survived the wire is not the seq it assigned.
+func (c *HoldClient) rejectJob(seq int64, reason string, err error) {
+	if seq <= 0 {
+		slog.Error("Dropping undecodable frame with no usable seq",
+			"reason", reason, "error", err)
+		return
+	}
+
+	slog.Error("Rejecting undecodable job frame",
+		"seq", seq, "reason", reason, "error", err)
+	c.SendSkipped(seq, fmt.Sprintf("%s: %v", reason, err))
+}
+
 // SendAck sends an acknowledgement for a received job
 func (c *HoldClient) SendAck(seq int64) {
 	c.sendJSON(scanner.AckMessage{Type: "ack", Seq: seq})
 }
 
+// SendStarted tells the hold a worker has begun this scan.
+//
+// The ack this job already got was sent from the reader goroutine on receipt,
+// before the job was queued. This is the signal the hold measures its scanning
+// deadline from; a hold too old to know the message logs and ignores it, and
+// falls back to its own budget measured from dispatch.
+func (c *HoldClient) SendStarted(seq int64) {
+	c.sendJSON(scanner.StartedMessage{Type: "started", Seq: seq})
+}
+
 // SendResult sends scan results back to the hold
 func (c *HoldClient) SendResult(seq int64, result *scanner.ScanResult) {
 	msg := scanner.ResultMessage{
@@ -227,11 +329,15 @@ func (c *HoldClient) Close() {
 
 // GetBlobPresignedURL gets a presigned download URL from the hold service.
 // If secret is non-empty, it is sent as a Bearer token for private hold access.
-func GetBlobPresignedURL(holdEndpoint, holdDID, digest, secret string) (string, error) {
+//
+// The digest is a scanner.Digest rather than a string so that only a validated
+// "sha256:<hex>" can ever be asked for. Callers parse once, at the boundary,
+// and the same value then names the blob on the wire and the file on disk.
+func GetBlobPresignedURL(holdEndpoint, holdDID string, digest scanner.Digest, secret string) (string, error) {
 	reqURL := fmt.Sprintf("%s/xrpc/com.atproto.sync.getBlob?did=%s&cid=%s&method=GET",
 		holdEndpoint,
 		url.QueryEscape(holdDID),
-		url.QueryEscape(digest))
+		url.QueryEscape(digest.String()))
 
 	req, err := http.NewRequest("GET", reqURL, nil)
 	if err != nil {
@@ -262,27 +368,93 @@ func GetBlobPresignedURL(holdEndpoint, holdDID, digest, secret string) (string,
 	return result.URL, nil
 }
 
-// DownloadBlob downloads a blob from a presigned URL to a local file
-func DownloadBlob(presignedURL, destPath string) error {
+// Errors a blob download can fail with that no retry can fix. Both are
+// decided by bytes the scanner has already seen, so a caller mapping them to a
+// permanent skip is not guessing.
+var (
+	// ErrBlobCorrupt means the bytes that arrived are not the bytes the
+	// descriptor described. It deliberately does not distinguish "the digest
+	// is wrong" from "the size is wrong": both fields come from the same
+	// user-writable manifest record, so a disagreement condemns the whole
+	// descriptor rather than proving anything about one field.
+	ErrBlobCorrupt = errors.New("digest mismatch")
+
+	// ErrBlobTooLarge means the transfer would take the job past the byte
+	// ceiling the configuration set.
+	ErrBlobTooLarge = errors.New("image too large")
+)
+
+// BlobExpectation is what a download has to turn out to be.
+type BlobExpectation struct {
+	// Digest is the validated digest the bytes must hash to. This is the
+	// authority: it is checked always, and it is what makes serving the wrong
+	// object detectable.
+	Digest scanner.Digest
+
+	// DeclaredSize is the descriptor's size field, corroborating evidence
+	// rather than authority. Zero or negative means the record claimed no
+	// size, in which case the digest alone decides.
+	DeclaredSize int64
+
+	// MaxBytes caps this transfer. Negative means unbounded. Zero is a real
+	// ceiling of zero bytes: it is what remains when an earlier blob in the
+	// same job has spent the whole budget.
+	MaxBytes int64
+}
+
+// DownloadBlob downloads a blob from a presigned URL to a local file and
+// verifies it against want, returning the number of bytes written.
+//
+// Verification happens while streaming, not by re-reading the file: the hash
+// and the byte count are both accumulated by the same io.Copy that writes the
+// blob, so a scan can never catalog bytes that were not checked, and a blob
+// over the ceiling stops costing bandwidth one byte past it.
+func DownloadBlob(presignedURL, destPath string, want BlobExpectation) (int64, error) {
 	resp, err := httpClient.Get(presignedURL)
 	if err != nil {
-		return fmt.Errorf("failed to download blob: %w", err)
+		return 0, fmt.Errorf("failed to download blob: %w", err)
 	}
 	defer resp.Body.Close()
 
 	if resp.StatusCode != http.StatusOK {
-		return fmt.Errorf("download returned status %d", resp.StatusCode)
+		return 0, fmt.Errorf("download returned status %d", resp.StatusCode)
 	}
 
 	out, err := os.Create(destPath)
 	if err != nil {
-		return fmt.Errorf("failed to create file: %w", err)
+		return 0, fmt.Errorf("failed to create file: %w", err)
 	}
 	defer out.Close()
 
-	if _, err := io.Copy(out, resp.Body); err != nil {
-		return fmt.Errorf("failed to write blob: %w", err)
+	// One byte past the ceiling is all the proof needed that the ceiling was
+	// broken, and reading no further is the point: the claimed sizes that let
+	// the job through the pre-check are the attacker's to choose.
+	body := io.Reader(resp.Body)
+	if want.MaxBytes >= 0 {
+		body = io.LimitReader(resp.Body, want.MaxBytes+1)
 	}
 
-	return nil
+	hasher := sha256.New()
+	n, err := io.Copy(io.MultiWriter(out, hasher), body)
+	if err != nil {
+		return n, fmt.Errorf("failed to write blob: %w", err)
+	}
+
+	if want.MaxBytes >= 0 && n > want.MaxBytes {
+		return n, fmt.Errorf("%w: blob %s exceeds the %d bytes left in the job's budget",
+			ErrBlobTooLarge, want.Digest, want.MaxBytes)
+	}
+
+	gotHex := hex.EncodeToString(hasher.Sum(nil))
+	sizeDisagrees := want.DeclaredSize > 0 && n != want.DeclaredSize
+	if gotHex != want.Digest.Hex || sizeDisagrees {
+		claimedSize := "no declared size"
+		if want.DeclaredSize > 0 {
+			claimedSize = fmt.Sprintf("%d bytes", want.DeclaredSize)
+		}
+		return n, fmt.Errorf("%w: descriptor claims %s and %s, received %d bytes hashing to %s:%s",
+			ErrBlobCorrupt, want.Digest, claimedSize, n, want.Digest.Algorithm, gotHex)
+	}
+
+	return n, nil
 }

scanner/internal/config/config_edge_test.go

@@ -0,0 +1,270 @@
+package config
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// writeYAML writes a scanner config file containing the two required keys plus
+// whatever extra YAML the case supplies, and returns its path.
+func writeYAML(t *testing.T, extra string) string {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "scanner.yaml")
+	body := "hold:\n  url: \"ws://hold.example\"\n  secret: \"s3cret\"\n" + extra
+	if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
+		t.Fatalf("write config: %v", err)
+	}
+	return path
+}
+
+// TestLoadConfigRejectsOnlyURLAndSecret is the positive control: these are the
+// only two fields LoadConfig validates, and they do fail loudly.
+func TestLoadConfigRejectsOnlyURLAndSecret(t *testing.T) {
+	clearScannerEnv(t)
+
+	if _, err := LoadConfig(""); err == nil {
+		t.Error("empty config was accepted; hold.url should be required")
+	}
+
+	path := filepath.Join(t.TempDir(), "scanner.yaml")
+	if err := os.WriteFile(path, []byte("hold:\n  url: \"ws://hold.example\"\n"), 0o600); err != nil {
+		t.Fatalf("write config: %v", err)
+	}
+	if _, err := LoadConfig(path); err == nil {
+		t.Error("config without hold.secret was accepted; it should be required")
+	}
+}
+
+// TestLoadConfigAcceptsSilentlyBrokenValues is the finding: every one of these
+// loads clean, logs nothing at boot, and produces a scanner that connects,
+// reports healthy on /healthz, and then either fails or strands every job it
+// is given. LoadConfig validates two strings for emptiness and nothing else.
+//
+// Each case names the runtime consequence; the behavioural proof for the ones
+// that reach the pipeline is in scanner/internal/e2e/protocol_test.go.
+func TestLoadConfigAcceptsSilentlyBrokenValues(t *testing.T) {
+	cases := []struct {
+		name     string
+		yaml     string
+		env      map[string]string
+		check    func(*testing.T, *Config)
+		breakage string
+	}{
+		{
+			name:     "workers: 0",
+			yaml:     "scanner:\n  workers: 0\n",
+			breakage: "WorkerPool.Start launches no goroutines; jobs are acked and then sit in the queue until the hold's 10 minute processing timeout fails them",
+			check: func(t *testing.T, c *Config) {
+				if c.Scanner.Workers != 0 {
+					t.Errorf("workers = %d, want 0", c.Scanner.Workers)
+				}
+			},
+		},
+		{
+			name:     "workers: negative",
+			yaml:     "scanner:\n  workers: -4\n",
+			breakage: "same as workers: 0 — the for loop simply does not run",
+			check: func(t *testing.T, c *Config) {
+				if c.Scanner.Workers != -4 {
+					t.Errorf("workers = %d, want -4", c.Scanner.Workers)
+				}
+			},
+		},
+		{
+			name:     "queue_size: 0",
+			yaml:     "scanner:\n  queue_size: 0\n",
+			breakage: "every Enqueue returns false, so every job is acked and then failed with \"scanner queue full\", which the hold retries forever",
+			check: func(t *testing.T, c *Config) {
+				if c.Scanner.QueueSize != 0 {
+					t.Errorf("queue_size = %d, want 0", c.Scanner.QueueSize)
+				}
+			},
+		},
+		{
+			name:     "queue_size: negative",
+			yaml:     "scanner:\n  queue_size: -1\n",
+			breakage: "same as queue_size: 0; Len() >= maxSize is true for an empty queue",
+			check: func(t *testing.T, c *Config) {
+				if c.Scanner.QueueSize != -1 {
+					t.Errorf("queue_size = %d, want -1", c.Scanner.QueueSize)
+				}
+			},
+		},
+		{
+			name:     "tmp_dir: empty",
+			yaml:     "vuln:\n  tmp_dir: \"\"\n",
+			breakage: "WorkerPool.Start skips the TMPDIR export, and processJob's ensureDir(\"\") fails, so every job errors retryably",
+			check: func(t *testing.T, c *Config) {
+				if c.Vuln.TmpDir != "" {
+					t.Errorf("tmp_dir = %q, want empty", c.Vuln.TmpDir)
+				}
+			},
+		},
+		{
+			name:     "db_path: empty with vuln enabled",
+			yaml:     "vuln:\n  enabled: true\n  db_path: \"\"\n",
+			breakage: "initializeVulnDatabase failure is only logged (\"scanning will be disabled\"), but nothing actually disables it: each scan then fails in scanVulnerabilities",
+			check: func(t *testing.T, c *Config) {
+				if c.Vuln.DBPath != "" {
+					t.Errorf("db_path = %q, want empty", c.Vuln.DBPath)
+				}
+			},
+		},
+		{
+			name:     "hold.url is not a URL",
+			env:      map[string]string{"SCANNER_HOLD_URL": "hold.example:8080"},
+			breakage: "url.Parse accepts it, the ws/wss switch does not match, and Connect redials a bad address every 5 seconds forever",
+			check: func(t *testing.T, c *Config) {
+				if c.Hold.URL != "hold.example:8080" {
+					t.Errorf("hold.url = %q", c.Hold.URL)
+				}
+			},
+		},
+		{
+			name:     "max_image_size: negative",
+			yaml:     "vuln:\n  max_image_size: -1\n",
+			breakage: "the guard is `if MaxImageSize > 0`, so a negative ceiling silently means no ceiling at all",
+			check: func(t *testing.T, c *Config) {
+				if c.Vuln.MaxImageSize != -1 {
+					t.Errorf("max_image_size = %d, want -1", c.Vuln.MaxImageSize)
+				}
+			},
+		},
+		{
+			name:     "log_level: nonsense",
+			yaml:     "log_level: \"verbose\"\n",
+			breakage: "InitLoggerWithShipper's default branch quietly falls back to info, so a typo in the level is invisible",
+			check: func(t *testing.T, c *Config) {
+				if c.LogLevel != "verbose" {
+					t.Errorf("log_level = %q", c.LogLevel)
+				}
+			},
+		},
+		{
+			name:     "server.addr is unbindable",
+			yaml:     "server:\n  addr: \"nope:not-a-port\"\n",
+			breakage: "ListenAndServe fails in a goroutine and only logs; the scanner keeps running with no health endpoint",
+			check: func(t *testing.T, c *Config) {
+				if c.Server.Addr != "nope:not-a-port" {
+					t.Errorf("server.addr = %q", c.Server.Addr)
+				}
+			},
+		},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			clearScannerEnv(t)
+			for k, v := range tc.env {
+				t.Setenv(k, v)
+			}
+			cfg, err := LoadConfig(writeYAML(t, tc.yaml))
+			if err != nil {
+				t.Fatalf("LoadConfig rejected %s — this finding is fixed: %v", tc.name, err)
+			}
+			tc.check(t, cfg)
+			t.Logf("accepted; runtime consequence: %s", tc.breakage)
+		})
+	}
+}
+
+// TestEmptyEnvVarDoesNotOverrideDefaults pins the one thing that saves
+// SCANNER_VULN_TMP_DIR="" from being the same hazard as the YAML key: Viper is
+// constructed without AllowEmptyEnv, so an env var set to the empty string is
+// treated as unset. The YAML path is the reachable one.
+//
+// It is worth pinning because it is accidental rather than intended: enabling
+// AllowEmptyEnv anywhere in pkg/config would silently make every empty-string
+// env var in every deployment start overriding defaults.
+func TestEmptyEnvVarDoesNotOverrideDefaults(t *testing.T) {
+	clearScannerEnv(t)
+	t.Setenv("SCANNER_HOLD_URL", "ws://hold.example")
+	t.Setenv("SCANNER_HOLD_SECRET", "s3cret")
+	t.Setenv("SCANNER_VULN_TMP_DIR", "")
+
+	cfg, err := LoadConfig("")
+	if err != nil {
+		t.Fatalf("LoadConfig: %v", err)
+	}
+	if cfg.Vuln.TmpDir != "/var/lib/atcr-scanner/tmp" {
+		t.Errorf("tmp_dir = %q; an empty env var now overrides the default, which makes "+
+			"SCANNER_VULN_TMP_DIR= as dangerous as the YAML key", cfg.Vuln.TmpDir)
+	}
+}
+
+// TestDefaultsAreSane guards the values the deployment relies on when nothing
+// is set, so a regression in setScannerDefaults shows up here rather than in
+// production.
+func TestDefaultsAreSane(t *testing.T) {
+	clearScannerEnv(t)
+	cfg := DefaultConfig()
+
+	if cfg.Scanner.Workers < 1 {
+		t.Errorf("default workers = %d, want at least 1", cfg.Scanner.Workers)
+	}
+	if cfg.Scanner.QueueSize < 1 {
+		t.Errorf("default queue_size = %d, want at least 1", cfg.Scanner.QueueSize)
+	}
+	if cfg.Vuln.TmpDir == "" {
+		t.Error("default tmp_dir is empty; extraction would land on /tmp")
+	}
+	if cfg.Vuln.TmpDir == "/tmp" {
+		t.Error("default tmp_dir is /tmp, which the code comments call out as a small tmpfs")
+	}
+}
+
+// clearScannerEnv removes SCANNER_* and LOG_SHIPPER_* from the environment for
+// the duration of a test, so a developer's shell cannot change what the
+// configuration loader sees.
+func clearScannerEnv(t *testing.T) {
+	t.Helper()
+	for _, kv := range os.Environ() {
+		key, _, ok := strings.Cut(kv, "=")
+		if !ok {
+			continue
+		}
+		if !strings.HasPrefix(key, "SCANNER_") && !strings.HasPrefix(key, "LOG_SHIPPER_") {
+			continue
+		}
+		// Setenv registers the restore; Unsetenv is what the test actually
+		// needs, since Viper treats an empty value as unset anyway.
+		t.Setenv(key, "")
+		os.Unsetenv(key)
+	}
+}
+
+// TestMalformedYAMLIsIgnoredEntirely is the loudest silent failure in the
+// configuration path, and it is not scanner-specific: NewViper in
+// pkg/config/viper.go discards the result of ReadInConfig, so a config file
+// that does not parse is skipped whole. Nothing logs, nothing fails; the
+// process boots on defaults plus environment.
+//
+// In the shipped deployment hold.url and hold.secret come from the
+// environment, so even LoadConfig's two checks pass. A typo in scanner.yaml
+// therefore silently reverts workers, queue_size, tmp_dir, db_path,
+// max_image_size, log level and log shipping to their defaults on a scanner
+// that reports itself healthy.
+func TestMalformedYAMLIsIgnoredEntirely(t *testing.T) {
+	clearScannerEnv(t)
+	t.Setenv("SCANNER_HOLD_URL", "ws://hold.example")
+	t.Setenv("SCANNER_HOLD_SECRET", "s3cret")
+
+	path := filepath.Join(t.TempDir(), "scanner.yaml")
+	body := "scanner:\n  workers: 8\n  workers: 9\nvuln:\n  tmp_dir: \"/data/scanner/tmp\"\n"
+	if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
+		t.Fatalf("write config: %v", err)
+	}
+
+	cfg, err := LoadConfig(path)
+	if err != nil {
+		t.Fatalf("LoadConfig rejected an unparseable file — this finding is fixed: %v", err)
+	}
+	if cfg.Vuln.TmpDir != "/data/scanner/tmp" {
+		t.Logf("unparseable YAML silently ignored: tmp_dir fell back to %q, workers to %d",
+			cfg.Vuln.TmpDir, cfg.Scanner.Workers)
+	} else {
+		t.Error("the file parsed after all; pick a different malformed body for this test")
+	}
+}

scanner/internal/e2e/blob_edge_test.go

@@ -0,0 +1,898 @@
+package e2e
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/json"
+	"fmt"
+	"math/rand"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"strings"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	scanner "atcr.io/scanner"
+	"atcr.io/scanner/internal/client"
+	"atcr.io/scanner/internal/config"
+	"atcr.io/scanner/internal/mockhold"
+	"atcr.io/scanner/internal/queue"
+	"atcr.io/scanner/internal/scan"
+)
+
+// This file covers the blob download and OCI layout construction stages:
+// what the scanner does when a blob is gone, truncated, corrupt, slow, or
+// named by a digest a user made up.
+//
+// Historical note on the shape of these scenarios. worker.go used to
+// dereference result.Summary unconditionally, and Summary is only populated
+// when cfg.Vuln.Enabled; with Grype off in the harness, a *successful* scan
+// panicked and took the test binary with it, so no e2e test could observe
+// anything but a terminal "error" or "skipped". That is fixed, and
+// TestZeroSizeLayerProducesACleanScan now runs a success through to the end.
+// Scenarios below that would otherwise succeed (a duplicated layer, a lying
+// size) still break one blob deliberately, but now only to keep the assertion
+// on the download stage rather than on Syft's verdict.
+//
+// A second move since: the scenarios that pinned the *absence* of digest
+// validation and content verification (the traversal digest, unverified bytes,
+// a short layer, malformed digests, the claimed-bytes ceiling) now live in
+// blob_integrity_test.go, where they assert the checks that replaced them.
+// What is left here is the transport: what happens when a blob is gone, slow,
+// redirected, duplicated, or served with a body the transport itself rejects.
+
+// --- local harness ---------------------------------------------------------
+
+// startScanner is Start with the mock hold supplied by the caller. Start builds
+// its own Hold and so cannot install a presign or blob-response hook, and every
+// transport fault here needs one. Everything else matches Start, including the
+// TMPDIR restore and the shortened cooldown.
+func startScanner(t *testing.T, hold *mockhold.Hold, opts ...Option) *Harness {
+	t.Helper()
+	t.Cleanup(hold.Close)
+
+	cfg := config.DefaultConfig()
+	cfg.Hold.URL = hold.URL()
+	cfg.Hold.Secret = testSecret
+	cfg.Scanner.Workers = 1
+	cfg.Vuln.Enabled = false
+	cfg.Vuln.TmpDir = t.TempDir()
+
+	origTmpDir, hadTmpDir := os.LookupEnv("TMPDIR")
+	t.Cleanup(func() {
+		if hadTmpDir {
+			os.Setenv("TMPDIR", origTmpDir)
+			return
+		}
+		os.Unsetenv("TMPDIR")
+	})
+	for _, opt := range opts {
+		opt(cfg)
+	}
+
+	restoreCooldown := scan.JobCooldown
+	scan.JobCooldown = testJobCooldown
+
+	q := queue.NewJobQueue(cfg.Scanner.QueueSize)
+	c := client.NewHoldClient(cfg.Hold.URL, cfg.Hold.Secret, q)
+	pool := scan.NewWorkerPool(cfg, q, c)
+
+	ctx, cancel := context.WithCancel(context.Background())
+	pool.Start(ctx)
+	go c.Connect()
+
+	// Teardown waits for the worker to actually exit before restoring
+	// JobCooldown. Start does not, and the race detector sees it: a worker from
+	// the finished test is still reading the package variable while the next
+	// test's cleanup writes it. Waiting also keeps one test's in-flight scan
+	// from writing into the next test's temp directory.
+	t.Cleanup(func() {
+		cancel()
+		c.Close()
+		q.Close()
+		pool.Wait()
+		scan.JobCooldown = restoreCooldown
+	})
+
+	if err := hold.WaitForScanner(10 * time.Second); err != nil {
+		t.Fatalf("scanner never connected: %v", err)
+	}
+	return &Harness{Hold: hold, Queue: q, Client: c, Cfg: cfg}
+}
+
+// newHold builds a mock hold with the harness secret plus any hooks.
+func newHold(blobs mockhold.BlobSource, opts ...mockhold.Option) *mockhold.Hold {
+	return mockhold.New(blobs, append([]mockhold.Option{mockhold.WithSecret(testSecret)}, opts...)...)
+}
+
+// --- synthetic blobs -------------------------------------------------------
+
+const (
+	layerType  = "application/vnd.oci.image.layer.v1.tar+gzip"
+	configType = "application/vnd.oci.image.config.v1+json"
+)
+
+func digestOf(b []byte) string {
+	return fmt.Sprintf("sha256:%x", sha256.Sum256(b))
+}
+
+// gzTar builds a one-file gzipped tar, the shape of a real image layer.
+func gzTar(t *testing.T, name string, content []byte) []byte {
+	t.Helper()
+	var buf bytes.Buffer
+	zw := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(zw)
+	if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0644, Size: int64(len(content))}); err != nil {
+		t.Fatalf("tar header: %v", err)
+	}
+	if _, err := tw.Write(content); err != nil {
+		t.Fatalf("tar write: %v", err)
+	}
+	if err := tw.Close(); err != nil {
+		t.Fatalf("tar close: %v", err)
+	}
+	if err := zw.Close(); err != nil {
+		t.Fatalf("gzip close: %v", err)
+	}
+	return buf.Bytes()
+}
+
+// unparseableConfig is a config blob that is not JSON at all. Scenarios that
+// would otherwise scan clean through use it to stop the pipeline at Syft, so
+// the assertion stays on the download stage (see the file comment).
+func unparseableConfig() []byte { return []byte("{ this is not a container config") }
+
+// validConfig is a well-formed OCI image config, for scenarios where the
+// *layer* is the thing under test and the config must not be what fails.
+func validConfig(t *testing.T, diffIDs ...string) []byte {
+	t.Helper()
+	cfg := map[string]any{
+		"architecture": "amd64",
+		"os":           "linux",
+		"config":       map[string]any{},
+		"rootfs":       map[string]any{"type": "layers", "diff_ids": diffIDs},
+	}
+	b, err := json.Marshal(cfg)
+	if err != nil {
+		t.Fatalf("marshal config: %v", err)
+	}
+	return b
+}
+
+func desc(digest string, size int64, mediaType string) scanner.BlobDescriptor {
+	return scanner.BlobDescriptor{Digest: digest, Size: size, MediaType: mediaType}
+}
+
+// jobFor assembles a scan job from raw descriptors. HoldEndpoint is left empty
+// so SendJob points it at the mock.
+func jobFor(cfg scanner.BlobDescriptor, layers ...scanner.BlobDescriptor) *scanner.ScanJob {
+	return &scanner.ScanJob{
+		ManifestDigest: "sha256:" + strings.Repeat("ab", 32),
+		Repository:     "edge-case",
+		Tag:            "latest",
+		UserDID:        "did:plc:testuser",
+		UserHandle:     "test.example",
+		HoldDID:        "did:web:hold.test",
+		Tier:           "deckhand",
+		Config:         cfg,
+		Layers:         layers,
+	}
+}
+
+// assertNoLeakedScanDirs checks the scanner's tmp dir is empty. The OCI layout
+// lives in tmpDir/scan-*, and stereoscope's extraction lands in the same
+// directory because WorkerPool.Start points TMPDIR at it, so a leak on any
+// error path shows up here.
+func assertNoLeakedScanDirs(t *testing.T, h *Harness) {
+	t.Helper()
+	entries, err := os.ReadDir(h.Cfg.Vuln.TmpDir)
+	if err != nil {
+		t.Fatalf("read tmp dir: %v", err)
+	}
+	for _, e := range entries {
+		t.Errorf("leftover in scanner tmp dir after failure: %s", e.Name())
+	}
+}
+
+func blobDigests(h *Harness) []string {
+	var out []string
+	for _, r := range h.Hold.BlobRequests() {
+		out = append(out, r.Digest)
+	}
+	return out
+}
+
+func countOf(list []string, want string) int {
+	n := 0
+	for _, s := range list {
+		if s == want {
+			n++
+		}
+	}
+	return n
+}
+
+// --- blob transport --------------------------------------------------------
+
+// TestMissingBlobIsRetryableError covers a layer garbage collected out from
+// under a queued job: getBlob answers 404 and there is nothing to download,
+// ever. The scanner reports "error", the hold records the scan as failed, and
+// failed records are re-offered by the stale-scan loop on every pass. Nothing
+// about the outcome can change, so this is an unbounded retry loop for a
+// permanently unscannable manifest.
+func TestMissingBlobIsRetryableError(t *testing.T) {
+	layer := gzTar(t, "usr/bin/app", []byte("hello"))
+	layerDigest := digestOf(layer)
+	cfgBytes := unparseableConfig()
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().Add(cfgDigest, cfgBytes)
+
+	var hold *mockhold.Hold
+	hold = newHold(blobs, mockhold.WithPresignHook(func(digest string) (string, bool) {
+		if digest == layerDigest {
+			return "", false // garbage collected
+		}
+		return hold.URL() + "/blobs/" + mockhold.DigestHex(digest), true
+	}))
+	h := startScanner(t, hold)
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc(layerDigest, int64(len(layer)), layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "error" {
+		t.Fatalf("want the current retryable-error behaviour, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if !strings.Contains(msg.Error, "404") {
+		t.Errorf("error does not mention the 404: %q", msg.Error)
+	}
+	t.Logf("missing blob -> retryable error: %q", msg.Error)
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestGetBlobNon200 points the job at a hold endpoint whose getBlob is broken
+// in two ways a real hold can be: a 5xx, and a 200 carrying something that is
+// not the expected JSON. Both surface as retryable errors, which is right for
+// the 5xx and wrong-but-harmless for the malformed body.
+func TestGetBlobNon200(t *testing.T) {
+	cases := []struct {
+		name    string
+		handler http.HandlerFunc
+		want    string
+	}{
+		{
+			name: "503 from getBlob",
+			handler: func(w http.ResponseWriter, r *http.Request) {
+				http.Error(w, "hold is restarting", http.StatusServiceUnavailable)
+			},
+			want: "status 503",
+		},
+		{
+			name: "200 with a non-JSON body",
+			handler: func(w http.ResponseWriter, r *http.Request) {
+				w.Write([]byte("<html>proxy error</html>"))
+			},
+			want: "failed to decode response",
+		},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			broken := httptest.NewServer(http.HandlerFunc(tc.handler))
+			defer broken.Close()
+
+			h := startScanner(t, newHold(mockhold.NewMemory()))
+
+			cfgBytes := unparseableConfig()
+			job := jobFor(desc(digestOf(cfgBytes), int64(len(cfgBytes)), configType))
+			job.HoldEndpoint = broken.URL
+
+			seq, err := h.Hold.SendJob(job)
+			if err != nil {
+				t.Fatalf("send job: %v", err)
+			}
+			msg := h.AwaitTerminal(t, seq, 30*time.Second)
+			if msg.Type != "error" {
+				t.Fatalf("want error, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+			}
+			if !strings.Contains(msg.Error, tc.want) {
+				t.Errorf("error %q does not mention %q", msg.Error, tc.want)
+			}
+			assertNoLeakedScanDirs(t, h)
+		})
+	}
+}
+
+// TestExpiredPresignedURL stands in for an S3 URL whose 15 minute window closed
+// while the job sat in the queue: getBlob succeeds, the download 403s.
+//
+// The assertion worth keeping is the second one. DownloadBlob reports only the
+// status code and drops the body, so the S3 <Error><Code>AccessDenied</Code>
+// explanation never reaches the hold's scan record: an operator sees
+// "download returned status 403" with no indication of which blob, from where,
+// or why. GetBlobPresignedURL includes the body in its error; DownloadBlob does
+// not.
+func TestExpiredPresignedURL(t *testing.T) {
+	s3 := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusForbidden)
+		w.Write([]byte(`<Error><Code>AccessDenied</Code><Message>Request has expired</Message></Error>`))
+	}))
+	defer s3.Close()
+
+	hold := newHold(mockhold.NewMemory(), mockhold.WithPresignHook(func(digest string) (string, bool) {
+		return s3.URL + "/bucket/blob", true
+	}))
+	h := startScanner(t, hold)
+
+	cfgBytes := unparseableConfig()
+	seq, err := h.Hold.SendJob(jobFor(desc(digestOf(cfgBytes), int64(len(cfgBytes)), configType)))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "error" {
+		t.Fatalf("want error, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if !strings.Contains(msg.Error, "403") {
+		t.Errorf("error does not mention the 403: %q", msg.Error)
+	}
+	if strings.Contains(msg.Error, "AccessDenied") || strings.Contains(msg.Error, "expired") {
+		t.Errorf("DownloadBlob now includes the response body; update this test and the finding: %q", msg.Error)
+	}
+	t.Logf("expired presigned URL -> %q (body discarded)", msg.Error)
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestTruncatedBodyIsDetected sends fewer bytes than the declared
+// Content-Length. Go's client turns that into an unexpected EOF on the body, so
+// io.Copy does notice and the blob never reaches Syft. This is the transport
+// fault the code handles correctly, and it is here so a regression is visible.
+func TestTruncatedBodyIsDetected(t *testing.T) {
+	layer := gzTar(t, "usr/bin/app", bytes.Repeat([]byte("payload"), 4096))
+	layerDigest := digestOf(layer)
+	cfgBytes := unparseableConfig()
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().Add(cfgDigest, cfgBytes)
+	hold := newHold(blobs, mockhold.WithBlobResponseHook(func(w http.ResponseWriter, r *http.Request, digest string) bool {
+		if digest != mockhold.DigestHex(layerDigest) {
+			return false
+		}
+		w.Header().Set("Content-Length", fmt.Sprintf("%d", len(layer)))
+		w.WriteHeader(http.StatusOK)
+		w.Write(layer[:len(layer)/4])
+		if f, ok := w.(http.Flusher); ok {
+			f.Flush()
+		}
+		return true // hang up short of Content-Length
+	}))
+	h := startScanner(t, hold)
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc(layerDigest, int64(len(layer)), layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "error" {
+		t.Fatalf("want error, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if !strings.Contains(msg.Error, "failed to write blob") {
+		t.Errorf("truncation was not caught by io.Copy; error was %q", msg.Error)
+	}
+	t.Logf("short body vs Content-Length -> %q", msg.Error)
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestStalledDownloadHasNoShortDeadline holds a response body open and shows
+// the scanner simply waits. client.httpClient's only bound is a 5 minute
+// per-request Timeout, and DownloadBlob builds no request context, so
+// cancelling the worker context (shutdown, SIGTERM) does not abort a download
+// in flight either.
+//
+// The stall here is deliberately short. The point is not to sit out the real
+// timeout but to show there is no shorter one: a 2 second stall costs the
+// worker 2 seconds, and a hold that accepts the connection and then says
+// nothing costs it five minutes per blob, times the number of layers.
+func TestStalledDownloadHasNoShortDeadline(t *testing.T) {
+	const stall = 2 * time.Second
+
+	cfgBytes := unparseableConfig()
+	cfgDigest := digestOf(cfgBytes)
+
+	hold := newHold(mockhold.NewMemory(), mockhold.WithBlobResponseHook(func(w http.ResponseWriter, r *http.Request, digest string) bool {
+		w.Header().Set("Content-Length", "4096")
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte("partial"))
+		if f, ok := w.(http.Flusher); ok {
+			f.Flush()
+		}
+		time.Sleep(stall)
+		return true
+	}))
+	h := startScanner(t, hold)
+
+	started := time.Now()
+	seq, err := h.Hold.SendJob(jobFor(desc(cfgDigest, int64(len(cfgBytes)), configType)))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 60*time.Second)
+	elapsed := time.Since(started)
+	if msg.Type != "error" {
+		t.Fatalf("want error, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if elapsed < stall {
+		t.Errorf("scanner gave up after %s, before the stall ended; a shorter deadline now exists", elapsed)
+	}
+	t.Logf("stalled body held the worker for %s, then: %q", elapsed.Round(100*time.Millisecond), msg.Error)
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestDownloadFollowsRedirectToAnotherHost shows the scanner will fetch blob
+// bytes from wherever the getBlob response, or any redirect from it, points.
+// DownloadBlob sends no credentials, so nothing leaks; what it means is that
+// the hold (or anything able to answer as the hold, or an open redirect on the
+// presigned URL's host) chooses which server the scanner talks to. That is
+// acceptable now for the reason B-10 gave: the bytes are verified against the
+// digest on arrival, so where they came from does not decide what they are.
+func TestDownloadFollowsRedirectToAnotherHost(t *testing.T) {
+	cfgBytes := unparseableConfig()
+	cfgDigest := digestOf(cfgBytes)
+
+	var elsewhereHits atomic.Int32
+	var sawAuth atomic.Bool
+	elsewhere := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		elsewhereHits.Add(1)
+		if r.Header.Get("Authorization") != "" {
+			sawAuth.Store(true)
+		}
+		w.Write(cfgBytes)
+	}))
+	defer elsewhere.Close()
+
+	hold := newHold(mockhold.NewMemory(), mockhold.WithBlobResponseHook(func(w http.ResponseWriter, r *http.Request, digest string) bool {
+		http.Redirect(w, r, elsewhere.URL+"/somewhere-else", http.StatusFound)
+		return true
+	}))
+	h := startScanner(t, hold)
+
+	seq, err := h.Hold.SendJob(jobFor(desc(cfgDigest, int64(len(cfgBytes)), configType)))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "error" {
+		t.Fatalf("want error (the redirected bytes are an unparseable config), got %s: %s%s",
+			msg.Type, msg.Error, msg.Reason)
+	}
+	if elsewhereHits.Load() == 0 {
+		t.Error("redirect to a different host was not followed")
+	}
+	if sawAuth.Load() {
+		t.Error("Authorization header was sent to the redirect target")
+	}
+	t.Logf("cross-host redirect followed %d time(s), no credentials attached", elsewhereHits.Load())
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestRedirectChainStopsAtTen pins the only bound on a redirect chain: Go's
+// default client policy. The scanner sets no policy of its own, so a hold
+// pointing a blob at a redirect loop costs eleven round trips per blob before
+// erroring, and the error names the URL rather than the blob.
+func TestRedirectChainStopsAtTen(t *testing.T) {
+	cfgBytes := unparseableConfig()
+	cfgDigest := digestOf(cfgBytes)
+
+	var hold *mockhold.Hold
+	hold = newHold(mockhold.NewMemory(), mockhold.WithBlobResponseHook(func(w http.ResponseWriter, r *http.Request, digest string) bool {
+		http.Redirect(w, r, hold.URL()+"/blobs/"+digest, http.StatusFound)
+		return true
+	}))
+	h := startScanner(t, hold)
+
+	seq, err := h.Hold.SendJob(jobFor(desc(cfgDigest, int64(len(cfgBytes)), configType)))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "error" {
+		t.Fatalf("want error, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if !strings.Contains(msg.Error, "redirect") {
+		t.Errorf("error does not mention redirects: %q", msg.Error)
+	}
+	if n := len(h.Hold.BlobRequests()); n < 10 {
+		t.Errorf("redirect loop cost %d requests, expected the client's 10-hop limit", n)
+	}
+	t.Logf("redirect loop: %d requests, then %q", len(h.Hold.BlobRequests()), msg.Error)
+	assertNoLeakedScanDirs(t, h)
+}
+
+// --- layout construction ---------------------------------------------------
+
+// TestDuplicateLayerIsDownloadedTwice feeds the same layer digest twice, which
+// a hand-written manifest record can do. Both copies are fetched over the
+// network and written to the same path, and the layout's manifest lists the
+// layer twice.
+//
+// The config is deliberately unparseable so the run ends at Syft rather than
+// in a scan verdict; the download accounting is what this asserts.
+func TestDuplicateLayerIsDownloadedTwice(t *testing.T) {
+	layer := gzTar(t, "usr/bin/app", []byte("hello"))
+	layerDigest := digestOf(layer)
+	cfgBytes := unparseableConfig()
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().
+		Add(cfgDigest, cfgBytes).
+		Add(layerDigest, layer)
+	h := startScanner(t, newHold(blobs))
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc(layerDigest, int64(len(layer)), layerType),
+		desc(layerDigest, int64(len(layer)), layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "error" {
+		t.Fatalf("want error, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if n := countOf(blobDigests(h), mockhold.DigestHex(layerDigest)); n != 2 {
+		t.Errorf("duplicate layer fetched %d times, want the current behaviour of 2", n)
+	}
+	t.Logf("duplicate layer digest fetched %d times", countOf(blobDigests(h), mockhold.DigestHex(layerDigest)))
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestUnparseableConfigBlobIsRetryableError states outright what several
+// scenarios above lean on: a config blob whose bytes are not JSON fails inside
+// generateSBOM and comes back as "error". That is a permanent property of the
+// image, so the hold's stale-scan loop will offer it again on every pass and
+// get the same answer forever. The same is true of a layer whose bytes are not
+// the tar its media type claims (TestDigestIsNeverVerified).
+func TestUnparseableConfigBlobIsRetryableError(t *testing.T) {
+	cfgBytes := unparseableConfig()
+	cfgDigest := digestOf(cfgBytes)
+	layer := gzTar(t, "usr/bin/app", []byte("hello"))
+	layerDigest := digestOf(layer)
+
+	blobs := mockhold.NewMemory().
+		Add(cfgDigest, cfgBytes).
+		Add(layerDigest, layer)
+	h := startScanner(t, newHold(blobs))
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc(layerDigest, int64(len(layer)), layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 60*time.Second)
+	if msg.Type != "error" {
+		t.Fatalf("want the current retryable-error behaviour, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	t.Logf("unparseable config -> retryable error: %q", msg.Error)
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestEmptyLayerDigestIsSkippedNotFetched pins the one malformed-digest case
+// the layout builder does handle: an empty digest among valid ones is dropped
+// from both the download loop and the layout manifest.
+//
+// The surviving layer is served as garbage so the run ends in an error, which
+// keeps the assertion on which blobs were fetched.
+func TestEmptyLayerDigestIsSkipped(t *testing.T) {
+	layer := []byte("not actually a gzip stream")
+	layerDigest := digestOf(layer)
+	cfgBytes := unparseableConfig()
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().
+		Add(cfgDigest, cfgBytes).
+		Add(layerDigest, layer)
+	h := startScanner(t, newHold(blobs))
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc("", 0, layerType), // empty digest
+		desc(layerDigest, int64(len(layer)), layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "error" {
+		t.Fatalf("want error, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if n := len(h.Hold.BlobRequests()); n != 2 {
+		t.Errorf("fetched %d blobs, want config + one real layer: %v", n, blobDigests(h))
+	}
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestZeroSizeLayerIsFetched covers a descriptor claiming size 0 whose blob is
+// genuinely empty. The size field is not consulted at download time, so the
+// blob is fetched and an empty file is written into the layout.
+func TestZeroSizeLayerIsFetched(t *testing.T) {
+	empty := []byte{}
+	emptyDigest := digestOf(empty)
+	cfgBytes := unparseableConfig()
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().
+		Add(cfgDigest, cfgBytes).
+		Add(emptyDigest, empty)
+	h := startScanner(t, newHold(blobs))
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc(emptyDigest, 0, layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "error" {
+		t.Fatalf("want error, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if n := countOf(blobDigests(h), mockhold.DigestHex(emptyDigest)); n != 1 {
+		t.Errorf("zero-size layer fetched %d times, want 1", n)
+	}
+	t.Logf("zero-size layer -> %q", msg.Error)
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestZeroSizeLayerProducesACleanScan is the same image with a config the
+// scanner can actually read, and it is the cheapest end-to-end proof that a
+// successful scan no longer kills the process.
+//
+// stereoscope accepts a zero-byte layer as a valid, empty layer, so the scan
+// succeeds with an SBOM containing no packages and, with Grype off, no
+// vulnerability summary. That combination used to reach the unconditional
+// result.Summary dereference in worker.go and panic the test binary. It is
+// worth keeping for a second reason too: an image whose layer descriptors are
+// all zero-size blobs comes back as a clean scan rather than a failure, and
+// nothing in the scanner distinguishes "no vulnerabilities" from "nothing was
+// there".
+func TestZeroSizeLayerProducesACleanScan(t *testing.T) {
+	empty := []byte{}
+	emptyDigest := digestOf(empty)
+	cfgBytes := validConfig(t, emptyDigest)
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().
+		Add(cfgDigest, cfgBytes).
+		Add(emptyDigest, empty)
+	h := startScanner(t, newHold(blobs))
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc(emptyDigest, 0, layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 60*time.Second)
+	if msg.Type != "result" {
+		t.Fatalf("want result, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if msg.Summary != nil {
+		t.Errorf("vuln scanning is disabled but the result carried a summary: %+v", msg.Summary)
+	}
+	t.Logf("zero-byte layer scanned clean, SBOM %d bytes", len(msg.SBOM))
+}
+
+// TestManyLayersAreFetchedSequentially runs an image as wide as the widest
+// manifest in the corpus (19 layers) through the download stage. Each layer is
+// one sequential HTTP round trip with its own 5 minute ceiling, so the
+// worst-case time for this one job is not five minutes but ninety-five, and
+// nothing caps it.
+//
+// The width comes from the corpus but the blobs are synthesised, because every
+// blob is now verified against its digest and no fixture can serve bytes that
+// hash to a real registry's digests. The layer count is what this measures.
+func TestManyLayersAreFetchedSequentially(t *testing.T) {
+	width := widestCorpusWidth(t)
+	if width < 10 {
+		t.Skipf("corpus has no wide manifest; widest is %d layers", width)
+	}
+
+	blobs := mockhold.NewMemory()
+	layers := make([]scanner.BlobDescriptor, 0, width)
+	diffIDs := make([]string, 0, width)
+	for i := 0; i < width; i++ {
+		b := gzTar(t, fmt.Sprintf("usr/bin/app%d", i), []byte(fmt.Sprintf("layer %d", i)))
+		d := digestOf(b)
+		blobs.Add(d, b)
+		layers = append(layers, desc(d, int64(len(b)), layerType))
+		diffIDs = append(diffIDs, d)
+	}
+	cfgBytes := validConfig(t, diffIDs...)
+	cfgDigest := digestOf(cfgBytes)
+	blobs.Add(cfgDigest, cfgBytes)
+
+	h := startScanner(t, newHold(blobs))
+
+	send := func() int64 {
+		t.Helper()
+		seq, err := h.Hold.SendJob(jobFor(desc(cfgDigest, int64(len(cfgBytes)), configType), layers...))
+		if err != nil {
+			t.Fatalf("send job: %v", err)
+		}
+		return seq
+	}
+
+	msg := h.AwaitTerminal(t, send(), 60*time.Second)
+	if msg.Type != "result" {
+		t.Fatalf("want result, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	reqs := h.Hold.BlobRequests()
+	if len(reqs) != width+1 {
+		t.Errorf("fetched %d blobs, want %d (config + %d layers)", len(reqs), width+1, width)
+	}
+	// Sequential, not concurrent: no two fetches overlap.
+	for i := 1; i < len(reqs); i++ {
+		if reqs[i].At.Before(reqs[i-1].At) {
+			t.Errorf("blob requests are out of order at %d", i)
+		}
+	}
+	t.Logf("%d-layer manifest: %d sequential fetches", width, len(reqs))
+
+	// Per-layer resource accounting: downloadBlob opens a file and a response
+	// body per layer. Run the same job twice more and require the process's
+	// file descriptor count to hold steady, which is what rules out a per-layer
+	// handle leak (a leak would grow by ~20 per run).
+	if fds, ok := openFDs(); ok {
+		for i := 0; i < 2; i++ {
+			h.AwaitTerminal(t, send(), 60*time.Second)
+		}
+		time.Sleep(200 * time.Millisecond)
+		after, _ := openFDs()
+		if after > fds+8 {
+			t.Errorf("file descriptors grew from %d to %d across two more %d-layer scans",
+				fds, after, width)
+		}
+		t.Logf("file descriptors: %d after one scan, %d after three", fds, after)
+	}
+
+	assertNoLeakedScanDirs(t, h)
+}
+
+// widestCorpusWidth reports the layer count of the widest manifest in the
+// corpus, so the shape under test stays tied to what a hold really holds.
+func widestCorpusWidth(t *testing.T) int {
+	t.Helper()
+	all, err := mockhold.Corpus()
+	if err != nil {
+		t.Fatalf("load corpus: %v", err)
+	}
+	widest := 0
+	for _, m := range all {
+		if len(m.Layers) > widest {
+			widest = len(m.Layers)
+		}
+	}
+	return widest
+}
+
+// openFDs returns the number of open file descriptors, and whether the count
+// was available (it is not on platforms without /proc).
+func openFDs() (int, bool) {
+	entries, err := os.ReadDir("/proc/self/fd")
+	if err != nil {
+		return 0, false
+	}
+	return len(entries), true
+}
+
+// --- size guard ------------------------------------------------------------
+
+// TestMaxImageSizeZeroDisablesTheGuard confirms the documented "0 = no limit"
+// reading of the config. The guard now spends a budget down as bytes arrive,
+// so the case worth pinning is that a zero limit spends nothing: an image far
+// larger than any of the other scenarios here transfers in full.
+//
+// The descriptors are honest, because a manifest that lies about its size is
+// now rejected by the descriptor check rather than by the ceiling, which would
+// prove nothing about the ceiling.
+func TestMaxImageSizeZeroDisablesTheGuard(t *testing.T) {
+	noise := make([]byte, 64*1024)
+	rand.New(rand.NewSource(2)).Read(noise)
+	big := gzTar(t, "usr/bin/app", noise)
+	bigDigest := digestOf(big)
+
+	// Unparseable on purpose: the run ends at Syft, so the assertion stays on
+	// the download stage rather than on a scan verdict.
+	cfgBytes := unparseableConfig()
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().
+		Add(cfgDigest, cfgBytes).
+		Add(bigDigest, big)
+	h := startScanner(t, newHold(blobs), WithMaxImageSize(0))
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc(bigDigest, int64(len(big)), layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "error" {
+		t.Fatalf("want error from Syft, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if strings.Contains(msg.Error, "too large") || strings.Contains(msg.Reason, "too large") {
+		t.Fatalf("MaxImageSize=0 rejected a job; the 0-means-unlimited contract changed: %q%q",
+			msg.Error, msg.Reason)
+	}
+	if n := len(h.Hold.BlobRequests()); n != 2 {
+		t.Errorf("fetched %d blobs, want config + layer with no ceiling in force", n)
+	}
+	assertNoLeakedScanDirs(t, h)
+}
+
+// --- cleanup ---------------------------------------------------------------
+
+// TestTmpDirIsCleanAfterAFailureInsideSyft covers the one cleanup path that is
+// not a direct return from buildOCILayout: the layout is built, downloads
+// succeed, and generateSBOM fails. processJob's deferred cleanup is responsible
+// for the directory, and stereoscope is responsible for its own extraction
+// scratch space under the same TMPDIR.
+func TestTmpDirIsCleanAfterAFailureInsideSyft(t *testing.T) {
+	layer := gzTar(t, "usr/bin/app", []byte("hello"))
+	layerDigest := digestOf(layer)
+	cfgBytes := unparseableConfig()
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().
+		Add(cfgDigest, cfgBytes).
+		Add(layerDigest, layer)
+	h := startScanner(t, newHold(blobs))
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc(layerDigest, int64(len(layer)), layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 60*time.Second)
+	if msg.Type != "error" {
+		t.Fatalf("want error, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if !strings.Contains(msg.Error, "SBOM") && !strings.Contains(msg.Error, "OCI image") {
+		t.Logf("failure came from somewhere other than Syft: %q", msg.Error)
+	}
+	assertNoLeakedScanDirs(t, h)
+}

scanner/internal/e2e/blob_integrity_test.go

@@ -0,0 +1,422 @@
+package e2e
+
+import (
+	"bytes"
+	"fmt"
+	"math/rand"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"atcr.io/scanner/internal/mockhold"
+)
+
+// This file states what the scanner must do with blob bytes and blob names it
+// did not choose. Both come from an io.atcr.manifest record in a user's own
+// PDS, so both are attacker-controlled text, and the hold's dispatch guards
+// check neither.
+//
+// Two invariants, and everything here is one of them:
+//
+//  1. A digest names a blob and a file. It is only ever "sha256:<64 hex>";
+//     anything else is rejected before it reaches a path or the network.
+//  2. The bytes that land on disk are the bytes the digest names, in the
+//     amount the descriptor claims, within the budget the config allows.
+//
+// A violation of either is a permanent property of the record or of the stored
+// blob, so it must come back as "skipped" (which the hold records once) rather
+// than "error" (which the stale-scan loop re-offers forever).
+
+// --- invariant 1: a digest is a name, never a path -------------------------
+
+// TestTraversalDigestNeverBecomesAPath is the security case. A layer digest of
+// "sha256:../../../escaped-marker" used to be split on ":" and filepath.Joined
+// onto blobs/sha256, which is exactly three levels below the scanner's tmp
+// dir, so the downloaded bytes landed in the tmp dir itself: outside the scan
+// directory, beyond the reach of cleanup, and over whatever file was already
+// there (os.Create truncates).
+//
+// The escape is aimed at the harness's own t.TempDir so the test proves the
+// boundary holds without writing anywhere real.
+func TestTraversalDigestNeverBecomesAPath(t *testing.T) {
+	payload := []byte("bytes the scanner must never place here")
+	traversal := "sha256:../../../escaped-marker"
+
+	cfgBytes := validConfig(t)
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().
+		Add(cfgDigest, cfgBytes).
+		Add("payload", payload)
+
+	// A hold willing to serve a 200 for the traversal digest, which is the
+	// precondition the old write needed.
+	var hold *mockhold.Hold
+	hold = newHold(blobs, mockhold.WithPresignHook(func(digest string) (string, bool) {
+		if digest == traversal {
+			return hold.URL() + "/blobs/payload", true
+		}
+		return hold.URL() + "/blobs/" + mockhold.DigestHex(digest), true
+	}))
+	h := startScanner(t, hold)
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc(traversal, int64(len(payload)), layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "skipped" {
+		t.Fatalf("want skipped (a malformed digest can never succeed), got %s: %s%s",
+			msg.Type, msg.Error, msg.Reason)
+	}
+	if !strings.Contains(msg.Reason, "digest") {
+		t.Errorf("skip reason %q does not name the digest as the problem", msg.Reason)
+	}
+
+	// Nothing may have been written outside the scan directory. The tmp dir is
+	// the escape's target, and after a scan it must hold nothing at all.
+	escaped := filepath.Join(h.Cfg.Vuln.TmpDir, "escaped-marker")
+	if got, err := os.ReadFile(escaped); err == nil {
+		os.Remove(escaped)
+		t.Fatalf("the traversal still wrote %d bytes to %s", len(got), escaped)
+	}
+	assertNoLeakedScanDirs(t, h)
+
+	// And the rejection happens before any network call: a digest that cannot
+	// name a file must not be sent to the hold as a blob name either.
+	if n := len(h.Hold.BlobRequests()); n != 0 {
+		t.Errorf("scanner made %d blob requests for a job it must reject up front: %v",
+			n, blobDigests(h))
+	}
+}
+
+// TestMalformedDigestsAreSkippedBeforeAnyDownload walks the digest shapes a
+// hand-written manifest record can carry. Every one is permanently
+// unscannable, so every one is a skip, and none of them costs a round trip.
+func TestMalformedDigestsAreSkippedBeforeAnyDownload(t *testing.T) {
+	layer := gzTar(t, "usr/bin/app", []byte("hello"))
+
+	cases := []struct {
+		name   string
+		digest string
+	}{
+		{"no algorithm prefix", strings.Repeat("ef", 32)},
+		{"unsupported algorithm", "sha512:" + strings.Repeat("cd", 64)},
+		{"uppercase hex", "sha256:" + strings.ToUpper(strings.Repeat("ab", 32))},
+		{"truncated hex", "sha256:abcd"},
+		{"path traversal", "sha256:../../../escaped-marker"},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			cfgBytes := validConfig(t)
+			cfgDigest := digestOf(cfgBytes)
+
+			blobs := mockhold.NewMemory().
+				Add(cfgDigest, cfgBytes).
+				Add(tc.digest, layer)
+			h := startScanner(t, newHold(blobs))
+
+			seq, err := h.Hold.SendJob(jobFor(
+				desc(cfgDigest, int64(len(cfgBytes)), configType),
+				desc(tc.digest, int64(len(layer)), layerType),
+			))
+			if err != nil {
+				t.Fatalf("send job: %v", err)
+			}
+			msg := h.AwaitTerminal(t, seq, 30*time.Second)
+			if msg.Type != "skipped" {
+				t.Fatalf("want skipped, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+			}
+			if n := len(h.Hold.BlobRequests()); n != 0 {
+				t.Errorf("made %d blob requests before rejecting a malformed digest: %v",
+					n, blobDigests(h))
+			}
+			t.Logf("%s -> skipped: %s", tc.name, msg.Reason)
+			assertNoLeakedScanDirs(t, h)
+		})
+	}
+}
+
+// TestMalformedConfigDigestIsSkipped covers the same validation on the config
+// descriptor, which reaches downloadBlob by a separate call site.
+func TestMalformedConfigDigestIsSkipped(t *testing.T) {
+	h := startScanner(t, newHold(mockhold.NewMemory()))
+
+	seq, err := h.Hold.SendJob(jobFor(desc("sha256:../../../escaped-config", 10, configType)))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "skipped" {
+		t.Fatalf("want skipped, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if got, err := os.ReadFile(filepath.Join(h.Cfg.Vuln.TmpDir, "escaped-config")); err == nil {
+		os.Remove(filepath.Join(h.Cfg.Vuln.TmpDir, "escaped-config"))
+		t.Fatalf("config traversal wrote %d bytes outside the scan directory", len(got))
+	}
+	assertNoLeakedScanDirs(t, h)
+}
+
+// --- invariant 2: the bytes are the bytes ----------------------------------
+
+// TestDigestMismatchIsDetectedAndSkipped serves bytes that do not hash to the
+// digest naming them, which is what a mixed-up S3 key, a poisoned cache or a
+// compromised BYOS hold produces. A compliant OCI client verifies on pull and
+// refuses these bytes; the scanner is the one consumer in the system that used
+// to trust them, and then vouched for them in a scan record.
+//
+// The failure must name the digest, not surface as whatever stereoscope makes
+// of a stream it cannot parse.
+func TestDigestMismatchIsDetectedAndSkipped(t *testing.T) {
+	layer := gzTar(t, "usr/bin/app", []byte("real content"))
+	layerDigest := digestOf(layer)
+	imposter := []byte("this is not a gzip stream at all")
+
+	cfgBytes := validConfig(t, digestOf(layer))
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().
+		Add(cfgDigest, cfgBytes).
+		Add(layerDigest, imposter) // wrong bytes under the right name
+	h := startScanner(t, newHold(blobs))
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc(layerDigest, int64(len(imposter)), layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "skipped" {
+		t.Fatalf("want skipped, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if !strings.Contains(msg.Reason, "digest mismatch") {
+		t.Errorf("skip reason %q does not say the bytes failed their digest", msg.Reason)
+	}
+	if !strings.Contains(msg.Reason, mockhold.DigestHex(layerDigest)[:16]) {
+		t.Errorf("skip reason %q does not name the offending blob", msg.Reason)
+	}
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestConfigDigestMismatchIsDetectedAndSkipped is the same check on the config
+// blob, whose download is a separate call site from the layer loop.
+func TestConfigDigestMismatchIsDetectedAndSkipped(t *testing.T) {
+	cfgBytes := validConfig(t)
+	cfgDigest := digestOf(cfgBytes)
+	imposter := []byte(`{"architecture":"amd64","os":"linux","rootfs":{"type":"layers"}}`)
+
+	blobs := mockhold.NewMemory().Add(cfgDigest, imposter)
+	h := startScanner(t, newHold(blobs))
+
+	seq, err := h.Hold.SendJob(jobFor(desc(cfgDigest, int64(len(imposter)), configType)))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "skipped" {
+		t.Fatalf("want skipped, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if !strings.Contains(msg.Reason, "digest mismatch") {
+		t.Errorf("skip reason %q does not say the bytes failed their digest", msg.Reason)
+	}
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestShortLayerIsRejectedByTheDescriptorCheck is the truncation the transport
+// cannot see: a body that is internally consistent (Content-Length matches
+// what is sent) but shorter than the size the descriptor declares.
+//
+// Both the digest and the size disagree with the bytes here, which is the
+// point: descriptor.Size is attacker-supplied too, so the right verdict is
+// that the descriptor as a whole does not describe the blob, and the message
+// says so rather than picking one field to blame.
+func TestShortLayerIsRejectedByTheDescriptorCheck(t *testing.T) {
+	layer := gzTar(t, "usr/bin/app", bytes.Repeat([]byte("payload"), 4096))
+	layerDigest := digestOf(layer)
+	short := layer[:len(layer)/4]
+
+	cfgBytes := validConfig(t, digestOf(layer))
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().
+		Add(cfgDigest, cfgBytes).
+		Add(layerDigest, short)
+	h := startScanner(t, newHold(blobs))
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc(layerDigest, int64(len(layer)), layerType), // declares the full size
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "skipped" {
+		t.Fatalf("want skipped, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if !strings.Contains(msg.Reason, fmt.Sprintf("%d", len(short))) {
+		t.Errorf("skip reason %q does not report the %d bytes actually received", msg.Reason, len(short))
+	}
+	if !strings.Contains(msg.Reason, fmt.Sprintf("%d", len(layer))) {
+		t.Errorf("skip reason %q does not report the %d bytes claimed", msg.Reason, len(layer))
+	}
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestHonestBlobsStillScan is the control. Verification must not reject an
+// image whose bytes are what they say they are: config, layer, digests and
+// sizes all agree, and the scan runs through to a result.
+func TestHonestBlobsStillScan(t *testing.T) {
+	layer := gzTar(t, "usr/bin/app", []byte("hello from a real layer"))
+	layerDigest := digestOf(layer)
+	cfgBytes := validConfig(t, digestOf(layer))
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().
+		Add(cfgDigest, cfgBytes).
+		Add(layerDigest, layer)
+	h := startScanner(t, newHold(blobs))
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc(layerDigest, int64(len(layer)), layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 60*time.Second)
+	if msg.Type != "result" {
+		t.Fatalf("verification rejected an honest image: %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestUndeclaredSizeIsAcceptedWhenTheDigestHolds pins the one relaxation in
+// the size check. A descriptor whose Size is zero has not claimed a length, so
+// the digest alone decides. Real records do carry sizes; this keeps a record
+// that omits one scannable rather than making the weaker field authoritative.
+func TestUndeclaredSizeIsAcceptedWhenTheDigestHolds(t *testing.T) {
+	layer := gzTar(t, "usr/bin/app", []byte("hello"))
+	layerDigest := digestOf(layer)
+	cfgBytes := validConfig(t, digestOf(layer))
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().
+		Add(cfgDigest, cfgBytes).
+		Add(layerDigest, layer)
+	h := startScanner(t, newHold(blobs))
+
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, 0, configType),
+		desc(layerDigest, 0, layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+	msg := h.AwaitTerminal(t, seq, 60*time.Second)
+	if msg.Type != "result" {
+		t.Fatalf("a size-less descriptor with correct bytes was rejected: %s: %s%s",
+			msg.Type, msg.Error, msg.Reason)
+	}
+	assertNoLeakedScanDirs(t, h)
+}
+
+// --- the size budget is measured, not claimed ------------------------------
+
+// TestMaxImageSizeCountsTransferredBytes closes the ceiling bypass. The guard
+// summed BlobDescriptor.Size, which comes from the same user-writable record
+// as the digests, so a manifest claiming one byte per blob passed any limit
+// and the scanner then wrote the real bytes to its tmp volume: 65,696 bytes
+// measured against a 1,024 byte limit. The layer below declares one byte and
+// the hold offers 65,696.
+//
+// The claimed-size pre-check stays (it is still worth refusing an honestly
+// large image before a byte moves), but the budget is now enforced against
+// what actually arrives.
+func TestMaxImageSizeCountsTransferredBytes(t *testing.T) {
+	const limit = 1024
+
+	noise := make([]byte, 64*1024)
+	rand.New(rand.NewSource(1)).Read(noise)
+	big := gzTar(t, "usr/bin/app", noise)
+	bigDigest := digestOf(big)
+	cfgBytes := validConfig(t, digestOf(big))
+	cfgDigest := digestOf(cfgBytes)
+
+	var served atomic.Int64
+	blobs := mockhold.NewMemory().
+		Add(cfgDigest, cfgBytes).
+		Add(bigDigest, big)
+
+	hold := newHold(blobs, mockhold.WithBlobResponseHook(func(w http.ResponseWriter, r *http.Request, digest string) bool {
+		if digest != mockhold.DigestHex(bigDigest) {
+			return false
+		}
+		w.Header().Set("Content-Length", fmt.Sprintf("%d", len(big)))
+		n, _ := w.Write(big)
+		served.Add(int64(n))
+		return true
+	}))
+	h := startScanner(t, hold, WithMaxImageSize(limit))
+
+	// The layer lies about its size, the way a hand-written record can. The
+	// config is honest, so the job reaches the download stage and the budget
+	// is what has to stop it rather than the descriptor check.
+	seq, err := h.Hold.SendJob(jobFor(
+		desc(cfgDigest, int64(len(cfgBytes)), configType),
+		desc(bigDigest, 1, layerType),
+	))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "skipped" {
+		t.Fatalf("want skipped (an image over the ceiling can never be scanned), got %s: %s%s",
+			msg.Type, msg.Error, msg.Reason)
+	}
+	if !strings.Contains(msg.Reason, "too large") {
+		t.Errorf("skip reason %q does not name the size ceiling", msg.Reason)
+	}
+	t.Logf("layer declared 1 byte against a %d byte limit, hold offered %d: %s",
+		limit, served.Load(), msg.Reason)
+	assertNoLeakedScanDirs(t, h)
+}
+
+// TestHonestlyOversizedImageIsSkipped keeps the cheap pre-check honest: a
+// record that declares its real, over-limit size is refused before a byte is
+// downloaded, and refused permanently.
+func TestHonestlyOversizedImageIsSkipped(t *testing.T) {
+	cfgBytes := validConfig(t)
+	cfgDigest := digestOf(cfgBytes)
+
+	blobs := mockhold.NewMemory().Add(cfgDigest, cfgBytes)
+	h := startScanner(t, newHold(blobs), WithMaxImageSize(1024))
+
+	seq, err := h.Hold.SendJob(jobFor(desc(cfgDigest, 1<<20, configType)))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "skipped" {
+		t.Fatalf("want skipped, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if n := len(h.Hold.BlobRequests()); n != 0 {
+		t.Errorf("downloaded %d blobs for an image refused by the pre-check", n)
+	}
+	assertNoLeakedScanDirs(t, h)
+}

scanner/internal/e2e/harness.go

@@ -0,0 +1,218 @@
+// Package e2e wires the real scanner components (WebSocket client, priority
+// queue, worker pool) to a mock hold, so scan scenarios run through the same
+// code path production uses.
+//
+// What this can and cannot cover is worth stating plainly. It drives the
+// scanner through hold behaviour: job dispatch, disconnects, blob faults,
+// artifact shapes. It cannot cover the hold's own state machine (the
+// pending/assigned/processing rows, the ack timeout, the stale-scan loop),
+// because a mock hold has no rows. That half belongs to the ScanBroadcaster
+// tests in pkg/hold/pds, and the two must be kept honest against each other:
+// whenever a scenario here encodes an assumption about what the real hold
+// does on reconnect, a hold-side test should pin that assumption.
+package e2e
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"atcr.io/scanner/internal/client"
+	"atcr.io/scanner/internal/config"
+	"atcr.io/scanner/internal/mockhold"
+	"atcr.io/scanner/internal/queue"
+	"atcr.io/scanner/internal/scan"
+)
+
+const testSecret = "test-scanner-secret"
+
+// Harness holds a running scanner attached to a mock hold.
+type Harness struct {
+	Hold   *mockhold.Hold
+	Queue  *queue.JobQueue
+	Client *client.HoldClient
+	Cfg    *config.Config
+
+	pool *scan.WorkerPool
+}
+
+// Option customizes the harness before the scanner starts.
+type Option func(*config.Config)
+
+// WithWorkers sets the worker pool size. The default of 1 matches the shipped
+// config default and keeps job ordering deterministic.
+func WithWorkers(n int) Option {
+	return func(c *config.Config) { c.Scanner.Workers = n }
+}
+
+// WithMaxImageSize sets the compressed-size ceiling checked before download.
+func WithMaxImageSize(n int64) Option {
+	return func(c *config.Config) { c.Vuln.MaxImageSize = n }
+}
+
+// VulnDBEnv gates every scenario that runs Grype against the real
+// vulnerability database. It follows the ATCR_SCANNER_PERF precedent: a clean
+// checkout runs the ordinary suite offline and in the same time it always did,
+// and the expensive coverage is opt-in.
+const VulnDBEnv = "ATCR_SCANNER_VULNDB"
+
+// VulnDBDir is where internal/mockhold/testdata/fetch-vulndb.sh caches the
+// Grype database. It is gitignored: ~2 GB of SQLite.
+func VulnDBDir() string {
+	return filepath.Join("..", "mockhold", "testdata", "vulndb")
+}
+
+// WithVulnDB turns Grype on and points it at the cached database, skipping the
+// test when either the env gate is unset or the database has not been fetched
+// — the same shape layoutFor uses for blob fixtures, for the same reason:
+// this fixture is too large to commit and too slow to build implicitly.
+//
+// The database is a package-level global in scan guarded by an RWMutex, so the
+// first scenario in a test binary pays the load (tens of seconds, and the
+// mmap'd file is ~2 GB) and every one after it reuses the provider. That is
+// per test binary, not per package: the stubbed-loader tests in
+// internal/scan/vulndb_refresh_test.go link their own binary and their own
+// copy of these globals, so nothing here reaches them and they stay offline.
+//
+// Note that db_path is deliberately the shared cache rather than a t.TempDir:
+// a per-test copy would re-download 2 GB every run. The consequence is that a
+// scenario running against a database more than 14 days old will refresh it
+// in-process (Grype's MaxAllowedBuiltAge), which is slow but correct. Run
+// fetch-vulndb.sh to refresh it out of band.
+func WithVulnDB(t *testing.T) Option {
+	t.Helper()
+
+	if os.Getenv(VulnDBEnv) != "1" {
+		t.Skipf("set %s=1 to run scenarios against the real Grype database", VulnDBEnv)
+	}
+	dir := VulnDBDir()
+	if _, err := os.Stat(filepath.Join(dir, "6", "import.json")); err != nil {
+		t.Skipf("no Grype database at %s; run scanner/internal/mockhold/testdata/fetch-vulndb.sh", dir)
+	}
+	abs, err := filepath.Abs(dir)
+	if err != nil {
+		t.Fatalf("resolve vulndb path: %v", err)
+	}
+
+	return func(c *config.Config) {
+		c.Vuln.Enabled = true
+		c.Vuln.DBPath = abs
+	}
+}
+
+// testJobCooldown replaces the production 10s inter-job pause. Scenarios here
+// run several jobs through one worker, and the real cooldown would dominate
+// their runtime entirely. A scenario that genuinely depends on the production
+// pause should restore it explicitly and say why.
+const testJobCooldown = 10 * time.Millisecond
+
+// Start brings up a mock hold serving blobs, then a real scanner connected to
+// it. Everything is torn down via t.Cleanup.
+//
+// Grype is disabled by default. Enabling it would pull a multi-hundred-MB
+// vulnerability database on first run and make results move as the upstream
+// feed changes; the pipeline still runs blob download plus a real Syft
+// catalog, which is what these scenarios exercise. Database refresh behaviour
+// is covered separately by stubbing loadVulnDB in the scan package, and the
+// matcher itself by the opt-in scenarios in vulnreport_test.go, which pass
+// WithVulnDB to turn Grype on against a locally cached database.
+func Start(t *testing.T, blobs mockhold.BlobSource, opts ...Option) *Harness {
+	t.Helper()
+
+	hold := mockhold.New(blobs, mockhold.WithSecret(testSecret))
+	t.Cleanup(hold.Close)
+
+	cfg := config.DefaultConfig()
+	cfg.Hold.URL = hold.URL()
+	cfg.Hold.Secret = testSecret
+	cfg.Scanner.Workers = 1
+	cfg.Vuln.Enabled = false
+	cfg.Vuln.TmpDir = t.TempDir()
+
+	// WorkerPool.Start exports TMPDIR process-wide and deliberately never
+	// restores it, which is right in production (Grype's database download and
+	// stereoscope's extraction must not land on a small tmpfs) but leaks
+	// between tests: the next test's t.TempDir() would resolve against this
+	// test's directory, which cleanup has already removed. Restore it here.
+	// These tests must not run in parallel for the same reason.
+	origTmpDir, hadTmpDir := os.LookupEnv("TMPDIR")
+	t.Cleanup(func() {
+		if hadTmpDir {
+			os.Setenv("TMPDIR", origTmpDir)
+			return
+		}
+		os.Unsetenv("TMPDIR")
+	})
+	for _, opt := range opts {
+		opt(cfg)
+	}
+
+	restoreCooldown := scan.JobCooldown
+	scan.JobCooldown = testJobCooldown
+
+	q := queue.NewJobQueue(cfg.Scanner.QueueSize)
+	c := client.NewHoldClient(cfg.Hold.URL, cfg.Hold.Secret, q)
+	pool := scan.NewWorkerPool(cfg, q, c)
+
+	ctx, cancel := context.WithCancel(context.Background())
+	pool.Start(ctx)
+	go c.Connect()
+
+	t.Cleanup(func() {
+		// Order matters. Workers read scan.JobCooldown on every loop
+		// iteration, so restoring it before they have exited is a data race
+		// that -race reports against worker.go. Cancelling the context and
+		// closing the queue is what releases them (Dequeue returns nil once
+		// closed, and the cooldown select wakes on ctx.Done), so join them
+		// before touching the global back.
+		cancel()
+		c.Close()
+		q.Close()
+		pool.Wait()
+		scan.JobCooldown = restoreCooldown
+	})
+
+	if err := hold.WaitForScanner(10 * time.Second); err != nil {
+		t.Fatalf("scanner never connected: %v", err)
+	}
+
+	return &Harness{Hold: hold, Queue: q, Client: c, Cfg: cfg, pool: pool}
+}
+
+// AwaitTerminal waits for the scanner's final word on a job: a result, an
+// error, or a skip. Returning whichever arrived (rather than asserting a type)
+// lets a test report what actually happened, which matters most for the
+// scenarios where the current behaviour is the thing under examination.
+func (h *Harness) AwaitTerminal(t *testing.T, seq int64, timeout time.Duration) mockhold.Message {
+	t.Helper()
+	msg, err := h.Hold.WaitForMessage(func(m mockhold.Message) bool {
+		return m.Seq == seq && (m.Type == "result" || m.Type == "error" || m.Type == "skipped")
+	}, timeout)
+	if err != nil {
+		t.Fatalf("no terminal message for seq %d within %s; transcript: %s",
+			seq, timeout, describe(h.Hold.Transcript()))
+	}
+	return msg
+}
+
+func describe(msgs []mockhold.Message) string {
+	if len(msgs) == 0 {
+		return "(empty)"
+	}
+	out := ""
+	for _, m := range msgs {
+		if out != "" {
+			out += ", "
+		}
+		out += m.Type
+		if m.Error != "" {
+			out += "(" + m.Error + ")"
+		}
+		if m.Reason != "" {
+			out += "(" + m.Reason + ")"
+		}
+	}
+	return out
+}

scanner/internal/e2e/pipeline_test.go

@@ -0,0 +1,223 @@
+package e2e
+
+import (
+	"os"
+	"path/filepath"
+	"slices"
+	"testing"
+	"time"
+
+	"atcr.io/scanner/internal/mockhold"
+)
+
+// layoutFor returns an OCI layout source for a fixture pulled by
+// testdata/fetch-blobs.sh, skipping the test when it is absent. Blob fixtures
+// are gitignored because they are megabytes of real container layers, so a
+// clean checkout runs every descriptor-only scenario and skips only these.
+func layoutFor(t *testing.T, name string) mockhold.BlobSource {
+	t.Helper()
+	dir := filepath.Join("..", "mockhold", "testdata", "blobs", name)
+	if _, err := os.Stat(filepath.Join(dir, "oci-layout")); err != nil {
+		t.Skipf("fixture %q not present; run scanner/internal/mockhold/testdata/fetch-blobs.sh", name)
+	}
+	return mockhold.NewOCILayout(dir)
+}
+
+// corpusOne returns the single corpus manifest with the given digest.
+func corpusOne(t *testing.T, digest string) mockhold.Manifest {
+	t.Helper()
+	all, err := mockhold.Corpus()
+	if err != nil {
+		t.Fatalf("load corpus: %v", err)
+	}
+	for _, m := range all {
+		if m.Digest == digest {
+			return m
+		}
+	}
+	t.Fatalf("digest %s not in corpus", digest)
+	return mockhold.Manifest{}
+}
+
+// hsmOperator is the fixture image fetch-blobs.sh pulls: a real single-layer
+// image whose descriptors are in the corpus and whose bytes are in the layout.
+const hsmOperator = "sha256:1cfa4e2b09e127b9c4ed43578d3f3c18e7d44ea47b9ea98475c0cbe9086525f8"
+
+// TestScanRealImage runs the whole pipeline against a real image: the scanner
+// downloads the config and layer through the mock's getBlob indirection,
+// reassembles an OCI layout, and hands it to Syft.
+//
+// The assertion that matters most is not "an SBOM came back" but that the
+// blobs the scanner asked for are exactly the ones the manifest declares. That
+// is the join between two independently sourced halves of the fixture: the
+// descriptors come from the PDS, the bytes from skopeo, and nothing keeps them
+// in agreement except the digests being content addresses.
+func TestScanRealImage(t *testing.T) {
+	// The harness disables Grype to avoid a multi-hundred-MB database
+	// download, so this scan completes with no vulnerability summary. That is
+	// the case worker.go used to dereference unconditionally, taking the whole
+	// test binary down with it; a successful summary-less scan must now log
+	// and return like any other.
+	blobs := layoutFor(t, "hsm-secrets-operator")
+	h := Start(t, blobs)
+
+	manifest := corpusOne(t, hsmOperator)
+	seq, err := h.Hold.SendJob(manifest.Job())
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 3*time.Minute)
+	if msg.Type != "result" {
+		t.Fatalf("want result, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if msg.SBOM == "" {
+		t.Error("result carried no SBOM")
+	}
+	// Grype is off, so there is no summary. A nil summary means "not scanned
+	// for vulnerabilities", which is not the same as "scanned, found none" —
+	// the scanner must not invent a zeroed one to fill the hole.
+	if msg.Summary != nil {
+		t.Errorf("vuln scanning is disabled but the result carried a summary: %+v", msg.Summary)
+	}
+
+	want := manifest.Digests()
+	var got []string
+	for _, r := range h.Hold.BlobRequests() {
+		got = append(got, r.Digest)
+	}
+	if len(got) != len(want) {
+		t.Errorf("fetched %d blobs, manifest declares %d: got %v", len(got), len(want), got)
+	}
+	for _, w := range want {
+		if !slices.Contains(got, mockhold.DigestHex(w)) {
+			t.Errorf("declared blob %s was never fetched", w)
+		}
+	}
+}
+
+// TestSkipsAttestation feeds the scanner the shape that reached production: an
+// in-toto SLSA provenance manifest carrying an ordinary image config, so the
+// config media type alone waves it through. It must come back as "skipped"
+// rather than "error", because the hold retries failures on the stale-scan
+// loop and never retries skips.
+//
+// No blob bytes are involved: the scanner refuses before it downloads
+// anything, which the blob-request count asserts directly.
+func TestSkipsAttestation(t *testing.T) {
+	h := Start(t, mockhold.NewMemory())
+
+	attestations, err := mockhold.CorpusByShape(mockhold.ShapeAttestation)
+	if err != nil {
+		t.Fatalf("load corpus: %v", err)
+	}
+	if len(attestations) == 0 {
+		t.Fatal("corpus contains no attestation manifests")
+	}
+
+	for _, m := range mockhold.Representatives(attestations) {
+		seq, err := h.Hold.SendJob(m.Job())
+		if err != nil {
+			t.Fatalf("send job: %v", err)
+		}
+		msg := h.AwaitTerminal(t, seq, 30*time.Second)
+		if msg.Type != "skipped" {
+			t.Errorf("%s (%s): want skipped, got %s: %s%s",
+				m.Repository, m.Digest[:19], msg.Type, msg.Error, msg.Reason)
+		}
+	}
+
+	if n := len(h.Hold.BlobRequests()); n != 0 {
+		t.Errorf("scanner downloaded %d blobs for artifacts it refused", n)
+	}
+}
+
+// TestSkipsHelm covers the other refusal path, where the config media type is
+// itself unscannable.
+func TestSkipsHelm(t *testing.T) {
+	h := Start(t, mockhold.NewMemory())
+
+	charts, err := mockhold.CorpusByShape(mockhold.ShapeHelm)
+	if err != nil {
+		t.Fatalf("load corpus: %v", err)
+	}
+	if len(charts) == 0 {
+		t.Fatal("corpus contains no helm manifests")
+	}
+
+	for _, m := range mockhold.Representatives(charts) {
+		seq, err := h.Hold.SendJob(m.Job())
+		if err != nil {
+			t.Fatalf("send job: %v", err)
+		}
+		msg := h.AwaitTerminal(t, seq, 30*time.Second)
+		if msg.Type != "skipped" {
+			t.Errorf("%s: want skipped, got %s: %s%s",
+				m.Repository, msg.Type, msg.Error, msg.Reason)
+		}
+	}
+}
+
+// TestIndexManifestIsRetriedForever documents current behaviour rather than
+// desired behaviour, and is deliberately written to pass today.
+//
+// A multi-arch index has no config and no layers. skipReason returns empty for
+// it: there is no config media type to match, and the layer check is skipped
+// when the layer list is empty (worker_skip_test.go asserts exactly that, as
+// "no layers at all is left to the pipeline"). The pipeline then fails in
+// buildOCILayout with "config blob has empty digest", and an error is
+// retryable, so the hold's stale-scan loop will re-offer this job forever.
+//
+// Nothing in the scanner prevents this. The only thing that does is on the far
+// side of the WebSocket: both of the hold's enqueue paths gate on
+// HasScannableContent (!IsMultiArch() && !IsReferrer()), so an index never
+// gets dispatched in production. That guard is load-bearing and nothing on the
+// scanner side would notice if it regressed, which is the point of pinning the
+// behaviour here.
+func TestIndexManifestIsRetriedForever(t *testing.T) {
+	h := Start(t, mockhold.NewMemory())
+
+	indexes, err := mockhold.CorpusByShape(mockhold.ShapeIndex)
+	if err != nil {
+		t.Fatalf("load corpus: %v", err)
+	}
+	if len(indexes) == 0 {
+		t.Fatal("corpus contains no index manifests")
+	}
+
+	m := indexes[0]
+	seq, err := h.Hold.SendJob(m.Job())
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "error" {
+		t.Fatalf("index manifest: expected the current retryable-error behaviour, got %s (%s%s)",
+			msg.Type, msg.Error, msg.Reason)
+	}
+	t.Logf("index manifest returns a retryable error, not a skip: %q", msg.Error)
+}
+
+// TestOversizeImageIsRejectedBeforeDownload pins the cheap half of the size
+// guard: it sums the sizes the manifest claims, so no blob source is needed
+// and nothing is fetched. An image does not shrink, so the verdict is a
+// permanent skip rather than a failure the stale loop re-offers forever. The
+// transferred-bytes half of the ceiling lives in blob_integrity_test.go.
+func TestOversizeImageIsRejectedBeforeDownload(t *testing.T) {
+	h := Start(t, mockhold.NewMemory(), WithMaxImageSize(1024))
+
+	manifest := corpusOne(t, hsmOperator)
+	seq, err := h.Hold.SendJob(manifest.Job())
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "skipped" {
+		t.Fatalf("want skipped for oversize image, got %s: %s%s", msg.Type, msg.Error, msg.Reason)
+	}
+	if n := len(h.Hold.BlobRequests()); n != 0 {
+		t.Errorf("oversize image still fetched %d blobs", n)
+	}
+}

scanner/internal/e2e/protocol_test.go

@@ -0,0 +1,722 @@
+package e2e
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"strings"
+	"testing"
+	"time"
+
+	scanner "atcr.io/scanner"
+	"atcr.io/scanner/internal/client"
+	"atcr.io/scanner/internal/config"
+	"atcr.io/scanner/internal/mockhold"
+	"atcr.io/scanner/internal/queue"
+	"atcr.io/scanner/internal/scan"
+)
+
+// --- helpers ---------------------------------------------------------------
+
+// silenceWindow is how long a test waits before concluding the scanner sent
+// nothing back. It is short on purpose: the failure mode being pinned is
+// "nothing, ever", and the hold's own ackTimeout is five minutes, so anything
+// the scanner has not said within a second here it will not say at all.
+const silenceWindow = 1 * time.Second
+
+// synthJob builds a descriptor-only container image job. The digests point at
+// nothing, which is fine for every scenario that either refuses the job before
+// downloading or deliberately stalls the download.
+func synthJob(repo string) *scanner.ScanJob {
+	return &scanner.ScanJob{
+		ManifestDigest: "sha256:" + strings.Repeat("a", 64),
+		Repository:     repo,
+		Tag:            "latest",
+		Tier:           "deckhand",
+		HoldDID:        "did:web:hold.example",
+		Config: scanner.BlobDescriptor{
+			Digest:    "sha256:" + strings.Repeat("c", 64),
+			Size:      100,
+			MediaType: "application/vnd.oci.image.config.v1+json",
+		},
+		Layers: []scanner.BlobDescriptor{{
+			Digest:    "sha256:" + strings.Repeat("1", 64),
+			Size:      200,
+			MediaType: "application/vnd.oci.image.layer.v1.tar+gzip",
+		}},
+	}
+}
+
+// helmJob returns a corpus Helm chart, the cheapest job that reaches a
+// terminal message: the scanner refuses it on config media type alone, before
+// any tmp dir or download work, so it is a reliable liveness probe.
+func helmJob(t *testing.T) *scanner.ScanJob {
+	t.Helper()
+	charts, err := mockhold.CorpusByShape(mockhold.ShapeHelm)
+	if err != nil {
+		t.Fatalf("load corpus: %v", err)
+	}
+	if len(charts) == 0 {
+		t.Fatal("corpus contains no helm manifests")
+	}
+	return charts[0].Job()
+}
+
+// rawJob renders a job frame from a map, so a test can omit a field, or give
+// it a shape the scanner cannot decode.
+func rawJob(t *testing.T, fields map[string]any) []byte {
+	t.Helper()
+	data, err := json.Marshal(fields)
+	if err != nil {
+		t.Fatalf("marshal raw job: %v", err)
+	}
+	return data
+}
+
+// expectSilence fails if any message matching match arrives within the window.
+func expectSilence(t *testing.T, h *Harness, match func(mockhold.Message) bool, window time.Duration, what string) {
+	t.Helper()
+	msg, err := h.Hold.WaitForMessage(match, window)
+	if err == nil {
+		t.Fatalf("%s: expected no reply, got %s for seq %d", what, msg.Type, msg.Seq)
+	}
+}
+
+// forSeq matches any message about one job.
+func forSeq(seq int64) func(mockhold.Message) bool {
+	return func(m mockhold.Message) bool { return m.Seq == seq }
+}
+
+// assertAlive proves the WebSocket survived whatever the previous step did to
+// it, by pushing a job the scanner is guaranteed to answer.
+func assertAlive(t *testing.T, h *Harness) {
+	t.Helper()
+	seq, err := h.Hold.SendJob(helmJob(t))
+	if err != nil {
+		t.Fatalf("connection did not survive: %v", err)
+	}
+	if msg := h.AwaitTerminal(t, seq, 30*time.Second); msg.Type != "skipped" {
+		t.Fatalf("liveness probe: want skipped, got %s", msg.Type)
+	}
+}
+
+// --- 1. frames the scanner cannot parse -------------------------------------
+
+// TestUnparseableFramesAreAnsweredWithSkipped is the central protocol finding,
+// and the shape of a nine-day outage.
+//
+// connectOnce decodes three things — the frame, then the config sub-document,
+// then the layers sub-document — and every failure branch used to be
+// slog.Error followed by continue, sending nothing back at all. The ack was
+// sent only after both sub-document unmarshals, so a job whose config or
+// layers did not decode was never even acknowledged.
+//
+// The hold has already written status='assigned' for that seq before it wrote
+// the frame. Its only escape was the five-minute ackTimeout in
+// reDispatchTimedOut, after which the row was re-offered — to the same
+// scanner, which dropped it again for exactly the same reason, because a
+// decoding disagreement is permanent. And because hasActiveJobs counts
+// 'assigned' rows and dispatchLoop admits one proactive candidate at a time
+// behind waitForCapacity, one such row meant no proactive scan was ever
+// dispatched again, deployment-wide.
+//
+// The scanner now answers "skipped" for any frame that carries a usable seq.
+// Skipped is the correct verdict rather than "error": the hold retries
+// failures on the rescan interval, and no retry of an undecodable frame can
+// ever succeed, whereas handleSkipped writes a terminal record and releases
+// the row and the in-flight digest for good. The hold-side half is
+// TestScanSkipped_RetiresAnUndecodableFrame in
+// pkg/hold/pds/scan_broadcaster_stuck_test.go.
+func TestUnparseableFramesAreAnsweredWithSkipped(t *testing.T) {
+	h := Start(t, mockhold.NewMemory())
+
+	base := func(seq int64) map[string]any {
+		return map[string]any{
+			"type":           "job",
+			"seq":            seq,
+			"manifestDigest": "sha256:" + strings.Repeat("a", 64),
+			"repository":     "probe/unparseable",
+			"tag":            "latest",
+			"userDid":        "did:plc:probe",
+			"holdDid":        "did:web:hold.example",
+			"holdEndpoint":   h.Hold.URL(),
+			"tier":           "deckhand",
+			"config":         map[string]any{"digest": "sha256:" + strings.Repeat("c", 64), "size": 1, "mediaType": "application/vnd.oci.image.config.v1+json"},
+			"layers":         []any{},
+		}
+	}
+
+	cases := []struct {
+		name  string
+		frame func(seq int64) []byte
+	}{
+		{"config is a string", func(seq int64) []byte {
+			f := base(seq)
+			f["config"] = "not-an-object"
+			return rawJob(t, f)
+		}},
+		{"config field absent", func(seq int64) []byte {
+			f := base(seq)
+			delete(f, "config")
+			return rawJob(t, f)
+		}},
+		{"layers is an object", func(seq int64) []byte {
+			f := base(seq)
+			f["layers"] = map[string]any{"oops": 1}
+			return rawJob(t, f)
+		}},
+		{"layers field absent", func(seq int64) []byte {
+			f := base(seq)
+			delete(f, "layers")
+			return rawJob(t, f)
+		}},
+		// encoding/json records the first type error and keeps decoding the
+		// rest of the object, so a frame the top-level unmarshal rejects can
+		// still yield the seq that addresses the hold's row. That row must be
+		// answered too, which is why the reply is keyed on the seq rather than
+		// on which of the three decodes failed.
+		{"frame has a type-mismatched field", func(seq int64) []byte {
+			f := base(seq)
+			f["tier"] = 12345
+			return rawJob(t, f)
+		}},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			seq := h.Hold.NextSeq()
+			if err := h.Hold.SendRaw(tc.frame(seq)); err != nil {
+				t.Fatalf("send raw frame: %v", err)
+			}
+			msg, err := h.Hold.WaitForMessage(func(m mockhold.Message) bool {
+				return m.Seq == seq && m.Type != "ack"
+			}, 10*time.Second)
+			if err != nil {
+				t.Fatalf("%s: no reply for seq %d; the hold's row stays "+
+					"assigned and is re-offered to this same scanner forever. "+
+					"transcript: %s", tc.name, seq, describe(h.Hold.Transcript()))
+			}
+			if msg.Type != "skipped" {
+				t.Fatalf("%s: reply for seq %d was %s (%s%s), want skipped: an "+
+					"undecodable frame is a permanent condition and the hold "+
+					"retries anything it records as a failure",
+					tc.name, seq, msg.Type, msg.Error, msg.Reason)
+			}
+			if msg.Reason == "" {
+				t.Errorf("%s: skipped with no reason; the hold stores it on the "+
+					"scan record and it is all a user ever sees", tc.name)
+			}
+		})
+	}
+
+	// The connection survives all of it, which is what made the retry loop
+	// infinite rather than self-limiting.
+	assertAlive(t, h)
+}
+
+// TestFramesWithNoUsableSeqAreDroppedSilently is the deliberate exception.
+//
+// A frame that does not decode far enough to yield a seq addresses no job:
+// there is no row to retire and no seq to put in a reply, so logging is the
+// only thing left. This is safe in a way the config/layers case never was —
+// the hold writes status='assigned' keyed by seq before it sends, so a frame
+// whose seq never made it onto the wire cannot be the frame that stranded a
+// row. The connection must survive, since the hold will keep using it.
+func TestFramesWithNoUsableSeqAreDroppedSilently(t *testing.T) {
+	h := Start(t, mockhold.NewMemory())
+
+	for _, tc := range []struct {
+		name  string
+		frame []byte
+	}{
+		{"frame is not JSON", []byte("{ this is not json")},
+		{"frame is a JSON array", []byte(`[1,2,3]`)},
+		{"frame carries seq 0", rawJob(t, map[string]any{"type": "job", "seq": 0, "config": "nope"})},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			before := len(h.Hold.Transcript())
+			if err := h.Hold.SendRaw(tc.frame); err != nil {
+				t.Fatalf("send raw frame: %v", err)
+			}
+			expectSilence(t, h, func(m mockhold.Message) bool {
+				return m.Seq == 0
+			}, silenceWindow, tc.name)
+			if got := len(h.Hold.Transcript()); got != before {
+				t.Errorf("%s: scanner sent %d messages for an unaddressable frame",
+					tc.name, got-before)
+			}
+		})
+	}
+
+	assertAlive(t, h)
+}
+
+// TestNullConfigIsAckedThenFailsRetryably covers the shape that *does* decode:
+// a JSON null unmarshals into a zero BlobDescriptor without error, so the job
+// is acked and enters the pipeline, then dies in buildOCILayout on the empty
+// config digest. That is reported as "error", which the hold treats as
+// transient and retries on the rescan interval forever, even though no retry
+// can ever succeed: nothing about a null config will change.
+func TestNullConfigIsAckedThenFailsRetryably(t *testing.T) {
+	h := Start(t, mockhold.NewMemory())
+
+	seq := h.Hold.NextSeq()
+	frame := rawJob(t, map[string]any{
+		"type":           "job",
+		"seq":            seq,
+		"manifestDigest": "sha256:" + strings.Repeat("b", 64),
+		"repository":     "probe/null-config",
+		"tag":            "latest",
+		"userDid":        "did:plc:probe",
+		"holdDid":        "did:web:hold.example",
+		"holdEndpoint":   h.Hold.URL(),
+		"tier":           "deckhand",
+		"config":         nil,
+		"layers": []any{map[string]any{
+			"digest":    "sha256:" + strings.Repeat("1", 64),
+			"size":      10,
+			"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
+		}},
+	})
+	if err := h.Hold.SendRaw(frame); err != nil {
+		t.Fatalf("send raw frame: %v", err)
+	}
+
+	if _, err := h.Hold.WaitForMessage(func(m mockhold.Message) bool {
+		return m.Seq == seq && m.Type == "ack"
+	}, 10*time.Second); err != nil {
+		t.Fatalf("null config was not acked: %v", err)
+	}
+
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "error" {
+		t.Fatalf("want the current retryable-error behaviour, got %s (%s%s)", msg.Type, msg.Error, msg.Reason)
+	}
+	if !strings.Contains(msg.Error, "empty digest") {
+		t.Errorf("unexpected error text %q", msg.Error)
+	}
+	t.Logf("null config is a permanent condition reported as a retryable error: %q", msg.Error)
+}
+
+// TestUnknownMessageTypeIsIgnored pins the benign half: a frame the scanner
+// does not recognise is logged and skipped, the connection survives, and
+// nothing is sent back. Harmless today because the hold only ever sends
+// "job", but it means any future message type is silently swallowed by an
+// older scanner rather than refused.
+func TestUnknownMessageTypeIsIgnored(t *testing.T) {
+	h := Start(t, mockhold.NewMemory())
+
+	seq := h.Hold.NextSeq()
+	if err := h.Hold.SendRaw(rawJob(t, map[string]any{"type": "cancel", "seq": seq})); err != nil {
+		t.Fatalf("send raw frame: %v", err)
+	}
+	expectSilence(t, h, forSeq(seq), silenceWindow, "unknown message type")
+	assertAlive(t, h)
+}
+
+// --- 2. duplicate delivery --------------------------------------------------
+
+// TestDuplicateSeqIsProcessedTwice shows the scanner has no idea it has seen a
+// job before. Nothing dedupes on seq or on manifest digest: the job is acked
+// twice, queued twice, and scanned twice.
+//
+// On the hold side the second ack is a no-op (handleAck's UPDATE is guarded by
+// status='assigned', which the first ack already cleared) but the second
+// terminal message is not: handleSkipped/handleResult/handleError re-run
+// unconditionally, writing a second scan record to the PDS for the same
+// manifest. For a real image this is also a full second download and Syft run.
+func TestDuplicateSeqIsProcessedTwice(t *testing.T) {
+	h := Start(t, mockhold.NewMemory())
+
+	job := helmJob(t)
+	job.Seq = h.Hold.NextSeq()
+	for i := 0; i < 2; i++ {
+		dup := *job
+		if _, err := h.Hold.SendJob(&dup); err != nil {
+			t.Fatalf("send job %d: %v", i, err)
+		}
+	}
+
+	deadline := time.Now().Add(30 * time.Second)
+	var acks, terminals int
+	for time.Now().Before(deadline) {
+		acks, terminals = 0, 0
+		for _, m := range h.Hold.Transcript() {
+			if m.Seq != job.Seq {
+				continue
+			}
+			switch m.Type {
+			case "ack":
+				acks++
+			case "result", "error", "skipped":
+				terminals++
+			}
+		}
+		if acks >= 2 && terminals >= 2 {
+			break
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+
+	if acks != 2 || terminals != 2 {
+		t.Fatalf("duplicate seq: got %d acks and %d terminal messages, want 2 and 2", acks, terminals)
+	}
+	t.Logf("seq %d was acked %d times and answered %d times: no dedup anywhere in the scanner",
+		job.Seq, acks, terminals)
+}
+
+// --- 3. queue capacity ------------------------------------------------------
+
+// TestQueueFullIsReportedAsRetryableError pins what happens past the queue's
+// high-water mark: the job is acked (hold: assigned -> processing) and then
+// immediately answered with "error: scanner queue full", which the hold
+// records as a *failure*. Failures are retryable, so the same job comes back
+// on the rescan interval and will overflow again for as long as the backlog
+// persists. A capacity signal is being reported through the channel reserved
+// for scan outcomes, and it lands in the user's scan history as a failed scan.
+func TestQueueFullIsReportedAsRetryableError(t *testing.T) {
+	const queueSize = 2
+	h := Start(t, mockhold.NewMemory(),
+		WithWorkers(0), // nothing drains the queue, so the Nth job is deterministic
+		func(c *config.Config) { c.Scanner.QueueSize = queueSize })
+
+	var seqs []int64
+	for i := 0; i < queueSize+2; i++ {
+		seq, err := h.Hold.SendJob(synthJob("probe/overflow"))
+		if err != nil {
+			t.Fatalf("send job %d: %v", i, err)
+		}
+		seqs = append(seqs, seq)
+	}
+
+	for _, seq := range seqs[queueSize:] {
+		msg := h.AwaitTerminal(t, seq, 10*time.Second)
+		if msg.Type != "error" || !strings.Contains(msg.Error, "queue full") {
+			t.Fatalf("seq %d: want a queue-full error, got %s (%s%s)", seq, msg.Type, msg.Error, msg.Reason)
+		}
+	}
+
+	// And the overflowed jobs were acked first, so the hold saw them go
+	// assigned -> processing -> failed for a condition that never involved the
+	// job at all.
+	for _, seq := range seqs[queueSize:] {
+		if _, err := h.Hold.WaitForMessage(func(m mockhold.Message) bool {
+			return m.Seq == seq && m.Type == "ack"
+		}, time.Second); err != nil {
+			t.Errorf("seq %d was rejected without ever being acked", seq)
+		}
+	}
+	t.Log("queue overflow is reported as a retryable per-job failure, not as backpressure")
+}
+
+// TestZeroQueueSizeRejectsEveryJob is the configuration corner of the same
+// path. scanner.queue_size = 0 passes validation, and NewJobQueue(0) then
+// refuses every Enqueue, so a scanner that looks healthy (connected, health
+// endpoint green, workers idle) fails 100% of jobs with "scanner queue full"
+// and the hold retries all of them forever.
+func TestZeroQueueSizeRejectsEveryJob(t *testing.T) {
+	h := Start(t, mockhold.NewMemory(), func(c *config.Config) { c.Scanner.QueueSize = 0 })
+
+	seq, err := h.Hold.SendJob(helmJob(t))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+	msg := h.AwaitTerminal(t, seq, 10*time.Second)
+	if msg.Type != "error" || !strings.Contains(msg.Error, "queue full") {
+		t.Fatalf("queue_size=0: want a queue-full error, got %s (%s%s)", msg.Type, msg.Error, msg.Reason)
+	}
+}
+
+// --- 4. worker configuration ------------------------------------------------
+
+// TestZeroWorkersAcksAndStrands: scanner.workers = 0 passes validation and
+// starts a pool with no workers at all. The client still acks everything it
+// receives, so the hold moves each job to 'processing' and then waits out the
+// ten-minute processing timeout in reDispatchTimedOut before failing it. The
+// scanner logs "Scanner worker pool started workers=0" once at boot and
+// nothing else; there is no health signal that distinguishes this from idle.
+func TestZeroWorkersAcksAndStrands(t *testing.T) {
+	h := Start(t, mockhold.NewMemory(), WithWorkers(0))
+
+	seq, err := h.Hold.SendJob(helmJob(t))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+
+	if _, err := h.Hold.WaitForMessage(func(m mockhold.Message) bool {
+		return m.Seq == seq && m.Type == "ack"
+	}, 10*time.Second); err != nil {
+		t.Fatalf("job was not even acked: %v", err)
+	}
+
+	expectSilence(t, h, func(m mockhold.Message) bool {
+		return m.Seq == seq && (m.Type == "result" || m.Type == "error" || m.Type == "skipped")
+	}, 2*time.Second, "workers=0")
+
+	if n := h.Queue.Len(); n != 1 {
+		t.Errorf("queue holds %d jobs, want 1 (acked and stranded)", n)
+	}
+}
+
+// TestEmptyTmpDirFailsEveryJob: vuln.tmp_dir = "" is accepted by config
+// loading, skips the TMPDIR export in WorkerPool.Start, and then fails every
+// single job in processJob's ensureDir, because os.MkdirAll("") is an error.
+// The failure is retryable, so every job in the deployment loops forever.
+func TestEmptyTmpDirFailsEveryJob(t *testing.T) {
+	h := Start(t, mockhold.NewMemory(), func(c *config.Config) { c.Vuln.TmpDir = "" })
+
+	seq, err := h.Hold.SendJob(synthJob("probe/no-tmpdir"))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+	msg := h.AwaitTerminal(t, seq, 30*time.Second)
+	if msg.Type != "error" || !strings.Contains(msg.Error, "tmp dir") {
+		t.Fatalf("empty tmp_dir: want a tmp dir error, got %s (%s%s)", msg.Type, msg.Error, msg.Reason)
+	}
+	if n := len(h.Hold.BlobRequests()); n != 0 {
+		t.Errorf("failed before download but still fetched %d blobs", n)
+	}
+	t.Logf("every job fails with %q, retryably", msg.Error)
+}
+
+// --- 5. priority ------------------------------------------------------------
+
+// gate is an HTTP stand-in for a hold whose getBlob hangs. A job pointed at it
+// occupies a worker for exactly as long as the test wants, which is how the
+// backlog scenarios below build a queue without needing real image bytes.
+type gate struct {
+	srv     *httptest.Server
+	entered chan struct{}
+	release chan struct{}
+}
+
+func newGate(t *testing.T) *gate {
+	t.Helper()
+	g := &gate{entered: make(chan struct{}, 8), release: make(chan struct{})}
+	g.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		select {
+		case g.entered <- struct{}{}:
+		default:
+		}
+		<-g.release
+		http.Error(w, "gate released", http.StatusNotFound)
+	}))
+	t.Cleanup(g.srv.Close)
+	return g
+}
+
+func (g *gate) waitEntered(t *testing.T, timeout time.Duration) {
+	t.Helper()
+	select {
+	case <-g.entered:
+	case <-time.After(timeout):
+		t.Fatal("worker never reached the stalled blob fetch")
+	}
+}
+
+func (g *gate) open() { close(g.release) }
+
+// TestHighTierJumpsQueuedBacklog confirms the priority heap does what it
+// claims across the real client/queue/worker path: while one worker is busy,
+// a later owner-tier job overtakes an earlier deckhand-tier one.
+//
+// It also shows the limit of that guarantee. Priority is consulted only at
+// Dequeue, so a high-tier job that arrives while the single worker is inside
+// a scan waits for that scan to finish plus the full JobCooldown. With the
+// production 10s cooldown and multi-minute scans, "priority" means position in
+// a queue, not preemption, and a saturated scanner starves the low tier
+// entirely: every owner job admitted during a scan is dequeued before any
+// deckhand job, no matter how long the deckhand job has waited.
+func TestHighTierJumpsQueuedBacklog(t *testing.T) {
+	h := Start(t, mockhold.NewMemory())
+	g := newGate(t)
+
+	// Occupy the single worker with a job whose blob fetch never returns.
+	blocker := synthJob("probe/blocker")
+	blocker.HoldEndpoint = g.srv.URL
+	blockerSeq, err := h.Hold.SendJob(blocker)
+	if err != nil {
+		t.Fatalf("send blocker: %v", err)
+	}
+	g.waitEntered(t, 15*time.Second)
+
+	// Queue a deckhand job first, then an owner job.
+	low := helmJob(t)
+	low.Tier = "deckhand"
+	lowSeq, err := h.Hold.SendJob(low)
+	if err != nil {
+		t.Fatalf("send low: %v", err)
+	}
+	high := helmJob(t)
+	high.Tier = "owner"
+	highSeq, err := h.Hold.SendJob(high)
+	if err != nil {
+		t.Fatalf("send high: %v", err)
+	}
+
+	// Both must be in the queue before the worker is freed, or the test would
+	// be measuring arrival order rather than priority.
+	for _, seq := range []int64{lowSeq, highSeq} {
+		if _, err := h.Hold.WaitForMessage(func(m mockhold.Message) bool {
+			return m.Seq == seq && m.Type == "ack"
+		}, 10*time.Second); err != nil {
+			t.Fatalf("seq %d never acked: %v", seq, err)
+		}
+	}
+	if n := h.Queue.Len(); n != 2 {
+		t.Fatalf("queue depth %d, want 2 backlogged jobs", n)
+	}
+
+	g.open()
+	h.AwaitTerminal(t, blockerSeq, 30*time.Second)
+
+	highMsg := h.AwaitTerminal(t, highSeq, 30*time.Second)
+	lowMsg := h.AwaitTerminal(t, lowSeq, 30*time.Second)
+	if !highMsg.At.Before(lowMsg.At) {
+		t.Errorf("owner-tier job finished at %s, deckhand at %s: priority did not apply",
+			highMsg.At, lowMsg.At)
+	}
+}
+
+// --- 6. shutdown ------------------------------------------------------------
+
+// TestQueueCloseDrainsRatherThanCancels documents queue.Close semantics, which
+// are not what "close" suggests. Dequeue returns nil only when the queue is
+// closed *and* empty, so a shutdown with a backlog hands every remaining job
+// to a worker rather than dropping it. Combined with HoldClient.Close having
+// already severed the socket, whatever those jobs produce is written into a
+// dead connection and lost, while the hold sits on them until the ten-minute
+// processing timeout.
+func TestQueueCloseDrainsRatherThanCancels(t *testing.T) {
+	q := queue.NewJobQueue(10)
+	for i := 0; i < 3; i++ {
+		if !q.Enqueue(&scanner.ScanJob{Seq: int64(i + 1), Tier: "deckhand"}) {
+			t.Fatalf("enqueue %d refused", i)
+		}
+	}
+
+	q.Close()
+
+	var drained []int64
+	for {
+		job := q.Dequeue()
+		if job == nil {
+			break
+		}
+		drained = append(drained, job.Seq)
+	}
+	if len(drained) != 3 {
+		t.Fatalf("Close() discarded the backlog: drained %v, want 3 jobs", drained)
+	}
+	t.Logf("Close() left %d jobs to be dequeued and scanned after shutdown began", len(drained))
+}
+
+// TestShutdownDoesNotInterruptInFlightDownload proves processJob ignores
+// context cancellation everywhere that matters. Syft and Grype take ctx, but
+// the blob fetches go through client.GetBlobPresignedURL / DownloadBlob, which
+// use a package-level http.Client with no request context at all. Cancelling
+// the pool's context while a fetch is in flight changes nothing: the worker
+// stays inside the download until the client's own five-minute timeout, and
+// WorkerPool.Wait (which cmd/scanner calls on SIGTERM, after cancel) blocks
+// for just as long. Under a typical 30-second termination grace period that is
+// a SIGKILL, with the scan directory left behind because cleanup never runs.
+func TestShutdownDoesNotInterruptInFlightDownload(t *testing.T) {
+	hold := mockhold.New(mockhold.NewMemory(), mockhold.WithSecret(testSecret))
+	t.Cleanup(hold.Close)
+
+	cfg := config.DefaultConfig()
+	cfg.Hold.URL = hold.URL()
+	cfg.Hold.Secret = testSecret
+	cfg.Scanner.Workers = 1
+	cfg.Vuln.Enabled = false
+	cfg.Vuln.TmpDir = t.TempDir()
+
+	origTmpDir, hadTmpDir := os.LookupEnv("TMPDIR")
+	t.Cleanup(func() {
+		if hadTmpDir {
+			os.Setenv("TMPDIR", origTmpDir)
+			return
+		}
+		os.Unsetenv("TMPDIR")
+	})
+
+	restoreCooldown := scan.JobCooldown
+	scan.JobCooldown = 10 * time.Millisecond
+	t.Cleanup(func() { scan.JobCooldown = restoreCooldown })
+
+	q := queue.NewJobQueue(cfg.Scanner.QueueSize)
+	c := client.NewHoldClient(cfg.Hold.URL, cfg.Hold.Secret, q)
+	pool := scan.NewWorkerPool(cfg, q, c)
+
+	ctx, cancel := context.WithCancel(context.Background())
+	pool.Start(ctx)
+	go c.Connect()
+
+	// HoldClient.Close is not idempotent (see TestHoldClientCloseIsNotIdempotent),
+	// so the cleanup must not repeat the shutdown the test itself performs.
+	var closed bool
+	t.Cleanup(func() {
+		cancel()
+		if !closed {
+			c.Close()
+		}
+		q.Close()
+	})
+
+	if err := hold.WaitForScanner(10 * time.Second); err != nil {
+		t.Fatalf("scanner never connected: %v", err)
+	}
+
+	g := newGate(t)
+	job := synthJob("probe/shutdown")
+	job.HoldEndpoint = g.srv.URL
+	if _, err := hold.SendJob(job); err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+	g.waitEntered(t, 15*time.Second)
+
+	// This is the shutdown sequence cmd/scanner runs on SIGTERM.
+	cancel()
+	c.Close()
+	closed = true
+	q.Close()
+
+	waited := make(chan struct{})
+	go func() { pool.Wait(); close(waited) }()
+
+	select {
+	case <-waited:
+		t.Fatal("worker exited on context cancellation; the download path now honours ctx")
+	case <-time.After(500 * time.Millisecond):
+		// Expected: the worker is still inside an uncancellable HTTP fetch.
+	}
+
+	g.open()
+	select {
+	case <-waited:
+	case <-time.After(30 * time.Second):
+		t.Fatal("worker never exited even after the download completed")
+	}
+	t.Log("context cancellation does not reach blob downloads; shutdown waits on the HTTP timeout")
+}
+
+// TestHoldClientCloseIsNotIdempotent pins a sharp edge rather than a live bug:
+// HoldClient.Close closes c.done unconditionally, so a second call panics the
+// process with "close of closed channel". cmd/scanner calls it exactly once
+// today, which is the only reason this is not already an incident, and there
+// is no guard if a future shutdown path (a health-check restart, a reconnect
+// supervisor) calls it again.
+func TestHoldClientCloseIsNotIdempotent(t *testing.T) {
+	c := client.NewHoldClient("ws://127.0.0.1:1", "secret", queue.NewJobQueue(1))
+	c.Close()
+
+	defer func() {
+		if recover() == nil {
+			t.Fatal("Close is idempotent now; this finding is fixed and the test should be inverted")
+		}
+	}()
+	c.Close()
+	t.Fatal("unreachable")
+}

scanner/internal/e2e/stuck_test.go

@@ -0,0 +1,571 @@
+package e2e
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"fmt"
+	"io"
+	"sync"
+	"testing"
+	"time"
+
+	scanner "atcr.io/scanner"
+	"atcr.io/scanner/internal/mockhold"
+)
+
+// Scenarios for the "the scanner gets stuck and never finishes" report.
+//
+// Everything here is a characterization test: it asserts what the scanner does
+// today, and the comment says where that is wrong. Nothing asserts the fixed
+// behaviour, so the suite stays green for whoever is working next door.
+//
+// Every scenario in this file is built so no job ever reaches a successful
+// result: each one fails on a blob download instead. That was originally
+// forced on them (a successful scan used to panic the test binary on
+// worker.go's unconditional result.Summary dereference) and is now simply what
+// keeps them focused on the dispatch and reconnect behaviour under test.
+//
+// The hold-side counterparts are in pkg/hold/pds/scan_broadcaster_stuck_test.go.
+
+// stuckSource is a BlobSource that can stall, delay, and selectively 404, and
+// records enough about each Open for a test to tell serial downloads from
+// concurrent ones.
+//
+// Blobs it does hold are served as junk bytes. Nothing verifies that a
+// downloaded blob hashes to the digest that asked for it (buildOCILayout writes
+// whatever arrives straight to blobs/sha256/<hex>), so junk is enough to make a
+// download "succeed" and move the pipeline on to the next one.
+type stuckSource struct {
+	mu        sync.Mutex
+	opens     []time.Time
+	perDigest map[string]int
+	active    int
+	maxActive int
+
+	// delay is slept inside Open, before answering.
+	delay time.Duration
+	// gate, when non-nil, blocks Open until it is closed.
+	gate chan struct{}
+	// have lists digests to answer with junk bytes. Anything else 404s.
+	have map[string]int
+}
+
+func newStuckSource() *stuckSource {
+	return &stuckSource{
+		perDigest: make(map[string]int),
+		have:      make(map[string]int),
+	}
+}
+
+// gated makes every Open block until Release is called.
+func (s *stuckSource) gated() *stuckSource {
+	s.gate = make(chan struct{})
+	return s
+}
+
+// slow makes every Open take d before answering.
+func (s *stuckSource) slow(d time.Duration) *stuckSource {
+	s.delay = d
+	return s
+}
+
+// servingJunk registers n junk bytes for download and returns the digest that
+// names them. Anything not registered answers 404.
+//
+// The digest is derived from the content rather than from a scenario label,
+// which it has to be: the scanner hashes what arrives and refuses bytes that
+// are not what the descriptor said, so a source can no longer answer an
+// arbitrary digest with arbitrary bytes.
+func (s *stuckSource) servingJunk(n int) string {
+	digest := digestOf(bytes.Repeat([]byte("x"), n))
+	s.have[mockhold.DigestHex(digest)] = n
+	return digest
+}
+
+// Release unblocks a gated source. Safe to call more than once, and always
+// registered as a cleanup: httptest.Server.Close waits for in-flight requests,
+// so a gate that is never opened deadlocks the teardown.
+func (s *stuckSource) Release() {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.gate != nil {
+		select {
+		case <-s.gate:
+		default:
+			close(s.gate)
+		}
+	}
+}
+
+func (s *stuckSource) Open(digest string) (io.ReadCloser, int64, error) {
+	hex := mockhold.DigestHex(digest)
+
+	s.mu.Lock()
+	s.opens = append(s.opens, time.Now())
+	s.perDigest[hex]++
+	s.active++
+	if s.active > s.maxActive {
+		s.maxActive = s.active
+	}
+	gate := s.gate
+	delay, size := s.delay, s.have[hex]
+	_, served := s.have[hex]
+	s.mu.Unlock()
+
+	defer func() {
+		s.mu.Lock()
+		s.active--
+		s.mu.Unlock()
+	}()
+
+	if gate != nil {
+		select {
+		case <-gate:
+		case <-time.After(90 * time.Second):
+			// Safety net: a test that forgets to Release must fail on its own
+			// assertions rather than wedging the whole package.
+		}
+	}
+	if delay > 0 {
+		time.Sleep(delay)
+	}
+
+	if !served {
+		return nil, 0, fmt.Errorf("%w: %s", mockhold.ErrBlobNotFound, digest)
+	}
+	return io.NopCloser(bytes.NewReader(bytes.Repeat([]byte("x"), size))), int64(size), nil
+}
+
+func (s *stuckSource) openCount() int {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return len(s.opens)
+}
+
+func (s *stuckSource) countFor(digest string) int {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return s.perDigest[mockhold.DigestHex(digest)]
+}
+
+func (s *stuckSource) peakConcurrency() int {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return s.maxActive
+}
+
+// waitForOpens blocks until the source has been asked for at least n blobs.
+func (s *stuckSource) waitForOpens(t *testing.T, n int, timeout time.Duration) {
+	t.Helper()
+
+	deadline := time.Now().Add(timeout)
+	for time.Now().Before(deadline) {
+		if s.openCount() >= n {
+			return
+		}
+		time.Sleep(5 * time.Millisecond)
+	}
+	t.Fatalf("only %d blob opens after %s, wanted %d", s.openCount(), timeout, n)
+}
+
+// stuckDigest builds a well-formed digest from a label so each synthetic job has
+// its own blobs and the accounting is unambiguous.
+func stuckDigest(label string) string {
+	sum := sha256.Sum256([]byte(label))
+	return fmt.Sprintf("sha256:%x", sum)
+}
+
+// stuckJob builds an ordinary-looking image job: one config, n tar layers.
+// skipReason waves it through, so the pipeline runs for real.
+func stuckJob(label string, layers int) *scanner.ScanJob {
+	job := &scanner.ScanJob{
+		ManifestDigest: stuckDigest(label + "/manifest"),
+		Repository:     label,
+		Tag:            "latest",
+		UserDID:        "did:plc:example",
+		UserHandle:     "user.example.com",
+		HoldDID:        "did:web:hold.example.com",
+		Tier:           "deckhand",
+		Config: scanner.BlobDescriptor{
+			Digest:    stuckDigest(label + "/config"),
+			Size:      64,
+			MediaType: "application/vnd.oci.image.config.v1+json",
+		},
+	}
+	for i := 0; i < layers; i++ {
+		job.Layers = append(job.Layers, scanner.BlobDescriptor{
+			Digest:    stuckDigest(fmt.Sprintf("%s/layer/%d", label, i)),
+			Size:      1024,
+			MediaType: "application/vnd.oci.image.layer.v1.tar+gzip",
+		})
+	}
+	return job
+}
+
+// stuckWaitForDials blocks until the mock hold has accepted at least n connections.
+func stuckWaitForDials(t *testing.T, h *Harness, n int, timeout time.Duration) {
+	t.Helper()
+
+	deadline := time.Now().Add(timeout)
+	for time.Now().Before(deadline) {
+		if len(h.Hold.Dials()) >= n {
+			return
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+	t.Fatalf("only %d dials after %s, wanted %d", len(h.Hold.Dials()), timeout, n)
+}
+
+// stuckTerminalFor reports the first result/error/skipped message for a seq, if any.
+func stuckTerminalFor(h *Harness, seq int64) (mockhold.Message, bool) {
+	for _, m := range h.Hold.Transcript() {
+		if m.Seq == seq && (m.Type == "result" || m.Type == "error" || m.Type == "skipped") {
+			return m, true
+		}
+	}
+	return mockhold.Message{}, false
+}
+
+// TestStuckJobBlocksEveryJobBehindIt is the first and simplest way scanning
+// stops: one job that never finishes, and a worker pool with nothing to
+// interrupt it.
+//
+// processJob takes a context and honours it nowhere. buildOCILayout does not
+// take one at all, and the worker's context is the process's, cancelled only at
+// shutdown — there is no per-job deadline anywhere in the scanner. So a job
+// that hangs holds the only worker (scanner.workers defaults to 1) for as long
+// as it hangs, and every job behind it sits in the priority queue having
+// already been acked.
+//
+// Here the hang is a blob the hold never answers. That one is bounded, at
+// 5 minutes per HTTP request by client.httpClient — see
+// TestBlobDownloadsAreSerialSoTheirTimeoutsAdd for how far that bound stretches.
+// The unbounded version is the same shape with no timeout at all: Syft's
+// stereoscope extraction and Grype's matching run outside any deadline, and a
+// layer that decompresses forever holds the worker forever.
+func TestStuckJobBlocksEveryJobBehindIt(t *testing.T) {
+	src := newStuckSource().gated()
+	h := Start(t, src)
+	t.Cleanup(src.Release)
+
+	stuck, err := h.Hold.SendJob(stuckJob("stuck", 1))
+	if err != nil {
+		t.Fatalf("send stuck job: %v", err)
+	}
+	src.waitForOpens(t, 1, 10*time.Second)
+
+	behind, err := h.Hold.SendJob(stuckJob("behind", 1))
+	if err != nil {
+		t.Fatalf("send second job: %v", err)
+	}
+
+	// The hold is told the second job is under way immediately: the scanner
+	// acks on receipt, before it even reaches the queue.
+	if _, err := h.Hold.WaitForMessage(func(m mockhold.Message) bool {
+		return m.Seq == behind && m.Type == "ack"
+	}, 5*time.Second); err != nil {
+		t.Fatalf("second job was never acked: %v", err)
+	}
+
+	// And then nothing happens to it, because the only worker is wedged.
+	if _, err := h.Hold.WaitForMessage(func(m mockhold.Message) bool {
+		return m.Seq == behind && m.Type != "ack"
+	}, 2*time.Second); err == nil {
+		t.Fatal("the queued job finished while the first job was stuck; " +
+			"head-of-line blocking is gone, update this test")
+	}
+
+	if n := src.openCount(); n != 1 {
+		t.Errorf("blob opens = %d, want 1: only the stuck job should have "+
+			"started downloading", n)
+	}
+	if _, ok := stuckTerminalFor(h, stuck); ok {
+		t.Error("the stuck job reported a terminal message while its download " +
+			"was still hanging")
+	}
+	t.Logf("seq %d acked and queued, seq %d holding the only worker, no "+
+		"deadline on either", behind, stuck)
+
+	// Let both unwind so the mock's teardown does not block on the open request.
+	src.Release()
+	h.AwaitTerminal(t, stuck, 30*time.Second)
+	h.AwaitTerminal(t, behind, 30*time.Second)
+}
+
+// TestAcksLandLongBeforeTheWorkDoes quantifies the ack timing gap from the
+// scanner's side, and pins the message that closes it.
+//
+// The scanner acks in handleFrame the moment a job is decoded, before
+// queue.Enqueue. That has not changed and should not: the ack means "I have
+// it". What it never meant is "a worker is on it", and the hold used to have
+// no other signal — handleAck left assigned_at at the dispatch time and the
+// ten-minute processing deadline was measured from there, so the deadline
+// covered queueing. With scanner.workers=1 and a JobCooldown of 10 seconds,
+// job N cannot start earlier than 10*(N-1) seconds after the burst is acked
+// even if every scan were instant, so job 61 in a burst was past the hold's
+// deadline before a worker touched it. The queue is 100 deep.
+//
+// A worker now sends 'started' when it dequeues, and the hold measures the
+// scanning deadline from that. This test sends a burst and shows the shape
+// that makes the two messages different signals: every ack lands up front,
+// while the starts are spread across the whole drain.
+func TestAcksLandLongBeforeTheWorkDoes(t *testing.T) {
+	const burst = 6
+
+	src := newStuckSource().slow(300 * time.Millisecond)
+	h := Start(t, src)
+
+	var seqs []int64
+	for i := 0; i < burst; i++ {
+		seq, err := h.Hold.SendJob(stuckJob(fmt.Sprintf("burst-%d", i), 1))
+		if err != nil {
+			t.Fatalf("send job %d: %v", i, err)
+		}
+		seqs = append(seqs, seq)
+	}
+
+	// Wait for every ack.
+	for _, seq := range seqs {
+		if _, err := h.Hold.WaitForMessage(func(m mockhold.Message) bool {
+			return m.Seq == seq && m.Type == "ack"
+		}, 10*time.Second); err != nil {
+			t.Fatalf("seq %d was never acked: %v", seq, err)
+		}
+	}
+
+	// Then for every job to actually finish.
+	for _, seq := range seqs {
+		h.AwaitTerminal(t, seq, 60*time.Second)
+	}
+
+	var firstAck, lastAck, firstStart, lastStart, firstTerminal, lastTerminal time.Time
+	starts := 0
+	for _, m := range h.Hold.Transcript() {
+		switch m.Type {
+		case "ack":
+			if firstAck.IsZero() {
+				firstAck = m.At
+			}
+			lastAck = m.At
+		case "started":
+			starts++
+			if firstStart.IsZero() {
+				firstStart = m.At
+			}
+			lastStart = m.At
+		default:
+			if firstTerminal.IsZero() {
+				firstTerminal = m.At
+			}
+			lastTerminal = m.At
+		}
+	}
+
+	if starts != burst {
+		t.Fatalf("%d 'started' messages for %d jobs: without one per job the "+
+			"hold is back to measuring its scanning deadline from dispatch",
+			starts, burst)
+	}
+	if !lastAck.Before(firstTerminal) {
+		t.Fatalf("the last ack (%s) did not precede the first terminal message (%s)",
+			lastAck, firstTerminal)
+	}
+
+	ackSpread := lastAck.Sub(firstAck)
+	startSpread := lastStart.Sub(firstStart)
+	drain := lastTerminal.Sub(lastAck)
+	perJob := drain / burst
+
+	// The point of the pair: the acks are a burst, the starts are the drain.
+	// Any deadline measured from the ack is measuring the queue.
+	if startSpread <= ackSpread {
+		t.Errorf("starts spread over %s and acks over %s; the two signals are "+
+			"not distinguishable in this run, so the test proves nothing",
+			startSpread.Round(time.Millisecond), ackSpread.Round(time.Millisecond))
+	}
+	if !lastStart.After(lastAck) {
+		t.Errorf("the last start (%s) did not follow the last ack (%s)",
+			lastStart, lastAck)
+	}
+
+	t.Logf("%d jobs acked within %s of each other, started over %s, and took "+
+		"%s to drain (%s per job at a %s cooldown)",
+		burst, ackSpread.Round(time.Millisecond), startSpread.Round(time.Millisecond),
+		drain.Round(time.Millisecond), perJob.Round(time.Millisecond), testJobCooldown)
+	t.Logf("measured from dispatch, at the production cooldown of 10s this "+
+		"same burst would take %s to drain", (perJob+10*time.Second)*burst)
+
+	// The arithmetic that used to make this a bug, kept as an assertion so a
+	// change to either constant shows up here. It is now the bound on the
+	// fallback budget a hold applies to a scanner that sends no 'started', not
+	// on healthy work.
+	const holdScanningDeadline = 10 * time.Minute
+	const productionCooldown = 10 * time.Second
+	if ceiling := int(holdScanningDeadline / productionCooldown); ceiling != 60 {
+		t.Errorf("a burst of more than %d jobs cannot be drained inside the "+
+			"hold's scanning deadline even with free scans; that number moved",
+			ceiling)
+	}
+}
+
+// TestBlobDownloadsAreSerialSoTheirTimeoutsAdd shows why the one timeout the
+// download path does have is not a bound on a job.
+//
+// buildOCILayout fetches the config and then every layer in sequence, each
+// through client.httpClient, whose Timeout is 5 minutes and applies per
+// request. A manifest with 19 layers therefore has a worst case of 20 × 5
+// minutes before the job fails — twice the hold's ten-minute processing
+// deadline, so the hold gives up on a job the scanner is still legitimately
+// working on, and does so without re-dispatching it.
+func TestBlobDownloadsAreSerialSoTheirTimeoutsAdd(t *testing.T) {
+	job := stuckJob("serial", 3)
+
+	// Config and the first two layers download and verify; the third keeps its
+	// label-derived digest, which the source does not have, so it 404s and the
+	// job fails there rather than reaching Syft. The sizes differ so the three
+	// served blobs are three distinct digests.
+	src := newStuckSource().slow(100 * time.Millisecond)
+	job.Config.Digest, job.Config.Size = src.servingJunk(64), 64
+	job.Layers[0].Digest, job.Layers[0].Size = src.servingJunk(65), 65
+	job.Layers[1].Digest, job.Layers[1].Size = src.servingJunk(66), 66
+
+	h := Start(t, src)
+
+	start := time.Now()
+	seq, err := h.Hold.SendJob(job)
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+	msg := h.AwaitTerminal(t, seq, 60*time.Second)
+	elapsed := time.Since(start)
+
+	if msg.Type != "error" {
+		t.Fatalf("want error, got %s (%s%s)", msg.Type, msg.Error, msg.Reason)
+	}
+	if n := src.openCount(); n != 4 {
+		t.Errorf("blob opens = %d, want 4 (config + 3 layers)", n)
+	}
+	if peak := src.peakConcurrency(); peak != 1 {
+		t.Errorf("peak concurrent downloads = %d, want 1: downloads within a "+
+			"job are serial", peak)
+	}
+	if elapsed < 4*100*time.Millisecond {
+		t.Errorf("job took %s, less than the sum of its four downloads", elapsed)
+	}
+
+	t.Logf("4 serial downloads took %s; each is bounded only by the 5-minute "+
+		"client timeout, so this job's worst case is 20 minutes against a "+
+		"10-minute hold deadline", elapsed.Round(time.Millisecond))
+}
+
+// TestReDispatchAfterDisconnectScansTheSameImageTwice proves the duplicate-scan
+// prediction.
+//
+// The hold's Unsubscribe flips the dropped scanner's assigned and processing
+// rows back to 'pending', and drainPendingJobs hands them to the next
+// connection (pinned in
+// pkg/hold/pds/scan_broadcaster_stuck_test.go:TestScanUnsubscribe_ReoffersAJobTheScannerIsStillRunning).
+// Nothing tells the scanner's worker pool the socket went away: the job it was
+// running is still running, and the re-offered copy is enqueued again with no
+// dedupe by seq or digest.
+//
+// With two workers that is two concurrent downloads of the same blob, which the
+// per-digest request count and the source's peak concurrency both show.
+func TestReDispatchAfterDisconnectScansTheSameImageTwice(t *testing.T) {
+	src := newStuckSource().gated()
+	h := Start(t, src, WithWorkers(2))
+	t.Cleanup(src.Release)
+
+	job := stuckJob("duplicated", 1)
+	seq, err := h.Hold.SendJob(job)
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+	src.waitForOpens(t, 1, 10*time.Second)
+
+	// The scanner is mid-download when the socket dies.
+	h.Hold.DropConnections(mockhold.DropAbrupt)
+	stuckWaitForDials(t, h, 2, 15*time.Second)
+
+	// What drainPendingJobs does on the new connection: the same seq again.
+	again := stuckJob("duplicated", 1)
+	again.Seq = seq
+	if _, err := h.Hold.SendJob(again); err != nil {
+		t.Fatalf("re-send job: %v", err)
+	}
+
+	src.waitForOpens(t, 2, 15*time.Second)
+
+	if n := src.countFor(job.Config.Digest); n < 2 {
+		t.Errorf("config blob fetched %d times, want 2: the re-offered job was "+
+			"deduplicated somewhere", n)
+	}
+	if peak := src.peakConcurrency(); peak < 2 {
+		t.Errorf("peak concurrent downloads = %d, want 2: the duplicate ran "+
+			"after the original rather than alongside it", peak)
+	}
+	t.Logf("seq %d is being downloaded by two workers at once; neither knows "+
+		"about the other and both will report a result", seq)
+
+	src.Release()
+	h.AwaitTerminal(t, seq, 30*time.Second)
+}
+
+// TestResultComputedWhileTheSocketIsDownIsLost covers the last way a job goes
+// quiet: the scanner finishes, and its answer goes nowhere.
+//
+// client.sendJSON guards only on `c.conn == nil`, and nothing ever nils conn —
+// connectOnce sets it on dial and the read loop just returns on error. So a
+// terminal message computed between a disconnect and the next dial is written
+// to a closed connection, WriteJSON fails, sendJSON logs it and returns. Ack,
+// SendResult, SendError and SendSkipped all return no error, so the worker has
+// no way to know and no path to retry: it moves straight on to the next job.
+//
+// The hold, meanwhile, put the row back to 'pending' when the scanner dropped,
+// so the work is simply done twice — and the second copy is the one that counts.
+// The loss window is the reconnect backoff, a flat 5 seconds (Connect's comment
+// claims exponential backoff to 30s; the code sleeps 5s and the cursor variable
+// it declares is never assigned).
+func TestResultComputedWhileTheSocketIsDownIsLost(t *testing.T) {
+	src := newStuckSource().gated()
+	h := Start(t, src)
+	t.Cleanup(src.Release)
+
+	seq, err := h.Hold.SendJob(stuckJob("lost", 1))
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+	src.waitForOpens(t, 1, 10*time.Second)
+
+	if _, err := h.Hold.WaitForMessage(func(m mockhold.Message) bool {
+		return m.Seq == seq && m.Type == "ack"
+	}, 5*time.Second); err != nil {
+		t.Fatalf("job was never acked: %v", err)
+	}
+
+	// Kill the socket, then let the job finish. The worker computes its answer
+	// with nowhere to send it.
+	h.Hold.DropConnections(mockhold.DropAbrupt)
+	src.Release()
+
+	// Give the worker time to finish and write into the dead conn, then wait
+	// for the reconnect so we can show nothing is re-sent afterwards either.
+	stuckWaitForDials(t, h, 2, 15*time.Second)
+	time.Sleep(500 * time.Millisecond)
+
+	if msg, ok := stuckTerminalFor(h, seq); ok {
+		t.Fatalf("the hold received a %s for seq %d after all; sendJSON now "+
+			"survives a dropped socket, update this test", msg.Type, seq)
+	}
+	if n := h.Queue.Len(); n != 0 {
+		t.Errorf("scanner queue holds %d jobs; the lost job was requeued, "+
+			"update this test", n)
+	}
+	t.Logf("seq %d was scanned to completion and its outcome discarded; the "+
+		"hold will only notice via its own ten-minute processing timeout", seq)
+
+	// The connection is healthy again — the loss was silent, not fatal.
+	if _, err := h.Hold.SendJob(stuckJob("after-reconnect", 1)); err != nil {
+		t.Fatalf("hold could not dispatch after the reconnect: %v", err)
+	}
+}

scanner/internal/e2e/vulnreport_test.go

@@ -0,0 +1,958 @@
+package e2e
+
+// Scan REPORT verification: what Grype actually matches, and whether the JSON
+// the scanner publishes is the JSON the appview reads back.
+//
+// Everything else in this package proves the pipeline mechanics — a job is
+// dispatched, blobs are fetched, an SBOM comes back. None of it exercises
+// Grype: the harness disables it precisely so no test has to download a
+// vulnerability database, which left the entire matching path, the matcher
+// configuration, and every vulnerability count the scanner has ever published
+// unverified (SCANNER_BUGS.md section 4). These scenarios close that.
+//
+// # Running them
+//
+//	scanner/internal/mockhold/testdata/fetch-vulndb.sh   # ~2 GB, once
+//	cd scanner && ATCR_SCANNER_VULNDB=1 go test ./internal/e2e -run TestVulnDB -v
+//
+// Without ATCR_SCANNER_VULNDB=1 every scenario here skips, so an ordinary
+// `go test ./...` stays fast and offline.
+//
+// # Why nothing here asserts a CVE count or a CVE ID
+//
+// The upstream feed is rebuilt daily. A test that pins "alpine:3.10 has 47
+// criticals" is wrong within a week and then gets muted, which is worse than
+// not having written it. Every assertion below is an invariant that holds
+// whatever the feed says: internal consistency of the counts, agreement
+// between the SBOM and the report, the JSON shape the appview parses,
+// determinism against a fixed database, and "more than zero" on an image that
+// has been end-of-life for years.
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"runtime"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	scanner "atcr.io/scanner"
+	"atcr.io/scanner/internal/mockhold"
+)
+
+// ---------------------------------------------------------------------------
+// Fixtures
+// ---------------------------------------------------------------------------
+
+// vulnFixtureOld is an image that has been end-of-life since 2021. Its apk
+// packages are frozen at versions the feed has accumulated advisories against
+// for years, so "this scan finds vulnerabilities" is a safe assertion in a way
+// it would not be for a current tag. fetch-vulndb.sh pulls it.
+const vulnFixtureOld = "vuln-alpine310"
+
+// vulnFixtureMinimal is distroless static: a scratch-like base with a handful
+// of dpkg status.d entries and no interpreter, no shell, no libc package. It
+// is the control. A matcher that invents results has nothing here to invent
+// them from, so a large finding count against it would mean the matcher is
+// reporting on packages the image does not contain.
+const vulnFixtureMinimal = "vuln-static"
+
+// vulnFixtureDebian is oldstable Debian. It is in the set for one reason: the
+// severity strings differ by ecosystem. Alpine's secdb resolves everything to
+// an NVD severity, so an Alpine image can accidentally satisfy a
+// bucket-completeness check that a Debian image does not — Debian's security
+// tracker carries "Negligible" and "Unknown" ratings, neither of which the
+// scanner buckets. A consistency check run against one distro would report
+// whichever answer that distro happens to give.
+const vulnFixtureDebian = "vuln-debian11"
+
+// severitySweepFixtures is every image on disk worth running the
+// bucket-consistency check against, cheapest first. It deliberately spans
+// package ecosystems (apk, dpkg, go modules) because the shortfall the check
+// looks for is a property of the severity vocabulary each feed uses, not of
+// the scanner's arithmetic.
+func severitySweepFixtures(t *testing.T) []string {
+	t.Helper()
+	var out []string
+	for _, name := range []string{
+		vulnFixtureMinimal,
+		vulnFixtureOld,
+		vulnFixtureDebian,
+		"hsm-secrets-operator", // a Go binary image: go-module matches, no distro packages
+		"perf-python",          // debian bookworm plus a python ecosystem
+	} {
+		if hasFixture(name) {
+			out = append(out, name)
+		}
+	}
+	if len(out) == 0 {
+		t.Skipf("no image fixtures present; run scanner/internal/mockhold/testdata/fetch-vulndb.sh")
+	}
+	return out
+}
+
+// ---------------------------------------------------------------------------
+// The report, decoded the way the appview decodes it
+// ---------------------------------------------------------------------------
+
+// appviewReport and friends are a deliberate restatement of the structs the
+// appview parses this blob with, so a change to the report's shape fails here
+// rather than silently rendering an empty vulnerabilities tab in production.
+//
+// The originals are pkg/appview/handlers/vuln_details.go:25-53 (and a verbatim
+// duplicate in cmd/image-advisor/main.go:62-86). They are restated rather than
+// imported because they are unexported, and because a copy that has to be kept
+// in sync is the point: this file is the thing that notices when it drifts.
+//
+// The one intentional difference: Metadata is a pointer here and a value there.
+// The scanner's own counting code branches on Metadata being nil
+// (scanner/internal/scan/grype.go, countVulnerabilitiesBySeverity), so the
+// tests below need to tell "no metadata" apart from "metadata with an empty
+// severity". The appview cannot: a null Metadata decodes to a zero struct and
+// arrives at the template as an empty severity string.
+type appviewReport struct {
+	Matches []appviewMatch `json:"matches"`
+}
+
+type appviewMatch struct {
+	Vulnerability appviewVuln    `json:"Vulnerability"`
+	Package       appviewPackage `json:"Package"`
+}
+
+type appviewVuln struct {
+	ID       string           `json:"ID"`
+	Metadata *appviewMetadata `json:"Metadata"`
+	Fix      appviewFix       `json:"Fix"`
+}
+
+type appviewMetadata struct {
+	Severity string `json:"Severity"`
+}
+
+type appviewFix struct {
+	Versions []string `json:"Versions"`
+	State    string   `json:"State"`
+}
+
+type appviewPackage struct {
+	Name    string `json:"Name"`
+	Version string `json:"Version"`
+	Type    string `json:"Type"`
+}
+
+// spdxDoc is the slice of the SPDX JSON SBOM these scenarios need: the package
+// list, so the report can be checked against it.
+type spdxDoc struct {
+	SPDXVersion string `json:"spdxVersion"`
+	Packages    []struct {
+		Name        string `json:"name"`
+		VersionInfo string `json:"versionInfo"`
+	} `json:"packages"`
+}
+
+// finding is one match reduced to the identity a user would recognise. It is
+// the unit the determinism scenario compares, because the full report JSON
+// carries a scan-local temp path (see TestVulnDBScanIsDeterministic).
+type finding struct {
+	ID      string
+	Package string
+	Version string
+}
+
+func (f finding) String() string { return f.ID + " " + f.Package + "@" + f.Version }
+
+// ---------------------------------------------------------------------------
+// Running one scan
+// ---------------------------------------------------------------------------
+
+// vulnScan is everything one real scan produced.
+type vulnScan struct {
+	Fixture string
+	Summary scanner.VulnerabilitySummary
+	Report  appviewReport
+	Raw     []byte
+	SBOM    spdxDoc
+	Digest  string
+
+	Elapsed time.Duration
+	PeakRSS uint64 // bytes, sampled from /proc/self/statm during the scan
+	HeapEnd uint64
+}
+
+// Findings returns the report's matches as comparable identities.
+func (v vulnScan) Findings() []finding {
+	out := make([]finding, 0, len(v.Report.Matches))
+	for _, m := range v.Report.Matches {
+		out = append(out, finding{m.Vulnerability.ID, m.Package.Name, m.Package.Version})
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].String() < out[j].String() })
+	return out
+}
+
+// severityHistogram counts every match by the exact severity string Grype put
+// on it, with a distinct key for a match that carries no metadata at all.
+// Both are cases countVulnerabilitiesBySeverity adds to Total and puts in no
+// bucket, so naming them separately is what makes the shortfall diagnosable.
+func (v vulnScan) severityHistogram() map[string]int {
+	hist := map[string]int{}
+	for _, m := range v.Report.Matches {
+		switch {
+		case m.Vulnerability.Metadata == nil:
+			hist["(no metadata)"]++
+		case m.Vulnerability.Metadata.Severity == "":
+			hist["(empty)"]++
+		default:
+			hist[m.Vulnerability.Metadata.Severity]++
+		}
+	}
+	return hist
+}
+
+// requireVulnFixture skips unless the pinned image has been pulled.
+func requireVulnFixture(t *testing.T, name string) {
+	t.Helper()
+	if !hasFixture(name) {
+		t.Skipf("fixture %q not present; run scanner/internal/mockhold/testdata/fetch-vulndb.sh", name)
+	}
+}
+
+// runVulnScan drives one image through the real pipeline with Grype enabled
+// and decodes both artifacts it produced.
+//
+// The timeout is generous because the first scan in a binary also pays the
+// database load: opening a ~2 GB SQLite file and building its indexes.
+// Each scan runs inside its own subtest, which is what bounds the harness's
+// lifetime. Start writes the package-level scan.JobCooldown and every worker
+// reads it once per loop iteration, so two harnesses alive at the same moment
+// is a data race -race reports against worker.go — and several scenarios here
+// scan a list of fixtures in one test. A subtest's t.Cleanup runs when t.Run
+// returns (and Start's cleanup joins the pool before restoring the global), so
+// harness N is fully torn down before harness N+1 starts.
+func runVulnScan(t *testing.T, fixture string) vulnScan {
+	t.Helper()
+	requireVulnFixture(t, fixture)
+
+	var out vulnScan
+	ok := t.Run(fixture, func(t *testing.T) {
+		dir := fixtureDir(fixture)
+		h := Start(t, mockhold.NewOCILayout(dir), WithVulnDB(t))
+		out = sendVulnScan(t, h, jobFromLayout(t, dir), fixture)
+	})
+	if !ok {
+		t.Fatalf("scan of %s did not complete", fixture)
+	}
+	if out.Fixture == "" {
+		t.Skipf("scan of %s was skipped", fixture)
+	}
+	return out
+}
+
+// sendVulnScan sends one job to an already-running harness and decodes the
+// result. Split out from runVulnScan so a scenario can put two scans through
+// one harness (and therefore one loaded provider).
+func sendVulnScan(t *testing.T, h *Harness, job *scanner.ScanJob, fixture string) vulnScan {
+	t.Helper()
+
+	job.Seq = 0 // let the mock allocate a fresh seq per send
+	stopRSS := watchRSS()
+	start := time.Now()
+
+	seq, err := h.Hold.SendJob(job)
+	if err != nil {
+		t.Fatalf("send job: %v", err)
+	}
+	msg := h.AwaitTerminal(t, seq, 15*time.Minute)
+	elapsed := time.Since(start)
+	peakRSS := stopRSS()
+
+	if msg.Type != "result" {
+		t.Fatalf("%s: want result, got %s: %s%s", fixture, msg.Type, msg.Error, msg.Reason)
+	}
+	if msg.Summary == nil {
+		t.Fatalf("%s: vuln scanning is enabled but the result carried no summary", fixture)
+	}
+	if msg.VulnReport == "" {
+		t.Fatalf("%s: result carried no vulnerability report", fixture)
+	}
+	if msg.SBOM == "" {
+		t.Fatalf("%s: result carried no SBOM", fixture)
+	}
+
+	out := vulnScan{
+		Fixture: fixture,
+		Summary: *msg.Summary,
+		Raw:     []byte(msg.VulnReport),
+		Elapsed: elapsed,
+		PeakRSS: peakRSS,
+	}
+	if err := json.Unmarshal(out.Raw, &out.Report); err != nil {
+		t.Fatalf("%s: the appview's own structs cannot parse the vulnerability report: %v", fixture, err)
+	}
+	if err := json.Unmarshal([]byte(msg.SBOM), &out.SBOM); err != nil {
+		t.Fatalf("%s: parse SBOM: %v", fixture, err)
+	}
+
+	var digest struct {
+		Descriptor struct {
+			Name    string `json:"name"`
+			Version string `json:"version"`
+		} `json:"descriptor"`
+	}
+	_ = json.Unmarshal(out.Raw, &digest)
+	out.Digest = digest.Descriptor.Version
+
+	var ms runtime.MemStats
+	runtime.ReadMemStats(&ms)
+	out.HeapEnd = ms.HeapAlloc
+
+	return out
+}
+
+// watchRSS samples this process's resident set every 50ms until the returned
+// stop func is called, and reports the peak.
+//
+// The scanner's production numbers all exclude Grype: FindMatches was never
+// run, so nothing measured what matching and the mmap'd database cost. This is
+// crude (one process, no cgroup) but it is the first measurement of that cost
+// there has been, and the database is mmap-backed, so RSS is the number that
+// matters rather than heap.
+func watchRSS() (stop func() uint64) {
+	var mu sync.Mutex
+	var high uint64
+	done := make(chan struct{})
+	exited := make(chan struct{})
+	var once sync.Once
+
+	sample := func() {
+		rss := currentRSS()
+		mu.Lock()
+		if rss > high {
+			high = rss
+		}
+		mu.Unlock()
+	}
+
+	sample()
+	go func() {
+		defer close(exited)
+		tick := time.NewTicker(50 * time.Millisecond)
+		defer tick.Stop()
+		for {
+			select {
+			case <-done:
+				return
+			case <-tick.C:
+				sample()
+			}
+		}
+	}()
+
+	// stop joins the sampler before reading the peak. Returning a pointer and
+	// letting the caller dereference it races with the last sample, which
+	// -race reports.
+	return func() uint64 {
+		once.Do(func() { close(done) })
+		<-exited
+		sample()
+		mu.Lock()
+		defer mu.Unlock()
+		return high
+	}
+}
+
+// currentRSS reads this process's resident set size. Resident rather than heap
+// because the vulnerability database is a mapped file: the Go runtime's own
+// accounting, and the GOMEMLIMIT built on it, cannot see it.
+func currentRSS() uint64 {
+	data, err := os.ReadFile("/proc/self/statm")
+	if err != nil {
+		return 0
+	}
+	fields := strings.Fields(string(data))
+	if len(fields) < 2 {
+		return 0
+	}
+	pages, err := strconv.ParseUint(fields[1], 10, 64)
+	if err != nil {
+		return 0
+	}
+	return pages * uint64(os.Getpagesize())
+}
+
+func mib(n uint64) string { return fmt.Sprintf("%.0f MiB", float64(n)/(1<<20)) }
+
+// ---------------------------------------------------------------------------
+// One-time priming
+// ---------------------------------------------------------------------------
+
+// The vulnerability provider is a package-level global in internal/scan behind
+// an RWMutex, so one load serves every scenario in this binary. primeVulnDB
+// makes that ordering explicit rather than leaving it to whichever test the
+// -run filter happened to select first: the first scan pays the load, and the
+// rest measure matching alone.
+//
+// This does not reach the stubbed-loader tests in
+// internal/scan/vulndb_refresh_test.go. Those are a different package and
+// therefore a different test binary, with their own copy of these globals and
+// their own stubbed loadVulnDB, so they stay offline whatever this file does.
+var (
+	primeOnce sync.Once
+	primed    vulnScan
+	primeFail string
+)
+
+// primeVulnDB loads the database once and returns the control scan it used to
+// do it. The minimal fixture is the cheapest image that still forces a real
+// match run, so priming costs a scan that a scenario wanted anyway.
+func primeVulnDB(t *testing.T) vulnScan {
+	t.Helper()
+	requireVulnFixture(t, vulnFixtureMinimal)
+
+	primeOnce.Do(func() {
+		defer func() {
+			if r := recover(); r != nil {
+				primeFail = fmt.Sprintf("panic while priming the vulnerability database: %v", r)
+			}
+		}()
+		before := currentRSS()
+		// runVulnScan already bounds its harness in a subtest, so the priming
+		// scan's worker pool is joined before the caller's own harness starts.
+		primed = runVulnScan(t, vulnFixtureMinimal)
+		t.Logf("primed vulnDB: first scan (%s) took %s; RSS %s before, %s peak, %s after — "+
+			"the database is a ~2 GB mmap'd SQLite file, so only the pages the matcher touches "+
+			"become resident",
+			vulnFixtureMinimal, primed.Elapsed.Round(time.Millisecond),
+			mib(before), mib(primed.PeakRSS), mib(currentRSS()))
+	})
+	if primeFail != "" {
+		t.Fatal(primeFail)
+	}
+	if primed.Fixture == "" {
+		// The priming scan skipped or failed inside the Once, which unwinds
+		// its goroutine rather than returning, so there is nothing to report
+		// here beyond "it did not happen". The scenario that ran it carries
+		// the reason.
+		t.Skip("vulnerability database was never primed; see the first scenario's output")
+	}
+	return primed
+}
+
+// ---------------------------------------------------------------------------
+// Scenarios
+// ---------------------------------------------------------------------------
+
+// TestVulnDBKnownVulnerableImageFindsSomething is the base case the whole file
+// rests on: an image that has been out of support for years, scanned against
+// the real feed, must produce findings. If this comes back empty the matcher
+// is not working, whatever the rest of the assertions say.
+//
+// No count is asserted. "More than zero on an EOL distro" is true of every
+// version of the feed that has ever existed; "47 criticals" is true of one
+// day's build.
+func TestVulnDBKnownVulnerableImageFindsSomething(t *testing.T) {
+	primeVulnDB(t)
+	scan := runVulnScan(t, vulnFixtureOld)
+
+	t.Logf("%s: %d packages, %d matches, summary C=%d H=%d M=%d L=%d total=%d in %s (peak RSS %s)",
+		scan.Fixture, len(scan.SBOM.Packages), len(scan.Report.Matches),
+		scan.Summary.Critical, scan.Summary.High, scan.Summary.Medium, scan.Summary.Low,
+		scan.Summary.Total, scan.Elapsed.Round(time.Millisecond), mib(scan.PeakRSS))
+
+	if scan.Summary.Total == 0 {
+		t.Fatal("an end-of-life image scanned clean; the matcher is not finding anything")
+	}
+	if len(scan.Report.Matches) == 0 {
+		t.Fatal("summary claims findings but the report carries no matches")
+	}
+
+	// The summary must describe the report it ships with. This is not a feed
+	// assertion: whatever the feed says, Total is counted from exactly the
+	// match set that gets marshalled.
+	if scan.Summary.Total != len(scan.Report.Matches) {
+		t.Errorf("summary.Total is %d but the report carries %d matches",
+			scan.Summary.Total, len(scan.Report.Matches))
+	}
+
+	// Severity is plausible rather than exact: an EOL base image always has
+	// something above Low. Asserting the pair rather than either alone keeps
+	// this true across the feed's periodic severity re-scoring.
+	if scan.Summary.Critical+scan.Summary.High == 0 {
+		t.Errorf("no critical or high findings on an end-of-life image; histogram: %v",
+			scan.severityHistogram())
+	}
+
+	// Every finding must name a vulnerability. An empty ID renders as a blank
+	// cell in the appview's table (partials/vuln-details.html), which is the
+	// shape of a report that scanned but shows nothing.
+	for i, m := range scan.Report.Matches {
+		if m.Vulnerability.ID == "" {
+			t.Errorf("match %d has no vulnerability ID", i)
+		}
+		if m.Package.Name == "" {
+			t.Errorf("match %d (%s) names no package", i, m.Vulnerability.ID)
+		}
+	}
+}
+
+// TestVulnDBSeverityBucketsAccountForEveryMatch is the internal-consistency
+// check, and it is written to pass against today's behaviour while recording
+// precisely how far off it is.
+//
+// countVulnerabilitiesBySeverity (scanner/internal/scan/grype.go) increments
+// Total for every match but buckets only matches whose Metadata is non-nil and
+// whose Severity is exactly one of Critical/High/Medium/Low. Grype also emits
+// "Negligible" and "Unknown", and neither has a bucket anywhere in the system:
+// not in the wire summary (pkg/hold/pds/scan_broadcaster.go), not in the
+// io.atcr.hold.scan record (pkg/atproto/lexicon.go), not in the CSS
+// (pkg/appview/src/css/main.css has exactly four .vuln-box-* classes).
+//
+// So the assertion is one-directional — the buckets may never exceed the total,
+// and no match may be counted twice — plus a log of the shortfall with the
+// severity strings that caused it. See the report accompanying this file.
+func TestVulnDBSeverityBucketsAccountForEveryMatch(t *testing.T) {
+	primeVulnDB(t)
+
+	unbucketedTotal := 0
+	for _, fixture := range severitySweepFixtures(t) {
+		scan := runVulnScan(t, fixture)
+
+		buckets := scan.Summary.Critical + scan.Summary.High + scan.Summary.Medium + scan.Summary.Low
+		hist := scan.severityHistogram()
+
+		t.Logf("%-22s buckets=%-5d total=%-5d unaccounted=%-4d | %s",
+			fixture, buckets, scan.Summary.Total, scan.Summary.Total-buckets, formatHistogram(hist))
+
+		if buckets > scan.Summary.Total {
+			t.Errorf("%s: severity buckets sum to %d, more than the total of %d — a match is counted twice",
+				fixture, buckets, scan.Summary.Total)
+		}
+
+		// Cross-check the counter against the report it summarises, bucket by
+		// bucket. This is what proves any shortfall is unbucketed severities
+		// rather than a miscount.
+		for _, sev := range []struct {
+			name string
+			got  int
+		}{
+			{"Critical", scan.Summary.Critical},
+			{"High", scan.Summary.High},
+			{"Medium", scan.Summary.Medium},
+			{"Low", scan.Summary.Low},
+		} {
+			if hist[sev.name] != sev.got {
+				t.Errorf("%s: summary says %d %s but the report carries %d",
+					fixture, sev.got, sev.name, hist[sev.name])
+			}
+		}
+
+		// Name what fell through, so the finding is actionable rather than a
+		// number. These are the matches the appview counts in "N
+		// vulnerabilities" (partials/vuln-details.html:36) and renders as a
+		// grey "?" row (:93), while the four coloured boxes beside them sum
+		// lower.
+		var unbucketed []string
+		for sev, n := range hist {
+			switch sev {
+			case "Critical", "High", "Medium", "Low":
+				continue
+			}
+			unbucketed = append(unbucketed, fmt.Sprintf("%s=%d", sev, n))
+		}
+		if len(unbucketed) > 0 {
+			sort.Strings(unbucketed)
+			unbucketedTotal += scan.Summary.Total - buckets
+			t.Logf("  FINDING %s: %d of %d matches are in no severity bucket: %s",
+				fixture, scan.Summary.Total-buckets, scan.Summary.Total, strings.Join(unbucketed, " "))
+		}
+	}
+
+	if unbucketedTotal == 0 {
+		t.Log("every match on every fixture landed in one of the four buckets. That is a property " +
+			"of these images and this feed build, not of the code: countVulnerabilitiesBySeverity " +
+			"has no default arm, so any Negligible, Unknown, empty or metadata-less severity would " +
+			"still be counted in Total and bucketed nowhere.")
+		return
+	}
+	t.Logf("%d matches across the sweep are counted in Total and shown in no severity box", unbucketedTotal)
+}
+
+// TestVulnDBMinimalImageIsNearlyClean is the control. A distroless static base
+// has a couple of dpkg entries and nothing else, so a matcher behaving itself
+// has almost nothing to report. The assertion is proportional rather than
+// absolute: findings must not outnumber what the SBOM says is installed by
+// more than a small factor, and every one must name a package the SBOM lists.
+//
+// That second half is the real control. A count ceiling can drift with the
+// feed; "the report may only mention packages the image contains" cannot.
+func TestVulnDBMinimalImageIsNearlyClean(t *testing.T) {
+	scan := primeVulnDB(t)
+
+	t.Logf("%s: %d packages in the SBOM, %d matches, total=%d",
+		scan.Fixture, len(scan.SBOM.Packages), len(scan.Report.Matches), scan.Summary.Total)
+	if len(scan.SBOM.Packages) == 0 {
+		t.Fatal("the control image catalogued no packages at all; it is not a control, it is an empty scan")
+	}
+
+	sbomNames := map[string]bool{}
+	for _, p := range scan.SBOM.Packages {
+		sbomNames[p.Name] = true
+	}
+	for _, m := range scan.Report.Matches {
+		if !sbomNames[m.Package.Name] {
+			t.Errorf("report names package %q, which is not in the SBOM: the matcher is reporting on "+
+				"something the image does not contain", m.Package.Name)
+		}
+	}
+
+	// One package can legitimately carry several advisories, so this is a
+	// sanity ceiling, not an expected value.
+	if max := 20 * len(scan.SBOM.Packages); scan.Summary.Total > max {
+		t.Errorf("%d findings against %d packages (ceiling %d) — implausible for a distroless base",
+			scan.Summary.Total, len(scan.SBOM.Packages), max)
+	}
+}
+
+// TestVulnDBSBOMAndReportAgree pins the join between the two artifacts a scan
+// publishes. They are produced by different libraries from one catalog and
+// stored as two separate blobs, and nothing downstream reconciles them: the
+// appview renders the SBOM tab from one and the vulnerabilities tab from the
+// other. A report naming a package the SBOM does not list would show a user a
+// vulnerability in something their image demonstrably does not have.
+//
+// Feed-independent: it says nothing about which vulnerabilities were found,
+// only that whatever was found is attributed to a real package at the version
+// the SBOM recorded.
+func TestVulnDBSBOMAndReportAgree(t *testing.T) {
+	primeVulnDB(t)
+	scan := runVulnScan(t, vulnFixtureOld)
+
+	type pkgKey struct{ name, version string }
+	sbomPkgs := map[pkgKey]bool{}
+	sbomNames := map[string]bool{}
+	for _, p := range scan.SBOM.Packages {
+		sbomPkgs[pkgKey{p.Name, p.VersionInfo}] = true
+		sbomNames[p.Name] = true
+	}
+
+	unknownName := map[string]bool{}
+	versionMismatch := map[string]bool{}
+	for _, m := range scan.Report.Matches {
+		if !sbomNames[m.Package.Name] {
+			unknownName[m.Package.Name] = true
+			continue
+		}
+		if !sbomPkgs[pkgKey{m.Package.Name, m.Package.Version}] {
+			versionMismatch[m.Package.Name+"@"+m.Package.Version] = true
+		}
+	}
+
+	if len(unknownName) > 0 {
+		t.Errorf("the vulnerability report names %d packages absent from the SBOM: %s",
+			len(unknownName), strings.Join(sortedKeys(unknownName), ", "))
+	}
+	// A version disagreement is softer: Grype normalises some ecosystems'
+	// versions (epoch handling, for one) and the SPDX encoder writes the raw
+	// syft value. Report it rather than failing, so the first real instance is
+	// visible without this rotting into a false alarm.
+	if len(versionMismatch) > 0 {
+		t.Logf("note: %d matched packages carry a version the SBOM records differently: %s",
+			len(versionMismatch), strings.Join(sortedKeys(versionMismatch), ", "))
+	}
+	t.Logf("%d matches across %d SBOM packages, all attributable", len(scan.Report.Matches), len(scan.SBOM.Packages))
+}
+
+// TestVulnDBReportIsTheShapeTheAppviewReads asserts the encoder's output
+// against the parser that consumes it.
+//
+// This is the failure mode nobody would notice. A report that scans correctly
+// but decodes to nothing renders as an empty vulnerabilities tab with a
+// healthy-looking summary beside it, because the summary comes from the
+// io.atcr.hold.scan record and never from the blob. The appview would show a
+// user "31 vulnerabilities" above an empty table and log nothing.
+//
+// The shape is unusual enough to be worth pinning explicitly: the top-level
+// key is lowercase ("matches"), because grype.go marshals it into a
+// map[string]any literal, while everything nested is PascalCase, because
+// match.Match has no JSON tags at all and Go falls back to field names. Adding
+// tags upstream, or switching to Grype's own presenter model (which is
+// lowercase throughout), would silently break every stored report.
+func TestVulnDBReportIsTheShapeTheAppviewReads(t *testing.T) {
+	primeVulnDB(t)
+	scan := runVulnScan(t, vulnFixtureOld)
+
+	// Parsed via the appview's structs already, in sendVulnScan. That it
+	// parsed is necessary but not sufficient: encoding/json is happy to
+	// produce an empty slice from a document with the wrong key.
+	if len(scan.Report.Matches) == 0 {
+		t.Fatal(`the appview's structs parsed the report into zero matches; the top-level "matches" key is gone`)
+	}
+
+	// Walk the raw JSON for the exact keys the appview's field tags name, so a
+	// rename is caught at the key rather than at the value.
+	var raw struct {
+		Matches []map[string]json.RawMessage `json:"matches"`
+	}
+	if err := json.Unmarshal(scan.Raw, &raw); err != nil {
+		t.Fatalf("parse report: %v", err)
+	}
+	if len(raw.Matches) != len(scan.Report.Matches) {
+		t.Fatalf("raw report has %d matches, typed parse produced %d", len(raw.Matches), len(scan.Report.Matches))
+	}
+	for _, key := range []string{"Vulnerability", "Package"} {
+		if _, ok := raw.Matches[0][key]; !ok {
+			t.Errorf("match objects have no %q key; appview handlers/vuln_details.go cannot read this report", key)
+		}
+	}
+
+	// Fields the templates actually render. Each is named with the template
+	// that would go blank without it.
+	var withMetadata, withFix, withType, cveLinkable int
+	for _, m := range scan.Report.Matches {
+		if m.Vulnerability.Metadata != nil && m.Vulnerability.Metadata.Severity != "" {
+			withMetadata++ // vuln-details.html:84-94, the severity badge
+		}
+		if len(m.Vulnerability.Fix.Versions) > 0 {
+			withFix++ // vuln-details.html:101-105, the "Fix" column
+		}
+		if m.Package.Type != "" {
+			withType++ // vuln-details.html:97, the "(type)" suffix
+		}
+		if strings.HasPrefix(m.Vulnerability.ID, "CVE-") || strings.HasPrefix(m.Vulnerability.ID, "GHSA-") {
+			cveLinkable++ // vuln_details.go:242-246, the CVE URL synthesis
+		}
+	}
+	t.Logf("of %d matches: %d carry a severity, %d a fixed-in version, %d a package type, %d a linkable ID",
+		len(scan.Report.Matches), withMetadata, withFix, withType, cveLinkable)
+
+	if withMetadata == 0 {
+		t.Error("no match carries a severity; every row would render as the grey \"?\" badge")
+	}
+	if withType == 0 {
+		t.Error("no match carries a package type; the Package column loses its qualifier for every row")
+	}
+
+	// The other three top-level keys the encoder writes. Two are unused
+	// downstream, which is itself worth pinning: if they ever start being read,
+	// something has to guarantee they are present.
+	var top map[string]json.RawMessage
+	if err := json.Unmarshal(scan.Raw, &top); err != nil {
+		t.Fatalf("parse report: %v", err)
+	}
+	for _, key := range []string{"matches", "source", "distro", "descriptor", "summary"} {
+		if _, ok := top[key]; !ok {
+			t.Errorf("report is missing top-level key %q", key)
+		}
+	}
+
+	// The report's embedded summary must agree with the one sent over the
+	// wire. Nothing downstream reads the embedded copy — the hold takes the
+	// counts from the separate WebSocket field and the appview takes them from
+	// the record — so a divergence would be invisible in production and would
+	// mean a report that contradicts the badge rendered over it.
+	var embedded struct {
+		Summary scanner.VulnerabilitySummary `json:"summary"`
+	}
+	if err := json.Unmarshal(scan.Raw, &embedded); err != nil {
+		t.Fatalf("parse embedded summary: %v", err)
+	}
+	if embedded.Summary != scan.Summary {
+		t.Errorf("the report's embedded summary %+v disagrees with the one sent to the hold %+v",
+			embedded.Summary, scan.Summary)
+	}
+
+	// The descriptor is what a stored report claims it was produced by.
+	if scan.Digest == "" {
+		t.Error("report descriptor carries no version")
+	} else {
+		t.Logf("report descriptor claims grype %s", scan.Digest)
+	}
+
+	// Size is not an assertion, but it belongs next to the shape. The encoder
+	// marshals whole match.Match values, so each row the appview renders from
+	// five fields ships with the vulnerability's full description, every CVSS
+	// vector, every CPE and every match detail. The appview fetches and parses
+	// this blob on each render of the vulnerabilities tab.
+	t.Logf("report is %d bytes for %d matches (%d bytes per rendered row, of which the "+
+		"appview reads ID, Metadata.Severity, Fix.Versions, Package.Name/Version/Type)",
+		len(scan.Raw), len(scan.Report.Matches), len(scan.Raw)/max(1, len(scan.Report.Matches)))
+}
+
+// TestVulnDBScanIsDeterministic scans the same image twice through one
+// harness, so both scans run against the same loaded provider. The findings
+// must be identical: same summary, same set of (vulnerability, package,
+// version) triples.
+//
+// Feed-independent by construction — a feed refresh between the two scans is
+// impossible, since the provider is loaded once and held in memory.
+//
+// It deliberately does not compare the report digests. See the log line at the
+// end for why.
+func TestVulnDBScanIsDeterministic(t *testing.T) {
+	primeVulnDB(t)
+	requireVulnFixture(t, vulnFixtureOld)
+
+	dir := fixtureDir(vulnFixtureOld)
+	h := Start(t, mockhold.NewOCILayout(dir), WithVulnDB(t))
+
+	first := sendVulnScan(t, h, jobFromLayout(t, dir), vulnFixtureOld)
+	second := sendVulnScan(t, h, jobFromLayout(t, dir), vulnFixtureOld)
+
+	if first.Summary != second.Summary {
+		t.Errorf("two scans of the same image against the same database disagree:\n first: %+v\nsecond: %+v",
+			first.Summary, second.Summary)
+	}
+
+	a, b := first.Findings(), second.Findings()
+	if len(a) != len(b) {
+		t.Fatalf("scan produced %d findings, rescan produced %d", len(a), len(b))
+	}
+	for i := range a {
+		if a[i] != b[i] {
+			t.Errorf("finding %d differs: %s vs %s", i, a[i], b[i])
+		}
+	}
+
+	// Content-identical scans, byte-identical reports? Not necessarily. The
+	// report's digest is what the hold stores as vulnDigest, so a report that
+	// changes when nothing about the image did means every rescan uploads a
+	// fresh blob under a fresh digest. Rather than assert either way — this is
+	// current behaviour, not a requirement — name the top-level keys that
+	// moved, so the cause is in the log instead of in a comment's guess.
+	if string(first.Raw) == string(second.Raw) {
+		t.Logf("the two reports are byte-identical (%d bytes), so a rescan of unchanged content "+
+			"reuses the same vulnDigest", len(first.Raw))
+	} else {
+		var ka, kb map[string]json.RawMessage
+		if err := json.Unmarshal(first.Raw, &ka); err != nil {
+			t.Fatalf("parse first report: %v", err)
+		}
+		if err := json.Unmarshal(second.Raw, &kb); err != nil {
+			t.Fatalf("parse second report: %v", err)
+		}
+		var moved []string
+		for k := range ka {
+			if string(ka[k]) != string(kb[k]) {
+				moved = append(moved, k)
+			}
+		}
+		sort.Strings(moved)
+		t.Logf("the two reports differ byte-for-byte despite identical findings (%d vs %d bytes); "+
+			"the top-level keys that changed are %v, so the vulnDigest the hold records changes "+
+			"on every rescan of unchanged content",
+			len(first.Raw), len(second.Raw), moved)
+		if len(moved) == 1 && moved[0] == "source" {
+			t.Logf("  \"source\" is s.Source from the SBOM, whose Name/Reference is the per-job "+
+				"OCI layout path: %s", firstLineOf(string(ka["source"])))
+		}
+	}
+
+	t.Logf("first scan %s, rescan %s (the difference is the database load, paid once)",
+		first.Elapsed.Round(time.Millisecond), second.Elapsed.Round(time.Millisecond))
+}
+
+// TestVulnDBMatchingCost measures what Grype adds to a scan, which every
+// performance number in SCANNER_BUGS.md section 3 explicitly excludes. It
+// asserts nothing about the numbers — a threshold here would be a flake on
+// slower hardware — it records them.
+//
+// Read the RSS figure with the database in mind: it is a ~2 GB SQLite file
+// opened mmap'd, so resident pages grow as the matcher touches the parts of it
+// the image's packages index into. That interacts directly with the 512 MiB
+// GOMEMLIMIT the scanner sets in cmd/scanner/main.go, which is a soft limit
+// the Go runtime applies to the heap and which does not see mapped file pages
+// at all.
+func TestVulnDBMatchingCost(t *testing.T) {
+	if os.Getenv(perfEnv) != "1" {
+		t.Skipf("set %s=1 (with %s=1) to measure Grype's cost", perfEnv, VulnDBEnv)
+	}
+	primeVulnDB(t)
+
+	t.Log("fixture                pkgs  matches  grypeOff   grypeOn    delta      peakRSS   report")
+	for _, fixture := range severitySweepFixtures(t) {
+		off, offRSS := runPipelineWithoutGrype(t, fixture)
+		scan := runVulnScan(t, fixture)
+		t.Logf("%-22s %-5d %-8d %-10s %-10s %-10s %-9s %s",
+			scan.Fixture, len(scan.SBOM.Packages), len(scan.Report.Matches),
+			off.Round(time.Millisecond),
+			scan.Elapsed.Round(time.Millisecond),
+			(scan.Elapsed - off).Round(time.Millisecond),
+			mib(scan.PeakRSS), mib(uint64(len(scan.Raw))))
+		// The two RSS figures are not an isolation of the database's cost: the
+		// provider is already loaded and its pages already resident by the
+		// time the Grype-off run happens, and nothing unmaps it. The number
+		// that does isolate it is the before/after pair logged by
+		// primeVulnDB, which straddles the only load in the process.
+		t.Logf("  peak RSS %s on the Grype-off run, %s on the Grype-on run (both after the "+
+			"database was loaded and its pages made resident)", mib(offRSS), mib(scan.PeakRSS))
+	}
+}
+
+// runPipelineWithoutGrype runs the same image through the same pipeline with
+// vuln.enabled false, which is the baseline every performance figure in
+// SCANNER_BUGS.md section 3 was measured against. The difference between this
+// and the Grype-enabled scan is the matching cost those figures excluded.
+func runPipelineWithoutGrype(t *testing.T, fixture string) (time.Duration, uint64) {
+	t.Helper()
+
+	var elapsed time.Duration
+	var peak uint64
+	t.Run(fixture+"/no-grype", func(t *testing.T) {
+		dir := fixtureDir(fixture)
+		h := Start(t, mockhold.NewOCILayout(dir))
+		job := jobFromLayout(t, dir)
+
+		stopRSS := watchRSS()
+		start := time.Now()
+		seq, err := h.Hold.SendJob(job)
+		if err != nil {
+			t.Fatalf("send job: %v", err)
+		}
+		msg := h.AwaitTerminal(t, seq, 15*time.Minute)
+		elapsed = time.Since(start)
+		peak = stopRSS()
+		if msg.Type != "result" {
+			t.Fatalf("%s: want result, got %s: %s%s", fixture, msg.Type, msg.Error, msg.Reason)
+		}
+	})
+	return elapsed, peak
+}
+
+// ---------------------------------------------------------------------------
+// Small helpers
+// ---------------------------------------------------------------------------
+
+func sortedKeys(m map[string]bool) []string {
+	out := make([]string, 0, len(m))
+	for k := range m {
+		out = append(out, k)
+	}
+	sort.Strings(out)
+	return out
+}
+
+func formatHistogram(hist map[string]int) string {
+	keys := make([]string, 0, len(hist))
+	for k := range hist {
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)
+	parts := make([]string, 0, len(keys))
+	for _, k := range keys {
+		parts = append(parts, fmt.Sprintf("%s=%d", k, hist[k]))
+	}
+	return strings.Join(parts, " ")
+}
+
+// firstLineOf collapses a pretty-printed JSON fragment onto one line and
+// truncates it to something a log line can carry.
+func firstLineOf(s string) string {
+	s = strings.Join(strings.Fields(s), " ")
+	if len(s) > 240 {
+		return s[:240] + "…"
+	}
+	return s
+}

scanner/internal/mockhold/blobsource.go

@@ -0,0 +1,144 @@
+// Package mockhold provides an in-process stand-in for the hold service so
+// scanner tests can drive the real scanner binary through failure scenarios
+// that are impractical to provoke against a live hold: mid-scan disconnects,
+// truncated blobs, stalled downloads, and artifact shapes the real hold's
+// dispatch guards currently refuse to send.
+//
+// The scanner touches exactly three hold endpoints, and this package serves
+// all three:
+//
+//	GET /xrpc/io.atcr.hold.subscribeScanJobs   WebSocket, jobs out / acks in
+//	GET /xrpc/com.atproto.sync.getBlob         returns {"url": "..."}
+//	GET /blobs/{hex}                           the bytes that URL points at
+//
+// Blob bytes come from a BlobSource, so the same scenario can run against a
+// real image pulled with skopeo or against synthetic bytes built in the test.
+package mockhold
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+)
+
+// ErrBlobNotFound is returned by a BlobSource that has no bytes for a digest.
+// The mock hold turns this into a 404 from getBlob, which is what a real hold
+// returns for a blob that was garbage collected out from under a queued job.
+var ErrBlobNotFound = errors.New("mockhold: blob not found")
+
+// BlobSource supplies blob bytes by digest. Implementations must be safe for
+// concurrent use: the scanner downloads a config and every layer in sequence,
+// but several workers may be scanning at once.
+//
+// Size is returned separately from the reader so the mock can set
+// Content-Length. A source that does not know the size ahead of time (a
+// generated stream, say) may return -1, and the mock will use chunked
+// encoding.
+type BlobSource interface {
+	Open(digest string) (io.ReadCloser, int64, error)
+}
+
+// DigestHex extracts the hex portion of a digest string, mirroring the
+// scanner's own digestHex so a source keyed either way resolves the same.
+func DigestHex(digest string) string {
+	if _, hex, ok := strings.Cut(digest, ":"); ok {
+		return hex
+	}
+	return digest
+}
+
+// OCILayout serves blobs out of an OCI image layout directory, which is what
+// `skopeo copy docker://<ref> oci:<dir>:<tag>` writes. The layout stores blobs
+// at blobs/sha256/<hex>, keyed by exactly the digest the scan job references,
+// so a layout pulled from a real registry can back a scan with no rewriting.
+type OCILayout struct {
+	Dir string
+}
+
+// NewOCILayout returns a source reading from an OCI layout directory.
+func NewOCILayout(dir string) *OCILayout { return &OCILayout{Dir: dir} }
+
+// Open implements BlobSource.
+func (o *OCILayout) Open(digest string) (io.ReadCloser, int64, error) {
+	path := filepath.Join(o.Dir, "blobs", "sha256", DigestHex(digest))
+
+	// Refuse a digest that escapes the layout. Digests are attacker-controlled
+	// in the sense that a test may deliberately feed a malformed one, and a
+	// traversal would read outside the fixture rather than failing the way a
+	// real hold would.
+	if !strings.HasPrefix(filepath.Clean(path), filepath.Clean(o.Dir)+string(os.PathSeparator)) {
+		return nil, 0, fmt.Errorf("%w: %q escapes the layout", ErrBlobNotFound, digest)
+	}
+
+	f, err := os.Open(path)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil, 0, fmt.Errorf("%w: %s", ErrBlobNotFound, digest)
+		}
+		return nil, 0, err
+	}
+
+	fi, err := f.Stat()
+	if err != nil {
+		f.Close()
+		return nil, 0, err
+	}
+
+	return f, fi.Size(), nil
+}
+
+// Memory serves blobs from an in-memory map keyed by digest. Use it for
+// synthetic content: a hand-built tar layer, a gzip bomb, bytes that do not
+// hash to the digest claiming them.
+type Memory struct {
+	blobs map[string][]byte
+}
+
+// NewMemory returns an empty in-memory source.
+func NewMemory() *Memory { return &Memory{blobs: make(map[string][]byte)} }
+
+// Add stores bytes under a digest. The digest is stored by hex, so callers may
+// pass either "sha256:abc" or "abc".
+func (m *Memory) Add(digest string, data []byte) *Memory {
+	m.blobs[DigestHex(digest)] = data
+	return m
+}
+
+// Open implements BlobSource.
+func (m *Memory) Open(digest string) (io.ReadCloser, int64, error) {
+	data, ok := m.blobs[DigestHex(digest)]
+	if !ok {
+		return nil, 0, fmt.Errorf("%w: %s", ErrBlobNotFound, digest)
+	}
+	return io.NopCloser(bytes.NewReader(data)), int64(len(data)), nil
+}
+
+// Multi tries each source in order and returns the first hit. It lets one
+// scenario span several skopeo-pulled layouts, or overlay synthetic bytes on
+// top of a real image to corrupt exactly one layer.
+type Multi struct {
+	sources []BlobSource
+}
+
+// NewMulti returns a source that consults each of sources in order.
+func NewMulti(sources ...BlobSource) *Multi { return &Multi{sources: sources} }
+
+// Open implements BlobSource.
+func (m *Multi) Open(digest string) (io.ReadCloser, int64, error) {
+	for _, s := range m.sources {
+		rc, n, err := s.Open(digest)
+		if err == nil {
+			return rc, n, nil
+		}
+		// A source that failed for a reason other than "I don't have it" is a
+		// real fault and must not be masked by a later source happening to.
+		if !errors.Is(err, ErrBlobNotFound) {
+			return nil, 0, err
+		}
+	}
+	return nil, 0, fmt.Errorf("%w: %s", ErrBlobNotFound, digest)
+}

scanner/internal/mockhold/corpus.go

@@ -0,0 +1,177 @@
+package mockhold
+
+import (
+	_ "embed"
+	"encoding/json"
+	"fmt"
+	"sort"
+	"strings"
+
+	scanner "atcr.io/scanner"
+)
+
+//go:embed testdata/corpus.json
+var corpusJSON []byte
+
+// Artifact shapes present in the corpus. The scanner treats each differently,
+// and three of the four are shapes the real hold's dispatch guards currently
+// refuse to send, which is precisely why they are worth being able to send.
+const (
+	// ShapeImage is an ordinary container image: a config plus tar layers.
+	// The only shape the scanner is expected to scan through to an SBOM.
+	ShapeImage = "image"
+
+	// ShapeAttestation is an in-toto or DSSE payload carried by a manifest
+	// with an ordinary image config. The config media type alone does not
+	// identify it, which is what let one reach production.
+	ShapeAttestation = "attestation"
+
+	// ShapeHelm is a Helm chart, identified by its config media type.
+	ShapeHelm = "helm"
+
+	// ShapeIndex is a multi-arch manifest list: no config, no layers, just a
+	// manifests array. Both of the hold's enqueue paths filter these out via
+	// HasScannableContent, so the scanner never sees one in production. Fed
+	// one directly it does not skip: skipReason returns empty (no config
+	// media type to match, and the layer check is skipped when there are no
+	// layers), then buildOCILayout fails with "config blob has empty digest",
+	// which is a retryable error rather than a permanent skip.
+	ShapeIndex = "index"
+
+	// ShapeReferrer is an artifact with a subject pointing at another
+	// manifest. Also filtered by HasScannableContent.
+	ShapeReferrer = "referrer"
+)
+
+// Manifest is one real io.atcr.manifest record, reduced to the fields a scan
+// job carries. Blob bytes are not included: descriptors are enough for every
+// scenario that does not run Syft, and the ones that do pair the corpus with
+// an OCILayout pulled by testdata/fetch-blobs.sh.
+type Manifest struct {
+	Digest     string                   `json:"digest"`
+	Repository string                   `json:"repository"`
+	Shape      string                   `json:"shape"`
+	MediaType  string                   `json:"mediaType"`
+	HoldDID    string                   `json:"holdDid"`
+	Config     *scanner.BlobDescriptor  `json:"config"`
+	Layers     []scanner.BlobDescriptor `json:"layers"`
+	Manifests  []scanner.BlobDescriptor `json:"manifests,omitempty"`
+	Subject    *scanner.BlobDescriptor  `json:"subject,omitempty"`
+}
+
+type corpusDoc struct {
+	SourceRepo   string     `json:"sourceRepo"`
+	SourceHandle string     `json:"sourceHandle"`
+	Manifests    []Manifest `json:"manifests"`
+}
+
+// Corpus returns every manifest in the embedded fixture.
+//
+// These are real records fetched anonymously from a live PDS, not
+// hand-written shapes, so the distribution reflects what a hold actually
+// holds: mostly ordinary images, a substantial minority of buildx
+// attestations, a handful of indexes and Helm charts.
+func Corpus() ([]Manifest, error) {
+	var doc corpusDoc
+	if err := json.Unmarshal(corpusJSON, &doc); err != nil {
+		return nil, fmt.Errorf("mockhold: parse corpus: %w", err)
+	}
+	return doc.Manifests, nil
+}
+
+// CorpusByShape returns the manifests matching one of the Shape constants.
+func CorpusByShape(shape string) ([]Manifest, error) {
+	all, err := Corpus()
+	if err != nil {
+		return nil, err
+	}
+	var out []Manifest
+	for _, m := range all {
+		if m.Shape == shape {
+			out = append(out, m)
+		}
+	}
+	return out, nil
+}
+
+// Job builds a scan job from a manifest record, the same conversion the hold
+// performs in dispatchCandidate. HoldEndpoint is left empty so SendJob points
+// it at the mock.
+func (m Manifest) Job() *scanner.ScanJob {
+	job := &scanner.ScanJob{
+		ManifestDigest: m.Digest,
+		Repository:     m.Repository,
+		Tag:            "latest",
+		HoldDID:        m.HoldDID,
+		Tier:           "deckhand",
+		Layers:         m.Layers,
+	}
+	if m.Config != nil {
+		job.Config = *m.Config
+	}
+	return job
+}
+
+// Digests returns every blob digest a scan of this manifest will request, in
+// the order buildOCILayout asks for them: config first, then layers. Tests use
+// it to build a Memory source, or to assert which blobs were actually fetched.
+func (m Manifest) Digests() []string {
+	var out []string
+	if m.Config != nil && m.Config.Digest != "" {
+		out = append(out, m.Config.Digest)
+	}
+	for _, l := range m.Layers {
+		if l.Digest != "" {
+			out = append(out, l.Digest)
+		}
+	}
+	return out
+}
+
+// Signature identifies a manifest's media-type shape: its config media type
+// plus the distinct set of layer media types. It is what the scanner's
+// skipReason actually branches on, so two manifests sharing a signature are
+// indistinguishable to the pipeline.
+func (m Manifest) Signature() string {
+	cfg := "(none)"
+	if m.Config != nil {
+		cfg = m.Config.MediaType
+	}
+
+	seen := map[string]bool{}
+	var types []string
+	for _, l := range m.Layers {
+		if seen[l.MediaType] {
+			continue
+		}
+		seen[l.MediaType] = true
+		types = append(types, l.MediaType)
+	}
+	sort.Strings(types)
+
+	return cfg + " | " + strings.Join(types, ",")
+}
+
+// Representatives returns one manifest per distinct Signature, preserving
+// corpus order.
+//
+// The corpus is real production data, so its distribution is lopsided: 16 of
+// the 84 manifests are buildx attestations of exactly the same shape. Running
+// all 16 through the pipeline tests one code path 16 times, and at the
+// production JobCooldown that costs two and a half minutes to learn nothing
+// the first one did not already establish. Scenarios that care about breadth
+// of shape should use this; scenarios that care about volume should say so
+// explicitly and set their own cooldown.
+func Representatives(manifests []Manifest) []Manifest {
+	seen := map[string]bool{}
+	var out []Manifest
+	for _, m := range manifests {
+		sig := m.Signature()
+		if seen[sig] {
+			continue
+		}
+		seen[sig] = true
+		out = append(out, m)
+	}
+	return out
+}

scanner/internal/mockhold/mockhold.go

@@ -0,0 +1,473 @@
+package mockhold
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"sync"
+	"sync/atomic"
+	"time"
+
+	scanner "atcr.io/scanner"
+	"github.com/gorilla/websocket"
+)
+
+// DropMode selects how DropConnections severs a scanner's WebSocket.
+//
+// The distinction is not cosmetic. The scanner's read loop passes errors
+// through websocket.IsUnexpectedCloseError, which treats CloseNormalClosure
+// and CloseGoingAway as expected and logs anything else as an error. Only
+// DropAbrupt resembles what a proxy timeout or an evicted pod does in
+// production, so a scenario that only ever closes cleanly tests the easy half.
+type DropMode int
+
+const (
+	// DropAbrupt closes the underlying TCP connection with no close frame.
+	DropAbrupt DropMode = iota
+	// DropClean sends a normal-closure frame before closing.
+	DropClean
+)
+
+// Message is a message received from the scanner. It mirrors the hold's own
+// ScannerMessage; the scanner module cannot import pkg/hold/pds without
+// dragging go-libsql into the scanner binary, so the shape is restated here.
+type Message struct {
+	Type       string                        `json:"type"`
+	Seq        int64                         `json:"seq"`
+	SBOM       string                        `json:"sbom,omitempty"`
+	VulnReport string                        `json:"vulnReport,omitempty"`
+	Summary    *scanner.VulnerabilitySummary `json:"summary,omitempty"`
+	Error      string                        `json:"error,omitempty"`
+	Reason     string                        `json:"reason,omitempty"`
+
+	// At records arrival time. Several scenarios assert on timing rather than
+	// content: how long the scanner waited before redialing, or whether a
+	// result landed before or after a disconnect.
+	At time.Time `json:"-"`
+}
+
+// BlobRequest records one blob fetch. Counting these is how a test proves the
+// scanner downloaded the same image twice after a reconnect, which no message
+// in the transcript would reveal on its own.
+type BlobRequest struct {
+	Digest string
+	At     time.Time
+}
+
+// Hold is an in-process stand-in for the hold service.
+type Hold struct {
+	// Secret, when non-empty, is required as the ?secret= query parameter on
+	// the subscribe endpoint, matching the real hold's ValidateScannerSecret.
+	Secret string
+
+	blobs  BlobSource
+	srv    *httptest.Server
+	seq    atomic.Int64
+	closed atomic.Bool
+
+	mu           sync.Mutex
+	conns        []*websocket.Conn
+	transcript   []Message
+	blobReqs     []BlobRequest
+	dials        []time.Time
+	msgWaiters   []*msgWaiter
+	connWaiters  []chan struct{}
+	presignHook  func(digest string) (string, bool)
+	blobRespHook func(w http.ResponseWriter, r *http.Request, digest string) bool
+}
+
+// msgWaiter is one in-flight WaitForMessage call. Waiters stay registered
+// until the call returns, rather than being consumed by the first delivery:
+// a scanner emits an ack before its terminal message, so a waiter that
+// unregistered on the first non-matching message would sleep through the one
+// it was waiting for.
+type msgWaiter struct {
+	ch chan Message
+}
+
+// Option configures a Hold at construction.
+type Option func(*Hold)
+
+// WithSecret requires the scanner to present this shared secret.
+func WithSecret(secret string) Option {
+	return func(h *Hold) { h.Secret = secret }
+}
+
+// WithPresignHook installs a hook consulted by getBlob before the default
+// response. Returning ok==false makes getBlob answer 404, standing in for a
+// blob the hold can no longer resolve. Returning a URL redirects the scanner
+// somewhere else entirely, which is how a test points one layer at a slow or
+// broken server while the rest come from the fixture.
+func WithPresignHook(fn func(digest string) (string, bool)) Option {
+	return func(h *Hold) { h.presignHook = fn }
+}
+
+// WithBlobResponseHook installs a hook that may write the blob response
+// itself. Returning true means the hook handled the request; the default
+// BlobSource path is then skipped. Use it to truncate, stall, or return a
+// 500 partway through a body.
+func WithBlobResponseHook(fn func(w http.ResponseWriter, r *http.Request, digest string) bool) Option {
+	return func(h *Hold) { h.blobRespHook = fn }
+}
+
+// New starts a mock hold serving blobs from the given source. Call Close when
+// finished. A nil source answers every getBlob with 404, which is a valid
+// configuration for scenarios that never reach a download.
+func New(blobs BlobSource, opts ...Option) *Hold {
+	h := &Hold{blobs: blobs}
+	for _, opt := range opts {
+		opt(h)
+	}
+
+	mux := http.NewServeMux()
+	mux.HandleFunc("/xrpc/io.atcr.hold.subscribeScanJobs", h.handleSubscribe)
+	mux.HandleFunc("/xrpc/com.atproto.sync.getBlob", h.handleGetBlob)
+	mux.HandleFunc("/blobs/", h.handleBlob)
+	h.srv = httptest.NewServer(mux)
+
+	return h
+}
+
+// URL returns the base URL of the mock hold, suitable for both the scanner's
+// hold.url config and a scan job's HoldEndpoint.
+func (h *Hold) URL() string { return h.srv.URL }
+
+// Close shuts down the server and severs any live scanner connections.
+func (h *Hold) Close() {
+	if !h.closed.CompareAndSwap(false, true) {
+		return
+	}
+	h.DropConnections(DropAbrupt)
+	h.srv.Close()
+}
+
+// --- WebSocket -------------------------------------------------------------
+
+var upgrader = websocket.Upgrader{
+	CheckOrigin: func(*http.Request) bool { return true },
+}
+
+func (h *Hold) handleSubscribe(w http.ResponseWriter, r *http.Request) {
+	if h.Secret != "" && r.URL.Query().Get("secret") != h.Secret {
+		http.Error(w, "invalid scanner secret", http.StatusUnauthorized)
+		return
+	}
+
+	conn, err := upgrader.Upgrade(w, r, nil)
+	if err != nil {
+		return
+	}
+
+	h.mu.Lock()
+	h.conns = append(h.conns, conn)
+	h.dials = append(h.dials, time.Now())
+	waiters := h.connWaiters
+	h.connWaiters = nil
+	h.mu.Unlock()
+
+	for _, ch := range waiters {
+		close(ch)
+	}
+
+	go h.readLoop(conn)
+}
+
+func (h *Hold) readLoop(conn *websocket.Conn) {
+	defer func() {
+		conn.Close()
+		h.mu.Lock()
+		for i, c := range h.conns {
+			if c == conn {
+				h.conns = append(h.conns[:i], h.conns[i+1:]...)
+				break
+			}
+		}
+		h.mu.Unlock()
+	}()
+
+	for {
+		_, data, err := conn.ReadMessage()
+		if err != nil {
+			return
+		}
+
+		var msg Message
+		if err := json.Unmarshal(data, &msg); err != nil {
+			continue
+		}
+		msg.At = time.Now()
+
+		h.mu.Lock()
+		h.transcript = append(h.transcript, msg)
+		waiters := append([]*msgWaiter(nil), h.msgWaiters...)
+		h.mu.Unlock()
+
+		// Buffered, non-blocking: a waiter that has already returned must not
+		// wedge the read loop, and losing a message for such a waiter is
+		// harmless since nobody is listening.
+		for _, w := range waiters {
+			select {
+			case w.ch <- msg:
+			default:
+			}
+		}
+	}
+}
+
+// SendJob dispatches a scan job to a connected scanner and returns the seq it
+// was assigned. HoldEndpoint is filled in automatically so blob downloads come
+// back to this server; an explicitly set HoldEndpoint is left alone.
+//
+// Unlike the real hold there is no round-robin, no persistence, and no
+// assignment state: the job goes to the first live connection. Anything a test
+// needs to assert about pending/assigned/processing bookkeeping belongs in a
+// hold-side test against the real ScanBroadcaster, not here.
+func (h *Hold) SendJob(job *scanner.ScanJob) (int64, error) {
+	if job.Seq == 0 {
+		job.Seq = h.seq.Add(1)
+	}
+	if job.HoldEndpoint == "" {
+		job.HoldEndpoint = h.srv.URL
+	}
+
+	h.mu.Lock()
+	if len(h.conns) == 0 {
+		h.mu.Unlock()
+		return 0, errors.New("mockhold: no scanner connected")
+	}
+	conn := h.conns[0]
+	h.mu.Unlock()
+
+	// The scanner decodes into ScanJobRaw, where Config and Layers are raw
+	// JSON, so marshal them separately rather than relying on ScanJob's own
+	// shape lining up with the wire format.
+	configJSON, err := json.Marshal(job.Config)
+	if err != nil {
+		return 0, fmt.Errorf("marshal config: %w", err)
+	}
+	layersJSON, err := json.Marshal(job.Layers)
+	if err != nil {
+		return 0, fmt.Errorf("marshal layers: %w", err)
+	}
+
+	raw := scanner.ScanJobRaw{
+		Type:           "job",
+		Seq:            job.Seq,
+		ManifestDigest: job.ManifestDigest,
+		Repository:     job.Repository,
+		Tag:            job.Tag,
+		UserDID:        job.UserDID,
+		UserHandle:     job.UserHandle,
+		HoldDID:        job.HoldDID,
+		HoldEndpoint:   job.HoldEndpoint,
+		Tier:           job.Tier,
+		Config:         configJSON,
+		Layers:         layersJSON,
+	}
+
+	data, err := json.Marshal(raw)
+	if err != nil {
+		return 0, fmt.Errorf("marshal job: %w", err)
+	}
+	if err := conn.WriteMessage(websocket.TextMessage, data); err != nil {
+		return 0, fmt.Errorf("write job: %w", err)
+	}
+	return job.Seq, nil
+}
+
+// NextSeq allocates a sequence number from the same counter SendJob uses, so
+// a hand-built raw frame cannot collide with a job sent the normal way.
+func (h *Hold) NextSeq() int64 { return h.seq.Add(1) }
+
+// SendRaw writes bytes to the first connected scanner verbatim, with no
+// marshaling and no validation.
+//
+// SendJob always produces a well-formed frame, which makes it useless for the
+// half of the protocol surface that matters most: what the scanner does with
+// a frame it cannot parse. Those paths (bad JSON, a config that is not an
+// object, an absent layers field) are reachable in production from any hold
+// whose stored config_json/layers_json columns disagree with what the scanner
+// expects, and the scanner's response to them is to say nothing at all.
+func (h *Hold) SendRaw(data []byte) error {
+	h.mu.Lock()
+	if len(h.conns) == 0 {
+		h.mu.Unlock()
+		return errors.New("mockhold: no scanner connected")
+	}
+	conn := h.conns[0]
+	h.mu.Unlock()
+
+	if err := conn.WriteMessage(websocket.TextMessage, data); err != nil {
+		return fmt.Errorf("write raw frame: %w", err)
+	}
+	return nil
+}
+
+// DropConnections severs every live scanner connection. The scanner's Connect
+// loop redials on its own, so a test drops and then waits for the next dial.
+func (h *Hold) DropConnections(mode DropMode) {
+	h.mu.Lock()
+	conns := append([]*websocket.Conn(nil), h.conns...)
+	h.mu.Unlock()
+
+	for _, c := range conns {
+		if mode == DropClean {
+			_ = c.WriteControl(websocket.CloseMessage,
+				websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""),
+				time.Now().Add(time.Second))
+			c.Close()
+			continue
+		}
+		// Abrupt: kill the TCP connection with no close frame at all.
+		_ = c.UnderlyingConn().Close()
+	}
+}
+
+// --- Observation -----------------------------------------------------------
+
+// Transcript returns every message received so far, in arrival order.
+func (h *Hold) Transcript() []Message {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	return append([]Message(nil), h.transcript...)
+}
+
+// BlobRequests returns every blob fetch so far, in arrival order.
+func (h *Hold) BlobRequests() []BlobRequest {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	return append([]BlobRequest(nil), h.blobReqs...)
+}
+
+// Dials returns the time of each scanner connection, which is how a test
+// measures reconnect cadence.
+func (h *Hold) Dials() []time.Time {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	return append([]time.Time(nil), h.dials...)
+}
+
+// WaitForScanner blocks until a scanner is connected or the timeout elapses.
+func (h *Hold) WaitForScanner(timeout time.Duration) error {
+	h.mu.Lock()
+	if len(h.conns) > 0 {
+		h.mu.Unlock()
+		return nil
+	}
+	ch := make(chan struct{})
+	h.connWaiters = append(h.connWaiters, ch)
+	h.mu.Unlock()
+
+	select {
+	case <-ch:
+		return nil
+	case <-time.After(timeout):
+		return errors.New("mockhold: timed out waiting for scanner to connect")
+	}
+}
+
+// WaitForMessage blocks until a message satisfying match arrives, or the
+// timeout elapses. Messages already in the transcript are considered first, so
+// a test cannot lose a race against a fast scanner.
+func (h *Hold) WaitForMessage(match func(Message) bool, timeout time.Duration) (Message, error) {
+	h.mu.Lock()
+	for _, m := range h.transcript {
+		if match(m) {
+			h.mu.Unlock()
+			return m, nil
+		}
+	}
+	w := &msgWaiter{ch: make(chan Message, 256)}
+	h.msgWaiters = append(h.msgWaiters, w)
+	h.mu.Unlock()
+
+	defer func() {
+		h.mu.Lock()
+		for i, cand := range h.msgWaiters {
+			if cand == w {
+				h.msgWaiters = append(h.msgWaiters[:i], h.msgWaiters[i+1:]...)
+				break
+			}
+		}
+		h.mu.Unlock()
+	}()
+
+	deadline := time.After(timeout)
+	for {
+		select {
+		case m := <-w.ch:
+			if match(m) {
+				return m, nil
+			}
+		case <-deadline:
+			return Message{}, errors.New("mockhold: timed out waiting for message")
+		}
+	}
+}
+
+// --- Blobs -----------------------------------------------------------------
+
+// handleGetBlob answers com.atproto.sync.getBlob with the URL the scanner
+// should download from. The real hold returns a presigned S3 URL; here it
+// points back at this server, or wherever a presign hook sends it.
+func (h *Hold) handleGetBlob(w http.ResponseWriter, r *http.Request) {
+	digest := r.URL.Query().Get("cid")
+	if digest == "" {
+		http.Error(w, "missing cid", http.StatusBadRequest)
+		return
+	}
+
+	if h.presignHook != nil {
+		url, ok := h.presignHook(digest)
+		if !ok {
+			http.Error(w, "blob not found", http.StatusNotFound)
+			return
+		}
+		writeJSON(w, map[string]string{"url": url})
+		return
+	}
+
+	writeJSON(w, map[string]string{"url": h.srv.URL + "/blobs/" + DigestHex(digest)})
+}
+
+func (h *Hold) handleBlob(w http.ResponseWriter, r *http.Request) {
+	digest := r.URL.Path[len("/blobs/"):]
+
+	h.mu.Lock()
+	h.blobReqs = append(h.blobReqs, BlobRequest{Digest: digest, At: time.Now()})
+	h.mu.Unlock()
+
+	if h.blobRespHook != nil && h.blobRespHook(w, r, digest) {
+		return
+	}
+
+	if h.blobs == nil {
+		http.Error(w, "no blob source", http.StatusNotFound)
+		return
+	}
+
+	rc, size, err := h.blobs.Open(digest)
+	if err != nil {
+		if errors.Is(err, ErrBlobNotFound) {
+			http.Error(w, "blob not found", http.StatusNotFound)
+			return
+		}
+		http.Error(w, err.Error(), http.StatusInternalServerError)
+		return
+	}
+	defer rc.Close()
+
+	w.Header().Set("Content-Type", "application/octet-stream")
+	if size >= 0 {
+		w.Header().Set("Content-Length", fmt.Sprintf("%d", size))
+	}
+	_, _ = io.Copy(w, rc)
+}
+
+func writeJSON(w http.ResponseWriter, v any) {
+	w.Header().Set("Content-Type", "application/json")
+	_ = json.NewEncoder(w).Encode(v)
+}

scanner/internal/mockhold/testdata/.gitignore

@@ -0,0 +1,2 @@
+blobs/
+vulndb/

scanner/internal/mockhold/testdata/fetch-blobs.sh

@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+# Pull real image blobs into OCI layouts that mockhold.OCILayout can serve.
+#
+# The layouts land in testdata/blobs/<name>/ and are gitignored: they are
+# megabytes of container layers, and every test that needs them skips cleanly
+# when they are absent. Descriptor-only scenarios (disconnect, reconnect,
+# skip classification, oversize) need nothing from here.
+#
+# Why skopeo rather than the scanner's own download path: `skopeo copy ...
+# oci:<dir>` writes blobs/sha256/<hex>, keyed by exactly the digest the scan
+# job references. That is the same layout buildOCILayout reconstructs at scan
+# time, so a pulled image drops straight in with no rewriting.
+#
+# Auth: the registry host must be mapped to the credential helper. seamark.cr
+# is NOT a registry you can log into by default (its handle appears under the
+# atcr.io appview, which is a different thing); buoy.cr works out of the box
+# if `docker-credential-atcr status` lists an account for it. The script writes
+# its own authfile so it never touches ~/.docker/config.json.
+#
+# Usage:
+#   ./fetch-blobs.sh                 # pull the default fixture set
+#   REGISTRY=atcr.io ./fetch-blobs.sh
+set -euo pipefail
+
+REGISTRY="${REGISTRY:-buoy.cr}"
+ACCOUNT="${ACCOUNT:-evan.jarrett.net}"
+DIR="$(cd "$(dirname "$0")" && pwd)/blobs"
+
+# name|repository|manifest digest
+# Digests come from corpus.json. Keep these small: the point is real layer
+# bytes for Syft to catalog, not coverage of every image on the hold.
+FIXTURES=(
+  "hsm-secrets-operator|hsm-secrets-operator|sha256:1cfa4e2b09e127b9c4ed43578d3f3c18e7d44ea47b9ea98475c0cbe9086525f8"
+)
+
+command -v skopeo >/dev/null || { echo "skopeo not found"; exit 1; }
+
+AUTHFILE=$(mktemp); trap 'rm -f "$AUTHFILE"' EXIT
+printf '{"credHelpers":{"%s":"atcr"}}\n' "$REGISTRY" > "$AUTHFILE"
+
+mkdir -p "$DIR"
+for entry in "${FIXTURES[@]}"; do
+  IFS='|' read -r name repo digest <<< "$entry"
+  dest="$DIR/$name"
+
+  if [ -f "$dest/oci-layout" ]; then
+    echo "→ $name already present, skipping"
+    continue
+  fi
+
+  echo "→ pulling ${REGISTRY}/${ACCOUNT}/${repo}@${digest:0:19}..."
+  rm -rf "$dest"
+  if ! skopeo copy --authfile "$AUTHFILE" \
+      "docker://${REGISTRY}/${ACCOUNT}/${repo}@${digest}" \
+      "oci:${dest}:img"; then
+    echo "   FAILED. If this is an auth error, check:"
+    echo "     docker-credential-atcr status"
+    echo "   and confirm an account is configured for ${REGISTRY}."
+    rm -rf "$dest"
+    exit 1
+  fi
+done
+
+echo
+echo "→ layouts in $DIR:"
+du -sh "$DIR"/*/ 2>/dev/null || true

scanner/internal/mockhold/testdata/fetch-corpus.sh

@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+# Regenerate corpus.json: real io.atcr.manifest records from a live PDS.
+#
+# Manifest records ARE the scan job payload. The config and layer descriptors
+# the hold sends to the scanner are stored verbatim in the user's PDS, and
+# com.atproto.repo.listRecords serves them without authentication. So the
+# corpus needs no registry credentials and no blob bytes, only a DID.
+#
+# Blob bytes are a separate concern; see fetch-blobs.sh.
+#
+# Usage:
+#   ./fetch-corpus.sh                      # default DID below
+#   ./fetch-corpus.sh did:plc:xxxx         # some other repo
+set -euo pipefail
+
+DID="${1:-did:plc:pddp4xt5lgnv2qsegbzzs4xg}"
+OUT="$(cd "$(dirname "$0")" && pwd)/corpus.json"
+
+# Resolve the DID's PDS from its DID document.
+PDS=$(curl -sS --max-time 20 "https://plc.directory/${DID}" \
+  | python3 -c 'import json,sys; d=json.load(sys.stdin); print(next(s["serviceEndpoint"] for s in d["service"] if s["id"]=="#atproto_pds"))')
+HANDLE=$(curl -sS --max-time 20 "https://plc.directory/${DID}" \
+  | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["alsoKnownAs"][0].replace("at://",""))')
+
+echo "→ repo   ${DID} (${HANDLE})"
+echo "→ pds    ${PDS}"
+
+TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
+: > "$TMP/all.jsonl"
+cursor=""
+while :; do
+  url="${PDS}/xrpc/com.atproto.repo.listRecords?repo=${DID}&collection=io.atcr.manifest&limit=100"
+  [ -n "$cursor" ] && url="${url}&cursor=${cursor}"
+  curl -sS --max-time 30 "$url" > "$TMP/page.json"
+  cursor=$(python3 -c '
+import json,sys
+d=json.load(open(sys.argv[1]))
+with open(sys.argv[2],"a") as f:
+    for r in d.get("records",[]): f.write(json.dumps(r["value"])+"\n")
+print(d.get("cursor",""))' "$TMP/page.json" "$TMP/all.jsonl")
+  [ -z "$cursor" ] && break
+done
+
+python3 - "$TMP/all.jsonl" "$OUT" "$DID" "$HANDLE" <<'PY'
+import json, sys, collections
+src, out, did, handle = sys.argv[1:5]
+
+records = [json.loads(l) for l in open(src)]
+seen, uniq = set(), []
+for v in records:
+    if v.get("digest") in seen:
+        continue
+    seen.add(v.get("digest"))
+    uniq.append(v)
+
+def shape(v):
+    layers = v.get("layers") or []
+    if v.get("subject"):
+        return "referrer"
+    if not layers and v.get("manifests"):
+        return "index"
+    if "helm" in (v.get("config") or {}).get("mediaType", ""):
+        return "helm"
+    media = {l.get("mediaType", "") for l in layers}
+    if any("in-toto" in m or "dsse" in m for m in media):
+        return "attestation"
+    if any("tar" in m for m in media):
+        return "image"
+    return "other"
+
+def desc(d):
+    return None if not d else {
+        "digest": d.get("digest"), "size": d.get("size"), "mediaType": d.get("mediaType")}
+
+entries = []
+for v in sorted(uniq, key=lambda v: (shape(v), v.get("repository") or "", v["digest"])):
+    e = {
+        "digest": v["digest"],
+        "repository": v.get("repository"),
+        "shape": shape(v),
+        "mediaType": v.get("mediaType"),
+        "holdDid": v.get("holdDid"),
+        "config": desc(v.get("config")),
+        "layers": [desc(l) for l in (v.get("layers") or [])],
+    }
+    if v.get("manifests"):
+        e["manifests"] = [desc(m) for m in v["manifests"]]
+    if v.get("subject"):
+        e["subject"] = desc(v["subject"])
+    entries.append(e)
+
+json.dump({
+    "_comment": ("Real io.atcr.manifest records, fetched anonymously via "
+                 "com.atproto.repo.listRecords. Regenerate with "
+                 "testdata/fetch-corpus.sh. Descriptors only, no blob bytes."),
+    "sourceRepo": did,
+    "sourceHandle": handle,
+    "manifests": entries,
+}, open(out, "w"), indent=2)
+
+print(f"→ wrote {out}: {len(entries)} manifests")
+for s, n in sorted(collections.Counter(e["shape"] for e in entries).items()):
+    print(f"    {s:<12} {n}")
+PY

scanner/internal/mockhold/testdata/fetch-vulndb.sh

@@ -0,0 +1,225 @@
+#!/usr/bin/env bash
+# Download the real Grype vulnerability database, plus the pinned images it is
+# matched against, so scan REPORTS can be verified and not just the pipeline
+# mechanics.
+#
+# Everything else in this testdata directory is descriptors and layer bytes.
+# This is the other half: without a real database, Grype's matcher, its
+# configuration, and every vulnerability count the scanner publishes are
+# entirely unexercised. The e2e scenarios in vulnreport_test.go are the ones
+# that need it, and they skip cleanly when it is absent.
+#
+# DISK COST: ~2.0 GB in testdata/vulndb/6/vulnerability.db (uncompressed
+# SQLite, mmap'd at match time), plus a transient ~350 MB archive and its
+# decompression scratch under testdata/vulndb/.tmp during the download. The
+# image fixtures add ~32 MB. Budget 3 GB free before running. Both directories
+# are gitignored.
+#
+# TIME: about 30-60s on a fast link; the download itself is a few hundred MB.
+#
+# FRESHNESS: grypeDBConfig (scanner/internal/scan/grype.go) sets
+# ValidateAge:true with MaxAllowedBuiltAge of 14 days, so a cached database
+# goes stale and then fails to load outright. Re-running this script refreshes
+# it: the same Grype curator the scanner uses checks the upstream listing and
+# downloads a newer build when the local one is superseded. Re-running on a
+# fresh database is a cheap no-op (one listing fetch).
+#
+# Usage:
+#   ./fetch-vulndb.sh            # download/refresh the database and the images
+#   ./fetch-vulndb.sh --force    # delete and re-download the database
+#   ./fetch-vulndb.sh --status   # report the cached database's build age only
+#   ./fetch-vulndb.sh --db-only  # skip the image pulls
+#
+# Why this shells into Go rather than curling the archive: the curator writes
+# import.json alongside the database during activation, and the scanner's
+# installation.Config sets ValidateChecksum:true. A hand-extracted archive is
+# missing that file and fails to load with "no import metadata file at ...",
+# which is a real production failure mode (SCANNER_BUGS.md section 6, finding
+# 6) and not something a fixture should reproduce by accident. Calling
+# grype.LoadVulnerabilityDB with the scanner's exact config is the only way to
+# land bytes on disk in the state the scanner expects to find them in.
+set -euo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+MODULE_ROOT="$(cd "$HERE/../../.." && pwd)"   # scanner/
+DB_DIR="${GRYPE_DB_DIR:-$HERE/vulndb}"
+TMP_DIR="$DB_DIR/.tmp"
+
+BLOB_DIR="$HERE/blobs"
+
+# name|reference. Pinned, public, and small. Each earns its place:
+#
+#   vuln-alpine310  alpine:3.10, end of life since 2021. The guarantee that a
+#                   scan finds something, whatever the feed says today.
+#   vuln-debian11   debian:11-slim. Debian's security tracker rates advisories
+#                   "Negligible" and "Unknown", which Alpine's does not, and
+#                   the severity-bucket consistency check gives a different
+#                   answer on each. One distro is not a sample.
+#   vuln-static     distroless static. The control: a handful of dpkg entries,
+#                   no shell, no libc. A matcher inventing results has nothing
+#                   here to invent them from.
+#
+# Tags rather than digests: these are the upstream's own frozen tags for
+# released images, and pinning a digest here would mean an unpullable fixture
+# the first time a registry re-pushes a manifest list.
+IMAGE_FIXTURES=(
+  "vuln-alpine310|docker://docker.io/library/alpine:3.10"
+  "vuln-debian11|docker://docker.io/library/debian:11-slim"
+  "vuln-static|docker://gcr.io/distroless/static-debian12:latest"
+)
+
+FORCE=0
+STATUS_ONLY=0
+DB_ONLY=0
+for arg in "$@"; do
+  case "$arg" in
+    --force)   FORCE=1 ;;
+    --status)  STATUS_ONLY=1 ;;
+    --db-only) DB_ONLY=1 ;;
+    *) echo "unknown argument: $arg" >&2; exit 2 ;;
+  esac
+done
+
+command -v go >/dev/null || { echo "go not found" >&2; exit 1; }
+
+if [ "$STATUS_ONLY" = 1 ]; then
+  if [ -f "$DB_DIR/6/import.json" ]; then
+    echo "→ cached database at $DB_DIR"
+    # import.json carries no build timestamp of its own; the archive name in
+    # its "source" field does, and that is the value Grype's ValidateAge check
+    # runs against.
+    built=$(grep -o '_[0-9]\{4\}-[0-9-]*T[0-9:]*Z_' "$DB_DIR/6/import.json" | head -1 | tr -d _)
+    echo "   built: ${built:-unknown}"
+    if [ -n "$built" ]; then
+      age=$(( ( $(date -u +%s) - $(date -u -d "$built" +%s) ) / 86400 ))
+      echo "   age:   ${age}d (Grype refuses to load one built more than 14d ago)"
+    fi
+    du -sh "$DB_DIR"
+  else
+    echo "→ no database at $DB_DIR; run this script with no arguments"
+  fi
+  echo "→ image fixtures:"
+  for entry in "${IMAGE_FIXTURES[@]}"; do
+    name="${entry%%|*}"
+    if [ -f "$BLOB_DIR/$name/oci-layout" ]; then
+      echo "   present: $name ($(du -sh "$BLOB_DIR/$name" | cut -f1))"
+    else
+      echo "   missing: $name"
+    fi
+  done
+  exit 0
+fi
+
+if [ "$FORCE" = 1 ]; then
+  echo "→ removing $DB_DIR"
+  rm -rf "$DB_DIR"
+fi
+
+mkdir -p "$TMP_DIR"
+
+# The loader is written to a scratch package under the module so `go run`
+# resolves grype from scanner/go.mod — the same version the scanner links, so
+# the on-disk schema is by construction the one it can open.
+RUNNER="$MODULE_ROOT/.fetch-vulndb.$$"
+mkdir -p "$RUNNER"
+trap 'rm -rf "$RUNNER"' EXIT
+
+cat > "$RUNNER/main.go" <<'GO'
+package main
+
+import (
+	"fmt"
+	"os"
+	"time"
+
+	"github.com/anchore/grype/grype"
+	"github.com/anchore/grype/grype/db/v6/distribution"
+	"github.com/anchore/grype/grype/db/v6/installation"
+)
+
+// Mirrors grypeDBConfig in scanner/internal/scan/grype.go. Keep in sync: a
+// database fetched under different settings is not the fixture the scanner
+// would have produced for itself.
+func main() {
+	root := os.Args[1]
+	if err := os.MkdirAll(root, 0o755); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+	start := time.Now()
+	provider, status, err := grype.LoadVulnerabilityDB(
+		distribution.DefaultConfig(),
+		installation.Config{
+			DBRootDir:          root,
+			ValidateAge:        true,
+			ValidateChecksum:   true,
+			MaxAllowedBuiltAge: 14 * 24 * time.Hour,
+		},
+		true, // update: check upstream, download when superseded, then open
+	)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "grype database load failed: %v\n", err)
+		os.Exit(1)
+	}
+	defer provider.Close()
+	fmt.Printf("built=%s age=%s schema=%s\npath=%s\nelapsed=%s\n",
+		status.Built.Format(time.RFC3339),
+		time.Since(status.Built).Round(time.Minute),
+		status.SchemaVersion,
+		status.Path,
+		time.Since(start).Round(time.Second))
+}
+GO
+
+echo "→ fetching Grype database into $DB_DIR"
+(
+  cd "$MODULE_ROOT"
+  # Same reason WorkerPool.Start exports TMPDIR in production: go-getter's zstd
+  # decompression is 1 GB+ and must not land on a small tmpfs.
+  TMPDIR="$TMP_DIR" go run "./$(basename "$RUNNER")" "$DB_DIR"
+)
+
+rm -rf "$TMP_DIR"
+
+if [ "$DB_ONLY" = 0 ]; then
+  if ! command -v skopeo >/dev/null; then
+    echo "→ skopeo not found; skipping the image fixtures. Install it, or pull them by hand:" >&2
+    for entry in "${IMAGE_FIXTURES[@]}"; do
+      echo "     skopeo copy --override-os linux --override-arch amd64 ${entry#*|} oci:$BLOB_DIR/${entry%%|*}:img" >&2
+    done
+  else
+    mkdir -p "$BLOB_DIR"
+    for entry in "${IMAGE_FIXTURES[@]}"; do
+      name="${entry%%|*}"
+      ref="${entry#*|}"
+      dest="$BLOB_DIR/$name"
+      if [ -f "$dest/oci-layout" ]; then
+        echo "→ $name already present, skipping"
+        continue
+      fi
+      echo "→ pulling $ref"
+      rm -rf "$dest"
+      # Pin the platform: an unqualified pull of a manifest list on a non-amd64
+      # host would produce a fixture whose findings differ from everyone
+      # else's, and the scenarios compare against no golden file precisely so
+      # that stays a non-issue.
+      if ! skopeo copy --override-os linux --override-arch amd64 "$ref" "oci:${dest}:img"; then
+        echo "   FAILED to pull $ref" >&2
+        rm -rf "$dest"
+        exit 1
+      fi
+    done
+  fi
+fi
+
+echo
+echo "→ on disk:"
+du -sh "$DB_DIR"
+for entry in "${IMAGE_FIXTURES[@]}"; do
+  d="$BLOB_DIR/${entry%%|*}"
+  [ -d "$d" ] && du -sh "$d"
+done
+echo
+echo "Run the scenarios that use it with:"
+echo "  cd $MODULE_ROOT && ATCR_SCANNER_VULNDB=1 go test ./internal/e2e -run TestVulnDB -v"
+echo "Add ATCR_SCANNER_PERF=1 for the Grype cost measurement."

scanner/internal/scan/extractor.go

@@ -3,6 +3,7 @@ package scan
 import (
 	"crypto/sha256"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"os"
@@ -46,7 +47,15 @@ type ociIndex struct {
 //	    ├── <manifest-hex>
 //	    ├── <config-hex>
 //	    └── <layer-hex>...
-func buildOCILayout(job *scanner.ScanJob, tmpDir, secret string) (string, func(), error) {
+//
+// Every blob is verified against its descriptor as it streams, and every
+// digest is validated before it becomes a path, so the layout describes bytes
+// that are on disk and hash to the names they are filed under.
+//
+// maxBytes is the ceiling on the total transferred for this job; zero or less
+// means unlimited. It is spent down blob by blob, so it bounds the real bytes
+// on disk rather than the numbers the manifest record claims.
+func buildOCILayout(job *scanner.ScanJob, tmpDir, secret string, maxBytes int64) (string, func(), error) {
 	scanDir, err := os.MkdirTemp(tmpDir, "scan-*")
 	if err != nil {
 		return "", nil, fmt.Errorf("failed to create temp directory: %w", err)
@@ -64,58 +73,54 @@ func buildOCILayout(job *scanner.ScanJob, tmpDir, secret string) (string, func()
 		return "", nil, fmt.Errorf("failed to create blobs directory: %w", err)
 	}
 
-	// Download config blob
 	if job.Config.Digest == "" {
 		cleanup()
 		return "", nil, fmt.Errorf("config blob has empty digest, cannot download")
 	}
-	slog.Info("Downloading config blob", "digest", job.Config.Digest)
-	if err := downloadBlob(job, job.Config.Digest, blobsDir, secret); err != nil {
-		cleanup()
-		return "", nil, fmt.Errorf("failed to download config blob: %w", err)
-	}
 
-	// Download layer blobs (no extraction — kept compressed)
-	for i, layer := range job.Layers {
-		if layer.Digest == "" {
-			slog.Warn("Skipping layer with empty digest", "index", i)
-			continue
-		}
-		// Skip non-tar layers (cosign signatures, in-toto attestations, etc.)
-		if layer.MediaType != "" && !strings.Contains(layer.MediaType, "tar") {
-			slog.Info("Skipping non-tar layer", "index", i, "digest", layer.Digest, "mediaType", layer.MediaType)
-			continue
-		}
-		slog.Info("Downloading layer", "index", i, "digest", layer.Digest, "size", layer.Size, "mediaType", layer.MediaType)
-		if err := downloadBlob(job, layer.Digest, blobsDir, secret); err != nil {
-			cleanup()
-			return "", nil, fmt.Errorf("failed to download layer %d: %w", i, err)
-		}
+	// Download every referenced blob and describe it in the same pass, so the
+	// layout can only ever list a blob that arrived and was verified, at the
+	// size it actually arrived in. Verification has already established that a
+	// declared size, where the record gave one, matches; recording the
+	// measured figure keeps the layout honest for the descriptors that
+	// declared none.
+	remaining := int64(-1) // -1 is unbounded
+	if maxBytes > 0 {
+		remaining = maxBytes
 	}
-
-	// Build OCI manifest from job descriptors
 	manifest := ociManifest{
 		SchemaVersion: 2,
 		MediaType:     "application/vnd.oci.image.manifest.v1+json",
-		Config: ociDescriptor{
-			MediaType: defaultMediaType(job.Config.MediaType, "application/vnd.oci.image.config.v1+json"),
-			Digest:    job.Config.Digest,
-			Size:      job.Config.Size,
-		},
-		Layers: make([]ociDescriptor, 0, len(job.Layers)),
+		Layers:        make([]ociDescriptor, 0, len(job.Layers)),
 	}
-	for _, layer := range job.Layers {
-		if layer.Digest == "" {
+	for _, ref := range referencedBlobs(job) {
+		digest, err := scanner.ParseDigest(ref.Descriptor.Digest)
+		if err != nil {
+			cleanup()
+			return "", nil, &SkipError{Reason: fmt.Sprintf("%s: %v", ref.what(), err)}
+		}
+		slog.Info("Downloading blob", "blob", ref.what(), "digest", digest,
+			"declaredSize", ref.Descriptor.Size, "mediaType", ref.Descriptor.MediaType)
+
+		n, err := downloadBlob(job, digest, ref.Descriptor.Size, remaining, blobsDir, secret)
+		if err != nil {
+			cleanup()
+			return "", nil, blobFailure(ref.what(), err)
+		}
+		if remaining >= 0 {
+			remaining -= n
+		}
+
+		d := ociDescriptor{
+			MediaType: defaultMediaType(ref.Descriptor.MediaType, ref.defaultMediaType()),
+			Digest:    digest.String(),
+			Size:      n,
+		}
+		if ref.isConfig() {
+			manifest.Config = d
 			continue
 		}
-		if layer.MediaType != "" && !strings.Contains(layer.MediaType, "tar") {
-			continue
-		}
-		manifest.Layers = append(manifest.Layers, ociDescriptor{
-			MediaType: defaultMediaType(layer.MediaType, "application/vnd.oci.image.layer.v1.tar+gzip"),
-			Digest:    layer.Digest,
-			Size:      layer.Size,
-		})
+		manifest.Layers = append(manifest.Layers, d)
 	}
 
 	// Write manifest blob
@@ -168,24 +173,95 @@ func buildOCILayout(job *scanner.ScanJob, tmpDir, secret string) (string, func()
 	return scanDir, cleanup, nil
 }
 
-// downloadBlob downloads a blob by digest to the blobs directory using presigned URLs.
-func downloadBlob(job *scanner.ScanJob, digest, blobsDir, secret string) error {
-	hex := digestHex(digest)
-	destPath := filepath.Join(blobsDir, hex)
+// blobRef is one blob the layout will contain: the descriptor that named it,
+// plus where it sits in the record so an error can say which blob it means.
+type blobRef struct {
+	Index      int // position in job.Layers; configIndex for the config blob
+	Descriptor scanner.BlobDescriptor
+}
+
+const configIndex = -1
+
+func (r blobRef) isConfig() bool { return r.Index == configIndex }
+
+func (r blobRef) what() string {
+	if r.isConfig() {
+		return "config blob"
+	}
+	return fmt.Sprintf("layer %d", r.Index)
+}
+
+func (r blobRef) defaultMediaType() string {
+	if r.isConfig() {
+		return "application/vnd.oci.image.config.v1+json"
+	}
+	return "application/vnd.oci.image.layer.v1.tar+gzip"
+}
+
+// referencedBlobs returns every blob a scan of this job touches, in the order
+// it touches them: the config first, then the layers that are not dropped.
+//
+// This is the single definition of "which blobs does this job reference".
+// Digest validation, the download loop, the byte budget and the layout
+// manifest all walk this list, so a layer dropped here is dropped everywhere
+// and a layer kept here is one that has been checked.
+func referencedBlobs(job *scanner.ScanJob) []blobRef {
+	refs := make([]blobRef, 0, len(job.Layers)+1)
+	if job.Config.Digest != "" {
+		refs = append(refs, blobRef{Index: configIndex, Descriptor: job.Config})
+	}
+	for i, layer := range job.Layers {
+		if layer.Digest == "" {
+			continue
+		}
+		// Non-tar layers (cosign signatures, in-toto attestations) are not
+		// something Syft can read, so they are never fetched or listed.
+		if layer.MediaType != "" && !strings.Contains(layer.MediaType, "tar") {
+			continue
+		}
+		refs = append(refs, blobRef{Index: i, Descriptor: layer})
+	}
+	return refs
+}
+
+// blobFailure labels a download error, converting the ones that can never
+// succeed on a retry into a SkipError.
+//
+// The hold's stale-scan loop re-offers "error" records on every pass and never
+// re-offers "skipped" ones, so a failure decided by the record itself or by
+// bytes already stored belongs on the skip side. A transport fault (a 5xx, a
+// dropped connection, an expired presigned URL) stays an error, because those
+// really can succeed next time.
+func blobFailure(what string, err error) error {
+	if errors.Is(err, client.ErrBlobCorrupt) || errors.Is(err, client.ErrBlobTooLarge) {
+		return &SkipError{Reason: fmt.Sprintf("%s: %v", what, err)}
+	}
+	return fmt.Errorf("failed to download %s: %w", what, err)
+}
+
+// downloadBlob fetches one validated blob into the blobs directory and returns
+// how many bytes arrived. maxBytes is the remaining job budget, negative for
+// unbounded.
+func downloadBlob(job *scanner.ScanJob, digest scanner.Digest, declaredSize, maxBytes int64, blobsDir, secret string) (int64, error) {
+	destPath := filepath.Join(blobsDir, digest.Hex)
+
+	// The invariant, asserted rather than assumed. ParseDigest has already
+	// constrained Hex to [0-9a-f], so this cannot fire; it is here so that any
+	// future path that reaches os.Create with something less constrained fails
+	// loudly instead of writing outside the scan directory.
+	if filepath.Dir(filepath.Clean(destPath)) != filepath.Clean(blobsDir) {
+		return 0, fmt.Errorf("refusing to write blob %s outside %s", digest, blobsDir)
+	}
 
 	presignedURL, err := client.GetBlobPresignedURL(job.HoldEndpoint, job.HoldDID, digest, secret)
 	if err != nil {
-		return fmt.Errorf("failed to get presigned URL for %s: %w", digest, err)
+		return 0, fmt.Errorf("failed to get presigned URL for %s: %w", digest, err)
 	}
-	return client.DownloadBlob(presignedURL, destPath)
-}
-
-// digestHex extracts the hex portion from a digest string (e.g., "sha256:abc123" → "abc123").
-func digestHex(digest string) string {
-	if _, hex, ok := strings.Cut(digest, ":"); ok {
-		return hex
-	}
-	return digest
+	return client.DownloadBlob(presignedURL, destPath, client.BlobExpectation{
+		Digest:       digest,
+		DeclaredSize: declaredSize,
+		MaxBytes:     maxBytes,
+	})
 }
 
 func defaultMediaType(mediaType, fallback string) string {

scanner/internal/scan/grype.go

@@ -4,9 +4,11 @@ import (
 	"context"
 	"crypto/sha256"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"os"
+	"strings"
 	"sync"
 	"sync/atomic"
 	"time"
@@ -36,6 +38,7 @@ var (
 	vulnDBLock    sync.RWMutex
 	vulnDBBuilt   time.Time    // build timestamp of the current vulnDB (not load time)
 	vulnDBAttempt time.Time    // last time we attempted a (re)load, success or fail
+	vulnDBLastErr error        // why the last attempt failed, replayed to throttled callers
 	vulnDBScans   atomic.Int64 // scan counter for periodic reload
 )
 
@@ -51,6 +54,24 @@ const vulnDBRefreshAge = 7 * 24 * time.Hour
 // every worker serializing through its own download timeout.
 const vulnDBRetryBackoff = 30 * time.Minute
 
+// vulnDBColdRetryBackoff throttles reload attempts when there is no usable
+// provider at all. Nothing can be scanned in that state, so it is worth
+// escaping quickly, but a scanner in it must not run one full download attempt
+// per queued scan while the hold's stale loop keeps re-queueing the failures.
+const vulnDBColdRetryBackoff = 2 * time.Minute
+
+// vulnDBMaxServeAge is the ceiling on the serve-the-old-DB fallback. Past it
+// the provider is refused rather than served, because a scan against a database
+// this old reports "0 critical" with a confidence it has not earned. It matches
+// the MaxAllowedBuiltAge that grypeDBConfig asks Grype to enforce on disk; the
+// in-memory provider never re-checks its own age, so the ceiling has to live
+// here.
+const vulnDBMaxServeAge = 14 * 24 * time.Hour
+
+// vulnDBReloadEvery is the periodic close-and-reopen interval, in scans, that
+// flushes SQLite's page cache and mmap region.
+const vulnDBReloadEvery = 50
+
 // scanVulnerabilities scans an SBOM for vulnerabilities using Grype
 func scanVulnerabilities(ctx context.Context, s *sbom.SBOM, vulnDBPath string) ([]byte, string, scanner.VulnerabilitySummary, error) {
 	slog.Info("Scanning for vulnerabilities with Grype")
@@ -166,51 +187,150 @@ func grypeDBConfig(vulnDBPath string) (distribution.Config, installation.Config)
 // real download.
 var loadVulnDB = grype.LoadVulnerabilityDB
 
-// loadVulnDatabase loads the Grype vulnerability database with caching and
-// automatic refresh. The cached DB is returned if loaded less than
-// vulnDBRefreshAge ago. On a stale or missing DB, Grype downloads a fresh copy
-// in the same call (update=true) — a single curator handles everything so
-// there is no chance of a double-curator update+load seeing different state.
+// vulnDBDecision is what a snapshot of the database state says the current
+// caller should do.
+type vulnDBDecision int
+
+const (
+	vulnDBServe    vulnDBDecision = iota // the provider in hand is usable and no reload is due
+	vulnDBReload                         // run a (re)load attempt now
+	vulnDBUnusable                       // nothing usable to serve, and the retry backoff has not elapsed
+)
+
+// vulnDBUsable reports whether a loaded provider may still be scanned against.
+// The provider itself cannot answer this: v6.NewVulnerabilityProvider closes
+// over a reader and nothing else, with no build timestamp and no age check on
+// any query path, which is why a stale-but-loaded DB keeps working and why the
+// ceiling has to be imposed out here.
+func vulnDBUsable(db vulnerability.Provider, built time.Time, now time.Time) bool {
+	return db != nil && now.Sub(built) < vulnDBMaxServeAge
+}
+
+// vulnDBDecide is the whole freshness/backoff policy, as a pure function of a
+// state snapshot. Both the read-lock fast path and the write-lock double-check
+// run it, so they cannot disagree — the previous version had a fast path that
+// returned on exactly the condition the write-lock branch required, which is
+// what made the periodic reload unreachable.
+func vulnDBDecide(db vulnerability.Provider, built, attempt time.Time, now time.Time) vulnDBDecision {
+	if vulnDBUsable(db, built, now) {
+		// Fresh, or stale but inside the retry backoff: either way this
+		// provider is what the scan will use.
+		if now.Sub(built) < vulnDBRefreshAge || now.Sub(attempt) < vulnDBRetryBackoff {
+			return vulnDBServe
+		}
+		return vulnDBReload
+	}
+	// No usable provider. There is no fallback to protect, so the throttle is
+	// the much shorter cold backoff — but it is still a throttle. Without one,
+	// every queued scan runs a complete load attempt (a 30s upstream check plus
+	// up to a 300s download) under the exclusive lock, and the hold's stale-scan
+	// loop feeds the failures straight back in.
+	if !attempt.IsZero() && now.Sub(attempt) < vulnDBColdRetryBackoff {
+		return vulnDBUnusable
+	}
+	return vulnDBReload
+}
+
+// vulnDBUnavailableErr explains a refusal to a caller that was throttled out of
+// attempting its own load, so the hold sees why rather than a bare "no database".
+func vulnDBUnavailableErr(db vulnerability.Provider, built time.Time, lastErr error) error {
+	if lastErr == nil {
+		lastErr = errors.New("no database loaded")
+	}
+	if db != nil {
+		// A provider is in hand but past the ceiling. This is the case worth
+		// naming precisely: it is the difference between "we cannot scan" and
+		// "we scanned against a four-month-old database and found nothing".
+		return fmt.Errorf("vulnerability database is too old to trust (built %s ago, max %s) and the refresh failed: %w",
+			time.Since(built).Round(time.Minute), vulnDBMaxServeAge, lastErr)
+	}
+	return fmt.Errorf("vulnerability database unavailable, retrying in at most %s: %w", vulnDBColdRetryBackoff, lastErr)
+}
+
+// vulnDBCorrupt reports whether a load failure is one that deleting the
+// database directory can fix. An expired database self-heals on its own
+// (curator.Update nils the current description when validateAge fails, so
+// isSupersededBy is unconditionally true and it always re-downloads). A
+// checksum or import-metadata failure does not: the on-disk description stays
+// valid, so an update only lands if the upstream has something strictly newer,
+// and until then every scan fails against a database nothing ever deletes.
+func vulnDBCorrupt(err error) bool {
+	if err == nil {
+		return false
+	}
+	msg := err.Error()
+	return strings.Contains(msg, "import metadata") || strings.Contains(msg, "bad db checksum")
+}
+
+// loadVulnDatabase returns the vulnerability database to scan against,
+// refreshing it when due.
+//
+// The state machine, in one place:
+//
+//   - a usable provider that is fresh, or stale but inside the retry backoff,
+//     is served from the read lock;
+//   - every vulnDBReloadEvery scans that path instead falls through to a
+//     periodic reload, which loads the replacement first and swaps second, so a
+//     failed reload leaves the working provider exactly where it was;
+//   - a stale provider past the backoff triggers a refresh, and if that refresh
+//     fails the provider keeps serving until it crosses vulnDBMaxServeAge;
+//   - past that ceiling, and on a cold start with nothing loaded, the caller
+//     gets an error rather than a silent success — but attempts are throttled
+//     on vulnDBColdRetryBackoff, so a scanner in that state probes occasionally
+//     instead of once per scan, and recovers on its own when the upstream
+//     returns.
 func loadVulnDatabase(ctx context.Context, vulnDBPath string) (vulnerability.Provider, error) {
 	vulnDBLock.RLock()
-	// Fresh, or stale but inside the retry backoff: either way this provider is
-	// what the scan will use, so take the shared lock only. Testing the backoff
-	// here as well as under the write lock matters — once the DB is stale the
-	// freshness test never passes again, and without this every scan would
-	// serialize through the exclusive lock just to reach the same conclusion.
-	if vulnDB != nil && (time.Since(vulnDBBuilt) < vulnDBRefreshAge || time.Since(vulnDBAttempt) < vulnDBRetryBackoff) {
-		vulnDBLock.RUnlock()
-		return vulnDB, nil
-	}
+	db, built, attempt, lastErr := vulnDB, vulnDBBuilt, vulnDBAttempt, vulnDBLastErr
 	vulnDBLock.RUnlock()
 
+	periodic := false
+	switch vulnDBDecide(db, built, attempt, time.Now()) {
+	case vulnDBServe:
+		// Count the scan here, on the path scans actually take. It used to be
+		// counted under the write lock, which a serviceable database never
+		// reaches, so the periodic reload never ran once in production.
+		n := vulnDBScans.Add(1)
+		// Only recycle a database that is genuinely fresh. When the DB is stale
+		// and we are serving it inside the retry backoff, the backoff is the
+		// authority on when to touch the upstream again; a scan counter must not
+		// smuggle an extra download attempt past it during an outage.
+		if n%vulnDBReloadEvery != 0 || time.Since(built) >= vulnDBRefreshAge {
+			return db, nil
+		}
+		periodic = true
+	case vulnDBUnusable:
+		return nil, vulnDBUnavailableErr(db, built, lastErr)
+	}
+
+	// Don't start a database download on behalf of a caller that has already
+	// gone away: the load runs under the exclusive lock and Grype bounds only
+	// its HTTP portion, so an abandoned refresh stalls the whole pool.
+	if err := ctx.Err(); err != nil {
+		if vulnDBUsable(db, built, time.Now()) {
+			return db, nil
+		}
+		return nil, err
+	}
+
 	vulnDBLock.Lock()
 	defer vulnDBLock.Unlock()
 
-	// Double-check after acquiring write lock. Freshness is measured from the
-	// DB's build timestamp, so a load that fell back to a stale-but-valid DB
-	// doesn't earn a fresh cache lease — it stays stale and keeps retrying.
-	if vulnDB != nil && time.Since(vulnDBBuilt) < vulnDBRefreshAge {
-		// Periodic reload: close and reopen DB every 50 scans to flush
-		// SQLite's page cache and mmap region.
-		n := vulnDBScans.Add(1)
-		if n%50 == 0 {
-			slog.Info("Periodic vulnDB reload to release memory", "scans", n)
-			vulnDB.Close()
-			vulnDB = nil
-			// Fall through to reload below
-		} else {
+	if periodic {
+		// A periodic reload must survive the double-check — re-deciding here is
+		// what made it unreachable before. Skip it only if another goroutine
+		// reloaded while we waited for the lock, which flushed the caches anyway.
+		if !vulnDBAttempt.Equal(attempt) {
 			return vulnDB, nil
 		}
-	}
-
-	// DB is stale (or absent). Throttle reload attempts: if we probed recently
-	// and still hold a usable provider, keep serving it rather than having every
-	// worker serialize through its own upstream timeout. The in-memory provider
-	// doesn't re-validate build age on queries, so a stale-but-loaded DB still
-	// scans fine until the upstream recovers.
-	if vulnDB != nil && time.Since(vulnDBAttempt) < vulnDBRetryBackoff {
-		return vulnDB, nil
+		slog.Info("Periodic vulnDB reload to release memory", "scans", vulnDBScans.Load())
+	} else {
+		switch vulnDBDecide(vulnDB, vulnDBBuilt, vulnDBAttempt, time.Now()) {
+		case vulnDBServe:
+			return vulnDB, nil
+		case vulnDBUnusable:
+			return nil, vulnDBUnavailableErr(vulnDB, vulnDBBuilt, vulnDBLastErr)
+		}
 	}
 
 	slog.Info("Loading Grype vulnerability database", "path", vulnDBPath, "tmpdir", os.Getenv("TMPDIR"))
@@ -227,35 +347,69 @@ func loadVulnDatabase(ctx context.Context, vulnDBPath string) (vulnerability.Pro
 	// back to serving the existing DB.
 	vulnDBAttempt = time.Now()
 	store, status, err := loadVulnDB(distConfig, installConfig, true)
+
+	if err != nil && vulnDBCorrupt(err) && !vulnDBUsable(vulnDB, vulnDBBuilt, time.Now()) {
+		// `grype db delete && grype db update`, in-process. Once per attempt
+		// window, so the self-heal cannot become its own retry storm, and only
+		// when there is no usable provider — nothing holds the read lock while
+		// we hold the write lock, but there is no reason to pull files out from
+		// under a database that is still scanning.
+		slog.Warn("Vulnerability database is corrupt; deleting it and re-downloading",
+			"error", err, "path", vulnDBPath)
+		if rmErr := os.RemoveAll(vulnDBPath); rmErr != nil {
+			slog.Error("Failed to delete corrupt vulnerability database", "error", rmErr, "path", vulnDBPath)
+		} else if mkErr := os.MkdirAll(vulnDBPath, 0o755); mkErr != nil {
+			slog.Error("Failed to recreate vulnerability database directory", "error", mkErr, "path", vulnDBPath)
+		} else {
+			store, status, err = loadVulnDB(distConfig, installConfig, true)
+		}
+	}
+
 	if err != nil {
-		// Reload failed (e.g. upstream down and on-disk DB past the 14-day max
-		// age). If we still hold a usable provider, keep serving it until the
-		// backoff elapses; only surface the error on a cold start with no DB.
-		if vulnDB != nil {
+		vulnDBLastErr = err
+		// Load-then-swap: the previous provider is still in place, so a failed
+		// refresh (transient or not, periodic or not) costs nothing.
+		if vulnDBUsable(vulnDB, vulnDBBuilt, time.Now()) {
 			slog.Warn("Vulnerability database reload failed; serving previously loaded DB",
 				"error", err,
 				"built", vulnDBBuilt,
 				"age", time.Since(vulnDBBuilt).Round(time.Minute).String())
 			return vulnDB, nil
 		}
-		return nil, fmt.Errorf("failed to load vulnerability database: %w", err)
+		if vulnDB != nil {
+			slog.Error("Vulnerability database is past the maximum age and could not be refreshed",
+				"error", err,
+				"built", vulnDBBuilt,
+				"age", time.Since(vulnDBBuilt).Round(time.Minute).String(),
+				"max", vulnDBMaxServeAge.String())
+		}
+		return nil, vulnDBUnavailableErr(vulnDB, vulnDBBuilt, err)
 	}
 
+	built = time.Time{}
+	if status != nil {
+		built = status.Built
+	}
 	age := "unknown"
-	if !status.Built.IsZero() {
-		age = time.Since(status.Built).Round(time.Minute).String()
+	if !built.IsZero() {
+		age = time.Since(built).Round(time.Minute).String()
+	}
+	if status != nil {
+		slog.Info("Vulnerability database loaded",
+			"built", built,
+			"age", age,
+			"schemaVersion", status.SchemaVersion,
+			"path", status.Path)
+	} else {
+		slog.Info("Vulnerability database loaded", "built", built, "age", age)
 	}
-	slog.Info("Vulnerability database loaded",
-		"built", status.Built,
-		"age", age,
-		"schemaVersion", status.SchemaVersion,
-		"path", status.Path)
 
-	if vulnDB != nil {
+	if vulnDB != nil && vulnDB != store {
 		vulnDB.Close()
 	}
 	vulnDB = store
-	vulnDBBuilt = status.Built
+	vulnDBBuilt = built
+	vulnDBLastErr = nil
 	return vulnDB, nil
 }
 

scanner/internal/scan/grype_test.go

@@ -33,14 +33,14 @@ func resetVulnDBState(t *testing.T) {
 	original := loadVulnDB
 	t.Cleanup(func() {
 		vulnDBLock.Lock()
-		vulnDB, vulnDBBuilt, vulnDBAttempt = nil, time.Time{}, time.Time{}
+		vulnDB, vulnDBBuilt, vulnDBAttempt, vulnDBLastErr = nil, time.Time{}, time.Time{}, nil
 		vulnDBLock.Unlock()
 		vulnDBScans.Store(0)
 		loadVulnDB = original
 	})
 
 	vulnDBLock.Lock()
-	vulnDB, vulnDBBuilt, vulnDBAttempt = nil, time.Time{}, time.Time{}
+	vulnDB, vulnDBBuilt, vulnDBAttempt, vulnDBLastErr = nil, time.Time{}, time.Time{}, nil
 	vulnDBLock.Unlock()
 	vulnDBScans.Store(0)
 }
@@ -125,11 +125,10 @@ func TestLoadVulnDatabase_FreshDBIsServedWithoutReloading(t *testing.T) {
 // serving, untouched.
 //
 // What this pins, precisely: the behaviour, not either check that implements
-// it. fa1dfb0 tests the backoff twice, once on the read-lock fast path and
-// again under the write lock, and mutation confirms they are redundant for
-// correctness — deleting either one alone leaves this test passing, and only
-// deleting both fails it. That redundancy is deliberate. The fast-path copy
-// exists so a stale DB does not push every scan through the exclusive lock, a
+// it. The policy is one function, vulnDBDecide, consulted twice — once on the
+// read-lock fast path so a stale DB does not push every scan through the
+// exclusive lock, and again under the write lock as the double-check. Deleting
+// either call alone leaves this test passing; the fast-path copy exists for a
 // contention property no unit test can assert without being flaky.
 func TestLoadVulnDatabase_StaleDBIsThrottled(t *testing.T) {
 	resetVulnDBState(t)
@@ -141,16 +140,25 @@ func TestLoadVulnDatabase_StaleDBIsThrottled(t *testing.T) {
 
 	_, calls := stubLoader(t, time.Now(), nil)
 
-	got, err := loadVulnDatabase(context.Background(), t.TempDir())
-	if err != nil {
-		t.Fatalf("loadVulnDatabase: %v", err)
-	}
-	if got != cached {
-		t.Error("the stale provider was replaced during the backoff window")
+	// Enough scans to cross the periodic-reload interval several times. The
+	// backoff outranks it: recycling the provider is memory hygiene, and it
+	// must not smuggle an extra upstream attempt past the throttle during an
+	// outage.
+	for i := 0; i < 3*vulnDBReloadEvery; i++ {
+		got, err := loadVulnDatabase(context.Background(), t.TempDir())
+		if err != nil {
+			t.Fatalf("scan %d: %v", i, err)
+		}
+		if got != cached {
+			t.Fatalf("scan %d: the stale provider was replaced during the backoff window", i)
+		}
 	}
 	if *calls != 0 {
 		t.Errorf("loader called %d times inside the retry backoff, want 0", *calls)
 	}
+	if cached.closed {
+		t.Error("the provider still in use was closed")
+	}
 }
 
 // TestLoadVulnDatabase_RetriesOnceTheBackoffElapses is the other half of the
@@ -225,6 +233,9 @@ func TestLoadVulnDatabase_FailedReloadKeepsServingTheOldDB(t *testing.T) {
 // TestLoadVulnDatabase_ColdStartFailureIsAnError is the one case that must fail
 // loudly. With no provider in hand there is nothing to scan against, and
 // returning success would report every image as clean.
+//
+// The first attempt is the one that reaches the loader; what the ones behind it
+// do is TestColdStart_ThrottlesRepeatedAttempts, in vulndb_refresh_test.go.
 func TestLoadVulnDatabase_ColdStartFailureIsAnError(t *testing.T) {
 	resetVulnDBState(t)
 

scanner/internal/scan/vulndb_refresh_test.go

@@ -0,0 +1,764 @@
+package scan
+
+import (
+	"context"
+	"errors"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"atcr.io/scanner/internal/client"
+	"atcr.io/scanner/internal/config"
+	"atcr.io/scanner/internal/mockhold"
+	"atcr.io/scanner/internal/queue"
+	"github.com/anchore/grype/grype"
+	v6 "github.com/anchore/grype/grype/db/v6"
+	v6dist "github.com/anchore/grype/grype/db/v6/distribution"
+	v6inst "github.com/anchore/grype/grype/db/v6/installation"
+	"github.com/anchore/grype/grype/vulnerability"
+)
+
+// This file covers the production complaint "grype is out of date, please
+// update the database": the vulnerability database stops refreshing and scans
+// start failing, with no way back to health short of a restart.
+//
+// grype_test.go pins the ordinary lifecycle of loadVulnDatabase. What follows
+// pins the failure lifecycle — cold start, upstream outage, an on-disk database
+// that is corrupt rather than merely old, the periodic reload, and the ceiling
+// on how old a database may get before the scanner refuses to vouch for its
+// results.
+//
+// Every test that reaches a real Grype code path does so without network
+// access. The only calls into grype.LoadVulnerabilityDB use update=false, so
+// no listing is fetched and no archive is downloaded; the on-disk database is
+// synthesised locally with the exported v6 low-level writer.
+//
+// These tests share the package-level vulnDB globals with grype_test.go and so
+// must not be run in parallel with anything.
+
+// slowLoader installs a loader that blocks for d before returning. It is how
+// the tests below stand in for a real refresh, which in production is a 30s
+// listing check plus an up-to-300s archive download plus hydration, all of it
+// inside loadVulnDatabase's exclusive lock.
+func slowLoader(t *testing.T, d time.Duration, built time.Time, err error) (*fakeProvider, *int) {
+	t.Helper()
+	loaded := &fakeProvider{name: "slow"}
+	calls := 0
+	var mu sync.Mutex
+	loadVulnDB = func(v6dist.Config, v6inst.Config, bool) (vulnerability.Provider, *vulnerability.ProviderStatus, error) {
+		mu.Lock()
+		calls++
+		mu.Unlock()
+		time.Sleep(d)
+		if err != nil {
+			return nil, nil, err
+		}
+		return loaded, &vulnerability.ProviderStatus{Built: built}, nil
+	}
+	return loaded, &calls
+}
+
+// scriptedLoader installs a loader that returns the given responses in order,
+// repeating the last one once the script runs out. It is how the tests below
+// express "the upstream fails and then recovers" without a network.
+type loaderResponse struct {
+	built time.Time
+	err   error
+}
+
+func scriptedLoader(t *testing.T, responses ...loaderResponse) (*fakeProvider, *int) {
+	t.Helper()
+	loaded := &fakeProvider{name: "scripted"}
+	calls := 0
+	var mu sync.Mutex
+	loadVulnDB = func(v6dist.Config, v6inst.Config, bool) (vulnerability.Provider, *vulnerability.ProviderStatus, error) {
+		mu.Lock()
+		i := calls
+		calls++
+		mu.Unlock()
+		if i >= len(responses) {
+			i = len(responses) - 1
+		}
+		r := responses[i]
+		if r.err != nil {
+			return nil, nil, r.err
+		}
+		return loaded, &vulnerability.ProviderStatus{Built: r.built}, nil
+	}
+	return loaded, &calls
+}
+
+// --- the database lifecycle under failure ---
+
+// TestColdStart_ThrottlesRepeatedAttempts. With no provider in hand the
+// retry backoff must still apply, or every queued scan runs a complete download
+// attempt (30s listing check plus up to a 300s archive fetch) under the
+// exclusive lock while the hold's stale loop keeps re-queueing the failures.
+//
+// The throttled calls must still report an error — the scan genuinely cannot
+// run — but they must not repeat the download.
+func TestColdStart_ThrottlesRepeatedAttempts(t *testing.T) {
+	resetVulnDBState(t)
+
+	_, calls := stubLoader(t, time.Time{}, errors.New("upstream unreachable"))
+
+	const attempts = 5
+	for i := 0; i < attempts; i++ {
+		got, err := loadVulnDatabase(context.Background(), t.TempDir())
+		if err == nil {
+			t.Fatalf("attempt %d: a cold start with no database returned success", i)
+		}
+		if got != nil {
+			t.Fatalf("attempt %d: a provider was returned alongside the error", i)
+		}
+	}
+
+	if *calls != 1 {
+		t.Errorf("loader called %d times across %d cold-start scans, want 1: the retry "+
+			"backoff must apply with no provider in hand, not only with one", *calls, attempts)
+	}
+}
+
+// TestColdStart_ConcurrentWorkersShareOneAttempt is the pool-level consequence
+// of B1. Four workers arriving together must produce one download attempt, not
+// four serialized ones, and must finish in about one attempt's time.
+func TestColdStart_ConcurrentWorkersShareOneAttempt(t *testing.T) {
+	resetVulnDBState(t)
+
+	const workers = 4
+	const loadDelay = 40 * time.Millisecond
+	_, calls := slowLoader(t, loadDelay, time.Time{}, errors.New("upstream unreachable"))
+
+	start := time.Now()
+	var wg sync.WaitGroup
+	for i := 0; i < workers; i++ {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			_, _ = loadVulnDatabase(context.Background(), t.TempDir())
+		}()
+	}
+	wg.Wait()
+	elapsed := time.Since(start)
+
+	if *calls != 1 {
+		t.Errorf("loader called %d times for %d concurrent workers, want 1", *calls, workers)
+	}
+	if max := time.Duration(workers-1) * loadDelay; elapsed >= max {
+		t.Errorf("elapsed %s for %d workers at %s each: the attempts are still being "+
+			"serialized through the exclusive lock", elapsed, workers, loadDelay)
+	}
+}
+
+// TestColdStart_RecoversWhenUpstreamReturns is the other half of the backoff:
+// it has to let go. A scanner that failed its cold start must load the database
+// on its own once the upstream comes back, with no process restart.
+func TestColdStart_RecoversWhenUpstreamReturns(t *testing.T) {
+	resetVulnDBState(t)
+
+	freshBuild := time.Now().Add(-1 * time.Hour)
+	loaded, calls := scriptedLoader(t,
+		loaderResponse{err: errors.New("upstream unreachable")},
+		loaderResponse{built: freshBuild},
+	)
+
+	if _, err := loadVulnDatabase(context.Background(), t.TempDir()); err == nil {
+		t.Fatal("the first cold-start attempt was expected to fail")
+	}
+
+	// Age out the cold-start backoff, standing in for the passage of time.
+	vulnDBLock.Lock()
+	vulnDBAttempt = time.Now().Add(-vulnDBColdRetryBackoff - time.Minute)
+	vulnDBLock.Unlock()
+
+	got, err := loadVulnDatabase(context.Background(), t.TempDir())
+	if err != nil {
+		t.Fatalf("the scanner did not recover once the upstream returned: %v", err)
+	}
+	if got != loaded {
+		t.Error("the newly loaded provider was not adopted")
+	}
+	if *calls != 2 {
+		t.Errorf("loader called %d times, want 2", *calls)
+	}
+	if !vulnDBBuilt.Equal(freshBuild) {
+		t.Errorf("vulnDBBuilt = %v, want %v", vulnDBBuilt, freshBuild)
+	}
+}
+
+// TestPeriodicReload_FiresOnSchedule. The "close and reopen the DB every N
+// scans to flush SQLite's page cache and mmap region" mitigation has to
+// actually run; for most of this file's history it never did. The counter therefore has to live on the path scans take — the
+// read-lock fast path — because a fresh database never reaches the write lock.
+func TestPeriodicReload_FiresOnSchedule(t *testing.T) {
+	resetVulnDBState(t)
+
+	cached := &fakeProvider{name: "cached"}
+	vulnDB = cached
+	vulnDBBuilt = time.Now().Add(-1 * time.Hour) // fresh
+
+	loaded, calls := stubLoader(t, time.Now().Add(-1*time.Hour), nil)
+
+	const scans = 200
+	for i := 0; i < scans; i++ {
+		got, err := loadVulnDatabase(context.Background(), t.TempDir())
+		if err != nil {
+			t.Fatalf("scan %d: %v", i, err)
+		}
+		if got == nil {
+			t.Fatalf("scan %d returned no provider", i)
+		}
+	}
+
+	if want := scans / vulnDBReloadEvery; *calls != want {
+		t.Errorf("loader called %d times in %d fresh scans, want %d (one every %d)",
+			*calls, scans, want, vulnDBReloadEvery)
+	}
+	if !cached.closed {
+		t.Error("the replaced provider was not closed: every periodic reload would leak its predecessor")
+	}
+	if vulnDB != loaded {
+		t.Error("the reloaded provider was not adopted")
+	}
+	if n := vulnDBScans.Load(); n != scans {
+		t.Errorf("vulnDBScans = %d after %d scans, want %d: the counter is not on the path scans take", n, scans, scans)
+	}
+}
+
+// TestPeriodicReload_FailureKeepsTheWorkingProvider is the interaction that
+// makes the reload dangerous to enable on its own. It used to null the provider
+// before loading its replacement, so one transient failure would drop a scanner
+// that was working perfectly into the no-provider state above.
+//
+// Load first, swap second: a failed periodic reload must be a no-op.
+func TestPeriodicReload_FailureKeepsTheWorkingProvider(t *testing.T) {
+	resetVulnDBState(t)
+
+	cached := &fakeProvider{name: "cached"}
+	vulnDB = cached
+	vulnDBBuilt = time.Now().Add(-1 * time.Hour) // fresh
+
+	_, calls := stubLoader(t, time.Time{}, errors.New("upstream blip"))
+
+	for i := 0; i < vulnDBReloadEvery+5; i++ {
+		got, err := loadVulnDatabase(context.Background(), t.TempDir())
+		if err != nil {
+			t.Fatalf("scan %d: a failed periodic reload must not fail the scan: %v", i, err)
+		}
+		if got != cached {
+			t.Fatalf("scan %d: the working provider was dropped for a failed reload", i)
+		}
+	}
+
+	if cached.closed {
+		t.Error("the working provider was closed before its replacement was in hand")
+	}
+	if *calls != 1 {
+		t.Errorf("loader called %d times, want 1", *calls)
+	}
+}
+
+// TestStaleFallback_RefusesPastTheCeiling. Serving a slightly old database
+// beats refusing to scan, which is why the fallback exists. Serving one of
+// unbounded age while reporting success is a different thing: a scanner whose
+// egress is blocked would publish confident "0 critical" verdicts for months.
+//
+// Grype's own MaxAllowedBuiltAge is the natural line, since that is the
+// guarantee grypeDBConfig already asks for.
+func TestStaleFallback_RefusesPastTheCeiling(t *testing.T) {
+	resetVulnDBState(t)
+
+	ancient := &fakeProvider{name: "ancient"}
+	vulnDB = ancient
+	vulnDBBuilt = time.Now().Add(-120 * 24 * time.Hour) // four months old
+
+	_, calls := stubLoader(t, time.Time{}, errors.New("upstream unreachable"))
+
+	got, err := loadVulnDatabase(context.Background(), t.TempDir())
+	if err == nil {
+		t.Fatal("a database built 120 days ago was served with a nil error")
+	}
+	if got != nil {
+		t.Error("a provider past the ceiling was returned alongside the error")
+	}
+	if !strings.Contains(err.Error(), "too old") {
+		t.Errorf("error %q does not say the database is too old to trust", err)
+	}
+	if *calls != 1 {
+		t.Errorf("loader called %d times, want 1", *calls)
+	}
+
+	// And the refusal is throttled like any other failure, rather than running
+	// a download per scan.
+	if _, err := loadVulnDatabase(context.Background(), t.TempDir()); err == nil {
+		t.Fatal("the second scan was served from the same ancient database")
+	}
+	if *calls != 1 {
+		t.Errorf("loader called %d times across two scans, want 1", *calls)
+	}
+}
+
+// TestStaleFallback_ServesInsideTheCeiling pins the other side of the trade: a
+// database that is stale but still inside Grype's max allowed age keeps
+// scanning through an upstream outage.
+func TestStaleFallback_ServesInsideTheCeiling(t *testing.T) {
+	resetVulnDBState(t)
+
+	old := &fakeProvider{name: "old"}
+	vulnDB = old
+	vulnDBBuilt = time.Now().Add(-10 * 24 * time.Hour) // stale, inside the 14-day ceiling
+
+	_, calls := stubLoader(t, time.Time{}, errors.New("upstream unreachable"))
+
+	got, err := loadVulnDatabase(context.Background(), t.TempDir())
+	if err != nil {
+		t.Fatalf("a stale but usable database must keep scanning: %v", err)
+	}
+	if got != old {
+		t.Error("the previously loaded provider was not served")
+	}
+	if old.closed {
+		t.Error("the provider still in use was closed")
+	}
+	if *calls != 1 {
+		t.Errorf("loader called %d times, want 1", *calls)
+	}
+}
+
+// TestCorruptDB_IsDeletedAndRetriedOnce covers the database that is current but
+// whose import metadata was lost — an interrupted activate, a truncated write,
+// a restored volume snapshot. curator.Update only installs something strictly
+// newer, and a checksum failure leaves the on-disk description in place, so
+// such a database never re-downloads: it fails every scan forever and nothing
+// deletes it. Delete it and retry once, which is what `grype db delete && grype
+// db update` does for CLI users.
+func TestCorruptDB_IsDeletedAndRetriedOnce(t *testing.T) {
+	resetVulnDBState(t)
+
+	root := t.TempDir()
+	marker := filepath.Join(root, "6", "vulnerability.db")
+	if err := os.MkdirAll(filepath.Dir(marker), 0o755); err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+	if err := os.WriteFile(marker, []byte("corrupt"), 0o644); err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	freshBuild := time.Now().Add(-1 * time.Hour)
+	loaded, calls := scriptedLoader(t,
+		loaderResponse{err: errors.New("no import metadata file at: " + filepath.Join(root, "6", "import.json"))},
+		loaderResponse{built: freshBuild},
+	)
+
+	got, err := loadVulnDatabase(context.Background(), root)
+	if err != nil {
+		t.Fatalf("a corrupt database must be deleted and re-downloaded, not served as a failure: %v", err)
+	}
+	if got != loaded {
+		t.Error("the re-downloaded provider was not adopted")
+	}
+	if *calls != 2 {
+		t.Fatalf("loader called %d times, want 2 (the load and one retry after the delete)", *calls)
+	}
+	if _, err := os.Stat(marker); !os.IsNotExist(err) {
+		t.Errorf("the corrupt database file still exists at %s (stat err %v)", marker, err)
+	}
+}
+
+// TestCorruptDB_RetriesOnlyOncePerAttempt keeps the self-heal from becoming its
+// own storm: if the retry fails too, the attempt ends there and the backoff
+// takes over.
+func TestCorruptDB_RetriesOnlyOncePerAttempt(t *testing.T) {
+	resetVulnDBState(t)
+
+	root := t.TempDir()
+	_, calls := stubLoader(t, time.Time{}, errors.New("no import metadata file at: "+root+"/6/import.json"))
+
+	if _, err := loadVulnDatabase(context.Background(), root); err == nil {
+		t.Fatal("a database that stays corrupt must still fail the scan")
+	}
+	if *calls != 2 {
+		t.Errorf("loader called %d times, want 2: the delete-and-retry must happen once per attempt", *calls)
+	}
+
+	// And the next scan is throttled rather than repeating the delete-and-retry.
+	if _, err := loadVulnDatabase(context.Background(), root); err == nil {
+		t.Fatal("the second scan unexpectedly succeeded")
+	}
+	if *calls != 2 {
+		t.Errorf("loader called %d times across two scans, want 2", *calls)
+	}
+}
+
+// TestLoadVulnDatabase_HonoursContextCancellation. A shutdown, or a hold that
+// has abandoned the job, must not start a fresh database download that then
+// holds the exclusive lock until Grype's own timeouts expire — up to 300s for
+// the archive alone, unbounded for hydration.
+func TestLoadVulnDatabase_HonoursContextCancellation(t *testing.T) {
+	resetVulnDBState(t)
+
+	_, calls := stubLoader(t, time.Now(), nil)
+
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+
+	if _, err := loadVulnDatabase(ctx, t.TempDir()); !errors.Is(err, context.Canceled) {
+		t.Errorf("loadVulnDatabase err = %v, want context.Canceled", err)
+	}
+	if *calls != 0 {
+		t.Errorf("loader called %d times with an already-cancelled context, want 0", *calls)
+	}
+
+	// A cancelled context must not take a usable database away from a caller
+	// that needs no load at all.
+	cached := &fakeProvider{name: "cached"}
+	vulnDBLock.Lock()
+	vulnDB = cached
+	vulnDBBuilt = time.Now().Add(-1 * time.Hour)
+	vulnDBLock.Unlock()
+
+	got, err := loadVulnDatabase(ctx, t.TempDir())
+	if err != nil {
+		t.Fatalf("a cached provider needs no load and must not be refused: %v", err)
+	}
+	if got != cached {
+		t.Error("the cached provider was not served")
+	}
+}
+
+// TestRefresh_BlocksAllInFlightScans quantifies the third finding: a refresh
+// holds the write lock for the entire download, and scanVulnerabilities holds
+// the read lock across FindMatches. A scan that arrives during a refresh
+// therefore waits out the whole download before it can even begin matching.
+//
+// The wait is measured on vulnDBLock directly rather than through
+// scanVulnerabilities, which would need a real SBOM and a real provider. It is
+// the same lock and the same acquisition scanVulnerabilities performs at
+// grype.go:105.
+func TestRefresh_BlocksAllInFlightScans(t *testing.T) {
+	resetVulnDBState(t)
+
+	old := &fakeProvider{name: "old"}
+	vulnDB = old
+	vulnDBBuilt = time.Now().Add(-10 * 24 * time.Hour)                // stale, so a reload is due
+	vulnDBAttempt = time.Now().Add(-vulnDBRetryBackoff - time.Minute) // backoff expired
+
+	const loadDelay = 150 * time.Millisecond
+	_, calls := slowLoader(t, loadDelay, time.Now(), nil)
+
+	refreshDone := make(chan struct{})
+	go func() {
+		defer close(refreshDone)
+		if _, err := loadVulnDatabase(context.Background(), t.TempDir()); err != nil {
+			t.Errorf("refresh: %v", err)
+		}
+	}()
+
+	// Give the refresher time to take the write lock before the scan tries for
+	// the read lock. Any interleaving still produces a valid measurement; this
+	// only makes the intended one likely.
+	time.Sleep(20 * time.Millisecond)
+
+	start := time.Now()
+	vulnDBLock.RLock()
+	blocked := time.Since(start)
+	vulnDBLock.RUnlock()
+
+	<-refreshDone
+
+	if *calls != 1 {
+		t.Fatalf("loader called %d times, want 1", *calls)
+	}
+	if blocked < loadDelay/2 {
+		t.Skipf("scan acquired the read lock in %s: the refresher had not taken the "+
+			"write lock yet, so this run measured nothing", blocked)
+	}
+	t.Logf("a scan arriving during a refresh waited %s for the read lock; in production "+
+		"the same wait is the full listing check plus archive download plus hydration, "+
+		"which grype bounds only by its 300s UpdateTimeout", blocked)
+}
+
+// --- what Grype itself does with the configuration this package passes it ---
+
+// writeSyntheticDB creates a minimal but genuine Grype v6 database on disk with
+// the given build timestamp, at the layout grypeDBConfig expects
+// (<root>/<ModelVersion>/vulnerability.db) and returns that directory.
+//
+// No import.json is written. That file is produced by the curator's activate
+// step, and its absence is deliberate: writing a valid one needs grype's
+// xxhash digest helper (and a new direct module dependency), and the tests
+// that call into the curator here are about Status(), which reports the age
+// failure whether or not the checksum check also fires.
+func writeSyntheticDB(t *testing.T, root string, built time.Time) string {
+	t.Helper()
+
+	dir := filepath.Join(root, strconv.Itoa(v6.ModelVersion))
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+
+	gdb, err := v6.NewLowLevelDB(filepath.Join(dir, v6.VulnerabilityDBFileName), true, true, false)
+	if err != nil {
+		t.Fatalf("create db: %v", err)
+	}
+	ts := built.UTC().Round(time.Second)
+	err = gdb.Create(&v6.DBMetadata{
+		BuildTimestamp: &ts,
+		Model:          v6.ModelVersion,
+		Revision:       v6.Revision,
+		Addition:       v6.Addition,
+	}).Error
+	if err != nil {
+		t.Fatalf("write metadata: %v", err)
+	}
+	sqlDB, err := gdb.DB()
+	if err != nil {
+		t.Fatalf("unwrap db: %v", err)
+	}
+	if err := sqlDB.Close(); err != nil {
+		t.Fatalf("close db: %v", err)
+	}
+	return dir
+}
+
+// TestGrype_ExpiredOnDiskDBIsAHardError is the production error message,
+// reproduced with no network.
+//
+// grypeDBConfig sets ValidateAge with a 14-day MaxAllowedBuiltAge. Once the
+// on-disk database crosses that line, curator.Status() fails, and
+// grype.LoadVulnerabilityDB returns that failure instead of a provider — after
+// the update attempt, not before it. So an expired database plus an upstream
+// that cannot be reached is a hard load failure, not a degraded scan.
+//
+// The call here uses update=false purely to keep the test off the network.
+// Production calls with update=true, and on the path that matters (the update
+// fails, or finds nothing to install) reaches the identical Status() check.
+func TestGrype_ExpiredOnDiskDBIsAHardError(t *testing.T) {
+	root := t.TempDir()
+	writeSyntheticDB(t, root, time.Now().Add(-20*24*time.Hour))
+
+	distCfg, instCfg := grypeDBConfig(root)
+	provider, _, err := grype.LoadVulnerabilityDB(distCfg, instCfg, false)
+	if err == nil {
+		if provider != nil {
+			provider.Close()
+		}
+		t.Fatal("a database built 20 days ago loaded successfully; MaxAllowedBuiltAge is not being enforced")
+	}
+	if provider != nil {
+		t.Error("a provider was returned alongside the error")
+	}
+
+	// This is Grype's own wording, from curator.validateAge
+	// (grype/db/v6/installation/curator.go:613). It is what reaches the hold,
+	// wrapped twice by grype.go, and what users report as "grype is out of
+	// date, please update the database".
+	if !strings.Contains(err.Error(), "the vulnerability database was built") ||
+		!strings.Contains(err.Error(), "max allowed age") {
+		t.Errorf("unexpected error text %q; the age check may have moved", err)
+	}
+	t.Logf("grype: %v", err)
+}
+
+// TestGrype_FreshDBWithoutImportMetadataIsAlsoAHardError is a second, quieter
+// way into the same symptom, and unlike the age cliff it does not self-heal.
+//
+// grypeDBConfig also sets ValidateChecksum. Status() joins the checksum failure
+// into the same error, so a database with a missing or corrupt import.json
+// fails to load even when its build timestamp is minutes old.
+//
+// The asymmetry is what makes this dangerous. curator.Update() nils out the
+// current description when validateAge fails, and isSupersededBy(nil, ...) is
+// unconditionally true, so an expired database is always re-downloaded. A
+// checksum failure leaves the description in place, so a download only happens
+// if the upstream has something strictly newer. A database that is current but
+// whose import metadata was lost (an interrupted activate, a truncated write, a
+// restored volume snapshot) therefore fails every scan and no update fixes it.
+func TestGrype_FreshDBWithoutImportMetadataIsAlsoAHardError(t *testing.T) {
+	root := t.TempDir()
+	writeSyntheticDB(t, root, time.Now().Add(-1*time.Hour))
+
+	distCfg, instCfg := grypeDBConfig(root)
+	provider, _, err := grype.LoadVulnerabilityDB(distCfg, instCfg, false)
+	if err == nil {
+		if provider != nil {
+			provider.Close()
+		}
+		t.Fatal("a database with no import metadata loaded successfully; ValidateChecksum is not being enforced")
+	}
+	if !strings.Contains(err.Error(), "import metadata") {
+		t.Errorf("unexpected error text %q; expected the checksum/import-metadata failure", err)
+	}
+	t.Logf("grype: %v", err)
+}
+
+// TestGrype_InMemoryProviderDoesNotRevalidateAge checks the claim grype.go
+// leans on at line 210: "the in-memory provider doesn't re-validate build age
+// on queries, so a stale-but-loaded DB still scans fine".
+//
+// The claim holds, and this is the mechanism the serve-the-old-DB fallback
+// depends on. v6.NewVulnerabilityProvider closes over a Reader and an
+// architecture-alias map and nothing else: no Config, no MaxAllowedBuiltAge, no
+// build timestamp. Age is checked by the curator when opening the file on disk,
+// never by the provider that results. Which is also why the fallback has no
+// ceiling: there is nothing in the object that could impose one.
+//
+// The provider is built the way LoadVulnerabilityDB builds it, but without the
+// curator, so no age or checksum gate runs at all.
+func TestGrype_InMemoryProviderDoesNotRevalidateAge(t *testing.T) {
+	root := t.TempDir()
+	// A year old: far past MaxAllowedBuiltAge, which the curator would refuse
+	// and the provider cannot see.
+	built := time.Now().Add(-365 * 24 * time.Hour)
+	dir := writeSyntheticDB(t, root, built)
+
+	rdr, err := v6.NewReader(v6.Config{DBDirPath: dir})
+	if err != nil {
+		t.Fatalf("open reader: %v", err)
+	}
+	defer rdr.Close()
+
+	meta, err := rdr.GetDBMetadata()
+	if err != nil {
+		t.Fatalf("read metadata: %v", err)
+	}
+	if age := time.Since(*meta.BuildTimestamp); age < 300*24*time.Hour {
+		t.Fatalf("test setup drifted: build age is %s", age)
+	}
+
+	provider := v6.NewVulnerabilityProvider(rdr)
+
+	// A query against a year-old database. It answers, rather than reporting
+	// that the database is out of date, and would answer identically at any
+	// age. That is the whole content of the comment at grype.go:210.
+	vulns, err := provider.FindVulnerabilities()
+	if err != nil {
+		t.Fatalf("query: %v", err)
+	}
+	if len(vulns) != 0 {
+		t.Fatalf("synthetic database returned %d vulnerabilities", len(vulns))
+	}
+}
+
+// --- the symptom, through the real worker pool ---
+
+// TestWorkerPool_EveryScanFailsWhileTheDBIsUnloadable drives the whole scan
+// pipeline against a mock hold with the database loader failing, which is what
+// production looks like during the outage. It is here rather than in
+// internal/e2e because loadVulnDB is unexported and the harness cannot stub it.
+//
+// Two things are asserted that the unit tests cannot: the failure reaches the
+// hold as an "error" message (which the hold's stale-scan loop retries, unlike
+// a "skipped"), and the cold-start backoff holds across jobs arriving from the
+// hold — so the hold's retry loop no longer drives one full download attempt
+// per queued scan for as long as the upstream is down.
+func TestWorkerPool_EveryScanFailsWhileTheDBIsUnloadable(t *testing.T) {
+	fixture := filepath.Join("..", "mockhold", "testdata", "blobs", "hsm-secrets-operator")
+	if _, err := os.Stat(filepath.Join(fixture, "oci-layout")); err != nil {
+		t.Skipf("fixture not present; run scanner/internal/mockhold/testdata/fetch-blobs.sh")
+	}
+
+	resetVulnDBState(t)
+	_, calls := stubLoader(t, time.Time{}, errors.New("upstream unreachable"))
+
+	const secret = "test-scanner-secret"
+	hold := mockhold.New(mockhold.NewOCILayout(fixture), mockhold.WithSecret(secret))
+	t.Cleanup(hold.Close)
+
+	cfg := config.DefaultConfig()
+	cfg.Hold.URL = hold.URL()
+	cfg.Hold.Secret = secret
+	cfg.Scanner.Workers = 1
+	cfg.Vuln.Enabled = true // the point of the test
+	cfg.Vuln.DBPath = t.TempDir()
+	cfg.Vuln.TmpDir = t.TempDir()
+
+	// WorkerPool.Start exports TMPDIR process-wide and never restores it.
+	origTmpDir, hadTmpDir := os.LookupEnv("TMPDIR")
+	t.Cleanup(func() {
+		if hadTmpDir {
+			os.Setenv("TMPDIR", origTmpDir)
+			return
+		}
+		os.Unsetenv("TMPDIR")
+	})
+
+	restoreCooldown := JobCooldown
+	JobCooldown = 10 * time.Millisecond
+
+	q := queue.NewJobQueue(cfg.Scanner.QueueSize)
+	c := client.NewHoldClient(cfg.Hold.URL, cfg.Hold.Secret, q)
+	pool := NewWorkerPool(cfg, q, c)
+
+	ctx, cancel := context.WithCancel(context.Background())
+	pool.Start(ctx)
+	go c.Connect()
+	t.Cleanup(func() {
+		cancel()
+		c.Close()
+		q.Close()
+		// Wait for the workers before restoring JobCooldown. The worker loop
+		// reads it on every iteration, so restoring it while a worker is still
+		// running is a genuine data race that -race reports. (The e2e harness
+		// restores it without waiting, and has the same latent race.)
+		pool.Wait()
+		JobCooldown = restoreCooldown
+	})
+
+	if err := hold.WaitForScanner(10 * time.Second); err != nil {
+		t.Fatalf("scanner never connected: %v", err)
+	}
+
+	const target = "sha256:1cfa4e2b09e127b9c4ed43578d3f3c18e7d44ea47b9ea98475c0cbe9086525f8"
+	all, err := mockhold.Corpus()
+	if err != nil {
+		t.Fatalf("load corpus: %v", err)
+	}
+	var manifest mockhold.Manifest
+	for _, m := range all {
+		if m.Digest == target {
+			manifest = m
+			break
+		}
+	}
+	if manifest.Digest == "" {
+		t.Fatalf("digest %s not in corpus", target)
+	}
+
+	// The startup goroutine in WorkerPool.Start also calls the loader once.
+	// Take a baseline after connecting so the per-job count is unambiguous.
+	baseline := *calls
+
+	const jobs = 2
+	for i := 0; i < jobs; i++ {
+		seq, err := hold.SendJob(manifest.Job())
+		if err != nil {
+			t.Fatalf("send job %d: %v", i, err)
+		}
+		msg, err := hold.WaitForMessage(func(m mockhold.Message) bool {
+			return m.Seq == seq && (m.Type == "result" || m.Type == "error" || m.Type == "skipped")
+		}, 3*time.Minute)
+		if err != nil {
+			t.Fatalf("no terminal message for seq %d: %v", seq, err)
+		}
+		if msg.Type != "error" {
+			t.Fatalf("job %d: want error while the vulnerability DB is unloadable, got %s", i, msg.Type)
+		}
+		if !strings.Contains(msg.Error, "failed to load vulnerability database") {
+			t.Errorf("job %d: error %q does not name the database failure", i, msg.Error)
+		}
+		t.Logf("job %d: %q", i, msg.Error)
+	}
+
+	// The jobs still fail — there is genuinely nothing to scan against — but
+	// they fail cheaply. Both arrive inside vulnDBColdRetryBackoff, so at most
+	// one of them reaches the loader.
+	if extra := *calls - baseline; extra > 1 {
+		t.Errorf("loader ran %d times across %d jobs, want at most 1: the cold-start "+
+			"backoff must throttle the hold's retry loop", extra, jobs)
+	}
+}

scanner/internal/scan/worker.go

@@ -9,7 +9,6 @@ import (
 	"log/slog"
 	"os"
 	"runtime"
-	"strings"
 	"sync"
 	"time"
 
@@ -104,6 +103,11 @@ func (wp *WorkerPool) worker(ctx context.Context, id int) {
 			"digest", job.ManifestDigest,
 			"tier", job.Tier)
 
+		// Tell the hold the scan is actually starting. The ack it already has
+		// was sent on receipt, before this job joined the queue, so it cannot
+		// tell queueing from scanning without this.
+		wp.client.SendStarted(job.Seq)
+
 		result, err := wp.processJob(ctx, job)
 		if err != nil {
 			var skipErr *SkipError
@@ -123,10 +127,21 @@ func (wp *WorkerPool) worker(ctx context.Context, id int) {
 		} else {
 			wp.client.SendResult(job.Seq, result)
 
-			slog.Info("Scan job completed",
-				"worker_id", id,
-				"repository", job.Repository,
-				"vulnerabilities", result.Summary.Total)
+			// A nil Summary means Grype never ran (vuln.enabled=false), which
+			// is not the same as "scanned, found nothing". Log the completion
+			// without a count rather than printing a zero the scan never
+			// established.
+			if result.Summary != nil {
+				slog.Info("Scan job completed",
+					"worker_id", id,
+					"repository", job.Repository,
+					"vulnerabilities", result.Summary.Total)
+			} else {
+				slog.Info("Scan job completed",
+					"worker_id", id,
+					"repository", job.Repository,
+					"vulnerabilities", "not scanned")
+			}
 		}
 
 		// Free large scan artifacts and trigger GC before the cooldown
@@ -139,11 +154,20 @@ func (wp *WorkerPool) worker(ctx context.Context, id int) {
 		select {
 		case <-ctx.Done():
 			return
-		case <-time.After(10 * time.Second):
+		case <-time.After(JobCooldown):
 		}
 	}
 }
 
+// JobCooldown is the pause a worker takes after each job so Go's GC can
+// reclaim what Syft and Grype allocated before the next scan starts.
+//
+// It is a variable, and exported, solely so tests in other packages can
+// shorten it: at the production value a scenario that runs a handful of jobs
+// through a single worker spends nearly all its runtime asleep. Production
+// code must not change it.
+var JobCooldown = 10 * time.Second
+
 // unscannable config media types — these are OCI artifacts that aren't
 // container images so Syft/Grype can't analyze their layers.
 var unscannableConfigTypes = map[string]bool{
@@ -153,6 +177,10 @@ var unscannableConfigTypes = map[string]bool{
 }
 
 // skipReason reports why a job cannot be scanned, or "" when it can be.
+//
+// Everything decided here is a permanent property of the manifest record, so
+// everything here is a skip rather than an error: the hold records a skip once
+// and re-offers a failure on every stale pass, forever.
 func skipReason(job *scanner.ScanJob) string {
 	if unscannableConfigTypes[job.Config.MediaType] {
 		return fmt.Sprintf("unscannable artifact type %s", job.Config.MediaType)
@@ -162,21 +190,31 @@ func skipReason(job *scanner.ScanJob) string {
 	// single in-toto or DSSE payload as its layer, so the config media type
 	// alone does not identify it. buildOCILayout drops every non-tar layer,
 	// which would hand Syft an image with nothing in it.
-	if len(job.Layers) > 0 && !hasScannableLayer(job.Layers) {
+	if len(job.Layers) > 0 && !hasScannableLayer(job) {
 		return fmt.Sprintf("no scannable layers (%s)", job.Layers[0].MediaType)
 	}
 
+	// Every digest the scan will use has to be a digest. Each one names a blob
+	// to ask the hold for and a file to write into the layout, and a string
+	// that is neither cannot start being one on a later attempt. Checking here
+	// rejects the job before a single request goes out; buildOCILayout parses
+	// the same digests again because it is what turns them into paths, and
+	// that boundary must hold on its own.
+	for _, ref := range referencedBlobs(job) {
+		if _, err := scanner.ParseDigest(ref.Descriptor.Digest); err != nil {
+			return fmt.Sprintf("%s: %v", ref.what(), err)
+		}
+	}
+
 	return ""
 }
 
-// hasScannableLayer mirrors the layer filter in buildOCILayout: anything that
-// is not a tar of some flavour is not something Syft can read.
-func hasScannableLayer(layers []scanner.BlobDescriptor) bool {
-	for _, layer := range layers {
-		if layer.Digest == "" {
-			continue
-		}
-		if layer.MediaType == "" || strings.Contains(layer.MediaType, "tar") {
+// hasScannableLayer reports whether any layer survives the filter
+// buildOCILayout applies. It asks referencedBlobs rather than repeating the
+// media-type rule, so this answer and the layout can never disagree.
+func hasScannableLayer(job *scanner.ScanJob) bool {
+	for _, ref := range referencedBlobs(job) {
+		if !ref.isConfig() {
 			return true
 		}
 	}
@@ -199,7 +237,13 @@ func (wp *WorkerPool) processJob(ctx context.Context, job *scanner.ScanJob) (*sc
 		return nil, fmt.Errorf("failed to create tmp dir: %w", err)
 	}
 
-	// Check total compressed image size before downloading
+	// The cheap guard: refuse an image that admits to being over the ceiling
+	// before a byte moves. It is only a pre-check, because the sizes it adds up
+	// come from the same user-writable record as the digests; buildOCILayout
+	// enforces the same ceiling against the bytes that actually arrive.
+	//
+	// Either way the verdict is permanent — an image does not shrink — so it is
+	// a skip, not a failure the stale loop will offer back forever.
 	if wp.cfg.Vuln.MaxImageSize > 0 {
 		var totalSize int64
 		for _, layer := range job.Layers {
@@ -207,13 +251,14 @@ func (wp *WorkerPool) processJob(ctx context.Context, job *scanner.ScanJob) (*sc
 		}
 		totalSize += job.Config.Size
 		if totalSize > wp.cfg.Vuln.MaxImageSize {
-			return nil, fmt.Errorf("image too large: %d bytes compressed (limit %d bytes)", totalSize, wp.cfg.Vuln.MaxImageSize)
+			return nil, &SkipError{Reason: fmt.Sprintf(
+				"image too large: %d bytes compressed (limit %d bytes)", totalSize, wp.cfg.Vuln.MaxImageSize)}
 		}
 	}
 
 	// Step 1: Build OCI image layout from hold via presigned URLs
 	slog.Info("Building OCI layout", "repository", job.Repository)
-	ociLayoutDir, cleanup, err := buildOCILayout(job, wp.cfg.Vuln.TmpDir, wp.cfg.Hold.Secret)
+	ociLayoutDir, cleanup, err := buildOCILayout(job, wp.cfg.Vuln.TmpDir, wp.cfg.Hold.Secret, wp.cfg.Vuln.MaxImageSize)
 	if err != nil {
 		return nil, fmt.Errorf("failed to build OCI layout: %w", err)
 	}

scanner/internal/scan/worker_skip_test.go

@@ -1,11 +1,19 @@
 package scan
 
 import (
+	"strings"
 	"testing"
 
 	scanner "atcr.io/scanner"
 )
 
+// hexDigest builds a well-formed sha256 digest from a one-character seed.
+// skipReason validates every digest it will use, so a placeholder like
+// "sha256:config" is now itself a skip and would mask what these cases test.
+func hexDigest(seed rune) string {
+	return "sha256:" + strings.Repeat(string(seed), scanner.HexLen)
+}
+
 // TestSkipReason covers the artifact shapes the scanner must refuse before it
 // spends a download on them. The attestation case is the one that reached
 // production: an in-toto SLSA provenance manifest carries an ordinary image
@@ -65,12 +73,12 @@ func TestSkipReason(t *testing.T) {
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			job := &scanner.ScanJob{
-				Config: scanner.BlobDescriptor{MediaType: tt.configType, Digest: "sha256:config"},
+				Config: scanner.BlobDescriptor{MediaType: tt.configType, Digest: hexDigest('c')},
 			}
 			for i, mt := range tt.layerTypes {
 				job.Layers = append(job.Layers, scanner.BlobDescriptor{
 					MediaType: mt,
-					Digest:    "sha256:layer" + string(rune('a'+i)),
+					Digest:    hexDigest(rune('a' + i)),
 				})
 			}
 
@@ -81,3 +89,83 @@ func TestSkipReason(t *testing.T) {
 		})
 	}
 }
+
+// TestSkipReasonRejectsMalformedDigests pins the other half of skipReason: a
+// digest that is not a digest is a permanent property of the record, so the
+// job is refused here rather than failing later as a retryable error.
+//
+// A malformed digest on a layer the layout would drop anyway is not a reason
+// to refuse the image, and the last case pins that: the check walks exactly the
+// blobs referencedBlobs will fetch.
+func TestSkipReasonRejectsMalformedDigests(t *testing.T) {
+	const layerType = "application/vnd.oci.image.layer.v1.tar+gzip"
+	const configType = "application/vnd.oci.image.config.v1+json"
+
+	tests := []struct {
+		name     string
+		job      *scanner.ScanJob
+		wantSkip bool
+	}{
+		{
+			name: "path traversal in a layer digest",
+			job: &scanner.ScanJob{
+				Config: scanner.BlobDescriptor{MediaType: configType, Digest: hexDigest('c')},
+				Layers: []scanner.BlobDescriptor{{MediaType: layerType, Digest: "sha256:../../../escaped"}},
+			},
+			wantSkip: true,
+		},
+		{
+			name: "path traversal in the config digest",
+			job: &scanner.ScanJob{
+				Config: scanner.BlobDescriptor{MediaType: configType, Digest: "sha256:../../../escaped"},
+				Layers: []scanner.BlobDescriptor{{MediaType: layerType, Digest: hexDigest('a')}},
+			},
+			wantSkip: true,
+		},
+		{
+			name: "no algorithm prefix",
+			job: &scanner.ScanJob{
+				Config: scanner.BlobDescriptor{MediaType: configType, Digest: hexDigest('c')},
+				Layers: []scanner.BlobDescriptor{{MediaType: layerType, Digest: strings.Repeat("ef", 32)}},
+			},
+			wantSkip: true,
+		},
+		{
+			name: "unsupported algorithm",
+			job: &scanner.ScanJob{
+				Config: scanner.BlobDescriptor{MediaType: configType, Digest: hexDigest('c')},
+				Layers: []scanner.BlobDescriptor{{MediaType: layerType, Digest: "sha512:" + strings.Repeat("cd", 64)}},
+			},
+			wantSkip: true,
+		},
+		{
+			name: "an empty layer digest is dropped, not refused",
+			job: &scanner.ScanJob{
+				Config: scanner.BlobDescriptor{MediaType: configType, Digest: hexDigest('c')},
+				Layers: []scanner.BlobDescriptor{
+					{MediaType: layerType, Digest: ""},
+					{MediaType: layerType, Digest: hexDigest('a')},
+				},
+			},
+		},
+		{
+			name: "a dropped non-tar layer's digest is not checked",
+			job: &scanner.ScanJob{
+				Config: scanner.BlobDescriptor{MediaType: configType, Digest: hexDigest('c')},
+				Layers: []scanner.BlobDescriptor{
+					{MediaType: "application/vnd.oci.image.layer.v1.zstd+odd", Digest: "not a digest"},
+					{MediaType: layerType, Digest: hexDigest('a')},
+				},
+			},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			reason := skipReason(tt.job)
+			if got := reason != ""; got != tt.wantSkip {
+				t.Errorf("skipReason = %q, wantSkip=%v", reason, tt.wantSkip)
+			}
+		})
+	}
+}

scanner/types.go

@@ -67,6 +67,20 @@ type AckMessage struct {
 	Seq  int64  `json:"seq"`
 }
 
+// StartedMessage is sent from scanner to hold when a worker actually begins a
+// scan, as distinct from AckMessage, which is sent from the WebSocket reader
+// the moment the job frame arrives.
+//
+// The gap between the two is the depth of this scanner's own queue multiplied
+// by the per-image scan time, and the hold cannot see into that queue. Without
+// this message its only clock is dispatch, so its scanning deadline budgets
+// queueing as well as scanning and cancels healthy work out from under a
+// backlogged scanner. A hold that does not know the message ignores it.
+type StartedMessage struct {
+	Type string `json:"type"` // "started"
+	Seq  int64  `json:"seq"`
+}
+
 // ResultMessage is sent from scanner to hold with scan results
 type ResultMessage struct {
 	Type       string                `json:"type"` // "result"