at-container-registry/docs/SBOM_SCANNING.md

SBOM Scanning and Vulnerability Analysis

ATCR generates Software Bills of Materials (SBOMs) and scans container images for vulnerabilities. Scanning runs in a separate atcr-scanner service that connects to a hold over a WebSocket, so the hold itself never runs Syft or Grype. Results are stored as io.atcr.hold.scan records in the hold's embedded PDS.

Overview

• Separate scanner binary: Scanning is performed by atcr-scanner (the scanner/ Go module), not by the hold. The scanner connects out to the hold and pulls jobs.
• Syft for SBOMs, Grype for vulnerabilities: Each job runs Syft to produce an SPDX-JSON SBOM, then Grype to scan that SBOM for CVEs. Grype is enabled by default.
• WebSocket dispatch: The hold pushes jobs to connected scanners over /xrpc/io.atcr.hold.subscribeScanJobs. A shared secret authenticates the scanner.
• ATProto result storage: Results land as io.atcr.hold.scan records in the hold's embedded PDS, with the SBOM and full Grype report uploaded as PDS blobs.
• Tier-gated scan-on-push plus proactive rescans: Pushes from eligible tiers trigger an immediate scan; the hold also discovers never-scanned manifests and re-scans stale ones on an interval.

Tools

• Anchore Syft generates the SBOM. Output format is SPDX JSON, hardcoded in scanner/internal/scan/syft.go (not configurable).
• Anchore Grype scans the SBOM for known vulnerabilities and produces critical/high/medium/low/total counts plus a full JSON report with CVE detail.

Architecture

Three pieces cooperate:

                   io.atcr.hold.subscribeScanJobs (WebSocket, ?secret=...)
   ┌───────────┐  ◄────────────────────────────────────────────  ┌──────────────┐
   │   Hold    │   job: {seq, manifestDigest, repo, tier,          │ atcr-scanner │
   │ (Scan     │         config, layers, holdEndpoint, ...}        │  (Syft +     │
   │ Broadcaster)│ ────────────────────────────────────────────►  │   Grype)     │
   │           │                                                   │              │
   │           │   result/error/skipped: {seq, sbom,               │              │
   │           │ ◄──── vulnReport, summary{critical,high,...}}      └──────────────┘
   └─────┬─────┘
         │ stores io.atcr.hold.scan record + SBOM/vuln blobs
         ▼
   Hold embedded PDS (CAR store)

1. Hold (pkg/hold/pds/scan_broadcaster.go) owns the ScanBroadcaster. It persists pending jobs in SQLite (scan_jobs table), accepts scanner WebSocket connections, and dispatches jobs round-robin across all connected scanners using a competing-consumer pattern. It re-dispatches timed-out jobs, and (when a rescan interval is set) runs background discovery and stale-scan loops. On receiving a result, the hold uploads the SBOM and vuln report as blobs and writes the io.atcr.hold.scan record.
2. Scanner (scanner/ module) dials the hold's WebSocket, acks jobs, runs the Syft → Grype pipeline, and sends back a result, error, or skipped message. It keeps a local priority queue so paid tiers jump ahead of free ones (see Scheduling).
3. AppView reads the scan records and blobs from the hold's PDS to render vulnerability badges, SBOM details, and download links in the web UI.

Why the hold's PDS?

Scan results are stored in the hold's embedded PDS rather than the user's PDS:

• No OAuth/service-token plumbing: the hold owns and signs its own records.
• Hold-scoped metadata (scanner version, scan time) stays with the operator.
• Different holds can independently scan the same image for cross-verification.
• The user's PDS stays lean: SBOM and Grype JSON live in hold blob storage.

The trust model is the same as Docker Hub: you trust the hold operator's scanner version and scan integrity. The hold's DID signs the records, and anyone can re-scan a digest to verify the result.

Configuration

Hold side

The hold's scanner integration is configured under scanner: in the hold config (pkg/hold/config.go). Env-var prefix is HOLD_.

YAML key	Env var	Default	Meaning
scanner.secret	HOLD_SCANNER_SECRET	""	Shared secret a scanner must present (as ?secret=) on the WebSocket. Empty disables scanning entirely — no scanner can connect and no jobs are dispatched.
scanner.rescan_interval	HOLD_SCANNER_RESCAN_INTERVAL	168h	Minimum interval between re-scans of the same manifest. When > 0 the hold runs proactive discovery + stale-scan loops. Set to 0 to disable proactive scanning (push-triggered scans still work).

# config-hold.yaml
scanner:
  secret: "a-long-random-shared-secret"
  rescan_interval: 168h

Whether a push triggers an immediate scan is decided by the quota tier (see Scan-on-push tier gate).

Scanner side

The scanner is configured via Viper (scanner/internal/config/config.go); it accepts a YAML file or pure env vars with the SCANNER_ prefix. Run with SCANNER_HOLD_URL=... SCANNER_HOLD_SECRET=... atcr-scanner serve.

YAML key	Env var	Default	Meaning
hold.url	SCANNER_HOLD_URL	— (required)	WebSocket URL of the hold, e.g. ws://localhost:8080 or wss://hold01.atcr.io. http(s) is auto-converted to ws(s).
hold.secret	SCANNER_HOLD_SECRET	— (required)	Must match the hold's scanner.secret. Sent as ?secret=.
scanner.workers	SCANNER_SCANNER_WORKERS	1	Number of concurrent scan workers. Declared to the hold on connect, which sizes the hold's dispatch budget for this process; raise it only alongside vuln.max_image_size and a cgroup memory cap.
scanner.queue_size	SCANNER_SCANNER_QUEUE_SIZE	100	Max depth of the local priority queue.
vuln.enabled	SCANNER_VULN_ENABLED	true	Run Grype after Syft. When false, only the SBOM is produced (no counts).
vuln.db_path	SCANNER_VULN_DB_PATH	/var/lib/atcr-scanner/vulndb	Directory for the Grype vulnerability database.
vuln.tmp_dir	SCANNER_VULN_TMP_DIR	/var/lib/atcr-scanner/tmp	Directory for layer extraction and DB download. Also exported as TMPDIR; point it at a large partition, not tmpfs.
vuln.max_image_size	SCANNER_VULN_MAX_IMAGE_SIZE	2147483648 (2 GiB)	Max total compressed image size. Larger images are skipped with an error. 0 = no limit.
vuln.sweep_max_age	SCANNER_VULN_SWEEP_MAX_AGE	1h	Age threshold for the startup sweep of vuln.tmp_dir. Leftover scan-*, syft-scan-* and syft-cataloger-* directories older than this are removed before workers start; a scan killed by a restart never cleans up after itself. Keep it above the longest scan so a second scanner sharing the directory keeps its live work. 0 disables the sweep.
server.addr	SCANNER_SERVER_ADDR	:9090	Listen address for the scanner's health endpoint.

Both hold.url and hold.secret are required; LoadConfig errors out if either is empty.

# Minimal scanner invocation (env-only)
SCANNER_HOLD_URL=wss://hold01.atcr.io \
SCANNER_HOLD_SECRET=a-long-random-shared-secret \
./bin/atcr-scanner serve

Scanning Workflow

1. Push → scan-on-push tier gate

When an image is pushed and the manifest is recorded, the hold's OCI XRPC handler (pkg/hold/oci/xrpc.go) decides whether to enqueue a scan. Multi-arch manifest lists and artifacts with a subject (attestations) are skipped — they have no scannable content. For everything else, the tier of the pusher decides:

• Captain / owner: always scanned.
• Quotas disabled (quotaMgr == nil or quotas not enabled): all pushes scanned (backwards compatible).
• Quotas enabled: scanned only if the pusher's tier has scan_on_push: true.

In the default config (pkg/hold/config.go), bosun and quartermaster have scan_on_push: true; deckhand does not. So a free-tier (deckhand) push is not scanned on push — it gets picked up later by the proactive discovery loop.

2. Dispatch

The ScanBroadcaster.Enqueue inserts the job into the scan_jobs SQLite table (status pending) and immediately tries to dispatch it to a connected scanner. Jobs survive hold restarts. If no scanner is connected, the job waits.

Which scanner gets it. Selection is by spare capacity, not position: each connection declares how many scans it runs at once (?workers=, default 1) and the hold prefers the scanner with the smallest fraction of its capacity committed, with ties resolved round-robin. A job that no connected scanner has room for stays pending rather than being pushed into a scanner's own queue, and is offered again the moment any scanner finishes something. Keeping the queue on the hold is what makes the job re-routable to whichever process frees up first, and what makes the deadlines below mean anything.

Deadlines. Assigned-but-unacked jobs time out after 5 minutes and are re-dispatched. Once a job is acked the scanner has it, but it may be queued behind that scanner's workers, so there are two further budgets: 10 minutes from the started message a worker sends when it actually begins the scan, and 60 minutes from dispatch for a job that was acked but never reported as started (which is also what a scanner too old to send started gets). Both write a failed scan record and release the manifest for re-scanning.

Disconnects. A dropped WebSocket does not return a scanner's in-flight jobs to the pool: its worker pool never learns the socket went away and keeps scanning, so handing that work to another process would have two scanners scanning the same image. The rows are marked instead. A scanner sends a stable per-process identity (?instance=) on every connect and resumes its own jobs on reconnect; a scanner that does not come back within 2 minutes has them reclaimed and re-offered. A scanner that actually restarted comes back with a new identity, so its old work is reclaimed rather than resumed — which is right, since a restart really did lose it.

3. Scan pipeline (scanner)

For each job (scanner/internal/scan/worker.go):

1. Artifact-type check — if config.mediaType is in unscannableConfigTypes the job returns a SkipError and the scanner sends a skipped message (see below).
2. Size check — if total compressed size exceeds vuln.max_image_size, the job fails.
3. Build OCI layout — layers are fetched from the hold via presigned URLs and assembled into an OCI image layout in vuln.tmp_dir.
4. Syft — generates the SBOM and encodes it to SPDX JSON.
5. Grype (if vuln.enabled) — scans the SBOM, producing the full JSON report and a severity summary (critical/high/medium/low/total).

A worker sends started when it dequeues a job, before step 1. This is distinct from the ack, which the WebSocket reader sends the instant a job frame arrives: the gap between them is however long the job waits in this scanner's own queue, and the hold measures its scanning deadline from started so that queueing does not count against it.

The scanner then sends one of three terminal messages back over the WebSocket: result (SBOM + optional vuln report + summary), error, or skipped (with a reason). All four messages are ignored by the hold unless the job is currently assigned to the scanner sending them.

4. Result storage (hold)

On result (scan_broadcaster.go handleResult):

1. Upload the SBOM bytes as a PDS blob (application/spdx+json).
2. Upload the Grype report as a PDS blob (application/vnd.atcr.vulnerabilities+json).
3. Create an io.atcr.hold.scan record (CreateScanRecord) keyed by the manifest digest, referencing both blobs and carrying the severity counts.
4. Mark the scan_jobs row completed.

On error, a failed scan record is written (NewFailedScanRecord) and the job is marked failed. On skipped, a skipped record is written (NewSkippedScanRecord) and the job is marked completed.

Scan Record Schema

Results are io.atcr.hold.scan records in the hold's embedded PDS (pkg/atproto/lexicon.go, ScanRecord). The record key is the manifest digest hex (without the sha256: prefix), so there is exactly one scan record per manifest and re-scans upsert it.

{
  "$type": "io.atcr.hold.scan",
  "manifest": "at://did:plc:alice123/io.atcr.manifest/abc123...",
  "repository": "myapp",
  "userDid": "did:plc:alice123",
  "sbomBlob": {
    "$type": "blob",
    "ref": { "$link": "bafkrei..." },
    "mimeType": "application/spdx+json",
    "size": 51234
  },
  "vulnReportBlob": {
    "$type": "blob",
    "ref": { "$link": "bafkrei..." },
    "mimeType": "application/vnd.atcr.vulnerabilities+json",
    "size": 18567
  },
  "critical": 2,
  "high": 15,
  "medium": 42,
  "low": 8,
  "total": 67,
  "scannerVersion": "atcr-scanner-v1.0.0",
  "scannedAt": "2026-06-11T12:34:56Z",
  "status": "ok",
  "reason": ""
}

Field	Notes
manifest	AT-URI of the scanned manifest in the user's PDS.
userDid	DID of the image owner.
sbomBlob	Reference to the SPDX-JSON SBOM in hold blob storage. Absent for failed/skipped scans.
vulnReportBlob	Reference to the full Grype JSON report. Absent if Grype disabled or scan failed/skipped.
critical/high/medium/low/total	Vulnerability counts from Grype. Zero on failed/skipped scans.
scannerVersion	Scanner identifier for reproducibility (currently atcr-scanner-v1.0.0).
scannedAt	RFC3339 scan completion timestamp.
status	ok, failed, or skipped.
reason	Populated for failed (error text) and skipped (why it was bypassed).

Status field

Status	Meaning	Rescan behavior
ok (or empty)	Scanner produced an SBOM; counts and SBOM blob populated.	Re-scanned on the rescan interval (default 7d).
failed	Scanner ran but errored (network, OOM, parse failure). No SBOM/counts.	Re-scanned on the rescan interval — failures may be transient.
skipped	Scanner intentionally bypassed the artifact (helm chart, in-toto, DSSE). reason explains why.	Never re-queued. Won't change without a code change in the scanner.

Records written before the status field existed have an empty status. The appview treats empty + nil-blob + zero-count as failed (legacy fallback).

Unscannable artifact types

The scanner skips artifacts whose config media type appears in unscannableConfigTypes (scanner/internal/scan/worker.go). Currently:

• application/vnd.cncf.helm.config.v1+json — Helm charts. Rendered with a helm-aware digest page (pkg/appview/handlers/digest.go) that shows Chart.yaml metadata instead of layers / vulns / SBOM.
• application/vnd.in-toto+json — in-toto attestations.
• application/vnd.dsse.envelope.v1+json — DSSE envelopes (SLSA provenance).

For these types the appview's vuln/SBOM tabs render "Vulnerability scanning isn't applied to this artifact type." — no retry hint.

To add a new unscannable type: append the media type to unscannableConfigTypes. Existing records won't auto-rewrite — run the backfill tool (below) once to convert any pre-existing failure records into skipped records.

Scheduling and Priority

Scanner-side priority queue

Each scanner keeps a local priority heap (scanner/internal/queue/priority_queue.go). Jobs are ordered by tier priority, FIFO within a tier (lower number = higher priority):

Tier	Priority
owner	0
quartermaster	1
bosun	2
anything else (deckhand)	3

So when a scanner has a backlog, owner and paid-tier jobs are processed before free-tier ones.

Hold-side dispatch

The hold dispatches jobs round-robin across connected scanners (no priority at the hold level — that is the scanner's job). Each scanner pulls its assigned jobs into its own priority queue. With multiple scanners, the competing-consumer pattern spreads load.

Proactive scanning

When scanner.rescan_interval > 0, the hold runs three background loops:

• Discovery loop: every 4 hours (and on scanner reconnect), queries relays for DIDs with io.atcr.manifest records, walks each user's PDS, and queues manifests that belong to this hold but have no scan record yet. These are dispatched at the deckhand tier.
• Stale-scan loop: walks the local scan records and re-queues any ok/failed record older than rescan_interval. Skipped records are left alone.
• Dispatch loop: drains the unscanned queue (higher priority) before the stale queue, throttled to one proactive job per connected scanner worker — the sum of every connected scanner's declared workers. The throttle counts only proactive jobs: push-triggered scans bypass it entirely, so counting them meant a hold with steady pushes never dispatched a proactive scan at all. With no scanner connected the budget is zero and nothing is dispatched.

Scaling this out is therefore a matter of running more scanner processes against the same hold, raising scanner.workers, or both: the dispatch budget, the drain on connect and the choice of scanner all follow the declared capacity. Do this only after bounding scanner memory — concurrency is what holds peak RSS down on a small host, and two concurrent scans of a node:22-class image measured 687 MiB with a 512 MiB GOMEMLIMIT in force and 1357 MiB without.

Accessing Results

There is no io.atcr.hold.getSBOM XRPC endpoint. Results are read directly from the hold's PDS using standard ATProto XRPC, and the appview UI wraps these calls.

From the AppView web UI

The appview exposes HTMX endpoints that render scan data on repository/digest pages (pkg/appview/routes/routes.go, handlers in pkg/appview/handlers/):

• GET /api/scan-result — vulnerability badge for a digest (scan_result.go).
• GET /api/scan-results — batch badges for a tag list (scan_result.go).
• GET /api/vuln-details — full vulnerability detail modal (vuln_details.go).
• GET /api/sbom-details — SBOM summary modal (sbom_details.go).
• GET /api/scan-download?digest=...&holdEndpoint=...&type=sbom|vuln — downloads the raw SBOM or Grype JSON as a file (scan_download.go).

These handlers resolve the hold, fetch the io.atcr.hold.scan record, and pull the SBOM/vuln blobs.

Directly from the hold's PDS

The appview handlers do exactly this under the hood:

# 1. Fetch the scan record (rkey = manifest digest hex, no "sha256:" prefix)
curl "https://hold01.atcr.io/xrpc/com.atproto.repo.getRecord?\
repo=did:web:hold01.atcr.io&\
collection=io.atcr.hold.scan&\
rkey=abc123..."

# Response value contains sbomBlob.ref.$link, vulnReportBlob.ref.$link, and counts.

# 2. Download the SBOM blob by its CID
curl "https://hold01.atcr.io/xrpc/com.atproto.sync.getBlob?\
did=did:web:hold01.atcr.io&\
cid=bafkrei..." > sbom.spdx.json

# 3. Scan locally with another tool if desired
grype sbom:./sbom.spdx.json
osv-scanner --sbom sbom.spdx.json

You can also list all scan records on a hold via com.atproto.repo.listRecords?repo=<holdDid>&collection=io.atcr.hold.scan.

Backfill and Rescan

Rescans

Re-scanning is automatic when scanner.rescan_interval > 0 — the stale-scan loop re-queues records older than the interval (default 7 days). Failed scans are retried; skipped scans are not.

Backfill tool

atcr-hold scan-backfill --config <path> walks every io.atcr.hold.scan record and rewrites legacy ones (empty status + nil SBOM blob + zero counts) by assigning a status from the manifest's layer media types:

• Layer media type contains helm.chart.content, in-toto, or dsse.envelope → status="skipped".
• Otherwise → status="failed".

The tool is idempotent and preserves each record's original scannedAt. It opens the hold's CAR store directly, so the hold service must be stopped first (the embedded PDS holds an exclusive lock). For zero-downtime backfill on a running hold, use the admin endpoint POST /admin/api/scan-backfill instead.

Troubleshooting

• No scans happening at all. Check that scanner.secret is set on the hold (empty disables scanning) and that a scanner is connected. Scanner connection failures log dial failed / WebSocket read error.
• Scanner connects then immediately disconnects. Usually a secret mismatch — SCANNER_HOLD_SECRET must equal the hold's scanner.secret.
• Free-tier pushes never get scanned on push. Expected: deckhand has scan_on_push: false by default. They are picked up by the discovery loop instead (requires rescan_interval > 0).
• Large images skipped. Total compressed size exceeds vuln.max_image_size (2 GiB default). Raise it or set 0 for no limit.
• Layer extraction or Grype DB download fails mid-process. vuln.tmp_dir is too small or on tmpfs. Point it at a large persistent partition; the scanner sets TMPDIR to this directory.
• vuln.tmp_dir fills up with scan-* / syft-scan-* directories. These are scans the process was killed in the middle of, which never ran their own cleanup. The startup sweep reclaims them on the next restart; if it is not doing so, check that vuln.sweep_max_age is not 0 and look for Swept scanner tmp dir in the startup logs.
• SBOM present but no vulnerability counts. vuln.enabled is false on the scanner, or the Grype DB failed to initialize (check startup logs).
• Helm/attestation artifacts show "scanning isn't applied". Expected — these are in unscannableConfigTypes and recorded as skipped.

References

• Syft
• Grype
• SPDX Specification
• Hold XRPC Endpoints
• Quotas
• ATProto Specification