mirror of
https://tangled.org/evan.jarrett.net/at-container-registry
synced 2026-09-25 03:34:14 +00:00
Evan JarrettandClaude Opus 5
a63f668de0
An audit of the scan pipeline and the hold side of scanning found several ways scanning stops without saying so. Each fix here was written test-first: a test expressing the wanted behaviour, confirmed failing for the right reason, then the change. A summary-less result crash-looped both processes. worker.go dereferenced result.Summary unconditionally, but processJob only sets it when Grype runs, and SendResult puts the nil on the wire before the scanner dies on it, so handleResult's unguarded log killed the hold too. A nil Summary now means "not scanned for vulnerabilities", deliberately distinct from "scanned, found zero" — inventing a zeroed summary would report every image as clean when Grype never ran. The hold writes a record rather than orphaning the uploaded SBOM, and the appview renders an "SBOM only" state instead of a green Clean badge. The Grype database could wedge with no way back short of a restart. All three throttles in loadVulnDatabase were guarded by vulnDB != nil, so a scanner holding no provider retried a full download on every scan under the exclusive lock. Two earlier attempts at this bug each added one more condition to the same chain; this replaces the chain with a single decision function over a state snapshot, consulted by both call sites so they cannot disagree. That disagreement was itself a bug: the 50-scan reload had never once executed. Two independent halts. An unparseable frame was dropped in silence, stranding a row that held the hold's only dispatch slot forever; it is now answered "skipped" on first delivery. The 10-minute sweep leaked the in-flight digest and wrote no record, permanently retiring one image per timeout. A digest went unvalidated into filepath.Join and os.Create, so a layer digest of sha256:../../../x wrote outside the scan directory, and nothing verified that downloaded bytes hashed to the digest naming them. Digests come from records in a user's own PDS. Both are fixed together: verification is what makes an escaping write self-defeating. Concurrency did not work on either axis. The proactive capacity gate was depth-one hold-wide, so neither extra workers nor extra scanner processes received work. Depth is now the sum of the worker counts scanners advertise on connect, the gate is scoped to proactive work, and dispatch prefers the least-loaded scanner. Disconnects no longer hand a running scan to someone else: a scanner keeps a stable per-process identity and reclaims its own rows within a grace window, while a process that truly restarted returns with a new identity and has its work reclaimed, which is correct because the restart did lose it. The hold's scanning deadline measured queueing rather than scanning, because the scanner acks on receipt and handleAck never refreshed assigned_at. A new "started" message, sent by the worker that dequeues the job, separates the two budgets. An older scanner never sends it and falls under the queueing budget, which is more forgiving than the deadline it gets today. Adds an in-process mock hold and an e2e harness that runs the real client, queue and worker pool, seeded with 84 real manifest records fetched from a live PDS. Real image layouts and the Grype database are fetched by scripts and gitignored; suites needing them skip cleanly, so the default run stays offline and fast. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF
67 lines
2.5 KiB
Bash
Executable File
67 lines
2.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Pull real image blobs into OCI layouts that mockhold.OCILayout can serve.
|
|
#
|
|
# The layouts land in testdata/blobs/<name>/ and are gitignored: they are
|
|
# megabytes of container layers, and every test that needs them skips cleanly
|
|
# when they are absent. Descriptor-only scenarios (disconnect, reconnect,
|
|
# skip classification, oversize) need nothing from here.
|
|
#
|
|
# Why skopeo rather than the scanner's own download path: `skopeo copy ...
|
|
# oci:<dir>` writes blobs/sha256/<hex>, keyed by exactly the digest the scan
|
|
# job references. That is the same layout buildOCILayout reconstructs at scan
|
|
# time, so a pulled image drops straight in with no rewriting.
|
|
#
|
|
# Auth: the registry host must be mapped to the credential helper. seamark.cr
|
|
# is NOT a registry you can log into by default (its handle appears under the
|
|
# atcr.io appview, which is a different thing); buoy.cr works out of the box
|
|
# if `docker-credential-atcr status` lists an account for it. The script writes
|
|
# its own authfile so it never touches ~/.docker/config.json.
|
|
#
|
|
# Usage:
|
|
# ./fetch-blobs.sh # pull the default fixture set
|
|
# REGISTRY=atcr.io ./fetch-blobs.sh
|
|
set -euo pipefail
|
|
|
|
REGISTRY="${REGISTRY:-buoy.cr}"
|
|
ACCOUNT="${ACCOUNT:-evan.jarrett.net}"
|
|
DIR="$(cd "$(dirname "$0")" && pwd)/blobs"
|
|
|
|
# name|repository|manifest digest
|
|
# Digests come from corpus.json. Keep these small: the point is real layer
|
|
# bytes for Syft to catalog, not coverage of every image on the hold.
|
|
FIXTURES=(
|
|
"hsm-secrets-operator|hsm-secrets-operator|sha256:1cfa4e2b09e127b9c4ed43578d3f3c18e7d44ea47b9ea98475c0cbe9086525f8"
|
|
)
|
|
|
|
command -v skopeo >/dev/null || { echo "skopeo not found"; exit 1; }
|
|
|
|
AUTHFILE=$(mktemp); trap 'rm -f "$AUTHFILE"' EXIT
|
|
printf '{"credHelpers":{"%s":"atcr"}}\n' "$REGISTRY" > "$AUTHFILE"
|
|
|
|
mkdir -p "$DIR"
|
|
for entry in "${FIXTURES[@]}"; do
|
|
IFS='|' read -r name repo digest <<< "$entry"
|
|
dest="$DIR/$name"
|
|
|
|
if [ -f "$dest/oci-layout" ]; then
|
|
echo "→ $name already present, skipping"
|
|
continue
|
|
fi
|
|
|
|
echo "→ pulling ${REGISTRY}/${ACCOUNT}/${repo}@${digest:0:19}..."
|
|
rm -rf "$dest"
|
|
if ! skopeo copy --authfile "$AUTHFILE" \
|
|
"docker://${REGISTRY}/${ACCOUNT}/${repo}@${digest}" \
|
|
"oci:${dest}:img"; then
|
|
echo " FAILED. If this is an auth error, check:"
|
|
echo " docker-credential-atcr status"
|
|
echo " and confirm an account is configured for ${REGISTRY}."
|
|
rm -rf "$dest"
|
|
exit 1
|
|
fi
|
|
done
|
|
|
|
echo
|
|
echo "→ layouts in $DIR:"
|
|
du -sh "$DIR"/*/ 2>/dev/null || true
|