Every blob went through the multipart machinery: an S3 multipart started
on Docker's initial POST, a hold round trip per part, and a complete on
the hold that finished the multipart, HEADed the temp object, copied it
to its final key, and deleted the temp. For a 2KB config blob that was
three hold calls and six S3 operations. On production data 86% of
distinct layers and every config blob fit in a 16MB buffer, and 49% of
image manifests have no layer larger than that.
The writer now buffers up to 16MB (also the multipart part size) and
makes no hold call until it has to. A blob that never overflows the
buffer is written at Commit with a single presigned PUT to its final
key, via the hold's existing method=PUT presign; the multipart only
starts on the first flush. The hold's completeUpload does nothing the
direct path skips: quota, layer records, stats and scan dispatch all
hang off notifyManifest, which is unchanged.
The buffer starts empty and grows on demand, with the doubling capped so
capacity never overshoots 16MB: a config blob costs kilobytes, and only
layers that approach the threshold fill it.
Bytes are hashed as they arrive. Commit compares the computed sha256 to
the digest the client claimed before any network call, and returns
DIGEST_INVALID on mismatch, aborting a multipart if one was started.
Previously nothing verified the content, so a pusher could store wrong
bytes under a digest in the shared content-addressed space.
Tests observe request counts on a fake hold and fake S3 rather than
return values. The growth test streams in 24KB chunks because
power-of-two chunks land on 16MB by luck and hid an earlier weaker guard.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
distribution calls Stat before every blob GET and HEAD. The appview's
Stat asked the hold for a presigned HEAD URL and then HEADed S3 with it
purely to read Content-Length for the descriptor: two round trips to
learn one number.
The hold's getBlob response for OCI digests on GET and HEAD now carries
"size". It comes from the records index when a layer record exists (a
SQLite lookup on a new digest index, no network) and from a HeadObject
otherwise, which is where config blobs land. If storage says the object
does not exist the hold answers 404 instead of signing a URL that can
only fail. The PUT and ATProto CID paths are untouched.
The appview builds the descriptor from the reported size and makes no
S3 request. When the field is absent it HEADs the presigned URL as
before, so a new appview works against a hold that has not been
upgraded, and an old appview ignores the extra field. A hold 404 maps
to ErrBlobUnknown.
Tests prove the index answered by leaving the mock bucket empty and
counting zero HeadObject calls, prove the fallback with exactly one, and
count requests reaching the fake S3 origin on the appview side rather
than trusting the returned size.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
An audit of the scan pipeline and the hold side of scanning found several
ways scanning stops without saying so. Each fix here was written test-first:
a test expressing the wanted behaviour, confirmed failing for the right
reason, then the change.
A summary-less result crash-looped both processes. worker.go dereferenced
result.Summary unconditionally, but processJob only sets it when Grype runs,
and SendResult puts the nil on the wire before the scanner dies on it, so
handleResult's unguarded log killed the hold too. A nil Summary now means
"not scanned for vulnerabilities", deliberately distinct from "scanned, found
zero" — inventing a zeroed summary would report every image as clean when
Grype never ran. The hold writes a record rather than orphaning the uploaded
SBOM, and the appview renders an "SBOM only" state instead of a green Clean
badge.
The Grype database could wedge with no way back short of a restart. All three
throttles in loadVulnDatabase were guarded by vulnDB != nil, so a scanner
holding no provider retried a full download on every scan under the exclusive
lock. Two earlier attempts at this bug each added one more condition to the
same chain; this replaces the chain with a single decision function over a
state snapshot, consulted by both call sites so they cannot disagree. That
disagreement was itself a bug: the 50-scan reload had never once executed.
Two independent halts. An unparseable frame was dropped in silence, stranding
a row that held the hold's only dispatch slot forever; it is now answered
"skipped" on first delivery. The 10-minute sweep leaked the in-flight digest
and wrote no record, permanently retiring one image per timeout.
A digest went unvalidated into filepath.Join and os.Create, so a layer digest
of sha256:../../../x wrote outside the scan directory, and nothing verified
that downloaded bytes hashed to the digest naming them. Digests come from
records in a user's own PDS. Both are fixed together: verification is what
makes an escaping write self-defeating.
Concurrency did not work on either axis. The proactive capacity gate was
depth-one hold-wide, so neither extra workers nor extra scanner processes
received work. Depth is now the sum of the worker counts scanners advertise on
connect, the gate is scoped to proactive work, and dispatch prefers the
least-loaded scanner. Disconnects no longer hand a running scan to someone
else: a scanner keeps a stable per-process identity and reclaims its own rows
within a grace window, while a process that truly restarted returns with a new
identity and has its work reclaimed, which is correct because the restart did
lose it.
The hold's scanning deadline measured queueing rather than scanning, because
the scanner acks on receipt and handleAck never refreshed assigned_at. A new
"started" message, sent by the worker that dequeues the job, separates the two
budgets. An older scanner never sends it and falls under the queueing budget,
which is more forgiving than the deadline it gets today.
Adds an in-process mock hold and an e2e harness that runs the real client,
queue and worker pool, seeded with 84 real manifest records fetched from a
live PDS. Real image layouts and the Grype database are fetched by scripts and
gitignored; suites needing them skip cleanly, so the default run stays offline
and fast.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF
- diff view gains a Packages tab with added/removed/changed/unchanged
package tables and purl-derived type/license/upstream links
- captain records verified against the DID's atcr_hold service before
caching (processor + batch backfill), preventing forged holds
- fix empty-handle updates clobbering cached handles and colliding on
the UNIQUE constraint
- move fillPrevCIDs into repo.go; DirectRepoOperator is now canonical,
repomgr kept as a test oracle
- surface read-only crew status in hold selector
- reconcile docs
Evan JarrettandClaude Fable 5.1