Every blob went through the multipart machinery: an S3 multipart started
on Docker's initial POST, a hold round trip per part, and a complete on
the hold that finished the multipart, HEADed the temp object, copied it
to its final key, and deleted the temp. For a 2KB config blob that was
three hold calls and six S3 operations. On production data 86% of
distinct layers and every config blob fit in a 16MB buffer, and 49% of
image manifests have no layer larger than that.
The writer now buffers up to 16MB (also the multipart part size) and
makes no hold call until it has to. A blob that never overflows the
buffer is written at Commit with a single presigned PUT to its final
key, via the hold's existing method=PUT presign; the multipart only
starts on the first flush. The hold's completeUpload does nothing the
direct path skips: quota, layer records, stats and scan dispatch all
hang off notifyManifest, which is unchanged.
The buffer starts empty and grows on demand, with the doubling capped so
capacity never overshoots 16MB: a config blob costs kilobytes, and only
layers that approach the threshold fill it.
Bytes are hashed as they arrive. Commit compares the computed sha256 to
the digest the client claimed before any network call, and returns
DIGEST_INVALID on mismatch, aborting a multipart if one was started.
Previously nothing verified the content, so a pusher could store wrong
bytes under a digest in the shared content-addressed space.
Tests observe request counts on a fake hold and fake S3 rather than
return values. The growth test streams in 24KB chunks because
power-of-two chunks land on 16MB by luck and hid an earlier weaker guard.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
distribution calls Stat before every blob GET and HEAD. The appview's
Stat asked the hold for a presigned HEAD URL and then HEADed S3 with it
purely to read Content-Length for the descriptor: two round trips to
learn one number.
The hold's getBlob response for OCI digests on GET and HEAD now carries
"size". It comes from the records index when a layer record exists (a
SQLite lookup on a new digest index, no network) and from a HeadObject
otherwise, which is where config blobs land. If storage says the object
does not exist the hold answers 404 instead of signing a URL that can
only fail. The PUT and ATProto CID paths are untouched.
The appview builds the descriptor from the reported size and makes no
S3 request. When the field is absent it HEADs the presigned URL as
before, so a new appview works against a hold that has not been
upgraded, and an old appview ignores the extra field. A hold 404 maps
to ErrBlobUnknown.
Tests prove the index answered by leaving the mock bucket empty and
counting zero HeadObject calls, prove the fallback with exactly one, and
count requests reaching the fake S3 origin on the appview side rather
than trusting the returned size.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
buildOCILayout already removes its scan dir on every error path, and syft.go
defers the stereoscope generator's Cleanup. What neither can do is clean up
after a process that dies mid-scan: the deferred call never runs, and nothing
afterwards ever looks at what was left. Every restart therefore leaks the
in-flight layout and extraction permanently, and a restart is routine — a
deploy is one.
On seamark-hold that reached 8.8 GB of orphaned scan-*, syft-scan-* and
syft-cataloger-* directories under a 20 GB disk, at which point the disk was
97% full and scans began failing on it:
failed to load OCI image: unable to populate layer cache
dir="/var/lib/seamark/scanner/tmp/syft-scan-1546187834/..."
: no space left on device
failed to download layer 5: failed to write blob:
write /var/lib/seamark/scanner/tmp/scan-4160414849/blobs/sha256/...
: no space left on device
The leaked directories cluster at the scanner's restart timestamps, which is
what identifies the killed process rather than the error paths as the source.
Manual removal reclaimed 8.8 GB and took the disk from 97% to 50%.
Startup is where this belongs: it is the one moment the previous process is
known to be gone, and it is immediately after the event that caused the leak.
The sweep runs in WorkerPool.Start after TMPDIR is set and before any worker
can dequeue, so nothing it removes can be work in progress here.
Three constraints shape what it will touch:
- Only the three per-job prefixes, only as direct children, only
directories. The Grype database lives beside the tmp dir at
<parent>/vulndb and go-getter unpacks into grype-dl underneath it; both
are state the scanner needs and neither matches a prefix. The prefixes now
have one definition each, used by both the creator and the sweeper, so
renaming a directory cannot silently take it out of the sweep's scope.
- An age threshold, vuln.sweep_max_age, default 1h. A second scanner sharing
the directory has an in-flight scan-* dir that is minutes old, and
scanner.job_timeout is 8m, so an hour clears both with room to spare. 0
disables the sweep rather than removing a peer's live work.
- Nothing is fatal. A stat or removal failure is a WARN and the sweep moves
on, so a permission problem in the tmp dir cannot keep the scanner from
starting.
The sweep only runs at startup, so a scanner that is killed twice between
deploys carries the first leak until its next restart. That is the tradeoff
for never racing a live peer; a periodic sweep would be the follow-up if
processes ever live long enough for it to matter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TA9D4DjaLZTvzQ7dJbu4eg
An audit of the scan pipeline and the hold side of scanning found several
ways scanning stops without saying so. Each fix here was written test-first:
a test expressing the wanted behaviour, confirmed failing for the right
reason, then the change.
A summary-less result crash-looped both processes. worker.go dereferenced
result.Summary unconditionally, but processJob only sets it when Grype runs,
and SendResult puts the nil on the wire before the scanner dies on it, so
handleResult's unguarded log killed the hold too. A nil Summary now means
"not scanned for vulnerabilities", deliberately distinct from "scanned, found
zero" — inventing a zeroed summary would report every image as clean when
Grype never ran. The hold writes a record rather than orphaning the uploaded
SBOM, and the appview renders an "SBOM only" state instead of a green Clean
badge.
The Grype database could wedge with no way back short of a restart. All three
throttles in loadVulnDatabase were guarded by vulnDB != nil, so a scanner
holding no provider retried a full download on every scan under the exclusive
lock. Two earlier attempts at this bug each added one more condition to the
same chain; this replaces the chain with a single decision function over a
state snapshot, consulted by both call sites so they cannot disagree. That
disagreement was itself a bug: the 50-scan reload had never once executed.
Two independent halts. An unparseable frame was dropped in silence, stranding
a row that held the hold's only dispatch slot forever; it is now answered
"skipped" on first delivery. The 10-minute sweep leaked the in-flight digest
and wrote no record, permanently retiring one image per timeout.
A digest went unvalidated into filepath.Join and os.Create, so a layer digest
of sha256:../../../x wrote outside the scan directory, and nothing verified
that downloaded bytes hashed to the digest naming them. Digests come from
records in a user's own PDS. Both are fixed together: verification is what
makes an escaping write self-defeating.
Concurrency did not work on either axis. The proactive capacity gate was
depth-one hold-wide, so neither extra workers nor extra scanner processes
received work. Depth is now the sum of the worker counts scanners advertise on
connect, the gate is scoped to proactive work, and dispatch prefers the
least-loaded scanner. Disconnects no longer hand a running scan to someone
else: a scanner keeps a stable per-process identity and reclaims its own rows
within a grace window, while a process that truly restarted returns with a new
identity and has its work reclaimed, which is correct because the restart did
lose it.
The hold's scanning deadline measured queueing rather than scanning, because
the scanner acks on receipt and handleAck never refreshed assigned_at. A new
"started" message, sent by the worker that dequeues the job, separates the two
budgets. An older scanner never sends it and falls under the queueing budget,
which is more forgiving than the deadline it gets today.
Adds an in-process mock hold and an e2e harness that runs the real client,
queue and worker pool, seeded with 84 real manifest records fetched from a
live PDS. Real image layouts and the Grype database are fetched by scripts and
gitignored; suites needing them skip cleanly, so the default run stays offline
and fast.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF
Around 2026-08-20 the Bluesky relay marked every ATCR hold offline and stopped
dialing. Nobody noticed for two weeks, and it surfaced only indirectly as "pull
and push counts are up to 24 h stale". The stats were a symptom; the fleet was
simply disconnected.
It cannot recover on its own. Indigo's relay gives up on a host after 16
consecutive dial failures and returns from the redialer, and only a fresh
requestCrawl revives it. The hold sends requestCrawl exactly once, at boot
(server.go:400), with no ticker and no check that any relay is subscribed. So a
dropped hold is invisible until its process restarts, silently.
Documents the current mechanics, the failure mode, how to diagnose it with
getHostStatus and a frozen repo rev, and how to recover. The automatic
re-crawl is described as a deferred proposal and explicitly NOT implemented,
by decision: a jittered ticker guarded on subscriber liveness, plus surfacing
the subscriber count, since the deeper problem is that this was silent.
Two things found while writing it, both recorded. The proposal needs plumbing
that does not exist: EventBroadcaster has no exported subscriber count, and
Subscriber does not retain the userAgent, so "is a relay listening" cannot
currently be answered. And ResubscribeAllHosts selects only active hosts, so an
offline host is not recovered even by a relay restart.
Carries a replay warning. ca539b1 fixed a panic on subscriber disconnect during
firehose backfill, and the exposure condition is that a backfill goroutine
exists at all, which Subscribe skips when the cursor is current. So a caught-up
relay never triggered it and a hold whose relays are far behind is exposed on
every reconnect. Verify a deployed hold contains ca539b1 before provoking a
re-crawl; efabb677 does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9
Finding 32: pushing an artifact with an unrecognised config media type
classifies as "unknown", and every template branches two ways on "helm-chart"
with the container-image page in the else. So an in-toto attestation is served
a docker pull command, Layers/Vulnerabilities/SBOM tabs, "Image layer history",
and a promise that scans run shortly after push, which for that artifact will
never be true. This is a design note rather than a fix, since the change is
larger than the symptom.
The inventory is the part worth keeping. The classification rule exists in four
places, keyed off three different inputs (appview config media type, hold config
media type with no unknown case, scanner config map plus layer shape, hold layer
substrings), and the appview's artifact_type feeds none of the scan decisions.
Manifest-level artifactType is discarded at parse time on every push: it is
absent from the record struct, the constructor and the lexicon, surviving only
inside the unindexed manifest blob.
Two corrections to the framing this started from, both verified rather than
assumed. The repo does not have referrers support: the pinned distribution
version has no referrers code and ATCR registers no such route, so what exists
is subject_digest persistence plus an attestation badge. And GetTopLevelManifests
filters artifact_type != 'unknown', so an untagged unknown artifact is invisible
while a tagged one renders as an image, which the finding did not mention.
Proposes a type set, spec precedence (manifest artifactType, then config media
type, then structural signals), a UI contract stating what such a page must not
show, and a six stage plan. Only stage 2 needs a migration, for the raw string
column; new slug values need no DDL and no data migration, since jetstream
upserts artifact_type on every record it sees.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9
seamark.dev's /install and /settings/devices told users to pipe
seamark.dev/static/install.sh into bash. That file was the unmodified ATCR
script: it announced itself as the "ATCR Credential Helper Installer",
installed docker-credential-atcr, and finished by telling the user to configure
credHelpers for atcr.io, the wrong registry for that deployment. Anyone
following the documented setup ended up pointed at another service. The
templates hardcoded docker-credential-atcr, "atcr" and ~/.atcr/device.json
alongside a correctly themed {{ .RegistryURL }}.
The scripts are now rendered from config by a handler, rather than forked per
brand. A theme overlay was the alternative and was worse: it needed a full copy
of both install.sh and install.ps1 per brand, four scripts to keep in sync, and
the operator asked for these values to come from config.
credential_helper.name is the single knob. Docker resolves a credHelpers value
x by exec'ing docker-credential-x, so the credHelpers value, the binary suffix
and the config directory are genuinely one word, not three that can drift. It
is validated against a strict pattern because it is interpolated into a shell
script.
install.sh renders byte-identical to the deleted static file under the atcr
default, so existing installs are unaffected. install.ps1 differs by one line,
where a stale usage comment named a path the script is not served at.
Two behaviour changes worth noting: these two URLs drop from a one-year
Cache-Control to five minutes, since the body now depends on deployment config;
and credential_helper.tangled_repo becomes a real overridable default. It was
previously assigned over unconditionally and read by nothing, while the shipped
script used a different URL form.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9
Update every direct dependency across all five workspace modules to
latest. Notable jumps: syft v1.43.0 -> v1.51.1, grype v0.111.1 ->
v0.118.0, stereoscope v0.1.23 -> v0.3.1, indigo -> 2026-09-01,
aws-sdk-go-v2/service/s3 v1.99.1 -> v1.110.0, grpc v1.80.0 -> v1.83.2,
x/crypto v0.50.0 -> v0.55.0.
Three deps needed more than a version bump:
go-libipfs could not be updated at all. The repo was renamed to boxo, so
every tag past v0.7.0 declares `module github.com/ipfs/boxo` and cannot
be required under the old path. sqlite_store.go already imported
go-block-format alongside it and used the archived package exactly once,
inside a function already returning blockformat.Block, so it was relying
on structural interface satisfaction. Collapsing to the native type drops
the archived dependency entirely.
go-didplc moved its package from the repo root into a didplc/ subdir in
v0.2.2. Package name is unchanged and every symbol we use (RegularOp,
OpEnum, OpService, Client.DirectoryURL, Submit) is intact, so this is an
import path change only.
The go-diskfs replace in scanner/go.mod had inverted. It pinned v1.7.0
because syft v1.43 passed diskfs entries as os.FileInfo; syft v1.51.1
fixed that upstream and now requires v1.9.4, so the workaround had become
the thing breaking the build. Removed per its own "Remove when syft ships
a fix" note, closing anchore/syft#4796 for us.
The indigo bump needed no code changes: of the 21 packages we import only
5 changed, and the repo/MST/CAR-store core is byte-identical. It does
bring a util/ssrf fix blocking 6to4 addresses (2002::/16), which we
inherit through atproto/auth/oauth.
Go 1.26.7 across go.work, all five go.mod files, the four Dockerfiles,
the three tangled workflows, and the stale references in
docs/DEVELOPMENT.md. Verified golang:1.26.7-trixie resolves on
mirror.gcr.io, which is what the Dockerfiles actually pull from.
Makefile's TRIXIE_BUILDER_IMAGE stays on the floating golang:1-trixie.
make test, make lint, and make test-race all pass, as do the scanner
module's tests and the integration-tagged build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KWoKzpgtBJ33sCyGxJGR7x
last_seen means "this user did something recently". The backfill walks every
historical record in the network, so stamping it there recorded when the
backfill ran, not when the user was active — for every user at once, on every
run. That destroys the only signal the column carries, and it is the one column
in users that nothing upstream can rebuild.
It is now written on the two paths that represent real activity: an interactive
login, and a live commit event on the firehose, which does mean the user just
wrote a record. The backfill still corrects handle, PDS endpoint and avatar,
which is why it re-resolves rather than trusting a cache; it just no longer
claims the user was present.
UpsertUser grows an options form rather than a fourth named variant, since the
avatar and last_seen decisions are independent and all four combinations occur.
Anyone computing MAU from this column should know it was unreliable for every
backfill run before this change.
Also corrects docs/HORIZONTAL_SCALING.md, which claimed oci_client and
registry_domain were local-only preferences. They are fields on
io.atcr.sailor.profile: settings writes them to the user's PDS and
ProcessSailorProfile refreshes the local cache. users is fully derived apart
from last_seen.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Neither is a correctness problem; both are round trips on hot paths for
timestamps nothing reads at that resolution.
UpdateUserLastSeen ran per Jetstream event for cached users, so once per indexed
record. DeviceStore.UpdateLastUsed ran per /auth/token call, so once per docker
push and pull including each layer's re-auth. Cheap against a local file, a
network round trip each against a remote primary, and the second sat on the
authentication path.
Both are now throttled to once per five minutes per subject. The MAU queries and
the admin views work in hours or days, so nothing loses meaning. The throttle
state is per-process and lost on restart, costing at most one extra write per
subject per boot; only the lease holder runs the consumer, so exactly one process
is doing the first of these at a time.
UpdateLastUsed stamps the throttle before writing rather than after, so a slow or
failing write cannot let every concurrent layer upload through to pile on more of
them.
Verified by disabling the throttle: 50 back-to-back calls then rewrite the
timestamp every time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Design for moving push bandwidth off the appview and onto the hold,
mirroring the pull path's 307 redirect. Proposal only, not implemented.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- diff view gains a Packages tab with added/removed/changed/unchanged
package tables and purl-derived type/license/upstream links
- captain records verified against the DID's atcr_hold service before
caching (processor + batch backfill), preventing forged holds
- fix empty-handle updates clobbering cached handles and colliding on
the UNIQUE constraint
- move fillPrevCIDs into repo.go; DirectRepoOperator is now canonical,
repomgr kept as a test oracle
- surface read-only crew status in hold selector
- reconcile docs
did:plc Identity Support (pkg/hold/pds/did.go, pkg/hold/config.go, pkg/hold/server.go)
The big feature — holds can now use did:plc identities instead of only did:web. This adds:
- LoadOrCreateDID() — resolves hold DID by priority: config DID > did.txt on disk > create new
- CreatePLCIdentity() — builds a genesis operation, signs with rotation key, submits to PLC directory
- EnsurePLCCurrent() — on boot, compares local signing key + URL against PLC directory and auto-updates if they've drifted (requires rotation key)
- New config fields: did_method (web/plc), did, plc_directory_url, rotation_key_path
- GenerateDIDDocument() now uses the stored DID instead of always deriving did:web from URL
- NewHoldServer wired up to call LoadOrCreateDID instead of GenerateDIDFromURL
CAR Export/Import (pkg/hold/pds/export.go, pkg/hold/pds/import.go, cmd/hold/repo.go)
New CLI subcommands for repo backup/restore:
- atcr-hold repo export — streams the hold's repo as a CAR file to stdout
- atcr-hold repo import <file>... — reads CAR files, upserts all records in a single atomic commit. Uses a bulkImportRecords method that opens a delta session, checks each record for
create vs update, commits once, and fires repo events.
- openHoldPDS() helper to spin up a HoldPDS from config for offline CLI operations
Admin UI Fixes (pkg/hold/admin/)
- Logout changed from GET to POST — nav template now uses a <form method=POST> instead of an <a> link (prevents CSRF on logout)
- Removed return_to parameter from login flow — simplified redirect logic, auth middleware now redirects to /admin/auth/login without query params
Config/Deploy
- config-hold.example.yaml and deploy/upcloud/configs/hold.yaml.tmpl updated with the four new did:plc config fields
- go.mod / go.sum — added github.com/did-method-plc/go-didplc dependency
1. Removing distribution/distribution from the Hold Service (biggest change)
The hold service previously used distribution's StorageDriver interface for all blob operations. This replaces it with direct AWS SDK v2 calls through ATCR's own pkg/s3.S3Service:
- New S3Service methods: Stat(), PutBytes(), Move(), Delete(), WalkBlobs(), ListPrefix() added to pkg/s3/types.go
- Pull zone fix: Presigned URLs are now generated against the real S3 endpoint, then the host is swapped to the CDN URL post-signing (previously the CDN URL was set as the endpoint, which
broke SigV4 signatures)
- All hold subsystems migrated: GC, OCI uploads, XRPC handlers, profile uploads, scan broadcaster, manifest posts — all now use *s3.S3Service instead of storagedriver.StorageDriver
- Config simplified: Removed configuration.Storage type and buildStorageConfigFromFields(); replaced with a simple S3Params() method
- Mock expanded: MockS3Client gains an in-memory object store + 5 new methods, replacing duplicate mockStorageDriver implementations in tests (~160 lines deleted from each test file)
2. Vulnerability Scan UI in AppView (new feature)
Displays scan results from the hold's PDS on the repository page:
- New lexicon: io/atcr/hold/scan.json with vulnReportBlob field for storing full Grype reports
- Two new HTMX endpoints: /api/scan-result (badge) and /api/vuln-details (modal with CVE table)
- New templates: vuln-badge.html (severity count chips) and vuln-details.html (full CVE table with NVD/GHSA links)
- Repository page: Lazy-loads scan badges per manifest via HTMX
- Tests: ~590 lines of test coverage for both handlers
3. S3 Diagnostic Tool
New cmd/s3-test/main.go (418 lines) — tests S3 connectivity with both SDK v1 and v2, including presigned URL generation, pull zone host swapping, and verbose signing debug output.
4. Deployment Tooling
- New syncServiceUnit() for comparing/updating systemd units on servers
- Update command now syncs config keys (adds missing keys from template) and service units with daemon-reload
5. DB Migration
0011_fix_captain_successor_column.yaml — rebuilds hold_captain_records to add the successor column that was missed in a previous migration.
6. Documentation
- APPVIEW-UI-FUTURE.md rewritten as a status-tracked feature inventory
- DISTRIBUTION.md renamed to CREDENTIAL_HELPER.md
- New REMOVING_DISTRIBUTION.md — 480-line analysis of fully removing distribution from the appview side
7. go.mod
aws-sdk-go v1 moved from indirect to direct (needed by cmd/s3-test).
Evan JarrettandClaude Fable 5.1