TestBenchRealImages pushes and pulls three images whose layer sizes are
copied from real manifests in the production appview database (the median,
p75 and p90 images by layer count) and reports, per operation, wall time and
the number of requests to the registry, the fake PDS, the hold and S3, broken
down by endpoint. Skipped unless BENCH_PROFILES is set, so the integration
target does not run it. BENCH_LAT_{PDS,HOLD,S3} inject per-request latency,
which is what makes byte-path changes visible in-process; request counts are
the reliable signal either way.
internal/reqcount counts and delays requests through a handler wrapper and a
client-side RoundTripper. testharness.WithBackendTap wraps the PDS and S3
handlers and puts a counting reverse proxy in front of the hold;
testpds.WithMiddleware is the hook that makes the PDS side possible.
The bench showed a pull costs three hold calls per blob, not two: distribution
installs its notifications listener unconditionally and it re-Stats every blob
after ServeBlob to build the pull event. The backlog's presign memoization
item is rewritten with the measured numbers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WTdBxLFU5TpwmqVdVsN1wq
server.test_mode survived the build-tag refactor only to feed five
behavioral branches: the registry's fall-back to the default hold when
the user's hold is unreachable, backfill warning suppression for
external holds, the appview listener close on shutdown, the hold's
relay-crawl skip, and the hold's appview-issuer tolerance. Every one of
them is a "this is a local development build" decision, which is what
the tag already says, and local development has to build with the tag
or nothing resolves. So they read atproto.TestModeBuild now, and the
flag, SetTestMode, IsTestMode, the middleware option, the backfill
constructor parameter, the never-read field on RemoteHoldAuthorizer,
the example and template YAML lines, and the docker-compose env vars
are gone. The registry keeps the fallback as a field seeded from the
constant so the production-path tests can pin it off under the tag.
The 24 SetTestMode calls in tests were dead already: stripping them and
running the affected packages tagged changed nothing.
Tests that resolve a loopback did:web used to t.Fatal naming the tag,
which left a bare `go test ./...` permanently red in five packages.
They now live under `//go:build testmode`: whole-file constraints where
every test needs it, and sibling *_testmode_test.go files holding the
moved tests plus their fixtures where a file mixed. The harness carries
the constraint too, with its package doc in an untagged doc.go so the
package still exists without it. An untagged run compiles those tests
out and passes; make test keeps the tag and runs everything.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UwYzaG3Yy7uA8FbZ5qk3tQ
indigo's identity directory refuses HTTP and IP-hosted did:web, and its
OAuth client is growing an SSRF-guarded transport that refuses loopback
and private addresses. Local development and the test suites need both,
and the workarounds were scattered: two did:web fallbacks in the
resolver, a hand-rolled appview key fetch on the hold, and the OAuth
client left on indigo's defaults so any test driving it against an
httptest server depended on the transport staying permissive.
Move every departure from indigo's defaults into one file pair in
pkg/atproto: indigo_prod.go (!testmode) returns indigo's directory and
OAuth client unchanged; indigo_local.go (testmode) wraps the directory
so a did:web naming an IP, localhost, or a host with a port resolves
over plain HTTP, and gives the OAuth client plain HTTP clients. All six
identity and OAuth constructor call sites go through NewDirectory and
NewOAuthClientApp. The resolver fallbacks, DIDWebToURL, and the hold's
scheme-guessing key fetch are gone; the hold resolves the appview key
through the directory, preferring #appview, and purges and retries once
on a signature failure so a re-keyed appview is not masked by the
24-hour cache.
There is no runtime switch for this: a production binary cannot be
configured to resolve local DIDs. The runtime test_mode flag still
gates the remaining behavioral branches only.
Tests, the harness, make dev, Air, Dockerfile.dev, and docker-compose
build with the tag; fixtures that need loopback did:web fail fast
naming it. Test hold servers now serve a did.json via pkg/testpds so
they resolve as real holds under the tag.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UwYzaG3Yy7uA8FbZ5qk3tQ
A large layer went up strictly one step at a time: fill 16MB from the
client, stop reading, fetch a part URL and PUT the part to S3, reset,
resume reading. While the part was in flight Docker sat on a full TCP
window; while the buffer filled S3 sat idle. Wall clock was receive
time plus send time.
The writer now hands a full buffer to a goroutine that does the hold
call and the PUT, and keeps filling a second buffer from the client.
When that one fills it waits for the previous part, takes its buffer
back, and hands the new one off. At most one part is in flight, so part
numbers and ETags stay ordered, and a blob that never fills a buffer
never allocates the second one. No network runs under the writer lock
on the happy path.
Peak memory for a large upload is now two buffers, 32MB. Both are
charged to the process budget through the existing accounting, the
second as it grows, and the budget floor rises to match so a large
upload can never be refused outright. The 512MB default holds sixteen.
A failed part records a sticky error, closes the writer, and aborts the
multipart from the goroutine that still holds the upload ID; the next
Write, hand-off, or Commit reports the cause. Commit verifies the digest
first, then waits for the flight, sends the final part, and completes.
Cancel waits for the flight, bounded, before aborting so the abort
cannot overtake a PUT that has not yet been issued its upload ID. The
sweeper refuses to reap a writer with a part in flight, since last
activity is only stamped when a part lands.
Tests observe the overlap directly: the fake S3 blocks the first PUT and
the second buffer's writes are asserted to return before it is released,
while the third buffer's writes block. Also covered: the one-part-late
error, Commit waiting, Cancel during flight, peak budget, the sweeper,
and concurrent Cancel and Write under the race detector. The
integration suite passed with a 72MB layer pushed through the pipeline
by three clients.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
The September 2026 read of the appview and hold request paths produced
more findings than were worth doing immediately. This records what
shipped, what remains with problem, location, fix and expected impact,
one non-performance bug found along the way, and the production size
distribution that informed the 16MB direct-PUT threshold.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
Hold discovery in the registry middleware called getRecord on the
repository owner's PDS for every request under /v2/: every HEAD, POST,
PATCH, PUT and GET. A 10-layer push was 40 or more PDS round trips, and
it was the last per-request network call on the push path that had
nothing to do with moving bytes. Only two profile fields are used
there: the default hold and the auto-remove-untagged flag.
The users row already caches the default hold, written by the Jetstream
processor on every profile event and prefilled by the backfill, and the
auth gate already reads it from there. This makes the row a faithful
copy of what the registry needs and switches the middleware to it.
The auto-remove flag gets a nullable users column. NULL means the value
has never been learned; the processor writes 0 or 1 on every profile
event and never NULL. On a request whose row is missing or still NULL,
the middleware does one live fetch, uses it, and writes both fields
back, including a 0 for a user with no profile at all, so the fallback
runs at most once per user. A failed fetch writes nothing and uses the
appview default for that request, so a network error is never cached.
That single mechanism covers the minutes after a deploy while the
startup backfill fills the column, a brand-new user, and a user the
backfill has not reached.
The processor also stops returning early on an empty default hold,
which left a user who removed their custom hold pushing to it forever.
Empty is now written through and means the appview default, matching
what the auth gate already reads.
Tests count PDS requests with a test server: a populated row makes
none, a NULL row makes exactly one and then none, a missing profile is
cached as known, and a failed fetch degrades without writing. The
migration was applied to a fresh database and to one built from the
previous schema.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
Each in-flight blob upload buffers up to 16MB, Docker pushes five layers
at once per client, and nothing bounded the total. Writers also lived in
the package-level map forever: a client that died mid-push left its
writer, its buffer, and any hold-side S3 multipart session behind with
no expiry.
A process-wide budget (golang.org/x/sync semaphore, default 512MB,
server.upload_buffer_budget_mb) now caps memory held in upload buffers.
A writer charges its buffer's projected backing capacity before growing,
so a config blob costs kilobytes and a full writer costs exactly one
buffer, and releases once, on Commit, Cancel, or reap. A write that
needs budget waits on the request's context with a five minute cap,
outside the writer's lock so Cancel and the sweeper cannot queue behind
it; that wait is backpressure on the client. The budget is clamped to
at least one buffer so a single upload can never deadlock.
A sweeper started with the other appview workers reaps writers idle
past server.upload_idle_timeout (default 1h), aborting the hold-side
multipart on a detached context and releasing the budget. It measures
inactivity, not age, so a slow push is never reaped, and it skips a
writer whose lock is held so it cannot race a live part upload.
Write also gains a fix the budget made visible. It appended a whole
chunk and checked afterwards, so the last chunk before a flush could
land a few bytes past 16MB, which did not fit the backing array;
bytes.Buffer doubled it to 32MB and Reset kept that for the rest of the
upload. Only chunk sizes that tile 16MB exactly avoided it, and the
network read loop promises no such thing. Every large layer could hold
32MB while the budget charged 16. Write now fills to exactly the
threshold, flushes, and continues with the remainder, so capacity is
pinned at 16MB for any chunk size, every part is exactly one buffer,
and a single oversized Write streams through as parts instead of
buffering whole. The test streams 24KB chunks across the boundary and
fails against the old code with cap 33554432.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
Every blob went through the multipart machinery: an S3 multipart started
on Docker's initial POST, a hold round trip per part, and a complete on
the hold that finished the multipart, HEADed the temp object, copied it
to its final key, and deleted the temp. For a 2KB config blob that was
three hold calls and six S3 operations. On production data 86% of
distinct layers and every config blob fit in a 16MB buffer, and 49% of
image manifests have no layer larger than that.
The writer now buffers up to 16MB (also the multipart part size) and
makes no hold call until it has to. A blob that never overflows the
buffer is written at Commit with a single presigned PUT to its final
key, via the hold's existing method=PUT presign; the multipart only
starts on the first flush. The hold's completeUpload does nothing the
direct path skips: quota, layer records, stats and scan dispatch all
hang off notifyManifest, which is unchanged.
The buffer starts empty and grows on demand, with the doubling capped so
capacity never overshoots 16MB: a config blob costs kilobytes, and only
layers that approach the threshold fill it.
Bytes are hashed as they arrive. Commit compares the computed sha256 to
the digest the client claimed before any network call, and returns
DIGEST_INVALID on mismatch, aborting a multipart if one was started.
Previously nothing verified the content, so a pusher could store wrong
bytes under a digest in the shared content-addressed space.
Tests observe request counts on a fake hold and fake S3 rather than
return values. The growth test streams in 24KB chunks because
power-of-two chunks land on 16MB by luck and hid an earlier weaker guard.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
distribution calls Stat before every blob GET and HEAD. The appview's
Stat asked the hold for a presigned HEAD URL and then HEADed S3 with it
purely to read Content-Length for the descriptor: two round trips to
learn one number.
The hold's getBlob response for OCI digests on GET and HEAD now carries
"size". It comes from the records index when a layer record exists (a
SQLite lookup on a new digest index, no network) and from a HeadObject
otherwise, which is where config blobs land. If storage says the object
does not exist the hold answers 404 instead of signing a URL that can
only fail. The PUT and ATProto CID paths are untouched.
The appview builds the descriptor from the reported size and makes no
S3 request. When the field is absent it HEADs the presigned URL as
before, so a new appview works against a hold that has not been
upgraded, and an old appview ignores the extra field. A hold 404 maps
to ErrBlobUnknown.
Tests prove the index answered by leaving the mock bucket empty and
counting zero HeadObject calls, prove the fallback with exactly one, and
count requests reaching the fake S3 origin on the appview side rather
than trusting the returned size.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5
buildOCILayout already removes its scan dir on every error path, and syft.go
defers the stereoscope generator's Cleanup. What neither can do is clean up
after a process that dies mid-scan: the deferred call never runs, and nothing
afterwards ever looks at what was left. Every restart therefore leaks the
in-flight layout and extraction permanently, and a restart is routine — a
deploy is one.
On seamark-hold that reached 8.8 GB of orphaned scan-*, syft-scan-* and
syft-cataloger-* directories under a 20 GB disk, at which point the disk was
97% full and scans began failing on it:
failed to load OCI image: unable to populate layer cache
dir="/var/lib/seamark/scanner/tmp/syft-scan-1546187834/..."
: no space left on device
failed to download layer 5: failed to write blob:
write /var/lib/seamark/scanner/tmp/scan-4160414849/blobs/sha256/...
: no space left on device
The leaked directories cluster at the scanner's restart timestamps, which is
what identifies the killed process rather than the error paths as the source.
Manual removal reclaimed 8.8 GB and took the disk from 97% to 50%.
Startup is where this belongs: it is the one moment the previous process is
known to be gone, and it is immediately after the event that caused the leak.
The sweep runs in WorkerPool.Start after TMPDIR is set and before any worker
can dequeue, so nothing it removes can be work in progress here.
Three constraints shape what it will touch:
- Only the three per-job prefixes, only as direct children, only
directories. The Grype database lives beside the tmp dir at
<parent>/vulndb and go-getter unpacks into grype-dl underneath it; both
are state the scanner needs and neither matches a prefix. The prefixes now
have one definition each, used by both the creator and the sweeper, so
renaming a directory cannot silently take it out of the sweep's scope.
- An age threshold, vuln.sweep_max_age, default 1h. A second scanner sharing
the directory has an in-flight scan-* dir that is minutes old, and
scanner.job_timeout is 8m, so an hour clears both with room to spare. 0
disables the sweep rather than removing a peer's live work.
- Nothing is fatal. A stat or removal failure is a WARN and the sweep moves
on, so a permission problem in the tmp dir cannot keep the scanner from
starting.
The sweep only runs at startup, so a scanner that is killed twice between
deploys carries the first leak until its next restart. That is the tradeoff
for never racing a live peer; a periodic sweep would be the follow-up if
processes ever live long enough for it to matter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TA9D4DjaLZTvzQ7dJbu4eg
An audit of the scan pipeline and the hold side of scanning found several
ways scanning stops without saying so. Each fix here was written test-first:
a test expressing the wanted behaviour, confirmed failing for the right
reason, then the change.
A summary-less result crash-looped both processes. worker.go dereferenced
result.Summary unconditionally, but processJob only sets it when Grype runs,
and SendResult puts the nil on the wire before the scanner dies on it, so
handleResult's unguarded log killed the hold too. A nil Summary now means
"not scanned for vulnerabilities", deliberately distinct from "scanned, found
zero" — inventing a zeroed summary would report every image as clean when
Grype never ran. The hold writes a record rather than orphaning the uploaded
SBOM, and the appview renders an "SBOM only" state instead of a green Clean
badge.
The Grype database could wedge with no way back short of a restart. All three
throttles in loadVulnDatabase were guarded by vulnDB != nil, so a scanner
holding no provider retried a full download on every scan under the exclusive
lock. Two earlier attempts at this bug each added one more condition to the
same chain; this replaces the chain with a single decision function over a
state snapshot, consulted by both call sites so they cannot disagree. That
disagreement was itself a bug: the 50-scan reload had never once executed.
Two independent halts. An unparseable frame was dropped in silence, stranding
a row that held the hold's only dispatch slot forever; it is now answered
"skipped" on first delivery. The 10-minute sweep leaked the in-flight digest
and wrote no record, permanently retiring one image per timeout.
A digest went unvalidated into filepath.Join and os.Create, so a layer digest
of sha256:../../../x wrote outside the scan directory, and nothing verified
that downloaded bytes hashed to the digest naming them. Digests come from
records in a user's own PDS. Both are fixed together: verification is what
makes an escaping write self-defeating.
Concurrency did not work on either axis. The proactive capacity gate was
depth-one hold-wide, so neither extra workers nor extra scanner processes
received work. Depth is now the sum of the worker counts scanners advertise on
connect, the gate is scoped to proactive work, and dispatch prefers the
least-loaded scanner. Disconnects no longer hand a running scan to someone
else: a scanner keeps a stable per-process identity and reclaims its own rows
within a grace window, while a process that truly restarted returns with a new
identity and has its work reclaimed, which is correct because the restart did
lose it.
The hold's scanning deadline measured queueing rather than scanning, because
the scanner acks on receipt and handleAck never refreshed assigned_at. A new
"started" message, sent by the worker that dequeues the job, separates the two
budgets. An older scanner never sends it and falls under the queueing budget,
which is more forgiving than the deadline it gets today.
Adds an in-process mock hold and an e2e harness that runs the real client,
queue and worker pool, seeded with 84 real manifest records fetched from a
live PDS. Real image layouts and the Grype database are fetched by scripts and
gitignored; suites needing them skip cleanly, so the default run stays offline
and fast.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF
Around 2026-08-20 the Bluesky relay marked every ATCR hold offline and stopped
dialing. Nobody noticed for two weeks, and it surfaced only indirectly as "pull
and push counts are up to 24 h stale". The stats were a symptom; the fleet was
simply disconnected.
It cannot recover on its own. Indigo's relay gives up on a host after 16
consecutive dial failures and returns from the redialer, and only a fresh
requestCrawl revives it. The hold sends requestCrawl exactly once, at boot
(server.go:400), with no ticker and no check that any relay is subscribed. So a
dropped hold is invisible until its process restarts, silently.
Documents the current mechanics, the failure mode, how to diagnose it with
getHostStatus and a frozen repo rev, and how to recover. The automatic
re-crawl is described as a deferred proposal and explicitly NOT implemented,
by decision: a jittered ticker guarded on subscriber liveness, plus surfacing
the subscriber count, since the deeper problem is that this was silent.
Two things found while writing it, both recorded. The proposal needs plumbing
that does not exist: EventBroadcaster has no exported subscriber count, and
Subscriber does not retain the userAgent, so "is a relay listening" cannot
currently be answered. And ResubscribeAllHosts selects only active hosts, so an
offline host is not recovered even by a relay restart.
Carries a replay warning. ca539b1 fixed a panic on subscriber disconnect during
firehose backfill, and the exposure condition is that a backfill goroutine
exists at all, which Subscribe skips when the cursor is current. So a caught-up
relay never triggered it and a hold whose relays are far behind is exposed on
every reconnect. Verify a deployed hold contains ca539b1 before provoking a
re-crawl; efabb677 does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9
Finding 32: pushing an artifact with an unrecognised config media type
classifies as "unknown", and every template branches two ways on "helm-chart"
with the container-image page in the else. So an in-toto attestation is served
a docker pull command, Layers/Vulnerabilities/SBOM tabs, "Image layer history",
and a promise that scans run shortly after push, which for that artifact will
never be true. This is a design note rather than a fix, since the change is
larger than the symptom.
The inventory is the part worth keeping. The classification rule exists in four
places, keyed off three different inputs (appview config media type, hold config
media type with no unknown case, scanner config map plus layer shape, hold layer
substrings), and the appview's artifact_type feeds none of the scan decisions.
Manifest-level artifactType is discarded at parse time on every push: it is
absent from the record struct, the constructor and the lexicon, surviving only
inside the unindexed manifest blob.
Two corrections to the framing this started from, both verified rather than
assumed. The repo does not have referrers support: the pinned distribution
version has no referrers code and ATCR registers no such route, so what exists
is subject_digest persistence plus an attestation badge. And GetTopLevelManifests
filters artifact_type != 'unknown', so an untagged unknown artifact is invisible
while a tagged one renders as an image, which the finding did not mention.
Proposes a type set, spec precedence (manifest artifactType, then config media
type, then structural signals), a UI contract stating what such a page must not
show, and a six stage plan. Only stage 2 needs a migration, for the raw string
column; new slug values need no DDL and no data migration, since jetstream
upserts artifact_type on every record it sees.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9
seamark.dev's /install and /settings/devices told users to pipe
seamark.dev/static/install.sh into bash. That file was the unmodified ATCR
script: it announced itself as the "ATCR Credential Helper Installer",
installed docker-credential-atcr, and finished by telling the user to configure
credHelpers for atcr.io, the wrong registry for that deployment. Anyone
following the documented setup ended up pointed at another service. The
templates hardcoded docker-credential-atcr, "atcr" and ~/.atcr/device.json
alongside a correctly themed {{ .RegistryURL }}.
The scripts are now rendered from config by a handler, rather than forked per
brand. A theme overlay was the alternative and was worse: it needed a full copy
of both install.sh and install.ps1 per brand, four scripts to keep in sync, and
the operator asked for these values to come from config.
credential_helper.name is the single knob. Docker resolves a credHelpers value
x by exec'ing docker-credential-x, so the credHelpers value, the binary suffix
and the config directory are genuinely one word, not three that can drift. It
is validated against a strict pattern because it is interpolated into a shell
script.
install.sh renders byte-identical to the deleted static file under the atcr
default, so existing installs are unaffected. install.ps1 differs by one line,
where a stale usage comment named a path the script is not served at.
Two behaviour changes worth noting: these two URLs drop from a one-year
Cache-Control to five minutes, since the body now depends on deployment config;
and credential_helper.tangled_repo becomes a real overridable default. It was
previously assigned over unconditionally and read by nothing, while the shipped
script used a different URL form.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9
Update every direct dependency across all five workspace modules to
latest. Notable jumps: syft v1.43.0 -> v1.51.1, grype v0.111.1 ->
v0.118.0, stereoscope v0.1.23 -> v0.3.1, indigo -> 2026-09-01,
aws-sdk-go-v2/service/s3 v1.99.1 -> v1.110.0, grpc v1.80.0 -> v1.83.2,
x/crypto v0.50.0 -> v0.55.0.
Three deps needed more than a version bump:
go-libipfs could not be updated at all. The repo was renamed to boxo, so
every tag past v0.7.0 declares `module github.com/ipfs/boxo` and cannot
be required under the old path. sqlite_store.go already imported
go-block-format alongside it and used the archived package exactly once,
inside a function already returning blockformat.Block, so it was relying
on structural interface satisfaction. Collapsing to the native type drops
the archived dependency entirely.
go-didplc moved its package from the repo root into a didplc/ subdir in
v0.2.2. Package name is unchanged and every symbol we use (RegularOp,
OpEnum, OpService, Client.DirectoryURL, Submit) is intact, so this is an
import path change only.
The go-diskfs replace in scanner/go.mod had inverted. It pinned v1.7.0
because syft v1.43 passed diskfs entries as os.FileInfo; syft v1.51.1
fixed that upstream and now requires v1.9.4, so the workaround had become
the thing breaking the build. Removed per its own "Remove when syft ships
a fix" note, closing anchore/syft#4796 for us.
The indigo bump needed no code changes: of the 21 packages we import only
5 changed, and the repo/MST/CAR-store core is byte-identical. It does
bring a util/ssrf fix blocking 6to4 addresses (2002::/16), which we
inherit through atproto/auth/oauth.
Go 1.26.7 across go.work, all five go.mod files, the four Dockerfiles,
the three tangled workflows, and the stale references in
docs/DEVELOPMENT.md. Verified golang:1.26.7-trixie resolves on
mirror.gcr.io, which is what the Dockerfiles actually pull from.
Makefile's TRIXIE_BUILDER_IMAGE stays on the floating golang:1-trixie.
make test, make lint, and make test-race all pass, as do the scanner
module's tests and the integration-tagged build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KWoKzpgtBJ33sCyGxJGR7x
last_seen means "this user did something recently". The backfill walks every
historical record in the network, so stamping it there recorded when the
backfill ran, not when the user was active — for every user at once, on every
run. That destroys the only signal the column carries, and it is the one column
in users that nothing upstream can rebuild.
It is now written on the two paths that represent real activity: an interactive
login, and a live commit event on the firehose, which does mean the user just
wrote a record. The backfill still corrects handle, PDS endpoint and avatar,
which is why it re-resolves rather than trusting a cache; it just no longer
claims the user was present.
UpsertUser grows an options form rather than a fourth named variant, since the
avatar and last_seen decisions are independent and all four combinations occur.
Anyone computing MAU from this column should know it was unreliable for every
backfill run before this change.
Also corrects docs/HORIZONTAL_SCALING.md, which claimed oci_client and
registry_domain were local-only preferences. They are fields on
io.atcr.sailor.profile: settings writes them to the user's PDS and
ProcessSailorProfile refreshes the local cache. users is fully derived apart
from last_seen.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Neither is a correctness problem; both are round trips on hot paths for
timestamps nothing reads at that resolution.
UpdateUserLastSeen ran per Jetstream event for cached users, so once per indexed
record. DeviceStore.UpdateLastUsed ran per /auth/token call, so once per docker
push and pull including each layer's re-auth. Cheap against a local file, a
network round trip each against a remote primary, and the second sat on the
authentication path.
Both are now throttled to once per five minutes per subject. The MAU queries and
the admin views work in hours or days, so nothing loses meaning. The throttle
state is per-process and lost on restart, costing at most one extra write per
subject per boot; only the lease holder runs the consumer, so exactly one process
is doing the first of these at a time.
UpdateLastUsed stamps the throttle before writing rather than after, so a slow or
failing write cannot let every concurrent layer upload through to pile on more of
them.
Verified by disabling the throttle: 50 back-to-back calls then rewrite the
timestamp every time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Design for moving push bandwidth off the appview and onto the hold,
mirroring the pull path's 307 redirect. Proposal only, not implemented.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- diff view gains a Packages tab with added/removed/changed/unchanged
package tables and purl-derived type/license/upstream links
- captain records verified against the DID's atcr_hold service before
caching (processor + batch backfill), preventing forged holds
- fix empty-handle updates clobbering cached handles and colliding on
the UNIQUE constraint
- move fillPrevCIDs into repo.go; DirectRepoOperator is now canonical,
repomgr kept as a test oracle
- surface read-only crew status in hold selector
- reconcile docs
did:plc Identity Support (pkg/hold/pds/did.go, pkg/hold/config.go, pkg/hold/server.go)
The big feature — holds can now use did:plc identities instead of only did:web. This adds:
- LoadOrCreateDID() — resolves hold DID by priority: config DID > did.txt on disk > create new
- CreatePLCIdentity() — builds a genesis operation, signs with rotation key, submits to PLC directory
- EnsurePLCCurrent() — on boot, compares local signing key + URL against PLC directory and auto-updates if they've drifted (requires rotation key)
- New config fields: did_method (web/plc), did, plc_directory_url, rotation_key_path
- GenerateDIDDocument() now uses the stored DID instead of always deriving did:web from URL
- NewHoldServer wired up to call LoadOrCreateDID instead of GenerateDIDFromURL
CAR Export/Import (pkg/hold/pds/export.go, pkg/hold/pds/import.go, cmd/hold/repo.go)
New CLI subcommands for repo backup/restore:
- atcr-hold repo export — streams the hold's repo as a CAR file to stdout
- atcr-hold repo import <file>... — reads CAR files, upserts all records in a single atomic commit. Uses a bulkImportRecords method that opens a delta session, checks each record for
create vs update, commits once, and fires repo events.
- openHoldPDS() helper to spin up a HoldPDS from config for offline CLI operations
Admin UI Fixes (pkg/hold/admin/)
- Logout changed from GET to POST — nav template now uses a <form method=POST> instead of an <a> link (prevents CSRF on logout)
- Removed return_to parameter from login flow — simplified redirect logic, auth middleware now redirects to /admin/auth/login without query params
Config/Deploy
- config-hold.example.yaml and deploy/upcloud/configs/hold.yaml.tmpl updated with the four new did:plc config fields
- go.mod / go.sum — added github.com/did-method-plc/go-didplc dependency
1. Removing distribution/distribution from the Hold Service (biggest change)
The hold service previously used distribution's StorageDriver interface for all blob operations. This replaces it with direct AWS SDK v2 calls through ATCR's own pkg/s3.S3Service:
- New S3Service methods: Stat(), PutBytes(), Move(), Delete(), WalkBlobs(), ListPrefix() added to pkg/s3/types.go
- Pull zone fix: Presigned URLs are now generated against the real S3 endpoint, then the host is swapped to the CDN URL post-signing (previously the CDN URL was set as the endpoint, which
broke SigV4 signatures)
- All hold subsystems migrated: GC, OCI uploads, XRPC handlers, profile uploads, scan broadcaster, manifest posts — all now use *s3.S3Service instead of storagedriver.StorageDriver
- Config simplified: Removed configuration.Storage type and buildStorageConfigFromFields(); replaced with a simple S3Params() method
- Mock expanded: MockS3Client gains an in-memory object store + 5 new methods, replacing duplicate mockStorageDriver implementations in tests (~160 lines deleted from each test file)
2. Vulnerability Scan UI in AppView (new feature)
Displays scan results from the hold's PDS on the repository page:
- New lexicon: io/atcr/hold/scan.json with vulnReportBlob field for storing full Grype reports
- Two new HTMX endpoints: /api/scan-result (badge) and /api/vuln-details (modal with CVE table)
- New templates: vuln-badge.html (severity count chips) and vuln-details.html (full CVE table with NVD/GHSA links)
- Repository page: Lazy-loads scan badges per manifest via HTMX
- Tests: ~590 lines of test coverage for both handlers
3. S3 Diagnostic Tool
New cmd/s3-test/main.go (418 lines) — tests S3 connectivity with both SDK v1 and v2, including presigned URL generation, pull zone host swapping, and verbose signing debug output.
4. Deployment Tooling
- New syncServiceUnit() for comparing/updating systemd units on servers
- Update command now syncs config keys (adds missing keys from template) and service units with daemon-reload
5. DB Migration
0011_fix_captain_successor_column.yaml — rebuilds hold_captain_records to add the successor column that was missed in a previous migration.
6. Documentation
- APPVIEW-UI-FUTURE.md rewritten as a status-tracked feature inventory
- DISTRIBUTION.md renamed to CREDENTIAL_HELPER.md
- New REMOVING_DISTRIBUTION.md — 480-line analysis of fully removing distribution from the appview side
7. go.mod
aws-sdk-go v1 moved from indirect to direct (needed by cmd/s3-test).
Evan JarrettandClaude Opus 4.8