mirror of
https://tangled.org/evan.jarrett.net/at-container-registry
synced 2026-09-20 17:24:16 +00:00
dcad0f8626a8194eebb2e526d2106e055725cfb2
666
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Evan JarrettandClaude Fable 5.1
|
dcad0f8626 |
deploy: snapshot binaries, configs, units and the appview database before every update
The update command deleted the remote binary before uploading the new one and took no backup, so a deploy that failed half way left nothing to roll back to. Previous production deploys made a predeploy-<stamp> directory by hand. Each target now gets that snapshot automatically before anything is replaced: the service binary and its companion (scanner with hold, labeler with appview), their configs and systemd units, the vcs.revision of each old binary, and for the appview a sqlite3 .backup of ui.db because migrations run on the next boot. A backup that does not complete aborts the deploy. The newest five per server are kept; --backup-keep changes that and --skip-backup turns it off. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EvFJr4Dwz8p2NDAeXmgmBt |
||
|
Evan JarrettandClaude Opus 4.8
|
7ecb09465d |
docs: move presign memoization to the done list
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WTdBxLFU5TpwmqVdVsN1wq |
||
|
Evan JarrettandClaude Opus 4.8
|
2a94f924af |
appview: memoize blob presigns within a request
A blob GET cost three hold calls, not two. distribution's blob handler calls Stat then ServeBlob, and its notifications listener (installed unconditionally, endpoints or not) calls Stat a third time after ServeBlob to build the pull event. Each Stat presigned method=HEAD and ServeBlob presigned the request method, so a p90 pull of 22 layers was 66 hold getBlob calls, 44 of them HEAD presigns whose URL was discarded. ProxyBlobStore is built once per request (the sync.Once in RoutingRepository.Blobs), so a per-instance memo keyed by digest+method is request-scoped. Stat reads the request method from the context and presigns for GET or HEAD accordingly (anything else, including the push existence check and manifest verification, still presigns HEAD, which the hold's read path requires). ServeBlob and the listener's second Stat then hit the memo. S3 signs the HTTP verb, so the key includes the method: a HEAD URL cannot serve a GET. Only successful presigns are cached, so error semantics are unchanged. The missing-size fallback (older hold) now probes with a HEAD-signed URL rather than GETting the blob body when Stat presigned for GET. Measured with TestBenchRealImages: p90 pull drops from 67 hold calls to 23, one per blob, and wall time at PDS 50ms / hold 5ms / S3 20ms drops 23%. Push counts are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WTdBxLFU5TpwmqVdVsN1wq |
||
|
Evan JarrettandClaude Fable 5.1
|
bd740060b2 |
make: lint the integration packages with their build tags
The lint target never passed the integration or stripe_integration tags, so every file under test/ was compiled out and unchecked. Add a pass for each and move the integration package comment onto the package's one non-test file, which is where staticcheck looks for it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WTdBxLFU5TpwmqVdVsN1wq |
||
|
Evan JarrettandClaude Fable 5.1
|
bf4e63e810 |
test: production-shaped push/pull benchmark with per-backend request counts
TestBenchRealImages pushes and pulls three images whose layer sizes are
copied from real manifests in the production appview database (the median,
p75 and p90 images by layer count) and reports, per operation, wall time and
the number of requests to the registry, the fake PDS, the hold and S3, broken
down by endpoint. Skipped unless BENCH_PROFILES is set, so the integration
target does not run it. BENCH_LAT_{PDS,HOLD,S3} inject per-request latency,
which is what makes byte-path changes visible in-process; request counts are
the reliable signal either way.
internal/reqcount counts and delays requests through a handler wrapper and a
client-side RoundTripper. testharness.WithBackendTap wraps the PDS and S3
handlers and puts a counting reverse proxy in front of the hold;
testpds.WithMiddleware is the hook that makes the PDS side possible.
The bench showed a pull costs three hold calls per blob, not two: distribution
installs its notifications listener unconditionally and it re-Stats every blob
after ServeBlob to build the pull event. The backlog's presign memoization
item is rewritten with the measured numbers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WTdBxLFU5TpwmqVdVsN1wq
|
||
|
Evan JarrettandClaude Fable 5.1
|
65db7945b2 |
appview: drop the testmode fall-back-to-default-hold probe
The registry middleware used to GET the user's hold's /.well-known/did.json on every registry request in testmode builds, and fall back to the appview default hold if it did not answer. The fallback only ever changed anything when the user's chosen hold differed from the default AND was down; in local development and in the integration harness the two are the same hold, so the probe's answer was discarded every time. In the production-shaped benchmark it was 94 of 150 hold requests per p90 push and 23 of 90 per pull, hiding the real hold traffic behind a testmode artifact. Remove isHoldReachable, the fallbackUnreachable field, and the probe branch. An empty choice still means the default hold. Testmode and production now take the same path here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WTdBxLFU5TpwmqVdVsN1wq |
||
|
Evan JarrettandClaude Fable 5.1
|
dcee8f6a62 |
deps: upgrade every module; gate loopback OAuth tests on testmode
go get -u across the root, scanner, and deploy modules, then tidy. The credential helpers pin atcr.io v0.1.4 for standalone go install and are left alone (go work sync tried to strip that pin; reverted). Direct upgrades in the root: indigo 20260901 to 20260903, aws-sdk-go-v2 core 1.45.1 to 1.47.0 with config, credentials, and s3 alongside, x/crypto 0.55 to 0.57, x/net, x/sync, x/sys, x/image, klauspost/compress 1.20, go-containerregistry 0.22.1, goldmark 1.8.6, regclient 0.11.6 (pinned only by the integration-tagged package, so the bulk upgrade skipped it). Scanner and deploy had no direct updates; their indirect sets moved. The indigo delta is a hardening series: identity.DefaultDirectory and oauth.NewClientApp now carry an SSRF-guarded transport that refuses loopback and private ranges, did:web and well-known bodies are size capped, all auth-server endpoints must be HTTPS URLs, and MST decoding validates PrefixLen on untrusted nodes. Production is unaffected. The testmode seam in pkg/atproto absorbs the rest: a probe confirmed an untagged build now refuses 127.0.0.1 with indigo's unsafe-address error and a tagged build dials through. Two OAuth tests drove the real client against httptest servers on loopback and failed untagged after the bump; three siblings in the same fixtures passed only because the refused dial happened to satisfy a "transient error" assertion. All five, with their fixtures and fake stores, move under //go:build testmode in sibling files. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UwYzaG3Yy7uA8FbZ5qk3tQ |
||
|
Evan JarrettandClaude Fable 5.1
|
0080957a21 |
remove the runtime test_mode switch; the testmode build tag is the only one
server.test_mode survived the build-tag refactor only to feed five behavioral branches: the registry's fall-back to the default hold when the user's hold is unreachable, backfill warning suppression for external holds, the appview listener close on shutdown, the hold's relay-crawl skip, and the hold's appview-issuer tolerance. Every one of them is a "this is a local development build" decision, which is what the tag already says, and local development has to build with the tag or nothing resolves. So they read atproto.TestModeBuild now, and the flag, SetTestMode, IsTestMode, the middleware option, the backfill constructor parameter, the never-read field on RemoteHoldAuthorizer, the example and template YAML lines, and the docker-compose env vars are gone. The registry keeps the fallback as a field seeded from the constant so the production-path tests can pin it off under the tag. The 24 SetTestMode calls in tests were dead already: stripping them and running the affected packages tagged changed nothing. Tests that resolve a loopback did:web used to t.Fatal naming the tag, which left a bare `go test ./...` permanently red in five packages. They now live under `//go:build testmode`: whole-file constraints where every test needs it, and sibling *_testmode_test.go files holding the moved tests plus their fixtures where a file mixed. The harness carries the constraint too, with its package doc in an untagged doc.go so the package still exists without it. An untagged run compiles those tests out and passes; make test keeps the tag and runs everything. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UwYzaG3Yy7uA8FbZ5qk3tQ |
||
|
Evan JarrettandClaude Fable 5.1
|
d8643ee03a |
config: describe what test_mode still controls
The comments on server.test_mode predated the testmode build tag and still claimed the flag allows HTTP DID resolution (appview) or changes OAuth redirects (hold). Neither is true: DID resolution is decided at build time now, and the hold's OAuth redirect never read the flag. Say what remains behind the runtime switch on each side and point at the build tag for the rest. Only the comment lines in the example YAMLs are updated; the examples carry hand-edited values and are not regenerated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UwYzaG3Yy7uA8FbZ5qk3tQ |
||
|
Evan JarrettandClaude Fable 5.1
|
a01b08b924 |
atproto: gate local indigo behavior behind a testmode build tag
indigo's identity directory refuses HTTP and IP-hosted did:web, and its OAuth client is growing an SSRF-guarded transport that refuses loopback and private addresses. Local development and the test suites need both, and the workarounds were scattered: two did:web fallbacks in the resolver, a hand-rolled appview key fetch on the hold, and the OAuth client left on indigo's defaults so any test driving it against an httptest server depended on the transport staying permissive. Move every departure from indigo's defaults into one file pair in pkg/atproto: indigo_prod.go (!testmode) returns indigo's directory and OAuth client unchanged; indigo_local.go (testmode) wraps the directory so a did:web naming an IP, localhost, or a host with a port resolves over plain HTTP, and gives the OAuth client plain HTTP clients. All six identity and OAuth constructor call sites go through NewDirectory and NewOAuthClientApp. The resolver fallbacks, DIDWebToURL, and the hold's scheme-guessing key fetch are gone; the hold resolves the appview key through the directory, preferring #appview, and purges and retries once on a signature failure so a re-keyed appview is not masked by the 24-hour cache. There is no runtime switch for this: a production binary cannot be configured to resolve local DIDs. The runtime test_mode flag still gates the remaining behavioral branches only. Tests, the harness, make dev, Air, Dockerfile.dev, and docker-compose build with the tag; fixtures that need loopback did:web fail fast naming it. Test hold servers now serve a did.json via pkg/testpds so they resolve as real holds under the tag. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UwYzaG3Yy7uA8FbZ5qk3tQ |
||
|
Evan JarrettandClaude Fable 5.1
|
7b67b3b383 |
appview: mark repo cards translate="no"
Repository cards hold user-written handles, names, descriptions and image references. Browser translators (Chrome, Edge, Safari, and Firefox once a translation is accepted) honor the translate attribute on elements below the root, so the card contents stay as written while the rest of the page translates. This does not suppress the browser's translate offer, which is a page-level decision driven by language detection. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011tf8SRvJiWHWkVcfbPUnDC |
||
|
Evan JarrettandClaude Fable 5.1
|
16375ce302 |
appview: name uploads with a UUID, not the nanosecond clock
ProxyBlobStore.Create named every writer fmt.Sprintf("upload-%d",
time.Now().UnixNano()) and used that as its key in globalUploads. The
nanosecond clock is not a unique source: Go's wall clock is coarser than
the spacing between goroutines, so two uploads opened at the same instant
read the same value. On this box, two goroutines released together
collided about 18% of the time (tsc clocksource).
That is reachable on every multi blob push, because Docker and crane POST
the config blob and several layers concurrently. When it happened the
second writer silently replaced the first in the map, both clients' PATCHes
resumed the same writer, and distribution rejected the second with a 416
"upload resumed at wrong offset: N != 0", which the client surfaces as
RANGE_INVALID: invalid content range. It showed up in 3 of 8 benchmark runs
with an instrumented Create: identical IDs, startedAt values 10 to 60ns
apart. The line predates the recent upload path work.
Writer IDs now come from newWriterID(), a package level func var returning
"upload-" + uuid.NewString(). google/uuid is already a direct dependency of
the module, so no new one is added, and the UUID's hex and hyphens keep the
ID URL safe: it travels in the upload URL and inside distribution's _state
token. The "upload-" prefix is kept because the existing ID test asserts it.
Create now also refuses to overwrite an occupied key, under globalUploadsMu,
rather than evicting the sitting writer. With a UUID a hit cannot be chance,
so failing loudly beats stranding a client that is midway through a layer.
The mock hold server's test upload ID moves to a UUID for the same reason.
Tests: TestCreate_ConcurrentIDsAreUnique releases three Creates from a
shared barrier over 300 rounds and asserts every ID is distinct and every
Create lands its own entry in globalUploads. That one does not reliably
fail against the old code, since the mock path desynchronises the
goroutines, so TestCreate_RefusesDuplicateID covers the guard directly by
overriding newWriterID to hand back an ID that is already taken, and
asserts Create errors and leaves the original writer in the map. Confirmed
it fails with the guard removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EvFJr4Dwz8p2NDAeXmgmBt
|
||
|
Evan JarrettandClaude Fable 5.1
|
831b7757bf |
appview: never block a writer on budget for its second buffer
Since |
||
|
Evan JarrettandClaude Fable 5.1
|
56bde61555 |
appview: keep one part upload in flight while the next buffer fills
A large layer went up strictly one step at a time: fill 16MB from the client, stop reading, fetch a part URL and PUT the part to S3, reset, resume reading. While the part was in flight Docker sat on a full TCP window; while the buffer filled S3 sat idle. Wall clock was receive time plus send time. The writer now hands a full buffer to a goroutine that does the hold call and the PUT, and keeps filling a second buffer from the client. When that one fills it waits for the previous part, takes its buffer back, and hands the new one off. At most one part is in flight, so part numbers and ETags stay ordered, and a blob that never fills a buffer never allocates the second one. No network runs under the writer lock on the happy path. Peak memory for a large upload is now two buffers, 32MB. Both are charged to the process budget through the existing accounting, the second as it grows, and the budget floor rises to match so a large upload can never be refused outright. The 512MB default holds sixteen. A failed part records a sticky error, closes the writer, and aborts the multipart from the goroutine that still holds the upload ID; the next Write, hand-off, or Commit reports the cause. Commit verifies the digest first, then waits for the flight, sends the final part, and completes. Cancel waits for the flight, bounded, before aborting so the abort cannot overtake a PUT that has not yet been issued its upload ID. The sweeper refuses to reap a writer with a part in flight, since last activity is only stamped when a part lands. Tests observe the overlap directly: the fake S3 blocks the first PUT and the second buffer's writes are asserted to return before it is released, while the third buffer's writes block. Also covered: the one-part-late error, Commit waiting, Cancel during flight, peak budget, the sweeper, and concurrent Cancel and Write under the race detector. The integration suite passed with a 72MB layer pushed through the pipeline by three clients. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5 |
||
|
Evan JarrettandClaude Fable 5.1
|
8b5f195f09 |
docs: record the remaining performance findings
The September 2026 read of the appview and hold request paths produced more findings than were worth doing immediately. This records what shipped, what remains with problem, location, fix and expected impact, one non-performance bug found along the way, and the production size distribution that informed the 16MB direct-PUT threshold. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5 |
||
|
Evan JarrettandClaude Fable 5.1
|
7045e84c00 |
appview: read the sailor profile from the local users row, not the PDS, per request
Hold discovery in the registry middleware called getRecord on the repository owner's PDS for every request under /v2/: every HEAD, POST, PATCH, PUT and GET. A 10-layer push was 40 or more PDS round trips, and it was the last per-request network call on the push path that had nothing to do with moving bytes. Only two profile fields are used there: the default hold and the auto-remove-untagged flag. The users row already caches the default hold, written by the Jetstream processor on every profile event and prefilled by the backfill, and the auth gate already reads it from there. This makes the row a faithful copy of what the registry needs and switches the middleware to it. The auto-remove flag gets a nullable users column. NULL means the value has never been learned; the processor writes 0 or 1 on every profile event and never NULL. On a request whose row is missing or still NULL, the middleware does one live fetch, uses it, and writes both fields back, including a 0 for a user with no profile at all, so the fallback runs at most once per user. A failed fetch writes nothing and uses the appview default for that request, so a network error is never cached. That single mechanism covers the minutes after a deploy while the startup backfill fills the column, a brand-new user, and a user the backfill has not reached. The processor also stops returning early on an empty default hold, which left a user who removed their custom hold pushing to it forever. Empty is now written through and means the appview default, matching what the auth gate already reads. Tests count PDS requests with a test server: a populated row makes none, a NULL row makes exactly one and then none, a missing profile is cached as known, and a failed fetch degrades without writing. The migration was applied to a fresh database and to one built from the previous schema. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5 |
||
|
Evan JarrettandClaude Fable 5.1
|
47a107058d |
appview: bound upload buffer memory, reap abandoned uploads, and pin the flush boundary
Each in-flight blob upload buffers up to 16MB, Docker pushes five layers at once per client, and nothing bounded the total. Writers also lived in the package-level map forever: a client that died mid-push left its writer, its buffer, and any hold-side S3 multipart session behind with no expiry. A process-wide budget (golang.org/x/sync semaphore, default 512MB, server.upload_buffer_budget_mb) now caps memory held in upload buffers. A writer charges its buffer's projected backing capacity before growing, so a config blob costs kilobytes and a full writer costs exactly one buffer, and releases once, on Commit, Cancel, or reap. A write that needs budget waits on the request's context with a five minute cap, outside the writer's lock so Cancel and the sweeper cannot queue behind it; that wait is backpressure on the client. The budget is clamped to at least one buffer so a single upload can never deadlock. A sweeper started with the other appview workers reaps writers idle past server.upload_idle_timeout (default 1h), aborting the hold-side multipart on a detached context and releasing the budget. It measures inactivity, not age, so a slow push is never reaped, and it skips a writer whose lock is held so it cannot race a live part upload. Write also gains a fix the budget made visible. It appended a whole chunk and checked afterwards, so the last chunk before a flush could land a few bytes past 16MB, which did not fit the backing array; bytes.Buffer doubled it to 32MB and Reset kept that for the rest of the upload. Only chunk sizes that tile 16MB exactly avoided it, and the network read loop promises no such thing. Every large layer could hold 32MB while the budget charged 16. Write now fills to exactly the threshold, flushes, and continues with the remainder, so capacity is pinned at 16MB for any chunk size, every part is exactly one buffer, and a single oversized Write streams through as parts instead of buffering whole. The test streams 24KB chunks across the boundary and fails against the old code with cap 33554432. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5 |
||
|
Evan JarrettandClaude Fable 5.1
|
f4343d7956 |
appview: upload small blobs with one presigned PUT, and verify every digest
Every blob went through the multipart machinery: an S3 multipart started on Docker's initial POST, a hold round trip per part, and a complete on the hold that finished the multipart, HEADed the temp object, copied it to its final key, and deleted the temp. For a 2KB config blob that was three hold calls and six S3 operations. On production data 86% of distinct layers and every config blob fit in a 16MB buffer, and 49% of image manifests have no layer larger than that. The writer now buffers up to 16MB (also the multipart part size) and makes no hold call until it has to. A blob that never overflows the buffer is written at Commit with a single presigned PUT to its final key, via the hold's existing method=PUT presign; the multipart only starts on the first flush. The hold's completeUpload does nothing the direct path skips: quota, layer records, stats and scan dispatch all hang off notifyManifest, which is unchanged. The buffer starts empty and grows on demand, with the doubling capped so capacity never overshoots 16MB: a config blob costs kilobytes, and only layers that approach the threshold fill it. Bytes are hashed as they arrive. Commit compares the computed sha256 to the digest the client claimed before any network call, and returns DIGEST_INVALID on mismatch, aborting a multipart if one was started. Previously nothing verified the content, so a pusher could store wrong bytes under a digest in the shared content-addressed space. Tests observe request counts on a fake hold and fake S3 rather than return values. The growth test streams in 24KB chunks because power-of-two chunks land on 16MB by luck and hid an earlier weaker guard. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5 |
||
|
Evan JarrettandClaude Fable 5.1
|
034ea5988b |
hold: report blob size on read presigns so the appview can skip its S3 HEAD
distribution calls Stat before every blob GET and HEAD. The appview's Stat asked the hold for a presigned HEAD URL and then HEADed S3 with it purely to read Content-Length for the descriptor: two round trips to learn one number. The hold's getBlob response for OCI digests on GET and HEAD now carries "size". It comes from the records index when a layer record exists (a SQLite lookup on a new digest index, no network) and from a HeadObject otherwise, which is where config blobs land. If storage says the object does not exist the hold answers 404 instead of signing a URL that can only fail. The PUT and ATProto CID paths are untouched. The appview builds the descriptor from the reported size and makes no S3 request. When the field is absent it HEADs the presigned URL as before, so a new appview works against a hold that has not been upgraded, and an old appview ignores the extra field. A hold 404 maps to ErrBlobUnknown. Tests prove the index answered by leaving the mock bucket empty and counting zero HeadObject calls, prove the fallback with exactly one, and count requests reaching the fake S3 origin on the appview side rather than trusting the returned size. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5 |
||
|
Evan JarrettandClaude Fable 5.1
|
61a934debb |
hold: look crew members up by rkey instead of walking the collection
ValidateBlobWriteAccess, ValidateBlobReadAccess, ValidateOwnerOrCrewAdmin and getCrewTier each listed every crew record to find one member: open a carstore session, walk the MST, CBOR-decode each record, compare DIDs. That ran on every multipart call from the appview, including the part URL request for every 10MB, and on every getBlob presign, so on a hold with hundreds of crew each part cost hundreds of decodes. lookupCrewMember tries the deterministic rkey first (one record read) and only falls back to the walk on a not-found miss. The fallback is required: records created before the hash-rkey scheme sit at a TID rkey, and the boot-time migration that rekeyed them only existed between |
||
|
Evan JarrettandClaude Fable 5.1
|
9228579b07 |
appview: share one HTTP transport across blob proxy requests
NewProxyBlobStore built a fresh http.Client and http.Transport on every
call, and it is called once per registry request because the routing
repository is created per request. So no connection to the hold or to a
presigned S3 URL was ever reused: every XRPC call and every part upload
paid a TCP and TLS handshake, HTTP/2 was never negotiated, the idle pool
settings were dead config, and each discarded transport kept its idle
sockets open for the full 90s timeout.
One package-level transport and client now back every store. Settings
are unchanged; ForceAttemptHTTP2 is explicit to document intent. The
load balancer negotiates h2 on the frontend (
|
||
|
Evan JarrettandClaude Opus 5
|
788b9e1053 |
hold/admin: bound the crew tab's identity lookups, and drop backend HTTP/2
Two corrections to the HTTP/2 change, both found by running it. UpCloud refuses http2_enabled on a backend that is not TLS: invalid_params_properties.http2_enabled='Tls must be enabled to enable HTTP2.' This load balancer reaches both origins over the private network in cleartext, so the option is not available here at all, and asking for it failed the whole provision run with a 400 after the frontend had already been changed. Backend HTTP/2 is removed rather than made conditional: it would need TLS terminated at the origins, which is a much larger change than it earns. The frontend is the leg that mattered, and it is enabled and confirmed live — all three domains now negotiate h2 by ALPN. The second correction is more interesting: enabling HTTP/2 did not fix the symptom it was aimed at, and briefly made it look worse. With the browser no longer rationing itself to ~6 connections, every row of the crew tab now reaches the server at once, and the 504s went from a flat 10s to a climbing ladder: [HTTP/2 504 10026ms] ... [HTTP/2 504 15230ms] ... [HTTP/2 504 25976ms] HTTP/1.1 had been hiding a server-side limit by throttling the client. The queue moved; it did not disappear. What is actually slow is resolveHandle, which calls live identity resolution per row with no bound of its own, so a DID whose resolution hangs — a did:web on a host that stopped answering, a PDS that accepts and then stalls — holds its request open until the proxy gives up. The crew tab issues one request per member, so a hold with hundreds of crew gets hundreds of chances to hit one. resolveHandle now bounds each lookup at 2s. A handle is decoration on a row whose DID is already rendered beside it, so waiting seconds for one and failing the row when it does not arrive trades something load-bearing for something cosmetic. Two seconds is far above a warm cached lookup and far below the proxy's 10s cut, so a slow DID costs its own row a handle and costs the page nothing. The quota lookup on the same handler was the other suspect and was measured out: records(collection, did) is indexed and the aggregation returns immediately on the production database. This does not remove the N+1 — the tab still issues one request per member, and a batch endpoint is the real fix. It removes the failure. Verified: the guard is covered by a test using a directory that hangs until its context is cancelled, asserting both that the call returns near the bound and that it did not return suspiciously early. make lint 0 issues, make test green across 44 packages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AV6Mk2AgghFsNo4HWQEaBV |
||
|
Evan JarrettandClaude Opus 5
|
b886a75532 |
deploy: make appview billing opt-in, and verify what was built
build-trixie hardcoded `-tags billing`, so the only way to build a non-billing
appview was to bypass the Makefile and drive the trixie container by hand. Both
production deploys of this deployment did exactly that, and the workspace notes
record it as "the deploy tool cannot reproduce this build" — a tool that cannot
produce the artifact you actually ship is a tool nobody can safely use.
Worse, the failure is silent and one-directional. Nothing about a billing binary
looks different: same name, same version stamp, same vcs.revision. A deploy that
forgot the manual path would quietly switch billing on for a deployment whose
operator had chosen to run without it, exposing a live Stripe webhook endpoint
and a paid tier ladder on a service configured for neither. Production has been
running a non-billing appview since May precisely because someone did not use
`make build-trixie`.
Billing is now opt-in and off by default:
make build-trixie # no billing
make build-trixie BILLING=1 # billing
deploy/upcloud update appview # no billing
deploy/upcloud update appview --with-billing
Only the appview is affected; hold, scanner, labeler and the credential helper
never reference pkg/billing.
The flag alone is not enough, so verifyAppviewBilling reads the built binary
before anything is uploaded and refuses to ship a mismatch. It observes rather
than trusts, because the flag and the artifact can disagree for reasons the flag
cannot see: a stale bin/atcr-appview from an earlier build, a Makefile that
hardcodes the tag, a builder image that ignored it. The Stripe SDK links only
under the tag, so its symbols are a direct measurement — verified as a
discriminator here: 2338 stripe-go strings with the tag, 0 without.
A missing binary is an error rather than an absence of symbols, so a deploy
cannot proceed on a file that was never built by reading it as "no billing".
Verified: the default build-trixie output has 0 stripe-go symbols and does carry
the "Billing is not enabled on this deployment" stub. Guard covered both
directions by test, plus the missing-binary case. make lint 0 issues across
root, deploy and credential-helper; make test green across 44 packages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AV6Mk2AgghFsNo4HWQEaBV
|
||
|
Evan JarrettandClaude Opus 5
|
8d7ccd7cb7 |
apply go fix modernizations across the workspace
`go fix` carries the modernize analyzers now, and the tree had drifted behind
them. This is the mechanical result, reviewed rather than trusted: the tool is
capable of rewriting code into something that no longer tests or does what it
did, so every non-test change was read individually and the concurrency-bearing
packages were re-run under -race.
Production code, four changes, all semantics-preserving:
- leases/manager.go: wg.Add(1) + go + defer wg.Done() becomes wg.Go. The
comment above that function turns on Add happening before the goroutine
starts, so that a Wait cannot return before the worker has run. wg.Go does
the Add synchronously on the calling goroutine, so the invariant it
describes still holds.
- auth/token/handler.go: strings.Fields -> strings.FieldsSeq, same splitting,
iterated rather than allocated.
- hold/gc/gc.go: a hand-written map copy -> maps.Copy.
- hold/pds/scan_broadcaster.go: three-clause loop -> range over int.
The rest are tests. The one worth naming is carstore_contention_test.go, where a
careless rewrite could have quietly stopped exercising contention: go fix
converted the reader and side-table goroutines to loopWG.Go but correctly
declined to touch the writer loop, which passes its index as a parameter. The
writer/reader/side-table shape and the stop channel are unchanged, so the test
still contends over the same carstore transactions.
Verified: go build for hold and appview, `make lint` 0 issues, the deploy and
credential-helper modules 0 issues, `make test` green across all 43 packages,
and -race green on leases, hold/pds, hold/gc and auth/token. The scanner module's
two lint findings are unchanged from HEAD and are in files go fix never touched.
Kept separate from the HTTP/2 commit so that one stays readable, and so this can
be reverted on its own if a modernization turns out to matter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TA9D4DjaLZTvzQ7dJbu4eg
|
||
|
Evan JarrettandClaude Opus 5
|
416ba4a2eb |
serve HTTP/2, and reach it through the load balancer
The admin crew tab renders one row per crew member and gives each row its own hx-get, so opening it on a hold with 551 crew issues 551 requests. Over HTTP/1.1 a browser runs at most ~6 per origin, so they queue six at a time and every other request to the same host queues behind them — which is why loading the relay page stalls while the crew rows are still resolving, and why the rows that lose the race come back as "Server error" toasts. The 504 behind that toast is the load balancer's, not the hold's: the hold logs those requests as 200. HTTP/2 multiplexes them over one connection and the queue disappears. It does not make the slow rows fast — that is a separate fix to the per-row identity lookup — but it stops one slow surface from blocking the rest of the panel. Two halves, because neither works alone. The load balancer terminates TLS and speaks cleartext to the origin, so ALPN never runs on the backend leg and net/http can only answer HTTP/1.1 there. Both servers now wrap their handler in h2c. The wrapper is opt-in per connection: it upgrades only for a client sending the h2c preface or "Upgrade: h2c", and passes everything else through untouched, so an HTTP/1.1 WebSocket upgrade is unaffected. Verified both directions against this wiring — HTTP/1.1 for a plain client, HTTP/2.0 with --http2-prior-knowledge. The frontend's http2_enabled was never set, so it sat at the UpCloud default of off. That is the half the browser actually sees. timeout_client is now stated explicitly at its current 10s rather than left implicit: it is the boundary that produces the 504s above, so it belongs somewhere visible. It is deliberately unchanged — raising it without fixing the slow lookup would only make a stalled row stall longer. The hold's *backend* stays on HTTP/1.1. It serves subscribeRepos over WebSocket to external relays and to the scanner, and WebSocket over HTTP/2 needs the RFC 8441 Extended CONNECT that Go's http2 server does not implement for Upgrade:. Routing that backend over h2 would break the firehose. The appview accepts no inbound WebSocket and has no such constraint. Both origins carry h2c regardless, so enabling it for the hold later is a config change, not a code change. createLoadBalancer only runs when there is no LB yet, so properties set there would reach a new deployment and never an existing one. ensureLBHTTP2 reconciles them onto an LB that already exists, following ensureLBForwardedHeaders: read what is there, change only what differs, report what it did, and no-op on a second run. Backend modifies carry the existing health check back, since Properties replaces the object wholesale. Also gofmt: provision.go was not gofmt-clean at HEAD, unrelated to this change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TA9D4DjaLZTvzQ7dJbu4eg |
||
|
Evan JarrettandClaude Opus 5
|
265e533ad3 |
scanner: reclaim scan directories a killed process left behind
buildOCILayout already removes its scan dir on every error path, and syft.go
defers the stereoscope generator's Cleanup. What neither can do is clean up
after a process that dies mid-scan: the deferred call never runs, and nothing
afterwards ever looks at what was left. Every restart therefore leaks the
in-flight layout and extraction permanently, and a restart is routine — a
deploy is one.
On seamark-hold that reached 8.8 GB of orphaned scan-*, syft-scan-* and
syft-cataloger-* directories under a 20 GB disk, at which point the disk was
97% full and scans began failing on it:
failed to load OCI image: unable to populate layer cache
dir="/var/lib/seamark/scanner/tmp/syft-scan-1546187834/..."
: no space left on device
failed to download layer 5: failed to write blob:
write /var/lib/seamark/scanner/tmp/scan-4160414849/blobs/sha256/...
: no space left on device
The leaked directories cluster at the scanner's restart timestamps, which is
what identifies the killed process rather than the error paths as the source.
Manual removal reclaimed 8.8 GB and took the disk from 97% to 50%.
Startup is where this belongs: it is the one moment the previous process is
known to be gone, and it is immediately after the event that caused the leak.
The sweep runs in WorkerPool.Start after TMPDIR is set and before any worker
can dequeue, so nothing it removes can be work in progress here.
Three constraints shape what it will touch:
- Only the three per-job prefixes, only as direct children, only
directories. The Grype database lives beside the tmp dir at
<parent>/vulndb and go-getter unpacks into grype-dl underneath it; both
are state the scanner needs and neither matches a prefix. The prefixes now
have one definition each, used by both the creator and the sweeper, so
renaming a directory cannot silently take it out of the sweep's scope.
- An age threshold, vuln.sweep_max_age, default 1h. A second scanner sharing
the directory has an in-flight scan-* dir that is minutes old, and
scanner.job_timeout is 8m, so an hour clears both with room to spare. 0
disables the sweep rather than removing a peer's live work.
- Nothing is fatal. A stat or removal failure is a WARN and the sweep moves
on, so a permission problem in the tmp dir cannot keep the scanner from
starting.
The sweep only runs at startup, so a scanner that is killed twice between
deploys carries the first leak until its next restart. That is the tradeoff
for never racing a live peer; a periodic sweep would be the follow-up if
processes ever live long enough for it to matter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TA9D4DjaLZTvzQ7dJbu4eg
|
||
|
Evan JarrettandClaude Opus 5
|
bd208661c6 |
build: run every workspace module's tests, not just the root's
go test ./... only ever covers the module it runs from, so the scanner's 17 test files were never reached by make test. Everything written for it this week was dormant: the mock hold, the manifest corpus, the blob integrity suite, the concurrency scenarios. The same shape as the billing tests a few commits back, which is why the module list is derived from go.work rather than written out, so adding a module to the workspace is enough to get it tested. Wiring it up immediately failed, which is the argument for having done it. TestSBOMIsStableAcrossScanDirectories asks the two SPDX keys that still vary between encodings to keep varying, so that whoever makes the digest stable is told to tighten the test rather than discovering the slack later. But creationInfo.created has one-second granularity, so two scans inside the same second produce an identical value and the assertion becomes a coin flip on how fast the machine is. It passed under -race, which is slow enough to straddle a second, and failed under -cover, which is not. Only documentNamespace can carry that signal, since its UUID is redrawn on every encode. creationInfo stays in the residue set, because the digest comparison must still exclude it; it is simply no longer asked to prove anything. Stable across repeated runs under -cover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
29fe24aa41 |
scanner: report the Grype version that is actually linked in
The vulnerability report's descriptor hardcoded "v0.107.1" while go.mod has required v0.118.0 for some time, so every report ever stored misreported which tool produced its findings. That field exists to answer the first question anyone asks when two scans of one image disagree: did the scanner change, or did the feed. A constant cannot answer it, and had been quietly wrong across eleven minor versions. It now reads the module version from debug.ReadBuildInfo, so a dependency bump carries into the reports on its own. A build-time ldflags stamp was the alternative and has the same failure mode as the constant, just moved to a step someone has to remember. Where the build genuinely cannot say, the descriptor carries "unknown" rather than a plausible-looking lie. The lookup is split into a pure function because a test binary's build info carries no dependency list, so the real call returns "unknown" under `go test` no matter what the code does; testing through it would have proved nothing. Verified against a real build instead: `go version -m bin/atcr-scanner` lists dep github.com/anchore/grype v0.118.0, which is what the split-out lookup returns for that input. A second test fails if the literal ever reappears in the source, since that is the regression rather than the lookup being wrong. Reports already stored keep the old string; there is no migration and nothing reads the field today, the appview decoders declaring only matches and dropping the rest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
96b48f4b3d |
hold: gate presigned PUT behind write access, and validate blob identifiers
com.atproto.sync.getBlob took its S3 operation straight from ?method= with no allowlist and passed it to GetPresignedURL, whose switch mints a presigned PUT. The ATProto sub-handler is public per the ATProto spec, and the OCI one gates on ValidateBlobReadAccess, which passes anonymous callers on a public hold by design. So an anonymous caller could obtain a fifteen-minute presigned PUT into the hold's own bucket, over the avatar, the OG-card thumbnails, stored SBOMs and vulnerability reports, or the OCI layer space. Holds sharing a bucket widened it. The defect is that a read check was gating a write. PUT now requires captain or crew with blob:write via the existing ValidateBlobWriteAccess, rather than a new parallel gate. Reads are untouched: ATProto GET and HEAD stay public, OCI GET and HEAD keep the read check. An unknown method is a 400 rather than falling through to a 500 from the presign switch. The identifiers reached object storage keys unvalidated, which the tests showed was not theoretical: cid=../../../../etc/passwd presigned a key containing that path, and an OCI digest of sha256:aa/bb presigned across a shard boundary. A digest is now sha256 plus exactly 64 lowercase hex, shaped after the scanner's ParseDigest, and a CID must decode and round-trip to its canonical encoding. Validation runs before the auth gate, so a malformed identifier is a 400 even for an authorised caller. Verified before changing anything that no legitimate caller reaches PUT presigning. Pushes upload through the multipart endpoints, which live inside requireBlobWriteAccess and use PresignUploadPart; every getBlob caller in the appview, the scanner and the handlers sends GET, HEAD or nothing, and the distribution route can only produce GET or HEAD. One existing test asserted that an unauthenticated PUT returns a presigned URL. It was pinning the vulnerability as intended behaviour and is replaced by coverage of both the refused and the authorised cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
22058cc5f4 |
scanner: bound a scan job, and lose the race to the hold on purpose
Nothing limited how long one job could take. The worker's context was the process's, buildOCILayout took none, and blob downloads used a package-level client whose five-minute timeout is per request with no context, so a 19-layer image had a hundred-minute worst case on downloads alone and cancellation could not touch it. At the default single worker, one wedged job stopped that scanner entirely. scanner.job_timeout, default 8m, against the hold's 10m scanning timeout. Both clocks start at the same instant: the worker sends "started" on dequeue and derives the job context on the next line, so the scanner loses by two minutes, which is enough for its terminal message to cross the socket and be recorded. If the hold wins instead it re-dispatches while this scanner is still working, which is duplicate work recorded under a generic reason. A scanner cannot read the hold's config, so the relation is a mirrored constant used only for a boot-time warning, and the same warning fires if the deadline is disabled. What is actually bounded, since a deadline the code cannot honour is worse than none: presign, download, stereoscope's Provide, Syft's CreateSBOM, and Grype, which does have FindMatchesContext even though FindMatches does not. stereoscope's img.Read takes no context and is 81% of a scan, so it is checked either side rather than interrupted. Abandoning it on a goroutine would trade a bounded overrun for one writing gigabytes into a directory the caller has already deleted. max_image_size remains the real bound on that stage. A timeout reports error, not skipped. It describes this host at this moment, a contended CPU or a slow bucket, not the image, and skips are never retried, so one bad afternoon would retire an image permanently with nothing in the record to say why. Retry cost is bounded on the other side by max_image_size and by the stale-scan schedule. The classification asks the job context rather than the error, because several stages replace the cause and the uninterruptible one knows nothing about the deadline, and a job that finishes after an overrun still reports its real result rather than throwing away completed work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
c44a874090 |
hold: set busy_timeout on every pooled connection, not just the first
A PRAGMA is per-connection, and database/sql opens one connection per concurrent caller. hold_db.go and scan_broadcaster.go each set busy_timeout = 5000 with a one-shot query against the pool, which configures whichever connection happened to serve it and leaves every other writer at zero. Probing the live pools showed exactly that: conn 0 at 5000, conns 1 and 2 at 0. That is the database is locked on COMMIT, and the carstore had no busy_timeout and no WAL on any connection at all. Opening a database now goes through a connector that runs the PRAGMA on every connection the pool creates, ported from the appview, which had already solved this. Applied to the carstore, the shared hold DB, the records index, the event broadcaster and the scan broadcaster. In-memory databases are left unwrapped, since libsql gives each connection its own, and the embedded replica path still skips PRAGMAs because a remote rejects them. WAL is not a new risk: production was already WAL, because journal mode is a persistent property of the file and OpenHoldDB has been setting it. This makes the carstore's own opener agree rather than leaving standalone and test databases in rollback-journal mode. A database that refuses WAL logs a warning instead of failing boot, so a network mount cannot stop the hold starting. NewHoldPDS's file mode opened the same file a second time for the records index; it now shares the carstore's pool. Two pools on one file buy nothing, since SQLite's write lock is per database, so the second only adds contenders. Production already shared a pool and was unaffected by that half. Note for anyone reading OpenHoldDB: its foreign_keys pragma has the same one-connection scope, but libsql reports foreign_keys on for every fresh connection anyway, so it is decorative rather than broken. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
3ceedc8bb5 |
hold/gc: time an orphaned record from when it was seen, not when it was written
The aux-record sweep decided whether a record was old enough to collect from a timestamp inside the record body, which for scan records is scannedAt. Every rescan rewrites that, so the grace clock reset continuously and a hold with a rescan_interval shorter than the grace period could never collect an orphaned scan record at all. Grace now measures how long a slot has been continuously observed orphaned across analysis passes. The key deliberately excludes the CID so a rescan rewriting the record in place does not restart the clock, and the check runs last, after the manifest, reachability and co-ownership tests, so a record judged live never accrues orphan age. A pass that errors partway commits nothing rather than resetting the clock on slots it never reached. The clock is in memory, so a restart forgets every observation and each surviving orphan starts its grace again. That delays collection and can never advance it, which is the safe direction, but it does mean a hold restarting more often than the grace period will not collect aux orphans. Persisting it wants a table in pkg/hold/db and is left for later. Applied to both aux collections rather than only to scan records. For image configs the new rule is strictly more conservative, since a record cannot be observed before it is written, and a per-collection table of which timestamps are safe to trust is a thing to maintain and to get quietly wrong. One widening beyond the reported bug, called out rather than left to be found: a record whose body timestamp is unparseable used to be kept forever, because the zero time read as in-grace. It is now collected on the normal schedule, having cleared every other guard plus a full grace period. Also corrects the comment claiming these records hold no blob references. Scan records carry sbomBlob and vulnReportBlob; they live under a prefix the blob sweep never walks, so deleting the record frees nothing, and a future sweep for that space must read those fields before the record goes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
61deadc124 |
hold: reuse stored scan blobs when a rescan finds nothing new
The stale-scan loop rescans an image on a schedule and handleResult uploaded both artifacts every time, so each pass wrote a fresh SBOM blob and orphaned the one before it. Nothing reclaims those, because GC walks only the OCI blob prefix and scan artifacts live under /repos. Keep the stored blobs when the record already on file was written by this same scanner version and carries a vulnerability-report reference equal to the one this report would be stored under, then keep each blob individually and only if it is still in S3. The report reference alone is not quite enough evidence on its own: the report lists matched packages, not every package, so an SBOM could gain a package with no known CVEs and leave the report byte-identical. Content is pinned by the manifest digest, so that can only happen across a scanner or Syft change, which is what the version clause closes. It is inert until that constant moves, and one comparison is cheaper than remembering to add it at the moment it first matters. Every failure path declines to reuse, which costs an upload and never costs correctness, including the reuse check's own timeout, which is carved out of the upload budget rather than given its own so a slow read cannot eat the deadline it stands in for. A record whose earlier upload failed heals on the next rescan, keeping the report and rewriting only the missing SBOM. scannedAt still advances. runStalePass selects on it, so freezing it would make every deduplicated record permanently stale and rescanned forever, which costs far more than the bytes saved. A scanner running with vulnerability scanning off sends no report, so there is no stable digest to compare and its SBOM still churns. Closing that means either parsing SPDX in the hold to normalise a timestamp and a UUID, or a lexicon change; a test pins the gap rather than leaving it to be rediscovered. Note for anyone extending GC to /repos later: a live SBOM object used to be rewritten on every rescan and so was always young. It now keeps its original mtime for the life of the content, so an age-based rule there would delete referenced blobs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
1853c0c1d3 |
hold/scanner: detect dead connections and get storage off the reader
Three defects that all turn on whether the hold knows what a scanner is doing. Neither end had a keepalive, a read deadline, or a read limit, so a half-open connection was invisible until some other timeout fired. That got worse with capacity-aware dispatch: a dead-but-connected scanner holds its advertised worker count out of the budget and keeps winning jobs. Both ends now ping every 30s against a 90s read deadline, so three unanswered pings condemn a connection. Detection takes about 90 seconds, after which the existing reconnect grace reclaims the rows, against the 60-minute queueing timeout that was previously the only escape. Liveness decides when a scanner is gone; the grace window decides when its work is reassignable. Read limits are asymmetric and deliberately generous, because exceeding one closes the connection rather than truncating, which would turn a large but legitimate result into a permanent retry loop. Write deadlines were absent everywhere; the scanner in particular held a mutex across an unbounded write, so a wedged write silenced it without disconnecting it. handleResult did two S3 uploads and a CAR commit inline on the reader goroutine, so a slow S3 looked exactly like a dead scanner. Terminal messages now go to a per-subscriber storage goroutine while acks and starts stay on the reader. One goroutine, not a pool: every path ends in CreateScanRecord, which serialises on the repo lock anyway, and ordering is worth more than parallelism that cannot be used. Uploads and the record write get separate budgets, so a stalled upload cannot spend the time the record needs and the record write stays unconditional. checkPredecessor cached an inconclusive answer as a definitive negative in a map that is never reset, so one unreachable hold meant its manifests were never scanned again for the life of the process. gc.go already carried the corrected logic for the same problem; this follows it rather than inventing a second approach, and also stops treating an unparseable captain record as definitive. The test PDS had to move off ":memory:", which go-libsql scopes per connection: writing a scan record from any goroutine but the caller's got a connection with no tables. That was invisible while every record write happened inline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
e57e31cefe |
scanner: name a scan by its manifest digest, not its temp directory
The Syft source reference was the per-job os.MkdirTemp path, which is different on every scan and reaches five places in the encoded SPDX document as well as the source object embedded in the vulnerability report. Rescanning unchanged content therefore produced byte-different artifacts under fresh digests, so the hold uploaded new blobs and orphaned the old ones on every pass of its stale-scan loop. The manifest digest is the identity the hold already uses: the scan record's rkey is derived from it, one record per digest whatever repository the job arrived under. "repository@digest" reads better but the same digest is dispatched under whichever user's manifest discovery walked first, so it would have reintroduced the same drift more slowly. An explicit source.Alias pins the name, because Describe() otherwise parses a bare sha256:<hex> as a repository called "sha256". Nothing reads the source field today; the only user-visible change is that a downloaded report names the image instead of leaking a temp path. This makes vulnDigest stable. sbomDigest still moves, and cannot be fixed here: Syft stamps creationInfo.created from time.Now() and appends a random UUID to documentNamespace, neither configurable through the encoder. Stamping a constant timestamp would be a falsehood in a security artifact and deriving the namespace UUID means mutating a package global, which races across workers. The test asserts equality outside those two keys and fails loudly if the residue ever stops moving, so the next person is told to tighten it rather than discovering it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
fcde9c879a |
docs: record why an all-negligible image is not rendered as clean
The previous wording called the 0 0 0 0 strip a visible consequence worth closing and named a route to close it, which invites exactly the change it should prevent. Clean is not the same as nothing worth reporting. A clean image has no findings; an image whose findings are all Negligible or Unknown has findings, none of which rise to a flagged severity, and the vulnerabilities tab lists what the strip does not. Rendering it as clean would assert something untrue. The headline exceeding the four counts is the same distinction: the total is how many were found, the boxes are how many merit action. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
7cf58de78e |
docs: record which scanner findings were reviewed and accepted
Two of the open findings were reviewed and are working as intended, which is worth writing down: an unrecorded decision reads identically to an oversight six months later, and both of these look like bugs from the code alone. The severity strip shows four buckets and does not account for Grype's Negligible and Unknown. That is deliberate. Two more colours would clutter the strip and those severities rarely merit attention, so the four boxes summarise what matters rather than partitioning the total, and the detail table still lists everything. The two places the gap stays visible are recorded with the cheaper route, should either ever be worth closing. GOMEMLIMIT not scaling with scanner.workers is a safeguard for a memory-constrained host, not a tuning parameter, and the intended direction is to remove it once scanners have their own nodes rather than to grow it. The separate open question about the cgroup ceiling is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
850c416dc8 |
docs: record the scanner audit, its fixes and what is still open
Section 1 is the status: what landed and what did not. Sections 2 to 6 are the consolidated analysis. Sections 7 onward are the original agent reports, kept unedited as dated evidence of the code as audited, which means they describe behaviour several of the fixes have since changed; section 1 is the authority and the renamed tests are listed in section 6. Findings are marked CONFIRMED where someone reproduced them and SUSPECTED where they were reasoned from source, because a reading pass and a failing test are not the same kind of claim and the difference should survive into whoever reads this next. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
0ea0e5494c |
deploy: bound the scanner's memory and yield the box to the hold
The scanner shares a 1 GiB host with the hold, and nothing stopped it taking the whole box down with it. max_image_size was absent from the template, so production ran the 2 GiB default. Peak RSS tracks image size and a 389 MiB compressed image already reaches 561 MiB, so 512 MiB is the ceiling that fits. Sizing matters in this order: max_image_size rejects before a byte is downloaded, while MemoryMax kills mid-scan, and the hold then returns the row to pending and the restarted scanner is handed the same job again, which is a permanent crash loop on one image. If the cgroup cap is ever seen firing, lower max_image_size rather than raising it. GOMEMLIMIT is soft: the runtime collects harder to respect it and never fails an allocation to honour it. With no cgroup cap the kernel picks its own OOM victim, and the other large process on that host is the hold, so the scanner's overshoot could kill the service it reports to. CPUWeight and IOWeight, because no unit here set any scheduling directive: a scan finishing a minute later costs nothing, a hold that cannot answer a pull costs a user. Weights apply only under contention, so the scanner still uses the whole box when nothing else wants it. workers drops to 1. Proactive dispatch was depth-one hold-wide until this release so the second worker never received anything, and now that it can, two concurrent scans of a node:22-class image need 687 MiB against a host this size. Raise it with MemoryMax and the host together. Note the cgroup ceiling still wants rechecking: it was sized from measurements taken with Grype disabled, and the vulnerability database's resident pages count against MemoryMax even though GOMEMLIMIT cannot see them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
a63f668de0 |
scanner: fix five crash and halt classes found by a pipeline audit
An audit of the scan pipeline and the hold side of scanning found several ways scanning stops without saying so. Each fix here was written test-first: a test expressing the wanted behaviour, confirmed failing for the right reason, then the change. A summary-less result crash-looped both processes. worker.go dereferenced result.Summary unconditionally, but processJob only sets it when Grype runs, and SendResult puts the nil on the wire before the scanner dies on it, so handleResult's unguarded log killed the hold too. A nil Summary now means "not scanned for vulnerabilities", deliberately distinct from "scanned, found zero" — inventing a zeroed summary would report every image as clean when Grype never ran. The hold writes a record rather than orphaning the uploaded SBOM, and the appview renders an "SBOM only" state instead of a green Clean badge. The Grype database could wedge with no way back short of a restart. All three throttles in loadVulnDatabase were guarded by vulnDB != nil, so a scanner holding no provider retried a full download on every scan under the exclusive lock. Two earlier attempts at this bug each added one more condition to the same chain; this replaces the chain with a single decision function over a state snapshot, consulted by both call sites so they cannot disagree. That disagreement was itself a bug: the 50-scan reload had never once executed. Two independent halts. An unparseable frame was dropped in silence, stranding a row that held the hold's only dispatch slot forever; it is now answered "skipped" on first delivery. The 10-minute sweep leaked the in-flight digest and wrote no record, permanently retiring one image per timeout. A digest went unvalidated into filepath.Join and os.Create, so a layer digest of sha256:../../../x wrote outside the scan directory, and nothing verified that downloaded bytes hashed to the digest naming them. Digests come from records in a user's own PDS. Both are fixed together: verification is what makes an escaping write self-defeating. Concurrency did not work on either axis. The proactive capacity gate was depth-one hold-wide, so neither extra workers nor extra scanner processes received work. Depth is now the sum of the worker counts scanners advertise on connect, the gate is scoped to proactive work, and dispatch prefers the least-loaded scanner. Disconnects no longer hand a running scan to someone else: a scanner keeps a stable per-process identity and reclaims its own rows within a grace window, while a process that truly restarted returns with a new identity and has its work reclaimed, which is correct because the restart did lose it. The hold's scanning deadline measured queueing rather than scanning, because the scanner acks on receipt and handleAck never refreshed assigned_at. A new "started" message, sent by the worker that dequeues the job, separates the two budgets. An older scanner never sends it and falls under the queueing budget, which is more forgiving than the deadline it gets today. Adds an in-process mock hold and an e2e harness that runs the real client, queue and worker pool, seeded with 84 real manifest records fetched from a live PDS. Real image layouts and the Grype database are fetched by scripts and gitignored; suites needing them skip cleanly, so the default run stays offline and fast. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1Km3N3uUmeGaj7VbaM8PF |
||
|
Evan JarrettandClaude Opus 5
|
f16a8eaa82 |
appview: warn about a deprecated chart on the tab people actually land on
|
||
|
Evan JarrettandClaude Opus 5
|
e542dd12f6 |
appview: keep the port in the registry host printed to users
On the dev stack (registry_domains: ["127.0.0.1:5000"]) the /auth/token failure guidance said `docker login 127.0.0.1`, dropping the port docker needs. The message exists to tell a stuck developer what to run, so a command that cannot work is the whole defect. Production is unaffected, both registry domains being port-free. The stripping itself is deliberate and stays: NormalizeService removes the port so JWT audiences match the port-stripped routing host. Removing it would desynchronize the service key from the routing key. The unstripped form turned out to survive in only one place. resolveService returns a key from h.services, which SetServices normalizes, and cfg.Auth.Services is normalized too by deriveServices, so neither holds the original. Only cfg.Server.RegistryDomains, straight off the YAML, does, and it was never reaching the token package. Adds a display-only map keyed by the same NormalizeService function the other two key on, so a resolved service always maps back to the entry it came from, and a multi-domain deployment prints the domain the client is authenticating against rather than the primary. Collisions take the first configured entry, matching deriveServices' own first-wins dedupe. A miss falls through to the normalized name, which is today's behaviour. Audiences and routing are untouched, proven by a test that mints a real token, parses the JWT and asserts aud is still the normalized host. The wiring line landed in the previous commit, both changes having been made in server.go at the same time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9 |
||
|
Evan JarrettandClaude Opus 5
|
3589473feb |
appview: run the hold purge on a worker pool, not the request context
Deleting a tag while over quota could leave the user in the worst available state. DeleteTagHandler deleted the tag and manifest rows first, then called PurgeOnHold, which bounded itself at 10s against the *request* context. The UpCloud load balancer in front of the appview cuts at its default backend timeout at about the same moment, wins the race, hands the client a 504 and cancels that context, killing the purge partway. The appview logged a warning and returned 200. So: gateway error, nothing freed, still locked out, the image gone from the UI so the purge cannot be retried through it, blobs orphaned until the hold's GC, and the appview considering it a success. Measured on production at 10.002367218s. Purges now go to a fixed pool of 4 workers rooted at context.Background(), so they survive the request ending. Following the shape of the hold's startJob helper, minus the progress fragment, since nobody is watching a purge. The buffer is bounded at 256 and sheds with an ERROR rather than growing: an unbounded queue turns a slow hold into an appview memory leak. Submissions are deduplicated on holdDID|manifestURI so a double-clicked delete does one purge and one service-token fetch. The channel send happens under the mutex that guards close, so a concurrent drain cannot send on a closed channel, and the drain is wired into both exit paths before logging shuts down. Failures are now classified and surfaced instead of swallowed: transient ones retry three times under a 90s budget (the hold's purge is idempotent), an unauthorized third-party hold logs at DEBUG since it is expected, and anything else that exhausts its retries logs at ERROR naming the manifest and hold, which is enough to re-drive by hand. Deliberately not reordered. Purge-first-then-delete requires waiting for the purge to know whether to delete, which puts the 10s call straight back on the request. So the orphaned-blob window remains, materially narrower but real: a purge that exhausts its retries still leaves blobs referenced by nothing until the hold's GC, and there is no row left to say so. Closing that needs a durable pending-purge record, which was judged out of proportion here. server.go in this commit also carries one line belonging to the next one, the token handler's display-name wiring, since the two changes landed in the same file concurrently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9 |
||
|
Evan JarrettandClaude Opus 5
|
b4fccce4d1 |
hold: actually send subject, so the attestation scan guard can fire
The hold declines to enqueue a scan when a pushed manifest has a subject, which
is how it means to skip attestations, signatures and other referrer artifacts.
The AppView never sent one: notifyHoldAboutManifest built mediaType, config,
layers and manifests, and #manifestInfo defined only those four. So the
condition was always true, the guard never fired, and every referrer artifact
was enqueued for scanning.
The AppView already parsed subject. NewManifestRecord unmarshals it into
ManifestRecord.Subject, and Put hands that same pointer to
notifyHoldAboutManifest. The value was in scope and simply never serialized, so
this is one missing marshal step rather than a missing parse.
Adds subject to the lexicon as a #blobInfo ref, and mediaType to #blobInfo,
which config has always sent and the hold has always parsed. That only makes the
schema honest about what is already on the wire.
Hoists the hold's anonymous request struct to a named type with IsMultiArch,
IsReferrer and HasScannableContent, so the predicate is written once and
testable without standing up a HoldPDS.
Both directions degrade safely. An older hold ignores the unknown key and
behaves exactly as today, so shipping the appview alone is harmless but achieves
nothing until the hold catches up. An older appview sends no subject, leaving
the manifest enqueued as before.
Complements
|
||
|
Evan JarrettandClaude Opus 5
|
9566201377 |
appview: give the confirm modal an accessible name and a focus trap
The modal guards tag deletion, device revocation and webhook deletion, so it is the wrong control to leave unlabelled and escapable. Note the original report was half wrong: role="dialog" and aria-modal="true" were already set on the outer element, and are in the deployed bundle. The observation was most likely taken against the inner .modal-box. Neither was added here. What was actually missing: the h2 existed but nothing pointed at it, so a screen reader announced a dialog with no name; and the only focus management was focusing Cancel on open, so Tab walked straight out into the skip link and page header behind the backdrop. Adds aria-labelledby and aria-describedby with sequence-suffixed ids so two modals cannot cross-reference. aria-describedby matters more than usual here: none of the three call sites' messages contain ". ", so the title is always the generic "Are you sure?" and the specific text is the body. The trap re-queries on each keypress rather than caching at open, and handles three cases: focus escaping to body gets pulled back, first plus Shift+Tab wraps to last, last plus Tab wraps to first. Focus is restored to the opener on every close path, guarded by document.contains, because confirming a tag deletion fires an htmx swap that can remove the button that opened the modal. Restoration happens before onConfirm so htmx sees a sane focus state. Native <dialog> would give the trap and Escape for free, but it renders in the top layer while this is styled entirely with daisyUI .modal classes that assume a positioned div, so converting means CSS work plus the seamark theme fork. Worth doing as its own change, not smuggled into this one. Includes the bundle rebuild, since nothing in the dev loop keeps that artifact current. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9 |
||
|
Evan JarrettandClaude Opus 5
|
dbb195a4ab |
appview: process daily stats, and drop the dead README branch fallback
io.atcr.hold.stats.daily was handled in the backfill collection list and in processor.go's dispatch, but missing from isRelevantCollection, which gates events before ProcessRecord ever sees them. So daily stats records arrived over the socket and were discarded at the worker, and the trend charts that read them got nothing live. This is the "present in one list, missing from the other" shape CLAUDE.md's firehose checklist warns about. Checked the whole class: this was the only gap. LayerCollection and ImageConfigCollection are absent deliberately, having no processor handler, and the test now pins that intent. Note this is currently masked by the relay outage, so fixing the relay alone would not have restored the charts. Separately, the README resolution tried "main" and fell back to "master", but DeriveReadmeURL never fetches: it parses the source URL and interpolates the branch, returning empty only for an unsupported platform, which is branch-independent. So if the main call returned empty the master call returned empty for the same reason, and the fallback could never fire. Removed, with a comment recording that a branch fallback has to happen at fetch time after a 404. The other two call sites already do exactly that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9 |
||
|
Evan JarrettandClaude Opus 5
|
a95c89aaef |
docs: record how a hold falls off the relay and stays off
Around 2026-08-20 the Bluesky relay marked every ATCR hold offline and stopped dialing. Nobody noticed for two weeks, and it surfaced only indirectly as "pull and push counts are up to 24 h stale". The stats were a symptom; the fleet was simply disconnected. It cannot recover on its own. Indigo's relay gives up on a host after 16 consecutive dial failures and returns from the redialer, and only a fresh requestCrawl revives it. The hold sends requestCrawl exactly once, at boot (server.go:400), with no ticker and no check that any relay is subscribed. So a dropped hold is invisible until its process restarts, silently. Documents the current mechanics, the failure mode, how to diagnose it with getHostStatus and a frozen repo rev, and how to recover. The automatic re-crawl is described as a deferred proposal and explicitly NOT implemented, by decision: a jittered ticker guarded on subscriber liveness, plus surfacing the subscriber count, since the deeper problem is that this was silent. Two things found while writing it, both recorded. The proposal needs plumbing that does not exist: EventBroadcaster has no exported subscriber count, and Subscriber does not retain the userAgent, so "is a relay listening" cannot currently be answered. And ResubscribeAllHosts selects only active hosts, so an offline host is not recovered even by a relay restart. Carries a replay warning. |
||
|
Evan JarrettandClaude Opus 5
|
25e2aa0228 |
docs: design note for artifact type classification
Finding 32: pushing an artifact with an unrecognised config media type classifies as "unknown", and every template branches two ways on "helm-chart" with the container-image page in the else. So an in-toto attestation is served a docker pull command, Layers/Vulnerabilities/SBOM tabs, "Image layer history", and a promise that scans run shortly after push, which for that artifact will never be true. This is a design note rather than a fix, since the change is larger than the symptom. The inventory is the part worth keeping. The classification rule exists in four places, keyed off three different inputs (appview config media type, hold config media type with no unknown case, scanner config map plus layer shape, hold layer substrings), and the appview's artifact_type feeds none of the scan decisions. Manifest-level artifactType is discarded at parse time on every push: it is absent from the record struct, the constructor and the lexicon, surviving only inside the unindexed manifest blob. Two corrections to the framing this started from, both verified rather than assumed. The repo does not have referrers support: the pinned distribution version has no referrers code and ATCR registers no such route, so what exists is subject_digest persistence plus an attestation badge. And GetTopLevelManifests filters artifact_type != 'unknown', so an untagged unknown artifact is invisible while a tagged one renders as an image, which the finding did not mention. Proposes a type set, spec precedence (manifest artifactType, then config media type, then structural signals), a UI contract stating what such a page must not show, and a six stage plan. Only stage 2 needs a migration, for the raw string column; new slug values need no DDL and no data migration, since jetstream upserts artifact_type on every record it sees. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9 |
||
|
Evan JarrettandClaude Opus 5
|
7a0769d8e4 |
appview: three small UI fixes, empty state, icon sizing, aria-selected
Artifacts filter had no empty state: filtering to zero matches left a blank panel, indistinguishable from something being broken. It now uses the existing state-empty partial, with no CTA since the filter input is right there. Rows appended by Load More now also obey the active filter, which is a change to existing behaviour but is required for the empty state to be truthful. The tag icon beside the tag selector computed to 5.44px by 24px on every repo page: the class was right, the flex parent was shrinking it. Adds shrink-0 to that instance only. 31 other icons are direct flex children without shrink-0 and are left alone, since a site-wide sweep found only this one squeezed. Digest-page scan tabs set role="tab" but never aria-selected, while the repo-page tabs do. Both are now set server-side so first paint is correct, with JS keeping them in sync afterwards, following switchRepoTab's pattern. The diff-content tabs had the identical defect and are fixed too, since the handler is generic over radio tabs and leaving them out would have meant attributes set by JS but never by the server. Includes the bundle rebuild for these and the two /auth/token-adjacent JS changes, since nothing in the dev loop keeps that artifact current on its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9 |
||
|
Evan JarrettandClaude Opus 5
|
9d8bd513da |
appview: render the install scripts from config instead of shipping ATCR's
seamark.dev's /install and /settings/devices told users to pipe
seamark.dev/static/install.sh into bash. That file was the unmodified ATCR
script: it announced itself as the "ATCR Credential Helper Installer",
installed docker-credential-atcr, and finished by telling the user to configure
credHelpers for atcr.io, the wrong registry for that deployment. Anyone
following the documented setup ended up pointed at another service. The
templates hardcoded docker-credential-atcr, "atcr" and ~/.atcr/device.json
alongside a correctly themed {{ .RegistryURL }}.
The scripts are now rendered from config by a handler, rather than forked per
brand. A theme overlay was the alternative and was worse: it needed a full copy
of both install.sh and install.ps1 per brand, four scripts to keep in sync, and
the operator asked for these values to come from config.
credential_helper.name is the single knob. Docker resolves a credHelpers value
x by exec'ing docker-credential-x, so the credHelpers value, the binary suffix
and the config directory are genuinely one word, not three that can drift. It
is validated against a strict pattern because it is interpolated into a shell
script.
install.sh renders byte-identical to the deleted static file under the atcr
default, so existing installs are unaffected. install.ps1 differs by one line,
where a stale usage comment named a path the script is not served at.
Two behaviour changes worth noting: these two URLs drop from a one-year
Cache-Control to five minutes, since the body now depends on deployment config;
and credential_helper.tangled_repo becomes a real overridable default. It was
previously assigned over unconditionally and read by nothing, while the shipped
script used a different URL form.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDqoCE1j3njokkZ9b1C5n9
|