mirror of
https://tangled.org/evan.jarrett.net/at-container-registry
synced 2026-09-23 10:44:16 +00:00
Evan JarrettandClaude Fable 5.1
56bde61555
A large layer went up strictly one step at a time: fill 16MB from the client, stop reading, fetch a part URL and PUT the part to S3, reset, resume reading. While the part was in flight Docker sat on a full TCP window; while the buffer filled S3 sat idle. Wall clock was receive time plus send time. The writer now hands a full buffer to a goroutine that does the hold call and the PUT, and keeps filling a second buffer from the client. When that one fills it waits for the previous part, takes its buffer back, and hands the new one off. At most one part is in flight, so part numbers and ETags stay ordered, and a blob that never fills a buffer never allocates the second one. No network runs under the writer lock on the happy path. Peak memory for a large upload is now two buffers, 32MB. Both are charged to the process budget through the existing accounting, the second as it grows, and the budget floor rises to match so a large upload can never be refused outright. The 512MB default holds sixteen. A failed part records a sticky error, closes the writer, and aborts the multipart from the goroutine that still holds the upload ID; the next Write, hand-off, or Commit reports the cause. Commit verifies the digest first, then waits for the flight, sends the final part, and completes. Cancel waits for the flight, bounded, before aborting so the abort cannot overtake a PUT that has not yet been issued its upload ID. The sweeper refuses to reap a writer with a part in flight, since last activity is only stamped when a part lands. Tests observe the overlap directly: the fake S3 blocks the first PUT and the second buffer's writes are asserted to return before it is released, while the third buffer's writes block. Also covered: the one-part-late error, Commit waiting, Cancel during flight, peak budget, the sweeper, and concurrent Cancel and Write under the race detector. The integration suite passed with a 72MB layer pushed through the pipeline by three clients. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Yf1ZVA7sXYhQNb9tCo1m5