mirror of
https://tangled.org/evan.jarrett.net/at-container-registry
synced 2026-09-02 00:06:58 +00:00
Evan JarrettandClaude Opus 5
8c9a85826d
6a7ddb8 moved RefreshHoldTiers under a lease with the comment "one-shot; the
lease is released when it returns". It never returns: past the startup retries
it sits on a 30-minute ticker forever. Three consequences, all observed on a
two-instance run against one database:
The billing-tiers lease is never released on a clean stop, so it survives to
its TTL and the replacement instance waits a full minute for a worker the old
one is no longer running. Worse, the goroutine stays in the shutdown WaitGroup,
so "Timed out waiting for leased workers to stop" now fires on EVERY clean
shutdown. The other four leases released in 12ms and the warning fired anyway.
A warning that is always present cannot report the case it exists for, which is
the jetstream lease genuinely failing to release.
And the lease was the wrong tool regardless. The commit justified it as
"RefreshHoldTiers writes tier state derived from Stripe" that instances would
race on. It writes holdTierCache, a per-process map, from read-only ListTiers
calls; there is no shared state anywhere in the path. Electing one refresher
means every other instance keeps an empty cache forever, so
aggregateHoldFeatures reports "no hold data" on all but one — a regression that
only appears at the scale the lease was added to support. This is the hold
health worker's situation exactly, and that one was deliberately left unleased
in the same commit.
So it runs on every instance again, with a context. The retry backoff was
time.Sleep for up to 45s total against an unreachable hold; it and the ticker
now select on ctx.Done, and ListTiers gets the context instead of
context.Background. LeaseBillingTiers is gone rather than left as a dead name.
Verified: with the backoff restored to time.Sleep the new test fails on its own
2s deadline rather than hanging, which is how a shutdown regression here should
present.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011rmjvU2gSRL9wFnmqVsWaF