mirror of
https://tangled.org/evan.jarrett.net/at-container-registry
synced 2026-09-20 17:24:16 +00:00
Evan JarrettandClaude Opus 5
f84e8ffa27
Groundwork for running more than one AppView instance. Nothing is wired to this
yet; the next commit moves the background workers onto it.
Several workers must run on exactly one instance. The Jetstream consumer is the
sharpest case: StatsCache is per-process in-memory state, and the aggregate it
produces is written to repository_stats as an absolute value rather than an
increment, so two consumers would each hold a partial view of the holds and each
write its partial sum as the whole truth, overwriting one another indefinitely.
The webhook dispatcher hangs off the same processor, so a second consumer also
means every webhook fires twice.
Instances contend for a named lease; only the holder runs the worker. Acquire is
a single INSERT ... ON CONFLICT ... WHERE, so two instances racing for the same
expired lease cannot both win: the loser's update matches no rows. The fence
token increments on every change of custody, so a process that stalled past its
TTL discovers on its next renewal that it was superseded, rather than continuing
to act as the holder.
A renewal blackout is treated as a loss. If the database has been unreachable
for longer than the TTL, another instance is entitled to steal the lease and we
must assume it has, even though we cannot ask. Continuing to work in that state
is the one outcome the lease exists to prevent.
Clean shutdown expires the lease in place rather than deleting the row, so a
replacement starts in seconds instead of waiting out the TTL, while the fence
token survives to keep a stalled former holder from matching again.
Timestamps are Unix milliseconds, not TIMESTAMP text. libSQL normalizes
date-like TEXT on the way in, and Go's driver and CURRENT_TIMESTAMP disagree on
format, so a stored expiry and a literal would compare as strings that sort
differently. That comparison is the whole safety property, so it does not get to
be subtle. The cost is a dependency on roughly-synced clocks, the same
assumption Kubernetes leases make; keep the TTL well above any plausible skew.
The lease tests are file-backed rather than :memory:. go-libsql gives every
connection to an in-memory DSN its own private database, so with MaxOpenConns of
8 a second goroutine lands on a connection where the schema was never applied
("no such table"). Every existing test in the package is sequential and reuses
one pooled connection, which is why this has stayed invisible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
32 lines
1.6 KiB
YAML
32 lines
1.6 KiB
YAML
description: |
|
|
Leader election for AppView background workers.
|
|
|
|
The Jetstream consumer, backfill, labeler subscriber and cleanup loops all
|
|
start unconditionally in every process today, which is fine for one instance
|
|
and actively harmful for two. The consumer is the clearest case: StatsCache is
|
|
per-process in-memory state whose aggregate is written to repository_stats as
|
|
an absolute value, so two consumers would each hold a partial view of the holds
|
|
and each write its partial sum as the whole truth, overwriting each other
|
|
forever. The webhook dispatcher hangs off the same processor, so a second
|
|
consumer also means duplicate deliveries.
|
|
|
|
Instances contend for a named lease and only the holder runs the worker. The
|
|
fence token increments on every change of custody so a process that stalled
|
|
past its TTL can detect it was superseded rather than continuing to act as the
|
|
holder.
|
|
|
|
Timestamps are Unix milliseconds rather than TIMESTAMP text. libSQL normalizes
|
|
date-like TEXT on the way in, and Go's driver and CURRENT_TIMESTAMP disagree on
|
|
format, so a stored expiry and a literal would be compared as strings that sort
|
|
differently. That comparison is the entire safety property of the acquire
|
|
statement, so it gets an integer.
|
|
query: |
|
|
CREATE TABLE IF NOT EXISTS instance_leases (
|
|
lease_name TEXT PRIMARY KEY,
|
|
holder_id TEXT NOT NULL,
|
|
fence INTEGER NOT NULL DEFAULT 0,
|
|
acquired_at INTEGER NOT NULL,
|
|
expires_at INTEGER NOT NULL
|
|
);
|
|
CREATE INDEX IF NOT EXISTS idx_instance_leases_expires ON instance_leases(expires_at);
|