Completes the swap 0033 set up. layers and manifest_references move onto
manifest_key and manifests.id is gone, which removes the last node-allocated
identifier in the AppView schema.
Statement order in 0034 is load-bearing. With foreign keys on, DROP TABLE
performs an implicit DELETE FROM, so dropping manifests while layers still holds
an ON DELETE CASCADE reference deletes every layer row. Migration 0009 did
exactly that; it went unnoticed because the Jetstream backfill rebuilds layers
from PDS records, so the damage healed itself. PRAGMA foreign_keys is no help:
it is a no-op inside a transaction and migrations run in one. So the new
children are built pointing at manifests_new, the old children are dropped
first, and only then is the old manifests table dropped, by which point nothing
references it. Verified both behaviors before relying on them.
manifest_key is declared NOT NULL as well as PRIMARY KEY, because in SQLite a
PRIMARY KEY column still accepts NULL unless it is INTEGER PRIMARY KEY. That
constraint immediately caught four test helpers inserting manifests without one.
Five queries used MAX(id) as "the newest manifest in this repo", which I had
previously reported as absent after grepping only for ORDER BY. A derived key
has no ordering, so recency now comes from created_at with manifest_key as a
deterministic tiebreak. This is a real behavior change, and a fix: the two
disagree whenever a manifest is indexed out of order, which the backfill does
routinely, and created_at is the push time these queries always wanted. Both
directions are tested, including that ties resolve the same way every run.
InsertManifest and BatchInsertManifests no longer read anything back. The key is
derived from (did, repository, digest), so the writer knows it before the
statement runs: the select-back, its per-DID IN list, and the "manifest missing
id after batch insert" branch all go away, along with the UNIQUE-conflict
fallback that existed only to recover a rowid.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First half of replacing manifests.id with a node-independent identity. Nothing
depends on the column yet: id is still the primary key, and layers and
manifest_references still reference it. Getting the column in place and filled
first means the eventual swap operates on data that is already complete and
already proven unique, instead of doing the fill and three table rebuilds in one
step.
The value cannot be computed by the migration. It is a truncated sha256, SQLite
has no hash builtin, and go-libsql exposes no way to register one. So the fill
uses machinery that already exists: the Jetstream backfill re-upserts every
manifest across the protocol on startup, and both upsert paths now write
manifest_key.
That only works because of one extra clause. Both upserts guard their DO UPDATE
with a WHERE that skips rows where nothing changed, which on a backfill re-run is
nearly every row, so they would have skipped the very manifests that need
filling. Adding "OR manifests.manifest_key IS NULL" is what makes an otherwise
no-op pass populate the column. Verified by removing it: the backfill then fills
zero of three manifests instead of three of three.
The index is UNIQUE even though the column is nullable. SQLite treats NULLs as
distinct, so unfilled rows coexist while every filled row is checked. That makes
production data verify the 16-byte truncation rather than us assuming it: if two
manifests ever derived the same key, it fails loudly at insert instead of
silently attaching one manifest's layers to another after the swap.
AppView logs the unfilled count at startup, since there is no single moment at
which this becomes complete and the follow-up migration is only safe at zero.
ManifestKey replaces the old fat "did|repo|digest" map key rather than sitting
beside it; they were always the same question, answered without asking the
database.
The jetstream tests hand-maintained their own CREATE TABLE statements, which is
the drift problem moved into a test: the copy had already fallen behind (it still
had tags.id) and only failed once a query touched the difference. They use
db.InitDB now, with foreign keys switched off to preserve the behavior the
hand-rolled schema had.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Evan JarrettandClaude Opus 5