Three scripts, split by what they cost to run.
batch07-stale-preview.mjs stages a preview and waits out the 30-minute
maxPreviewAgeForDelete constant. batch07-stale-click.mjs is the resumable
half: it re-renders whatever preview the hold already holds and clicks delete
on it. GET /admin/api/gc/status re-renders lastPreview WITHOUT touching
lastPreviewAt, so showing an old preview does not reset its age — which is
what makes a failed run cost seconds instead of another 31 minutes.
Result against the dev hold: "preview is 34m0s old (limit 30m0s) — run Scan
again before deleting" rendered through the progress-to-error fragment chain,
with all 387 records still there afterwards. That chain is the point; the
refusal logic itself already has a Go test, but a refusal that renders as
nothing is indistinguishable from "there was nothing to delete".
batch07-sweep.mjs then runs the destructive path for real: 387 records
deleted of 387 staged, orphaned count to zero, referenced blobs unchanged at
15. Safe only against the dev hold on Storj; production is Bunny + UpCloud
and is not reachable from here.
page.on('dialog') did not reliably intercept hx-confirm on this page, and an
unaccepted native dialog blocks every later evaluate() and innerText(), so
the script hangs rather than fails — the worst failure mode for an unattended
check. Both scripts now strip the hx-confirm attribute before clicking. The
confirm is not what is under test.
Two findings worth carrying, neither introduced by this range:
* deleteOrphanedBlobs is still unexercised. The bucket holds 19 objects,
of which 8 are past the 7-day blob grace, and none are unreferenced — so
there is nothing for it to collect. More pushes cannot help: fresh blobs
are inside the grace window by definition.
* Storage accounting is derived from layer records, so this sweep moved the
dashboard from 1.3 GB to 1.1 KB while the bucket held 147 MB throughout.
It was overstating by ~9x before (records for blobs held by another hold)
and understates now (referenced blobs with no layer records). Quotas and
billing read the same number. Belongs to batch 12.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drives "Scan for Orphans" through the admin panel and asserts the wiring the
Go tests structurally cannot: that the progress fragment swaps into
#gc-results and hands off to the preview fragment, that every advertised stat
renders a value, and that each table's row count agrees with the stat above
it. A GC that decides correctly and renders a blank panel still gets someone
to click delete on the wrong thing.
Then it hand-checks the claim itself, which is the part that matters: for
each distinct manifest behind an orphaned record, resolve the owner's PDS and
ask whether that manifest is really gone.
Two instrument bugs were found writing this, both in the script rather than
the product, and both worth keeping as comments:
* The three tables overlap on a Digest column, so classifying by "has
Digest but no RKey" swallowed Missing Records as orphaned blobs and
reported 3 blobs against a stat of 0. Classification is now by exact
header set.
* Asserting the manifest is ABSENT from the PDS is too strong. A manifest
can be alive and name a different hold, which is exactly what happens
when defaultHold is repointed and the image re-pushed. Those records are
legitimately orphaned here. The only state that means GC is staged to
destroy live data is a manifest that exists AND still names this hold.
Against the dev hold: 387 orphaned records over 68 distinct manifests, 25
sampled — 17 gone, 8 alive but now pointing at the production hold, 0 still
naming this hold. Orphaned blobs 0, referenced 15, and the counts agree with
the tables.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
oauth-refresh-e2e.sh covers the refresh mechanics; this covers the symptom the
range exists to stop — a signed-in browser being thrown out when the access
token rotates underneath it.
The test is only meaningful because ui_sessions is a table carrying an
oauth_session_id rather than an in-memory map, so restarting the appview to
clear the refresher's cache does not by itself log the browser out. Verified
against the row the browser actually uses: one oauth_sessions row, and the
ui_sessions row created by the login points at it. rev advances 1 to 2 while
/settings/user keeps rendering.
Drives the login itself. Given a handle it fills the field and clicks through
consent, which is the whole flow whenever the PDS already has a session. Two
things it must not do, both learned by doing them:
* Never navigate while waiting for a human. The first version re-issued
goto() every two seconds and wiped the login form out from under whoever
was typing into it.
* Never bail permanently at a password field. Returning there left the flow
stranded on the Authorize screen once the password had been submitted,
because nothing was left to click it. It now pauses and resumes.
ATCR_E2E_FRESH clears only the appview's cookies. Clearing all of them takes
the PDS session with it, which turns a handle-and-consent flow into a password
prompt and makes the login impossible to drive unattended.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
client_test.go reproduces the cancellation precisely, but in-process against an
httptest PDS. The failure this range fixes — mass sign-outs — happened against a
real one, through the real refresher and the real oauth_sessions row, so the
unit tests alone are a thinner sign-off than the batch deserves.
Staling the access token in the live row and restarting the appview (the
refresher caches sessions in memory, so editing the DB alone changes nothing)
forces the real refresh path. Four concurrent pulls then advance rev 1 to 3
rather than 1 to 4: the compare-and-swap collapses four racing refreshes into
two rotations, with the losers adopting the winner's token instead of each
burning one. Killing a pull 150ms into that window and retrying still succeeds.
Backs ui.db up first, since a burned refresh token would otherwise leave the
account signed out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Migrations are forward-only and the appview applies whatever is missing on
boot, so moving to the next batch does not need a reset at all — Air rebuilds
into the new code and the live DB migrates in place. Verified moving onto
val/04-oauth: 0031 appeared in schema_migrations on its own, on top of a
level-27 database, with the appview healthy afterwards.
That matters more than it sounds. ui.db holds the OAuth sessions and the
appview's signing keys, so the old wipe-on-every-switch cost an interactive
`docker-credential-atcr login` per batch, which is most of what made the stack
awkward to hand to an agent. Validating in stack order is all forward motion,
so in the normal case there is now no login at all.
A reset is still done when the live DB carries migrations the branch's code has
never heard of, which is what going backward means, and the per-set snapshot is
still banked so that case can restore rather than start empty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Batching reorders migrations. val/04-oauth carries e75b2e2 (commit 48 of the
range), whose migration is 0031, while 0028-0030 only arrive in batches 09, 12
and 14. So val/04 holds {..0027, 0031} and val/09 holds {..0027, 0028, 0031}.
Both have a max version of 0031, so keying the snapshot on the max would
restore val/04's database onto val/09 — a database missing 0028's schema while
schema_migrations claims otherwise.
Snapshots are now keyed on a fingerprint of every migration file present, which
distinguishes those two states. The max version is kept in the filename purely
so the directory stays readable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wiping ui.db on every switch also wipes the OAuth sessions and the appview's
oauth_p256/jwt_rsa keys, so each batch cost an interactive
`docker-credential-atcr login`. There are only five distinct migration levels
across the stack (27 for batches 00-08, 28 for 09-11, 29 for 12-13, 32 for 14,
34 for 15), so the DB is snapshotted per level and restored instead of
re-migrated. One login now serves every batch sharing a level.
Also skip the compose pin from val/01 onward: a7c7db6 lands there, so the
branch's own compose already has the shared netns and pinning would drag in
later batches' changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jobs_test.go covers the job framework thoroughly, but nothing covers the
wiring: whether the kickoff handler renders the progress fragment into the
right hx-target, and whether the loop actually outlives the request it was
started from. Both are what ab4a4eb changed, and both are invisible to Go
tests — a typo in an hx-target or a fragment that renders blank passes every
assertion we have.
The load-bearing check drives crew import rather than the tier remap. A
one-member remap completes in under a second, so closing the tab "mid-run"
proves nothing; import does a PDS write plus a network PLC lookup per entry,
which leaves a real window to close the browser and watch the job keep going.
It is caught mid-flight at a progress tick with no admin page open.
Seeded members are created on the local-only dev hold and removed in a
finally block. README records the environment traps found while building
this: 127.0.0.1 vs localhost, in-memory sessions dying on every hold rebuild,
UA/IP pinning that makes curl log you out, and the forward-only appview
migrations that require a per-batch DB reset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Evan JarrettandClaude Opus 5