Vulnerability scanning produced nothing across the whole deployment for
nine days, from 2026-08-25 01:20:48 until a scanner restart on 2026-09-03.
The scanner was connected and idle, the hold's discovery pass kept
reporting unscannedFound=15 every four hours, and no scan_jobs row was
created in that entire window.
hasActiveJobs counted pending, assigned and processing rows globally with
no age bound, and waitForCapacity spins while it is true. dispatchLoop
calls it before popping any candidate, so a single pending row that never
reached a terminal state reported "busy" forever: discovery kept pushing
candidates into unscannedQueue and nothing ever popped them. That is why
the symptom was an empty queue rather than a growing one.
Nothing papered over it because push-triggered enqueue only fires for
owner or a tier with scan_on_push, which in production means pro alone.
All 210 manifests pushed to this hold in that window came from free,
supporter, or accounts with no crew row, so the frozen proactive loop was
the only source of jobs.
Nor could it recover on its own. Only Enqueue and drainPendingJobs
dispatch a pending row, and drainPendingJobs runs only when a scanner
newly connects; reDispatchTimedOut considered assigned rows only. The
hold had been up since Aug 14 and the scanner since Aug 21 on the same
websocket, so the drain path had not run since the row appeared.
So bound the capacity gate to pending rows younger than pendingStaleAfter,
give reDispatchTimedOut a pending reclaim, and check RowsAffected on the
assign UPDATE now that two dispatchers can race for a row. waitForCapacity
warns and names the blocking jobs after ten minutes without capacity,
because the failure mode above was completely silent.
Two adjacent fixes for the same outage. The scanner never called
InitLogger, so log_level and log_shipper were dead config and an idle
scanner was mute, which is what made nine days invisible. And skipReason
now also skips a job whose layers contain nothing tar-shaped: the job that
wedged this queue was an in-toto attestation whose config mediaType is an
ordinary image config, so the existing config-type check missed it and
buildOCILayout would have handed Syft an empty image.
The regression tests were verified against the old logic first: three of
them fail on it and pass on the fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPWkeCKcbtGoyXyyeMhSps
scanner/internal/scan had no test file at all, which is why fa1dfb0 could be
written and reviewed without anyone being able to state its defect as an
assertion. Six tests now do.
The load path is only reachable from a test through an indirection, so
grype.LoadVulnerabilityDB is now behind a loadVulnDB package var. Everything
worth testing here is what happens when that call returns a stale database or
fails outright, and neither is reachable from a test that has to perform a
real download.
Covered: freshness is taken from the DB's own build timestamp; a fresh DB is
reused without a download; a stale DB inside the retry backoff keeps serving
without one; the backoff expires and the replacement is adopted with its
predecessor closed; a failed reload with a usable provider in hand keeps
scanning and still advances the attempt timestamp; and a cold-start failure is
an error rather than a scan that silently finds nothing.
Verified by mutation. Restoring `vulnDBBuilt = time.Now()` — the original
defect — fails the freshness test with the stale build time in the message.
Removing the serve-the-old-DB fallback fails the outage test.
One honest limit, recorded in the test file. The backoff is tested twice in
the production code, on the read-lock fast path and again under the write
lock, and mutation shows they are redundant for correctness: deleting either
alone leaves the throttle test passing, and only deleting both fails it. The
fast-path copy exists so a stale DB does not push every scan through the
exclusive lock, which is a contention property, not a behavioural one, and no
unit test can assert it without being flaky.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Freshness was measured from load time, so a load that fell back to a
stale-but-valid on-disk DB earned a fresh cache lease and could ride past
Grype's MaxAllowedBuiltAge cliff. Measure from the DB's own build
timestamp instead, and throttle reload attempts with a 30m backoff so a
down upstream doesn't make every worker pay its own download timeout.
Two locking fixes come with it, both reachable only once the DB is stale
and so newly relevant now that staleness is tracked honestly:
- FindMatches ran on a provider fetched outside the lock while a reload
could Close() it under the write lock. The scan now holds the read
lock across matching and reads the provider under it, so a reload
waits for in-flight scans instead of closing a store mid-scan.
- The retry backoff is also tested on the read-lock fast path. Once the
DB is stale the freshness test never passes again, so every scan was
taking the exclusive lock purely to reach the backoff return.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Evan JarrettandClaude Opus 5