mirror of
https://tangled.org/evan.jarrett.net/at-container-registry
synced 2026-09-10 04:06:06 +00:00
Evan JarrettandClaude Opus 5
dfd604b106
Vulnerability scanning produced nothing across the whole deployment for nine days, from 2026-08-25 01:20:48 until a scanner restart on 2026-09-03. The scanner was connected and idle, the hold's discovery pass kept reporting unscannedFound=15 every four hours, and no scan_jobs row was created in that entire window. hasActiveJobs counted pending, assigned and processing rows globally with no age bound, and waitForCapacity spins while it is true. dispatchLoop calls it before popping any candidate, so a single pending row that never reached a terminal state reported "busy" forever: discovery kept pushing candidates into unscannedQueue and nothing ever popped them. That is why the symptom was an empty queue rather than a growing one. Nothing papered over it because push-triggered enqueue only fires for owner or a tier with scan_on_push, which in production means pro alone. All 210 manifests pushed to this hold in that window came from free, supporter, or accounts with no crew row, so the frozen proactive loop was the only source of jobs. Nor could it recover on its own. Only Enqueue and drainPendingJobs dispatch a pending row, and drainPendingJobs runs only when a scanner newly connects; reDispatchTimedOut considered assigned rows only. The hold had been up since Aug 14 and the scanner since Aug 21 on the same websocket, so the drain path had not run since the row appeared. So bound the capacity gate to pending rows younger than pendingStaleAfter, give reDispatchTimedOut a pending reclaim, and check RowsAffected on the assign UPDATE now that two dispatchers can race for a row. waitForCapacity warns and names the blocking jobs after ten minutes without capacity, because the failure mode above was completely silent. Two adjacent fixes for the same outage. The scanner never called InitLogger, so log_level and log_shipper were dead config and an idle scanner was mute, which is what made nine days invisible. And skipReason now also skips a job whose layers contain nothing tar-shaped: the job that wedged this queue was an in-toto attestation whose config mediaType is an ordinary image config, so the existing config-type check missed it and buildOCILayout would have handed Syft an empty image. The regression tests were verified against the old logic first: three of them fail on it and pass on the fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DPWkeCKcbtGoyXyyeMhSps