Files
at-container-registry/scanner/internal/scan/syft.go
T
Evan JarrettandClaude Opus 5 265e533ad3 scanner: reclaim scan directories a killed process left behind
buildOCILayout already removes its scan dir on every error path, and syft.go
defers the stereoscope generator's Cleanup. What neither can do is clean up
after a process that dies mid-scan: the deferred call never runs, and nothing
afterwards ever looks at what was left. Every restart therefore leaks the
in-flight layout and extraction permanently, and a restart is routine — a
deploy is one.

On seamark-hold that reached 8.8 GB of orphaned scan-*, syft-scan-* and
syft-cataloger-* directories under a 20 GB disk, at which point the disk was
97% full and scans began failing on it:

  failed to load OCI image: unable to populate layer cache
    dir="/var/lib/seamark/scanner/tmp/syft-scan-1546187834/..."
    : no space left on device

  failed to download layer 5: failed to write blob:
    write /var/lib/seamark/scanner/tmp/scan-4160414849/blobs/sha256/...
    : no space left on device

The leaked directories cluster at the scanner's restart timestamps, which is
what identifies the killed process rather than the error paths as the source.
Manual removal reclaimed 8.8 GB and took the disk from 97% to 50%.

Startup is where this belongs: it is the one moment the previous process is
known to be gone, and it is immediately after the event that caused the leak.
The sweep runs in WorkerPool.Start after TMPDIR is set and before any worker
can dequeue, so nothing it removes can be work in progress here.

Three constraints shape what it will touch:

  - Only the three per-job prefixes, only as direct children, only
    directories. The Grype database lives beside the tmp dir at
    <parent>/vulndb and go-getter unpacks into grype-dl underneath it; both
    are state the scanner needs and neither matches a prefix. The prefixes now
    have one definition each, used by both the creator and the sweeper, so
    renaming a directory cannot silently take it out of the sweep's scope.

  - An age threshold, vuln.sweep_max_age, default 1h. A second scanner sharing
    the directory has an in-flight scan-* dir that is minutes old, and
    scanner.job_timeout is 8m, so an hour clears both with room to spare. 0
    disables the sweep rather than removing a peer's live work.

  - Nothing is fatal. A stat or removal failure is a WARN and the sweep moves
    on, so a permission problem in the tmp dir cannot keep the scanner from
    starting.

The sweep only runs at startup, so a scanner that is killed twice between
deploys carries the first leak until its next restart. That is the tradeoff
for never racing a live peer; a periodic sweep would be the follow-up if
processes ever live long enough for it to matter.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TA9D4DjaLZTvzQ7dJbu4eg
2026-09-08 21:59:58 -05:00

140 lines
5.4 KiB
Go

package scan
import (
"context"
"crypto/sha256"
"fmt"
"log/slog"
"github.com/anchore/stereoscope/pkg/file"
"github.com/anchore/stereoscope/pkg/image/oci"
"github.com/anchore/syft/syft"
"github.com/anchore/syft/syft/format"
"github.com/anchore/syft/syft/format/spdxjson"
"github.com/anchore/syft/syft/sbom"
"github.com/anchore/syft/syft/source"
"github.com/anchore/syft/syft/source/stereoscopesource"
)
// generateSBOM generates an SBOM using Syft from an OCI image layout directory.
// Returns the SBOM object, SBOM JSON bytes, and its digest.
//
// sourceRef names the thing being scanned. It is what Syft records as the
// source, and it must be a function of the content and nothing else: it lands
// in the SBOM's documentName, documentNamespace, root package and describing
// relationship, and grype.go copies the whole source description into the
// vulnerability report. Callers pass the manifest digest — see the comment on
// the stereoscopesource config below.
func generateSBOM(ctx context.Context, ociLayoutDir, sourceRef string) (*sbom.SBOM, []byte, string, error) {
slog.Info("Generating SBOM with Syft", "ociLayout", ociLayoutDir, "source", sourceRef)
// Create stereoscope OCI directory provider
tmpGen := file.NewTempDirGenerator(syftTempDirName)
defer tmpGen.Cleanup()
provider := oci.NewDirectoryProvider(tmpGen, ociLayoutDir)
img, err := provider.Provide(ctx)
if err != nil {
return nil, nil, "", fmt.Errorf("failed to load OCI image: %w", err)
}
// The one stage of the pipeline no deadline reaches.
//
// img.Read() is stereoscope's layer extraction: it takes no context, and
// there is no supported way to interrupt it. It is also the most expensive
// thing the scanner does, measured at 81% of a node:22 scan. Checking
// before it starts is what can be done honestly — a job that has already
// spent its budget on downloads does not go on to spend another thirteen
// minutes extracting — and checking after is how an overrun is noticed at
// all. In between, the job runs to completion no matter what the deadline
// says.
//
// The alternative would be to run it on its own goroutine and return when
// the context fires, which trades a bounded overrun for a leaked goroutine
// still writing gigabytes into a directory the caller has cleaned up. That
// is worse. vuln.max_image_size is the real bound on this stage: it caps
// the input, and with it the extraction.
if err := ctx.Err(); err != nil {
_ = img.Cleanup()
return nil, nil, "", fmt.Errorf("before reading OCI image: %w", err)
}
if err := img.Read(); err != nil {
_ = img.Cleanup()
return nil, nil, "", fmt.Errorf("failed to read OCI image: %w", err)
}
if err := ctx.Err(); err != nil {
_ = img.Cleanup()
return nil, nil, "", fmt.Errorf("after reading OCI image: %w", err)
}
// Wrap in Syft source — src.Close() calls img.Cleanup() internally,
// so we don't defer img.Cleanup() separately.
//
// The reference used to be ociLayoutDir, which is the per-job
// os.MkdirTemp("scan-*") path buildOCILayout hands us. That path is
// different on every scan, and it reaches five places in the encoded SPDX
// document (documentName, documentNamespace, the DocumentRoot package's
// name and SPDXID, and the DESCRIBES relationship) plus the "source"
// object grype.go embeds in the vulnerability report. Rescanning unchanged
// content therefore produced byte-different artifacts under fresh digests,
// so the hold uploaded new blobs and orphaned the old ones on every pass of
// its stale-scan loop.
//
// The manifest digest is the identity the hold already uses: the
// io.atcr.hold.scan record's rkey is atproto.ScanRecordKey(manifestDigest),
// one record per digest regardless of which repository the job arrived
// under. "repository@digest" reads better, but the same digest is
// dispatched under whichever user's manifest discovery walked first, so it
// would reintroduce the same drift at a slower rate.
//
// Alias pins Name and Version rather than letting Describe() infer them:
// it runs the reference through distribution/reference, where a bare
// "sha256:<hex>" parses as the repository "sha256" tagged with the hex.
src := stereoscopesource.New(img, stereoscopesource.ImageConfig{
Reference: sourceRef,
Alias: source.Alias{
Name: sourceRef,
Version: sourceRef,
},
})
defer src.Close()
slog.Info("Running Syft cataloging")
sbomResult, err := syft.CreateSBOM(ctx, src, nil)
if err != nil {
return nil, nil, "", fmt.Errorf("failed to generate SBOM: %w", err)
}
if sbomResult == nil {
return nil, nil, "", fmt.Errorf("Syft returned nil SBOM")
}
slog.Info("SBOM generated",
"packages", sbomResult.Artifacts.Packages.PackageCount(),
"distro", func() string {
if sbomResult.Artifacts.LinuxDistribution != nil {
return fmt.Sprintf("%s %s", sbomResult.Artifacts.LinuxDistribution.Name, sbomResult.Artifacts.LinuxDistribution.Version)
}
return "none"
}())
encoder, err := spdxjson.NewFormatEncoderWithConfig(spdxjson.DefaultEncoderConfig())
if err != nil {
return nil, nil, "", fmt.Errorf("failed to create SPDX encoder: %w", err)
}
sbomJSON, err := format.Encode(*sbomResult, encoder)
if err != nil {
return nil, nil, "", fmt.Errorf("failed to encode SBOM to SPDX JSON: %w", err)
}
hash := sha256.Sum256(sbomJSON)
digest := fmt.Sprintf("sha256:%x", hash)
slog.Info("SBOM encoded", "format", "spdx-json", "size", len(sbomJSON), "digest", digest)
return sbomResult, sbomJSON, digest, nil
}