Add ordered content sources and targets to the shared copy pipeline. Observe
fs.File metadata before target creation, bind fs.FS sources, and expose
fatal-stop notification while preserving native copy behavior.
Overlap source reads, hashing, ordered linear writes, and asynchronous cleanup with bounded buffers. Preserve conflicting output order, API behavior, and no-space diagnostics. Add an LTFS drive-buffer model and reproducible performance checks.
Three review follow-ups that the last release review asked for:
- TESTING.md no longer claims a formulation vet accepts for the Windows mapped
slice conversion; measured on go1.26.8/windows-amd64, the unsafe.Slice call,
a typed pointer in a variable and an array cast are all flagged, and the one
accepted shape is the deprecated reflect.SliceHeader field assignment, which
trades a false positive for a less safe conversion.
- The report chapter drops the byte-identity claim about rows written without
full_path and signature_cache_hit and states what the fields are instead.
- The mmap divergence note and File's own comment now say that ReadAt past the
end reports an invalid offset rather than io.EOF, and that a caller keeping
the descriptor must keep the reader reachable because the finalizer owns it.
- TestManagedSignatureIsNotOrdinaryCopiedXattr now exercises the copy path: a
decoy managed entry on the source and an ordinary control attribute prove the
metadata restore ran, the target carries no managed attribute, and the source
keeps its own entry.
TestRunReportsEveryAcceptedItemExactlyOnce required that at least one accepted item happened to be
abandoned after the results callback cancelled the context, and that Wait returned the cancellation.
Both claims describe a run that observed the stop, which depends on how far the pipeline got before
the callback ran: a fast run finishes every item, and the sibling test
TestRunDoesNotReportAStopAfterACompleteRun pins that such a run reports success. The assertions, not
the pipeline, were wrong: the test failed roughly once in two hundred runs on an unmodified tree.
The test keeps the cancellation and now asserts the accounting that holds either way: every accepted
item owns exactly one terminal outcome, an item the stop reached carries the stopping reason instead
of a completion, a completed item has its target on disk, Wait is nil or that same stopping reason
and never another error, and at least the item whose delivery stopped the feed completed. Deliberate
mutations (dropping an item the stop reached, publishing one item twice) still fail all three modes,
so the accounting assertions remain non-vacuous.
- The event table said "The whole set is:", which reads as if every run emits every event. It now
says what a run can emit and which parts depend on the run: EventUpdateJob comes from the
af05f05c shell alone, EventSignatureCacheSummary from a run with a hash policy other than the
default HashOff, and EventReportError from a run that recorded a pipeline-level error.
- protectCall still named a batch source and an "item callback", both removed with the streaming
API. It now names the calls it actually wraps: the results callback, an event handler, and
Item.Source/Item.Targets.
- The compatibility section presented post-af05f05c symbols as "the public surface of af05f05c".
It now says the shell restores that surface and carries today's additive symbols beside it, which
is what the symbol list and the following bullet actually describe.
- "registering the same handler twice is not supported" contradicted the last-wins rule stated a
few lines above. WithEventHandler replaces the previous registration, so registering the same
handler twice is harmless and it is still called once.
Each of these described behaviour the code does not have:
- The nil-item note on Submit and the comment on its test claimed a rejected submission ends the
run, while the feed keeps accepting items; the test now pins that follow-up submissions are still
accepted and complete.
- The buildOption comment claimed every repeated option is last-wins, although the job and device
options accumulate.
- The usesCache comment claimed stored hashes are always read, while HashRead never looks one up;
the function decides whether a run needs a cache at all.
- TESTING.md claimed the non-linear pre-allocation on /dev/full reports ErrTargetIO. Measured on
Linux, fallocate there returns ENODEV, which matches none of the classified errors (ENOSPC,
EROFS, EIO), so the target carries the device's own error.
- TESTING.md's "every signature cache test" filter missed TestRunHoldsTargetCacheUntilTheHashIsComplete
and TestManagedSignatureIsNotOrdinaryCopiedXattr.
- The Windows vet diagnostic that make check does not run is documented where the cross-platform
commands live, with the reason it is accepted instead of worked around.
Three findings from the release review, all in the package's platform surface:
- An open mapping of an empty file was indistinguishable from a closed reader, because both were a
nil slice, so ReadAt and Slice answered `mmap: closed` to a standard-library io.ReaderAt consumer
that probed an empty source. The empty mapping is a non-nil empty slice now, which makes it read
like an empty *os.File or bytes.Reader while a closed reader still reports itself as closed.
- The Linux prefetch passed `MADV_SEQUENTIAL|MADV_WILLNEED` as one advice. An madvise advice is a
value, not a flag bit, so that expression is simply MADV_WILLNEED and the sequential hint was
silently dropped. Each hint is its own call now, and the syscall is indirected so a Linux-only
test can pin the advice sequence and the release-on-failure path that needed syscall injection
before.
- The descriptor-only fallback installed no finalizer, so an abandoned reader leaked its descriptor
where the mapping platforms release theirs. It installs the same finalizer, and its Close comment
now describes what all platforms do.
NewReportGetter rendered its rows by ranging over a map, so two runs over the same items could
write different documents even though every row was identical. The getter now emits the rows in the
order of the joined relative path that keys them, and a test renders the same set of rows eight
times so a map order cannot pass by chance.
The README states what the document actually contains while it is being corrected: an empty files or
errors array is omitted, so a run without rows and without pipeline problems writes `{}`.
The hash goroutine closed cacheGate at the end of its body, so a panic in the hasher skipped the
close and left every target writer parked on the gate. The panic was recovered and recorded as the
run error, but the copy stage, Close, Wait and WaitErr never returned, which contradicts the
fatal-panic guarantee that a stopped pipeline leaves no blocked handoff behind.
The release is deferred now, so it runs on the panicking path as well. The regression test installs
a hasher whose Write panics and asserts that the run ends with that panic instead of hanging; the
focused-command list in TESTING.md pins it beside the fatal-panic test.
refreshCacheEntry built its entry from the facts recorded at indexing while the hash came from
the bytes the run actually read. A source replaced between indexing and the read was therefore
described by the entry as the old version, so a later run that saw those old facts took a hit and
reported a SHA-256 for content it never read.
The publication is verified through the descriptor the item still owns: the source descriptor
must still show the size the item read and the mtime it indexed, and a target descriptor must
hold exactly the length that was hashed, because the run stamps the item's metadata onto it after
the entry is published. A source that cannot be shown to be the version the run hashed publishes
nothing instead of a guess, which is also what the docs now state, together with the one
path-based cache step those invariants already had.
Both were invisible on Darwin: one needs a real Linux ENOSPC, the other needs
the command's own stderr.
- A non-linear target pre-allocates with fallocate, and a failure there was
reported as a bare error: it never ran through mappingError, never aborted
the device and never ended a linear target, unlike every neighbouring path.
A volume that fills up after the disk-usage snapshot therefore reported a
target that was "not written" instead of ErrTargetNoSpace. Classify it like
the rest. TestFullVolumePreallocationReportsNoSpace pins it with a real
ENOSPC on a small tmpfs (it fails on the previous revision), and
TestNonLinearTargetOnRefusingDeviceReportsATargetFailure records that a
device refusing fallocate reports its own refusal rather than claiming no
space. The drain test now uses a linear target, which is the shape that
actually queues buffers before the write fails.
- cmd/acp registered the progress bar and then the report collector, and a run
keeps one event handler, so the bar received nothing: the command printed its
construction frame and no update. Compose the two handlers in the command and
pin it end to end (the new test fails on the previous revision). README now
says a command that wants both composes them.
- item.go still promised that an empty Targets slice hashes the source; the
default HashOff reads nothing. TESTING.md and TODO.md now describe the Linux
suite that actually exists.
A release review of the v0.2.1 branch found one data race and a set of
documentation claims the code does not support.
- Make the feed counters atomic: Submit writes them while the pipeline reads
them for its terminal event, and a caller may Close from another goroutine.
The new test reproduces the race on the previous revision under -race.
- Correct the withdrawn-version mapping: WithSignatureCache(true) becomes
HashCachedOrReadRefresh (HashReadRefresh never reuses a stored hash, so the
old mapping silently lost every cache hit), and add the SourceWithPath and
Job.Path string-to-segments rows.
- Correct the targetless-hash promise: an empty Targets slice writes nothing,
and it only hashes when the policy produces a hash, so the default HashOff
records none.
- Make the report chapter honest about row keying, the restored shell carrying
today's additive options, the second deliberate difference from af05f05c
(the managed signature cache), the rewrite tool's tab indent, the
duplicate-path rule, and the mmap divergence from upstream.
- Fix TESTING.md: the drain test is portable, and only the FreeBSD build
compiles mmap_other.go.
- Fix AGENTS.md: only value options are last-wins, a reuse-only hash policy is
an item error rather than an option error, and the event fan-out is the one
handoff without a hardStop escape.
- Pin the behaviours the review found uncovered: the interval flush, the
result handoff's hardStop escape, and an Item.Targets panic.
- Harden the read-buffer pool instrumentation: two collections drop the buffers
cached in another processor's private slot, which a drain loop cannot reach,
so the refcount assertions stop being vacuous and stop flaking.
Reshape the public API around one push engine and restore the af05f05c
surface as a compatibility shell over it.
- NewStream takes caller-owned items (Source/Targets), reports every accepted
item through one results callback, and Wait returns the run's terminal error.
Run, BatchSource and the item callbacks are gone.
- The engine owns the result path: WithResultBuffer is the depth of the result
queue, WithResultBatch is how many results one delivery carries, and
WithResultFlushInterval is how long a successful result may wait. A result
that carries any error is delivered immediately as its own batch.
- Restore the af05f05c API byte-compatibly in the same package: New, Copyer,
Wait/WaitErr, AccurateJob, WildcardJob, the report events and the JSON report
round-trip through encoding/json alone.
- One descriptor serves a whole item; the stored hash is read through the
descriptor that reads the content and the computed hash is published through
it before it closes.
- Make every handoff escapable by a hard stop: the read buffer, a chunk send,
an event send and the result buffer.
- Fix the recorded sweep findings: report indent and round-trip, cmd/acp exit
status, acp-rewrite final paths, mmap EOF/close/slice, disk-usage
reservations, ErrTargetIO identity, the fatal-panic safety net, option
validation, and the FreeBSD/Windows build paths.
- Cover the contract with semantic tests, add Makefile check/race/vet/cross
targets, and document the withdrawn v0.1.0/v0.2.0 migration in README.
Records two measurements as repeatable benchmarks: the same hash-only read
through the buffered and the mapped source read mode, and the cache
publication path where an unchanged stored hash is compared and left alone
while a changed one is written back. The second pair is what keeps a Scan from
paying a physical attribute write per file.
Feeds the exactly-one-callback test through four batches with a recording
source, so the stop assertions cover the items the run actually produced
instead of depending on how far the index raced ahead of the cancel, and
anchors the TESTING.md filter that selects every signature cache test.
Replaces the streaming request/result API with caller-owned items and
consolidates the content options, following the Job execution performance
design.
- `Item`, `BatchSource` and `Run` replace `StreamRequest`, `StreamResult`,
`StreamSource`, `StreamSink` and `RunStream`. Every accepted item receives
exactly one terminal callback from a single goroutine and without doing
I/O; an item ACP cannot process reports `Failed`, and a completion carries
one outcome per requested target in request order with its error identity
preserved. A graceful stop stops feeding, finishes the items in flight and
reports the rest, and a hard stop abandons per-item accounting.
- One `HashPolicy` replaces `WithHash`, `WithSignatureCache` and
`ForceRehash`. A stored hash is read whenever the policy uses the cache,
reused only where the policy allows it, and refreshed only when it differs.
- A source-only `WithReadMode` selects buffered (the default) or mapped
reads, and buffered sources open with `O_NOATIME` where the platform has
it. `LinearDevice` keeps request ordering, append targets and abort-on-error
but no longer selects the read mode.
- `Overwrite` becomes a target-device option, `WithReadBuffer` sets the read
buffer, and `SelectFiles` returns the entries the command line copies,
replacing `AccurateJob`, `WildcardJob` and the walk options.
- Copy consumers share one reference-counted chunk buffer each from a common
pool, and the per-item write goroutines finish before the result channel
closes, so no finished item can be dropped.
The command line tools, the documentation and the test suite move with the
API, including semantic tests for the callback contract, the stop and drain
paths, the hash policy matrix and the read modes.
The writer no longer re-derives or revalidates live size and mtime around
the xattr write. The reader already compares the stored snapshot against
the file and treats a mismatch as stale, so an entry queued for a version
that has since changed can never read back as a hit, while rewriting it
from live metadata would bind a hash to a version it was not computed for.
The regression test seeds a decoy attribute, asserts that the queued
snapshot lands unchanged, and asserts that the reader reports a miss for a
file whose metadata moved on.
Concurrent source preparation can no longer reorder writes to a linear
target. Preparation reports one outcome per indexed request; the ordering
stage buffers later outcomes until the request's turn and only then
transfers the reader, so a source waits for its consumer after the
ordering stage has taken it. Skipped and failed requests advance the order
without a write, so one unresolvable source no longer stalls the requests
behind it. Random targets keep completion-order concurrency.
Samuel Cui
Samuel CUI
Samuel N Cui