* fix(scrub): don't flag offset-0 logical tombstones in volume scrub
A remote-tier delete records a tombstone at .idx offset 0 with no physical .dat
bytes. Full scrub double-flagged a healthy remote-tiered volume with deletes:
scrubVolumeData counted the tombstone's GetActualSize(-1)=32 toward totalRead
(want > physical .dat), and CheckIndexFile treated it as occupying [0,31] and
flagged the first live needle as overlapping. Skip offset-0 logical tombstones
from both the size reconcile and the overlap check; they are still counted for
the index-size check. Local deletes (offset != 0) are unaffected.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(scrub): mirror offset-0 logical tombstone handling into Rust
Same fix as the Go volume_checking.go + idx/check.go change: Volume::scrub skips
offset-0 logical tombstones from total_read, and check_index_file excludes them
from the overlap check (still counted for the index-size check).
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(volume): verify the .dat-tail needle in the integrity check
CheckVolumeDataIntegrity checked the last entry by file position in the .idx
and, for a live needle, flipped the volume read-only when fileSize > fileTailOffset.
That entry is the .dat tail only when the .idx is in append order; a key-sorted
.idx (weed fix and other rebuilds listed entries by key) puts the highest-key
needle last, whose tail sits mid-file, so healthy volumes went read-only on every
load and re-running weed fix only reproduced the sorted index.
Locate the needle at the maximum offset — the one physically last in the .dat —
and verify the .dat ends exactly at it, regardless of .idx ordering. The
append-ordered common case stays O(1) (the last entry's on-disk end matches the
.dat size); only a key-sorted index pays a single linear scan. Deletion
tombstones at the tail are now verified too, instead of skipping the file-size
check.
* fix(command): weed fix rebuilds the .idx in .dat offset order
SaveToIdx wrote entries via AscendingVisit — sorted by key, the .sdx/.ecx shape
— so the rebuilt .idx put the highest-key needle last instead of the .dat-tail
needle, and dropped tombstones whose live needle was gone. Collect the live and
deleted entries, sort by .dat offset, and write them in append order so the .idx
stays a faithful log whose last entry is the real .dat tail.
* fix(volume): pass on-disk tombstone size to ReadData in verifyDeletedNeedleIntegrity
verifyDeletedNeedleIntegrity was forwarding TombstoneFileSize (-1) into
Needle.ReadData. A deletion tombstone is appended to .dat with DataSize=0
so the on-disk needle header carries Size=0; TombstoneFileSize is only
the .idx sentinel for "this entry is deleted" and is never written into
a needle header.
ReadBytes' size check therefore mismatched on every tombstone
(-1 != 0), returned ErrorSizeMismatch, and triggered the
4-byte-offset wrap-around retry in ReadData (offset + 32 GB). On any
volume large enough that offset+32 GB exceeds dat fileSize the retry
read EOF, CheckVolumeDataIntegrity reported corruption, and the loader
set noWriteOrDelete = true. Every volume whose last 10 .idx entries
included a deletion went read-only on startup — i.e. any healthy
volume where the most recent operations included a delete.
Pass Size(0) so the size check matches the on-disk tombstone header.
Add a regression test that writes three needles, deletes one, and
asserts CheckVolumeDataIntegrity succeeds with a tombstone at the .idx
tail. Without this fix the test reproduces the exact log shape from
the bug report:
read 0 dataSize 32 offset <orig+32GB> fileSize <much smaller>: EOF
verifyDeletedNeedleIntegrity ...idx failed: read data [N,N+32) : EOF
The Rust port guards its integrity-check size comparison with
!size.is_deleted() (seaweed-volume/src/storage/volume.rs) and never
hits this path, so no Rust mirror change is needed.
* test(seaweed-volume): mirror Go regression for deletion-tombstone integrity
The Rust integrity check already guards its size-mismatch comparison
with !size.is_deleted() (volume.rs:1859) and reads tombstone AppendAtNs
with body_size=0, so the Go regression fixed in the previous commit
does not apply. Lock that guarantee in with a parallel reload test:
write three needles, delete one, sync, reopen via Volume::new, assert
the volume is not flipped read-only.
Catches any future change that removes the deleted-entry guard or
re-introduces a size-strict path in check_volume_data_integrity for
tombstones.
* fix(volume): propagate io.EOF and ErrorSizeMismatch from verifyDeletedNeedleIntegrity
CheckVolumeDataIntegrity relies on identity comparison against io.EOF
and ErrorSizeMismatch to walk back through the last ten .idx entries
and tolerate a partial truncation at the tail (the "fix and continue"
loop). The live-needle branch in doCheckAndFixVolumeData already
returns those sentinels unwrapped; the deletion branch wrapped them
in fmt.Errorf, so a genuine .dat truncation past a tombstone offset
broke the recovery and flipped the volume read-only.
Mirror the live-needle handling: both verifyDeletedNeedleIntegrity
and doCheckAndFixVolumeData now short-circuit on io.EOF /
ErrorSizeMismatch and pass them through unwrapped. Other errors keep
their existing context wrapping.
Also tighten the regression test to capture lastAppendAtNs and assert
it's non-zero, so a future regression that skips the tombstone body
(and therefore never populates AppendAtNs) is caught even when the
err check still passes.
* fix(volume): keep vacuum running past dangling .idx entries
Vacuum compaction aborted entirely on the first .idx entry whose offset
pointed past the end of the .dat file, surfacing as `cannot hydrate
needle from file: EOF` and stalling progress on every other volume.
In both Go and Rust:
- During compaction, skip an unreadable needle and continue. The bytes
it pointed at were already unreachable via reads, so dropping the
index reference makes the post-vacuum volume consistent. Real EIO
still bails out so a disk fault is not silently papered over.
- At volume load, do a single linear scan of the .idx and confirm
every (offset + actual size) fits inside .dat. The pre-existing
integrity check only looked at the last 10 entries, so deeper
corruption (e.g. left over from a crashed batched write) went
undetected and only surfaced later as a vacuum EOF. A failure now
marks the volume read-only at load time so an operator can react.
Refs #8928
* fix(volume): only skip permanent-corruption needle reads during vacuum
Address PR review feedback (gemini-code-assist + coderabbit):
The original patch skipped any non-EIO read failure, which would silently
drop needles on transient errors — Windows hardware bad-sector errors
(ERROR_CRC etc.) never surface as syscall.EIO; tiered-storage network
timeouts and EROFS would also slip through and shrink the volume.
Switch to an explicit whitelist of permanent-corruption shapes:
- Add needle.ErrorCorrupted sentinel and wrap CRC and "index out of
range" errors with %w so callers can match via errors.Is.
- copyDataBasedOnIndexFile now skips only when the read failure is
io.EOF, io.ErrUnexpectedEOF, ErrorSizeMismatch, ErrorSizeInvalid,
or ErrorCorrupted. Anything else (real disk faults, environmental
errors, Windows hardware codes) aborts the compaction so an
operator notices.
- Mirror the same whitelist in the Rust volume server, matching on
io::ErrorKind::UnexpectedEof and the NeedleError corruption variants
(SizeMismatch, CrcMismatch, IndexOutOfRange, TailTooShort).
Also add `defer v.Close()` in TestVerifyIndexFitsInDat so Windows
t.TempDir() cleanup can release the .dat/.idx handles.
Refs #8928
* fix(volume): wrap entry-not-found size-mismatch with ErrorSizeMismatch
Address PR review: the fallback branch in ReadBytes returned an
unwrapped fmt.Errorf, so isSkippableNeedleReadError (and any caller
using errors.Is(..., ErrorSizeMismatch)) could not match it. Wrap
with %w so the whitelist applies, while leaving the existing direct
sentinel return for the OffsetSize==4 / offset<MaxPossibleVolumeSize
retry path unchanged so ReadData's `err == ErrorSizeMismatch` retry
still triggers.
Refs #8928
* fix(volume): integrate dangling-idx check into existing index load walk
Address PR review (gemini-code-assist, medium): the structural .idx
check used to do a second linear scan of the index file at every volume
load, doubling the disk-I/O cost on servers managing many volumes.
Track the largest (offset + actual size) seen during the existing
needle-map load walks (`LoadCompactNeedleMap`, `NewLevelDbNeedleMap`,
`NewSortedFileNeedleMap`'s `newNeedleMapMetricFromIndexFile`,
`DoOffsetLoading`) on a new `MaximumNeedleEnd` field on `mapMetric`,
exposed as `MaxNeedleEnd()` on the NeedleMapper interface.
`volume.load()` then compares `nm.MaxNeedleEnd()` to the .dat size
after the load is complete — pure numeric comparison, no extra I/O.
The standalone `verifyIndexFitsInDat` helper and its caller in
`CheckVolumeDataIntegrity` are removed; the test that used to drive
the helper directly now exercises the new path via
`LoadCompactNeedleMap`.
Mirror the same change in the Rust volume server: track
`max_needle_end` on `NeedleMapMetric`, expose via `max_needle_end()`
on `CompactNeedleMap`, `RedbNeedleMap`, and the `NeedleMap` enum.
The Rust load walk already happens in `load_from_idx` for both map
kinds, so the structural check becomes free.
Refs #8928
Chris LuandGitHub
Lisandro PinandGitHub