* storage: make vacuum/compaction commit crash-safe with a durable .cpc marker
A crash mid-compaction-commit could lose or corrupt volume data. The
two-rename commit (.cpd->.dat, .cpx->.idx) was not atomic, fsync results
were discarded before renaming over a healthy .dat, a stale .ldb could
poison the needle map, and a duplicate/late commit could delete the live
.dat/.idx outright.
Introduce a durable .cpc commit marker so the swap is atomic across a
crash:
- CommitCompact writes and fsyncs the .cpc marker after makeupDiff
fsyncs the .cpd/.cpx, then runs applyCompactSwap: an existence-guarded
rename of .cpd->.dat and .cpx->.idx, a directory fsync, removal of the
stale .ldb/.rdb, and finally removal of the marker.
- reconcileCompactState recovers an interrupted commit on load: roll
forward (finish the renames) when the marker is present, roll back
(delete the orphan .cpd/.cpx) when it is absent. It runs from a
directory pre-pass keyed on .cpd/.cpc existence, since the per-volume
loader is keyed on .idx/.vif and misses the marker-only and
already-renamed-.idx states.
- applyCompactSwap verifies BOTH .cpd and .cpx exist before touching the
live files, so a stale-state commit (including the Windows
RemoveAll-then-rename path) errors without deleting anything.
- Error-check the fsyncs that gate the swap: the .cpd close-fsync and
.cpx fsync in copyDataBasedOnIndexFile, the makeupDiff .idx fsync, and
MemDb.SaveToIdx.
- generateLevelDbFile rebuilds from offset 0 when the stored watermark
sits past the end of the .idx, instead of replaying zero entries and
poisoning the needle map.
- removeVolumeFiles and cleanupCompact sweep the .cpc marker; cleanup
refuses to unlink the temp files while a marker is present.
Mirror the commit-marker, fsync-before-rename, guard, and
load/reconcile logic in the Rust volume server.
* storage: don't reconcile an already-loaded volume's compaction state on reload
reconcileCompactStates runs in loadExistingVolumes, which is re-invoked at
runtime on SIGHUP (Store.LoadNewVolumes). For a volume that is already loaded
and mid-vacuum, its .cpd/.cpx are live temp files, not crash leftovers --
rolling them back would clobber the in-flight compaction (and remove a live
.ldb out from under an open handle). Skip any vid already present in the
volume map; genuine startup recovery runs before any volume is loaded, so the
map is empty then. Mirrored in the Rust volume server.
Also drop the .note keepVif change that crept into this branch; it belongs to
the replica-copy/verify workstream and is restored to master's behavior here
so the two changes don't collide.
* storage: roll a compaction commit forward per-file, not all-or-nothing
A crash after the .cpd->.dat rename but before .cpx->.idx leaves .cpd gone,
.cpx and .cpc present, and a stale .idx. The roll-forward required BOTH temp
files, so it skipped the swap and cleared the marker, pairing the fresh .dat
with the stale .idx (index corruption). Finish whichever temp file remains:
extract finishCompactSwap to rename .cpd->.dat and/or .cpx->.idx independently;
applyCompactSwap keeps the both-present guard for the normal commit. Existence
in the Rust mirror is checked robustly so a transient error never skips the swap.
* seaweed-volume: propagate directory fsync failures on the compaction commit path
fsync_dir dropped every sync_all error, so the commit could proceed with an
undurable marker or rename and a later restart could recover the wrong
generation. Return the error and check it at the commit call sites (marker write
and the swap), matching the Go fsyncDir which already propagates. Directory
fsync stays a no-op on Windows, where it is unsupported.
* storage: overflow-safe stale-watermark check when rebuilding the leveldb index
watermark*NeedleMapEntrySize can overflow uint64 for a corrupted watermark and
wrap below the file size, defeating the stale-.ldb guard. Compare in entries
(watermark > size/NeedleMapEntrySize) instead, which is equivalent and cannot
overflow. LevelDb-backed needle map is Go-only; no Rust mirror.
* storage: propagate idxFile.Close error when writing the compacted index
SaveToIdx writes the .cpx that is renamed to .idx at commit; a discarded Close
error (buffered data not flushed) could leave a partially-written index after a
crash. Surface it in the same durability gate as the fsync.
Fixed critical race condition in CompactMap where Set(), Delete(), and
Get() methods had issues with concurrent map access.
Root cause: segmentForKey() can create new map segments, which modifies
the cm.segments map. Calling this under a read lock caused concurrent
map write panics when multiple goroutines accessed the map simultaneously
(e.g., during VolumeNeedleStatus gRPC calls).
Changes:
- Set() method: Changed RLock/RUnlock to Lock/Unlock
- Delete() method: Changed RLock/RUnlock to Lock/Unlock, optimized to
avoid creating empty segments when key doesn't exist
- Get() method: Removed segmentForKey() call to avoid race condition,
now checks segment existence directly and returns early if segment
doesn't exist (optimization: avoids unnecessary segment creation)
This fix resolves the runtime/maps.fatal panic that occurred under
concurrent load.
Tested with race detector: go test -v -race ./weed/storage/needle_map/...
This is a follow-up fix to PR #7332 which partially addressed the issue.
The problem is that size=0 needles are in a gray area:
- IsValid() returns false for size=0 (because size must be > 0)
- IsDeleted() returns false for size=0 (because size must be < 0 or == TombstoneFileSize)
PR #7332 only fixed 2 places, but several other places still had the same bug:
1. needle_map_memory.go:doLoading - line 43 still used oldSize.IsValid()
2. needle_map_memory.go:DoOffsetLoading - used during vacuum and incremental loading
3. needle_map_leveldb.go:generateLevelDbFile - used when generating LevelDB needle maps
4. needle_map_leveldb.go:DoOffsetLoading - used during incremental loading for LevelDB
5. needle_map/compact_map.go:delete - couldn't delete size=0 entries because:
- The condition 'size > 0' failed for size=0
- Even if it passed, negating 0 gives 0 (not marking as deleted)
Changes:
- Changed size.IsValid() to !size.IsDeleted() in doLoading and DoOffsetLoading functions
- Fixed compact_map delete to use TombstoneFileSize for size=0 entries
Fixes#7293
* Rework `needle_map.CompactMap()` to maximize memory efficiency.
* Use a memory-efficient structure for `CompactMap` needle value entries.
This slightly complicates the code, but makes a **massive** difference
in memory efficiency - preliminary results show a ~30% reduction in
heap usage, with no measurable performance impact otherwise.
* Clean up type for `CompactMap` chunk IDs.
* Add a small comment description for `CompactMap()`.
* Add the old version of `CompactMap()` for comparison purposes.
Per-entry memory usage is based on `TotalAllocs`, which is incorrect - that
value is a cummulative of heap usage, which doesn't decrease when objects
are freeed.
`Allocs` is instead an accurate represeentation of actual memory usage
at the time metrics are reported.
Chris LuandGitHub
Chris Lu
Lisandro PinandGitHub
SmsS4andChris Lu
Konstantin LebedevandGitHub
guol-fnst