mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-07-19 22:42:20 +00:00
Chris Lu
339a597e7e
* storage: make vacuum/compaction commit crash-safe with a durable .cpc marker A crash mid-compaction-commit could lose or corrupt volume data. The two-rename commit (.cpd->.dat, .cpx->.idx) was not atomic, fsync results were discarded before renaming over a healthy .dat, a stale .ldb could poison the needle map, and a duplicate/late commit could delete the live .dat/.idx outright. Introduce a durable .cpc commit marker so the swap is atomic across a crash: - CommitCompact writes and fsyncs the .cpc marker after makeupDiff fsyncs the .cpd/.cpx, then runs applyCompactSwap: an existence-guarded rename of .cpd->.dat and .cpx->.idx, a directory fsync, removal of the stale .ldb/.rdb, and finally removal of the marker. - reconcileCompactState recovers an interrupted commit on load: roll forward (finish the renames) when the marker is present, roll back (delete the orphan .cpd/.cpx) when it is absent. It runs from a directory pre-pass keyed on .cpd/.cpc existence, since the per-volume loader is keyed on .idx/.vif and misses the marker-only and already-renamed-.idx states. - applyCompactSwap verifies BOTH .cpd and .cpx exist before touching the live files, so a stale-state commit (including the Windows RemoveAll-then-rename path) errors without deleting anything. - Error-check the fsyncs that gate the swap: the .cpd close-fsync and .cpx fsync in copyDataBasedOnIndexFile, the makeupDiff .idx fsync, and MemDb.SaveToIdx. - generateLevelDbFile rebuilds from offset 0 when the stored watermark sits past the end of the .idx, instead of replaying zero entries and poisoning the needle map. - removeVolumeFiles and cleanupCompact sweep the .cpc marker; cleanup refuses to unlink the temp files while a marker is present. Mirror the commit-marker, fsync-before-rename, guard, and load/reconcile logic in the Rust volume server. * storage: don't reconcile an already-loaded volume's compaction state on reload reconcileCompactStates runs in loadExistingVolumes, which is re-invoked at runtime on SIGHUP (Store.LoadNewVolumes). For a volume that is already loaded and mid-vacuum, its .cpd/.cpx are live temp files, not crash leftovers -- rolling them back would clobber the in-flight compaction (and remove a live .ldb out from under an open handle). Skip any vid already present in the volume map; genuine startup recovery runs before any volume is loaded, so the map is empty then. Mirrored in the Rust volume server. Also drop the .note keepVif change that crept into this branch; it belongs to the replica-copy/verify workstream and is restored to master's behavior here so the two changes don't collide. * storage: roll a compaction commit forward per-file, not all-or-nothing A crash after the .cpd->.dat rename but before .cpx->.idx leaves .cpd gone, .cpx and .cpc present, and a stale .idx. The roll-forward required BOTH temp files, so it skipped the swap and cleared the marker, pairing the fresh .dat with the stale .idx (index corruption). Finish whichever temp file remains: extract finishCompactSwap to rename .cpd->.dat and/or .cpx->.idx independently; applyCompactSwap keeps the both-present guard for the normal commit. Existence in the Rust mirror is checked robustly so a transient error never skips the swap. * seaweed-volume: propagate directory fsync failures on the compaction commit path fsync_dir dropped every sync_all error, so the commit could proceed with an undurable marker or rename and a later restart could recover the wrong generation. Return the error and check it at the commit call sites (marker write and the swap), matching the Go fsyncDir which already propagates. Directory fsync stays a no-op on Windows, where it is unsupported. * storage: overflow-safe stale-watermark check when rebuilding the leveldb index watermark*NeedleMapEntrySize can overflow uint64 for a corrupted watermark and wrap below the file size, defeating the stale-.ldb guard. Compare in entries (watermark > size/NeedleMapEntrySize) instead, which is equivalent and cannot overflow. LevelDb-backed needle map is Go-only; no Rust mirror. * storage: propagate idxFile.Close error when writing the compacted index SaveToIdx writes the .cpx that is renamed to .idx at commit; a discarded Close error (buffered data not flushed) could leave a partially-written index after a crash. Surface it in the same durability gate as the fsync.