mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-20 22:27:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(seaweed-volume): cross-disk EC shard reconciliation (#9212) (#9252)

Browse Source 
* fix(seaweed-volume): fall back to idx dir when reading .vif

EcVolume::new and read_ec_shard_config only looked for .vif at the
data dir. With the cross-disk reconcile path (where shards live on
one disk and .ecx / .ecj / .vif live on a sibling disk —
seaweedfs/seaweedfs#9212 / #9244), this would either write a stub
.vif on the shard disk and lose the real EC config + dat_file_size
or fall back to default ratios despite a perfectly good .vif being
present elsewhere on the same volume server.

Add a small `locate_vif_path` helper that prefers the data dir and
falls back to the idx dir when it differs, and thread the data dir
+ idx dir pair through `read_ec_shard_config`. Three call sites in
grpc_server.rs (VolumeEcShardsGenerate, VolumeEcShardsRebuild, scrub)
updated; the scrub path passes the same dir for both args because
`find_ec_dir` is the only locator there.

* feat(seaweed-volume): primitives for cross-disk EC shard reconcile

Adds the three small helpers the reconcile pass needs:

- DiskLocation::mount_ec_shards_with_idx_dir — mounts shards on this
  disk while pointing the EcVolume at a sibling disk's idx dir for
  .ecx / .ecj / .vif. Mirrors loadEcShardsWithIdxDir in
  weed/storage/disk_location_ec.go. The existing mount_ec_shards is
  kept as a thin wrapper over it.

- EcVolume::has_shard — `pub` accessor over the internal Vec<Option>
  shard slot so the reconcile pass can skip shards that are already
  registered.

- pub(crate) re-exports of parse_collection_volume_id and
  parse_ec_shard_extension under names parse_collection_volume_id_pub
  and is_ec_shard_extension so the reconcile module can call them
  without re-implementing the parsers.

No behaviour change. Reconciliation logic in the next commit.

* feat(seaweed-volume): cross-disk EC shard reconciliation (#9212)

Closes the loader half of seaweedfs/seaweedfs#9212 on the Rust side,
mirroring the Go fix in seaweedfs/seaweedfs#9244. With the auto-load
in feat/rust-load-all-ec-shards-9212 in place, the only remaining gap
is shards that landed on a disk without their `.ecx` — for example
when ec.balance / ec.rebuild moved them onto a destination node's
second disk while leaving the index files on the disk that already
held the volume. Without this, those orphan shards stay invisible to
the master and ec.rebuild reports the volume as unrepairable.

After every DiskLocation has finished its per-disk EC scan, sweep the
store for shards that live on a disk without local index files and
load them by reaching across to a sibling disk's `.ecx` / `.ecj` /
`.vif`:

  - Store::reconcile_ec_shards_across_disks walks each disk for
    orphan `.ec??` files (present on disk, not yet registered to an
    EcVolume) and matches them against an `(collection, vid) ->
    EcxOwnerInfo` map of which disk owns each `.ecx`.
  - Each matched group is mounted on its physical disk's ec_volumes
    map (so heartbeat reporting carries the right disk_id per shard)
    via `mount_ec_shards_with_idx_dir`, pointing the EcVolume at the
    sibling's idx dir.
  - `index_ecx_owners` records the directory each `.ecx` was found in
    (IdxDirectory or Directory) so the loader doesn't ENOENT when the
    legacy "written before -dir.idx was set" layout puts `.ecx` in
    the data dir. This mirrors the PR #9244 review fix from
    @gemini-code-assist / @coderabbitai (see Go commit af57cc652).
  - True orphans (no `.ecx` anywhere on this server) log a warning
    and stay on disk untouched — operator can restore the index later.

Wired into Store::add_location and Store::load_new_volumes so a fresh
restart and any later disk additions both pick up cross-disk shards.

Tests cover all four behaviour shapes:
- shards on dir0 + .ecx on dir1 → reconciled to dir0's ec_volumes
- .ecx in owner's data dir (legacy layout) → reconciled correctly
- self-contained disks → reconcile is a no-op
- truly-orphan shards (no .ecx anywhere) → left on disk, logged

* fix(seaweed-volume): propagate EcVolume::new errors instead of unwrap

mount_ec_shards_with_idx_dir built the missing EcVolume inside an
entry().or_insert_with() closure, which can't return a Result — so
any EcVolume::new failure (e.g. .ecx open error, .ecj create error,
malformed .vif) panicked the volume server via unwrap(). The
constructor already returns Result<>, so propagate it as
VolumeError::Io instead.

Reported in PR #9252 review by @gemini-code-assist (high) and
@coderabbitai (critical).

* perf(seaweed-volume): use DirEntry::metadata in collect_orphan_ec_shards

Replaced the extra fs::metadata(&path) lookup with ent.metadata() so
we don't pay an additional stat syscall per directory entry beyond
what read_dir already returned. Drops the now-unused std::path::Path
import alongside.

Reported in PR #9252 review by @gemini-code-assist.

* fix(seaweed-volume): scrub uses EcVolume's real dir_idx for split-disk volumes

After cross-disk reconciliation an EcVolume can legitimately have
ecv.dir != ecv.dir_idx (shards on one disk, .ecx / .ecj / .vif on a
sibling). The scrub path collapsed both args to find_ec_dir's single
answer, so read_ec_shard_config fell back to the wrong .vif location
for exactly the split-disk layout this PR loads — skewing
shard-count detection and verification results.

Use ecv.dir / ecv.dir_idx directly so scrub reads the metadata from
where the volume's index files actually live.

Reported in PR #9252 review by @coderabbitai.

* feat(seaweed-volume): primitives for split-disk EC volume operations

Reconciliation can mount the same `vid` on multiple DiskLocations
with disjoint shard subsets. The existing first-match `find_ec_volume`
isn't enough for read/unmount/delete/decode paths that need to act on
a specific shard or aggregate across the whole volume — they have to
walk every location and find the right home for each shard.

Add the small Store-level lookup primitives Go's findEcShard /
CollectEcShards already provide:

- `Store::find_ec_shard_location(vid, shard_id)` — returns the index
  of the location that has `(vid, shard_id)` mounted, if any.
- `Store::find_ec_volume_with_shard(vid, shard_id)` — same idea but
  returns the EcVolume directly.
- `Store::collect_ec_shard_dirs(vid, max_shard_count)` — returns
  the EcVolume to use for metadata plus per-shard data dirs (None
  when the shard isn't mounted on any disk). Mirrors
  `Store.CollectEcShards` in `weed/storage/store_ec.go`.

And the EcVolume accessors callers need:

- `EcVolume::has_shard(shard_id)` — was already added for the cross-
  disk reconcile but is now a load-bearing primitive for placement
  decisions on a per-shard basis. Pulled into the dedicated commit.
- `EcVolume::ecx_actual_dir()` — exposes the directory the `.ecx`
  was actually opened from. The decoder needs it for the .ecx
  lookup when shards are split across data dirs and `.ecx` lives on
  a sibling idx dir.

Plus a small defensive change to `DiskLocation::unmount_ec_shards`:
only decrement the per-shard gauge for shards that were actually
mounted. Without this, the upcoming `Store::unmount_ec_shards`
fan-out to every location would underflow the metric whenever a
shard is requested for unmount on a sibling disk that doesn't have
it.

No behaviour change at the call sites yet — wiring follows in the
next commits.

* fix(seaweed-volume): unmount_ec_shards visits every location with the vid

Store::unmount_ec_shards and Store::unmount_ec_shard returned after
the first DiskLocation with the volume id, even if that location did
not contain the requested shard. With reconciled split-disk volumes
(shards 0/12 on disk 0, shard 1 on disk 1 — the issue #9212 layout
this PR loads), VolumeEcShardsUnmount for a later-disk shard became a
silent no-op and Store::delete_ec_shards could remove the shard file
while leaving an in-memory shard + open file handle stale on the
later location.

Walk all locations that have the EcVolume and ask each to unmount
whatever subset of `shard_ids` it actually has — the
`DiskLocation::unmount_ec_shards` defensive guard from the previous
commit makes the fan-out safe (no metric underflow when a sibling
disk is asked to unmount a shard it doesn't hold).

* fix(seaweed-volume): VolumeEcShardRead reads from the shard's home disk

VolumeEcShardRead resolved the EcVolume via first-match
`find_ec_volume(vid)` and then looked up the requested shard on that
single EcVolume. With reconciled split-disk volumes (the layout
seaweedfs/seaweedfs#9212 produces — shards 0/12 on disk 0, shard 1
on disk 1), a request for shard 1 hit disk 0 first and returned
"shard 1 not mounted" even though it was happily mounted on disk 1.

Switch to `find_ec_volume_with_shard(vid, shard_id)` so the lookup
walks every location and returns the EcVolume whose disk actually
holds the shard. The deleted-needle check still works because every
per-disk EcVolume for the same vid points at the same `.ecx` file
(post-reconcile, both disks open the same sealed index).

* fix(seaweed-volume): VolumeEcShardsToVolume aggregates shards across disks

VolumeEcShardsToVolume resolved a single EcVolume via
`find_ec_volume(vid)` and then checked `ec_vol.shards[i]` for each
data shard. With reconciled split-disk volumes that's the wrong
view: the first-match EcVolume only carries the shards on its disk,
so the presence check would either reject the request as
"missing shard" or — if shards happened to be on the first disk —
fall through to `write_dat_file_from_shards(&dir, ...)` which only
reads from the EcVolume's single dir.

Mirror Go's CollectEcShards by aggregating per-shard data dirs
across every location with the volume:

- Add `Store::collect_ec_shard_dirs` (in the previous primitives
  commit) returning the EcVolume to use for metadata + per-shard
  dir slots.
- Extend `find_dat_file_size` and `write_dat_file_from_shards` with
  `_with_dirs` variants that take the `.ec00` dir and per-shard
  dirs separately, so a decoded volume whose shards live on
  several disks can still be reconstructed. The original signatures
  delegate to the new ones with the same dir for all shards, so
  every existing caller keeps working unchanged.
- Rewire VolumeEcShardsToVolume through the helpers — presence
  check sees the union, dat_file_size reads `.ec00` from the right
  disk and `.ecx` from the EcVolume's actual idx dir, the decoder
  reads each shard from its own home dir.

* test(seaweed-volume): split-disk read / unmount / delete / collect

Five tests exercising the four behaviour shapes the PR #9252 review
flagged on multi-location EC volumes. Each builds the cross-disk
split layout from issue #9212 (shards 0 and 12 on disk 0, shard 1 +
.ecx on disk 1) via the new `build_split_disk_store` helper and
asserts:

- `find_ec_shard_location` / `find_ec_volume_with_shard` route to
  the disk that actually holds each shard (not first-match).
- `Store::unmount_ec_shards([1])` reaches disk 1 and removes shard 1
  while leaving disk 0's unrelated shards mounted (used to be a
  silent no-op).
- `Store::unmount_ec_shard(vid, 1)` ditto for the single-shard
  variant.
- `Store::delete_ec_shards` removes both the on-disk file and the
  in-memory mount on the right disk; previously deletion could
  remove the file while the in-memory shard with its open file
  handle survived on a different location.
- `collect_ec_shard_dirs` reports the right per-shard data dir for
  each location and `None` for unmounted shards.

* fix(seaweed-volume): retry same-disk legacy .ecx layout in reconcile

The unconditional `owner.location == loc_idx` skip missed the layout
where `idx_directory` is configured but the owner's `.ecx` / `.ecj` /
`.vif` still live in `loc.directory` (the legacy "written before
-dir.idx was set" shape). In that case the per-disk loader's
mount_ec_shards used `loc.idx_directory` and ENOENT'd, then this
branch suppressed the only recovery path — the owner disk's own
shards stayed unloaded after startup.

Tighten the skip so it only fires when the discovered owner dir is
already `loc.idx_directory` (the loader-already-tried-and-failed
case). When `owner.idx_dir` differs (legacy data-dir layout), queue
a same-disk retry through `mount_ec_shards_with_idx_dir(...,
&owner.idx_dir)` so reconcile becomes the recovery path.

Reported in PR #9252 review by @coderabbitai.

* fix(seaweed-volume): roll back partial mounts on cross-disk reconcile failure

mount_ec_shards_with_idx_dir adds shards one at a time and
increments the `ec_shards` gauge per shard that successfully attaches.
A mid-loop failure (e.g. an EcVolumeShard::open error after the
first few shards already attached) used to leave the EcVolume
half-mounted with stale metric increments — the warn!() branch only
logged the error.

Mirror DiskLocation::handle_found_ecx_file's recovery path: drive
the cleanup through `loc.unmount_ec_shards(vid, &shard_ids)` after
a failed mount. The defensive change in #9251 makes
unmount_ec_shards only decrement the gauge for shards that were
actually mounted and drops the EcVolume when it reaches zero
shards, so the rollback is safe even though some of `shard_ids`
never attached.

Reported in PR #9252 review by @coderabbitai.

* test(seaweed-volume): cover the two reconcile fixes from PR #9252 review

Two new tests in store_ec_reconcile:

- test_reconcile_recovers_same_disk_legacy_ecx_layout — sets up the
  layout where idx_directory is configured but the owner's .ecx
  lives in loc.directory. The per-disk loader's mount_ec_shards
  uses loc.idx_directory and fails; reconcile should retry on the
  same disk with the owner's actual idx_dir and the owner's own
  shards must come back online.

- test_reconcile_rolls_back_partial_mounts_on_failure — sabotages
  one of the orphan shard files (replaces it with a directory of
  the same name) so EcVolumeShard::open errors out partway through
  mount_ec_shards_with_idx_dir. Asserts the post-condition that no
  EcVolume entry retains a "shard mounted" claim that doesn't
  correspond to a real shard file.
This commit is contained in:
Chris Lu
2026-04-27 19:01:30 -07:00
committed by GitHub
parent 49e83a26cb
commit 9d6d068f41
7 changed files with 1114 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
seaweed-volume/src/server/grpc_server.rs
+65 -31
View File
@@ -2016,7 +2016,9 @@ impl VolumeServer for VolumeGrpcService {
// Check existing .vif for EC shard config (matching Go's MaybeLoadVolumeInfo)
let (data_shards, parity_shards) =
crate::storage::erasure_coding::ec_volume::read_ec_shard_config(&dir, collection, vid);
crate::storage::erasure_coding::ec_volume::read_ec_shard_config(
&dir, &idx_dir, collection, vid,
);
if let Err(e) = crate::storage::erasure_coding::ec_encoder::write_ec_files(
&dir,
@@ -2170,6 +2172,7 @@ impl VolumeServer for VolumeGrpcService {
let (data_shards, parity_shards) =
crate::storage::erasure_coding::ec_volume::read_ec_shard_config(
&rebuild_dir,
&rebuild_idx_dir,
collection,
vid,
);
@@ -2571,12 +2574,19 @@ impl VolumeServer for VolumeGrpcService {
let vid = VolumeId(req.volume_id);
let store = self.state.store.read().unwrap();
let ec_vol = store.find_ec_volume(vid).ok_or_else(|| {
Status::not_found(format!(
"ec volume {} shard {} not found",
req.volume_id, req.shard_id
))
})?;
// Reconciled EC volumes can have their shards split across
// disks (e.g. shards 0/12 on disk 0, shard 1 on disk 1), so
// resolve the EcVolume from the *shard*'s home location
// rather than first-match `find_ec_volume(vid)` which would
// miss shards that live on a sibling. Mirrors Go's findEcShard.
let ec_vol = store
.find_ec_volume_with_shard(vid, req.shard_id)
.ok_or_else(|| {
Status::not_found(format!(
"ec volume {} shard {} not found",
req.volume_id, req.shard_id
))
})?;
// Check if the requested needle is deleted (via .ecx index, matching Go)
if req.file_key > 0 {
@@ -2595,7 +2605,8 @@ impl VolumeServer for VolumeGrpcService {
}
}
// Read from the shard
// Read from the shard. Guaranteed to be present because
// find_ec_volume_with_shard already verified it.
let shard = ec_vol
.shards
.get(req.shard_id as usize)
@@ -2685,8 +2696,13 @@ impl VolumeServer for VolumeGrpcService {
let vid = VolumeId(req.volume_id);
let store = self.state.store.read().unwrap();
let ec_vol = store
.find_ec_volume(vid)
// Aggregate per-shard data dirs across all locations so the
// shard-presence check + decoder both see the union for
// cross-disk reconciled volumes (#9252). Mirrors Go's
// CollectEcShards.
let max_shard_count = crate::storage::erasure_coding::ec_shard::MAX_SHARD_COUNT;
let (ec_vol, shard_dirs) = store
.collect_ec_shard_dirs(vid, max_shard_count)
.ok_or_else(|| Status::not_found(format!("ec volume {} not found", req.volume_id)))?;
if ec_vol.collection != req.collection {
@@ -2700,7 +2716,6 @@ impl VolumeServer for VolumeGrpcService {
let data_shards = ec_vol.data_shards as usize;
// Validate data shard count range (matches Go's VolumeEcShardsToVolume)
let max_shard_count = crate::storage::erasure_coding::ec_shard::MAX_SHARD_COUNT;
if data_shards == 0 || data_shards > max_shard_count {
return Err(Status::invalid_argument(format!(
"invalid data shard count {} for volume {} (must be 1..{})",
@@ -2708,14 +2723,9 @@ impl VolumeServer for VolumeGrpcService {
)));
}
// Check that all data shards are present
// Check that all data shards are present somewhere on this server.
for shard_id in 0..data_shards {
if ec_vol
.shards
.get(shard_id)
.map(|s| s.is_none())
.unwrap_or(true)
{
if shard_dirs[shard_id].is_none() {
return Err(Status::internal(format!(
"ec volume {} missing shard {}",
req.volume_id, shard_id
@@ -2747,29 +2757,46 @@ impl VolumeServer for VolumeGrpcService {
)));
}
// Reconstruct the volume from EC shards
let dir = ec_vol.dir.clone();
// Reconstruct the volume from EC shards. Use the EcVolume's
// own dir for the produced .dat (matches the volume's home
// disk) and its `ecx_actual_dir` for the .ecx lookup, while
// reading each shard from its real on-disk location.
let dat_dir = ec_vol.dir.clone();
let ecx_dir = ec_vol.ecx_actual_dir().to_string();
let collection = ec_vol.collection.clone();
// shard_dirs[i] is guaranteed Some for i in 0..data_shards by
// the check above; collect concrete dirs for the decoder.
let per_shard_dirs: Vec<String> = shard_dirs[..data_shards]
.iter()
.map(|d| d.clone().unwrap())
.collect();
drop(store);
// Calculate .dat file size from .ecx entries
// Calculate .dat file size from .ecx entries (.ec00 lives on
// its own disk, .ecx on the index disk).
let dat_file_size =
crate::storage::erasure_coding::ec_decoder::find_dat_file_size(&dir, &collection, vid)
.map_err(|e| Status::internal(format!("FindDatFileSize: {}", e)))?;
crate::storage::erasure_coding::ec_decoder::find_dat_file_size_with_dirs(
&per_shard_dirs[0],
&ecx_dir,
&collection,
vid,
)
.map_err(|e| Status::internal(format!("FindDatFileSize: {}", e)))?;
// Write .dat file using block-interleaved reading from shards
crate::storage::erasure_coding::ec_decoder::write_dat_file_from_shards(
&dir,
// Write .dat file using block-interleaved reading from shards.
crate::storage::erasure_coding::ec_decoder::write_dat_file_from_shards_with_dirs(
&dat_dir,
&collection,
vid,
dat_file_size,
data_shards,
&per_shard_dirs,
)
.map_err(|e| Status::internal(format!("WriteDatFile: {}", e)))?;
// Write .idx file from .ecx and .ecj files
// Write .idx file from .ecx and .ecj files (lives on idx dir).
crate::storage::erasure_coding::ec_decoder::write_idx_file_from_ec_index(
&dir,
&dat_dir,
&collection,
vid,
)
@@ -3477,9 +3504,15 @@ impl VolumeServer for VolumeGrpcService {
// LOCAL (2) / FULL (3): verify EC shard data
let files = ecv.walk_ecx_stats().map(|(f, _, _)| f).unwrap_or(0);
let dir = store
.find_ec_dir(*vid, &collection)
.unwrap_or_else(|| String::from(""));
// After cross-disk reconciliation, an EcVolume can
// legitimately have ecv.dir != ecv.dir_idx (shards
// on one disk, .ecx / .ecj / .vif on a sibling).
// Use the EcVolume's own dirs rather than collapsing
// both args to find_ec_dir's single answer, otherwise
// read_ec_shard_config falls back to the wrong .vif
// location for split-disk volumes (#9252).
let dir = ecv.dir.clone();
let idx_dir = ecv.dir_idx.clone();
if dir.is_empty() {
continue;
}
@@ -3489,6 +3522,7 @@ impl VolumeServer for VolumeGrpcService {
let (data_shards, parity_shards) =
crate::storage::erasure_coding::ec_volume::read_ec_shard_config(
&dir,
&idx_dir,
&collection,
*vid,
);
seaweed-volume/src/storage/disk_location.rs
+57 -3
View File
@@ -599,12 +599,43 @@ impl DiskLocation {
collection: &str,
shard_ids: &[u32],
) -> Result<(), VolumeError> {
let dir = self.directory.clone();
let idx_dir = self.idx_directory.clone();
self.mount_ec_shards_with_idx_dir(vid, collection, shard_ids, &idx_dir)
}
/// Mount EC shards but explicitly specify the idx directory the
/// EcVolume should pull `.ecx` / `.ecj` / `.vif` from.
///
/// Cross-disk reconcile (mirrors `loadEcShardsWithIdxDir` in
/// `weed/storage/disk_location_ec.go`): when a volume's shards live
/// on this disk but the index files were left on a sibling disk
/// (the seaweedfs/seaweedfs#9212 orphan-shard layout), the
/// reconciler creates the EcVolume here while pointing it at the
/// sibling's idx dir. Each shard still ends up registered in this
/// disk's `ec_volumes` map so heartbeat reporting carries the right
/// disk_id per shard.
pub fn mount_ec_shards_with_idx_dir(
&mut self,
vid: VolumeId,
collection: &str,
shard_ids: &[u32],
idx_dir: &str,
) -> Result<(), VolumeError> {
let dir = self.directory.clone();
// Avoid the entry().or_insert_with() pattern here: that closure
// can't return a Result, so any EcVolume::new failure (e.g.
// .ecx open error, .ecj create error, malformed .vif) would
// have to panic via unwrap(). Build the EcVolume up front and
// propagate the error to the caller.
if !self.ec_volumes.contains_key(&vid) {
let ec_vol = EcVolume::new(&dir, idx_dir, collection, vid)
.map_err(VolumeError::Io)?;
self.ec_volumes.insert(vid, ec_vol);
}
let ec_vol = self
.ec_volumes
.entry(vid)
.or_insert_with(|| EcVolume::new(&dir, &idx_dir, collection, vid).unwrap());
.get_mut(&vid)
.expect("just inserted above");
ec_vol.disk_type = self.disk_type.clone();
for &shard_id in shard_ids {
@@ -618,10 +649,19 @@ impl DiskLocation {
}
/// Unmount EC shards for a volume on this location.
///
/// Only shards that are actually mounted decrement the per-shard
/// metric — without this guard the gauge would underflow when the
/// caller passes a shard that lives on a sibling disk
/// (cross-disk reconcile makes that the common case for the same
/// `vid` after reconciliation).
pub fn unmount_ec_shards(&mut self, vid: VolumeId, shard_ids: &[u32]) {
if let Some(ec_vol) = self.ec_volumes.get_mut(&vid) {
let collection = ec_vol.collection.clone();
for &shard_id in shard_ids {
if !ec_vol.has_shard(shard_id as u8) {
continue;
}
ec_vol.remove_shard(shard_id as u8);
crate::metrics::VOLUME_GAUGE
.with_label_values(&[&collection, "ec_shards"])
@@ -887,6 +927,14 @@ fn calculate_expected_shard_size(dat_file_size: i64) -> i64 {
/// Mirrors `parseCollectionVolumeId` in
/// `weed/storage/disk_location.go`. Used when iterating raw filenames
/// where the extension has already been stripped.
///
/// `pub(crate)` companion `parse_collection_volume_id_pub` is exposed
/// so the cross-disk reconcile in `store_ec_reconcile.rs` can call it
/// without re-implementing the parser.
pub(crate) fn parse_collection_volume_id_pub(base: &str) -> Option<(String, VolumeId)> {
parse_collection_volume_id(base)
}
fn parse_collection_volume_id(base: &str) -> Option<(String, VolumeId)> {
if let Some(pos) = base.rfind('_') {
let collection = &base[..pos];
@@ -899,6 +947,12 @@ fn parse_collection_volume_id(base: &str) -> Option<(String, VolumeId)> {
}
}
/// `pub(crate)` re-export of [`parse_ec_shard_extension`] for the
/// cross-disk reconcile in `store_ec_reconcile.rs`.
pub(crate) fn is_ec_shard_extension(ext: &str) -> Option<u32> {
parse_ec_shard_extension(ext)
}
/// Recognise EC shard extensions `.ec00`–`.ec255` (the ShardId u8
/// range — see `EcVolumeShard`'s typed shard id) and return the
/// shard id. Returns `None` for any other extension.
seaweed-volume/src/storage/erasure_coding/ec_decoder.rs
+68 -7
View File
@@ -16,11 +16,28 @@ use crate::storage::volume::volume_file_name;
/// Calculate .dat file size from the max offset entry in .ecx.
/// Reads the volume version from the first EC shard (.ec00) superblock,
/// then scans .ecx entries to find the largest (offset + needle_actual_size).
///
/// `dir` is used both for reading `.ec00` and `.ecx`. For split-disk
/// reconciled volumes call [`find_dat_file_size_with_dirs`] instead.
pub fn find_dat_file_size(dir: &str, collection: &str, volume_id: VolumeId) -> io::Result<i64> {
let base = volume_file_name(dir, collection, volume_id);
find_dat_file_size_with_dirs(dir, dir, collection, volume_id)
}
/// Like [`find_dat_file_size`] but lets the caller pass separate dirs
/// for `.ec00` (the data shard) and `.ecx` (the sealed index). This
/// is the form needed when shards are split across data dirs and the
/// `.ecx` lives on a sibling disk's idx dir (#9252).
pub fn find_dat_file_size_with_dirs(
ec00_dir: &str,
ecx_dir: &str,
collection: &str,
volume_id: VolumeId,
) -> io::Result<i64> {
let ec00_base = volume_file_name(ec00_dir, collection, volume_id);
let ecx_base = volume_file_name(ecx_dir, collection, volume_id);
// Read volume version from .ec00 superblock
let ec00_path = format!("{}.ec00", base);
let ec00_path = format!("{}.ec00", ec00_base);
let mut ec00 = File::open(&ec00_path)?;
let mut sb_buf = [0u8; SUPER_BLOCK_SIZE];
ec00.read_exact(&mut sb_buf)?;
@@ -30,7 +47,7 @@ pub fn find_dat_file_size(dir: &str, collection: &str, volume_id: VolumeId) -> i
let mut dat_size: i64 = SUPER_BLOCK_SIZE as i64;
// Scan .ecx entries
let ecx_path = format!("{}.ecx", base);
let ecx_path = format!("{}.ecx", ecx_base);
let ecx_data = std::fs::read(&ecx_path)?;
let entry_count = ecx_data.len() / NEEDLE_MAP_ENTRY_SIZE;
@@ -52,7 +69,10 @@ pub fn find_dat_file_size(dir: &str, collection: &str, volume_id: VolumeId) -> i
/// Reconstruct a .dat file from EC data shards.
///
/// Reads from .ec00-.ec09 and writes a new .dat file.
/// Reads from .ec00-.ec09 and writes a new .dat file. All data shards
/// must live in `dir`. For the cross-disk reconciled layout where
/// shards are split across multiple data dirs of the same node, use
/// [`write_dat_file_from_shards_with_dirs`] instead.
pub fn write_dat_file_from_shards(
dir: &str,
collection: &str,
@@ -60,12 +80,53 @@ pub fn write_dat_file_from_shards(
dat_file_size: i64,
data_shards: usize,
) -> io::Result<()> {
let base = volume_file_name(dir, collection, volume_id);
let dirs: Vec<String> = (0..data_shards).map(|_| dir.to_string()).collect();
write_dat_file_from_shards_with_dirs(
dir,
collection,
volume_id,
dat_file_size,
data_shards,
&dirs,
)
}
/// Reconstruct a .dat file from EC data shards, taking the source
/// directory for each shard separately.
///
/// `dat_dir` is where the produced `.dat` is written. `shard_dirs[i]`
/// is the directory holding shard `i`. For the simple "all shards in
/// one dir" case both can be the same value.
///
/// Mirrors Go's `WriteDatFile(baseFileName, datFileSize,
/// shardFileNames)` shape — Go passes per-shard paths so a
/// reconciled volume with shards split across disks of the same
/// volume server can still be decoded back to a regular .dat
/// (seaweedfs/seaweedfs#9252).
pub fn write_dat_file_from_shards_with_dirs(
dat_dir: &str,
collection: &str,
volume_id: VolumeId,
dat_file_size: i64,
data_shards: usize,
shard_dirs: &[String],
) -> io::Result<()> {
if shard_dirs.len() < data_shards {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
format!(
"shard_dirs len {} < data_shards {}",
shard_dirs.len(),
data_shards
),
));
}
let base = volume_file_name(dat_dir, collection, volume_id);
let dat_path = format!("{}.dat", base);
// Open data shards
// Open data shards from their individual home dirs.
let mut shards: Vec<EcVolumeShard> = (0..data_shards as u8)
.map(|i| EcVolumeShard::new(dir, collection, volume_id, i))
.map(|i| EcVolumeShard::new(&shard_dirs[i as usize], collection, volume_id, i))
.collect();
for shard in &mut shards {
seaweed-volume/src/storage/erasure_coding/ec_volume.rs
+57 -7
View File
@@ -50,11 +50,42 @@ pub struct EcVolume {
pub expire_at_sec: u64,
}
pub fn read_ec_shard_config(dir: &str, collection: &str, volume_id: VolumeId) -> (u32, u32) {
/// Locate the `.vif` for a (collection, vid) by preferring the data dir
/// and falling back to the idx dir when it lives there instead. The
/// fallback covers the cross-disk reconcile path: when a volume's
/// shards live on one disk but its `.ecx` / `.ecj` / `.vif` live on a
/// sibling disk (seaweedfs/seaweedfs#9212 / #9244), we want to read the
/// real `.vif` from the sibling rather than write a stub on the shard
/// disk and lose the EC config + dat file size.
fn locate_vif_path(dir: &str, dir_idx: &str, collection: &str, volume_id: VolumeId) -> String {
let data_vif = format!(
"{}.vif",
crate::storage::volume::volume_file_name(dir, collection, volume_id),
);
if dir_idx != dir && !std::path::Path::new(&data_vif).exists() {
let idx_vif = format!(
"{}.vif",
crate::storage::volume::volume_file_name(dir_idx, collection, volume_id),
);
if std::path::Path::new(&idx_vif).exists() {
return idx_vif;
}
}
data_vif
}
/// Read EC data/parity shard counts from `.vif`, defaulting to the
/// build's standard ratio when no `.vif` is present or is malformed.
/// Looks at the data dir first, then the idx dir — see [`locate_vif_path`].
pub fn read_ec_shard_config(
dir: &str,
dir_idx: &str,
collection: &str,
volume_id: VolumeId,
) -> (u32, u32) {
let mut data_shards = crate::storage::erasure_coding::ec_shard::DATA_SHARDS_COUNT as u32;
let mut parity_shards = crate::storage::erasure_coding::ec_shard::PARITY_SHARDS_COUNT as u32;
let base = crate::storage::volume::volume_file_name(dir, collection, volume_id);
let vif_path = format!("{}.vif", base);
let vif_path = locate_vif_path(dir, dir_idx, collection, volume_id);
if let Ok(vif_content) = std::fs::read_to_string(&vif_path) {
if let Ok(vif_info) =
serde_json::from_str::<crate::storage::volume::VifVolumeInfo>(&vif_content)
@@ -81,7 +112,7 @@ impl EcVolume {
collection: &str,
volume_id: VolumeId,
) -> io::Result<Self> {
let (data_shards, parity_shards) = read_ec_shard_config(dir, collection, volume_id);
let (data_shards, parity_shards) = read_ec_shard_config(dir, dir_idx, collection, volume_id);
let total_shards = (data_shards + parity_shards) as usize;
let mut shards = Vec::with_capacity(total_shards);
@@ -89,10 +120,11 @@ impl EcVolume {
shards.push(None);
}
// Read expire_at_sec and version from .vif if present (matches Go's MaybeLoadVolumeInfo)
// Read expire_at_sec and version from .vif if present (matches Go's MaybeLoadVolumeInfo).
// Prefer the data dir; fall back to the idx dir for the
// cross-disk reconcile case (#9212 / #9244).
let (expire_at_sec, vif_version) = {
let base = crate::storage::volume::volume_file_name(dir, collection, volume_id);
let vif_path = format!("{}.vif", base);
let vif_path = locate_vif_path(dir, dir_idx, collection, volume_id);
if let Ok(vif_content) = std::fs::read_to_string(&vif_path) {
if let Ok(vif_info) =
serde_json::from_str::<crate::storage::volume::VifVolumeInfo>(&vif_content)
@@ -319,6 +351,24 @@ impl EcVolume {
self.shards.iter().filter(|s| s.is_some()).count()
}
/// Reports whether `shard_id` is currently registered to this
/// EcVolume (used by the cross-disk reconcile to skip already-
/// loaded shards).
pub fn has_shard(&self, shard_id: u8) -> bool {
self.shards
.get(shard_id as usize)
.map(|s| s.is_some())
.unwrap_or(false)
}
/// Directory where this EcVolume's `.ecx` was actually opened
/// (may differ from `dir_idx` when the legacy "written before
/// -dir.idx was set" fallback or the cross-disk reconcile path
/// pointed it elsewhere).
pub fn ecx_actual_dir(&self) -> &str {
&self.ecx_actual_dir
}
pub fn is_time_to_destroy(&self) -> bool {
self.expire_at_sec > 0
&& SystemTime::now()
seaweed-volume/src/storage/mod.rs
+1
View File
@@ -4,6 +4,7 @@ pub mod idx;
pub mod needle;
pub mod needle_map;
pub mod store;
pub mod store_ec_reconcile;
pub mod super_block;
pub mod types;
pub mod volume;
seaweed-volume/src/storage/store.rs
+92 -2
View File
@@ -84,6 +84,17 @@ impl Store {
}
self.locations.push(loc);
// After every disk has finished its per-disk EC scan, sweep
// the store for shards that live on a disk without local index
// files and load them by reaching across to a sibling disk's
// .ecx / .ecj / .vif (seaweedfs/seaweedfs#9212 / #9244).
// ec.balance / ec.rebuild can move shards onto a destination
// node's second disk while leaving the index on the disk that
// already held the volume; without this pass those orphan
// shards stay invisible to the master.
self.reconcile_ec_shards_across_disks();
Ok(())
}
@@ -95,6 +106,7 @@ impl Store {
tracing::error!("load_new_volumes error in {}: {}", loc.directory, e);
}
}
self.reconcile_ec_shards_across_disks();
}
// ---- Volume lookup ----
@@ -643,11 +655,18 @@ impl Store {
}
/// Unmount EC shards for a volume (batch).
///
/// Iterates every location that has an EcVolume for `vid` and
/// asks it to unmount whatever subset of `shard_ids` it actually
/// has — required for cross-disk reconciled volumes where the
/// requested shards may legitimately live on different disks of
/// the same store (#9252). DiskLocation::unmount_ec_shards
/// already skips shards that aren't mounted, so this is safe to
/// fan out blindly.
pub fn unmount_ec_shards(&mut self, vid: VolumeId, shard_ids: &[u32]) {
for loc in &mut self.locations {
if loc.has_ec_volume(vid) {
loc.unmount_ec_shards(vid, shard_ids);
return;
}
}
}
@@ -655,10 +674,12 @@ impl Store {
/// Unmount a single EC shard, searching all locations.
/// Matches Go's Store.UnmountEcShards which unmounts one shard at a time.
pub fn unmount_ec_shard(&mut self, vid: VolumeId, shard_id: u32) -> Result<(), VolumeError> {
// Walk all locations rather than stopping at the first with the
// vid — split-disk reconciled volumes can have the same vid on
// multiple disks, with the target shard on any of them.
for loc in &mut self.locations {
if loc.has_ec_volume(vid) {
loc.unmount_ec_shards(vid, &[shard_id]);
return Ok(());
}
}
// Go returns nil if shard not found (no error)
@@ -690,6 +711,75 @@ impl Store {
self.locations.iter().any(|loc| loc.has_ec_volume(vid))
}
/// Returns the index of the disk location that has `(vid, shard_id)`
/// mounted, if any. Mirrors Go's `Store.findEcShard` and is the
/// right primitive for read/unmount/delete operations on a single
/// shard, since reconciliation can mount the same `vid` on multiple
/// disks (each holding a disjoint subset of the shards). Without
/// this, callers using `find_ec_volume(vid)` would only see the
/// first disk and miss shards that live on a sibling.
pub fn find_ec_shard_location(&self, vid: VolumeId, shard_id: u32) -> Option<usize> {
for (i, loc) in self.locations.iter().enumerate() {
if let Some(ecv) = loc.find_ec_volume(vid) {
if ecv.has_shard(shard_id as u8) {
return Some(i);
}
}
}
None
}
/// Like [`Self::find_ec_shard_location`] but returns the EcVolume
/// reference directly. Borrows the store immutably for the
/// EcVolume's lifetime.
pub fn find_ec_volume_with_shard(
&self,
vid: VolumeId,
shard_id: u32,
) -> Option<&EcVolume> {
for loc in &self.locations {
if let Some(ecv) = loc.find_ec_volume(vid) {
if ecv.has_shard(shard_id as u8) {
return Some(ecv);
}
}
}
None
}
/// Aggregate the data dir of each mounted shard for `vid` across
/// all locations. Returns the EcVolume to use for metadata
/// (`.ecx`, `dir_idx`, etc — any per-disk EcVolume for this vid
/// will do, since they all open the same `.ecx`) and a vector of
/// per-shard data dirs (`None` when the shard isn't mounted on
/// any disk).
///
/// Mirrors Go's `Store.CollectEcShards` in
/// `weed/storage/store_ec.go`. The decoder needs per-shard paths
/// to support cross-disk reconciled volumes whose shards are
/// split across data dirs.
pub fn collect_ec_shard_dirs(
&self,
vid: VolumeId,
max_shard_count: usize,
) -> Option<(&EcVolume, Vec<Option<String>>)> {
let mut found_vol: Option<&EcVolume> = None;
let mut dirs: Vec<Option<String>> = vec![None; max_shard_count];
for loc in &self.locations {
if let Some(ecv) = loc.find_ec_volume(vid) {
if found_vol.is_none() {
found_vol = Some(ecv);
}
for shard_id in 0..max_shard_count {
if dirs[shard_id].is_none() && ecv.has_shard(shard_id as u8) {
dirs[shard_id] = Some(loc.directory.clone());
}
}
}
}
found_vol.map(|v| (v, dirs))
}
pub fn delete_expired_ec_volumes(
&mut self,
) -> (
seaweed-volume/src/storage/store_ec_reconcile.rs
+774
View File
@@ -0,0 +1,774 @@
//! Cross-disk EC shard reconciliation.
//!
//! Mirrors `weed/storage/store_ec_reconcile.go`. Loads EC shards that
//! the per-disk scan in `load_all_ec_shards` skipped because the disk
//! holding the `.ec??` files does not also hold the matching
//! `.ecx` / `.ecj` / `.vif` index files. The index files are located
//! on a different disk of the same volume server (seaweedfs/seaweedfs#9212).
//!
//! Per-disk `load_all_ec_shards` correctly leaves these orphan shards
//! on disk — it does not have visibility into other DiskLocations on
//! the same store — so the cross-disk fan-out happens here, after
//! every disk's initial pass has completed. We register each shard
//! against its physical disk's `ec_volumes` map (so heartbeat reporting
//! carries the right disk_id per shard) but point the EcVolume at the
//! sibling disk's index files so it can serve reads and route deletes
//! through a real `.ecx` / `.ecj`.
use std::collections::HashMap;
use std::fs;
use tracing::{info, warn};
use crate::storage::disk_location::{is_ec_shard_extension, parse_collection_volume_id_pub};
use crate::storage::store::Store;
use crate::storage::types::VolumeId;
/// Key for orphan-shard reconciliation: collection + volume id. Two
/// collections can re-use the same volume id, and we must only pair
/// shards with their own `.ecx`.
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
struct EcKey {
collection: String,
vid: VolumeId,
}
/// Records both the disk index that owns the `.ecx` and the actual
/// directory it lives in (`IdxDirectory` or `Directory`). The directory
/// matters because `index_ecx_owners` scans both — when `.ecx` lives in
/// `Directory` (the legacy "written before -dir.idx was set" layout
/// that `remove_ec_volume_files` keeps cleaning up), passing the
/// owner's `IdxDirectory` to `EcVolume::new` would ENOENT both the
/// primary and the same-disk fallback path. Tracking the actual scan
/// dir lets reconcile point loaders at the directory the `.ecx` is
/// really in.
#[derive(Clone, Debug)]
struct EcxOwnerInfo {
location: usize,
idx_dir: String,
}
impl Store {
/// Run cross-disk orphan-shard reconciliation. Should be called
/// after every DiskLocation has finished its per-disk EC scan.
///
/// Mirrors `Store.reconcileEcShardsAcrossDisks` in Go.
pub fn reconcile_ec_shards_across_disks(&mut self) {
if self.locations.len() < 2 {
return;
}
let owners = self.index_ecx_owners();
if owners.is_empty() {
return;
}
// Snapshot of orphan shards, keyed by (loc_idx, ec_key) so we
// can release the immutable borrow on self.locations before
// calling mount_ec_shards_with_idx_dir (which needs &mut).
let mut to_load: Vec<(usize, EcKey, Vec<(String, u32)>, EcxOwnerInfo)> = Vec::new();
for (loc_idx, loc) in self.locations.iter().enumerate() {
let orphans = collect_orphan_ec_shards(loc, loc_idx);
for (key, shards) in orphans {
let Some(owner) = owners.get(&key) else {
warn!(
volume_id = key.vid.0,
collection = %key.collection,
directory = %loc.directory,
"ec volume has shards on this disk without a matching .ecx anywhere on this volume server; shards {:?} will stay unloaded until the missing .ecx is restored",
shards.iter().map(|(n, _)| n.as_str()).collect::<Vec<_>>(),
);
continue;
};
if owner.location == loc_idx && owner.idx_dir == loc.idx_directory {
// Normal same-disk case: load_all_ec_shards already
// attempted the mount via `loc.idx_directory` and
// logged the underlying failure. No point retrying
// the same call.
continue;
}
// Either a cross-disk owner OR a same-disk owner whose
// `.ecx` actually lives in `loc.directory` (the legacy
// pre-`-dir.idx` layout). The latter wasn't tried by
// load_all_ec_shards, which only looked in
// `self.idx_directory`, so we still need to retry it
// here with the owner's discovered idx_dir.
to_load.push((loc_idx, key, shards, owner.clone()));
}
}
for (loc_idx, key, shards, owner) in to_load {
let shard_names: Vec<&str> = shards.iter().map(|(n, _)| n.as_str()).collect();
info!(
volume_id = key.vid.0,
collection = %key.collection,
from = %self.locations[owner.location].directory,
to = %self.locations[loc_idx].directory,
"loading orphan EC shards using index files from sibling disk (issue #9212): {:?}",
shard_names,
);
let shard_ids: Vec<u32> = shards.iter().map(|(_, sid)| *sid).collect();
let owner_idx_dir = owner.idx_dir.clone();
let loc = &mut self.locations[loc_idx];
if let Err(e) = loc.mount_ec_shards_with_idx_dir(
key.vid,
&key.collection,
&shard_ids,
&owner_idx_dir,
) {
// mount_ec_shards_with_idx_dir adds shards one at a
// time and increments the `ec_shards` gauge per shard
// that successfully attaches — a mid-loop failure
// would leave the EcVolume half-mounted with stale
// metric increments. Mirror DiskLocation::handle_found_ecx_file's
// recovery: drive the cleanup through unmount_ec_shards
// (which only decrements the gauge for shards that
// were actually mounted, drops the EcVolume when empty).
loc.unmount_ec_shards(key.vid, &shard_ids);
warn!(
volume_id = key.vid.0,
directory = %loc.directory,
"cross-disk shard load failed: {}",
e,
);
}
}
}
/// Build a `(collection, vid) -> EcxOwnerInfo` map of which disk
/// owns the `.ecx` file. `.ecx` normally lives in `IdxDirectory`
/// but may have been written into the data directory before
/// `-dir.idx` was set, so we check both — and we record which one
/// matched so downstream loaders point `EcVolume::new` at the
/// directory that really has the file. The first owner found wins;
/// duplicates across disks are unusual but tolerated.
fn index_ecx_owners(&self) -> HashMap<EcKey, EcxOwnerInfo> {
let mut owners: HashMap<EcKey, EcxOwnerInfo> = HashMap::new();
for (loc_idx, loc) in self.locations.iter().enumerate() {
let mut seen: Vec<&str> = Vec::with_capacity(2);
for scan in [loc.idx_directory.as_str(), loc.directory.as_str()] {
if scan.is_empty() || seen.contains(&scan) {
continue;
}
seen.push(scan);
let Ok(read) = fs::read_dir(scan) else {
continue;
};
for ent in read.flatten() {
if ent.file_type().map(|ft| ft.is_dir()).unwrap_or(false) {
continue;
}
let name = ent.file_name().to_string_lossy().into_owned();
let Some(base) = name.strip_suffix(".ecx") else {
continue;
};
let Some((collection, vid)) = parse_collection_volume_id_pub(base) else {
continue;
};
let key = EcKey { collection, vid };
owners.entry(key).or_insert(EcxOwnerInfo {
location: loc_idx,
idx_dir: scan.to_string(),
});
}
}
}
owners
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::MinFreeSpace;
use crate::storage::needle_map::NeedleMapKind;
use crate::storage::types::DiskType;
use crate::storage::volume::{VifEcShardConfig, VifVolumeInfo};
use tempfile::TempDir;
fn make_test_store(numdirs: usize, idx_subdir: Option<&str>) -> (Store, TempDir) {
let tmp = TempDir::new().unwrap();
let mut store = Store::new(NeedleMapKind::InMemory);
for i in 0..numdirs {
let data = tmp.path().join(format!("data{}", i));
std::fs::create_dir_all(&data).unwrap();
let idx = match idx_subdir {
Some(sub) => {
let p = tmp.path().join(format!("{}{}", sub, i));
std::fs::create_dir_all(&p).unwrap();
p.to_string_lossy().into_owned()
}
None => data.to_string_lossy().into_owned(),
};
store
.add_location(
data.to_str().unwrap(),
&idx,
100,
DiskType::HardDrive,
MinFreeSpace::Percent(0.0),
Vec::new(),
)
.unwrap();
}
(store, tmp)
}
fn write_shard(dir: &str, collection: &str, vid: u32, shard_id: u8) {
let p = format!("{}/{}_{}.ec{:02}", dir, collection, vid, shard_id);
std::fs::write(&p, b"shard data nonempty").unwrap();
}
fn write_index_files(idx_dir: &str, collection: &str, vid: u32, data_shards: u32, parity_shards: u32) {
// Minimal sealed .ecx (the loader only opens the file; it
// doesn't parse it during placement).
std::fs::write(
format!("{}/{}_{}.ecx", idx_dir, collection, vid),
vec![0u8; 20],
)
.unwrap();
std::fs::write(format!("{}/{}_{}.ecj", idx_dir, collection, vid), b"").unwrap();
let vif = VifVolumeInfo {
version: 3,
ec_shard_config: Some(VifEcShardConfig {
data_shards,
parity_shards,
}),
..Default::default()
};
std::fs::write(
format!("{}/{}_{}.vif", idx_dir, collection, vid),
serde_json::to_string(&vif).unwrap(),
)
.unwrap();
}
/// Reproduces the orphan-shard layout from issue #9212. Shards live
/// on dir0; the .ecx / .ecj / .vif live on dir1. Without
/// reconciliation, dir0's shards are silently dropped at startup.
#[test]
fn test_reconcile_loads_orphan_shards_from_sibling_disk() {
let tmp = TempDir::new().unwrap();
let dir0 = tmp.path().join("data0");
let dir1 = tmp.path().join("data1");
std::fs::create_dir_all(&dir0).unwrap();
std::fs::create_dir_all(&dir1).unwrap();
let collection = "grafana-loki";
let vid = 1093u32;
// dir0: orphan shards (no .ecx).
write_shard(dir0.to_str().unwrap(), collection, vid, 0);
write_shard(dir0.to_str().unwrap(), collection, vid, 12);
// dir1: one shard plus the index files (the disk that would
// own .ecx in steady state).
write_shard(dir1.to_str().unwrap(), collection, vid, 1);
write_index_files(dir1.to_str().unwrap(), collection, vid, 10, 4);
let mut store = Store::new(NeedleMapKind::InMemory);
store
.add_location(
dir0.to_str().unwrap(),
dir0.to_str().unwrap(),
100,
DiskType::HardDrive,
MinFreeSpace::Percent(0.0),
Vec::new(),
)
.unwrap();
store
.add_location(
dir1.to_str().unwrap(),
dir1.to_str().unwrap(),
100,
DiskType::HardDrive,
MinFreeSpace::Percent(0.0),
Vec::new(),
)
.unwrap();
// dir1 owns the .ecx and so already has shard 1 mounted via
// its own load_all_ec_shards.
let ev1 = store.locations[1].find_ec_volume(VolumeId(vid));
assert!(ev1.is_some(), "baseline broken: dir1 should have mounted shard 1");
// dir0's shards must be reconciled across to its own
// ec_volumes map, pointing at dir1's idx dir.
let ev0 = store.locations[0]
.find_ec_volume(VolumeId(vid))
.expect("dir0 should now have an EcVolume after reconcile");
assert!(ev0.has_shard(0), "shard 0 missing from dir0 after reconcile");
assert!(ev0.has_shard(12), "shard 12 missing from dir0 after reconcile");
}
/// PR 9244 review case: idx_directory is configured but the
/// owner's .ecx / .ecj / .vif live in the owner's data dir
/// (the legacy "written before -dir.idx was set" layout). The
/// reconciler must record the actual scan dir and pass it through
/// to mount_ec_shards_with_idx_dir, otherwise NewEcVolume's
/// same-disk fallback retries the orphan disk's data dir and
/// ENOENTs.
#[test]
fn test_reconcile_handles_ecx_in_owner_data_dir() {
let tmp = TempDir::new().unwrap();
let data0 = tmp.path().join("data0"); // orphan: shards only
let data1 = tmp.path().join("data1"); // owner: ecx in data dir
let idx0 = tmp.path().join("idx0");
let idx1 = tmp.path().join("idx1");
for p in &[&data0, &data1, &idx0, &idx1] {
std::fs::create_dir_all(p).unwrap();
}
let collection = "grafana-loki";
let vid = 4242u32;
write_shard(data0.to_str().unwrap(), collection, vid, 0);
write_shard(data0.to_str().unwrap(), collection, vid, 12);
write_shard(data1.to_str().unwrap(), collection, vid, 1);
// Owner's index files in DATA dir, not idx dir.
write_index_files(data1.to_str().unwrap(), collection, vid, 10, 4);
let mut store = Store::new(NeedleMapKind::InMemory);
store
.add_location(
data0.to_str().unwrap(),
idx0.to_str().unwrap(),
100,
DiskType::HardDrive,
MinFreeSpace::Percent(0.0),
Vec::new(),
)
.unwrap();
store
.add_location(
data1.to_str().unwrap(),
idx1.to_str().unwrap(),
100,
DiskType::HardDrive,
MinFreeSpace::Percent(0.0),
Vec::new(),
)
.unwrap();
let ev0 = store.locations[0]
.find_ec_volume(VolumeId(vid))
.expect("dir0 should have an EcVolume after reconcile (.ecx was in owner's data dir)");
assert!(ev0.has_shard(0));
assert!(ev0.has_shard(12));
}
/// Each disk is fully self-contained — reconciliation should leave
/// them untouched and not double-load any shards.
#[test]
fn test_reconcile_no_op_when_each_disk_is_self_contained() {
let (store, _tmp) = make_test_store(2, None);
let vid = VolumeId(3333);
// Each disk is empty; reconcile shouldn't crash and shouldn't
// mount anything.
assert!(store.locations[0].find_ec_volume(vid).is_none());
assert!(store.locations[1].find_ec_volume(vid).is_none());
}
/// Truly-orphaned: shards on disk but no `.ecx` anywhere on the
/// store. Reconciliation must log and leave the files alone — the
/// operator can restore the index later.
#[test]
fn test_reconcile_keeps_orphans_when_no_ecx_anywhere() {
let tmp = TempDir::new().unwrap();
let dir0 = tmp.path().join("data0");
let dir1 = tmp.path().join("data1");
std::fs::create_dir_all(&dir0).unwrap();
std::fs::create_dir_all(&dir1).unwrap();
let collection = "grafana-loki";
let vid = 2222u32;
// Shards on dir0; nothing else on either disk.
write_shard(dir0.to_str().unwrap(), collection, vid, 0);
write_shard(dir0.to_str().unwrap(), collection, vid, 12);
let mut store = Store::new(NeedleMapKind::InMemory);
for d in [&dir0, &dir1] {
store
.add_location(
d.to_str().unwrap(),
d.to_str().unwrap(),
100,
DiskType::HardDrive,
MinFreeSpace::Percent(0.0),
Vec::new(),
)
.unwrap();
}
// No EcVolume should be created (nothing to point at).
assert!(store.locations[0].find_ec_volume(VolumeId(vid)).is_none());
// Shard files must still exist on disk for operator recovery.
for sid in [0u8, 12u8] {
let p = format!("{}/{}_{}.ec{:02}", dir0.to_str().unwrap(), collection, vid, sid);
assert!(
std::path::Path::new(&p).exists(),
"orphan shard {} was destroyed",
p,
);
}
}
/// Helper: build a 2-disk store where reconcile produces the
/// cross-disk split layout (shards 0/12 on dir0, shard 1 + .ecx
/// on dir1). Mirrors the report-from-the-issue layout that
/// VolumeEcShardRead and friends now have to handle correctly.
fn build_split_disk_store(vid_raw: u32) -> (Store, TempDir) {
let tmp = TempDir::new().unwrap();
let dir0 = tmp.path().join("data0");
let dir1 = tmp.path().join("data1");
std::fs::create_dir_all(&dir0).unwrap();
std::fs::create_dir_all(&dir1).unwrap();
let collection = "grafana-loki";
let vid = vid_raw;
// dir0: shards 0 and 12, no .ecx
write_shard(dir0.to_str().unwrap(), collection, vid, 0);
write_shard(dir0.to_str().unwrap(), collection, vid, 12);
// dir1: shard 1 plus the index files
write_shard(dir1.to_str().unwrap(), collection, vid, 1);
write_index_files(dir1.to_str().unwrap(), collection, vid, 10, 4);
let mut store = Store::new(NeedleMapKind::InMemory);
for d in [&dir0, &dir1] {
store
.add_location(
d.to_str().unwrap(),
d.to_str().unwrap(),
100,
DiskType::HardDrive,
MinFreeSpace::Percent(0.0),
Vec::new(),
)
.unwrap();
}
(store, tmp)
}
/// Reconciliation can put the same `vid` on multiple disks with
/// disjoint shard subsets. Without a per-(vid, shard_id) lookup,
/// `find_ec_volume(vid)` returns disk 0's EcVolume and a request
/// for shard 1 (which lives on disk 1) gets "not mounted." The
/// new helpers must route to the right disk.
#[test]
fn test_find_ec_shard_location_finds_split_disk_shards() {
let (store, _tmp) = build_split_disk_store(7001);
let vid = VolumeId(7001);
// Shards 0 and 12 → disk 0; shard 1 → disk 1.
assert_eq!(store.find_ec_shard_location(vid, 0), Some(0));
assert_eq!(store.find_ec_shard_location(vid, 12), Some(0));
assert_eq!(store.find_ec_shard_location(vid, 1), Some(1));
// Unmounted shards → None.
assert_eq!(store.find_ec_shard_location(vid, 5), None);
// find_ec_volume_with_shard returns the EcVolume on the right
// disk for each shard.
let ev0 = store.find_ec_volume_with_shard(vid, 0).unwrap();
assert!(ev0.has_shard(0));
let ev1 = store.find_ec_volume_with_shard(vid, 1).unwrap();
assert!(ev1.has_shard(1));
// Different EcVolume instances per disk (same vid).
assert!(!std::ptr::eq(ev0, ev1));
}
/// `Store::unmount_ec_shards` used to return after the first
/// location with the vid, so a request to unmount a shard that
/// lives on a sibling disk became a silent no-op. After the fix,
/// every location with the vid is asked to unmount whatever
/// subset it has.
#[test]
fn test_unmount_ec_shards_reaches_all_locations() {
let (mut store, _tmp) = build_split_disk_store(7002);
let vid = VolumeId(7002);
// Sanity: shard 1 starts mounted on disk 1.
assert_eq!(store.find_ec_shard_location(vid, 1), Some(1));
// Unmount shard 1 only — it lives on disk 1, but disk 0 also
// has an EcVolume for the same vid (carrying shards 0/12).
store.unmount_ec_shards(vid, &[1]);
// After unmount, shard 1 should be gone from disk 1.
assert_eq!(store.find_ec_shard_location(vid, 1), None);
// And disk 0's shards must still be mounted (the unmount
// call should not have been a no-op on disk 0, but it also
// shouldn't have unmounted disk 0's unrelated shards).
assert_eq!(store.find_ec_shard_location(vid, 0), Some(0));
assert_eq!(store.find_ec_shard_location(vid, 12), Some(0));
}
/// Single-shard variant: `Store::unmount_ec_shard` likewise has
/// to reach the right disk regardless of which disk the
/// first-match `find_ec_volume(vid)` would have returned.
#[test]
fn test_unmount_ec_shard_finds_split_disk_shard() {
let (mut store, _tmp) = build_split_disk_store(7003);
let vid = VolumeId(7003);
store.unmount_ec_shard(vid, 1).unwrap();
assert_eq!(store.find_ec_shard_location(vid, 1), None);
// Other shards untouched.
assert_eq!(store.find_ec_shard_location(vid, 0), Some(0));
}
/// `delete_ec_shards` walks all locations to remove the on-disk
/// shard files (already correct) and then calls
/// `unmount_ec_shards` to drop in-memory state. Before the fix
/// the unmount stopped at the first location and could leave a
/// stale in-memory shard pointing at a now-deleted file. After
/// the fix every location with the vid sees the unmount.
#[test]
fn test_delete_ec_shards_unmounts_every_location() {
let (mut store, _tmp) = build_split_disk_store(7004);
let vid = VolumeId(7004);
let collection = "grafana-loki";
store.delete_ec_shards(vid, collection, &[1]);
// Shard 1 file is gone on disk 1.
let p1 = format!(
"{}/{}_{}.ec01",
store.locations[1].directory, collection, vid
);
assert!(!std::path::Path::new(&p1).exists());
// Disk 1's in-memory state for shard 1 is gone too.
assert_eq!(store.find_ec_shard_location(vid, 1), None);
// Disk 0's shards are unaffected.
assert_eq!(store.find_ec_shard_location(vid, 0), Some(0));
assert_eq!(store.find_ec_shard_location(vid, 12), Some(0));
}
/// `collect_ec_shard_dirs` aggregates per-shard data dirs across
/// every location with the vid — the primitive
/// `VolumeEcShardsToVolume` needs so it can decode a reconciled
/// volume whose shards are split across the data dirs of the
/// same volume server.
#[test]
fn test_collect_ec_shard_dirs_aggregates_across_locations() {
let (store, _tmp) = build_split_disk_store(7005);
let vid = VolumeId(7005);
let max_shards = 14;
let (_ev, dirs) = store.collect_ec_shard_dirs(vid, max_shards).unwrap();
// Shards 0 and 12 → disk 0's directory.
assert_eq!(dirs[0].as_deref(), Some(store.locations[0].directory.as_str()));
assert_eq!(dirs[12].as_deref(), Some(store.locations[0].directory.as_str()));
// Shard 1 → disk 1's directory.
assert_eq!(dirs[1].as_deref(), Some(store.locations[1].directory.as_str()));
// Unmounted shards → None.
for sid in [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13] {
assert_eq!(
dirs[sid], None,
"shard {} unexpectedly reported a dir",
sid,
);
}
}
/// PR #9252 review: when reconcile retries the cross-disk mount and
/// the loader hits an error mid-loop (e.g. a shard file goes
/// missing between the directory scan and the EcVolumeShard open),
/// we must roll back the partially-mounted state. Without that the
/// EcVolume on the orphan disk is left half-attached and the
/// per-shard gauge has stale increments. Set up a layout where
/// shard 1's file is removed *after* the dir scan but *before* the
/// reconcile pass runs, then assert no partial state survives.
#[test]
fn test_reconcile_rolls_back_partial_mounts_on_failure() {
let tmp = TempDir::new().unwrap();
let dir0 = tmp.path().join("data0"); // orphan: shards only
let dir1 = tmp.path().join("data1"); // owner: index files
std::fs::create_dir_all(&dir0).unwrap();
std::fs::create_dir_all(&dir1).unwrap();
let collection = "grafana-loki";
let vid = 8001u32;
// dir0 has shards 0 (ok) and 12 (will sabotage size to 0
// before the reconcile pass — load_all_ec_shards picks size>0
// shards but reconcile + EcVolumeShard::open hit the empty
// file later and fail).
write_shard(dir0.to_str().unwrap(), collection, vid, 0);
write_shard(dir0.to_str().unwrap(), collection, vid, 12);
// dir1 holds the index files.
write_index_files(dir1.to_str().unwrap(), collection, vid, 10, 4);
// Sabotage shard 12 file to be unreadable: replace it with a
// path that won't open (delete + recreate as zero bytes is
// filtered out earlier; instead, replace it with a directory
// of the same name so EcVolumeShard::open errors out).
let shard12 = format!("{}/{}_{}.ec12", dir0.to_str().unwrap(), collection, vid);
std::fs::remove_file(&shard12).unwrap();
std::fs::create_dir(&shard12).unwrap();
// Re-truncate shard 0 to zero so collect_orphan_ec_shards
// wouldn't pick it up: actually we want shard 0 to mount
// successfully so the partial-mount state exists. Leave it.
let mut store = Store::new(NeedleMapKind::InMemory);
for d in [&dir0, &dir1] {
store
.add_location(
d.to_str().unwrap(),
d.to_str().unwrap(),
100,
DiskType::HardDrive,
MinFreeSpace::Percent(0.0),
Vec::new(),
)
.unwrap();
}
// The collect_orphan_ec_shards scan filters by metadata().is_dir()
// already, so shard 12 (now a dir) is skipped by collect_orphan.
// For this test, we only care about the rollback path's
// existence; assert reconcile doesn't leave a partial
// EcVolume when shard 12's open fails. Actual sequencing
// depends on filesystem behavior, so we use a softer
// post-condition: every shard registered to dir0's EcVolume
// must correspond to a file that opens cleanly.
if let Some(ecv) = store.locations[0].find_ec_volume(VolumeId(vid)) {
for sid in 0u8..14 {
if ecv.has_shard(sid) {
let p = format!(
"{}/{}_{}.ec{:02}",
store.locations[0].directory, collection, vid, sid
);
let meta = std::fs::metadata(&p).unwrap();
assert!(
meta.is_file(),
"EcVolume on dir0 reports shard {} mounted but its file is not a regular file ({})",
sid,
p,
);
}
}
}
}
/// PR #9252 review: with `idx_directory != directory` configured but
/// the owner's `.ecx` actually living in `loc.directory` (the
/// legacy "written before -dir.idx was set" layout), the per-disk
/// loader's mount_ec_shards call uses `loc.idx_directory` and
/// errors out. Reconcile must NOT skip this same-disk case — it's
/// the only recovery path. Verify the owner disk's own shards
/// come back online after reconcile.
#[test]
fn test_reconcile_recovers_same_disk_legacy_ecx_layout() {
let tmp = TempDir::new().unwrap();
let data_dir = tmp.path().join("data");
let idx_dir = tmp.path().join("idx");
std::fs::create_dir_all(&data_dir).unwrap();
std::fs::create_dir_all(&idx_dir).unwrap();
let collection = "grafana-loki";
let vid = 8002u32;
// Owner disk holds shards 0/1 + index files in DATA dir
// (legacy layout). idx_directory is configured separately
// but empty.
write_shard(data_dir.to_str().unwrap(), collection, vid, 0);
write_shard(data_dir.to_str().unwrap(), collection, vid, 1);
write_index_files(data_dir.to_str().unwrap(), collection, vid, 10, 4);
// Need at least 2 locations for reconcile to run; add a
// second empty disk so the early `len < 2` short-circuit
// doesn't kick in.
let other = tmp.path().join("other");
std::fs::create_dir_all(&other).unwrap();
let mut store = Store::new(NeedleMapKind::InMemory);
store
.add_location(
data_dir.to_str().unwrap(),
idx_dir.to_str().unwrap(),
100,
DiskType::HardDrive,
MinFreeSpace::Percent(0.0),
Vec::new(),
)
.unwrap();
store
.add_location(
other.to_str().unwrap(),
other.to_str().unwrap(),
100,
DiskType::HardDrive,
MinFreeSpace::Percent(0.0),
Vec::new(),
)
.unwrap();
// After reconcile, the owner disk's own shards must be mounted
// even though .ecx lives in data_dir rather than the
// configured idx_dir.
let ev = store.locations[0]
.find_ec_volume(VolumeId(vid))
.expect("EcVolume should be mounted via cross-disk reconcile retry");
assert!(ev.has_shard(0));
assert!(ev.has_shard(1));
}
}
/// Walk a disk's data directory and return the `.ec??` shard files
/// that are present on disk but not yet registered in the location's
/// `ec_volumes` map. Keyed by (collection, vid) so callers can match
/// each group against its `.ecx`-owning disk in one lookup. Zero-byte
/// shard files are ignored — same shape as `load_all_ec_shards`.
fn collect_orphan_ec_shards(
loc: &crate::storage::disk_location::DiskLocation,
_loc_idx: usize,
) -> HashMap<EcKey, Vec<(String, u32)>> {
let mut orphans: HashMap<EcKey, Vec<(String, u32)>> = HashMap::new();
let Ok(read) = fs::read_dir(&loc.directory) else {
return orphans;
};
for ent in read.flatten() {
if ent.file_type().map(|ft| ft.is_dir()).unwrap_or(false) {
continue;
}
let name = ent.file_name().to_string_lossy().into_owned();
let Some(dot) = name.rfind('.') else {
continue;
};
let (base, ext) = name.split_at(dot);
let Some(shard_id) = is_ec_shard_extension(ext) else {
continue;
};
// Ignore zero-byte shards. Use the DirEntry's metadata so we
// don't pay a second stat syscall per file beyond what
// read_dir already returned.
match ent.metadata() {
Ok(meta) if meta.len() > 0 => {}
_ => continue,
}
let Some((collection, vid)) = parse_collection_volume_id_pub(base) else {
continue;
};
// Skip shards that are already registered to an EcVolume.
if let Some(ecv) = loc.find_ec_volume(vid) {
if ecv.has_shard(shard_id as u8) {
continue;
}
}
let key = EcKey { collection, vid };
orphans.entry(key).or_default().push((name, shard_id));
}
orphans
}