test: EC lifecycle chaos harness, with four fixes it found (#10763)

* ec: let the encode's balance see a migrating volume's shards across disk-type buckets

Shard generation writes beside the source .dat, so a cross-tier encode
(source on hdd, -diskType=ssd) leaves the fresh shards in the source
disk-type bucket. The encode's internal balance ingested only the target
bucket, saw no shards, and planned no moves; the spread guard then
correctly aborted the encode (and before that guard existed, the shards
silently stayed clumped on the generation host in the wrong tier).

EcBalance now takes the encode batch as migratingVolumeIds and ingests
those volumes' shards from every bucket, while everything else keeps the
bucket filter so a plain ec.balance never drags deliberately tiered
shards onto another disk type. The in-memory model delete also becomes
bucket-agnostic: a node holds a given shard in exactly one bucket, and a
bucket-scoped delete missed cross-bucket moves in the dry-run model.

* volume: decode reads shard 0 from its resolved path, not the EC volume's base dir

On a multi-disk server a volume's shards can sit on several disks; the
store registers each shard with its own path and CollectEcShards resolves
them, but FindDatFileSize derived the .ec00 path from the EcVolume's base
directory. When shard 0 lived on a sibling disk, VolumeEcShardsToVolume
failed with 'open ...ec00: no such file or directory' and ec.decode
aborted.

* ec: decode re-copies shards the topology claims but the target does not hold

An interrupted earlier decode or balance can leave the master believing
the decode target holds a shard whose file never landed: the mount
registered but the partial copy was cleaned, or the file was swept. The
collect step took the topology's word for it, excluded the shard from
the copy set, and the decode failed with 'missing shard'. Probe the
target's live inventory (VolumeEcShardsInfo) and treat anything it
cannot serve as still-to-copy.

* ec: decode discovers shards across disk-type buckets

Shards sit wherever encode generation and balance left them: a
cross-tier encode leaves them in the source disk-type bucket, a partial
migration straddles buckets. ec.decode scoped its shard discovery to the
-diskType bucket and reported a decodable volume as having no shards at
all. Union across buckets, the way the encode's shard verification
already does.

* test: EC chaos lifecycle harness

Randomized, seeded sequences of the EC lifecycle against a live cluster
in the production-shaped layout: multiple data disks per server, a
separate -dir.idx directory so .ecx/.ecj sidecars are shared across
disks, and a tagged ssd tier. Operations cover encode (hdd and ssd
targets), balance, shard damage plus rebuild, decode, re-encode,
deletes, scrub, tier moves, crash-restarts, sidecar fault injections
(a data-dir .vif pushed into the shared idx dir; a stale-generation
shard planted beside a newer encode), and interruptions: a real weed
shell subprocess killed mid-encode, mid-decode, and mid-balance, with
the recovery re-run required to converge.

One invariant holds after every step: every stored byte reads back
identical and every deleted needle stays deleted. EC_CHAOS_SEED and
EC_CHAOS_STEPS make runs reproducible and scalable.

A known gap is tolerated and logged rather than fixed here: a shard
mounted on two disks of one node (orphan adoption after an interrupted
copy) is invisible to ec.balance's dedup and unaddressable by
ec.shard.unmount's shard@address form, so no cleanup path exists yet.

* test: fail payload-corruption checks on the test goroutine

t.Fatalf inside require.Eventually's condition runs on the poller's
goroutine, where Goexit kills only that goroutine and the corruption
message can be lost behind a generic timeout. Record the mismatch, end
the polling, and fail on the test goroutine. Also assert the full shard
count in the cross-bucket decode-discovery test.

weed/ec/ec_balance_migrate_test.go

@@ -0,0 +1,63 @@
+package ec
+
+import (
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
+	"github.com/seaweedfs/seaweedfs/weed/storage/types"
+)
+
+// TestEcBalanceMigratesCrossDiskTypeShards: a cross-tier encode generates its
+// shards beside the source .dat — in the SOURCE disk-type bucket — while the
+// encode's balance targets -diskType. The balance must still see and spread
+// those shards when the volume is named as migrating; without that, the
+// planner finds nothing in the target bucket, plans no moves, and the encode's
+// spread guard aborts a perfectly good encode.
+func TestEcBalanceMigratesCrossDiskTypeShards(t *testing.T) {
+	allShards := []erasure_coding.ShardId{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13}
+
+	build := func() []*EcNode {
+		// Volume 1's fresh shards sit in the default (hdd, "") bucket of dn1;
+		// the balance runs with -diskType=ssd.
+		return []*EcNode{
+			newEcNode("dc1", "rack1", "dn1", 100).addEcVolumeAndShardsForTest(1, "c1", allShards),
+			newEcNode("dc1", "rack2", "dn2", 100),
+			newEcNode("dc1", "rack3", "dn3", 100),
+		}
+	}
+
+	// Without the migrating hint the target-bucket filter hides the shards:
+	// nothing moves. This is the standalone ec.balance semantic — shards of
+	// other tiers stay where they are.
+	ecb := &ecBalancer{
+		ecNodes:        build(),
+		applyBalancing: false,
+		diskType:       types.SsdType,
+	}
+	if err := ecb.balance([]string{"c1"}); err != nil {
+		t.Fatalf("balance without migrating hint: %v", err)
+	}
+	if got := ecb.ecNodes[0].LocalShardIdCount(1); got != len(allShards) {
+		t.Fatalf("without migrating hint, cross-type shards must stay put; dn1 has %d", got)
+	}
+
+	// With the volume named as migrating, the balance ingests the shards from
+	// the source bucket and spreads them.
+	ecb = &ecBalancer{
+		ecNodes:            build(),
+		applyBalancing:     false,
+		diskType:           types.SsdType,
+		migratingVolumeIds: map[uint32]bool{1: true},
+	}
+	if err := ecb.balance([]string{"c1"}); err != nil {
+		t.Fatalf("balance with migrating hint: %v", err)
+	}
+	onDn1 := ecb.ecNodes[0].LocalShardIdCount(1)
+	spread := ecb.ecNodes[1].LocalShardIdCount(1) + ecb.ecNodes[2].LocalShardIdCount(1)
+	if onDn1 == len(allShards) || spread == 0 {
+		t.Fatalf("migrating volume's shards did not spread: dn1=%d others=%d", onDn1, spread)
+	}
+	if onDn1+spread != len(allShards) {
+		t.Fatalf("shards lost or duplicated during dry-run spread: dn1=%d others=%d", onDn1, spread)
+	}
+}

weed/ec/ec_common.go

@@ -298,7 +298,7 @@ func MoveMountedShardToEcNode(env *Env, existingLocation *EcNode, collection str
 	}
 
 	destinationEcNode.AddEcVolumeShards(vid, collection, copiedShardIds, diskType)
-	existingLocation.DeleteEcVolumeShards(vid, copiedShardIds, diskType)
+	existingLocation.DeleteEcVolumeShards(vid, copiedShardIds)
 
 	return nil
 
@@ -634,9 +634,18 @@ func (ecNode *EcNode) AddEcVolumeShards(vid needle.VolumeId, collection string,
 	return ecNode
 }
 
-func (ecNode *EcNode) DeleteEcVolumeShards(vid needle.VolumeId, shardIds []erasure_coding.ShardId, diskType types.DiskType) *EcNode {
+// DeleteEcVolumeShards removes the shards from the node model wherever they
+// sit. A node holds a given shard in exactly one disk-type bucket, but which
+// bucket is not the caller's to know: a mid-migration (cross-tier encode)
+// volume keeps its fresh shards in the SOURCE disk's bucket while the balance
+// runs against the target type, so a bucket-scoped delete would miss them and
+// the dry-run model would count a moved shard twice.
+func (ecNode *EcNode) DeleteEcVolumeShards(vid needle.VolumeId, shardIds []erasure_coding.ShardId) *EcNode {
 
-	if diskInfo, found := ecNode.Info.DiskInfos[string(diskType)]; found {
+	for _, diskInfo := range ecNode.Info.DiskInfos {
+		if diskInfo == nil {
+			continue
+		}
 		for _, eci := range diskInfo.EcShardInfos {
 			if needle.VolumeId(eci.Id) == vid {
 				si := erasure_coding.ShardsInfoFromVolumeEcShardInformationMessage(eci)
@@ -750,6 +759,9 @@ type ecBalancer struct {
 	// volumeIds narrows the plan to these ec volume ids; nil balances every volume
 	// of the selected collections.
 	volumeIds map[uint32]bool
+	// migratingVolumeIds are mid-encode volumes whose shards are ingested from
+	// every disk-type bucket; see EcBalance.
+	migratingVolumeIds map[uint32]bool
 }
 
 // excludeNodes is a set of server addresses kept out of the balance as copy/move
@@ -760,7 +772,16 @@ type ecBalancer struct {
 //
 // volumeIds, when non-empty, restricts the plan to those ec volume ids; empty
 // balances every volume of the given collections.
-func EcBalance(env *Env, collections []string, dc string, ecReplicaPlacement *super_block.ReplicaPlacement, diskType types.DiskType, maxParallelization int, ioBytePerSecond int64, applyBalancing bool, excludeNodes map[pb.ServerAddress]struct{}, volumeIds []needle.VolumeId) (err error) {
+//
+// migratingVolumeIds names volumes whose shards are ingested from EVERY
+// disk-type bucket, not just the diskType one. ec.encode passes its batch:
+// shard generation writes beside the source .dat, so a cross-tier encode
+// (source on hdd, -diskType=ssd) leaves the fresh shards in the source bucket,
+// where a target-bucket-only balance cannot see them — it plans no moves and
+// the encode's spread guard aborts. Everything else keeps the bucket filter,
+// so a plain ec.balance -diskType=X never drags deliberately tiered shards of
+// other types onto X disks.
+func EcBalance(env *Env, collections []string, dc string, ecReplicaPlacement *super_block.ReplicaPlacement, diskType types.DiskType, maxParallelization int, ioBytePerSecond int64, applyBalancing bool, excludeNodes map[pb.ServerAddress]struct{}, volumeIds []needle.VolumeId, migratingVolumeIds []needle.VolumeId) (err error) {
 	// collect all ec nodes
 	allEcNodes, totalFreeEcSlots, err := CollectEcNodesForDC(env, dc, diskType)
 	if err != nil {
@@ -795,6 +816,13 @@ func EcBalance(env *Env, collections []string, dc string, ecReplicaPlacement *su
 			volumeIdFilter[uint32(vid)] = true
 		}
 	}
+	var migrating map[uint32]bool
+	if len(migratingVolumeIds) > 0 {
+		migrating = make(map[uint32]bool, len(migratingVolumeIds))
+		for _, vid := range migratingVolumeIds {
+			migrating[uint32(vid)] = true
+		}
+	}
 
 	ecb := &ecBalancer{
 		env:                env,
@@ -805,6 +833,7 @@ func EcBalance(env *Env, collections []string, dc string, ecReplicaPlacement *su
 		ioBytePerSecond:    ioBytePerSecond,
 		diskType:           diskType,
 		volumeIds:          volumeIdFilter,
+		migratingVolumeIds: migrating,
 	}
 
 	if len(collections) == 0 {
@@ -823,7 +852,7 @@ func defaultECRatio(_ string) (int, int) {
 // balance plans EC shard moves with the shared planner and executes them. When
 // collections is empty all collections present are balanced.
 func (ecb *ecBalancer) balance(collections []string) error {
-	topo, volumeRatio, selected := toBalancerTopology(ecb.ecNodes, collections, ecb.diskType, ecb.volumeIds)
+	topo, volumeRatio, selected := toBalancerTopology(ecb.ecNodes, collections, ecb.diskType, ecb.volumeIds, ecb.migratingVolumeIds)
 	if len(ecb.volumeIds) > 0 {
 		requested := make([]uint32, 0, len(ecb.volumeIds))
 		for vid := range ecb.volumeIds {
@@ -875,7 +904,7 @@ func (ecb *ecBalancer) balance(collections []string) error {
 // (0,0 when unreported, e.g. always in OSS), which Plan prefers over the
 // collection ratio for mixed-ratio clusters, and the set of volume ids that made
 // it into the topology.
-func toBalancerTopology(ecNodes []*EcNode, collections []string, diskType types.DiskType, volumeIds map[uint32]bool) (*ecbalancer.Topology, func(collection string, vid uint32) (int, int), map[uint32]bool) {
+func toBalancerTopology(ecNodes []*EcNode, collections []string, diskType types.DiskType, volumeIds map[uint32]bool, migratingVolumeIds map[uint32]bool) (*ecbalancer.Topology, func(collection string, vid uint32) (int, int), map[uint32]bool) {
 	allowed := make(map[string]bool, len(collections))
 	for _, c := range collections {
 		allowed[c] = true
@@ -898,21 +927,29 @@ func toBalancerTopology(ecNodes []*EcNode, collections []string, diskType types.
 		for diskId, d := range en.Disks {
 			node.AddDisk(diskId, d.DiskType, d.FreeEcSlots, d.EcShardCount)
 		}
-		diskInfo, found := en.Info.DiskInfos[string(diskType)]
-		if !found {
-			continue
-		}
-		for _, eci := range diskInfo.EcShardInfos {
-			if len(allowed) > 0 && !allowed[eci.Collection] {
+		for diskTypeKey, diskInfo := range en.Info.DiskInfos {
+			if diskInfo == nil {
 				continue
 			}
-			if volumeIds != nil && !volumeIds[eci.Id] {
-				continue
-			}
-			selected[eci.Id] = true
-			node.AddShards(eci.Id, eci.Collection, eci.DiskId, erasure_coding.ShardBits(eci.EcIndexBits))
-			if d, p := ecbalancer.VolumeShardRatio(eci); d > 0 || p > 0 {
-				volRatios[volRatioKey{eci.Collection, eci.Id}] = [2]int{d, p}
+			for _, eci := range diskInfo.EcShardInfos {
+				// A migrating (mid-encode) volume's fresh shards sit beside the
+				// source .dat, in whatever bucket that disk belongs to; ingest
+				// them regardless so the balance can move them onto the target
+				// disk type. All other volumes keep the bucket filter.
+				if diskTypeKey != string(diskType) && !migratingVolumeIds[eci.Id] {
+					continue
+				}
+				if len(allowed) > 0 && !allowed[eci.Collection] {
+					continue
+				}
+				if volumeIds != nil && !volumeIds[eci.Id] {
+					continue
+				}
+				selected[eci.Id] = true
+				node.AddShards(eci.Id, eci.Collection, eci.DiskId, erasure_coding.ShardBits(eci.EcIndexBits))
+				if d, p := ecbalancer.VolumeShardRatio(eci); d > 0 || p > 0 {
+					volRatios[volRatioKey{eci.Collection, eci.Id}] = [2]int{d, p}
+				}
 			}
 		}
 	}
@@ -1021,7 +1058,7 @@ func (ecb *ecBalancer) executeMove(byID map[string]*EcNode, m ecbalancer.Move) e
 	if m.Phase == "dedup" {
 		fmt.Printf("dedup: delete ec shard %d.%d on %s\n", vid, shardId, m.SourceNode)
 		if !ecb.applyBalancing {
-			src.DeleteEcVolumeShards(vid, shardIds, ecb.diskType)
+			src.DeleteEcVolumeShards(vid, shardIds)
 			return nil
 		}
 		grpcDialOption := ecb.env.GrpcDialOption

weed/ec/ec_decode.go

@@ -27,12 +27,12 @@ func DoEcDecode(env *Env, topoInfo *master_pb.TopologyInfo, collection string, v
 	}
 
 	// find volume location
-	nodeToEcShardsInfo, dataShards := collectEcNodeShardsInfo(topoInfo, vid, diskType)
+	nodeToEcShardsInfo, dataShards := collectEcNodeShardsInfo(topoInfo, vid)
 
 	fmt.Printf("ec volume %d shard locations: %+v\n", vid, nodeToEcShardsInfo)
 
 	if len(nodeToEcShardsInfo) == 0 {
-		return fmt.Errorf("no EC shards found for volume %d (diskType %s)", vid, diskType.ReadableString())
+		return fmt.Errorf("no EC shards found for volume %d", vid)
 	}
 
 	var originalShardCounts map[pb.ServerAddress]int
@@ -214,6 +214,23 @@ func collectEcShards(env *Env, nodeToShardsInfo map[pb.ServerAddress]*erasure_co
 		return "", fmt.Errorf("no eligible target datanodes available to decode volume %d", vid)
 	}
 
+	// Trust only shards the target can actually serve: an interrupted earlier
+	// decode or balance can leave the master believing the target holds a
+	// shard whose file never landed. A phantom entry here would exclude the
+	// shard from the copy set and the decode would then fail with "missing
+	// shard"; probing the target's live inventory makes the re-run re-copy it.
+	if present, probeErr := erasure_coding.CollectShardsOnServer(context.Background(), collection, uint32(vid), string(targetNodeLocation), env.GrpcDialOption); probeErr == nil {
+		confirmed := erasure_coding.NewShardsInfo()
+		for _, sid := range existingShardsInfo.Ids() {
+			if present.Has(sid) {
+				confirmed.Set(erasure_coding.NewShardInfo(sid, 0))
+			}
+		}
+		existingShardsInfo = confirmed
+	} else {
+		fmt.Printf("collectEcShards: probe %s inventory for volume %d: %v (keeping the topology's view)\n", targetNodeLocation, vid, probeErr)
+	}
+
 	fmt.Printf("collectEcShards: ec volume %d collect shards to %s from: %+v\n", vid, targetNodeLocation, nodeToShardsInfo)
 
 	copiedShardsInfo := erasure_coding.NewShardsInfo()
@@ -294,14 +311,19 @@ func CollectEcShardIds(topoInfo *master_pb.TopologyInfo, collectionRegex *regexp
 	return
 }
 
-func collectEcNodeShardsInfo(topoInfo *master_pb.TopologyInfo, vid needle.VolumeId, diskType types.DiskType) (map[pb.ServerAddress]*erasure_coding.ShardsInfo, int) {
+func collectEcNodeShardsInfo(topoInfo *master_pb.TopologyInfo, vid needle.VolumeId) (map[pb.ServerAddress]*erasure_coding.ShardsInfo, int) {
 	res := make(map[pb.ServerAddress]*erasure_coding.ShardsInfo)
 	EachDataNode(topoInfo, func(dc DataCenterId, rack RackId, dn *master_pb.DataNodeInfo) {
-		if diskInfo, found := dn.DiskInfos[string(diskType)]; found {
-			// A node may report several EcShardInfos for one volume — one per
-			// physical disk holding shards of it (multi-disk nodes). Union them
-			// rather than overwriting, or only the last disk's shards survive and
-			// the node looks like it is missing shards it actually has.
+		// Union across ALL disk-type buckets and, within a node, across its
+		// physical disks. Shards sit wherever encode generation and balance
+		// left them — a cross-tier encode leaves them in the source bucket, a
+		// partial migration straddles buckets — and a decode that only looks
+		// at one bucket reports a decodable volume as having no shards at all.
+		// (Same rationale as the encode's shard verification.)
+		for _, diskInfo := range dn.DiskInfos {
+			if diskInfo == nil {
+				continue
+			}
 			for _, v := range diskInfo.EcShardInfos {
 				if v.Id == uint32(vid) {
 					addr := pb.NewServerAddressFromDataNode(dn)

weed/ec/ec_encode.go

@@ -117,7 +117,7 @@ func ProcessEcEncodeBatch(env *Env, writer io.Writer, volumeIds []needle.VolumeI
 	// safely verified and deleted without waiting for all batches to finish.
 	// skippedNodes are excluded so a recovered node's stale orphan is never
 	// paired with a new-generation shard.
-	if err := EcBalance(env, balanceCollections, "", rp, diskType, maxParallelization, 0, applyBalancing, skippedNodes, nil); err != nil {
+	if err := EcBalance(env, balanceCollections, "", rp, diskType, maxParallelization, 0, applyBalancing, skippedNodes, nil, volumeIds); err != nil {
 		return fmt.Errorf("re-balance ec shards for collection(s) %v: %w", balanceCollections, err)
 	}
 	if err := verifyEcShardsBeforeDelete(env, volumeIds, diskType, applyBalancing); err != nil {

weed/ec/ec_encode_test.go

@@ -449,10 +449,15 @@ func TestEcShardCountIgnoresDiskTypeOfTheShards(t *testing.T) {
 	assert.NoError(t, err, "a complete shard set must not read as unrecoverable")
 	assert.False(t, degraded)
 
-	// The disk-scoped view is what the check used to consult, and it is blind
-	// to this volume entirely.
-	scoped, _ := collectEcNodeShardsInfo(topo, needle.VolumeId(1), types.ToDiskType(""))
-	assert.Empty(t, scoped, "the hdd-scoped view cannot see ssd shards")
+	// The decode's node view unions across buckets for the same reason: a
+	// decode scoped to the hdd bucket would report this decodable volume as
+	// having no shards at all.
+	byNode, _ := collectEcNodeShardsInfo(topo, needle.VolumeId(1))
+	assert.Len(t, byNode, 1, "decode discovery must see shards on a non-default medium")
+	for _, si := range byNode {
+		assert.Equal(t, erasure_coding.TotalShardsCount, si.Count(),
+			"decode discovery must include every shard on the non-default medium")
+	}
 }
 
 // The message an aborted deletion leaves behind is all the operator has to go

weed/server/volume_grpc_erasure_coding.go

@@ -990,10 +990,12 @@ func (vs *VolumeServer) VolumeEcShardsToVolume(ctx context.Context, req *volume_
 		return nil, status.Errorf(codes.FailedPrecondition, "ec volume %d %s", req.VolumeId, erasure_coding.EcNoLiveEntriesSubstring)
 	}
 
-	// calculate .dat file size
-	datFileSize, err := erasure_coding.FindDatFileSize(dataBaseFileName, indexBaseFileName)
+	// calculate .dat file size. Pass shard 0's resolved path: on a multi-disk
+	// server the volume's shards can sit on several disks, so the .ec00 is not
+	// necessarily beside the EcVolume's own base path.
+	datFileSize, err := erasure_coding.FindDatFileSize(shardFileNames[0], indexBaseFileName)
 	if err != nil {
-		return nil, fmt.Errorf("FindDatFileSize %s: %v", dataBaseFileName, err)
+		return nil, fmt.Errorf("FindDatFileSize %s: %v", shardFileNames[0], err)
 	}
 
 	// The shard block layout was fixed by the .dat size at encode time (recorded

weed/shell/command_ec_common.go

@@ -138,7 +138,7 @@ func moveMountedShardToEcNode(commandEnv *CommandEnv, existingLocation *EcNode,
 // EcBalance balances EC shards across the cluster; see ec.EcBalance for the
 // excludeNodes and volumeIds semantics.
 func EcBalance(commandEnv *CommandEnv, collections []string, dc string, ecReplicaPlacement *super_block.ReplicaPlacement, diskType types.DiskType, maxParallelization int, ioBytePerSecond int64, applyBalancing bool, excludeNodes map[pb.ServerAddress]struct{}, volumeIds []needle.VolumeId) (err error) {
-	return ec.EcBalance(commandEnv.ecEnv(), collections, dc, ecReplicaPlacement, diskType, maxParallelization, ioBytePerSecond, applyBalancing, excludeNodes, volumeIds)
+	return ec.EcBalance(commandEnv.ecEnv(), collections, dc, ecReplicaPlacement, diskType, maxParallelization, ioBytePerSecond, applyBalancing, excludeNodes, volumeIds, nil)
 }
 
 // compileCollectionPattern compiles a regex pattern for collection matching.

weed/storage/erasure_coding/ec_decoder.go

@@ -93,11 +93,15 @@ func WriteIdxFileFromEcIndex(baseFileName string) (err error) {
 // FindDatFileSize calculate .dat file size from max offset entry
 // there may be extra deletions after that entry
 // but they are deletions anyway
-func FindDatFileSize(dataBaseFileName, indexBaseFileName string) (datSize int64, err error) {
+// shard0FileName is the actual path of the .ec00 shard file, which on a
+// multi-disk server may sit on a different disk than the EcVolume's own base
+// path — the store registers shards per disk, so the caller must pass the
+// path CollectEcShards resolved rather than deriving it from a base name.
+func FindDatFileSize(shard0FileName, indexBaseFileName string) (datSize int64, err error) {
 
-	version, err := readEcVolumeVersion(dataBaseFileName)
+	version, err := readEcVolumeVersion(shard0FileName)
 	if err != nil {
-		return 0, fmt.Errorf("read ec volume %s version: %v", dataBaseFileName, err)
+		return 0, fmt.Errorf("read ec volume %s version: %v", shard0FileName, err)
 	}
 
 	// Safety: ensure datSize is at least SuperBlockSize. While the caller typically
@@ -122,19 +126,19 @@ func FindDatFileSize(dataBaseFileName, indexBaseFileName string) (datSize int64,
 	return
 }
 
-func readEcVolumeVersion(baseFileName string) (version needle.Version, err error) {
+func readEcVolumeVersion(shard0FileName string) (version needle.Version, err error) {
 
 	// find volume version
-	datFile, err := os.OpenFile(baseFileName+".ec00", os.O_RDONLY, 0644)
+	datFile, err := os.OpenFile(shard0FileName, os.O_RDONLY, 0644)
 	if err != nil {
-		return 0, fmt.Errorf("open ec volume %s superblock: %v", baseFileName, err)
+		return 0, fmt.Errorf("open ec volume %s superblock: %v", shard0FileName, err)
 	}
 	datBackend := backend.NewDiskFile(datFile)
 
 	superBlock, err := super_block.ReadSuperBlock(datBackend)
 	datBackend.Close()
 	if err != nil {
-		return 0, fmt.Errorf("read ec volume %s superblock: %v", baseFileName, err)
+		return 0, fmt.Errorf("read ec volume %s superblock: %v", shard0FileName, err)
 	}
 
 	return superBlock.Version, nil

weed/storage/erasure_coding/ec_decoder_multidisk_test.go

@@ -0,0 +1,54 @@
+package erasure_coding
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+// TestFindDatFileSizeWithRelocatedShard0: on a multi-disk server a volume's
+// shards can sit on several disks, so the .ec00 is not necessarily beside the
+// EcVolume's base path. FindDatFileSize takes the resolved shard-0 path; it
+// must work when that path points to a different directory than the index.
+func TestFindDatFileSizeWithRelocatedShard0(t *testing.T) {
+	diskA := t.TempDir()
+	diskB := t.TempDir()
+
+	for _, ext := range []string{".dat", ".idx"} {
+		data, err := os.ReadFile("1" + ext)
+		if err != nil {
+			t.Fatalf("read fixture 1%s: %v", ext, err)
+		}
+		if err := os.WriteFile(filepath.Join(diskA, "1"+ext), data, 0o644); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	base := filepath.Join(diskA, "1")
+	ctx := NewDefaultECContext("", 0)
+	if _, err := generateEcFiles(base, 50, largeBlockSize, smallBlockSize, ctx); err != nil {
+		t.Fatalf("generateEcFiles: %v", err)
+	}
+	if err := WriteSortedFileFromIdx(base, ".ecx"); err != nil {
+		t.Fatalf("WriteSortedFileFromIdx: %v", err)
+	}
+
+	wantSize, err := FindDatFileSize(base+".ec00", base)
+	if err != nil {
+		t.Fatalf("FindDatFileSize with co-located shard 0: %v", err)
+	}
+
+	// Relocate shard 0 to a sibling disk, as a balance or copy can leave it.
+	moved := filepath.Join(diskB, "1.ec00")
+	if err := os.Rename(base+".ec00", moved); err != nil {
+		t.Fatal(err)
+	}
+
+	gotSize, err := FindDatFileSize(moved, base)
+	if err != nil {
+		t.Fatalf("FindDatFileSize with relocated shard 0: %v", err)
+	}
+	if gotSize != wantSize {
+		t.Fatalf("dat size changed with relocated shard 0: got %d want %d", gotSize, wantSize)
+	}
+}

weed/storage/erasure_coding/verification.go

@@ -150,15 +150,12 @@ func SummarizeShardInventory(perServer map[string]ServerShardInventory) string {
 // A server that cannot be queried is unknown, not confirmed, and returns an
 // error: treating an unreachable peer as proof of a surviving copy is how a
 // network blip becomes data loss.
-func VerifyShardsOnServer(ctx context.Context, collection string, volumeID uint32,
-	server string, shardIDs []uint32, dialOption grpc.DialOption) error {
-
-	if server == "" {
-		return fmt.Errorf("no server given to verify volume %d shard(s) %v", volumeID, shardIDs)
-	}
-
-	var present ShardBits
-	callErr := operation.WithVolumeServerClient(false, pb.ServerAddress(server), dialOption,
+// CollectShardsOnServer asks one volume server which shards of the volume it
+// actually serves right now — its live inventory, as opposed to what the
+// master's possibly stale topology claims for it.
+func CollectShardsOnServer(ctx context.Context, collection string, volumeID uint32,
+	server string, dialOption grpc.DialOption) (present ShardBits, err error) {
+	err = operation.WithVolumeServerClient(false, pb.ServerAddress(server), dialOption,
 		func(client volume_server_pb.VolumeServerClient) error {
 			resp, e := client.VolumeEcShardsInfo(ctx, &volume_server_pb.VolumeEcShardsInfoRequest{
 				VolumeId: volumeID,
@@ -174,6 +171,17 @@ func VerifyShardsOnServer(ctx context.Context, collection string, volumeID uint3
 			}
 			return nil
 		})
+	return present, err
+}
+
+func VerifyShardsOnServer(ctx context.Context, collection string, volumeID uint32,
+	server string, shardIDs []uint32, dialOption grpc.DialOption) error {
+
+	if server == "" {
+		return fmt.Errorf("no server given to verify volume %d shard(s) %v", volumeID, shardIDs)
+	}
+
+	present, callErr := CollectShardsOnServer(ctx, collection, volumeID, server, dialOption)
 	if callErr != nil {
 		return fmt.Errorf("verify volume %d shard(s) %v on %s: %w", volumeID, shardIDs, server, callErr)
 	}