refactor: share volume and EC shard move logic between shell and workers (#10727)

* operation: add shared volume_move package for volume and EC shard moves

The shell commands (volume.move, volume.balance, ec.balance, tier moves)
and the maintenance workers (balance, ec_balance) each carried their own
copy of the move RPC sequences, and the copies had drifted: the worker
verified the target before deleting the source but dropped the disk
type and IO throttle; the shell passed those but deleted the source
unverified.

volume_move.Mover carries the merged sequences, keeping the stricter
behavior from each side:

- LiveMoveVolume: check-then-hard-freeze the source (VolumeStatus's
  IsReadOnly also covers low-disk and readonly-but-can-delete states,
  which still accept needle deletes), copy with disk type and IO
  throttle, tail, verify the target is not behind the source before the
  destructive source delete (a target that is ahead holds writes it
  accepted during the tail and the move commits to keep them), and
  restore the source's writability when a failure precedes the delete
  and this move did the freezing. Aborts clean up the incomplete target
  copy; a failed cleanup or an ambiguous source delete keeps the source
  readonly (ErrSourceKeptReadonly) so callers do not thaw a source next
  to a possibly-authoritative copy. With a readonly source, an existing
  or unknown-state target refuses the move outright: no client-side
  observation can prove such a copy is a stale remnant rather than the
  authoritative copy of an unfinished move.
- MoveEcShards: copy with the .ecx/.ecj/.vif/.ecsum sidecars, mount,
  verify the target registered every shard before unmount+delete on the
  source, and reject same-server moves (the EC delete is server-wide).

Server identity is the grpc endpoint (SameServer), so node:8080 and
node:8080.18080 compare equal while test servers sharing a degenerate
HTTP address stay distinct; addresses are validated non-fatally before
dialing and before being embedded in copy/tail requests, since both the
client dialer and the receiving server normalize them through a parser
that aborts the process on a malformed port. The Rust volume server's
codes.NotFound counts as a definitively absent probe answer alongside
the Go server's plain-error code Unknown.

All RPCs go through an injectable ClientFunc, so the sequences are unit
tested against a fake volume server client: RPC order, request fields,
and that verification failures keep the source intact.

* shell, worker: delegate volume and EC shard moves to operation/volume_move

LiveMoveVolume and the copy/tail/delete/mark-writable helpers become
thin wrappers over the shared mover, keeping their signatures; the EC
helpers keep their per-step output and delegate the RPCs. BalanceTask
and ECBalanceTask keep their parameter validation, progress reporting,
and guards (same-node cross-disk rejection, dedup keep-node
verification, shard ids range-checked before the uint8 narrowing) and
hand the RPC sequences to the mover. volume.tier.move skips its
thaw-on-failure when the mover deliberately kept the source readonly,
since reopening the replicas beside a possibly-authoritative target
copy would fork the volume.

The tail-failure tolerance moves inside the mover: a failed tail is
tolerated only when the volume was already readonly before the move
began, backstopped by a stability re-read across the idle window, so
volume.balance's -skipTailError-by-readonly heuristic and tier-move's
unconditional skip both become the same authoritative rule.

* volume_move: keep the source readonly when a failed copy leaves a target of unknown origin

A failed copy can leave a complete, mounted copy on the target (the
server finishes after the client loses the stream). The abort probed
the target only when its pre-copy state was known-absent; an unknown
prior state skipped both the probe and the cleanup and then reopened
the source - two writable replicas of one volume, diverging from the
next write on.

The abort now probes the target on every failed copy and restores the
source only when the target provably holds nothing. A copy whose
provenance cannot be proven (unknown prior state, a pre-existing
replica, or an unreachable target) is never deleted, and the source
stays readonly with ErrSourceKeptReadonly naming the recovery.

* test: teach the plugin worker harness the shared move sequence

The fake volume server lacked VolumeStatus, which the shared mover now
issues before freezing the source, and the batch execution test's
status-read accounting predates the pre-copy target probe and the
verification reads. Mirrors the harness the enterprise tree already
carries.

test/plugin_workers/fake_volume_server.go

@@ -32,6 +32,7 @@ type VolumeServer struct {
 
 	mu                   sync.Mutex
 	receivedFiles        map[string]uint64
+	readonlyVolumes      map[uint32]bool
 	mountRequests        []*volume_server_pb.VolumeEcShardsMountRequest
 	deleteRequests       []*volume_server_pb.VolumeDeleteRequest
 	markReadonlyCalls    int
@@ -67,12 +68,13 @@ func NewVolumeServer(t *testing.T, baseDir string) *VolumeServer {
 	grpcPort := listener.Addr().(*net.TCPAddr).Port
 	server := pb.NewGrpcServer()
 	vs := &VolumeServer{
-		t:             t,
-		server:        server,
-		listener:      listener,
-		address:       fmt.Sprintf("127.0.0.1:0.%d", grpcPort),
-		baseDir:       baseDir,
-		receivedFiles: make(map[string]uint64),
+		t:               t,
+		server:          server,
+		listener:        listener,
+		address:         fmt.Sprintf("127.0.0.1:0.%d", grpcPort),
+		baseDir:         baseDir,
+		receivedFiles:   make(map[string]uint64),
+		readonlyVolumes: make(map[uint32]bool),
 	}
 
 	volume_server_pb.RegisterVolumeServerServer(server, vs)
@@ -386,6 +388,9 @@ func (v *VolumeServer) VolumeDelete(ctx context.Context, req *volume_server_pb.V
 func (v *VolumeServer) VolumeMarkReadonly(ctx context.Context, req *volume_server_pb.VolumeMarkReadonlyRequest) (*volume_server_pb.VolumeMarkReadonlyResponse, error) {
 	v.mu.Lock()
 	v.markReadonlyCalls++
+	if req != nil {
+		v.readonlyVolumes[req.VolumeId] = true
+	}
 	v.mu.Unlock()
 	return &volume_server_pb.VolumeMarkReadonlyResponse{}, nil
 }
@@ -393,10 +398,19 @@ func (v *VolumeServer) VolumeMarkReadonly(ctx context.Context, req *volume_serve
 func (v *VolumeServer) VolumeMarkWritable(ctx context.Context, req *volume_server_pb.VolumeMarkWritableRequest) (*volume_server_pb.VolumeMarkWritableResponse, error) {
 	v.mu.Lock()
 	v.markWritableCalls++
+	if req != nil {
+		v.readonlyVolumes[req.VolumeId] = false
+	}
 	v.mu.Unlock()
 	return &volume_server_pb.VolumeMarkWritableResponse{}, nil
 }
 
+func (v *VolumeServer) VolumeStatus(ctx context.Context, req *volume_server_pb.VolumeStatusRequest) (*volume_server_pb.VolumeStatusResponse, error) {
+	v.mu.Lock()
+	defer v.mu.Unlock()
+	return &volume_server_pb.VolumeStatusResponse{IsReadOnly: v.readonlyVolumes[req.GetVolumeId()]}, nil
+}
+
 func (v *VolumeServer) ReadVolumeFileStatus(ctx context.Context, req *volume_server_pb.ReadVolumeFileStatusRequest) (*volume_server_pb.ReadVolumeFileStatusResponse, error) {
 	v.mu.Lock()
 	v.readFileStatusCalls++

test/plugin_workers/volume_balance/execution_test.go

@@ -146,13 +146,14 @@ func TestVolumeBalanceBatchExecutionIntegration(t *testing.T) {
 		require.True(t, deletedVols[vid], "volume %d should have been deleted from source", vid)
 	}
 
-	// Each move reads source status once before copy and once inside the
-	// target's fake VolumeCopy implementation, then reads target status once
-	// before deleting the source.
+	// Each move reads source status once inside the target's fake VolumeCopy
+	// implementation and once for the pre-delete verification, and reads
+	// target status once probing for a pre-existing copy and once for the
+	// verification.
 	require.Equal(t, len(volumeIDs)*2, source.ReadFileStatusCount(),
-		"each move should read source volume status before copy and during target copy")
-	require.Equal(t, len(volumeIDs), target.ReadFileStatusCount(),
-		"each move should read target volume status before delete")
+		"each move should read source volume status during target copy and at verification")
+	require.Equal(t, len(volumeIDs)*2, target.ReadFileStatusCount(),
+		"each move should probe the target and read its status at verification")
 
 	// Target should have received copy and tail calls for all 3 volumes.
 	copyCalls, _, tailCalls := target.BalanceStats()

weed/operation/volume_move/ec_move.go

@@ -0,0 +1,200 @@
+package volume_move
+
+import (
+	"context"
+	"fmt"
+	"io"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb"
+	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
+	"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
+	"github.com/seaweedfs/seaweedfs/weed/storage/needle"
+)
+
+// EcShardMove is one planned relocation of mounted shards to another server.
+type EcShardMove struct {
+	VolumeId   needle.VolumeId
+	Collection string
+	ShardIds   []erasure_coding.ShardId
+	Source     pb.ServerAddress
+	Target     pb.ServerAddress
+	// TargetDisk picks the destination disk (0 lets the server pick).
+	TargetDisk uint32
+}
+
+// EcMoveOptions control MoveEcShards.
+type EcMoveOptions struct {
+	// Writer receives human-readable progress lines (nil discards them).
+	Writer io.Writer
+	// Progress, when set, receives percent/stage callbacks as the move advances.
+	Progress func(percent float64, stage string)
+}
+
+// MoveEcShards relocates mounted shards: copy+mount on the target, verify the
+// target registered them, then unmount+delete on the source. The verification
+// gates the destructive half — a copy/mount RPC can return OK while the shard
+// is not loadable on the target, and deleting the source then would lose it;
+// on a mismatch the source is kept so the caller can retry.
+func (m *Mover) MoveEcShards(ctx context.Context, move EcShardMove, opts EcMoveOptions) error {
+	writer := opts.Writer
+	if writer == nil {
+		writer = io.Discard
+	}
+	progress := opts.Progress
+	if progress == nil {
+		progress = func(float64, string) {}
+	}
+
+	// A same-server "move" cannot be expressed with these RPCs: the source
+	// delete is server-wide, so it would erase the just-copied shard. Removing
+	// a duplicate shard in place is RemoveEcShards. SameServer, not ==:
+	// "node:8080" and "node:8080.18080" are one server.
+	if SameServer(move.Source, move.Target) {
+		return fmt.Errorf("refusing EC shard move of volume %d shard(s) %v onto its own server %s: the source delete is server-wide", move.VolumeId, move.ShardIds, move.Source)
+	}
+
+	progress(10, fmt.Sprintf("copying EC shard(s) %d.%v from %s to %s", move.VolumeId, move.ShardIds, move.Source, move.Target))
+	if err := m.CopyAndMountEcShards(ctx, move.VolumeId, move.Collection, move.ShardIds, move.Source, move.Target, move.TargetDisk, writer); err != nil {
+		return err
+	}
+
+	progress(40, fmt.Sprintf("verifying EC shard(s) %d.%v on %s", move.VolumeId, move.ShardIds, move.Target))
+	if err := m.VerifyEcShards(ctx, move.VolumeId, move.Target, move.ShardIds); err != nil {
+		return err
+	}
+
+	progress(50, fmt.Sprintf("unmounting EC shard(s) %d.%v from %s", move.VolumeId, move.ShardIds, move.Source))
+	fmt.Fprintf(writer, "unmount %d.%v from %s\n", move.VolumeId, move.ShardIds, move.Source)
+	if err := m.UnmountEcShards(ctx, move.VolumeId, move.Source, move.ShardIds); err != nil {
+		return fmt.Errorf("unmount %d.%v from %s: %v", move.VolumeId, move.ShardIds, move.Source, err)
+	}
+
+	progress(75, fmt.Sprintf("deleting EC shard(s) %d.%v from %s", move.VolumeId, move.ShardIds, move.Source))
+	fmt.Fprintf(writer, "delete %d.%v from %s\n", move.VolumeId, move.ShardIds, move.Source)
+	if err := m.DeleteEcShards(ctx, move.VolumeId, move.Collection, move.Source, move.ShardIds); err != nil {
+		return fmt.Errorf("delete %d.%v from %s: %v", move.VolumeId, move.ShardIds, move.Source, err)
+	}
+
+	progress(100, fmt.Sprintf("moved EC shard(s) %d.%v from %s to %s", move.VolumeId, move.ShardIds, move.Source, move.Target))
+	return nil
+}
+
+// CopyAndMountEcShards has the target copy the shards (with their .ecx/.ecj/
+// .vif/.ecsum sidecars) from source and mount them. A same-address call skips
+// the copy and just mounts — ec.encode uses that to bring freshly generated
+// shards online in place.
+func (m *Mover) CopyAndMountEcShards(ctx context.Context, volumeId needle.VolumeId, collection string, shardIds []erasure_coding.ShardId, source, target pb.ServerAddress, targetDisk uint32, writer io.Writer) error {
+	if writer == nil {
+		writer = io.Discard
+	}
+	// The target dials the embedded source itself; a malformed one would
+	// abort the target server, not this client.
+	if !SameServer(target, source) {
+		if err := checkDialable(source); err != nil {
+			return err
+		}
+	}
+	return m.withClient(false, target, func(client volume_server_pb.VolumeServerClient) error {
+		if !SameServer(target, source) {
+			fmt.Fprintf(writer, "copy %d.%v %s => %s\n", volumeId, shardIds, source, target)
+			_, copyErr := client.VolumeEcShardsCopy(ctx, &volume_server_pb.VolumeEcShardsCopyRequest{
+				VolumeId:       uint32(volumeId),
+				Collection:     collection,
+				ShardIds:       erasure_coding.ShardIdsToUint32(shardIds),
+				CopyEcxFile:    true,
+				CopyEcjFile:    true,
+				CopyVifFile:    true,
+				CopyEcsumFile:  true, // propagate the bitrot sidecar with the shards (no-op if the source has none)
+				SourceDataNode: string(source),
+				DiskId:         targetDisk,
+			})
+			if copyErr != nil {
+				return fmt.Errorf("copy %d.%v %s => %s: %v", volumeId, shardIds, source, target, copyErr)
+			}
+		}
+		fmt.Fprintf(writer, "mount %d.%v on %s\n", volumeId, shardIds, target)
+		_, mountErr := client.VolumeEcShardsMount(ctx, &volume_server_pb.VolumeEcShardsMountRequest{
+			VolumeId:   uint32(volumeId),
+			Collection: collection,
+			ShardIds:   erasure_coding.ShardIdsToUint32(shardIds),
+		})
+		if mountErr != nil {
+			return fmt.Errorf("mount %d.%v on %s: %v", volumeId, shardIds, target, mountErr)
+		}
+		return nil
+	})
+}
+
+// VerifyEcShards confirms server has every shard in shardIds registered for
+// the volume.
+func (m *Mover) VerifyEcShards(ctx context.Context, volumeId needle.VolumeId, server pb.ServerAddress, shardIds []erasure_coding.ShardId) error {
+	return m.withClient(false, server, func(client volume_server_pb.VolumeServerClient) error {
+		resp, err := client.VolumeEcShardsInfo(ctx, &volume_server_pb.VolumeEcShardsInfoRequest{
+			VolumeId: uint32(volumeId),
+		})
+		if err != nil {
+			return fmt.Errorf("verify EC shard(s) on %s for volume %d: %v", server, volumeId, err)
+		}
+		var bits erasure_coding.ShardBits
+		for _, s := range resp.EcShardInfos {
+			if s.VolumeId != uint32(volumeId) || s.ShardId >= erasure_coding.MaxShardCount {
+				continue
+			}
+			bits = bits.Set(erasure_coding.ShardId(s.ShardId))
+		}
+		for _, sid := range shardIds {
+			if !bits.Has(sid) {
+				return fmt.Errorf("%s missing EC shard %d.%d after copy/mount; keeping source", server, volumeId, sid)
+			}
+		}
+		return nil
+	})
+}
+
+// MountEcShards mounts shards already on server.
+func (m *Mover) MountEcShards(ctx context.Context, volumeId needle.VolumeId, collection string, server pb.ServerAddress, shardIds []erasure_coding.ShardId) error {
+	return m.withClient(false, server, func(client volume_server_pb.VolumeServerClient) error {
+		_, mountErr := client.VolumeEcShardsMount(ctx, &volume_server_pb.VolumeEcShardsMountRequest{
+			VolumeId:   uint32(volumeId),
+			Collection: collection,
+			ShardIds:   erasure_coding.ShardIdsToUint32(shardIds),
+		})
+		return mountErr
+	})
+}
+
+// UnmountEcShards unmounts shards on server.
+func (m *Mover) UnmountEcShards(ctx context.Context, volumeId needle.VolumeId, server pb.ServerAddress, shardIds []erasure_coding.ShardId) error {
+	return m.withClient(false, server, func(client volume_server_pb.VolumeServerClient) error {
+		_, unmountErr := client.VolumeEcShardsUnmount(ctx, &volume_server_pb.VolumeEcShardsUnmountRequest{
+			VolumeId: uint32(volumeId),
+			ShardIds: erasure_coding.ShardIdsToUint32(shardIds),
+		})
+		return unmountErr
+	})
+}
+
+// DeleteEcShards deletes shards on server. The delete is server-wide: it
+// removes the shards from every disk of the server.
+func (m *Mover) DeleteEcShards(ctx context.Context, volumeId needle.VolumeId, collection string, server pb.ServerAddress, shardIds []erasure_coding.ShardId) error {
+	return m.withClient(false, server, func(client volume_server_pb.VolumeServerClient) error {
+		_, deleteErr := client.VolumeEcShardsDelete(ctx, &volume_server_pb.VolumeEcShardsDeleteRequest{
+			VolumeId:   uint32(volumeId),
+			Collection: collection,
+			ShardIds:   erasure_coding.ShardIdsToUint32(shardIds),
+		})
+		return deleteErr
+	})
+}
+
+// RemoveEcShards unmounts then deletes shards in place — the dedup path for a
+// shard that already has a copy elsewhere.
+func (m *Mover) RemoveEcShards(ctx context.Context, volumeId needle.VolumeId, collection string, server pb.ServerAddress, shardIds []erasure_coding.ShardId) error {
+	if err := m.UnmountEcShards(ctx, volumeId, server, shardIds); err != nil {
+		return fmt.Errorf("unmount %d.%v from %s: %v", volumeId, shardIds, server, err)
+	}
+	if err := m.DeleteEcShards(ctx, volumeId, collection, server, shardIds); err != nil {
+		return fmt.Errorf("delete %d.%v from %s: %v", volumeId, shardIds, server, err)
+	}
+	return nil
+}

weed/operation/volume_move/ec_move_test.go

@@ -0,0 +1,117 @@
+package volume_move
+
+import (
+	"context"
+	"strings"
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb"
+	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
+	"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
+)
+
+func ecMove(shardIds ...erasure_coding.ShardId) EcShardMove {
+	return EcShardMove{
+		VolumeId:   7,
+		Collection: "c1",
+		ShardIds:   shardIds,
+		Source:     srcAddr,
+		Target:     dstAddr,
+		TargetDisk: 2,
+	}
+}
+
+func dstShards(shardIds ...uint32) []*volume_server_pb.EcShardInfo {
+	var infos []*volume_server_pb.EcShardInfo
+	for _, sid := range shardIds {
+		infos = append(infos, &volume_server_pb.EcShardInfo{VolumeId: 7, ShardId: sid})
+	}
+	return infos
+}
+
+func TestMoveEcShardsSequence(t *testing.T) {
+	cluster := newFakeCluster()
+	cluster.ecShards[string(dstAddr)] = dstShards(3, 4)
+
+	err := cluster.mover().MoveEcShards(context.Background(), ecMove(3, 4), EcMoveOptions{})
+	if err != nil {
+		t.Fatalf("MoveEcShards: %v", err)
+	}
+
+	assertCalls(t, cluster.callList(), []string{
+		"dst:8080 VolumeEcShardsCopy",
+		"dst:8080 VolumeEcShardsMount",
+		"dst:8080 VolumeEcShardsInfo",
+		"src:8080 VolumeEcShardsUnmount",
+		"src:8080 VolumeEcShardsDelete",
+	})
+
+	copyReq := cluster.ecCopyReqs[0]
+	if !copyReq.CopyEcxFile || !copyReq.CopyEcjFile || !copyReq.CopyVifFile || !copyReq.CopyEcsumFile {
+		t.Errorf("shard sidecars not all copied: %+v", copyReq)
+	}
+	if copyReq.DiskId != 2 || copyReq.SourceDataNode != string(srcAddr) || copyReq.Collection != "c1" {
+		t.Errorf("copy request not propagated: %+v", copyReq)
+	}
+}
+
+func TestMoveEcShardsVerifyFailureKeepsSource(t *testing.T) {
+	cluster := newFakeCluster()
+	cluster.ecShards[string(dstAddr)] = dstShards(3) // shard 4 didn't register
+
+	err := cluster.mover().MoveEcShards(context.Background(), ecMove(3, 4), EcMoveOptions{})
+	if err == nil || !strings.Contains(err.Error(), "missing EC shard 7.4") {
+		t.Fatalf("expected missing-shard error, got: %v", err)
+	}
+
+	for _, call := range cluster.callList() {
+		if call == "src:8080 VolumeEcShardsUnmount" || call == "src:8080 VolumeEcShardsDelete" {
+			t.Fatalf("source touched despite verification failure: %v", cluster.callList())
+		}
+	}
+}
+
+func TestMoveEcShardsRejectsSameServer(t *testing.T) {
+	// The second target is the same server written with an explicit grpc port;
+	// the guard must see through the representation difference.
+	for _, target := range []pb.ServerAddress{srcAddr, pb.ServerAddress("src:8080.18080")} {
+		cluster := newFakeCluster()
+		move := ecMove(3)
+		move.Target = target
+
+		err := cluster.mover().MoveEcShards(context.Background(), move, EcMoveOptions{})
+		if err == nil || !strings.Contains(err.Error(), "its own server") {
+			t.Fatalf("target %q: expected same-server rejection, got: %v", target, err)
+		}
+		if len(cluster.callList()) != 0 {
+			t.Fatalf("target %q: RPCs issued for a rejected move: %v", target, cluster.callList())
+		}
+	}
+}
+
+func TestRemoveEcShards(t *testing.T) {
+	cluster := newFakeCluster()
+
+	err := cluster.mover().RemoveEcShards(context.Background(), 7, "c1", srcAddr, []erasure_coding.ShardId{3})
+	if err != nil {
+		t.Fatalf("RemoveEcShards: %v", err)
+	}
+
+	assertCalls(t, cluster.callList(), []string{
+		"src:8080 VolumeEcShardsUnmount",
+		"src:8080 VolumeEcShardsDelete",
+	})
+}
+
+func TestCopyAndMountEcShardsSameAddressMountsOnly(t *testing.T) {
+	cluster := newFakeCluster()
+
+	err := cluster.mover().CopyAndMountEcShards(context.Background(), 7, "c1", []erasure_coding.ShardId{3}, srcAddr, srcAddr, 0, nil)
+	if err != nil {
+		t.Fatalf("CopyAndMountEcShards: %v", err)
+	}
+
+	assertCalls(t, cluster.callList(), []string{
+		"src:8080 VolumeEcShardsMount",
+	})
+}

weed/operation/volume_move/fake_client_test.go

@@ -0,0 +1,230 @@
+package volume_move
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"sync"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb"
+	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// fakeCluster fakes the volume servers a move touches. It records every RPC as
+// "<server> <method>" so tests can assert the exact sequence, and lets tests
+// inject per-RPC errors and per-server state.
+type fakeCluster struct {
+	mu    sync.Mutex
+	calls []string
+
+	// errs fails an RPC, keyed "<server> <method>".
+	errs map[string]error
+	// readonly is each server's VolumeStatus.IsReadOnly answer.
+	readonly map[string]bool
+	// status is each server's ReadVolumeFileStatus answer.
+	status map[string]*volume_server_pb.ReadVolumeFileStatusResponse
+	// statusFailures fails a server's next N ReadVolumeFileStatus calls, e.g.
+	// to model a target that has no volume before the copy but one after. The
+	// failures use statusFailureErr, defaulting to a server-side (gRPC code
+	// Unknown) volume-not-found error.
+	statusFailures   map[string]int
+	statusFailureErr error
+	// statusSeq overrides a server's next ReadVolumeFileStatus answers in
+	// order, e.g. to model a source that changes between two reads.
+	statusSeq map[string][]*volume_server_pb.ReadVolumeFileStatusResponse
+	// ecShards is each server's VolumeEcShardsInfo answer.
+	ecShards map[string][]*volume_server_pb.EcShardInfo
+
+	lastAppendAtNs uint64
+
+	copyReqs   []*volume_server_pb.VolumeCopyRequest
+	deleteReqs []*volume_server_pb.VolumeDeleteRequest
+	ecCopyReqs []*volume_server_pb.VolumeEcShardsCopyRequest
+}
+
+func newFakeCluster() *fakeCluster {
+	return &fakeCluster{
+		errs:           make(map[string]error),
+		readonly:       make(map[string]bool),
+		status:         make(map[string]*volume_server_pb.ReadVolumeFileStatusResponse),
+		statusFailures: make(map[string]int),
+		statusSeq:      make(map[string][]*volume_server_pb.ReadVolumeFileStatusResponse),
+		ecShards:       make(map[string][]*volume_server_pb.EcShardInfo),
+	}
+}
+
+func (c *fakeCluster) mover() *Mover {
+	return NewMoverWithClientFunc(func(streamingMode bool, addr pb.ServerAddress, fn func(client volume_server_pb.VolumeServerClient) error) error {
+		return fn(&fakeClient{cluster: c, addr: string(addr)})
+	})
+}
+
+func (c *fakeCluster) record(addr, method string) error {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.calls = append(c.calls, addr+" "+method)
+	return c.errs[addr+" "+method]
+}
+
+func (c *fakeCluster) callList() []string {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	return append([]string(nil), c.calls...)
+}
+
+// fakeClient answers as the volume server at addr. Unused VolumeServerClient
+// methods panic via the embedded nil interface.
+type fakeClient struct {
+	volume_server_pb.VolumeServerClient
+	cluster *fakeCluster
+	addr    string
+}
+
+func (f *fakeClient) VolumeStatus(ctx context.Context, req *volume_server_pb.VolumeStatusRequest, opts ...grpc.CallOption) (*volume_server_pb.VolumeStatusResponse, error) {
+	if err := f.cluster.record(f.addr, "VolumeStatus"); err != nil {
+		return nil, err
+	}
+	return &volume_server_pb.VolumeStatusResponse{IsReadOnly: f.cluster.readonly[f.addr]}, nil
+}
+
+func (f *fakeClient) VolumeMarkReadonly(ctx context.Context, req *volume_server_pb.VolumeMarkReadonlyRequest, opts ...grpc.CallOption) (*volume_server_pb.VolumeMarkReadonlyResponse, error) {
+	if err := f.cluster.record(f.addr, "VolumeMarkReadonly"); err != nil {
+		return nil, err
+	}
+	return &volume_server_pb.VolumeMarkReadonlyResponse{}, nil
+}
+
+func (f *fakeClient) VolumeMarkWritable(ctx context.Context, req *volume_server_pb.VolumeMarkWritableRequest, opts ...grpc.CallOption) (*volume_server_pb.VolumeMarkWritableResponse, error) {
+	if err := f.cluster.record(f.addr, "VolumeMarkWritable"); err != nil {
+		return nil, err
+	}
+	return &volume_server_pb.VolumeMarkWritableResponse{}, nil
+}
+
+func (f *fakeClient) ReadVolumeFileStatus(ctx context.Context, req *volume_server_pb.ReadVolumeFileStatusRequest, opts ...grpc.CallOption) (*volume_server_pb.ReadVolumeFileStatusResponse, error) {
+	if err := f.cluster.record(f.addr, "ReadVolumeFileStatus"); err != nil {
+		return nil, err
+	}
+	f.cluster.mu.Lock()
+	failures := f.cluster.statusFailures[f.addr]
+	if failures > 0 {
+		f.cluster.statusFailures[f.addr] = failures - 1
+	}
+	failureErr := f.cluster.statusFailureErr
+	var seqResp *volume_server_pb.ReadVolumeFileStatusResponse
+	if failures == 0 {
+		if seq := f.cluster.statusSeq[f.addr]; len(seq) > 0 {
+			seqResp = seq[0]
+			f.cluster.statusSeq[f.addr] = seq[1:]
+		}
+	}
+	resp := f.cluster.status[f.addr]
+	f.cluster.mu.Unlock()
+	if failures > 0 {
+		if failureErr != nil {
+			return nil, failureErr
+		}
+		return nil, status.Error(codes.Unknown, fmt.Sprintf("not found volume id %d", req.VolumeId))
+	}
+	if seqResp != nil {
+		return seqResp, nil
+	}
+	if resp == nil {
+		return nil, status.Error(codes.Unknown, fmt.Sprintf("not found volume id %d", req.VolumeId))
+	}
+	return resp, nil
+}
+
+func (f *fakeClient) VolumeCopy(ctx context.Context, req *volume_server_pb.VolumeCopyRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[volume_server_pb.VolumeCopyResponse], error) {
+	if err := f.cluster.record(f.addr, "VolumeCopy"); err != nil {
+		return nil, err
+	}
+	f.cluster.mu.Lock()
+	f.cluster.copyReqs = append(f.cluster.copyReqs, req)
+	lastAppendAtNs := f.cluster.lastAppendAtNs
+	f.cluster.mu.Unlock()
+	return &fakeCopyStream{resps: []*volume_server_pb.VolumeCopyResponse{
+		{ProcessedBytes: 1024},
+		{LastAppendAtNs: lastAppendAtNs},
+	}}, nil
+}
+
+func (f *fakeClient) VolumeTailReceiver(ctx context.Context, req *volume_server_pb.VolumeTailReceiverRequest, opts ...grpc.CallOption) (*volume_server_pb.VolumeTailReceiverResponse, error) {
+	if err := f.cluster.record(f.addr, "VolumeTailReceiver"); err != nil {
+		return nil, err
+	}
+	return &volume_server_pb.VolumeTailReceiverResponse{}, nil
+}
+
+func (f *fakeClient) VolumeDelete(ctx context.Context, req *volume_server_pb.VolumeDeleteRequest, opts ...grpc.CallOption) (*volume_server_pb.VolumeDeleteResponse, error) {
+	if err := f.cluster.record(f.addr, "VolumeDelete"); err != nil {
+		return nil, err
+	}
+	f.cluster.mu.Lock()
+	f.cluster.deleteReqs = append(f.cluster.deleteReqs, req)
+	f.cluster.mu.Unlock()
+	return &volume_server_pb.VolumeDeleteResponse{}, nil
+}
+
+func (f *fakeClient) VolumeConfigure(ctx context.Context, req *volume_server_pb.VolumeConfigureRequest, opts ...grpc.CallOption) (*volume_server_pb.VolumeConfigureResponse, error) {
+	if err := f.cluster.record(f.addr, "VolumeConfigure"); err != nil {
+		return nil, err
+	}
+	return &volume_server_pb.VolumeConfigureResponse{}, nil
+}
+
+func (f *fakeClient) VolumeEcShardsCopy(ctx context.Context, req *volume_server_pb.VolumeEcShardsCopyRequest, opts ...grpc.CallOption) (*volume_server_pb.VolumeEcShardsCopyResponse, error) {
+	if err := f.cluster.record(f.addr, "VolumeEcShardsCopy"); err != nil {
+		return nil, err
+	}
+	f.cluster.mu.Lock()
+	f.cluster.ecCopyReqs = append(f.cluster.ecCopyReqs, req)
+	f.cluster.mu.Unlock()
+	return &volume_server_pb.VolumeEcShardsCopyResponse{}, nil
+}
+
+func (f *fakeClient) VolumeEcShardsMount(ctx context.Context, req *volume_server_pb.VolumeEcShardsMountRequest, opts ...grpc.CallOption) (*volume_server_pb.VolumeEcShardsMountResponse, error) {
+	if err := f.cluster.record(f.addr, "VolumeEcShardsMount"); err != nil {
+		return nil, err
+	}
+	return &volume_server_pb.VolumeEcShardsMountResponse{}, nil
+}
+
+func (f *fakeClient) VolumeEcShardsUnmount(ctx context.Context, req *volume_server_pb.VolumeEcShardsUnmountRequest, opts ...grpc.CallOption) (*volume_server_pb.VolumeEcShardsUnmountResponse, error) {
+	if err := f.cluster.record(f.addr, "VolumeEcShardsUnmount"); err != nil {
+		return nil, err
+	}
+	return &volume_server_pb.VolumeEcShardsUnmountResponse{}, nil
+}
+
+func (f *fakeClient) VolumeEcShardsDelete(ctx context.Context, req *volume_server_pb.VolumeEcShardsDeleteRequest, opts ...grpc.CallOption) (*volume_server_pb.VolumeEcShardsDeleteResponse, error) {
+	if err := f.cluster.record(f.addr, "VolumeEcShardsDelete"); err != nil {
+		return nil, err
+	}
+	return &volume_server_pb.VolumeEcShardsDeleteResponse{}, nil
+}
+
+func (f *fakeClient) VolumeEcShardsInfo(ctx context.Context, req *volume_server_pb.VolumeEcShardsInfoRequest, opts ...grpc.CallOption) (*volume_server_pb.VolumeEcShardsInfoResponse, error) {
+	if err := f.cluster.record(f.addr, "VolumeEcShardsInfo"); err != nil {
+		return nil, err
+	}
+	return &volume_server_pb.VolumeEcShardsInfoResponse{EcShardInfos: f.cluster.ecShards[f.addr]}, nil
+}
+
+// fakeCopyStream feeds the canned VolumeCopy responses, then io.EOF.
+type fakeCopyStream struct {
+	grpc.ClientStream
+	resps []*volume_server_pb.VolumeCopyResponse
+}
+
+func (s *fakeCopyStream) Recv() (*volume_server_pb.VolumeCopyResponse, error) {
+	if len(s.resps) == 0 {
+		return nil, io.EOF
+	}
+	resp := s.resps[0]
+	s.resps = s.resps[1:]
+	return resp, nil
+}

weed/operation/volume_move/mover.go

@@ -0,0 +1,113 @@
+// Package volume_move implements the volume and EC shard move sequences shared
+// by the interactive shell commands (volume.move, volume.balance, ec.balance,
+// volume.tier.move, ...) and the maintenance workers (balance, ec_balance,
+// volume_tiering). The RPCs go through an injectable ClientFunc so every
+// sequence can be unit tested against a fake volume server client.
+package volume_move
+
+import (
+	"fmt"
+	"strconv"
+	"strings"
+
+	"github.com/seaweedfs/seaweedfs/weed/operation"
+	"github.com/seaweedfs/seaweedfs/weed/pb"
+	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
+	"github.com/seaweedfs/seaweedfs/weed/util"
+	"google.golang.org/grpc"
+)
+
+// ClientFunc runs fn with a client for the volume server at addr. It matches
+// operation.WithVolumeServerClient, which production movers use to dial real
+// servers; tests substitute a fake client.
+type ClientFunc func(streamingMode bool, addr pb.ServerAddress, fn func(client volume_server_pb.VolumeServerClient) error) error
+
+// Mover executes volume and EC shard moves against volume servers.
+type Mover struct {
+	withClient ClientFunc
+}
+
+func NewMover(grpcDialOption grpc.DialOption) *Mover {
+	return &Mover{withClient: func(streamingMode bool, addr pb.ServerAddress, fn func(client volume_server_pb.VolumeServerClient) error) error {
+		// Validated here, the single point every mover RPC dials through:
+		// handing a malformed address (an unvalidated -source/-target flag)
+		// to the dialer would abort the whole process instead of failing the
+		// move.
+		if err := checkDialable(addr); err != nil {
+			return err
+		}
+		return operation.WithVolumeServerClient(streamingMode, addr, grpcDialOption, fn)
+	}}
+}
+
+// checkDialable rejects an address whose grpc normalization would abort the
+// process: for the "host:port" form, ServerAddress.ToGrpcAddress falls back
+// to a parser that calls glog.Fatalf when the port is not numeric. Everything
+// else either normalizes cleanly or fails at dial time as an ordinary error.
+// It also guards the source addresses embedded in copy/tail requests: the
+// receiving volume server dials those through the same fatal parser, so an
+// unchecked malformed source would terminate the destination server.
+func checkDialable(addr pb.ServerAddress) error {
+	s := string(addr)
+	colon := strings.LastIndex(s, ":")
+	if colon < 0 || colon+1 >= len(s) {
+		return nil // no port part; handed to the dialer untouched
+	}
+	ports := s[colon+1:]
+	if dot := strings.LastIndex(ports, "."); dot >= 0 {
+		// "port.grpcPort": different dial paths canonicalize a half-malformed
+		// form differently (the method splices the grpc part in unparsed; the
+		// string helper falls back to the http part + 10000), so a bad
+		// component could dial an unintended server or reach the fatal
+		// parser. Require both components numeric.
+		if _, err := strconv.ParseUint(ports[:dot], 10, 64); err != nil {
+			return fmt.Errorf("invalid volume server address %q: port %q is not a number", s, ports[:dot])
+		}
+		if _, err := strconv.ParseUint(ports[dot+1:], 10, 64); err != nil {
+			return fmt.Errorf("invalid volume server address %q: grpc port %q is not a number", s, ports[dot+1:])
+		}
+		return nil
+	}
+	if _, err := strconv.ParseUint(ports, 10, 64); err != nil {
+		return fmt.Errorf("invalid volume server address %q: port %q is not a number", s, ports)
+	}
+	return nil
+}
+
+// NewMoverWithClientFunc builds a Mover on a custom transport.
+func NewMoverWithClientFunc(withClient ClientFunc) *Mover {
+	return &Mover{withClient: withClient}
+}
+
+// SameServer reports whether two addresses name the same volume server. The
+// gRPC endpoint identifies the server process — each has exactly one — so
+// "node:8080" and "node:8080.18080" compare equal, while two servers sharing
+// a degenerate HTTP address (e.g. port 0 in test harnesses) stay distinct.
+// The normalization is non-fatal, unlike ServerAddress.ToGrpcAddress, whose
+// parser exits the process on a malformed port; anything unparsable compares
+// as its literal self.
+func SameServer(a, b pb.ServerAddress) bool {
+	return grpcEndpoint(string(a)) == grpcEndpoint(string(b))
+}
+
+// grpcEndpoint mirrors ServerAddress.ToGrpcAddress ("host:port" gets the
+// +10000 grpc port; "host:port.grpcPort" names it explicitly) but returns a
+// malformed address unchanged instead of exiting.
+func grpcEndpoint(addr string) string {
+	colon := strings.LastIndex(addr, ":")
+	if colon < 0 || colon+1 >= len(addr) {
+		return addr
+	}
+	host, ports := addr[:colon], addr[colon+1:]
+	if dot := strings.LastIndex(ports, "."); dot >= 0 {
+		if grpcPort, err := strconv.Atoi(ports[dot+1:]); err == nil {
+			return util.JoinHostPort(host, grpcPort)
+		}
+		return addr
+	}
+	httpPort, err := strconv.Atoi(ports)
+	if err != nil {
+		return addr
+	}
+	return util.JoinHostPort(host, httpPort+10000)
+}

weed/operation/volume_move/volume_move.go

@@ -0,0 +1,501 @@
+package volume_move
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/glog"
+	"github.com/seaweedfs/seaweedfs/weed/pb"
+	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
+	"github.com/seaweedfs/seaweedfs/weed/storage/needle"
+	"github.com/seaweedfs/seaweedfs/weed/util"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// ErrSourceKeptReadonly marks a move failure whose recovery deliberately keeps
+// the source volume readonly — the target copy may hold data the source lacks
+// (an ambiguous source delete, or an undeletable stale copy). Callers that
+// froze the source themselves must not thaw it on this error.
+var ErrSourceKeptReadonly = errors.New("the source volume is deliberately kept readonly")
+
+// VolumeMoveOptions control LiveMoveVolume.
+type VolumeMoveOptions struct {
+	// DiskType changes the volume's disk type on the target ("" keeps it).
+	DiskType string
+	// IoBytePerSecond throttles the copy (0 = unlimited).
+	IoBytePerSecond int64
+	// IdleTimeout is how long the tail phase waits for in-flight requests to
+	// drain (default 5s).
+	IdleTimeout time.Duration
+	// Writer receives human-readable progress lines (nil discards them).
+	Writer io.Writer
+	// Progress, when set, receives percent/stage callbacks as the move advances.
+	Progress func(percent float64, stage string)
+}
+
+func (o *VolumeMoveOptions) fillDefaults() {
+	if o.IdleTimeout <= 0 {
+		o.IdleTimeout = 5 * time.Second
+	}
+	if o.Writer == nil {
+		o.Writer = io.Discard
+	}
+	if o.Progress == nil {
+		o.Progress = func(float64, string) {}
+	}
+}
+
+// LiveMoveVolume moves one volume between volume servers while it keeps serving
+// reads: freeze the source (readonly), copy, tail to drain in-flight requests,
+// verify the target matches the source, then delete the source. A failure
+// before the source delete restores the source's writability if this move is
+// what froze it.
+func (m *Mover) LiveMoveVolume(ctx context.Context, volumeId needle.VolumeId, source, target pb.ServerAddress, opts VolumeMoveOptions) (err error) {
+	opts.fillDefaults()
+
+	// VolumeCopy tears down any existing copy on the target first, so a
+	// same-server "move" would delete the volume and then fail to read it.
+	// SameServer, not ==: "node:8080" and "node:8080.18080" are one server.
+	if SameServer(source, target) {
+		return fmt.Errorf("refusing to move volume %d onto its own server %s", volumeId, source)
+	}
+
+	opts.Progress(10, fmt.Sprintf("marking volume %d readonly on %s", volumeId, source))
+	sourceWasWritable, err := m.ensureVolumeReadonly(ctx, volumeId, source, true)
+	// The source stays authoritative until its delete succeeds; on any earlier
+	// failure undo the freeze this move added and clean up the target copy.
+	// Installed before the error check: the marking RPC itself can fail after
+	// the server applied the mark (e.g. its master notification failed).
+	var copyStarted, copyCompleted, sourceDeleteStarted bool
+	var targetHadVolume, targetStateKnown bool
+	defer func() {
+		if err != nil {
+			if sourceDeleteStarted {
+				// Past verification: the target is complete and may already hold
+				// new writes, and the delete may or may not have reached the
+				// source. Deleting the target or reopening the source could fork
+				// the volume; keep the source readonly.
+				fmt.Fprintf(opts.Writer, "volume %d is left readonly on %s: the source delete failed with the verified copy on %s mounted\n", volumeId, source, target)
+				glog.Warningf("volume %d is left readonly on %s: the source delete failed with the verified copy on %s mounted", volumeId, source, target)
+				err = fmt.Errorf("%w: %w", ErrSourceKeptReadonly, err)
+				return
+			}
+			// The cleanup runs on its own deadline so an abort via cancelled
+			// context still cleans up.
+			cleanupCtx, cleanupCancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
+			defer cleanupCancel()
+			cleanupTarget := copyCompleted
+			if copyStarted && !copyCompleted {
+				// The server can finish the copy and mount the target even when
+				// the client loses the stream, so probe rather than assume.
+				exists, known := m.probeVolume(cleanupCtx, volumeId, target)
+				switch {
+				case known && !exists:
+					// nothing mounted on the target; just undo the freeze below
+				case known && exists && targetStateKnown && !targetHadVolume:
+					// the failed copy created it; remove it so the source stays
+					// the only replica
+					cleanupTarget = true
+				default:
+					// A copy may sit mounted on the target and its provenance
+					// cannot be proven: the prior state is unknown, the target
+					// held a replica before the move, or the probe failed.
+					// Deleting it risks someone else's replica; reopening the
+					// source beside it risks two writable replicas taking
+					// divergent writes. Keep the source readonly.
+					fmt.Fprintf(opts.Writer, "volume %d is left readonly on %s: a copy may exist on %s but its origin cannot be determined; delete one side explicitly, then re-run the move\n", volumeId, source, target)
+					glog.Warningf("volume %d is left readonly on %s: a copy may exist on %s but its origin cannot be determined; delete one side explicitly, then re-run the move", volumeId, source, target)
+					err = fmt.Errorf("%w: %w", ErrSourceKeptReadonly, err)
+					return
+				}
+			}
+			if cleanupTarget {
+				// The target copy may be missing tailed entries; remove it so
+				// the source stays the only replica.
+				if dErr := m.DeleteVolume(cleanupCtx, volumeId, target, false, true); dErr != nil {
+					// Restoring the source while the stale target stays mounted
+					// risks divergent replicas; keep the source readonly. A
+					// re-run refuses while the copy exists, so name the fix.
+					fmt.Fprintf(opts.Writer, "volume %d is left readonly on %s: failed to delete the incomplete copy on %s: %v; delete that copy, then re-run the move\n", volumeId, source, target, dErr)
+					glog.Warningf("volume %d is left readonly on %s: failed to delete the incomplete copy on %s: %v; delete that copy, then re-run the move", volumeId, source, target, dErr)
+					err = fmt.Errorf("%w: %w", ErrSourceKeptReadonly, err)
+					return
+				}
+			}
+			if sourceWasWritable {
+				m.restoreVolumeWritable(volumeId, source)
+			} else {
+				// The hard freeze added on top of a readonly-reporting status is
+				// kept: it is indistinguishable from an operator's mark. It is
+				// not persisted, so a volume server restart clears it.
+				fmt.Fprintf(opts.Writer, "volume %d on %s keeps the readonly mark the move added; volume.mark -writable clears it if the prior readonly state was transient\n", volumeId, source)
+			}
+		}
+	}()
+	if err != nil {
+		return fmt.Errorf("mark volume %d readonly on %s: %v", volumeId, source, err)
+	}
+
+	targetHadVolume, targetStateKnown = m.probeVolume(ctx, volumeId, target)
+	if !sourceWasWritable {
+		// A readonly source next to an existing copy is the signature of a
+		// previous move whose source delete failed — that copy may be the
+		// authoritative one, serving writes the source never saw, and the
+		// copy below would tear it down. No client-side observation proves
+		// otherwise: compaction revision is shared by ordinary replicas,
+		// aggregate sizes shrink under compaction, and any snapshot can be
+		// invalidated by a write right after it. Fail closed on any existing
+		// copy or unknown state; the operator deletes one side explicitly.
+		if !targetStateKnown {
+			err = fmt.Errorf("%w: cannot determine whether %s already holds a copy of volume %d; refusing to overwrite it while the source is readonly", ErrSourceKeptReadonly, target, volumeId)
+			return err
+		}
+		if targetHadVolume {
+			err = fmt.Errorf("%w: volume %d already has a copy on %s while the source on %s is readonly — likely a previous move that did not finish; delete the source to keep that copy, or delete the copy before re-running the move", ErrSourceKeptReadonly, volumeId, target, source)
+			return err
+		}
+	}
+
+	opts.Progress(20, fmt.Sprintf("copying volume %d from %s to %s", volumeId, source, target))
+	fmt.Fprintf(opts.Writer, "copying volume %d from %s to %s\n", volumeId, source, target)
+	copyStarted = true
+	lastAppendAtNs, err := m.copyVolumeData(ctx, volumeId, source, target, opts.DiskType, opts.IoBytePerSecond, opts.Writer)
+	if err != nil {
+		return fmt.Errorf("copy volume %d from %s to %s: %v", volumeId, source, target, err)
+	}
+	copyCompleted = true
+
+	opts.Progress(70, fmt.Sprintf("tailing volume %d from %s to %s", volumeId, source, target))
+	fmt.Fprintf(opts.Writer, "tailing volume %d from %s to %s\n", volumeId, source, target)
+	tailFailed := false
+	if tailErr := m.TailVolume(ctx, volumeId, source, target, lastAppendAtNs, opts.IdleTimeout); tailErr != nil {
+		// A tail failure is tolerable only when the volume was already readonly
+		// before this move began: frozen for the whole copy, it should have no
+		// in-flight writes for the tail to drain — and the stability check
+		// below still proves it. A volume this move froze can have stragglers
+		// admitted just before the freeze that only the tail delivers, so
+		// losing the tail there must abort the move.
+		if sourceWasWritable {
+			return fmt.Errorf("tail volume %d from %s to %s: %v", volumeId, source, target, tailErr)
+		}
+		tailFailed = true
+		fmt.Fprintf(opts.Writer, "tail volume %d from %s to %s: %v\n", volumeId, source, target, tailErr)
+		glog.Warningf("tail volume %d from %s to %s: %v", volumeId, source, target, tailErr)
+	}
+
+	// Verify before the point of no return: the source is deleted only when the
+	// target holds at least everything the source does. The source status is
+	// read here, after the tail — a write in flight when the source was frozen
+	// can still land after an earlier read, and a stale snapshot would wave
+	// through an incomplete target.
+	opts.Progress(85, fmt.Sprintf("verifying volume %d on %s", volumeId, target))
+	sourceStatus, err := m.ReadVolumeFileStatus(ctx, volumeId, source)
+	if err != nil {
+		return fmt.Errorf("read volume %d status on %s: %v", volumeId, source, err)
+	}
+	if tailFailed {
+		// A successful tail proves the source held still for the idle window; a
+		// tolerated tail failure proved nothing, so substitute the same drain
+		// barrier here: the source status must not change across the window.
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(opts.IdleTimeout):
+		}
+		secondStatus, statusErr := m.ReadVolumeFileStatus(ctx, volumeId, source)
+		if statusErr != nil {
+			return fmt.Errorf("re-read volume %d status on %s: %v", volumeId, source, statusErr)
+		}
+		if secondStatus.DatFileSize != sourceStatus.DatFileSize || secondStatus.IdxFileSize != sourceStatus.IdxFileSize || secondStatus.FileCount != sourceStatus.FileCount {
+			return fmt.Errorf("volume %d on %s is still changing after a failed tail; aborting the move", volumeId, source)
+		}
+		sourceStatus = secondStatus
+	}
+	targetStatus, err := m.ReadVolumeFileStatus(ctx, volumeId, target)
+	if err != nil {
+		return fmt.Errorf("verify volume %d on target %s before deleting source: %v", volumeId, target, err)
+	}
+	if err = verifyTargetNotBehind(volumeId, sourceStatus, targetStatus); err != nil {
+		return err
+	}
+	if targetStatus.DatFileSize > sourceStatus.DatFileSize || targetStatus.IdxFileSize > sourceStatus.IdxFileSize || targetStatus.FileCount > sourceStatus.FileCount {
+		// The target announced itself writable when the copy mounted it, so
+		// clients may have written to it during the tail. Those writes live on
+		// the surviving copy; committing keeps them.
+		fmt.Fprintf(opts.Writer, "volume %d on %s has writes beyond the source; they stay with the moved volume\n", volumeId, target)
+	}
+
+	opts.Progress(90, fmt.Sprintf("deleting volume %d from %s", volumeId, source))
+	fmt.Fprintf(opts.Writer, "deleting volume %d from %s\n", volumeId, source)
+	sourceDeleteStarted = true
+	if err = m.DeleteVolume(ctx, volumeId, source, false, true); err != nil {
+		return fmt.Errorf("delete volume %d from %s: %v", volumeId, source, err)
+	}
+
+	opts.Progress(100, fmt.Sprintf("moved volume %d from %s to %s", volumeId, source, target))
+	fmt.Fprintf(opts.Writer, "moved volume %d from %s to %s\n", volumeId, source, target)
+	return nil
+}
+
+// verifyTargetNotBehind fails when the target holds less than the (frozen)
+// source — the copy or tail missed data and deleting the source would lose it.
+// A target that is ahead is not an error: the target serves writes during the
+// tail, and those belong to the copy that survives the move.
+func verifyTargetNotBehind(volumeId needle.VolumeId, source, target *volume_server_pb.ReadVolumeFileStatusResponse) error {
+	if target.DatFileSize < source.DatFileSize {
+		return fmt.Errorf("volume %d target is behind the source: .dat %d < %d bytes", volumeId, target.DatFileSize, source.DatFileSize)
+	}
+	if target.IdxFileSize < source.IdxFileSize {
+		return fmt.Errorf("volume %d target is behind the source: .idx %d < %d bytes", volumeId, target.IdxFileSize, source.IdxFileSize)
+	}
+	if target.FileCount < source.FileCount {
+		return fmt.Errorf("volume %d target is behind the source: %d < %d files", volumeId, target.FileCount, source.FileCount)
+	}
+	return nil
+}
+
+// CopyVolume freezes the volume on source, copies it to target, and reports the
+// stamp of the last entry copied. restoreWritable also restores the source's
+// writability on success, for copies that leave the source serving; a failed
+// copy always undoes the freeze this call added.
+func (m *Mover) CopyVolume(ctx context.Context, volumeId needle.VolumeId, source, target pb.ServerAddress, diskType string, ioBytePerSecond int64, restoreWritable bool, writer io.Writer) (lastAppendAtNs uint64, err error) {
+	if writer == nil {
+		writer = io.Discard
+	}
+	if SameServer(source, target) {
+		return 0, fmt.Errorf("refusing to copy volume %d onto its own server %s", volumeId, source)
+	}
+	// The copy is non-destructive, so a readonly-reporting source is taken as
+	// is (force=false): missed concurrent deletes only make the new replica
+	// trivially stale, and hard-marking here could pin a transiently readonly
+	// (e.g. low-disk) source readonly with nothing to gate on it.
+	sourceWasWritable, err := m.ensureVolumeReadonly(ctx, volumeId, source, false)
+	defer func() {
+		if sourceWasWritable && (err != nil || restoreWritable) {
+			m.restoreVolumeWritable(volumeId, source)
+		}
+	}()
+	if err != nil {
+		return 0, err
+	}
+	return m.copyVolumeData(ctx, volumeId, source, target, diskType, ioBytePerSecond, writer)
+}
+
+// ensureVolumeReadonly freezes the volume on server and reports whether it was
+// writable beforehand — only then may a failure path undo the freeze, since a
+// volume that was already readonly (e.g. full, or operator-set) must stay so.
+// With force, the mark is issued even when the status already reports
+// readonly: that answer also covers transient low-disk state and the
+// readonly-but-can-delete flag, and neither blocks needle deletes, so only the
+// hard mark makes the volume immutable — required before deleting the source
+// of a move. Without force, a readonly-reporting volume is left untouched.
+func (m *Mover) ensureVolumeReadonly(ctx context.Context, volumeId needle.VolumeId, server pb.ServerAddress, force bool) (wasWritable bool, err error) {
+	err = m.withClient(false, server, func(client volume_server_pb.VolumeServerClient) error {
+		resp, statusErr := client.VolumeStatus(ctx, &volume_server_pb.VolumeStatusRequest{
+			VolumeId: uint32(volumeId),
+		})
+		if statusErr != nil {
+			return statusErr
+		}
+		wasWritable = !resp.IsReadOnly
+		if !wasWritable && !force {
+			return nil
+		}
+		_, readonlyErr := client.VolumeMarkReadonly(ctx, &volume_server_pb.VolumeMarkReadonlyRequest{
+			VolumeId: uint32(volumeId),
+			Persist:  false,
+		})
+		return readonlyErr
+	})
+	return
+}
+
+// probeVolume reports whether server currently has the volume, and whether the
+// answer is trustworthy. The Go server answers a missing volume with a plain
+// error (gRPC code Unknown — the only way this RPC fails there); the Rust
+// server answers with codes.NotFound. Both mean the server responded and the
+// volume is absent. A transport-level failure means the state is unknown, and
+// callers deciding to delete must treat unknown as hands-off.
+func (m *Mover) probeVolume(ctx context.Context, volumeId needle.VolumeId, server pb.ServerAddress) (exists bool, known bool) {
+	_, err := m.ReadVolumeFileStatus(ctx, volumeId, server)
+	if err == nil {
+		return true, true
+	}
+	if s, ok := status.FromError(err); ok && (s.Code() == codes.Unknown || s.Code() == codes.NotFound) {
+		return false, true
+	}
+	return false, false
+}
+
+// restoreVolumeWritable undoes a readonly mark after a failed move. It runs on
+// its own deadline so an abort via cancelled context still restores the source,
+// and only logs a failure — the caller is already returning the move error.
+func (m *Mover) restoreVolumeWritable(volumeId needle.VolumeId, server pb.ServerAddress) {
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+	if err := m.MarkVolumeWritable(ctx, volumeId, server, true, false); err != nil {
+		glog.Warningf("restore volume %d writable on %s: %v", volumeId, server, err)
+	}
+}
+
+// copyVolumeData streams a VolumeCopy on the target, which pulls the volume
+// from source and mounts it, and returns the stamp of the last entry copied
+// for the tail phase.
+func (m *Mover) copyVolumeData(ctx context.Context, volumeId needle.VolumeId, source, target pb.ServerAddress, diskType string, ioBytePerSecond int64, writer io.Writer) (lastAppendAtNs uint64, err error) {
+	// The target dials the embedded source itself; a malformed one would
+	// abort the target server, not this client.
+	if err = checkDialable(source); err != nil {
+		return 0, err
+	}
+	err = m.withClient(true, target, func(client volume_server_pb.VolumeServerClient) error {
+		stream, replicateErr := client.VolumeCopy(ctx, &volume_server_pb.VolumeCopyRequest{
+			VolumeId:        uint32(volumeId),
+			SourceDataNode:  string(source),
+			DiskType:        diskType,
+			IoBytePerSecond: ioBytePerSecond,
+		})
+		if replicateErr != nil {
+			return replicateErr
+		}
+		for {
+			resp, recvErr := stream.Recv()
+			if recvErr != nil {
+				if recvErr == io.EOF {
+					break
+				}
+				return recvErr
+			}
+			if resp.LastAppendAtNs != 0 {
+				lastAppendAtNs = resp.LastAppendAtNs
+			} else {
+				fmt.Fprintf(writer, "%s => %s volume %d processed %s\n", source, target, volumeId, util.BytesToHumanReadable(uint64(resp.ProcessedBytes)))
+			}
+		}
+		return nil
+	})
+	return
+}
+
+// TailVolume has the target follow the source's appends since sinceNs until the
+// source stays idle for idleTimeout, draining requests in flight when the move
+// froze the source.
+func (m *Mover) TailVolume(ctx context.Context, volumeId needle.VolumeId, source, target pb.ServerAddress, sinceNs uint64, idleTimeout time.Duration) error {
+	// The target dials the embedded source itself; a malformed one would
+	// abort the target server, not this client.
+	if err := checkDialable(source); err != nil {
+		return err
+	}
+	return m.withClient(true, target, func(client volume_server_pb.VolumeServerClient) error {
+		_, replicateErr := client.VolumeTailReceiver(ctx, &volume_server_pb.VolumeTailReceiverRequest{
+			VolumeId:           uint32(volumeId),
+			SinceNs:            sinceNs,
+			IdleTimeoutSeconds: uint32(idleTimeout.Seconds()),
+			SourceVolumeServer: string(source),
+		})
+		return replicateErr
+	})
+}
+
+// ReadVolumeFileStatus reads the volume's file sizes and needle count on server.
+func (m *Mover) ReadVolumeFileStatus(ctx context.Context, volumeId needle.VolumeId, server pb.ServerAddress) (resp *volume_server_pb.ReadVolumeFileStatusResponse, err error) {
+	err = m.withClient(false, server, func(client volume_server_pb.VolumeServerClient) error {
+		var statusErr error
+		resp, statusErr = client.ReadVolumeFileStatus(ctx, &volume_server_pb.ReadVolumeFileStatusRequest{
+			VolumeId: uint32(volumeId),
+		})
+		return statusErr
+	})
+	return
+}
+
+// DeleteVolume removes the volume from server. When keepRemoteData is true, the
+// cloud-tier object backing the volume is left intact — used on the source side
+// of a move where another server is taking over the same .vif.
+func (m *Mover) DeleteVolume(ctx context.Context, volumeId needle.VolumeId, server pb.ServerAddress, onlyEmpty bool, keepRemoteData bool) error {
+	return m.withClient(false, server, func(client volume_server_pb.VolumeServerClient) error {
+		_, deleteErr := client.VolumeDelete(ctx, &volume_server_pb.VolumeDeleteRequest{
+			VolumeId:       uint32(volumeId),
+			OnlyEmpty:      onlyEmpty,
+			KeepRemoteData: keepRemoteData,
+		})
+		return deleteErr
+	})
+}
+
+// MarkVolumeWritable marks the volume writable (or readonly when writable is
+// false, persisted per persist) on server.
+func (m *Mover) MarkVolumeWritable(ctx context.Context, volumeId needle.VolumeId, server pb.ServerAddress, writable, persist bool) (err error) {
+	return m.withClient(false, server, func(client volume_server_pb.VolumeServerClient) error {
+		if writable {
+			_, err = client.VolumeMarkWritable(ctx, &volume_server_pb.VolumeMarkWritableRequest{
+				VolumeId: uint32(volumeId),
+			})
+		} else {
+			_, err = client.VolumeMarkReadonly(ctx, &volume_server_pb.VolumeMarkReadonlyRequest{
+				VolumeId: uint32(volumeId),
+				Persist:  persist,
+			})
+		}
+		return err
+	})
+}
+
+// ReplicateVolume copies a volume from source to target without touching the
+// source — the replica-creation half of a move.
+func (m *Mover) ReplicateVolume(ctx context.Context, volumeId needle.VolumeId, source, target pb.ServerAddress, diskType string, writer io.Writer) error {
+	if writer == nil {
+		writer = io.Discard
+	}
+	if SameServer(source, target) {
+		return fmt.Errorf("refusing to replicate volume %d onto its own server %s", volumeId, source)
+	}
+	// The target dials the embedded source itself; a malformed one would
+	// abort the target server, not this client.
+	if err := checkDialable(source); err != nil {
+		return err
+	}
+	return m.withClient(false, target, func(client volume_server_pb.VolumeServerClient) error {
+		stream, replicateErr := client.VolumeCopy(ctx, &volume_server_pb.VolumeCopyRequest{
+			VolumeId:       uint32(volumeId),
+			SourceDataNode: string(source),
+			DiskType:       diskType,
+		})
+		if replicateErr != nil {
+			return replicateErr
+		}
+		for {
+			resp, recvErr := stream.Recv()
+			if recvErr != nil {
+				if recvErr == io.EOF {
+					break
+				}
+				return recvErr
+			}
+			if resp.ProcessedBytes > 0 {
+				fmt.Fprintf(writer, "volume %d processed %s bytes\n", volumeId, util.BytesToHumanReadable(uint64(resp.ProcessedBytes)))
+			}
+		}
+		return nil
+	})
+}
+
+// ConfigureVolumeReplication sets the replication setting on the volume at server.
+func (m *Mover) ConfigureVolumeReplication(ctx context.Context, volumeId needle.VolumeId, server pb.ServerAddress, replication string) error {
+	return m.withClient(false, server, func(client volume_server_pb.VolumeServerClient) error {
+		resp, configureErr := client.VolumeConfigure(ctx, &volume_server_pb.VolumeConfigureRequest{
+			VolumeId:    uint32(volumeId),
+			Replication: replication,
+		})
+		if configureErr != nil {
+			return configureErr
+		}
+		if resp.Error != "" {
+			return errors.New(resp.Error)
+		}
+		return nil
+	})
+}

weed/operation/volume_move/volume_move_test.go

@@ -0,0 +1,621 @@
+package volume_move
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb"
+	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
+	"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/credentials/insecure"
+	"google.golang.org/grpc/status"
+)
+
+const (
+	srcAddr = pb.ServerAddress("src:8080")
+	dstAddr = pb.ServerAddress("dst:8080")
+)
+
+func volumeStatus(datSize, idxSize, fileCount uint64) *volume_server_pb.ReadVolumeFileStatusResponse {
+	return &volume_server_pb.ReadVolumeFileStatusResponse{
+		DatFileSize: datSize,
+		IdxFileSize: idxSize,
+		FileCount:   fileCount,
+	}
+}
+
+func TestSameServer(t *testing.T) {
+	if !SameServer("node:8080", "node:8080.18080") {
+		t.Error("one server written with and without an explicit grpc port compared unequal")
+	}
+	// Test harnesses run several servers on HTTP port 0 with distinct grpc
+	// ports; the grpc endpoint is the server's identity.
+	if SameServer("127.0.0.1:0.36825", "127.0.0.1:0.38609") {
+		t.Error("two servers sharing a degenerate HTTP address compared equal")
+	}
+	// Malformed addresses (an unvalidated -source/-target flag) must compare
+	// without touching the fatal ToGrpcAddress parser.
+	if !SameServer("node:abc", "node:abc") {
+		t.Error("identical malformed addresses compared unequal")
+	}
+	if SameServer("node:abc", "node:8080") {
+		t.Error("malformed and well-formed addresses compared equal")
+	}
+	if SameServer("", "node:8080") {
+		t.Error("empty and well-formed addresses compared equal")
+	}
+}
+
+func TestCheckDialable(t *testing.T) {
+	// Rejected: the class the fatal parser aborts on, plus dotted forms with
+	// a non-numeric component — those canonicalize differently across dial
+	// paths ("node:8080.bad" dials as node:bad on one and node:18080 on the
+	// other, silently aliasing another server).
+	for _, addr := range []pb.ServerAddress{"node:abc", "host.domain.com:x8080", "[::1]:abc",
+		"node:8080.bad", "node:bad.8080", "node:.bad", "node:8080."} {
+		if err := checkDialable(addr); err == nil {
+			t.Errorf("%q accepted although it is unsafe to dial", addr)
+		}
+	}
+	// Accepted: normalized cleanly, or handed to the dialer untouched where
+	// a bad address fails as an ordinary error.
+	for _, addr := range []pb.ServerAddress{"node:8080", "node:8080.18080", "127.0.0.1:0.36825", "", "node", "node:"} {
+		if err := checkDialable(addr); err != nil {
+			t.Errorf("%q rejected: %v", addr, err)
+		}
+	}
+}
+
+func TestEmbeddedSourceAddressValidated(t *testing.T) {
+	// The target dials the source embedded in copy/tail requests through the
+	// same fatal parser; the client must reject a malformed source before
+	// issuing any RPC.
+	bad := pb.ServerAddress("node:abc")
+
+	ops := map[string]func(m *Mover) error{
+		"ReplicateVolume": func(m *Mover) error {
+			return m.ReplicateVolume(context.Background(), 7, bad, dstAddr, "", nil)
+		},
+		"TailVolume": func(m *Mover) error {
+			return m.TailVolume(context.Background(), 7, bad, dstAddr, 0, time.Second)
+		},
+		"CopyAndMountEcShards": func(m *Mover) error {
+			return m.CopyAndMountEcShards(context.Background(), 7, "c1", []erasure_coding.ShardId{3}, bad, dstAddr, 0, nil)
+		},
+		"MoveEcShards": func(m *Mover) error {
+			move := ecMove(3)
+			move.Source = bad
+			return m.MoveEcShards(context.Background(), move, EcMoveOptions{})
+		},
+	}
+	for name, op := range ops {
+		cluster := newFakeCluster()
+		err := op(cluster.mover())
+		if err == nil || !strings.Contains(err.Error(), "invalid volume server address") {
+			t.Errorf("%s: expected invalid-address error, got: %v", name, err)
+		}
+		if len(cluster.callList()) != 0 {
+			t.Errorf("%s: RPCs issued with a malformed embedded source: %v", name, cluster.callList())
+		}
+	}
+}
+
+func TestNewMoverRejectsMalformedAddressWithoutDialing(t *testing.T) {
+	// The production dial path must fail the move, not the process, when a
+	// caller passes an unvalidated malformed address.
+	mover := NewMover(grpc.WithTransportCredentials(insecure.NewCredentials()))
+	err := mover.LiveMoveVolume(context.Background(), 7, "node:abc", "node:8080", VolumeMoveOptions{})
+	if err == nil || !strings.Contains(err.Error(), "invalid volume server address") {
+		t.Fatalf("expected invalid-address error, got: %v", err)
+	}
+}
+
+func assertCalls(t *testing.T, got, want []string) {
+	t.Helper()
+	if fmt.Sprint(got) != fmt.Sprint(want) {
+		t.Fatalf("RPC sequence mismatch:\n got:  %v\n want: %v", got, want)
+	}
+}
+
+func TestLiveMoveVolumeSequence(t *testing.T) {
+	cluster := newFakeCluster()
+	cluster.lastAppendAtNs = 12345
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(1000, 100, 10)
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{
+		DiskType:        "ssd",
+		IoBytePerSecond: 42,
+	})
+	if err != nil {
+		t.Fatalf("LiveMoveVolume: %v", err)
+	}
+
+	// The source status read must come after the tail: a write in flight when
+	// the source was frozen can land after an earlier read, and verifying
+	// against that stale snapshot would miss a missed tail. The first target
+	// status read probes whether the target already held the volume, so a
+	// failed copy never cleans up someone else's replica.
+	assertCalls(t, cluster.callList(), []string{
+		"src:8080 VolumeStatus",
+		"src:8080 VolumeMarkReadonly",
+		"dst:8080 ReadVolumeFileStatus",
+		"dst:8080 VolumeCopy",
+		"dst:8080 VolumeTailReceiver",
+		"src:8080 ReadVolumeFileStatus",
+		"dst:8080 ReadVolumeFileStatus",
+		"src:8080 VolumeDelete",
+	})
+
+	copyReq := cluster.copyReqs[0]
+	if copyReq.DiskType != "ssd" || copyReq.IoBytePerSecond != 42 || copyReq.SourceDataNode != string(srcAddr) {
+		t.Errorf("copy request not propagated: %+v", copyReq)
+	}
+	deleteReq := cluster.deleteReqs[0]
+	if deleteReq.OnlyEmpty || !deleteReq.KeepRemoteData {
+		t.Errorf("source delete must keep remote data and not be only-empty: %+v", deleteReq)
+	}
+}
+
+func TestLiveMoveVolumeVerifyMismatchKeepsSource(t *testing.T) {
+	cluster := newFakeCluster()
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(900, 100, 10)
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err == nil || !strings.Contains(err.Error(), "behind the source") {
+		t.Fatalf("expected target-behind error, got: %v", err)
+	}
+
+	calls := cluster.callList()
+	for _, call := range calls {
+		if call == "src:8080 VolumeDelete" {
+			t.Fatalf("source deleted despite verification failure: %v", calls)
+		}
+	}
+	// The incomplete target copy is removed so the restored source stays the
+	// only replica, then the source is made writable again.
+	if fmt.Sprint(calls[len(calls)-2:]) != fmt.Sprint([]string{"dst:8080 VolumeDelete", "src:8080 VolumeMarkWritable"}) {
+		t.Fatalf("expected target cleanup then source restore, got: %v", calls)
+	}
+}
+
+func TestLiveMoveVolumeAbortKeepsSourceReadonlyWhenTargetCleanupFails(t *testing.T) {
+	// Restoring the source while the stale target stays mounted risks
+	// divergent replicas; the source stays readonly until a re-run recovers.
+	cluster := newFakeCluster()
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(900, 100, 10)
+	cluster.errs["dst:8080 VolumeDelete"] = errors.New("target unreachable")
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err == nil {
+		t.Fatal("expected verification failure")
+	}
+	if !errors.Is(err, ErrSourceKeptReadonly) {
+		t.Fatalf("error does not mark the source as deliberately kept readonly: %v", err)
+	}
+
+	for _, call := range cluster.callList() {
+		if call == "src:8080 VolumeMarkWritable" {
+			t.Fatalf("source made writable with the stale target still mounted: %v", cluster.callList())
+		}
+	}
+}
+
+func TestLiveMoveVolumeAlreadyReadonlyStaysReadonly(t *testing.T) {
+	cluster := newFakeCluster()
+	cluster.readonly[string(srcAddr)] = true
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.errs["dst:8080 VolumeCopy"] = errors.New("copy failed")
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err == nil {
+		t.Fatal("expected copy failure")
+	}
+
+	// The hard mark is issued even for a readonly-reporting volume: that
+	// status also covers low-disk and readonly-but-can-delete states, which
+	// still accept needle deletes.
+	marked := false
+	for _, call := range cluster.callList() {
+		if call == "src:8080 VolumeMarkReadonly" {
+			marked = true
+		}
+		// A volume that was readonly before the move (e.g. full) must stay so.
+		if call == "src:8080 VolumeMarkWritable" {
+			t.Fatal("made an already-readonly volume writable after a failed move")
+		}
+	}
+	if !marked {
+		t.Fatalf("readonly-reporting source not hard-frozen: %v", cluster.callList())
+	}
+}
+
+func TestLiveMoveVolumePreFrozenAbortCleansTarget(t *testing.T) {
+	// tiering.run freezes replicas itself and thaws them after a failed move,
+	// so even a move that did not do the freezing must remove its target copy
+	// on abort — a stale mounted target plus a thawed source is divergence.
+	cluster := newFakeCluster()
+	cluster.readonly[string(srcAddr)] = true
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(900, 100, 10)
+	cluster.statusFailures[string(dstAddr)] = 1 // no volume on the target before the copy
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err == nil {
+		t.Fatal("expected verification failure")
+	}
+
+	cleaned := false
+	for _, call := range cluster.callList() {
+		if call == "dst:8080 VolumeDelete" {
+			cleaned = true
+		}
+		if call == "src:8080 VolumeMarkWritable" {
+			t.Fatalf("thawed a source this move did not freeze: %v", cluster.callList())
+		}
+	}
+	if !cleaned {
+		t.Fatalf("incomplete target copy not cleaned up: %v", cluster.callList())
+	}
+}
+
+func TestLiveMoveVolumeCopyErrorCleansMountedTarget(t *testing.T) {
+	// The server can finish the copy and mount the target even when the client
+	// loses the stream; the abort probes the target and cleans up the copy.
+	cluster := newFakeCluster()
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(1000, 100, 10)
+	cluster.statusFailures[string(dstAddr)] = 1 // no volume on the target before the copy
+	cluster.errs["dst:8080 VolumeCopy"] = errors.New("stream lost")
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err == nil {
+		t.Fatal("expected copy failure")
+	}
+
+	calls := cluster.callList()
+	if fmt.Sprint(calls[len(calls)-2:]) != fmt.Sprint([]string{"dst:8080 VolumeDelete", "src:8080 VolumeMarkWritable"}) {
+		t.Fatalf("expected target cleanup then source restore, got: %v", calls)
+	}
+}
+
+func TestLiveMoveVolumeCopyErrorUnknownTargetKeepsSourceReadonly(t *testing.T) {
+	// The pre-copy probe failed at the transport level, so whether the target
+	// held a replica before the move is unknown; the failed copy may still
+	// have mounted a complete copy there (the server can finish after the
+	// client loses the stream). Deleting it risks a healthy pre-existing
+	// replica, and reopening the source beside it risks two writable replicas
+	// taking divergent writes — so the source stays readonly.
+	cluster := newFakeCluster()
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(1000, 100, 10)
+	cluster.statusFailures[string(dstAddr)] = 1
+	cluster.statusFailureErr = status.Error(codes.Unavailable, "connection refused")
+	cluster.errs["dst:8080 VolumeCopy"] = errors.New("stream lost")
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err == nil || !errors.Is(err, ErrSourceKeptReadonly) {
+		t.Fatalf("expected kept-readonly failure, got: %v", err)
+	}
+
+	calls := cluster.callList()
+	for _, call := range calls {
+		if call == "dst:8080 VolumeDelete" {
+			t.Fatalf("deleted the target with its prior state unknown: %v", calls)
+		}
+		if call == "src:8080 VolumeMarkWritable" {
+			t.Fatalf("reopened the source beside a possibly-mounted copy: %v", calls)
+		}
+	}
+}
+
+func TestLiveMoveVolumeCopyErrorLeavesPreexistingTarget(t *testing.T) {
+	// A target that held the volume before the move is someone else's replica;
+	// a failed copy must not delete it.
+	cluster := newFakeCluster()
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(1000, 100, 10)
+	cluster.errs["dst:8080 VolumeCopy"] = errors.New("copy failed")
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err == nil || !errors.Is(err, ErrSourceKeptReadonly) {
+		t.Fatalf("expected kept-readonly failure, got: %v", err)
+	}
+
+	calls := cluster.callList()
+	for _, call := range calls {
+		if call == "dst:8080 VolumeDelete" {
+			t.Fatalf("deleted a pre-existing target replica after a failed copy: %v", calls)
+		}
+		if call == "src:8080 VolumeMarkWritable" {
+			t.Fatalf("reopened the source with a copy of unprovable origin on the target: %v", calls)
+		}
+	}
+}
+
+func TestLiveMoveVolumeTailErrorAborts(t *testing.T) {
+	cluster := newFakeCluster()
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(1000, 100, 10)
+	cluster.errs["dst:8080 VolumeTailReceiver"] = errors.New("tail failed")
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err == nil || !strings.Contains(err.Error(), "tail volume") {
+		t.Fatalf("expected tail error, got: %v", err)
+	}
+
+	calls := cluster.callList()
+	for _, call := range calls {
+		if call == "src:8080 VolumeDelete" {
+			t.Fatalf("source deleted despite tail failure: %v", calls)
+		}
+	}
+	if fmt.Sprint(calls[len(calls)-2:]) != fmt.Sprint([]string{"dst:8080 VolumeDelete", "src:8080 VolumeMarkWritable"}) {
+		t.Fatalf("expected target cleanup then source restore, got: %v", calls)
+	}
+}
+
+func TestLiveMoveVolumeTailErrorToleratedForReadonlySource(t *testing.T) {
+	// A volume that was already readonly before the move was frozen for the
+	// whole copy, so a failed tail has nothing to deliver.
+	cluster := newFakeCluster()
+	cluster.readonly[string(srcAddr)] = true
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(1000, 100, 10)
+	cluster.statusFailures[string(dstAddr)] = 1 // no volume on the target before the copy
+	cluster.errs["dst:8080 VolumeTailReceiver"] = errors.New("tail failed")
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{IdleTimeout: time.Millisecond})
+	if err != nil {
+		t.Fatalf("LiveMoveVolume on readonly source with failed tail: %v", err)
+	}
+
+	calls := cluster.callList()
+	if calls[len(calls)-1] != "src:8080 VolumeDelete" {
+		t.Fatalf("move did not complete with source delete: %v", calls)
+	}
+}
+
+func TestLiveMoveVolumeTailFailureAbortsWhenSourceStillChanging(t *testing.T) {
+	// A tolerated tail failure substitutes a drain barrier: the source status
+	// must hold still across the idle window, or the move aborts.
+	cluster := newFakeCluster()
+	cluster.readonly[string(srcAddr)] = true
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(1000, 100, 10)
+	cluster.statusFailures[string(dstAddr)] = 1 // no volume on the target before the copy
+	cluster.statusSeq[string(srcAddr)] = []*volume_server_pb.ReadVolumeFileStatusResponse{
+		volumeStatus(1000, 100, 10),
+		volumeStatus(1024, 116, 11), // a straggler landed between the two reads
+	}
+	cluster.errs["dst:8080 VolumeTailReceiver"] = errors.New("tail failed")
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{IdleTimeout: time.Millisecond})
+	if err == nil || !strings.Contains(err.Error(), "still changing") {
+		t.Fatalf("expected still-changing abort, got: %v", err)
+	}
+
+	for _, call := range cluster.callList() {
+		if call == "src:8080 VolumeDelete" {
+			t.Fatalf("source deleted while still changing: %v", cluster.callList())
+		}
+	}
+}
+
+func TestLiveMoveVolumeSourceDeleteFailureKeepsBothReadonly(t *testing.T) {
+	// Past verification the target is complete and may hold new writes, and
+	// the failed delete may or may not have reached the source. Deleting the
+	// target or reopening the source could fork the volume; keep everything
+	// readonly for a re-run.
+	cluster := newFakeCluster()
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(1000, 100, 10)
+	cluster.errs["src:8080 VolumeDelete"] = errors.New("delete timed out")
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err == nil {
+		t.Fatal("expected source delete failure")
+	}
+	if !errors.Is(err, ErrSourceKeptReadonly) {
+		t.Fatalf("error does not mark the source as deliberately kept readonly: %v", err)
+	}
+
+	for _, call := range cluster.callList() {
+		if call == "dst:8080 VolumeDelete" {
+			t.Fatalf("deleted the verified target copy after a failed source delete: %v", cluster.callList())
+		}
+		if call == "src:8080 VolumeMarkWritable" {
+			t.Fatalf("restored source writability next to a mounted verified copy: %v", cluster.callList())
+		}
+	}
+}
+
+func TestLiveMoveVolumeRefusesRecopyOverCompleteTarget(t *testing.T) {
+	// A readonly source next to a target already holding a copy that is not
+	// behind it is the signature of a previous move whose source delete
+	// failed; recopying would tear down the possibly-authoritative target.
+	cluster := newFakeCluster()
+	cluster.readonly[string(srcAddr)] = true
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(1000, 100, 10)
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err == nil || !strings.Contains(err.Error(), "already has a copy") {
+		t.Fatalf("expected recopy refusal, got: %v", err)
+	}
+	if !errors.Is(err, ErrSourceKeptReadonly) {
+		t.Fatalf("refusal does not mark the source as deliberately kept readonly: %v", err)
+	}
+
+	for _, call := range cluster.callList() {
+		if call == "dst:8080 VolumeCopy" || call == "dst:8080 VolumeDelete" {
+			t.Fatalf("touched the possibly-authoritative target: %v", cluster.callList())
+		}
+	}
+}
+
+func TestLiveMoveVolumeReadonlySourceUnknownTargetRefused(t *testing.T) {
+	// With a readonly source, an unreachable target probe must fail closed:
+	// the target could hold the authoritative copy of a previous move.
+	cluster := newFakeCluster()
+	cluster.readonly[string(srcAddr)] = true
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.statusFailures[string(dstAddr)] = 1
+	cluster.statusFailureErr = status.Error(codes.Unavailable, "connection refused")
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err == nil || !errors.Is(err, ErrSourceKeptReadonly) {
+		t.Fatalf("expected fail-closed refusal, got: %v", err)
+	}
+
+	for _, call := range cluster.callList() {
+		if call == "dst:8080 VolumeCopy" {
+			t.Fatalf("copied over a target in unknown state: %v", cluster.callList())
+		}
+	}
+}
+
+func TestLiveMoveVolumePreFrozenSourceMissingRustTargetProceeds(t *testing.T) {
+	// The Rust volume server answers a missing volume with codes.NotFound
+	// (the Go server uses a plain error, code Unknown). Both are definitive
+	// absence: with a pre-frozen source (tiering freezes before moving), a
+	// NotFound misclassified as "unknown" would refuse the move with
+	// ErrSourceKeptReadonly — and tiering callers then deliberately skip
+	// thawing, stranding every replica readonly after an ordinary move.
+	cluster := newFakeCluster()
+	cluster.readonly[string(srcAddr)] = true
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(1000, 100, 10)
+	cluster.statusFailures[string(dstAddr)] = 1
+	cluster.statusFailureErr = status.Error(codes.NotFound, "not found volume id 7")
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err != nil {
+		t.Fatalf("LiveMoveVolume to an empty Rust target: %v", err)
+	}
+
+	calls := cluster.callList()
+	if calls[len(calls)-1] != "src:8080 VolumeDelete" {
+		t.Fatalf("move did not complete with source delete: %v", calls)
+	}
+}
+
+func TestLiveMoveVolumeReadonlySourceExistingTargetRefused(t *testing.T) {
+	// No client-side observation can prove an existing copy is a stale
+	// remnant rather than the authoritative copy of an unfinished move —
+	// compaction shrinks the authoritative copy below the stale source, and
+	// any snapshot can be invalidated by a write right after it. Even a
+	// target that reads as behind is refused.
+	cluster := newFakeCluster()
+	cluster.readonly[string(srcAddr)] = true
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(900, 90, 9)
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err == nil || !errors.Is(err, ErrSourceKeptReadonly) {
+		t.Fatalf("expected fail-closed refusal, got: %v", err)
+	}
+
+	for _, call := range cluster.callList() {
+		if call == "dst:8080 VolumeCopy" || call == "dst:8080 VolumeDelete" {
+			t.Fatalf("touched a possibly-authoritative target: %v", cluster.callList())
+		}
+	}
+}
+
+func TestCopyVolumeLeavesReadonlySourceUntouched(t *testing.T) {
+	// The copy is non-destructive, so a readonly-reporting source (possibly
+	// only transiently, e.g. low disk) is not hard-marked.
+	cluster := newFakeCluster()
+	cluster.readonly[string(srcAddr)] = true
+
+	_, err := cluster.mover().CopyVolume(context.Background(), 7, srcAddr, dstAddr, "", 0, true, nil)
+	if err != nil {
+		t.Fatalf("CopyVolume: %v", err)
+	}
+
+	for _, call := range cluster.callList() {
+		if call == "src:8080 VolumeMarkReadonly" || call == "src:8080 VolumeMarkWritable" {
+			t.Fatalf("readonly source touched by a non-destructive copy: %v", cluster.callList())
+		}
+	}
+}
+
+func TestLiveMoveVolumeTargetAheadCommits(t *testing.T) {
+	// The target serves writes while the move tails; a target that got ahead
+	// holds those acknowledged writes, and the move must commit so they stay
+	// on the surviving copy.
+	cluster := newFakeCluster()
+	cluster.status[string(srcAddr)] = volumeStatus(1000, 100, 10)
+	cluster.status[string(dstAddr)] = volumeStatus(1100, 110, 11)
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err != nil {
+		t.Fatalf("LiveMoveVolume with target ahead: %v", err)
+	}
+
+	calls := cluster.callList()
+	if calls[len(calls)-1] != "src:8080 VolumeDelete" {
+		t.Fatalf("move did not commit with source delete: %v", calls)
+	}
+}
+
+func TestLiveMoveVolumeRejectsSameServer(t *testing.T) {
+	// The second target is the same server written with an explicit grpc port;
+	// the guard must see through the representation difference.
+	for _, target := range []pb.ServerAddress{srcAddr, pb.ServerAddress("src:8080.18080")} {
+		cluster := newFakeCluster()
+
+		err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, target, VolumeMoveOptions{})
+		if err == nil || !strings.Contains(err.Error(), "its own server") {
+			t.Fatalf("target %q: expected same-server rejection, got: %v", target, err)
+		}
+		if len(cluster.callList()) != 0 {
+			t.Fatalf("target %q: RPCs issued for a rejected move: %v", target, cluster.callList())
+		}
+	}
+}
+
+func TestLiveMoveVolumeReadonlyMarkFailureRestores(t *testing.T) {
+	// The marking RPC can fail after the server applied the mark (e.g. its
+	// master notification failed), so a mark error must still restore.
+	cluster := newFakeCluster()
+	cluster.errs["src:8080 VolumeMarkReadonly"] = errors.New("master notification failed")
+
+	err := cluster.mover().LiveMoveVolume(context.Background(), 7, srcAddr, dstAddr, VolumeMoveOptions{})
+	if err == nil || !strings.Contains(err.Error(), "mark volume") {
+		t.Fatalf("expected mark-readonly error, got: %v", err)
+	}
+
+	calls := cluster.callList()
+	if calls[len(calls)-1] != "src:8080 VolumeMarkWritable" {
+		t.Fatalf("source writability not restored after failed readonly mark: %v", calls)
+	}
+}
+
+func TestCopyVolumeRestoreWritable(t *testing.T) {
+	for _, restoreWritable := range []bool{true, false} {
+		cluster := newFakeCluster()
+		_, err := cluster.mover().CopyVolume(context.Background(), 7, srcAddr, dstAddr, "", 0, restoreWritable, nil)
+		if err != nil {
+			t.Fatalf("CopyVolume(restoreWritable=%v): %v", restoreWritable, err)
+		}
+		restored := false
+		for _, call := range cluster.callList() {
+			if call == "src:8080 VolumeMarkWritable" {
+				restored = true
+			}
+		}
+		if restored != restoreWritable {
+			t.Errorf("restoreWritable=%v but writability restored=%v", restoreWritable, restored)
+		}
+	}
+}

weed/shell/command_ec_common.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"os"
 	"regexp"
 	"slices"
 	"sort"
@@ -13,6 +14,7 @@ import (
 
 	"github.com/seaweedfs/seaweedfs/weed/glog"
 	"github.com/seaweedfs/seaweedfs/weed/operation"
+	"github.com/seaweedfs/seaweedfs/weed/operation/volume_move"
 	"github.com/seaweedfs/seaweedfs/weed/pb"
 	"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
 	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
@@ -389,48 +391,19 @@ func oneServerCopyAndMountEcShardsFromSource(grpcDialOption grpc.DialOption,
 	fmt.Printf("allocate %d.%v %s => %s\n", volumeId, shardIdsToCopy, existingLocation, targetServer.info.Id)
 
 	targetAddress := pb.NewServerAddressFromDataNode(targetServer.info)
-	err = operation.WithVolumeServerClient(false, targetAddress, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
-
-		if targetAddress != existingLocation {
-			fmt.Printf("copy %d.%v %s => %s\n", volumeId, shardIdsToCopy, existingLocation, targetServer.info.Id)
-			_, copyErr := volumeServerClient.VolumeEcShardsCopy(context.Background(), &volume_server_pb.VolumeEcShardsCopyRequest{
-				VolumeId:       uint32(volumeId),
-				Collection:     collection,
-				ShardIds:       erasure_coding.ShardIdsToUint32(shardIdsToCopy),
-				CopyEcxFile:    true,
-				CopyEcjFile:    true,
-				CopyVifFile:    true,
-				CopyEcsumFile:  true, // propagate the bitrot sidecar with the shards (no-op if the source has none)
-				SourceDataNode: string(existingLocation),
-				DiskId:         destDiskId,
-			})
-			if copyErr != nil {
-				return fmt.Errorf("copy %d.%v %s => %s : %v\n", volumeId, shardIdsToCopy, existingLocation, targetServer.info.Id, copyErr)
-			}
-		}
-
-		fmt.Printf("mount %d.%v on %s\n", volumeId, shardIdsToCopy, targetServer.info.Id)
-		_, mountErr := volumeServerClient.VolumeEcShardsMount(context.Background(), &volume_server_pb.VolumeEcShardsMountRequest{
-			VolumeId:   uint32(volumeId),
-			Collection: collection,
-			ShardIds:   erasure_coding.ShardIdsToUint32(shardIdsToCopy),
-		})
-		if mountErr != nil {
-			return fmt.Errorf("mount %d.%v on %s : %v\n", volumeId, shardIdsToCopy, targetServer.info.Id, mountErr)
-		}
-
-		if targetAddress != existingLocation {
-			copiedShardIds = shardIdsToCopy
-			glog.V(0).Infof("%s ec volume %d deletes shards %+v", existingLocation, volumeId, copiedShardIds)
-		}
-
-		return nil
-	})
-
+	err = volume_move.NewMover(grpcDialOption).CopyAndMountEcShards(context.Background(), volumeId, collection, shardIdsToCopy, existingLocation, targetAddress, destDiskId, os.Stdout)
 	if err != nil {
 		return
 	}
 
+	// SameServer, not ==: a representation mismatch here would report a
+	// same-server mount-in-place as a copy and have the caller delete the
+	// shards it kept.
+	if !volume_move.SameServer(targetAddress, existingLocation) {
+		copiedShardIds = shardIdsToCopy
+		glog.V(0).Infof("%s ec volume %d deletes shards %+v", existingLocation, volumeId, copiedShardIds)
+	}
+
 	return
 }
 
@@ -582,15 +555,7 @@ func sourceServerDeleteEcShards(grpcDialOption grpc.DialOption, collection strin
 
 	fmt.Printf("delete %d.%v from %s\n", volumeId, toBeDeletedShardIds, sourceLocation)
 
-	return operation.WithVolumeServerClient(false, sourceLocation, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
-		_, deleteErr := volumeServerClient.VolumeEcShardsDelete(context.Background(), &volume_server_pb.VolumeEcShardsDeleteRequest{
-			VolumeId:   uint32(volumeId),
-			Collection: collection,
-			ShardIds:   erasure_coding.ShardIdsToUint32(toBeDeletedShardIds),
-		})
-		return deleteErr
-	})
-
+	return volume_move.NewMover(grpcDialOption).DeleteEcShards(context.Background(), volumeId, collection, sourceLocation, toBeDeletedShardIds)
 }
 
 // errFullTeardownNotAcked marks a reachable server that completed the delete RPC
@@ -649,27 +614,14 @@ func unmountEcShards(grpcDialOption grpc.DialOption, volumeId needle.VolumeId, s
 
 	fmt.Printf("unmount %d.%v from %s\n", volumeId, toBeUnmountedShardIds, sourceLocation)
 
-	return operation.WithVolumeServerClient(false, sourceLocation, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
-		_, deleteErr := volumeServerClient.VolumeEcShardsUnmount(context.Background(), &volume_server_pb.VolumeEcShardsUnmountRequest{
-			VolumeId: uint32(volumeId),
-			ShardIds: erasure_coding.ShardIdsToUint32(toBeUnmountedShardIds),
-		})
-		return deleteErr
-	})
+	return volume_move.NewMover(grpcDialOption).UnmountEcShards(context.Background(), volumeId, sourceLocation, toBeUnmountedShardIds)
 }
 
 func mountEcShards(grpcDialOption grpc.DialOption, collection string, volumeId needle.VolumeId, sourceLocation pb.ServerAddress, toBeMountedShardIds []erasure_coding.ShardId) error {
 
 	fmt.Printf("mount %d.%v on %s\n", volumeId, toBeMountedShardIds, sourceLocation)
 
-	return operation.WithVolumeServerClient(false, sourceLocation, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
-		_, mountErr := volumeServerClient.VolumeEcShardsMount(context.Background(), &volume_server_pb.VolumeEcShardsMountRequest{
-			VolumeId:   uint32(volumeId),
-			Collection: collection,
-			ShardIds:   erasure_coding.ShardIdsToUint32(toBeMountedShardIds),
-		})
-		return mountErr
-	})
+	return volume_move.NewMover(grpcDialOption).MountEcShards(context.Background(), volumeId, collection, sourceLocation, toBeMountedShardIds)
 }
 
 func ceilDivide(a, b int) int {
@@ -1157,23 +1109,26 @@ func (ecb *ecBalancer) executeMove(byID map[string]*EcNode, m ecbalancer.Move) e
 	return ecb.applyShardMoveRPC(src, dst, m.Collection, vid, shardId, m.TargetDisk)
 }
 
-// applyShardMoveRPC copies a shard to the destination disk, then unmounts and
-// deletes it on the source. It does not touch the in-memory model, so it is safe
-// to run concurrently across the moves of a phase.
+// applyShardMoveRPC copies a shard to the destination disk, verifies the
+// destination registered it, then unmounts and deletes it on the source. It
+// does not touch the in-memory model, so it is safe to run concurrently across
+// the moves of a phase.
 func (ecb *ecBalancer) applyShardMoveRPC(src, dst *EcNode, collection string, vid needle.VolumeId, shardId erasure_coding.ShardId, destDiskId uint32) error {
-	grpcDialOption := ecb.commandEnv.option.GrpcDialOption
 	srcAddr := pb.NewServerAddressFromDataNode(src.info)
-	copiedShardIds, err := oneServerCopyAndMountEcShardsFromSource(grpcDialOption, dst, []erasure_coding.ShardId{shardId}, vid, collection, srcAddr, destDiskId)
-	if err != nil {
-		return err
-	}
-	if len(copiedShardIds) == 0 {
+	dstAddr := pb.NewServerAddressFromDataNode(dst.info)
+	if volume_move.SameServer(srcAddr, dstAddr) {
+		// A same-server (cross-disk) move cannot be expressed with these RPCs;
+		// leave the shard where it is.
 		return nil
 	}
-	if err := unmountEcShards(grpcDialOption, vid, srcAddr, copiedShardIds); err != nil {
-		return err
-	}
-	return sourceServerDeleteEcShards(grpcDialOption, collection, vid, srcAddr, copiedShardIds)
+	return volume_move.NewMover(ecb.commandEnv.option.GrpcDialOption).MoveEcShards(context.Background(), volume_move.EcShardMove{
+		VolumeId:   vid,
+		Collection: collection,
+		ShardIds:   []erasure_coding.ShardId{shardId},
+		Source:     srcAddr,
+		Target:     dstAddr,
+		TargetDisk: destDiskId,
+	}, volume_move.EcMoveOptions{Writer: os.Stdout})
 }
 
 // parseVolumeIdsFlag parses a comma-separated -volumeIds flag value, dropping

weed/shell/command_volume_balance.go

@@ -657,7 +657,7 @@ func moveVolume(commandEnv *CommandEnv, v *master_pb.VolumeInformationMessage, f
 	}
 	fmt.Fprintf(os.Stdout, "  moving %s volume %s%d %s => %s\n", v.DiskType, collectionPrefix, v.Id, fullNode.info.Id, emptyNode.info.Id)
 	if applyChange {
-		return LiveMoveVolume(context.Background(), commandEnv.option.GrpcDialOption, os.Stderr, needle.VolumeId(v.Id), pb.NewServerAddressFromDataNode(fullNode.info), pb.NewServerAddressFromDataNode(emptyNode.info), 5*time.Second, v.DiskType, 0, v.ReadOnly)
+		return LiveMoveVolume(context.Background(), commandEnv.option.GrpcDialOption, os.Stderr, needle.VolumeId(v.Id), pb.NewServerAddressFromDataNode(fullNode.info), pb.NewServerAddressFromDataNode(emptyNode.info), 5*time.Second, v.DiskType, 0)
 	}
 	return nil
 }

weed/shell/command_volume_copy.go

@@ -61,6 +61,6 @@ func (c *commandVolumeCopy) Do(args []string, commandEnv *CommandEnv, writer io.
 		return fmt.Errorf("source and target volume servers are the same!")
 	}
 
-	_, _, err = copyVolume(context.Background(), commandEnv.option.GrpcDialOption, writer, volumeId, sourceVolumeServer, targetVolumeServer, "", 0, true)
+	_, err = copyVolume(context.Background(), commandEnv.option.GrpcDialOption, writer, volumeId, sourceVolumeServer, targetVolumeServer, "", 0, true)
 	return
 }

weed/shell/command_volume_merge.go

@@ -187,7 +187,7 @@ func (c *commandVolumeMerge) Do(args []string, commandEnv *CommandEnv, writer io
 
 	for i, replica := range replicas {
 		sourceServer := pb.NewServerAddressFromDataNode(replica.location.dataNode)
-		if _, _, err = copyVolume(context.Background(), commandEnv.option.GrpcDialOption, writer, volumeId, targetServer, sourceServer, "", 0, false); err != nil {
+		if _, err = copyVolume(context.Background(), commandEnv.option.GrpcDialOption, writer, volumeId, targetServer, sourceServer, "", 0, false); err != nil {
 			return fmt.Errorf("rebuild replica %d/%d on %s from merged volume %d: %w; merged copy kept on %s, re-run to finish", i+1, len(replicas), sourceServer, volumeId, err, targetServer)
 		}
 	}

weed/shell/command_volume_move.go

@@ -2,20 +2,18 @@ package shell
 
 import (
 	"context"
-	"errors"
 	"flag"
 	"fmt"
 	"io"
-	"log"
 	"time"
 
 	"github.com/seaweedfs/seaweedfs/weed/pb"
 	"github.com/seaweedfs/seaweedfs/weed/wdclient"
 
 	"github.com/seaweedfs/seaweedfs/weed/operation"
+	"github.com/seaweedfs/seaweedfs/weed/operation/volume_move"
 	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
 	"github.com/seaweedfs/seaweedfs/weed/storage/needle"
-	"github.com/seaweedfs/seaweedfs/weed/util"
 
 	"google.golang.org/grpc"
 )
@@ -42,7 +40,7 @@ func (c *commandVolumeMove) Help() string {
 	1. This command marks the source volume as read-only, copies it to the target volume server, and records the last entry timestamp.
 	2. This command asks the target volume server to mount the new volume.
 	3. This command asks the target volume server to tail the source volume for updates after the timestamp, for 1 minutes to drain any in-flight requests.
-	4. This command asks the source volume server to delete the source volume.
+	4. This command verifies the target volume matches the source, then asks the source volume server to delete the source volume.
 
 	The option "-disk [hdd|ssd|<tag>]" can be used to change the volume disk type.
 	The option "-timeout" fails the whole move if it does not finish in time.
@@ -81,7 +79,7 @@ func (c *commandVolumeMove) Do(args []string, commandEnv *CommandEnv, writer io.
 
 	volumeId := needle.VolumeId(*volumeIdInt)
 
-	if sourceVolumeServer == targetVolumeServer {
+	if volume_move.SameServer(sourceVolumeServer, targetVolumeServer) {
 		return fmt.Errorf("source and target volume servers are the same!")
 	}
 
@@ -92,170 +90,32 @@ func (c *commandVolumeMove) Do(args []string, commandEnv *CommandEnv, writer io.
 		defer cancel()
 	}
 
-	return LiveMoveVolume(ctx, commandEnv.option.GrpcDialOption, writer, volumeId, sourceVolumeServer, targetVolumeServer, 5*time.Second, *diskTypeStr, *ioBytePerSecond, false)
+	return LiveMoveVolume(ctx, commandEnv.option.GrpcDialOption, writer, volumeId, sourceVolumeServer, targetVolumeServer, 5*time.Second, *diskTypeStr, *ioBytePerSecond)
 }
 
 // LiveMoveVolume moves one volume from one source volume server to one target volume server, with idleTimeout to drain the incoming requests.
-func LiveMoveVolume(ctx context.Context, grpcDialOption grpc.DialOption, writer io.Writer, volumeId needle.VolumeId, sourceVolumeServer, targetVolumeServer pb.ServerAddress, idleTimeout time.Duration, diskType string, ioBytePerSecond int64, skipTailError bool) (err error) {
-
-	log.Printf("copying volume %d from %s to %s", volumeId, sourceVolumeServer, targetVolumeServer)
-	lastAppendAtNs, leftReadonly, err := copyVolume(ctx, grpcDialOption, writer, volumeId, sourceVolumeServer, targetVolumeServer, diskType, ioBytePerSecond, false)
-	if err != nil {
-		return fmt.Errorf("copy volume %d from %s to %s: %v", volumeId, sourceVolumeServer, targetVolumeServer, err)
-	}
-
-	// A move aborted after the copy must restore the source writability, or
-	// the source volume is left permanently readonly.
-	var sourceDeleteStarted bool
-	defer func() {
-		if err == nil || !leftReadonly {
-			return
-		}
-		if !sourceDeleteStarted {
-			// The target copy may be missing tailed entries; remove it so the
-			// restored source stays the only replica.
-			deleteCtx, deleteCancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
-			defer deleteCancel()
-			if dErr := deleteVolume(deleteCtx, grpcDialOption, volumeId, targetVolumeServer, false, true); dErr != nil {
-				// Restoring the source while the stale target stays mounted
-				// risks divergent replicas; re-running the move recovers both.
-				log.Printf("volume %d is left readonly on %s: failed to delete the incomplete copy on %s: %v; re-run volume.move to recover", volumeId, sourceVolumeServer, targetVolumeServer, dErr)
-				return
-			}
-		}
-		restoreCtx, restoreCancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
-		defer restoreCancel()
-		if wErr := markVolumeWritable(restoreCtx, grpcDialOption, volumeId, sourceVolumeServer, true, false); wErr != nil {
-			log.Printf("failed to restore volume %d writable on %s: %v", volumeId, sourceVolumeServer, wErr)
-		}
-	}()
-
-	log.Printf("tailing volume %d from %s to %s", volumeId, sourceVolumeServer, targetVolumeServer)
-	if err = tailVolume(ctx, grpcDialOption, volumeId, sourceVolumeServer, targetVolumeServer, lastAppendAtNs, idleTimeout); err != nil {
-		if skipTailError {
-			fmt.Fprintf(writer, "tail volume %d from %s to %s: %v\n", volumeId, sourceVolumeServer, targetVolumeServer, err)
-		} else {
-			return fmt.Errorf("tail volume %d from %s to %s: %v", volumeId, sourceVolumeServer, targetVolumeServer, err)
-		}
-	}
-
-	log.Printf("deleting volume %d from %s", volumeId, sourceVolumeServer)
-	sourceDeleteStarted = true
-	if err = deleteVolume(ctx, grpcDialOption, volumeId, sourceVolumeServer, false, true); err != nil {
-		return fmt.Errorf("delete volume %d from %s: %v", volumeId, sourceVolumeServer, err)
-	}
-
-	log.Printf("moved volume %d from %s to %s", volumeId, sourceVolumeServer, targetVolumeServer)
-	return nil
+func LiveMoveVolume(ctx context.Context, grpcDialOption grpc.DialOption, writer io.Writer, volumeId needle.VolumeId, sourceVolumeServer, targetVolumeServer pb.ServerAddress, idleTimeout time.Duration, diskType string, ioBytePerSecond int64) (err error) {
+	return volume_move.NewMover(grpcDialOption).LiveMoveVolume(ctx, volumeId, sourceVolumeServer, targetVolumeServer, volume_move.VolumeMoveOptions{
+		DiskType:        diskType,
+		IoBytePerSecond: ioBytePerSecond,
+		IdleTimeout:     idleTimeout,
+		Writer:          writer,
+	})
 }
 
-func copyVolume(ctx context.Context, grpcDialOption grpc.DialOption, writer io.Writer, volumeId needle.VolumeId, sourceVolumeServer, targetVolumeServer pb.ServerAddress, diskType string, ioBytePerSecond int64, restoreWritable bool) (lastAppendAtNs uint64, leftReadonly bool, err error) {
-
-	// check to see if the volume is already read-only and if its not then we need
-	// to mark it as read-only and then before we return we need to undo what we
-	// did
-	var shouldMarkWritable bool
-	defer func() {
-		if !shouldMarkWritable {
-			return
-		}
-		if !restoreWritable && err == nil {
-			leftReadonly = true
-			return
-		}
-
-		// Restoring writability must outlive a cancelled copy, or the source
-		// is left permanently readonly on abort, as balance_task.go already
-		// guards against.
-		restoreCtx, restoreCancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
-		defer restoreCancel()
-		clientErr := operation.WithVolumeServerClient(false, sourceVolumeServer, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
-			_, writableErr := volumeServerClient.VolumeMarkWritable(restoreCtx, &volume_server_pb.VolumeMarkWritableRequest{
-				VolumeId: uint32(volumeId),
-			})
-			return writableErr
-		})
-		if clientErr != nil {
-			log.Printf("failed to mark volume %d as writable after copy from %s: %v", volumeId, sourceVolumeServer, clientErr)
-		}
-	}()
-
-	err = operation.WithVolumeServerClient(false, sourceVolumeServer, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
-		resp, statusErr := volumeServerClient.VolumeStatus(ctx, &volume_server_pb.VolumeStatusRequest{
-			VolumeId: uint32(volumeId),
-		})
-		if statusErr == nil && !resp.IsReadOnly {
-			shouldMarkWritable = true
-			_, readonlyErr := volumeServerClient.VolumeMarkReadonly(ctx, &volume_server_pb.VolumeMarkReadonlyRequest{
-				VolumeId: uint32(volumeId),
-				Persist:  false,
-			})
-			return readonlyErr
-		}
-		return statusErr
-	})
-	if err != nil {
-		return
-	}
-
-	err = operation.WithVolumeServerClient(true, targetVolumeServer, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
-		stream, replicateErr := volumeServerClient.VolumeCopy(ctx, &volume_server_pb.VolumeCopyRequest{
-			VolumeId:        uint32(volumeId),
-			SourceDataNode:  string(sourceVolumeServer),
-			DiskType:        diskType,
-			IoBytePerSecond: ioBytePerSecond,
-		})
-		if replicateErr != nil {
-			return replicateErr
-		}
-		for {
-			resp, recvErr := stream.Recv()
-			if recvErr != nil {
-				if recvErr == io.EOF {
-					break
-				} else {
-					return recvErr
-				}
-			}
-			if resp.LastAppendAtNs != 0 {
-				lastAppendAtNs = resp.LastAppendAtNs
-			} else {
-				fmt.Fprintf(writer, "%s => %s volume %d processed %s\n", sourceVolumeServer, targetVolumeServer, volumeId, util.BytesToHumanReadable(uint64(resp.ProcessedBytes)))
-			}
-		}
-
-		return nil
-	})
-
-	return
+func copyVolume(ctx context.Context, grpcDialOption grpc.DialOption, writer io.Writer, volumeId needle.VolumeId, sourceVolumeServer, targetVolumeServer pb.ServerAddress, diskType string, ioBytePerSecond int64, restoreWritable bool) (lastAppendAtNs uint64, err error) {
+	return volume_move.NewMover(grpcDialOption).CopyVolume(ctx, volumeId, sourceVolumeServer, targetVolumeServer, diskType, ioBytePerSecond, restoreWritable, writer)
 }
 
 func tailVolume(ctx context.Context, grpcDialOption grpc.DialOption, volumeId needle.VolumeId, sourceVolumeServer, targetVolumeServer pb.ServerAddress, lastAppendAtNs uint64, idleTimeout time.Duration) (err error) {
-
-	return operation.WithVolumeServerClient(true, targetVolumeServer, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
-		_, replicateErr := volumeServerClient.VolumeTailReceiver(ctx, &volume_server_pb.VolumeTailReceiverRequest{
-			VolumeId:           uint32(volumeId),
-			SinceNs:            lastAppendAtNs,
-			IdleTimeoutSeconds: uint32(idleTimeout.Seconds()),
-			SourceVolumeServer: string(sourceVolumeServer),
-		})
-		return replicateErr
-	})
-
+	return volume_move.NewMover(grpcDialOption).TailVolume(ctx, volumeId, sourceVolumeServer, targetVolumeServer, lastAppendAtNs, idleTimeout)
 }
 
 // deleteVolume removes the volume from sourceVolumeServer. When keepRemoteData
 // is true, the cloud-tier object backing the volume is left intact — used on
 // the source side of a move where another server is taking over the same .vif.
 func deleteVolume(ctx context.Context, grpcDialOption grpc.DialOption, volumeId needle.VolumeId, sourceVolumeServer pb.ServerAddress, onlyEmpty bool, keepRemoteData bool) (err error) {
-	return operation.WithVolumeServerClient(false, sourceVolumeServer, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
-		_, deleteErr := volumeServerClient.VolumeDelete(ctx, &volume_server_pb.VolumeDeleteRequest{
-			VolumeId:       uint32(volumeId),
-			OnlyEmpty:      onlyEmpty,
-			KeepRemoteData: keepRemoteData,
-		})
-		return deleteErr
-	})
+	return volume_move.NewMover(grpcDialOption).DeleteVolume(ctx, volumeId, sourceVolumeServer, onlyEmpty, keepRemoteData)
 }
 
 func markVolumeWritable(ctx context.Context, grpcDialOption grpc.DialOption, volumeId needle.VolumeId, sourceVolumeServer pb.ServerAddress, writable, persist bool) (err error) {
@@ -300,44 +160,10 @@ func markVolumeReplicasWritable(ctx context.Context, grpcDialOption grpc.DialOpt
 
 // replicateVolumeToServer copies a volume from sourceAddress to targetAddress via the VolumeCopy gRPC stream.
 func replicateVolumeToServer(ctx context.Context, grpcDialOption grpc.DialOption, writer io.Writer, volumeId needle.VolumeId, sourceAddress, targetAddress pb.ServerAddress, diskType string) error {
-	return operation.WithVolumeServerClient(false, targetAddress, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
-		stream, replicateErr := volumeServerClient.VolumeCopy(ctx, &volume_server_pb.VolumeCopyRequest{
-			VolumeId:       uint32(volumeId),
-			SourceDataNode: string(sourceAddress),
-			DiskType:       diskType,
-		})
-		if replicateErr != nil {
-			return replicateErr
-		}
-		for {
-			resp, recvErr := stream.Recv()
-			if recvErr != nil {
-				if recvErr == io.EOF {
-					break
-				}
-				return recvErr
-			}
-			if resp.ProcessedBytes > 0 {
-				fmt.Fprintf(writer, "volume %d processed %s bytes\n", volumeId, util.BytesToHumanReadable(uint64(resp.ProcessedBytes)))
-			}
-		}
-		return nil
-	})
+	return volume_move.NewMover(grpcDialOption).ReplicateVolume(ctx, volumeId, sourceAddress, targetAddress, diskType, writer)
 }
 
 // configureVolumeReplication sets the replication setting on a volume at the given server.
 func configureVolumeReplication(ctx context.Context, grpcDialOption grpc.DialOption, volumeId needle.VolumeId, targetAddress pb.ServerAddress, replicationString string) error {
-	return operation.WithVolumeServerClient(false, targetAddress, grpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
-		resp, configureErr := volumeServerClient.VolumeConfigure(ctx, &volume_server_pb.VolumeConfigureRequest{
-			VolumeId:    uint32(volumeId),
-			Replication: replicationString,
-		})
-		if configureErr != nil {
-			return configureErr
-		}
-		if resp.Error != "" {
-			return errors.New(resp.Error)
-		}
-		return nil
-	})
+	return volume_move.NewMover(grpcDialOption).ConfigureVolumeReplication(ctx, volumeId, targetAddress, replicationString)
 }

weed/shell/command_volume_tier_move.go

@@ -2,15 +2,18 @@ package shell
 
 import (
 	"context"
+	"errors"
 	"flag"
 	"fmt"
-	"github.com/seaweedfs/seaweedfs/weed/placement"
 	"io"
 	"path/filepath"
 	"sync"
 	"time"
 
+	"github.com/seaweedfs/seaweedfs/weed/placement"
+
 	"github.com/seaweedfs/seaweedfs/weed/glog"
+	"github.com/seaweedfs/seaweedfs/weed/operation/volume_move"
 	"github.com/seaweedfs/seaweedfs/weed/pb"
 	"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
 	"github.com/seaweedfs/seaweedfs/weed/storage/super_block"
@@ -327,13 +330,18 @@ func (c *commandVolumeTierMove) doMoveOneVolume(commandEnv *CommandEnv, writer i
 	deletedSource := sourceVolumeServer
 	if alreadyPlaced {
 		deletedSource = ""
-	} else if err = LiveMoveVolume(context.Background(), commandEnv.option.GrpcDialOption, writer, vid, sourceVolumeServer, newAddress, 5*time.Second, toDiskType.ReadableString(), ioBytePerSecond, true); err != nil {
-		// mark all replicas as writable
-		if err = markVolumeReplicasWritable(context.Background(), commandEnv.option.GrpcDialOption, vid, locations, true, false); err != nil {
-			glog.Errorf("mark volume %d as writable on %s: %v", vid, locations[0].Url, err)
+	} else if moveErr := LiveMoveVolume(context.Background(), commandEnv.option.GrpcDialOption, writer, vid, sourceVolumeServer, newAddress, 5*time.Second, toDiskType.ReadableString(), ioBytePerSecond); moveErr != nil {
+		// A move that deliberately kept the source readonly (its delete may
+		// have happened, leaving the target authoritative) must not be thawed
+		// — reopening the replicas beside that copy would fork the volume.
+		if !errors.Is(moveErr, volume_move.ErrSourceKeptReadonly) {
+			// mark all replicas as writable
+			if err = markVolumeReplicasWritable(context.Background(), commandEnv.option.GrpcDialOption, vid, locations, true, false); err != nil {
+				glog.Errorf("mark volume %d as writable on %s: %v", vid, locations[0].Url, err)
+			}
 		}
 
-		return fmt.Errorf("move volume %d %s => %s : %v", vid, locations[0].Url, dst.dataNode.Id, err)
+		return fmt.Errorf("move volume %d %s => %s : %v", vid, locations[0].Url, dst.dataNode.Id, moveErr)
 	}
 
 	// If move is successful and replication is not empty, alter moved volume's replication setting

weed/worker/tasks/balance/balance_task.go

@@ -3,16 +3,13 @@ package balance
 import (
 	"context"
 	"fmt"
-	"io"
 	"time"
 
 	"github.com/seaweedfs/seaweedfs/weed/glog"
-	"github.com/seaweedfs/seaweedfs/weed/operation"
+	"github.com/seaweedfs/seaweedfs/weed/operation/volume_move"
 	"github.com/seaweedfs/seaweedfs/weed/pb"
-	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
 	"github.com/seaweedfs/seaweedfs/weed/pb/worker_pb"
 	"github.com/seaweedfs/seaweedfs/weed/storage/needle"
-	"github.com/seaweedfs/seaweedfs/weed/util"
 	"github.com/seaweedfs/seaweedfs/weed/worker/types"
 	"github.com/seaweedfs/seaweedfs/weed/worker/types/base"
 	"google.golang.org/grpc"
@@ -75,83 +72,21 @@ func (t *BalanceTask) Execute(ctx context.Context, params *worker_pb.TaskParams)
 		"collection":  t.collection,
 	}).Info("Starting balance task - moving volume")
 
-	sourceServer := pb.ServerAddress(sourceNode)
-	targetServer := pb.ServerAddress(destNode)
-	volumeId := needle.VolumeId(t.volumeID)
-
-	// Step 1: Mark volume readonly
-	t.ReportProgress(10.0)
-	t.GetLogger().Info("Marking volume readonly for move")
-	if err := t.markVolumeReadonly(ctx, sourceServer, volumeId); err != nil {
-		return fmt.Errorf("failed to mark volume readonly: %w", err)
-	}
-	// Restore source writability if any subsequent step fails, so the
-	// source volume is not left permanently readonly on abort.
-	sourceMarkedReadonly := true
-	defer func() {
-		if sourceMarkedReadonly {
-			cleanupCtx, cleanupCancel := context.WithTimeout(context.Background(), 30*time.Second)
-			defer cleanupCancel()
-			if wErr := t.markVolumeWritable(cleanupCtx, sourceServer, volumeId); wErr != nil {
-				glog.Warningf("failed to restore volume %d writability on %s: %v", volumeId, sourceServer, wErr)
-			}
-		}
-	}()
-
-	// Step 2: Read source volume size before copy (for post-copy verification)
-	t.ReportProgress(15.0)
-	sourceStatus, err := t.readVolumeFileStatus(ctx, sourceServer, volumeId)
+	// The move sequence — freeze the source, copy, tail, verify the target
+	// matches the source before the destructive delete — is shared with the
+	// shell's volume.move/volume.balance commands.
+	mover := volume_move.NewMover(t.grpcDialOption)
+	err := mover.LiveMoveVolume(ctx, needle.VolumeId(t.volumeID), pb.ServerAddress(sourceNode), pb.ServerAddress(destNode), volume_move.VolumeMoveOptions{
+		IdleTimeout: 60 * time.Second,
+		Progress: func(percent float64, stage string) {
+			t.ReportProgress(percent)
+			t.GetLogger().Info(stage)
+		},
+	})
 	if err != nil {
-		return fmt.Errorf("failed to read source volume status: %w", err)
+		return fmt.Errorf("move volume %d from %s to %s: %w", t.volumeID, sourceNode, destNode, err)
 	}
 
-	// Step 3: Copy volume to destination (VolumeCopy also mounts the volume)
-	t.ReportProgress(20.0)
-	t.GetLogger().Info("Copying volume to destination")
-	lastAppendAtNs, err := t.copyVolume(ctx, sourceServer, targetServer, volumeId)
-	if err != nil {
-		return fmt.Errorf("failed to copy volume: %w", err)
-	}
-
-	// Step 4: Tail for updates
-	t.ReportProgress(70.0)
-	t.GetLogger().Info("Syncing final updates")
-	if err := t.tailVolume(ctx, sourceServer, targetServer, volumeId, lastAppendAtNs); err != nil {
-		glog.Warningf("Tail operation failed (may be normal): %v", err)
-	}
-
-	// Step 5: Verify the volume on target before deleting source.
-	// This is a critical safety check — once the source is deleted, data loss
-	// is irreversible. We verify the target has the volume with matching size.
-	t.ReportProgress(85.0)
-	t.GetLogger().Info("Verifying volume on target before deleting source")
-	targetStatus, err := t.readVolumeFileStatus(ctx, targetServer, volumeId)
-	if err != nil {
-		return fmt.Errorf("aborting: cannot verify volume %d on target %s before deleting source: %v", volumeId, targetServer, err)
-	}
-	if targetStatus.DatFileSize != sourceStatus.DatFileSize {
-		return fmt.Errorf("aborting: volume %d .dat size mismatch — source %d bytes, target %d bytes",
-			volumeId, sourceStatus.DatFileSize, targetStatus.DatFileSize)
-	}
-	if targetStatus.FileCount != sourceStatus.FileCount {
-		return fmt.Errorf("aborting: volume %d file count mismatch — source %d, target %d",
-			volumeId, sourceStatus.FileCount, targetStatus.FileCount)
-	}
-	if targetStatus.IdxFileSize != sourceStatus.IdxFileSize {
-		return fmt.Errorf("aborting: volume %d .idx size mismatch — source %d bytes, target %d bytes",
-			volumeId, sourceStatus.IdxFileSize, targetStatus.IdxFileSize)
-	}
-
-	// Step 6: Delete from source — after this, the move is committed.
-	// Clear the readonly flag so the defer doesn't try to restore writability.
-	t.ReportProgress(90.0)
-	t.GetLogger().Info("Deleting volume from source server")
-	if err := t.deleteVolume(ctx, sourceServer, volumeId); err != nil {
-		return fmt.Errorf("failed to delete volume from source: %w", err)
-	}
-	sourceMarkedReadonly = false
-
-	t.ReportProgress(100.0)
 	glog.Infof("Balance task completed successfully: volume %d moved from %s to %s",
 		t.volumeID, sourceNode, destNode)
 	return nil
@@ -197,109 +132,3 @@ func (t *BalanceTask) EstimateTime(params *worker_pb.TaskParams) time.Duration {
 func (t *BalanceTask) GetProgress() float64 {
 	return t.progress
 }
-
-// Helper methods for real balance operations
-
-// markVolumeReadonly marks the volume readonly on the source server.
-func (t *BalanceTask) markVolumeReadonly(ctx context.Context, server pb.ServerAddress, volumeId needle.VolumeId) error {
-	return operation.WithVolumeServerClient(false, server, t.grpcDialOption,
-		func(client volume_server_pb.VolumeServerClient) error {
-			_, err := client.VolumeMarkReadonly(ctx, &volume_server_pb.VolumeMarkReadonlyRequest{
-				VolumeId: uint32(volumeId),
-			})
-			return err
-		})
-}
-
-// markVolumeWritable restores the volume to writable on the source server.
-func (t *BalanceTask) markVolumeWritable(ctx context.Context, server pb.ServerAddress, volumeId needle.VolumeId) error {
-	return operation.WithVolumeServerClient(false, server, t.grpcDialOption,
-		func(client volume_server_pb.VolumeServerClient) error {
-			_, err := client.VolumeMarkWritable(ctx, &volume_server_pb.VolumeMarkWritableRequest{
-				VolumeId: uint32(volumeId),
-			})
-			return err
-		})
-}
-
-// copyVolume copies volume from source to target server.
-func (t *BalanceTask) copyVolume(ctx context.Context, sourceServer, targetServer pb.ServerAddress, volumeId needle.VolumeId) (uint64, error) {
-	var lastAppendAtNs uint64
-
-	err := operation.WithVolumeServerClient(true, targetServer, t.grpcDialOption,
-		func(client volume_server_pb.VolumeServerClient) error {
-			stream, err := client.VolumeCopy(ctx, &volume_server_pb.VolumeCopyRequest{
-				VolumeId:       uint32(volumeId),
-				SourceDataNode: string(sourceServer),
-			})
-			if err != nil {
-				return err
-			}
-
-			for {
-				resp, recvErr := stream.Recv()
-				if recvErr != nil {
-					if recvErr == io.EOF {
-						break
-					}
-					return recvErr
-				}
-
-				if resp.LastAppendAtNs != 0 {
-					lastAppendAtNs = resp.LastAppendAtNs
-				} else {
-					// Report copy progress
-					glog.V(1).Infof("Volume %d copy progress: %s", volumeId,
-						util.BytesToHumanReadable(uint64(resp.ProcessedBytes)))
-				}
-			}
-
-			return nil
-		})
-
-	return lastAppendAtNs, err
-}
-
-// tailVolume syncs remaining updates from source to target.
-func (t *BalanceTask) tailVolume(ctx context.Context, sourceServer, targetServer pb.ServerAddress, volumeId needle.VolumeId, sinceNs uint64) error {
-	return operation.WithVolumeServerClient(true, targetServer, t.grpcDialOption,
-		func(client volume_server_pb.VolumeServerClient) error {
-			_, err := client.VolumeTailReceiver(ctx, &volume_server_pb.VolumeTailReceiverRequest{
-				VolumeId:           uint32(volumeId),
-				SinceNs:            sinceNs,
-				IdleTimeoutSeconds: 60, // 1 minute timeout
-				SourceVolumeServer: string(sourceServer),
-			})
-			return err
-		})
-}
-
-// readVolumeFileStatus reads the volume's file status (sizes, file count) from a server.
-func (t *BalanceTask) readVolumeFileStatus(ctx context.Context, server pb.ServerAddress, volumeId needle.VolumeId) (*volume_server_pb.ReadVolumeFileStatusResponse, error) {
-	var resp *volume_server_pb.ReadVolumeFileStatusResponse
-	err := operation.WithVolumeServerClient(false, server, t.grpcDialOption,
-		func(client volume_server_pb.VolumeServerClient) error {
-			var err error
-			resp, err = client.ReadVolumeFileStatus(ctx,
-				&volume_server_pb.ReadVolumeFileStatusRequest{
-					VolumeId: uint32(volumeId),
-				})
-			return err
-		})
-	return resp, err
-}
-
-// deleteVolume deletes the volume from the source server after a successful
-// move. KeepRemoteData=true prevents the source from removing the cloud-tier
-// object that the destination's freshly-copied .vif now points at.
-func (t *BalanceTask) deleteVolume(ctx context.Context, server pb.ServerAddress, volumeId needle.VolumeId) error {
-	return operation.WithVolumeServerClient(false, server, t.grpcDialOption,
-		func(client volume_server_pb.VolumeServerClient) error {
-			_, err := client.VolumeDelete(ctx, &volume_server_pb.VolumeDeleteRequest{
-				VolumeId:       uint32(volumeId),
-				OnlyEmpty:      false,
-				KeepRemoteData: true,
-			})
-			return err
-		})
-}

weed/worker/tasks/ec_balance/ec_balance_task.go

@@ -8,18 +8,20 @@ import (
 	"time"
 
 	"github.com/seaweedfs/seaweedfs/weed/glog"
-	"github.com/seaweedfs/seaweedfs/weed/operation"
+	"github.com/seaweedfs/seaweedfs/weed/operation/volume_move"
 	"github.com/seaweedfs/seaweedfs/weed/pb"
-	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
 	"github.com/seaweedfs/seaweedfs/weed/pb/worker_pb"
 	"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
+	"github.com/seaweedfs/seaweedfs/weed/storage/needle"
 	"github.com/seaweedfs/seaweedfs/weed/worker/types"
 	"github.com/seaweedfs/seaweedfs/weed/worker/types/base"
 	"google.golang.org/grpc"
 )
 
 // ECBalanceTask implements a single EC shard move operation.
-// The move sequence is: copy+mount on dest → unmount on source → delete on source.
+// The move sequence — copy+mount on dest, verify the dest registered the
+// shards, then unmount+delete on the source — is shared with the shell's
+// ec.balance command via weed/operation/volume_move.
 type ECBalanceTask struct {
 	*base.BaseTask
 	volumeID       uint32
@@ -39,8 +41,7 @@ func NewECBalanceTask(id string, volumeID uint32, collection string, grpcDialOpt
 	}
 }
 
-// Execute performs the EC shard move operation using the same RPC sequence
-// as the shell ec.balance command's moveMountedShardToEcNode function.
+// Execute performs the EC shard move operation.
 func (t *ECBalanceTask) Execute(ctx context.Context, params *worker_pb.TaskParams) error {
 	if params == nil {
 		return fmt.Errorf("task parameters are required")
@@ -62,6 +63,13 @@ func (t *ECBalanceTask) Execute(ctx context.Context, params *worker_pb.TaskParam
 
 	sourceAddr := pb.ServerAddress(source.Node)
 	targetAddr := pb.ServerAddress(target.Node)
+	// Range-check before the uint8 narrowing in Uint32ToShardIds: a malformed
+	// id like 259 would otherwise alias shard 3 and copy/delete a real,
+	// unrelated shard.
+	if err := checkShardIdRange(source.ShardIds); err != nil {
+		return err
+	}
+	shardIds := erasure_coding.Uint32ToShardIds(source.ShardIds)
 
 	ecParams := params.GetEcBalanceParams()
 
@@ -74,12 +82,12 @@ func (t *ECBalanceTask) Execute(ctx context.Context, params *worker_pb.TaskParam
 
 	isDedupDelete := ecParams != nil && isDedupPhase(params)
 
-	// Guard against a same-node, cross-disk "move". copyAndMountShard skips the
-	// copy when source and target addresses match, but deleteShard is node-wide
-	// (it removes the shard from every disk on the node), so this sequence would
-	// erase the shard after never copying it. EC shards also cannot be relocated
-	// between disks of one node via these RPCs, so such a move is meaningless.
-	// Reject it rather than lose data.
+	// Guard against a same-node, cross-disk "move". The shared mover skips the
+	// copy when source and target addresses match, but the EC shard delete is
+	// node-wide (it removes the shard from every disk on the node), so this
+	// sequence would erase the shard after never copying it. EC shards also
+	// cannot be relocated between disks of one node via these RPCs, so such a
+	// move is meaningless. Reject it rather than lose data.
 	if source.Node == target.Node && source.DiskId != target.DiskId {
 		return fmt.Errorf("refusing same-node cross-disk EC shard move for volume %d shard(s) %v on %s (source disk %d, target disk %d): EC shard delete is node-wide and would erase the shard after a skipped copy",
 			params.VolumeId, source.ShardIds, source.Node, source.DiskId, target.DiskId)
@@ -88,72 +96,45 @@ func (t *ECBalanceTask) Execute(ctx context.Context, params *worker_pb.TaskParam
 	glog.Infof("EC balance: moving shard(s) %v of volume %d from %s to %s",
 		source.ShardIds, params.VolumeId, source.Node, target.Node)
 
+	mover := volume_move.NewMover(t.grpcDialOption)
+
 	// For dedup, we only unmount+delete from source (no copy needed)
 	if isDedupDelete {
-		return t.executeDedupDelete(ctx, params.VolumeId, sourceAddr, source.ShardIds, ecParams.GetDedupKeepNode())
+		// Nothing is copied first, so the shard surviving elsewhere is the
+		// only thing making this safe — and the plan asserting so is not
+		// evidence. The topology can name a location that holds nothing, and
+		// deleting on that basis removes the last copy. Confirm the keep node
+		// has it before deleting here.
+		if err := t.verifyShardsOnKeepNode(ctx, params.VolumeId, ecParams.GetDedupKeepNode(), source.ShardIds); err != nil {
+			return err
+		}
+		t.reportProgress(25.0, "Removing duplicate EC shard")
+		if err := mover.RemoveEcShards(ctx, needle.VolumeId(params.VolumeId), t.collection, sourceAddr, shardIds); err != nil {
+			return fmt.Errorf("remove duplicate shard: %w", err)
+		}
+		t.reportProgress(100.0, "Duplicate shard removed")
+		return nil
 	}
 
-	// Step 1: Copy shard to destination and mount
-	t.reportProgress(10.0, "Copying EC shard to destination")
-	if err := t.copyAndMountShard(ctx, params.VolumeId, sourceAddr, targetAddr, source.ShardIds, target.DiskId); err != nil {
-		return fmt.Errorf("copy and mount shard: %w", err)
-	}
-
-	// Step 1.5: confirm the destination actually registered the shard(s)
-	// before removing them from the source. A copy/mount RPC can return OK
-	// while the shard isn't loadable on the destination; deleting the source
-	// then would lose the shard. On a mismatch we keep the source so the
-	// scanner retries (the move is reported failed).
-	t.reportProgress(40.0, "Verifying EC shard(s) on destination")
-	if err := t.verifyShardsOnDestination(ctx, params.VolumeId, targetAddr, source.ShardIds); err != nil {
+	err := mover.MoveEcShards(ctx, volume_move.EcShardMove{
+		VolumeId:   needle.VolumeId(params.VolumeId),
+		Collection: params.Collection,
+		ShardIds:   shardIds,
+		Source:     sourceAddr,
+		Target:     targetAddr,
+		TargetDisk: target.DiskId,
+	}, volume_move.EcMoveOptions{
+		Progress: t.reportProgress,
+	})
+	if err != nil {
 		return err
 	}
 
-	// Step 2: Unmount shard on source
-	t.reportProgress(50.0, "Unmounting EC shard from source")
-	if err := t.unmountShard(ctx, params.VolumeId, sourceAddr, source.ShardIds); err != nil {
-		return fmt.Errorf("unmount shard on source: %w", err)
-	}
-
-	// Step 3: Delete shard from source
-	t.reportProgress(75.0, "Deleting EC shard from source")
-	if err := t.deleteShard(ctx, params.VolumeId, params.Collection, sourceAddr, source.ShardIds); err != nil {
-		return fmt.Errorf("delete shard on source: %w", err)
-	}
-
-	t.reportProgress(100.0, "EC shard move complete")
 	glog.Infof("EC balance: successfully moved shard(s) %v of volume %d from %s to %s",
 		source.ShardIds, params.VolumeId, source.Node, target.Node)
 	return nil
 }
 
-// executeDedupDelete removes a duplicate shard without copying. Because nothing
-// is copied first, the only thing standing between this and data loss is that
-// another node really holds the shard -- and the plan asserting so is not
-// evidence. The topology can name a location that holds nothing (such a server
-// answers "not found ec volume id" when asked for the file), and deleting on the
-// strength of that removes the last copy while reporting success. So confirm the
-// shard on the node the plan chose to keep, and keep this copy if that cannot be
-// established. An unreachable peer is unknown, not confirmed.
-func (t *ECBalanceTask) executeDedupDelete(ctx context.Context, volumeID uint32, sourceAddr pb.ServerAddress, shardIDs []uint32, keepNode string) error {
-	if err := t.verifyShardsOnKeepNode(ctx, volumeID, keepNode, shardIDs); err != nil {
-		return err
-	}
-
-	t.reportProgress(25.0, "Unmounting duplicate EC shard")
-	if err := t.unmountShard(ctx, volumeID, sourceAddr, shardIDs); err != nil {
-		return fmt.Errorf("unmount duplicate shard: %w", err)
-	}
-
-	t.reportProgress(75.0, "Deleting duplicate EC shard")
-	if err := t.deleteShard(ctx, volumeID, t.collection, sourceAddr, shardIDs); err != nil {
-		return fmt.Errorf("delete duplicate shard: %w", err)
-	}
-
-	t.reportProgress(100.0, "Duplicate shard removed")
-	return nil
-}
-
 // verifyShardsOnKeepNode confirms the node the plan wants to keep the shard on
 // actually has every shard about to be deleted elsewhere, for this collection.
 func (t *ECBalanceTask) verifyShardsOnKeepNode(ctx context.Context, volumeID uint32, keepNode string, shardIDs []uint32) error {
@@ -167,85 +148,6 @@ func (t *ECBalanceTask) verifyShardsOnKeepNode(ctx context.Context, volumeID uin
 	return nil
 }
 
-// copyAndMountShard copies EC shard from source to destination and mounts it
-func (t *ECBalanceTask) copyAndMountShard(ctx context.Context, volumeID uint32, sourceAddr, targetAddr pb.ServerAddress, shardIDs []uint32, destDiskID uint32) error {
-	return operation.WithVolumeServerClient(false, targetAddr, t.grpcDialOption,
-		func(client volume_server_pb.VolumeServerClient) error {
-			// Copy shard data (if source != target)
-			if sourceAddr != targetAddr {
-				_, err := client.VolumeEcShardsCopy(ctx, &volume_server_pb.VolumeEcShardsCopyRequest{
-					VolumeId:       volumeID,
-					Collection:     t.collection,
-					ShardIds:       shardIDs,
-					CopyEcxFile:    true,
-					CopyEcjFile:    true,
-					CopyVifFile:    true,
-					SourceDataNode: string(sourceAddr),
-					DiskId:         destDiskID,
-				})
-				if err != nil {
-					return fmt.Errorf("copy shard(s) %v from %s to %s: %v", shardIDs, sourceAddr, targetAddr, err)
-				}
-			}
-
-			// Mount the shard on destination
-			_, err := client.VolumeEcShardsMount(ctx, &volume_server_pb.VolumeEcShardsMountRequest{
-				VolumeId:   volumeID,
-				Collection: t.collection,
-				ShardIds:   shardIDs,
-			})
-			if err != nil {
-				return fmt.Errorf("mount shard(s) %v on %s: %v", shardIDs, targetAddr, err)
-			}
-
-			return nil
-		})
-}
-
-// unmountShard unmounts EC shards from a server
-func (t *ECBalanceTask) unmountShard(ctx context.Context, volumeID uint32, addr pb.ServerAddress, shardIDs []uint32) error {
-	return operation.WithVolumeServerClient(false, addr, t.grpcDialOption,
-		func(client volume_server_pb.VolumeServerClient) error {
-			_, err := client.VolumeEcShardsUnmount(ctx, &volume_server_pb.VolumeEcShardsUnmountRequest{
-				VolumeId: volumeID,
-				ShardIds: shardIDs,
-			})
-			return err
-		})
-}
-
-// verifyShardsOnDestination confirms targetAddr has registered every shard in
-// shardIDs for volumeID, so the caller can safely delete the source copies.
-func (t *ECBalanceTask) verifyShardsOnDestination(ctx context.Context, volumeID uint32, targetAddr pb.ServerAddress, shardIDs []uint32) error {
-	_, perServer := erasure_coding.VerifyShardsAcrossServers(ctx, volumeID, []string{string(targetAddr)}, t.grpcDialOption)
-	inv, ok := perServer[string(targetAddr)]
-	if !ok {
-		return fmt.Errorf("verify shard(s) on destination %s for volume %d: no inventory returned", targetAddr, volumeID)
-	}
-	if inv.QueryError != nil {
-		return fmt.Errorf("verify shard(s) on destination %s for volume %d: %v", targetAddr, volumeID, inv.QueryError)
-	}
-	for _, sid := range shardIDs {
-		if !inv.Bits.Has(erasure_coding.ShardId(sid)) {
-			return fmt.Errorf("destination %s missing EC shard %d.%d after copy/mount; keeping source", targetAddr, volumeID, sid)
-		}
-	}
-	return nil
-}
-
-// deleteShard deletes EC shards from a server
-func (t *ECBalanceTask) deleteShard(ctx context.Context, volumeID uint32, collection string, addr pb.ServerAddress, shardIDs []uint32) error {
-	return operation.WithVolumeServerClient(false, addr, t.grpcDialOption,
-		func(client volume_server_pb.VolumeServerClient) error {
-			_, err := client.VolumeEcShardsDelete(ctx, &volume_server_pb.VolumeEcShardsDeleteRequest{
-				VolumeId:   volumeID,
-				Collection: collection,
-				ShardIds:   shardIDs,
-			})
-			return err
-		})
-}
-
 // Validate validates the task parameters.
 // ECBalanceTask handles exactly one source→target shard move per execution.
 func (t *ECBalanceTask) Validate(params *worker_pb.TaskParams) error {
@@ -264,10 +166,15 @@ func (t *ECBalanceTask) Validate(params *worker_pb.TaskParams) error {
 	if len(params.Targets[0].ShardIds) == 0 {
 		return fmt.Errorf("ECBalanceTask.Validate: Targets[0].ShardIds is empty")
 	}
+	if err := checkShardIdRange(params.Sources[0].ShardIds); err != nil {
+		return fmt.Errorf("ECBalanceTask.Validate: %v", err)
+	}
+	if err := checkShardIdRange(params.Targets[0].ShardIds); err != nil {
+		return fmt.Errorf("ECBalanceTask.Validate: %v", err)
+	}
 	// A same-node, cross-disk move is unsafe: the node-wide EC shard delete would
-	// erase the shard after copyAndMountShard skips the same-address copy. Such a
-	// move cannot be expressed by these RPCs anyway. Dedup (same node and disk) is
-	// allowed.
+	// erase the shard after the skipped same-address copy. Such a move cannot be
+	// expressed by these RPCs anyway. Dedup (same node and disk) is allowed.
 	if params.Sources[0].Node == params.Targets[0].Node && params.Sources[0].DiskId != params.Targets[0].DiskId {
 		return fmt.Errorf("ECBalanceTask.Validate: refusing same-node cross-disk move on %s (source disk %d, target disk %d): EC shard delete is node-wide",
 			params.Sources[0].Node, params.Sources[0].DiskId, params.Targets[0].DiskId)
@@ -296,6 +203,17 @@ func (t *ECBalanceTask) reportProgress(progress float64, stage string) {
 	glog.Infof("EC balance volume %d: [%.2f] %s", t.volumeID, progress, stage)
 }
 
+// checkShardIdRange rejects shard ids that would alias a real shard when
+// narrowed to the uint8 ShardId (e.g. 259 → 3).
+func checkShardIdRange(ids []uint32) error {
+	for _, id := range ids {
+		if id >= erasure_coding.MaxShardCount {
+			return fmt.Errorf("shard id %d out of range (max %d)", id, erasure_coding.MaxShardCount-1)
+		}
+	}
+	return nil
+}
+
 // isDedupPhase checks if this is a dedup-phase task: an unmount+delete on a
 // single location, encoded by detection as source==target on the same node AND
 // the same disk. Comparing the disk too is essential — VolumeEcShardsDelete is

weed/worker/tasks/ec_balance/ec_balance_task_test.go

@@ -0,0 +1,27 @@
+package ec_balance
+
+import (
+	"context"
+	"strings"
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/worker_pb"
+)
+
+func TestECBalanceTaskRejectsOutOfRangeShardIds(t *testing.T) {
+	// ShardId is a uint8; an unchecked id like 259 would alias shard 3 and
+	// copy/delete a real, unrelated shard.
+	params := &worker_pb.TaskParams{
+		VolumeId: 7,
+		Sources:  []*worker_pb.TaskSource{{Node: "src:8080", ShardIds: []uint32{259}}},
+		Targets:  []*worker_pb.TaskTarget{{Node: "dst:8080", ShardIds: []uint32{259}}},
+	}
+	task := NewECBalanceTask("t1", 7, "c1", nil)
+
+	if err := task.Validate(params); err == nil || !strings.Contains(err.Error(), "out of range") {
+		t.Fatalf("Validate: expected out-of-range rejection, got: %v", err)
+	}
+	if err := task.Execute(context.Background(), params); err == nil || !strings.Contains(err.Error(), "out of range") {
+		t.Fatalf("Execute: expected out-of-range rejection, got: %v", err)
+	}
+}