* volume.tier.move: fulfill target replication before deleting old replicas
When -toReplication is specified, volume.tier.move now creates all
required replicas on the destination tier before deleting old replicas.
This closes the data-loss window where only one copy existed on the
target tier while awaiting volume.fix.replication.
If replication fulfillment fails, old replicas are preserved and marked
writable so the volume remains accessible.
Also extracts replicateVolumeToServer and configureVolumeReplication
helpers to reduce duplication across volume.tier.move and
volume.fix.replication.
Fixes#8937
* volume.tier.move: always fulfill replication before deleting old replicas
When -toReplication is specified, use that replication setting.
Otherwise, read the volume's existing replication from the super block.
In both cases, all required replicas are created on the destination
tier before old replicas are deleted.
If replication fulfillment fails (e.g. not enough destination nodes),
old replicas are preserved and marked writable so no data is lost.
* volume.tier.move: address review feedback on ensureReplicationFulfilled
- Add 5s delay before re-collecting topology to allow master heartbeat
propagation after the move
- Add nil guard for targetTierReplicas to prevent panic if the moved
replica is not yet visible in the topology
- Treat configureVolumeReplication failure as a hard error instead of a
warning, so the rollback logic preserves old replicas
* volume.tier.move: harden replication config error handling
- Make configureVolumeReplication failure on the primary moved replica a
hard error that aborts the move, instead of logging and continuing
- Configure replication metadata on all existing target-tier replicas
(not just newly created ones) when -toReplication is specified
- Deletion of old replicas cannot affect new replicas since the
locations list only contains pre-move servers (verified, no change)
* volume.tier.move: fix cleanup deleting fulfilled replicas and broken recovery
Fix 1: The cleanup loop now preserves pre-existing target-tier replicas
that ensureReplicationFulfilled counted toward the replication target.
Previously, a mixed-tier volume with an existing replica on the target
tier could have that replica deleted right after being counted as
fulfilled, leaving the volume under-replicated.
ensureReplicationFulfilled now returns a preserveServers set that the
deletion loop checks before removing any old replica.
Fix 2: Failure paths after LiveMoveVolume (which deletes the source
replica) now use restoreSurvivingReplicasWritable instead of
markVolumeReplicasWritable. The old helper stopped on first error, so
attempting to mark the already-deleted source writable would prevent
all surviving replicas from being restored. The new helper skips the
deleted source and continues through all remaining locations, logging
per-replica errors instead of aborting.
* volume.tier.move: mark preserved replicas writable, skip nodes with existing volume
Fix 1: Preserved pre-existing target-tier replicas were left read-only
after the move completed. They were marked read-only at the start
(along with all other replicas) but never restored since the old code
deleted them. Now they are explicitly marked writable before cleanup.
Fix 2: The fulfillment loop could pick a candidate node that already
hosts this volume on a different disk type, causing a VolumeCopy
conflict. Added a guard that skips any node already hosting the volume
(on any disk) before attempting replication.
* feat(weed.move): add a speed limit parameter of moving files
* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
