* feat: throughput limits for replicate, EC shard, and worker-driven moves
VolumeCopy was the only rate-limitable transfer; EC shard copies,
replica creation, and worker-driven moves all ran at whatever the
receiving server's maintenance rate allowed, with no per-operation
control.
- proto: VolumeEcShardsCopyRequest and the balance / ec_balance task
params and configs gain io_byte_per_second; 0 keeps today's behavior
(the volume server's own maintenance rate governs).
- volume server: VolumeEcShardsCopy throttles with one WriteThrottler
per request, shared across the shard, .ecx, .ecj, .vif, and .ecsum
copies so the limit caps the transfer as a whole - the same shape as
VolumeCopy.
- volume_move: ReplicateVolume accepts the limit; EcMoveOptions carries
it through MoveEcShards/CopyAndMountEcShards into the copy request,
with fake-client tests asserting propagation.
- shell: ec.balance gains -ioBytePerSecond; volume.tier.move's
replication top-up honors the command's existing -ioBytePerSecond
instead of running unthrottled.
- worker: balance and ec_balance configs gain io_byte_per_second
(surfaced in the admin config schema), carried through detection and
plugin job parameters into task params and handed to the shared
mover; batch balance jobs inherit the limit from their detection
results.
The limit is per copy stream, so maxParallelization multiplies the
aggregate ceiling.
* worker plugins: expose io_byte_per_second in the plugin config and derive it
The plugin-driven detection path derives its task Config from the
plugin configuration values, and both balance and ec_balance left
IoBytePerSecond at zero there - a configured limit silently reverted
to the server maintenance rate. Both derive functions now read the
field (clamped at zero), and the plugin descriptors expose it with
defaults so the configuration form carries it.
* operation: add shared volume_move package for volume and EC shard moves
The shell commands (volume.move, volume.balance, ec.balance, tier moves)
and the maintenance workers (balance, ec_balance) each carried their own
copy of the move RPC sequences, and the copies had drifted: the worker
verified the target before deleting the source but dropped the disk
type and IO throttle; the shell passed those but deleted the source
unverified.
volume_move.Mover carries the merged sequences, keeping the stricter
behavior from each side:
- LiveMoveVolume: check-then-hard-freeze the source (VolumeStatus's
IsReadOnly also covers low-disk and readonly-but-can-delete states,
which still accept needle deletes), copy with disk type and IO
throttle, tail, verify the target is not behind the source before the
destructive source delete (a target that is ahead holds writes it
accepted during the tail and the move commits to keep them), and
restore the source's writability when a failure precedes the delete
and this move did the freezing. Aborts clean up the incomplete target
copy; a failed cleanup or an ambiguous source delete keeps the source
readonly (ErrSourceKeptReadonly) so callers do not thaw a source next
to a possibly-authoritative copy. With a readonly source, an existing
or unknown-state target refuses the move outright: no client-side
observation can prove such a copy is a stale remnant rather than the
authoritative copy of an unfinished move.
- MoveEcShards: copy with the .ecx/.ecj/.vif/.ecsum sidecars, mount,
verify the target registered every shard before unmount+delete on the
source, and reject same-server moves (the EC delete is server-wide).
Server identity is the grpc endpoint (SameServer), so node:8080 and
node:8080.18080 compare equal while test servers sharing a degenerate
HTTP address stay distinct; addresses are validated non-fatally before
dialing and before being embedded in copy/tail requests, since both the
client dialer and the receiving server normalize them through a parser
that aborts the process on a malformed port. The Rust volume server's
codes.NotFound counts as a definitively absent probe answer alongside
the Go server's plain-error code Unknown.
All RPCs go through an injectable ClientFunc, so the sequences are unit
tested against a fake volume server client: RPC order, request fields,
and that verification failures keep the source intact.
* shell, worker: delegate volume and EC shard moves to operation/volume_move
LiveMoveVolume and the copy/tail/delete/mark-writable helpers become
thin wrappers over the shared mover, keeping their signatures; the EC
helpers keep their per-step output and delegate the RPCs. BalanceTask
and ECBalanceTask keep their parameter validation, progress reporting,
and guards (same-node cross-disk rejection, dedup keep-node
verification, shard ids range-checked before the uint8 narrowing) and
hand the RPC sequences to the mover. volume.tier.move skips its
thaw-on-failure when the mover deliberately kept the source readonly,
since reopening the replicas beside a possibly-authoritative target
copy would fork the volume.
The tail-failure tolerance moves inside the mover: a failed tail is
tolerated only when the volume was already readonly before the move
began, backstopped by a stability re-read across the idle window, so
volume.balance's -skipTailError-by-readonly heuristic and tier-move's
unconditional skip both become the same authoritative rule.
* volume_move: keep the source readonly when a failed copy leaves a target of unknown origin
A failed copy can leave a complete, mounted copy on the target (the
server finishes after the client loses the stream). The abort probed
the target only when its pre-copy state was known-absent; an unknown
prior state skipped both the probe and the cleanup and then reopened
the source - two writable replicas of one volume, diverging from the
next write on.
The abort now probes the target on every failed copy and restores the
source only when the target provably holds nothing. A copy whose
provenance cannot be proven (unknown prior state, a pre-existing
replica, or an unreachable target) is never deleted, and the source
stays readonly with ErrSourceKeptReadonly naming the recovery.
* test: teach the plugin worker harness the shared move sequence
The fake volume server lacked VolumeStatus, which the shared mover now
issues before freezing the source, and the batch execution test's
status-read accounting predates the pre-copy target probe and the
verification reads. Mirrors the harness the enterprise tree already
carries.
The command chose destinations by walking its location list in order, so it
neither preferred a node near the source nor spread a burst of copies. Replica
placement and "this node already holds the volume" move into the Accept
predicate; the ranking and the per-pick reservation come from placement.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
LiveMoveVolume and the copy, tail, delete, mark, replicate, and
configure helpers around it issued every RPC on context.Background(), so
a caller had no way to bound or abort a move once it started. They now
take a context, which the exported LiveMoveVolume in particular needs:
callers outside the shell drive long moves and want to stop them.
The deferred restore in copyVolume runs on a detached, bounded context
rather than the caller's. Marking the source writable again is cleanup,
and cancelling the copy must not skip it and leave the volume readonly —
the same guard balance_task.go already applies for the same reason.
Shell commands pass context.Background(): their Do signature carries no
context, and changing it would touch every command in the package.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* tiering: stop a shared remote object being deleted while replicas still point at it
A remote-tiered volume's .dat content lives only in one cloud object that all
N replica .vif files point at. Deleting that object while destroying any one
replica, or before a downloaded replica is durable, bricks the survivors.
- volume.tier.move cleanup now deletes old replicas with keepRemoteData=true so
surviving replicas keep the shared object. Document why the alreadyPlaced
anchor needs no replica sync (same-object replicas are byte-identical).
- VolumeTierMoveDatFromRemote now fsyncs the downloaded .dat, fsyncs the
containing directory, trims the .vif (fsynced) and swaps to the local DiskFile
BEFORE deleting the remote object, on both the keep-remote and delete paths.
Only the final DeleteFile is gated by keep_remote_dat_file, so a keep-remote
download leaves the replica served from local disk rather than the shared
object, and a crash before delete merely leaks the object.
- volume.tier.download keeps the shared object for every replica except the
last, which deletes it.
- s3 and rclone download paths fsync the .dat before close.
* storage: swap the volume data backend under the data lock
The tier-download swap closed v.DataBackend and assigned the new local DiskFile
without holding dataFileAccessLock, racing concurrent reads/writes (use of a
closed file / nil deref). Add an exported Volume.SwapDataBackend that performs
the close-and-replace under the lock, and call it from the tier download.
* server: skip directory fsync on Windows in the tier download path
os.Open(dir).Sync() is unsupported on Windows and returns an error, which would
fail VolumeTierMoveDatFromRemote entirely there. Skip the directory fsync on
Windows, matching how the storage-side helper tolerates the unsupported case.
* shell: make multi-replica tier.download resilient to already-local replicas
If a multi-replica download is interrupted and retried, a replica made local
in the prior attempt returns "already on local disk", which aborted the whole
command and left the remaining remote replicas dangling. Treat that case as a
skip-and-continue so a retry completes the rest.
* server: assert downloaded .dat content, not just length, in the tier test
A length-only check passes even if the bytes are corrupted; compare the full
content of the local .dat against the original.
* shell: volume.tier.move can move volumes between data centers
-fromDataCenter scopes volume selection to volumes with a replica in
that data center. -toDataCenter constrains move destinations and
replication fulfillment. With identical disk types both flags are
required, moving full volumes between data centers on the same tier.
* shell: assert node identity in data center filter test
* shell: tier move resumes when the volume is already on the target
A replica already on the target tier and data center, typically left by
an interrupted earlier run, anchors the move: skip the copy and only
complete replication fulfillment and old replica cleanup. Previously
such volumes hit the no-destination path and the stale source replicas
were never removed.
* refactor(volume): extract replica sync/select into shared volume_replica package
Move the volume replica reconciliation helpers (status, union builder,
SyncAndSelectBestReplica, ReadNeedleMeta) out of the shell into a new
weed/storage/volume_replica package so both the shell (ec.encode, volume.tier.move,
volume.check.disk) and the EC encode worker can reuse them. No behavior change.
* fix(ec): bring ec.encode worker to parity with the shell
- Sync replicas and encode the most-complete one (via the shared
volume_replica.SyncAndSelectBestReplica) instead of a possibly-stale replica,
marking all replicas readonly first. Prevents silent data loss when a stale
replica is encoded and the originals deleted.
- Skip remote/tiered volumes in detection (shell ec.encode excludes them).
- Min-node safety gate: refuse to encode when cluster nodes < parity shards.
- Align default thresholds with the shell (fullness 0.95, quiet 1h).
* fix(vacuum): plugin path honors min_volume_age_seconds override
deriveVacuumConfig hard-coded MinVolumeAgeSeconds=0, dropping any configured
value. Read it from worker config (default 0, matching the shell/master vacuum
which has no age gate) so an explicit override is honored.
* address review feedback
- config.go: align GetConfigSpec schema defaults (quiet_for_seconds=3600,
fullness_ratio=0.95) with the runtime defaults so UI/bootstrap flows match the
shell (coderabbitai).
- ec_task.go: roll back readonly when markReplicasReadonly fails partway, so
already-marked replicas don't stay readonly (coderabbitai).
- volume_replica: pass the caller's replica statuses into buildUnionReplica instead
of re-fetching them, and skip the per-needle ReadNeedleMeta RPC when the source
replica is read-only (gemini-code-assist).
* test(plugin_workers/ec): make fixtures eligible under the new defaults
The default EC encode thresholds were raised to match the shell (fullness 0.95,
quiet 1h), but the plugin-worker integration fixtures still used 90%-full /
10-minute-old volumes, so detection found no eligible volumes and the tests failed
in CI. Bump the eligible fixtures to 96% full and 2h old.
2026-05-21 02:16:28 -07:00
Chris LuGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix(volume): don't fatal on missing .idx for remote-tiered volume
A .vif left behind without its .idx (orphaned by a crashed move, partial
copy, or hand-edit) would trip glog.Fatalf in checkIdxFile and take the
whole volume server down on boot, killing every healthy volume on it
too. For remote-tiered volumes treat it as a per-volume load error so
the server can come up and the operator can clean up the stray .vif.
Refs #9331.
* fix(balance): skip remote-tiered volumes in admin balance detection
The admin/worker balance detector had no equivalent of the shell-side
guard ("does not move volume in remote storage" in
command_volume_balance.go), so it scheduled moves on remote-tiered
volumes. The "move" copies .idx/.vif to the destination and then calls
Volume.Destroy on the source, which calls backendStorage.DeleteFile —
deleting the remote object the destination's new .vif now points at.
Populate HasRemoteCopy on the metrics emitted by both the admin
maintenance scanner and the worker's master poll, then drop those
volumes at the top of Detection.
Fixes#9331.
* Apply suggestion from @gemini-code-assist[bot]
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix(volume): keep remote data on volume-move-driven delete
The on-source delete after a volume move (admin/worker balance and
shell volume.move) ran Volume.Destroy with no way to opt out of the
remote-object cleanup. Volume.Destroy unconditionally calls
backendStorage.DeleteFile for remote-tiered volumes, so a successful
move would copy .idx/.vif to the destination and then nuke the cloud
object the destination's new .vif was already pointing at.
Add VolumeDeleteRequest.keep_remote_data and plumb it through
Store.DeleteVolume / DiskLocation.DeleteVolume / Volume.Destroy. The
balance task and shell volume.move set it to true; the post-tier-upload
cleanup of other replicas and the over-replication trim in
volume.fix.replication also set it to true since the remote object is
still referenced. Other real-delete callers keep the default. The
delete-before-receive path in VolumeCopy also sets it: the inbound copy
carries a .vif that may reference the same cloud object as the
existing volume.
Refs #9331.
* test(storage): in-process remote-tier integration tests
Cover the four operations the user is most likely to run against a
cloud-tiered volume — balance/move, vacuum, EC encode, EC decode — by
registering a local-disk-backed BackendStorage as the "remote" tier and
exercising the real Volume / DiskLocation / EC encoder code paths.
Locks in:
- Destroy(keepRemoteData=true) preserves the remote object (move case)
- Destroy(keepRemoteData=false) deletes it (real-delete case)
- Vacuum/compact on a remote-tier volume never deletes the remote object
- EC encode requires the local .dat (callers must download first)
- EC encode + rebuild round-trips after a tier-down
Tests run in-process and finish in under a second total — no cluster,
binary, or external storage required.
* fix(rust-volume): keep remote data on volume-move-driven delete
Mirror the Go fix in seaweed-volume: plumb keep_remote_data through
grpc volume_delete → Store.delete_volume → DiskLocation.delete_volume
→ Volume.destroy, and skip the s3-tier delete_file call when the flag
is set. The pre-receive cleanup in volume_copy passes true for the
same reason as the Go side: the inbound copy carries a .vif that may
reference the same cloud object as the existing volume.
The Rust loader already warns rather than fataling on a stray .vif
without an .idx (volume.rs load_index_inmemory / load_index_redb), so
no counterpart to the Go fatal-on-missing-idx fix is needed.
Refs #9331.
* fix(volume): preserve remote tier on IO-error eviction; fix EC test target
Two review nits:
- Store.MaybeAddVolumes' periodic cleanup pass deleted IO-errored
volumes with keepRemoteData=false, so a transient local fault on a
remote-tiered volume would also nuke the cloud object. Track the
delete reason via a parallel slice and pass keepRemoteData=v.HasRemoteFile()
for IO-error evictions; TTL-expired evictions still pass false.
- TestRemoteTier_ECEncodeDecode_AfterDownload deleted shards 0..3 but
called them "parity" — by the klauspost/reedsolomon convention shards
0..DataShardsCount-1 are data and DataShardsCount..TotalShardsCount-1
are parity. Switch the loop to delete the parity range so the
intent matches the indices.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* volume.tier.move: fulfill target replication before deleting old replicas
When -toReplication is specified, volume.tier.move now creates all
required replicas on the destination tier before deleting old replicas.
This closes the data-loss window where only one copy existed on the
target tier while awaiting volume.fix.replication.
If replication fulfillment fails, old replicas are preserved and marked
writable so the volume remains accessible.
Also extracts replicateVolumeToServer and configureVolumeReplication
helpers to reduce duplication across volume.tier.move and
volume.fix.replication.
Fixes#8937
* volume.tier.move: always fulfill replication before deleting old replicas
When -toReplication is specified, use that replication setting.
Otherwise, read the volume's existing replication from the super block.
In both cases, all required replicas are created on the destination
tier before old replicas are deleted.
If replication fulfillment fails (e.g. not enough destination nodes),
old replicas are preserved and marked writable so no data is lost.
* volume.tier.move: address review feedback on ensureReplicationFulfilled
- Add 5s delay before re-collecting topology to allow master heartbeat
propagation after the move
- Add nil guard for targetTierReplicas to prevent panic if the moved
replica is not yet visible in the topology
- Treat configureVolumeReplication failure as a hard error instead of a
warning, so the rollback logic preserves old replicas
* volume.tier.move: harden replication config error handling
- Make configureVolumeReplication failure on the primary moved replica a
hard error that aborts the move, instead of logging and continuing
- Configure replication metadata on all existing target-tier replicas
(not just newly created ones) when -toReplication is specified
- Deletion of old replicas cannot affect new replicas since the
locations list only contains pre-move servers (verified, no change)
* volume.tier.move: fix cleanup deleting fulfilled replicas and broken recovery
Fix 1: The cleanup loop now preserves pre-existing target-tier replicas
that ensureReplicationFulfilled counted toward the replication target.
Previously, a mixed-tier volume with an existing replica on the target
tier could have that replica deleted right after being counted as
fulfilled, leaving the volume under-replicated.
ensureReplicationFulfilled now returns a preserveServers set that the
deletion loop checks before removing any old replica.
Fix 2: Failure paths after LiveMoveVolume (which deletes the source
replica) now use restoreSurvivingReplicasWritable instead of
markVolumeReplicasWritable. The old helper stopped on first error, so
attempting to mark the already-deleted source writable would prevent
all surviving replicas from being restored. The new helper skips the
deleted source and continues through all remaining locations, logging
per-replica errors instead of aborting.
* volume.tier.move: mark preserved replicas writable, skip nodes with existing volume
Fix 1: Preserved pre-existing target-tier replicas were left read-only
after the move completed. They were marked read-only at the start
(along with all other replicas) but never restored since the old code
deleted them. Now they are explicitly marked writable before cleanup.
Fix 2: The fulfillment loop could pick a candidate node that already
hosts this volume on a different disk type, causing a VolumeCopy
conflict. Added a guard that skips any node already hosting the volume
(on any disk) before attempting replication.
* Fix#8040: Support 'default' keyword in collectionPattern to match default collection
The default collection in SeaweedFS is represented as an empty string internally.
Previously, it was impossible to specifically target only the default collection
because:
- Empty collectionPattern matched ALL collections (filter was skipped)
- Using collectionPattern="default" tried to match the literal string "default"
This commit adds special handling for the keyword "default" in collectionPattern
across multiple shell commands:
- volume.tier.move
- volume.list
- volume.fix.replication
- volume.configure.replication
Now users can use -collectionPattern="default" to specifically target volumes
in the default collection (empty collection name), while maintaining backward
compatibility where empty pattern matches all collections.
Updated help text to document this feature.
* Update compileCollectionPattern to support 'default' keyword
This extends the fix to all commands that use regex-based collection
pattern matching:
- ec.encode
- ec.decode
- volume.tier.download
- volume.balance
The compileCollectionPattern function now treats "default" as a special
keyword that compiles to the regex "^$" (matching empty strings), making
it consistent with the other commands that use filepath.Match.
* Use CollectionDefault constant instead of hardcoded "default" string
Refactored the collection pattern matching logic to use a central constant
CollectionDefault defined in weed/shell/common.go. This improves maintainability
and ensures consistency across all shell commands.
* Address PR review feedback: simplify logic and use '_default' keyword
Changes:
1. Changed CollectionDefault from "default" to "_default" to avoid collision
with literal collection names
2. Simplified pattern matching logic to reduce code duplication across all
affected commands
3. Fixed error handling in command_volume_tier_move.go to properly propagate
filepath.Match errors instead of swallowing them
4. Updated documentation to clarify how to match a literal "default"
collection using regex patterns like "^default$"
This addresses all feedback from PR review comments.
* Remove unnecessary documentation about matching literal 'default'
Since we changed the keyword to '_default', users can now simply use
'default' to match a literal collection named "default". The previous
documentation about using regex patterns was confusing and no longer needed.
* Fix error propagation and empty pattern handling
1. command_volume_tier_move.go: Added early termination check after
eachDataNode callback to stop processing remaining nodes if a pattern
matching error occurred, improving efficiency
2. command_volume_configure_replication.go: Fixed empty pattern handling
to match all collections (collectionMatched = true when pattern is empty),
mirroring the behavior in other commands
These changes address the remaining PR review feedback.
* fix: sync replica entries before ec.encode and volume.tier.move (#7797)
This addresses the data inconsistency risk in multi-replica volumes.
When ec.encode or volume.tier.move operates on a multi-replica volume:
1. Find the replica with the highest file count (the 'best' one)
2. Copy missing entries from other replicas INTO this best replica
3. Use this union replica for the destructive operation
This ensures no data is lost due to replica inconsistency before
EC encoding or tier moving.
Added:
- command_volume_replica_check.go: Core sync and select logic
- command_volume_replica_check_test.go: Test coverage
Modified:
- command_ec_encode.go: Call syncAndSelectBestReplica before encoding
- command_volume_tier_move.go: Call syncAndSelectBestReplica before moving
Fixes#7797
* test: add integration test for replicated volume sync during ec.encode
* test: improve retry logic for replicated volume integration test
* fix: resolve JWT issue in integration tests by using empty security.toml
* address review comments: add readNeedleMeta, parallelize status fetch, fix collection param, fix test issues
* test: use collection parameter consistently in replica sync test
* fix: convert weed binary path to absolute to work with changed working directory
* fix: remove skip behavior, keep tests failing on missing binary
* fix: always check recency for each needle, add divergent replica test
* Unify the parameter to disable dry-run on weed shell commands to --apply (instead of --force).
* lint
* refactor
* Execution Order Corrected
* handle deprecated force flag
* fix help messages
* Refactoring]: Using flag.FlagSet.Visit()
* consistent with other commands
* Checks for both flags
* fix toml files
---------
Co-authored-by: chrislu <chris.lu@gmail.com>
* feat(weed.move): add a speed limit parameter of moving files
* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
Chris LuandGitHub
Max DenushevandGitHub
skycopeandGitHub
BrianandGitHub
divanikus
chrislu
Chris Lu