* worker: move the collection filter parser into weed/util/wildcard
The parser sits beside the volume-list filtering it was written for, in
weed/plugin/worker, which imports weed/shell — so the shell commands that
parse the same filter three other ways can never call it. Move it down to
weed/util/wildcard, next to the comma-separated wildcard helper it already
replaced, leaving the behavior unchanged.
* shell: parse every collection filter the same way
The shell parsed a collection filter three ways: compileCollectionPattern
compiled one regex for ec.encode, ec.decode, volume.balance and the tier
commands; volume.list and volume.deleteEmpty matched a single wildcard; and
volume.tier.move, volume.fix.replication and volume.configure.replication
called filepath.Match on their own. None of them took a list, so
"ec.encode -collection=a,b" selected nothing, the same way the admin UI did.
They all go through the shared matcher now: a comma-separated list of names,
"*" and "?" wildcards, "_default" for the collection with no name, and regex
entries. The one thing that stays per-command is what an empty value means -
every collection for -collectionPattern, the unnamed collection for the ec
and tier -collection flag - so compileCollectionPattern keeps that mapping.
The matchers are compiled once per command instead of once per volume, and a
regex entry now has to match the whole name unless it anchors itself, so
-collection=bucket no longer picks up mybucket2.
* shell: keep dots in collection names, and commas inside a regex
A dot no longer marks an entry as a regex, so a collection named "my.bucket"
matches itself and not "my-bucket" - the difference decides which volumes
volume.deleteEmpty and volume.tier.move touch. A dot still counts when it is
quantified, so "bucket.*" stays a prefix regex.
The comma split also leaves alone the commas inside a character class or a
repetition count, so "bucket[0-9]{1,3}" stays one entry instead of becoming
two broken fragments.
* shell: let a regex entry match its own spelling
A collection named after regex syntax, say "logs(2024)", was unreachable:
the entry compiled to a pattern that matches "logs2024" instead. Match the
entry verbatim as well, so naming a collection always selects it, whatever
characters it holds.
* shell: reject a collection filter that names no collection
A value of "," parsed to no entries and then matched every collection, so a
typo widened ec.encode or volume.deleteEmpty to the whole cluster. Only a
genuinely empty filter means "all collections"; anything else has to name one.
* shell: keep commas inside a regex group out of the entry split
The split already left alone the commas inside a character class or a
repetition count, but not the ones inside a group, so "bucket(foo,bar)"
was cut into two fragments that no longer compile.
* shell: cover escaping a collection name that is not a regex
A name like "logs(2024" does not parse as a regex on its own; escaping it,
"logs\(2024", reaches it. Pin that so the escape hatch does not regress.
* shell: split entries only on commas inside a closed regex construct
An unmatched "{" or "[" made the splitter swallow every comma after it, so
"foo{bar,videos" became one entry that matches neither collection - the
silent no-op this filter work exists to remove. A construct now has to close
before its commas stop separating entries.
* shell: skip character classes while scanning a regex group
A ")" inside a class is a literal, so "(a[)],b)" ended its group early and
split into two fragments that no longer compile.
* shell: cover escaping a comma inside a collection name
A comma separates entries, so a name holding one is reached by escaping it.
* shell: follow the regexp parser when scanning a character class
A "]" leading a class is a member of it, and a POSIX class such as
"[:alpha:]" carries its own "]", so stopping at the first one cut a valid
filter like "(a[]),],b)" into fragments and rejected it.
LiveMoveVolume and the copy, tail, delete, mark, replicate, and
configure helpers around it issued every RPC on context.Background(), so
a caller had no way to bound or abort a move once it started. They now
take a context, which the exported LiveMoveVolume in particular needs:
callers outside the shell drive long moves and want to stop them.
The deferred restore in copyVolume runs on a detached, bounded context
rather than the caller's. Marking the source writable again is cleanup,
and cancelling the copy must not skip it and leave the volume readonly —
the same guard balance_task.go already applies for the same reason.
Shell commands pass context.Background(): their Do signature carries no
context, and changing it would touch every command in the package.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* feat: add collection pattern to delete empty volumes
Co-authored-by: Codex <noreply@openai.com>
* shell: match collection pattern with wildcard matcher
Use wildcard.MatchesWildcard in the shared collection-pattern helper,
matching command_volume_fix_replication's matchCollectionPattern. The
flag only advertises '*' and '?', which is exactly what the matcher
supports.
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-06-29 12:43:19 -07:00
Chris LuGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix(volume): don't fatal on missing .idx for remote-tiered volume
A .vif left behind without its .idx (orphaned by a crashed move, partial
copy, or hand-edit) would trip glog.Fatalf in checkIdxFile and take the
whole volume server down on boot, killing every healthy volume on it
too. For remote-tiered volumes treat it as a per-volume load error so
the server can come up and the operator can clean up the stray .vif.
Refs #9331.
* fix(balance): skip remote-tiered volumes in admin balance detection
The admin/worker balance detector had no equivalent of the shell-side
guard ("does not move volume in remote storage" in
command_volume_balance.go), so it scheduled moves on remote-tiered
volumes. The "move" copies .idx/.vif to the destination and then calls
Volume.Destroy on the source, which calls backendStorage.DeleteFile —
deleting the remote object the destination's new .vif now points at.
Populate HasRemoteCopy on the metrics emitted by both the admin
maintenance scanner and the worker's master poll, then drop those
volumes at the top of Detection.
Fixes#9331.
* Apply suggestion from @gemini-code-assist[bot]
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix(volume): keep remote data on volume-move-driven delete
The on-source delete after a volume move (admin/worker balance and
shell volume.move) ran Volume.Destroy with no way to opt out of the
remote-object cleanup. Volume.Destroy unconditionally calls
backendStorage.DeleteFile for remote-tiered volumes, so a successful
move would copy .idx/.vif to the destination and then nuke the cloud
object the destination's new .vif was already pointing at.
Add VolumeDeleteRequest.keep_remote_data and plumb it through
Store.DeleteVolume / DiskLocation.DeleteVolume / Volume.Destroy. The
balance task and shell volume.move set it to true; the post-tier-upload
cleanup of other replicas and the over-replication trim in
volume.fix.replication also set it to true since the remote object is
still referenced. Other real-delete callers keep the default. The
delete-before-receive path in VolumeCopy also sets it: the inbound copy
carries a .vif that may reference the same cloud object as the
existing volume.
Refs #9331.
* test(storage): in-process remote-tier integration tests
Cover the four operations the user is most likely to run against a
cloud-tiered volume — balance/move, vacuum, EC encode, EC decode — by
registering a local-disk-backed BackendStorage as the "remote" tier and
exercising the real Volume / DiskLocation / EC encoder code paths.
Locks in:
- Destroy(keepRemoteData=true) preserves the remote object (move case)
- Destroy(keepRemoteData=false) deletes it (real-delete case)
- Vacuum/compact on a remote-tier volume never deletes the remote object
- EC encode requires the local .dat (callers must download first)
- EC encode + rebuild round-trips after a tier-down
Tests run in-process and finish in under a second total — no cluster,
binary, or external storage required.
* fix(rust-volume): keep remote data on volume-move-driven delete
Mirror the Go fix in seaweed-volume: plumb keep_remote_data through
grpc volume_delete → Store.delete_volume → DiskLocation.delete_volume
→ Volume.destroy, and skip the s3-tier delete_file call when the flag
is set. The pre-receive cleanup in volume_copy passes true for the
same reason as the Go side: the inbound copy carries a .vif that may
reference the same cloud object as the existing volume.
The Rust loader already warns rather than fataling on a stray .vif
without an .idx (volume.rs load_index_inmemory / load_index_redb), so
no counterpart to the Go fatal-on-missing-idx fix is needed.
Refs #9331.
* fix(volume): preserve remote tier on IO-error eviction; fix EC test target
Two review nits:
- Store.MaybeAddVolumes' periodic cleanup pass deleted IO-errored
volumes with keepRemoteData=false, so a transient local fault on a
remote-tiered volume would also nuke the cloud object. Track the
delete reason via a parallel slice and pass keepRemoteData=v.HasRemoteFile()
for IO-error evictions; TTL-expired evictions still pass false.
- TestRemoteTier_ECEncodeDecode_AfterDownload deleted shards 0..3 but
called them "parity" — by the klauspost/reedsolomon convention shards
0..DataShardsCount-1 are data and DataShardsCount..TotalShardsCount-1
are parity. Switch the loop to delete the parity range so the
intent matches the indices.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Unify the parameter to disable dry-run on weed shell commands to --apply (instead of --force).
* lint
* refactor
* Execution Order Corrected
* handle deprecated force flag
* fix help messages
* Refactoring]: Using flag.FlagSet.Visit()
* consistent with other commands
* Checks for both flags
* fix toml files
---------
Co-authored-by: chrislu <chris.lu@gmail.com>
Chris LuandGitHub
柏杰andGitHub
chrislu
byunghwa.yun