mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-31 21:31:24 +00:00
da087f77b392e8de065f9863cfec609eee23bf3e
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Chris LuandGitHub
|
627b5e9d59 |
shell: parse every collection filter the same way (#10955)
* worker: move the collection filter parser into weed/util/wildcard
The parser sits beside the volume-list filtering it was written for, in
weed/plugin/worker, which imports weed/shell — so the shell commands that
parse the same filter three other ways can never call it. Move it down to
weed/util/wildcard, next to the comma-separated wildcard helper it already
replaced, leaving the behavior unchanged.
* shell: parse every collection filter the same way
The shell parsed a collection filter three ways: compileCollectionPattern
compiled one regex for ec.encode, ec.decode, volume.balance and the tier
commands; volume.list and volume.deleteEmpty matched a single wildcard; and
volume.tier.move, volume.fix.replication and volume.configure.replication
called filepath.Match on their own. None of them took a list, so
"ec.encode -collection=a,b" selected nothing, the same way the admin UI did.
They all go through the shared matcher now: a comma-separated list of names,
"*" and "?" wildcards, "_default" for the collection with no name, and regex
entries. The one thing that stays per-command is what an empty value means -
every collection for -collectionPattern, the unnamed collection for the ec
and tier -collection flag - so compileCollectionPattern keeps that mapping.
The matchers are compiled once per command instead of once per volume, and a
regex entry now has to match the whole name unless it anchors itself, so
-collection=bucket no longer picks up mybucket2.
* shell: keep dots in collection names, and commas inside a regex
A dot no longer marks an entry as a regex, so a collection named "my.bucket"
matches itself and not "my-bucket" - the difference decides which volumes
volume.deleteEmpty and volume.tier.move touch. A dot still counts when it is
quantified, so "bucket.*" stays a prefix regex.
The comma split also leaves alone the commas inside a character class or a
repetition count, so "bucket[0-9]{1,3}" stays one entry instead of becoming
two broken fragments.
* shell: let a regex entry match its own spelling
A collection named after regex syntax, say "logs(2024)", was unreachable:
the entry compiled to a pattern that matches "logs2024" instead. Match the
entry verbatim as well, so naming a collection always selects it, whatever
characters it holds.
* shell: reject a collection filter that names no collection
A value of "," parsed to no entries and then matched every collection, so a
typo widened ec.encode or volume.deleteEmpty to the whole cluster. Only a
genuinely empty filter means "all collections"; anything else has to name one.
* shell: keep commas inside a regex group out of the entry split
The split already left alone the commas inside a character class or a
repetition count, but not the ones inside a group, so "bucket(foo,bar)"
was cut into two fragments that no longer compile.
* shell: cover escaping a collection name that is not a regex
A name like "logs(2024" does not parse as a regex on its own; escaping it,
"logs\(2024", reaches it. Pin that so the escape hatch does not regress.
* shell: split entries only on commas inside a closed regex construct
An unmatched "{" or "[" made the splitter swallow every comma after it, so
"foo{bar,videos" became one entry that matches neither collection - the
silent no-op this filter work exists to remove. A construct now has to close
before its commas stop separating entries.
* shell: skip character classes while scanning a regex group
A ")" inside a class is a literal, so "(a[)],b)" ended its group early and
split into two fragments that no longer compile.
* shell: cover escaping a comma inside a collection name
A comma separates entries, so a name holding one is reached by escaping it.
* shell: follow the regexp parser when scanning a character class
A "]" leading a class is a member of it, and a POSIX class such as
"[:alpha:]" carries its own "]", so stopping at the first one cut a valid
filter like "(a[]),],b)" into fragments and rejected it.
|
||
|
Chris LuandGitHub
|
f09e8345c6 |
storage: stop keeping the remote storage key on the master (#10672)
A master decides nothing from it. Every caller that read it was asking whether a volume is remote, which the backend name answers, and the value itself is reported on demand by the server holding the volume, through the volume info in ReadVolumeFileStatus. It is also the one string here that cannot be shared: unique per volume, so unlike the collection and backend names it carries its own characters for every volume a master tracks. VolumeInfo goes from 136 bytes to 120. 800k volumes registered from a heartbeat that has been over the wire go from 214 to 163 B/volume when tiered. The volume server's own status page keeps showing the key, now read from the volume it holds rather than relayed through a master, which is also where the other volume server implementation reads it. The heartbeat digest drops it on the same grounds: a change to something the master does not hold cannot make its copy stale. Both implementations and their shared vectors move together, and the field-coverage test now names what is deliberately not retained rather than being loosened. |
||
|
Chris LuandGitHub
|
4fb3e22a01 |
fix(tiering): never delete a shared remote object while replicas still reference it (#9942)
* tiering: stop a shared remote object being deleted while replicas still point at it A remote-tiered volume's .dat content lives only in one cloud object that all N replica .vif files point at. Deleting that object while destroying any one replica, or before a downloaded replica is durable, bricks the survivors. - volume.tier.move cleanup now deletes old replicas with keepRemoteData=true so surviving replicas keep the shared object. Document why the alreadyPlaced anchor needs no replica sync (same-object replicas are byte-identical). - VolumeTierMoveDatFromRemote now fsyncs the downloaded .dat, fsyncs the containing directory, trims the .vif (fsynced) and swaps to the local DiskFile BEFORE deleting the remote object, on both the keep-remote and delete paths. Only the final DeleteFile is gated by keep_remote_dat_file, so a keep-remote download leaves the replica served from local disk rather than the shared object, and a crash before delete merely leaks the object. - volume.tier.download keeps the shared object for every replica except the last, which deletes it. - s3 and rclone download paths fsync the .dat before close. * storage: swap the volume data backend under the data lock The tier-download swap closed v.DataBackend and assigned the new local DiskFile without holding dataFileAccessLock, racing concurrent reads/writes (use of a closed file / nil deref). Add an exported Volume.SwapDataBackend that performs the close-and-replace under the lock, and call it from the tier download. * server: skip directory fsync on Windows in the tier download path os.Open(dir).Sync() is unsupported on Windows and returns an error, which would fail VolumeTierMoveDatFromRemote entirely there. Skip the directory fsync on Windows, matching how the storage-side helper tolerates the unsupported case. * shell: make multi-replica tier.download resilient to already-local replicas If a multi-replica download is interrupted and retried, a replica made local in the prior attempt returns "already on local disk", which aborted the whole command and left the remaining remote replicas dangling. Treat that case as a skip-and-continue so a retry completes the rest. * server: assert downloaded .dat content, not just length, in the tier test A length-only check passes even if the bytes are corrupted; compare the full content of the local .dat against the original. |
||
|
Chris LuandGitHub
|
7fbdb9b7b7 |
feat(shell): add volume.tier.compact command to reclaim cloud storage space (#8715)
* feat(shell): add volume.tier.compact command to reclaim cloud storage space Adds a new shell command that automates compaction of cloud tier volumes. When files are deleted from remote-tiered volumes, space is not reclaimed on the cloud storage. This command orchestrates: download from remote, compact locally, and re-upload to reclaim deleted space. Closes #8563 * fix: log cleanup errors in compactVolumeOnServer instead of discarding them Helps operators diagnose leftover temp files (.cpd/.cpx) if cleanup fails after a compaction or commit failure. * fix: return aggregate error from loop and use regex for collection filter - Track and return error count when one or more volumes fail to compact, so callers see partial failures instead of always getting nil. - Use compileCollectionPattern for -collection in -volumeId mode too, so regex patterns work consistently with the flag description. Empty pattern (no -collection given) matches all collections. |