4 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandGitHub	627b5e9d59	shell: parse every collection filter the same way (#10955) ... * worker: move the collection filter parser into weed/util/wildcard The parser sits beside the volume-list filtering it was written for, in weed/plugin/worker, which imports weed/shell — so the shell commands that parse the same filter three other ways can never call it. Move it down to weed/util/wildcard, next to the comma-separated wildcard helper it already replaced, leaving the behavior unchanged. * shell: parse every collection filter the same way The shell parsed a collection filter three ways: compileCollectionPattern compiled one regex for ec.encode, ec.decode, volume.balance and the tier commands; volume.list and volume.deleteEmpty matched a single wildcard; and volume.tier.move, volume.fix.replication and volume.configure.replication called filepath.Match on their own. None of them took a list, so "ec.encode -collection=a,b" selected nothing, the same way the admin UI did. They all go through the shared matcher now: a comma-separated list of names, "*" and "?" wildcards, "_default" for the collection with no name, and regex entries. The one thing that stays per-command is what an empty value means - every collection for -collectionPattern, the unnamed collection for the ec and tier -collection flag - so compileCollectionPattern keeps that mapping. The matchers are compiled once per command instead of once per volume, and a regex entry now has to match the whole name unless it anchors itself, so -collection=bucket no longer picks up mybucket2. * shell: keep dots in collection names, and commas inside a regex A dot no longer marks an entry as a regex, so a collection named "my.bucket" matches itself and not "my-bucket" - the difference decides which volumes volume.deleteEmpty and volume.tier.move touch. A dot still counts when it is quantified, so "bucket.*" stays a prefix regex. The comma split also leaves alone the commas inside a character class or a repetition count, so "bucket[0-9]{1,3}" stays one entry instead of becoming two broken fragments. * shell: let a regex entry match its own spelling A collection named after regex syntax, say "logs(2024)", was unreachable: the entry compiled to a pattern that matches "logs2024" instead. Match the entry verbatim as well, so naming a collection always selects it, whatever characters it holds. * shell: reject a collection filter that names no collection A value of "," parsed to no entries and then matched every collection, so a typo widened ec.encode or volume.deleteEmpty to the whole cluster. Only a genuinely empty filter means "all collections"; anything else has to name one. * shell: keep commas inside a regex group out of the entry split The split already left alone the commas inside a character class or a repetition count, but not the ones inside a group, so "bucket(foo,bar)" was cut into two fragments that no longer compile. * shell: cover escaping a collection name that is not a regex A name like "logs(2024" does not parse as a regex on its own; escaping it, "logs\(2024", reaches it. Pin that so the escape hatch does not regress. * shell: split entries only on commas inside a closed regex construct An unmatched "{" or "[" made the splitter swallow every comma after it, so "foo{bar,videos" became one entry that matches neither collection - the silent no-op this filter work exists to remove. A construct now has to close before its commas stop separating entries. * shell: skip character classes while scanning a regex group A ")" inside a class is a literal, so "(a[)],b)" ended its group early and split into two fragments that no longer compile. * shell: cover escaping a comma inside a collection name A comma separates entries, so a name holding one is reached by escaping it. * shell: follow the regexp parser when scanning a character class A "]" leading a class is a member of it, and a POSIX class such as "[:alpha:]" carries its own "]", so stopping at the first one cut a valid filter like "(a[]),],b)" into fragments and rejected it.	2026-08-25 18:03:52 -07:00
Chris LuandGitHub	9c8d3b6a81	ec: refund the cleared leftover shards' slots in the encode source health check (#10903) ... * erasure_coding: one home for the shard-count to volume-slots conversion * ec: refund the cleared leftover shards' slots in the encode source health check	2026-08-23 21:48:20 -07:00
Chris LuandGitHub	602746f51d	test: EC lifecycle chaos harness, with four fixes it found (#10763) ... * ec: let the encode's balance see a migrating volume's shards across disk-type buckets Shard generation writes beside the source .dat, so a cross-tier encode (source on hdd, -diskType=ssd) leaves the fresh shards in the source disk-type bucket. The encode's internal balance ingested only the target bucket, saw no shards, and planned no moves; the spread guard then correctly aborted the encode (and before that guard existed, the shards silently stayed clumped on the generation host in the wrong tier). EcBalance now takes the encode batch as migratingVolumeIds and ingests those volumes' shards from every bucket, while everything else keeps the bucket filter so a plain ec.balance never drags deliberately tiered shards onto another disk type. The in-memory model delete also becomes bucket-agnostic: a node holds a given shard in exactly one bucket, and a bucket-scoped delete missed cross-bucket moves in the dry-run model. * volume: decode reads shard 0 from its resolved path, not the EC volume's base dir On a multi-disk server a volume's shards can sit on several disks; the store registers each shard with its own path and CollectEcShards resolves them, but FindDatFileSize derived the .ec00 path from the EcVolume's base directory. When shard 0 lived on a sibling disk, VolumeEcShardsToVolume failed with 'open ...ec00: no such file or directory' and ec.decode aborted. * ec: decode re-copies shards the topology claims but the target does not hold An interrupted earlier decode or balance can leave the master believing the decode target holds a shard whose file never landed: the mount registered but the partial copy was cleaned, or the file was swept. The collect step took the topology's word for it, excluded the shard from the copy set, and the decode failed with 'missing shard'. Probe the target's live inventory (VolumeEcShardsInfo) and treat anything it cannot serve as still-to-copy. * ec: decode discovers shards across disk-type buckets Shards sit wherever encode generation and balance left them: a cross-tier encode leaves them in the source disk-type bucket, a partial migration straddles buckets. ec.decode scoped its shard discovery to the -diskType bucket and reported a decodable volume as having no shards at all. Union across buckets, the way the encode's shard verification already does. * test: EC chaos lifecycle harness Randomized, seeded sequences of the EC lifecycle against a live cluster in the production-shaped layout: multiple data disks per server, a separate -dir.idx directory so .ecx/.ecj sidecars are shared across disks, and a tagged ssd tier. Operations cover encode (hdd and ssd targets), balance, shard damage plus rebuild, decode, re-encode, deletes, scrub, tier moves, crash-restarts, sidecar fault injections (a data-dir .vif pushed into the shared idx dir; a stale-generation shard planted beside a newer encode), and interruptions: a real weed shell subprocess killed mid-encode, mid-decode, and mid-balance, with the recovery re-run required to converge. One invariant holds after every step: every stored byte reads back identical and every deleted needle stays deleted. EC_CHAOS_SEED and EC_CHAOS_STEPS make runs reproducible and scalable. A known gap is tolerated and logged rather than fixed here: a shard mounted on two disks of one node (orphan adoption after an interrupted copy) is invisible to ec.balance's dedup and unaddressable by ec.shard.unmount's shard@address form, so no cleanup path exists yet. * test: fail payload-corruption checks on the test goroutine t.Fatalf inside require.Eventually's condition runs on the poller's goroutine, where Goexit kills only that goroutine and the corruption message can be lost behind a generic timeout. Record the mismatch, end the polling, and fail on the test goroutine. Also assert the full shard count in the cross-bucket decode-discovery test.	2026-08-14 17:26:54 -07:00
Chris LuandGitHub	944d967502	refactor: extract EC orchestration into a shared weed/ec package (#10760) ... * shell: move ErrorWaitGroup to weed/util * shell: remove unused CandidateEcNode and EcRack types * ec: extract EC orchestration logic from weed/shell into weed/ec Move the EC node/topology model, balance engine, encode pipeline, decode pipeline, and rebuild engine into a new weed/ec package so shell commands and maintenance workers can share the logic. Shell commands keep flag parsing and delegate through a small ec.Env (dial option, topology fetch, volume locations, lock check). Tests move along with the code. * shell: remove unused proportional-rebalance type stubs * ec: move scrub, replication check, and shard unmount engines into weed/ec * worker: share the EC generation-aware shard counter from weed/ec * ec: gofmt * shell: drop EC aliases with no remaining callers * ec: guard a missing topology hook and nil disk entries in topology helpers * ec: drop trailing newlines from decode error strings * ec: re-check the shell lock before applying shard unmounts * shell: trim -node entries in ec.scrub	2026-08-14 13:54:12 -07:00