3 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandGitHub	76a1983c86	test: re-lock and retry every chaos command, not just the balance (#10770) ... The harness kills shells mid-command, and the master releases the dead session's lock only when it notices the connection is gone. That cleanup lands after the harness has already re-acquired the lock, so it can clear the lock this run holds and the next command refuses with need to run "lock" first to continue recoverInterruptedBalance answered that the way an operator would -- run lock again and retry -- but the encode and decode recoveries called shellCommand once and required success, so the same reap failed the run outright. Move the retry into shellCommand: the reap can land during any command that follows a kill, not only a balance.	2026-08-15 14:13:37 -07:00
Chris LuandGitHub	1c926e8fac	test: systematic EC interruption verification — exhaustive model check + deterministic kill matrix (#10764) ... * ec: bounded-exhaustive model check of the volume lifecycle The randomized chaos harness samples the state space; this enumerates it. The lifecycle is a state machine whose steps mirror the pipelines in this package, and the checker explores every schedule within the bound: a crash at every step boundary, an error return running the rollback (itself crashable at every step), a volume-server restart applying the startup reconciliation rules in every quiescent state, and the prescribed restart-based recovery from every crashed state. Checked in every reachable state: durability (a readable copy always exists), at most one generation mounted, and — a property the sweep discipline turns out to guarantee — at most one generation's files on disk. From every quiescent state the recovery must converge to a clean volume. Runs in well under a second. * test: deterministic EC interruption matrix Enumerate every phase of every interruptible EC operation and kill a real weed shell exactly when the phase announces itself on the command output, instead of at a random moment: four encode phases, four decode phases, and the balance's move phase (set up with -rebalance=false so a move is guaranteed). Each scenario prepares its precondition, kills at the marker, runs the prescribed recovery, and verifies every stored byte still reads back identical. The interruption recoveries move out of the randomized ops into shared chaosRun helpers both drivers use. * test: make the randomized EC chaos walk opt-in The systematic layers — the interruption matrix and the lifecycle model check — carry the CI coverage deterministically; the randomized walk stays for exploratory runs, behind EC_CHAOS_SEED. * ci: bound the EC integration suite by the job budget, not go test's default The suite with the interruption matrix runs close to the default 10m binary timeout on slower runners. * test: require every interruption-matrix marker to appear A marker that never prints means a pipeline refactor renamed or dropped the progress line; silently degenerating into a no-interruption run would let CI pass without exercising the boundary the scenario names. Also recheck the marker channel after the wait: a shell that prints and exits at once makes both channels ready, and select picking the exit case must not report a printed marker as missed.	2026-08-14 17:45:11 -07:00
Chris LuandGitHub	602746f51d	test: EC lifecycle chaos harness, with four fixes it found (#10763) ... * ec: let the encode's balance see a migrating volume's shards across disk-type buckets Shard generation writes beside the source .dat, so a cross-tier encode (source on hdd, -diskType=ssd) leaves the fresh shards in the source disk-type bucket. The encode's internal balance ingested only the target bucket, saw no shards, and planned no moves; the spread guard then correctly aborted the encode (and before that guard existed, the shards silently stayed clumped on the generation host in the wrong tier). EcBalance now takes the encode batch as migratingVolumeIds and ingests those volumes' shards from every bucket, while everything else keeps the bucket filter so a plain ec.balance never drags deliberately tiered shards onto another disk type. The in-memory model delete also becomes bucket-agnostic: a node holds a given shard in exactly one bucket, and a bucket-scoped delete missed cross-bucket moves in the dry-run model. * volume: decode reads shard 0 from its resolved path, not the EC volume's base dir On a multi-disk server a volume's shards can sit on several disks; the store registers each shard with its own path and CollectEcShards resolves them, but FindDatFileSize derived the .ec00 path from the EcVolume's base directory. When shard 0 lived on a sibling disk, VolumeEcShardsToVolume failed with 'open ...ec00: no such file or directory' and ec.decode aborted. * ec: decode re-copies shards the topology claims but the target does not hold An interrupted earlier decode or balance can leave the master believing the decode target holds a shard whose file never landed: the mount registered but the partial copy was cleaned, or the file was swept. The collect step took the topology's word for it, excluded the shard from the copy set, and the decode failed with 'missing shard'. Probe the target's live inventory (VolumeEcShardsInfo) and treat anything it cannot serve as still-to-copy. * ec: decode discovers shards across disk-type buckets Shards sit wherever encode generation and balance left them: a cross-tier encode leaves them in the source disk-type bucket, a partial migration straddles buckets. ec.decode scoped its shard discovery to the -diskType bucket and reported a decodable volume as having no shards at all. Union across buckets, the way the encode's shard verification already does. * test: EC chaos lifecycle harness Randomized, seeded sequences of the EC lifecycle against a live cluster in the production-shaped layout: multiple data disks per server, a separate -dir.idx directory so .ecx/.ecj sidecars are shared across disks, and a tagged ssd tier. Operations cover encode (hdd and ssd targets), balance, shard damage plus rebuild, decode, re-encode, deletes, scrub, tier moves, crash-restarts, sidecar fault injections (a data-dir .vif pushed into the shared idx dir; a stale-generation shard planted beside a newer encode), and interruptions: a real weed shell subprocess killed mid-encode, mid-decode, and mid-balance, with the recovery re-run required to converge. One invariant holds after every step: every stored byte reads back identical and every deleted needle stays deleted. EC_CHAOS_SEED and EC_CHAOS_STEPS make runs reproducible and scalable. A known gap is tolerated and logged rather than fixed here: a shard mounted on two disks of one node (orphan adoption after an interrupted copy) is invisible to ec.balance's dedup and unaddressable by ec.shard.unmount's shard@address form, so no cleanup path exists yet. * test: fail payload-corruption checks on the test goroutine t.Fatalf inside require.Eventually's condition runs on the poller's goroutine, where Goexit kills only that goroutine and the corruption message can be lost behind a generic timeout. Record the mismatch, end the polling, and fail on the test goroutine. Also assert the full shard count in the cross-bucket decode-discovery test.	2026-08-14 17:26:54 -07:00