mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-20 06:07:05 +00:00
Chris LuandGitHub
602746f51d
* ec: let the encode's balance see a migrating volume's shards across disk-type buckets Shard generation writes beside the source .dat, so a cross-tier encode (source on hdd, -diskType=ssd) leaves the fresh shards in the source disk-type bucket. The encode's internal balance ingested only the target bucket, saw no shards, and planned no moves; the spread guard then correctly aborted the encode (and before that guard existed, the shards silently stayed clumped on the generation host in the wrong tier). EcBalance now takes the encode batch as migratingVolumeIds and ingests those volumes' shards from every bucket, while everything else keeps the bucket filter so a plain ec.balance never drags deliberately tiered shards onto another disk type. The in-memory model delete also becomes bucket-agnostic: a node holds a given shard in exactly one bucket, and a bucket-scoped delete missed cross-bucket moves in the dry-run model. * volume: decode reads shard 0 from its resolved path, not the EC volume's base dir On a multi-disk server a volume's shards can sit on several disks; the store registers each shard with its own path and CollectEcShards resolves them, but FindDatFileSize derived the .ec00 path from the EcVolume's base directory. When shard 0 lived on a sibling disk, VolumeEcShardsToVolume failed with 'open ...ec00: no such file or directory' and ec.decode aborted. * ec: decode re-copies shards the topology claims but the target does not hold An interrupted earlier decode or balance can leave the master believing the decode target holds a shard whose file never landed: the mount registered but the partial copy was cleaned, or the file was swept. The collect step took the topology's word for it, excluded the shard from the copy set, and the decode failed with 'missing shard'. Probe the target's live inventory (VolumeEcShardsInfo) and treat anything it cannot serve as still-to-copy. * ec: decode discovers shards across disk-type buckets Shards sit wherever encode generation and balance left them: a cross-tier encode leaves them in the source disk-type bucket, a partial migration straddles buckets. ec.decode scoped its shard discovery to the -diskType bucket and reported a decodable volume as having no shards at all. Union across buckets, the way the encode's shard verification already does. * test: EC chaos lifecycle harness Randomized, seeded sequences of the EC lifecycle against a live cluster in the production-shaped layout: multiple data disks per server, a separate -dir.idx directory so .ecx/.ecj sidecars are shared across disks, and a tagged ssd tier. Operations cover encode (hdd and ssd targets), balance, shard damage plus rebuild, decode, re-encode, deletes, scrub, tier moves, crash-restarts, sidecar fault injections (a data-dir .vif pushed into the shared idx dir; a stale-generation shard planted beside a newer encode), and interruptions: a real weed shell subprocess killed mid-encode, mid-decode, and mid-balance, with the recovery re-run required to converge. One invariant holds after every step: every stored byte reads back identical and every deleted needle stays deleted. EC_CHAOS_SEED and EC_CHAOS_STEPS make runs reproducible and scalable. A known gap is tolerated and logged rather than fixed here: a shard mounted on two disks of one node (orphan adoption after an interrupted copy) is invisible to ec.balance's dedup and unaddressable by ec.shard.unmount's shard@address form, so no cleanup path exists yet. * test: fail payload-corruption checks on the test goroutine t.Fatalf inside require.Eventually's condition runs on the poller's goroutine, where Goexit kills only that goroutine and the corruption message can be lost behind a generic timeout. Record the mismatch, end the polling, and fail on the test goroutine. Also assert the full shard count in the cross-bucket decode-discovery test.
64 lines
2.4 KiB
Go
64 lines
2.4 KiB
Go
package ec
|
|
|
|
import (
|
|
"testing"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
|
|
"github.com/seaweedfs/seaweedfs/weed/storage/types"
|
|
)
|
|
|
|
// TestEcBalanceMigratesCrossDiskTypeShards: a cross-tier encode generates its
|
|
// shards beside the source .dat — in the SOURCE disk-type bucket — while the
|
|
// encode's balance targets -diskType. The balance must still see and spread
|
|
// those shards when the volume is named as migrating; without that, the
|
|
// planner finds nothing in the target bucket, plans no moves, and the encode's
|
|
// spread guard aborts a perfectly good encode.
|
|
func TestEcBalanceMigratesCrossDiskTypeShards(t *testing.T) {
|
|
allShards := []erasure_coding.ShardId{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13}
|
|
|
|
build := func() []*EcNode {
|
|
// Volume 1's fresh shards sit in the default (hdd, "") bucket of dn1;
|
|
// the balance runs with -diskType=ssd.
|
|
return []*EcNode{
|
|
newEcNode("dc1", "rack1", "dn1", 100).addEcVolumeAndShardsForTest(1, "c1", allShards),
|
|
newEcNode("dc1", "rack2", "dn2", 100),
|
|
newEcNode("dc1", "rack3", "dn3", 100),
|
|
}
|
|
}
|
|
|
|
// Without the migrating hint the target-bucket filter hides the shards:
|
|
// nothing moves. This is the standalone ec.balance semantic — shards of
|
|
// other tiers stay where they are.
|
|
ecb := &ecBalancer{
|
|
ecNodes: build(),
|
|
applyBalancing: false,
|
|
diskType: types.SsdType,
|
|
}
|
|
if err := ecb.balance([]string{"c1"}); err != nil {
|
|
t.Fatalf("balance without migrating hint: %v", err)
|
|
}
|
|
if got := ecb.ecNodes[0].LocalShardIdCount(1); got != len(allShards) {
|
|
t.Fatalf("without migrating hint, cross-type shards must stay put; dn1 has %d", got)
|
|
}
|
|
|
|
// With the volume named as migrating, the balance ingests the shards from
|
|
// the source bucket and spreads them.
|
|
ecb = &ecBalancer{
|
|
ecNodes: build(),
|
|
applyBalancing: false,
|
|
diskType: types.SsdType,
|
|
migratingVolumeIds: map[uint32]bool{1: true},
|
|
}
|
|
if err := ecb.balance([]string{"c1"}); err != nil {
|
|
t.Fatalf("balance with migrating hint: %v", err)
|
|
}
|
|
onDn1 := ecb.ecNodes[0].LocalShardIdCount(1)
|
|
spread := ecb.ecNodes[1].LocalShardIdCount(1) + ecb.ecNodes[2].LocalShardIdCount(1)
|
|
if onDn1 == len(allShards) || spread == 0 {
|
|
t.Fatalf("migrating volume's shards did not spread: dn1=%d others=%d", onDn1, spread)
|
|
}
|
|
if onDn1+spread != len(allShards) {
|
|
t.Fatalf("shards lost or duplicated during dry-run spread: dn1=%d others=%d", onDn1, spread)
|
|
}
|
|
}
|