mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-07-30 03:53:18 +00:00
* admin/topology: carry the volume server address on DiskInfo The planning DiskInfo exposed only the node id, which can be an opaque label rather than ip:port. Record the address too so callers can resolve the physical machine a disk sits on. * ec.balance: spread a volume's shards across machines, not just nodes Volume servers sharing a host are one fault domain, but the within-rack spread treated them as independent nodes, so one box could end up holding more shards of a volume than EC can afford to lose. Add a machine (host) tier between rack and node: the within-rack pass spreads each volume across machines, and the global load phase no longer re-concentrates a volume onto a machine it already sits on. Host defaults to the node id, so clusters with one server per host are unchanged. * ec placement: prefer machines holding fewer of a volume's shards EC allocation and repair picked the least-loaded node in a rack with no regard for which physical machine it sits on, so a volume's shards could pile onto several servers of one box. Rank candidate nodes by their machine's shard count first, then the node's own. The machine is derived from the volume server address carried on DiskInfo, falling back to the node id, matching how the balancer resolves it. * volume.balance: don't move a replica onto a machine already holding one isGoodMove only rejected a move onto the same data node, so two replicas could land on two volume servers of one box and a single machine failure would lose both. Reject a target whose host already holds another replica of the volume. Best-effort: balancing simply skips and tries the next target. * volume allocation: spread same-rack replicas across machines PickNodesByWeight filled the same-rack replica picks by weight alone, so replicas could co-locate on one box. Prefer candidates on not-yet-used hosts, falling back when too few distinct machines exist. Data-center and rack tiers have no host, so their ordering is unchanged. * ec.balance: harden machine spread against re-concentration and capped machines Two cases where the machine-aware spread could still leave a volume badly placed: - The global load phase could move a shard of a volume onto a machine that already held it, raising that machine's count and undoing the within-rack spread (a 4/4/3/3 layout could become 3/5/3/3, past parity for 10+4). Limit the load-only fallback to same-machine moves, which leave a machine's count unchanged; cross-machine concentration is no longer allowed for load alone. - The within-rack spread chose a destination machine by free slots alone, so if that machine's only nodes were already at the SameRackCount cap it skipped the move instead of trying another machine. Require a machine to have a node that can actually take the shard before selecting it. * reduce comments across the machine-affinity change Trim narration down to the non-obvious why; one terse line where a block was overkill. * ec.balance: gate machine spread on fault-tolerance feasibility Spreading a volume evenly across machines only helps when there are enough that each can stay within EC's parity tolerance (numMachines >= ceil(total/parity)). With fewer -- or wildly unequal -- machines it can't make a machine loss survivable anyway, and forcing it fights capacity: e.g. a cluster of 12 volume servers on one host and 2 on another would have half of every volume crammed onto the 2-server box. So spread across machines only when it's achievable; otherwise fall back to per-node spread and let capacity/global balancing decide. The global load phase applies the same test: it protects a volume's machine spread (no cross-machine move that raises a machine's count past the source's) only where that spread is achievable, so heterogeneous clusters still level by fullness. * ec.balance worker: group servers by host when planning The worker built its planner topology without recording each server's host, so automated ec.balance treated ports on one machine as independent nodes and could concentrate a volume's shards on one physical box. Set the host from the volume server address, matching the shell path. * volume.balance worker: don't move a replica onto a machine holding one The worker compared only node ids, and the replica map dropped the server address, so it could move replicas onto different ports of one machine. Carry the host on ReplicaLocation (from the server address) and reject a target whose host already holds another replica of the volume. Best-effort, matching the shell. * ec.balance: judge machine-spread feasibility by the rack's shards The within-rack and global feasibility checks compared the whole volume's shard count against a rack's machine count, so a rack holding only part of a volume after cross-rack spreading -- e.g. 7 of a 10+4 volume across 2 machines -- was wrongly judged infeasible and fell back to node spread, which could pile 6 shards onto one host, past parity. Gate on the rack's own shard count of the volume instead. * ec.balance: spread a volume's shards across machines by combined count EC recovers from any loss within parity regardless of shard type, so what bounds a machine's exposure is its total shards of the volume, not data and parity separately. Spreading the two independently let each type's remainder land on the same machine -- ceil(d/M)+ceil(p/M) can exceed ceil(total/M), e.g. a 5/3 split where 4/4 was achievable, past parity. Balance the combined count in one pass; disk-level data/parity anti-affinity stays in pickBestDiskOnNode. * ec.balance: don't let the imbalance threshold skip an over-parity machine The within-rack spread gated on relative skew ((max-min)/avg > threshold), so a worker threshold of 0.5 skipped an exactly-50%-skewed layout like 5/4/3 for a 10+4 volume, leaving 5 shards -- past parity -- on one machine. The even cap (ceil(shards/groups)) is the real bound and the move loop already sheds only what exceeds it, so drop the threshold gate from the within-rack phase (machine and node): a balanced rack stays a no-op while any over-cap machine is always fixed. * ec.balance: keep the imbalance threshold for the node fallback Dropping the threshold from the whole within-rack phase made the node fallback too eager: it runs only when machine fault tolerance is unachievable, so it is cosmetic load distribution that should defer to the global utilization phase. Without the gate it would, for a one-server-per-host 6/4 split at threshold 0.5, schedule a count move that worsens utilization balance. Restore the threshold there; machine spreading keeps bypassing it, since that bound is durability, not cosmetic skew.
127 lines
4.9 KiB
Go
127 lines
4.9 KiB
Go
package topology
|
|
|
|
import (
|
|
"sync"
|
|
"time"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
|
|
)
|
|
|
|
// TaskSource represents a single source in a multi-source task (for replicated volume cleanup)
|
|
type TaskSource struct {
|
|
SourceServer string `json:"source_server"`
|
|
SourceDisk uint32 `json:"source_disk"`
|
|
StorageChange StorageSlotChange `json:"storage_change"` // Storage impact on this source
|
|
EstimatedSize int64 `json:"estimated_size"` // Estimated size for this source
|
|
}
|
|
|
|
// TaskDestination represents a single destination in a multi-destination task
|
|
type TaskDestination struct {
|
|
TargetServer string `json:"target_server"`
|
|
TargetDisk uint32 `json:"target_disk"`
|
|
StorageChange StorageSlotChange `json:"storage_change"` // Storage impact on this destination
|
|
EstimatedSize int64 `json:"estimated_size"` // Estimated size for this destination
|
|
}
|
|
|
|
// taskState represents the current state of tasks affecting the topology (internal)
|
|
// Uses unified multi-source/multi-destination design:
|
|
// - Single-source tasks (balance, vacuum, replication): 1 source, 1 destination
|
|
// - Multi-source EC tasks (replicated volumes): N sources, M destinations
|
|
type taskState struct {
|
|
VolumeID uint32 `json:"volume_id"`
|
|
TaskType TaskType `json:"task_type"`
|
|
Status TaskStatus `json:"status"`
|
|
StartedAt time.Time `json:"started_at"`
|
|
CompletedAt time.Time `json:"completed_at,omitempty"`
|
|
EstimatedSize int64 `json:"estimated_size"` // Total estimated size of task
|
|
|
|
// Unified source and destination arrays (always used)
|
|
Sources []TaskSource `json:"sources"` // Source locations (1+ for all task types)
|
|
Destinations []TaskDestination `json:"destinations"` // Destination locations (1+ for all task types)
|
|
}
|
|
|
|
// DiskInfo represents a disk with its current state and ongoing tasks (public for external access)
|
|
type DiskInfo struct {
|
|
NodeID string `json:"node_id"`
|
|
Address string `json:"address"` // volume server ip:port; NodeID may be an opaque id, so this is the source for the physical-machine host
|
|
DiskID uint32 `json:"disk_id"`
|
|
DiskType string `json:"disk_type"`
|
|
DataCenter string `json:"data_center"`
|
|
Rack string `json:"rack"`
|
|
DiskInfo *master_pb.DiskInfo `json:"disk_info"`
|
|
LoadCount int `json:"load_count"` // Number of active tasks
|
|
}
|
|
|
|
// activeDisk represents internal disk state (private)
|
|
type activeDisk struct {
|
|
*DiskInfo
|
|
pendingTasks []*taskState
|
|
assignedTasks []*taskState
|
|
recentTasks []*taskState // Completed in last N seconds
|
|
}
|
|
|
|
// activeNode represents a node with its disks (private)
|
|
type activeNode struct {
|
|
nodeID string
|
|
dataCenter string
|
|
rack string
|
|
nodeInfo *master_pb.DataNodeInfo
|
|
disks map[uint32]*activeDisk // DiskID -> activeDisk
|
|
}
|
|
|
|
// ActiveTopology provides a real-time view of cluster state with task awareness
|
|
type ActiveTopology struct {
|
|
// Core topology from master
|
|
topologyInfo *master_pb.TopologyInfo
|
|
lastUpdated time.Time
|
|
|
|
// Structured topology for easy access (private)
|
|
nodes map[string]*activeNode // NodeID -> activeNode
|
|
disks map[string]*activeDisk // "NodeID:DiskID" -> activeDisk
|
|
|
|
// Performance indexes for O(1) lookups (private)
|
|
volumeIndex map[uint32][]string // VolumeID -> list of "NodeID:DiskID" where volume replicas exist
|
|
ecShardIndex map[uint32][]string // VolumeID -> list of "NodeID:DiskID" where EC shards exist
|
|
|
|
// Task states affecting the topology (private)
|
|
pendingTasks map[string]*taskState
|
|
assignedTasks map[string]*taskState
|
|
recentTasks map[string]*taskState
|
|
|
|
// Configuration
|
|
recentTaskWindowSeconds int
|
|
|
|
// Synchronization
|
|
mutex sync.RWMutex
|
|
}
|
|
|
|
// DestinationPlan represents a planned destination for a volume/shard operation
|
|
type DestinationPlan struct {
|
|
TargetNode string `json:"target_node"`
|
|
TargetAddress string `json:"target_address"`
|
|
TargetDisk uint32 `json:"target_disk"`
|
|
TargetRack string `json:"target_rack"`
|
|
TargetDC string `json:"target_dc"`
|
|
ExpectedSize uint64 `json:"expected_size"`
|
|
PlacementScore float64 `json:"placement_score"`
|
|
}
|
|
|
|
// MultiDestinationPlan represents multiple planned destinations for operations like EC
|
|
type MultiDestinationPlan struct {
|
|
Plans []*DestinationPlan `json:"plans"`
|
|
TotalShards int `json:"total_shards"`
|
|
SuccessfulRack int `json:"successful_racks"`
|
|
SuccessfulDCs int `json:"successful_dcs"`
|
|
}
|
|
|
|
// VolumeReplica represents a replica location with server and disk information.
|
|
// ShardIds is populated only by GetECShardLocations — it lists the EC shards
|
|
// the disk holds for the volume.
|
|
type VolumeReplica struct {
|
|
ServerID string `json:"server_id"`
|
|
DiskID uint32 `json:"disk_id"`
|
|
DataCenter string `json:"data_center"`
|
|
Rack string `json:"rack"`
|
|
ShardIds []uint32 `json:"shard_ids,omitempty"`
|
|
}
|