mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-19 05:36:58 +00:00
0de7ff5eb8fa00a2cd0e5b748259b41b83348390
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Chris LuandGitHub
|
0799084e98 |
refactor: share volume and EC shard move logic between shell and workers (#10727)
* operation: add shared volume_move package for volume and EC shard moves The shell commands (volume.move, volume.balance, ec.balance, tier moves) and the maintenance workers (balance, ec_balance) each carried their own copy of the move RPC sequences, and the copies had drifted: the worker verified the target before deleting the source but dropped the disk type and IO throttle; the shell passed those but deleted the source unverified. volume_move.Mover carries the merged sequences, keeping the stricter behavior from each side: - LiveMoveVolume: check-then-hard-freeze the source (VolumeStatus's IsReadOnly also covers low-disk and readonly-but-can-delete states, which still accept needle deletes), copy with disk type and IO throttle, tail, verify the target is not behind the source before the destructive source delete (a target that is ahead holds writes it accepted during the tail and the move commits to keep them), and restore the source's writability when a failure precedes the delete and this move did the freezing. Aborts clean up the incomplete target copy; a failed cleanup or an ambiguous source delete keeps the source readonly (ErrSourceKeptReadonly) so callers do not thaw a source next to a possibly-authoritative copy. With a readonly source, an existing or unknown-state target refuses the move outright: no client-side observation can prove such a copy is a stale remnant rather than the authoritative copy of an unfinished move. - MoveEcShards: copy with the .ecx/.ecj/.vif/.ecsum sidecars, mount, verify the target registered every shard before unmount+delete on the source, and reject same-server moves (the EC delete is server-wide). Server identity is the grpc endpoint (SameServer), so node:8080 and node:8080.18080 compare equal while test servers sharing a degenerate HTTP address stay distinct; addresses are validated non-fatally before dialing and before being embedded in copy/tail requests, since both the client dialer and the receiving server normalize them through a parser that aborts the process on a malformed port. The Rust volume server's codes.NotFound counts as a definitively absent probe answer alongside the Go server's plain-error code Unknown. All RPCs go through an injectable ClientFunc, so the sequences are unit tested against a fake volume server client: RPC order, request fields, and that verification failures keep the source intact. * shell, worker: delegate volume and EC shard moves to operation/volume_move LiveMoveVolume and the copy/tail/delete/mark-writable helpers become thin wrappers over the shared mover, keeping their signatures; the EC helpers keep their per-step output and delegate the RPCs. BalanceTask and ECBalanceTask keep their parameter validation, progress reporting, and guards (same-node cross-disk rejection, dedup keep-node verification, shard ids range-checked before the uint8 narrowing) and hand the RPC sequences to the mover. volume.tier.move skips its thaw-on-failure when the mover deliberately kept the source readonly, since reopening the replicas beside a possibly-authoritative target copy would fork the volume. The tail-failure tolerance moves inside the mover: a failed tail is tolerated only when the volume was already readonly before the move began, backstopped by a stability re-read across the idle window, so volume.balance's -skipTailError-by-readonly heuristic and tier-move's unconditional skip both become the same authoritative rule. * volume_move: keep the source readonly when a failed copy leaves a target of unknown origin A failed copy can leave a complete, mounted copy on the target (the server finishes after the client loses the stream). The abort probed the target only when its pre-copy state was known-absent; an unknown prior state skipped both the probe and the cleanup and then reopened the source - two writable replicas of one volume, diverging from the next write on. The abort now probes the target on every failed copy and restores the source only when the target provably holds nothing. A copy whose provenance cannot be proven (unknown prior state, a pre-existing replica, or an unreachable target) is never deleted, and the source stays readonly with ErrSourceKeptReadonly naming the recovery. * test: teach the plugin worker harness the shared move sequence The fake volume server lacked VolumeStatus, which the shared mover now issues before freezing the source, and the batch execution test's status-read accounting predates the pre-copy target probe and the verification reads. Mirrors the harness the enterprise tree already carries. |
||
|
Chris LuandGitHub
|
d4d8e097dd |
shell: volume.move cleans up when aborted after the copy phase (#10704)
* shell: volume.move restores source writability when aborted after the copy phase * shell: volume.move removes the incomplete target copy when aborted before the source delete * shell: give each abort cleanup RPC its own timeout |
||
|
Chris LuandGitHub
|
312cfe5ae1 |
Fix volume.merge corrupting every needle it copies (#10565)
* Give volume.merge the needle size the target actually indexes by needleBlobFromNeedle returned the size Append reports, which is Size(n.DataSize) - payload bytes only. The .dat header, the needle map and WriteNeedleBlobRequest.Size all use n.Size, which additionally covers the flags, name, mime and lastModified fields. Every needle volume.merge copied therefore landed with a too-small size. The target indexed it at that length, so every later read failed the header check in ReadBytes with a size mismatch, and on v3 the fresh AppendAtNs stamp landed NeedleHeaderSize+DataSize+NeedleChecksumSize into the blob - exactly on the flags byte - overwriting flags, name size, mime size and the first mime bytes with the top of a timestamp. Needles came back with flags 0x18, no name, no mime and a phantom TTL parsed from two arbitrary timestamp bytes; the ones that decoded as expired 404 and vacuum would drop them. Since merge rebuilds every replica from the merged copy, no clean replica survives. Return n.Size, which Append fills in as it serializes, matching what the normal write path stores via nm.Put. * Reject needle blobs whose size disagrees with their own header WriteNeedleBlob trusts the caller's size for two destructive things: it is what goes into the needle map, and it is where the v3 AppendAtNs stamp is written inside the caller's buffer. A caller passing the payload-only DataSize convention corrupts both, and nothing surfaces until the needle is read back - by which point every replica may already have been rebuilt from it. Parse the blob's own header and refuse the write when the two disagree. Mirrored in the Rust volume server. |
||
|
Chris LuandGitHub
|
6e6255b58e |
shell: accept a context in the volume move helpers (#10415)
LiveMoveVolume and the copy, tail, delete, mark, replicate, and configure helpers around it issued every RPC on context.Background(), so a caller had no way to bound or abort a move once it started. They now take a context, which the exported LiveMoveVolume in particular needs: callers outside the shell drive long moves and want to stop them. The deferred restore in copyVolume runs on a detached, bounded context rather than the caller's. Marking the source writable again is cleanup, and cancelling the copy must not skip it and leave the volume readonly — the same guard balance_task.go already applies for the same reason. Shell commands pass context.Background(): their Do signature carries no context, and changing it would touch every command in the package. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris LuandGitHub
|
5955972fe6 |
fix(shell): verify volume.merge output before overwriting replicas (#9731)
* fix(shell): verify volume.merge output before overwriting replicas volume.merge overwrote every replica with the merged copy without checking it was complete. Read back the merged copy and refuse to overwrite unless it holds at least as many live needles as the most complete source replica, leaving the originals intact on a short or empty merge. * fix(shell): keep merged volume until all replicas are rebuilt On a copy failure partway through the overwrite loop, the temporary merged copy was deleted along with the half-rebuilt replicas. Stop deleting it until every replica has been rebuilt; on failure the verified copy is kept so the merge can be re-run to completion. * refactor(shell): reuse readVolumeStatus in ensureVolumeReadonly * fix(shell): guard against nil volume status response |
||
|
1c0e24f06a |
fix(balance): don't move remote-tiered volumes; don't fatal on missing .idx (#9335)
* fix(volume): don't fatal on missing .idx for remote-tiered volume A .vif left behind without its .idx (orphaned by a crashed move, partial copy, or hand-edit) would trip glog.Fatalf in checkIdxFile and take the whole volume server down on boot, killing every healthy volume on it too. For remote-tiered volumes treat it as a per-volume load error so the server can come up and the operator can clean up the stray .vif. Refs #9331. * fix(balance): skip remote-tiered volumes in admin balance detection The admin/worker balance detector had no equivalent of the shell-side guard ("does not move volume in remote storage" in command_volume_balance.go), so it scheduled moves on remote-tiered volumes. The "move" copies .idx/.vif to the destination and then calls Volume.Destroy on the source, which calls backendStorage.DeleteFile — deleting the remote object the destination's new .vif now points at. Populate HasRemoteCopy on the metrics emitted by both the admin maintenance scanner and the worker's master poll, then drop those volumes at the top of Detection. Fixes #9331. * Apply suggestion from @gemini-code-assist[bot] Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * fix(volume): keep remote data on volume-move-driven delete The on-source delete after a volume move (admin/worker balance and shell volume.move) ran Volume.Destroy with no way to opt out of the remote-object cleanup. Volume.Destroy unconditionally calls backendStorage.DeleteFile for remote-tiered volumes, so a successful move would copy .idx/.vif to the destination and then nuke the cloud object the destination's new .vif was already pointing at. Add VolumeDeleteRequest.keep_remote_data and plumb it through Store.DeleteVolume / DiskLocation.DeleteVolume / Volume.Destroy. The balance task and shell volume.move set it to true; the post-tier-upload cleanup of other replicas and the over-replication trim in volume.fix.replication also set it to true since the remote object is still referenced. Other real-delete callers keep the default. The delete-before-receive path in VolumeCopy also sets it: the inbound copy carries a .vif that may reference the same cloud object as the existing volume. Refs #9331. * test(storage): in-process remote-tier integration tests Cover the four operations the user is most likely to run against a cloud-tiered volume — balance/move, vacuum, EC encode, EC decode — by registering a local-disk-backed BackendStorage as the "remote" tier and exercising the real Volume / DiskLocation / EC encoder code paths. Locks in: - Destroy(keepRemoteData=true) preserves the remote object (move case) - Destroy(keepRemoteData=false) deletes it (real-delete case) - Vacuum/compact on a remote-tier volume never deletes the remote object - EC encode requires the local .dat (callers must download first) - EC encode + rebuild round-trips after a tier-down Tests run in-process and finish in under a second total — no cluster, binary, or external storage required. * fix(rust-volume): keep remote data on volume-move-driven delete Mirror the Go fix in seaweed-volume: plumb keep_remote_data through grpc volume_delete → Store.delete_volume → DiskLocation.delete_volume → Volume.destroy, and skip the s3-tier delete_file call when the flag is set. The pre-receive cleanup in volume_copy passes true for the same reason as the Go side: the inbound copy carries a .vif that may reference the same cloud object as the existing volume. The Rust loader already warns rather than fataling on a stray .vif without an .idx (volume.rs load_index_inmemory / load_index_redb), so no counterpart to the Go fatal-on-missing-idx fix is needed. Refs #9331. * fix(volume): preserve remote tier on IO-error eviction; fix EC test target Two review nits: - Store.MaybeAddVolumes' periodic cleanup pass deleted IO-errored volumes with keepRemoteData=false, so a transient local fault on a remote-tiered volume would also nuke the cloud object. Track the delete reason via a parallel slice and pass keepRemoteData=v.HasRemoteFile() for IO-error evictions; TTL-expired evictions still pass false. - TestRemoteTier_ECEncodeDecode_AfterDownload deleted shards 0..3 but called them "parity" — by the klauspost/reedsolomon convention shards 0..DataShardsCount-1 are data and DataShardsCount..TotalShardsCount-1 are parity. Switch the loop to delete the parity range so the intent matches the indices. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
979c54f693 |
fix(wdclient,volume): compare master leader with ServerAddress.Equals (#9089)
* fix(wdclient,volume): compare master leader with ServerAddress.Equals Raft leader is advertised as host:httpPort.grpcPort, but clients dial host:httpPort. Raw string comparison against VolumeLocation.Leader / HeartbeatResponse.Leader therefore never matches, causing the masterclient and the volume server heartbeat loop to continuously "redirect" to the already-connected master, tearing down the stream and reconnecting. Use ServerAddress.Equals, which normalizes the grpc-port suffix. * fix(filer,mq): compare ServerAddress via Equals in two more sites filer bootstrap skip (MaybeBootstrapFromOnePeer) and the broker's local partition assignment check both compared a wire-supplied address string against the local self ServerAddress with raw string equality. Both are vulnerable to the same plain-vs-host:port.grpcPort mismatch as the masterclient/volume heartbeat sites: filer would bootstrap from itself, and the broker would fail to claim a partition it was actually assigned. Route both through ServerAddress.Equals. * fix(master,shell): more ServerAddress comparisons via Equals - raft_server_handlers.go HealthzHandler: s.serverAddr == leader would skip the child-lock check on the real leader when the two carry different plain/grpc-suffix forms, returning 200 OK instead of 423. - master_server.go SetRaftServer leader-change callback: the Leader() == Name() guard for ensureTopologyId could disagree with topology.IsLeader() (which already uses Equals), so leader-only initialization could be skipped after an election. - command_volume_merge.go isReplicaServer: the -target guard compared user-supplied host:port against NewServerAddressFromDataNode(...) with ==, letting an existing replica slip through when topology carries the embedded gRPC port. All routed through pb.ServerAddress.Equals. * fix(mq,cluster): more ServerAddress comparisons via Equals - broker_grpc_lookup.go GetTopicPublishers/GetTopicSubscribers: the partition ownership check gated listing on raw LeaderBroker == BrokerAddress().String(), so listings silently omitted partitions hosted locally when the assignment carried the other host:port / host:port.grpcPort form. - lock_client.go: LockHostMovedTo comparison and the seedFiler fallback guard both used raw string equality against configured filer addresses (which may be plain host:port while LockHostMovedTo comes back suffixed), causing spurious host-change churn and blocking the seed-filer fallback. * fix(mq): more ServerAddress comparisons via Equals - pub_balancer/allocate.go EnsureAssignmentsToActiveBrokers: direct activeBrokers.Get() lookup missed brokers when a persisted assignment carried a different address encoding than the registered broker key, triggering a bogus reassignment on every read/write cycle. Added a findActiveBroker helper that falls back to an Equals-based scan and canonicalizes the assignment in place so later writes are stable. - broker_grpc_lookup.go isLockOwner: used raw string equality between LockOwner() and BrokerAddress().String(), so a lock owner could fail to recognize itself and proxy local lookup/config/admin RPCs away. - pub_client/scheduler.go onEachAssignments: reused publisher jobs only on exact LeaderBroker match, so an encoding flip in lookup results tore down and recreated a stream to the same broker. |
||
|
Chris LuandGitHub
|
b565a0cc86 |
Adds volume.merge command with deduplication and disk-based backend (#8441)
* Enhance volume.merge command with deduplication and disk-based backend
* Fix copyVolume function call with correct argument order and missing bool parameter
* Revert "Fix copyVolume function call with correct argument order and missing bool parameter"
This reverts commit
|