mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-22 06:22:38 +00:00
Lisandro PinGitHubClaude Opus 5.5Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Chris Lu* volume: route lookups to the leader master only (#11887) After a leader change, the volume server kept sending volume and EC shard lookups to masters that were not the leader: * currentMaster was set as soon as a heartbeat stream opened, before the peer said whether it was leader, and kept a follower through the redirect and its 3s sleep. A follower answers LookupVolume from its own vid cache, which right after an election can be empty, so reads went 404 rather than failing over. * EC lookups used Store.MasterAddress, a second, unlocked copy that the heartbeat loop set before each dial and cleared on errors, so EC and plain reads could disagree on the master or have none. * the receive goroutine called stream.Send itself when the volume options changed, racing the send loop, and exited silently on failure, after which leader redirects were no longer read. Publish currentMaster only once a master is known to be leader: the peer answered a new stream without naming a leader, which the leader always does, or a follower named the leader. Compare redirects against the connected peer. Replace Store.MasterAddress with MasterFn, backed by getCurrentMaster. Have the receiver ask the send loop for the extra heartbeat instead of sending it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: ask the masters for the leader after losing the heartbeat When the heartbeat stream to the leader failed, currentMaster kept the lost master until a new stream answered, so lookups on a cache miss kept dialing it through the retry delay even with a new leader already elected. On a regular heartbeat error, ask the configured masters for the raft leader and publish it, within the retry delay rather than on top of it. Without an answer the last known leader stays: clearing it would only trade a dead address for an empty one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: stop the heartbeat on a failed assertion The leader test closed stopChan only once every assertion had passed, so a failure left the stream open and the fake master's GracefulStop waited for the whole test run's timeout. Start the heartbeat through a helper whose cleanup stops it, safe to call twice, and waits for it to exit. It runs before the cleanups registered earlier, so the stream is closed before the store and the fake master go away. Both heartbeat tests use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: clear the leader when no master names one When the heartbeat lost its master and no configured master named a leader, refreshLeaderFromPeers kept the lost one. That master is dead or no longer leader, and a follower answers LookupVolume from its own cache instead of refusing, so lookups could return locations no leader vouched for. Clear currentMaster instead. Lookups then fail fast ("missing address") until a leader is known again, rather than reading a follower's cache or waiting on a dial to a dead master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: keep volumeSizeLimit 64-bit aligned on 32-bit builds MasterFn (a func, 4 bytes on 386) replaced MasterAddress (a string, 8 bytes) ahead of volumeSizeLimit and moved it to a 4-byte-aligned offset, where atomic.LoadUint64 panics with an unaligned 64-bit operation. The atomic.Uint64 type carries the alignment itself. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: publish the answering leader before a duplicate-uuid retry Read the leader field first on each heartbeat reply. A follower reply carries nothing else, so it no longer runs through the option updates; and a leader that rejects the registration over duplicate uuids still answers lookups, so it stays the lookup target through the retry wait instead of leaving the previous master in place. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: publish the master only once it is known leader (rust) * volume: ask the masters for the leader after losing the heartbeat (rust) * volume: read the leader field first on each heartbeat reply (rust) * volume: drop the seed fallback from ec shard lookups (rust) cached_lookup_ec_shard_locations fell back to the static first seed when no leader was published, so a mid-election EC lookup could land on a follower and read its stale shard map. Error out like Go's masterAddress() returning "" instead, via a current_master() getter matching getCurrentMaster. * volume: send volume lookups to the confirmed leader, not the seed (rust) Every HTTP lookup path — proxy/redirect targets, replicated writes and deletes, batch chunk deletes, remote chunk reads — dialed the static first seed, so after a failover they kept asking a follower whose vid map can be stale. Go routes all of these through GetMaster; switch the lookup calls and their leader-known gates to current_master(). * volume: mark volumes readonly on the leader, retrying the named one (rust) notify_master_volume_readonly fell back to the seed when no leader was published, which the peer answers with "Not current leader". It now uses only the confirmed leader, skips the call in masterless mode, and on NotLeader asks the seed masters who leads and retries there — the same path Go's notifyMasterVolumeReadonly takes. The peer leader query is shared with refresh_leader_from_peers as lookup_leader_from_peers. * volume: ask the confirmed leader for copy prealloc options (rust) VolumeCopy's preallocate query went to the static seed; Go issues it through GetMaster. Route it through current_master() too — when no leader is known the lookup fails fast and the copy proceeds without preallocating, as before. * volume: keep the announced master port forms, not just http (rust) publish_current_master stripped to http form, so a leader configured with an explicit gRPC port (host:9333.19999) was redialed on the default offset. Store the announced host:port[.grpcPort] form like Go's currentMaster; callers pick to_http_address / to_grpc_address / parse_grpc_address per use, and Ping admission compares on the http form. * volume: fail replicated writes and deletes when no leader is known (rust) Gating replication on a known leader meant a write could be acknowledged on fewer copies than required, and a delete could leave the data readable on the other replicas. Go fails the request when the replica-location lookup cannot run; drop the gate so the empty-leader lookup errors the same way. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>