A Canceled/DeadlineExceeded from the caller's per-request context was
treated like a dead channel: it closed the shared cached ClientConn and
cancelled every other in-flight RPC on it with "the client connection is
closing". Under a burst of concurrent chunk assigns (e.g. a large S3
multipart upload) one slow assign hitting its 10s attempt timeout could
poison the connection for all the rest, cascading into a flood of 500s.
Thread the caller's context into shouldInvalidateConnection and only
invalidate on Canceled/DeadlineExceeded while that context is still live,
which isolates the genuine stale-channel signal (a peer restart behind a
k8s Service VIP). To carry the context, add a ctx parameter to the
existing WithGrpcClient, WithMasterClient, and WithMasterServerClient; the
master assign and volume-lookup paths pass their per-attempt context and
every other caller passes context.Background().
cluster.check asks every master to ping every volume server, but the
Ping gate validated volume-server targets only against the local
topology. Only the leader receives volume-server heartbeats, so a
follower's topology is empty and every probe through it failed with
"unknown ping target ... of type volumeServer".
Fall back to the volume-server set the master learns over its own
MasterClient subscription to the leader, the same source the filer gate
already trusts. The anti-SSRF intent is preserved: Ping still only dials
recognized cluster members.
Ping previously dialled whatever host:port the caller asked for. Gate
each server's Ping handler on cluster membership: masters check the
topology, registered cluster nodes, and configured master peers; volume
servers only accept their seed/current masters; filers accept tracked
peer filers, the master-learned volume server set, and configured
masters.
Use address-indexed peer lookups to keep Ping target validation O(1):
- topology maintains a pb.ServerAddress -> *DataNode index alongside
the dc/rack/node tree, kept in sync from doLinkChildNode and
UnlinkChildNode plus the ip/port-rewrite branch in
GetOrCreateDataNode. GetTopology now returns nil on a detached
subtree instead of panicking, so the linkage hooks can no-op safely.
- vid_map tracks a refcount per volume-server address so
hasVolumeServer answers without scanning every vid location. The
add path skips empty-address entries the same way the delete path
already does, so a zero-value Location cannot leak a permanent
serverRefCount[""] bucket.
- masters reuse a cached master-address set from MasterClient instead
of walking the configured peer slice on every request.
- volume servers compare against a pre-built seed-master set and
protect currentMaster reads/writes with an RWMutex, fixing the
data race with the heartbeat goroutine. The seed slice is copied
on construction so external mutation cannot desync it from the
frozen lookup set.
- cluster.check drops the direct volume-to-volume sweep; volume
servers no longer carry a peer-volume list, and the note next to
the dropped probe is reworded to make clear that direct
volume-to-volume reachability is intentionally not validated by
this command.
Update the volume-server integration tests that drove Ping through the
new admission gate: success-path coverage now targets the master peer
(the only type a volume server tracks), and the unknown/unreachable
path asserts the InvalidArgument the gate now returns instead of the
old downstream dial error.
Mirror the same admission gate in the Rust volume server crate: a
seed-master HashSet built once at startup plus a tokio RwLock over the
heartbeat-tracked current master, both consulted in is_known_ping_target
on every Ping, with InvalidArgument returned for any target that isn't
a recognised master.
* Fix stale admin lock metric when lock expires and is reacquired (#8857)
When a lock expired without an explicit unlock and a different client
acquired it, the old client's metric was never cleared, causing
multiple clients to appear as simultaneously holding the lock.
* Use DeleteLabelValues instead of Set(0) to remove stale metric series
Avoids cardinality explosion from accumulated stale series when
client names are dynamic.
* remove old raft servers if they don't answer to pings for too long
add ping durations as options
rename ping fields
fix some todos
get masters through masterclient
raft remove server from leader
use raft servers to ping them
CheckMastersAlive for hashicorp raft only
* prepare blocking ping
* pass waitForReady as param
* pass waitForReady through all functions
* waitForReady works
* refactor
* remove unneeded params
* rollback unneeded changes
* fix
Chris LuandGitHub
Konstantin LebedevandGitHub
askeipxandGitHub
chrislu