mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-29 03:05:59 +00:00
Chris LuandGitHub
50b388771a
* grpc: a non-cancellable context is no evidence of a stale channel shouldInvalidateConnection only invalidates on Canceled/DeadlineExceeded while the context handed to WithGrpcClient is still live, so that an RPC timing out on its own does not close the shared cached ClientConn and cancel every other in-flight RPC on it. context.Background()/TODO never expire, so Err() stays nil forever and that guard always answered "invalidate" - and Background is what almost every caller passes, the S3 gateway included. One S3 request whose RPC rode an abandoned HTTP request context therefore closed the shared filer connection, and every multipart part in flight died with "the client connection is closing", surfacing to the client as 400 InvalidRequest. Only a cancellable context bounds an RPC attempt, so require one before reading it. A genuinely stale channel (a peer restart behind a stable L4 endpoint) surfaces as Unavailable, which invalidates on its own branch. * grpc: a bystander of a connection teardown is not a stale-channel witness gRPC raises ErrClientConnClosing locally, before an RPC reaches the wire, when this process has already closed the ClientConn. Every caller that touches a channel during another goroutine's teardown gets it, so reading it as a stale-channel signal lets one teardown re-arm itself across the whole herd of callers it just cancelled. The cached-connection version check keeps those callers from closing a replacement channel, but the streaming path invalidates by address alone and has no such guard. * grpc: end a stream without dropping the peer connection under it A streaming caller gets its own ClientConn, but on any error it also drops the cached non-streaming ClientConn every request handler shares with that peer, to recover a peer restart hidden behind a stable L4 endpoint. Any error includes the ordinary ones: a metadata subscription that reached its stop point, a follow callback that refused an event, a caller that gave up. The S3 gateway follows filer metadata on such a stream and reconnects forever, so each ordinary end of it cancelled every S3 request in flight against the filer. Drop the shared channel only for errors that say the peer went away, which is what invalidation is for. * test: close the connections the cascade tests leave cached Each test swaps in a fresh connection cache and restores the previous one, dropping its own entries without closing them, so the ClientConn's transport and reconnect goroutines outlive the fake filer they dialed. * grpc: say why ErrClientConnClosing's deprecation notice does not apply It points at codes.Canceled, which is the code this function exists to disambiguate. Only the message distinguishes a teardown a caller merely walked into, so the sentinel stays.