mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-25 16:47:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

Pin grpc-go to v1.82.0 to avoid a Unix-socket transport deadlock

Browse Source 
grpc-go v1.82.1 broadened the control-buffer throttle to count registerStream
and cleanupStream. Those are per-RPC bookkeeping items that produce no wire
traffic, so any connection carrying enough concurrent RPCs latches throttling
in both directions at once: both peers stop reading, both loopyWriters block
writing to a full socket, and the connection never recovers.

Local gRPC goes over Unix sockets, whose send buffers are small and do not
autotune, so loopyWriter blocks readily and the deadlock is easy to reach. TCP
loopback does not trigger it. weed mini wedges at roughly 320 concurrent S3
PUTs: throughput collapses to tens of ops/s with 65s stalls and no filer
handlers running. Pinned back until upstream is fixed.
This commit is contained in:
Chris Lu
2026-08-18 18:33:53 -07:00
parent f41595fb10
commit 911d4e5f31
2 changed files with 18 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
go.mod
+16
View File
@@ -527,3 +527,19 @@ replace github.com/apache/thrift => github.com/apache/thrift v0.23.1-0.202604291
// through rclone's internxt backend, which calls IsMnemonicValid and NewSeed.
// cosmos/go-bip39 is a maintained, API-compatible fork.
replace github.com/tyler-smith/go-bip39 => github.com/cosmos/go-bip39 v1.0.0
// grpc-go v1.82.1 started counting registerStream/cleanupStream toward the
// control-buffer throttle limit. Those are per-RPC bookkeeping items that emit
// no wire traffic, so a connection carrying enough concurrent RPCs latches
// throttling in both directions at once and deadlocks for good: both peers stop
// reading, both loopyWriters block writing to a full socket.
//
// It triggers whenever the socket send buffer is too small to absorb the burst
// loopyWriter emits for the in-flight streams, so it is about buffer size, not
// transport: TCP forced to a 64KB buffer deadlocks, a Unix socket with a 2MB
// buffer does not. Local gRPC rides Unix sockets, whose buffers are small and
// fixed (208KB on Linux, 8KB on macOS), which is why weed mini wedges under
// concurrent S3 writes; a remote filer over TCP is exposed too wherever
// send buffers are constrained. Pin to the last unaffected release until
// grpc/grpc-go#9330 is fixed.
replace google.golang.org/grpc => google.golang.org/grpc v1.82.0
go.sum
+2
View File
@@ -2838,6 +2838,8 @@ google.golang.org/grpc v1.51.0/go.mod h1:wgNDFcnuBGmxLKI/qn4T+m5BtEBYXJPvibbUPsA
google.golang.org/grpc v1.52.0/go.mod h1:pu6fVzoFb+NBYNAvQL08ic+lvB2IojljRYuun5vorUY=
google.golang.org/grpc v1.53.0/go.mod h1:OnIrk0ipVdj4N5d9IUoFUx72/VlD7+jUsHwZgwSMQpw=
google.golang.org/grpc v1.55.0/go.mod h1:iYEXKGkEBhg1PjZQvoYEVPTDkHo1/bjTnfwTeGONTY8=
google.golang.org/grpc v1.82.0 h1:vguDnZUPjE26w09A63VoxZPnvPjB5Riyc0mkXPFmAIU=
google.golang.org/grpc v1.82.0/go.mod h1:yzTZ1TB1Z3SG+LIYaI+WiE8D5+PZ3ArnrSp8zF3+/ZA=
google.golang.org/grpc v1.84.0-dev.0.20260723093437-b6eac429d7b6 h1:HfjjkdGIa8u9sP9EW5WCygy0kQDuTI/Tax4j//t24Fo=
google.golang.org/grpc v1.84.0-dev.0.20260723093437-b6eac429d7b6/go.mod h1:ljCht0DrxQrXBDRTZp52Qxh3Ffk8CdYm2sj4O2QN2C0=
google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0/go.mod h1:6Kw0yEErY5E/yWrBtf03jp27GLLJujG4z/JK95pnjjw=