Files
seaweedfs/weed/pb/grpc_client_server_test.go
T
Chris LuandGitHub da2e90aefd fix(mount): sanitize non-UTF-8 filenames; keep marshal errors per-request (#9207)
* fix(mount): sanitize non-UTF-8 filenames; keep marshal errors per-request (#9139)

A single file with invalid-UTF-8 bytes in its name (e.g. a GNOME Trash
"partial" like \x10\x98=\\\x8a\x7f.trashinfo.9a51454f.partial) made every
FUSE-initiated filer RPC fail with:

  rpc error: code = Internal desc = grpc: error while marshaling:
  string field contains invalid UTF-8

and then produced an avalanche of "connection is closing" errors on
unrelated LookupEntry / ReadDirAll / UpdateEntry calls, causing the
volume-server QPS dips reported in #9139.

Root cause is twofold:

1. Proto3 `string` fields require valid UTF-8, but the FUSE kernel passes
   raw name bytes. Create/Mknod/Mkdir/Unlink/Rmdir/Rename/Lookup/Link/
   Symlink all forwarded those bytes directly into CreateEntryRequest.Name,
   DeleteEntryRequest.Name, StreamRenameEntryRequest.{Old,New}Name and
   Entry.Name. saveDataAsChunk also copied the FullPath into
   AssignVolumeRequest.Path unchecked.

2. When the marshal failed, shouldInvalidateConnection treated the
   resulting codes.Internal as a connection problem and dropped the
   shared cached ClientConn — canceling every other in-flight RPC on it.

Fix:

- Add sanitizeFuseName (strings.ToValidUTF8 with '?' replacement, matching
  util.FullPath.DirAndName) and make checkName return the sanitized name.
  Apply at every FUSE entry point that passes a name to the filer RPC,
  including Unlink/Rmdir (which did not previously call checkName) and
  both oldName/newName in Rename. Add a backstop scrub for
  AssignVolumeRequest.Path so async flush paths cannot reintroduce
  invalid bytes from a pre-sanitization cached FullPath.

- In weed/pb.shouldInvalidateConnection, detect client-side marshal
  errors via the gRPC library's "error while marshaling" prefix and
  return false: the connection is healthy, only the request is bad.

Refs: https://github.com/seaweedfs/seaweedfs/issues/9139#issuecomment-4301184231

* fix(mount,util): use '_' for invalid-UTF-8 replacement (URL-safe)

Sanitized filenames flow downstream into HTTP URLs (volume-server uploads,
filer HTTP API, S3/WebDAV gateways). '?' is the URL query-string
delimiter and would split the path the first time the name lands in one,
so swap every invalid-UTF-8 replacement to '_'. This covers the two
pre-existing sites in weed/util/fullpath.go as well, keeping all paths
sanitized the same way.

* refactor(pb): detect client-side marshal errors via errors.As, not substring

Replace the raw `strings.Contains(err.Error(), ...)` check with a
type-based carve-out: use errors.As against the `GRPCStatus() *Status`
interface to pull the original Status out of any fmt.Errorf("...: %w")
wrapping, then match the library-owned "grpc:" prefix on that Status's
Message.

Why not errors.Is against a proto-level sentinel: gRPC's encode()
collapses the inner proto error with "%v" (stringification) before
wrapping it in a Status, so the original error type does not survive
into the caller. The Status itself is the structural signal that does
survive.

Why not status.FromError: when the caller wraps the Status error with
fmt.Errorf("...: %w", ...), status.FromError rewrites Status.Message
with the full err.Error() of the outermost wrapper, which defeats a
prefix check on the library-owned message. errors.As gives us the
original Status whose Message is still verbatim from the gRPC library.

A new test asserts that a plain errors.New("grpc: error while marshaling: …")
— i.e. the same text attached to something that is NOT a gRPC status —
does not short-circuit invalidation, so we never silently keep a cached
connection alive based on a coincidental substring match.

* refactor(util): centralize UTF-8 sanitization; add FullPath.Sanitized

Addresses review feedback on PR #9207.

Nitpick: every invalid-UTF-8 replacement across the codebase (DirAndName,
Name, mount.sanitizeFuseName, the weedfs_write.go backstop) now goes
through a single util.SanitizeUTF8Name helper, so the replacement char
('_' — URL-safe) is chosen in one place.

Outside-diff: three proto fields took raw FullPath strings that could
break marshaling if an entry ever carried invalid UTF-8
(CreateEntryRequest.Directory in Mkdir, DeleteEntryRequest.Directory in
Unlink, AssignVolumeRequest.Path in command_fs_merge_volumes). The
reviewer's suggested fix — using DirAndName() — would have silently
changed Directory from parent to grandparent, because DirAndName
sanitizes only the trailing component. Added FullPath.Sanitized(), which
scrubs every component, and applied it at the three sites. Exposure is
narrow in practice (FUSE-boundary sanitization and the gRPC-side
isClientSideMarshalError carve-out already cover the #9139 cascade),
but the defense-in-depth is cheap and consistent with the existing
AssignVolume backstop.

New tests in weed/util/fullpath_test.go document:
- SanitizeUTF8Name: valid UTF-8 passes through unchanged; invalid bytes
  become '_' (not '?', which is URL-special).
- FullPath.Sanitized: scrubs bytes in any component, not just the last.
- FullPath.DirAndName: dir remains raw on purpose — callers needing a
  clean full path must use Sanitized(). The test pins this behavior so
  it is not accidentally "fixed" in a way that changes the (dir, name)
  semantics callers depend on.
2026-04-23 19:17:35 -07:00

69 lines
2.9 KiB
Go

package pb
import (
"fmt"
"testing"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
// TestShouldInvalidateConnection_MarshalErrorIsPerRequest ensures that a
// client-side proto marshal failure does NOT cause the shared cached
// ClientConn to be torn down. Tearing it down would cancel every other
// in-flight RPC (seaweedfs#9139: one file with invalid-UTF-8 bytes triggered
// an avalanche of "connection is closing" errors on unrelated operations).
func TestShouldInvalidateConnection_MarshalErrorIsPerRequest(t *testing.T) {
// Reproduces the exact error gRPC returns when a string field in the
// outgoing request contains invalid UTF-8 bytes.
marshalErr := status.Error(codes.Internal,
"grpc: error while marshaling: string field contains invalid UTF-8")
if shouldInvalidateConnection(marshalErr) {
t.Fatalf("client-side marshal error must not invalidate the shared connection")
}
// Same error wrapped with fmt.Errorf (common when callers add context).
wrapped := fmt.Errorf("upload data: %w", marshalErr)
if shouldInvalidateConnection(wrapped) {
t.Fatalf("wrapped marshal error must not invalidate the shared connection")
}
}
// TestShouldInvalidateConnection_GenuineInternalStillInvalidates ensures the
// marshal-error carve-out does not swallow real server-side Internal errors,
// which previously caused — and should continue to cause — connection
// invalidation.
func TestShouldInvalidateConnection_GenuineInternalStillInvalidates(t *testing.T) {
serverInternal := status.Error(codes.Internal, "stream terminated by RST_STREAM with code 2")
if !shouldInvalidateConnection(serverInternal) {
t.Fatalf("genuine server-side Internal must still invalidate the connection")
}
}
// TestShouldInvalidateConnection_TransportErrorsStillInvalidate is a
// regression guard for the string-matching fallback path (e.g. a raw
// "connection refused" from net.Dial that never acquired a gRPC status).
func TestShouldInvalidateConnection_TransportErrorsStillInvalidate(t *testing.T) {
for _, msg := range []string{
"rpc error: code = Unavailable desc = transport is closing",
"dial tcp: connection refused",
"read: connection reset by peer",
} {
if !shouldInvalidateConnection(fmt.Errorf("%s", msg)) {
t.Fatalf("transport error %q must still invalidate", msg)
}
}
}
// TestIsClientSideMarshalError_RequiresGrpcStatus ensures the carve-out is
// type-based (via errors.As on the grpc status interface), not a naive
// string match against arbitrary errors that happen to mention marshaling.
// A plain errors.New(...) with the same prefix must NOT be treated as a
// per-request marshal error — we have no evidence the connection is healthy.
func TestIsClientSideMarshalError_RequiresGrpcStatus(t *testing.T) {
impostor := fmt.Errorf("grpc: error while marshaling: synthetic non-status error")
if isClientSideMarshalError(impostor) {
t.Fatalf("plain error must not match the marshal-error carve-out")
}
}