* pb: stop exiting the process on malformed server addresses
ServerToGrpcAddress and GrpcAddressToServerAddress called glog.Fatalf
when hostAndPort could not parse the port, which os.Exit(255)ed the whole
process. A caller-supplied copy or tail source address reached this path
synchronously in the serving goroutine, so one anonymous VolumeCopy with
a non-numeric port terminated the volume server.
Log the parse error and return the input unchanged instead: the dial or
request that consumes the address then fails as an ordinary error.
* volume: validate copy and tail source addresses before dialing
VolumeCopy, VolumeEcShardsCopy and VolumeTailReceiver dial a
caller-supplied source address (SourceDataNode / SourceVolumeServer)
with no endpoint validation, so an anonymous caller could aim the volume
server at loopback, link-local (cloud metadata) or other unintended
destinations and read dial behavior back as a connectivity oracle.
Apply the same peer-target deny list FetchAndWriteNeedle uses for
replica targets: the source must be a bare host:port whose host is not
loopback, link-local or unspecified; cluster peers stay reachable on
private networks, and -volume.allowUntrustedRemoteEndpoints opts out.
The loopback-using copy tests set the flag to keep exercising the copy
path in process.
* rust volume: validate copy and tail source addresses before dialing
Mirror the Go guard on the Rust volume server: volume_copy,
volume_ec_shards_copy and volume_tail_receiver dial a caller-supplied
source address, so run it through validate_replica_target first (bare
host:port; no loopback, link-local or unspecified hosts; private peers
stay allowed). --volume.allowUntrustedRemoteEndpoints opts out; the test
fixture and the Rust test-cluster launcher set it so loopback sources in
tests keep working.
* volume: pin validated copy/tail source addresses at dial time
validateReplicaTarget resolves the source hostname once, but the gRPC
client resolved it again at connect, leaving a DNS-rebinding window for
hostname sources. The copy and tail source dials now run through the
same guardedDialerPolicy the remote-storage path uses, so every resolved
address is re-checked against the replica deny list (private peers
allowed) immediately before the TCP connect. guardedDialerPolicy also
moves to util.OutboundDialContext so the guarded path keeps the -ip.bind
source binding the default gRPC dialer had.
The Rust volume server mirrors this with connect_guarded, a tonic
connector that resolves, re-checks each address, and connects to the
first passing IP; handlers use it whenever the untrusted-endpoint
opt-out is off. A handler-level test now exercises the enabled
validation branches for all three source-taking RPCs.
* pb: return empty server address for malformed grpc addresses
GrpcAddressToServerAddress used to return the unparseable input on a
hostAndPort failure, so a malformed raft address (e.g. "host:abc")
flowed into admin dashboard master maps unchanged. Return an empty
string instead, skip empty conversions at the two raft-cluster merge
sites, and drop the now-stale comment about the fatal exit the earlier
commit removed.
* test: opt erasure-coding loopback clusters out of the remote endpoint guard
The erasure-coding suites drive VolumeEcShardsCopy / VolumeCopy between
volume servers bound to 127.0.0.1, which the copy/tail source guard now
rejects by default. Pass -volume.allowUntrustedRemoteEndpoints to the
test volume launches, matching what the volume_server framework
harnesses already do.
* admin: only claim fallback master leadership on an empty raft response
A nonempty RaftListClusterServers response whose entries were all
rejected left masterMap empty, so the fallback marked the reachable
current master as leader the same way a genuinely empty (non-raft)
response does. Track whether the successful response returned zero
servers and only promote the fallback master then.
* filer: add guarded remote-storage client builder hook for lazy fetch
The lazy-remote fetch path (maybeLazyFetchFromRemote) resolved its
remote-storage client through the unguarded shared cache, bypassing the
SSRF chokepoint (BuildGuardedRemoteStorageClient) that the CVE-2026-73080
remediation wired into the volume, filer stream and s3 stream dial paths.
Add a RemoteStorageClientBuilder hook on Filer plus conf-only lookups on
FilerRemoteStorage, and route the lazy fetch through the builder when set
(endpoint deny-list + DNS-rebinding-safe dialer), falling back to the
shared cache otherwise. The filer server wires the builder in a follow-up.
* filer: route lazy directory listing through the guarded remote client
maybeLazyListFromRemote shared the unguarded client resolution of the
fetch path, so a caller-supplied remote endpoint was dialed without the
SSRF deny-list or rebinding-safe dialer. Resolve the conf and build the
client through buildRemoteStorageClient so the same guard covers listing.
* filer: route lazy remote delete through the guarded remote client
maybeDeleteFromRemote issued outbound DELETE/RemoveDirectory requests
through the unguarded client, giving a write-side SSRF to a caller-chosen
endpoint. Resolve the conf and build the client through
buildRemoteStorageClient so the endpoint deny-list and rebinding-safe
dialer apply to the delete path as well.
* filer server: wire the guarded remote client builder into the filer
Set Filer.BuildGuardedRemoteClient to BuildGuardedRemoteStorageClient and
forward AllowUntrustedRemoteEndpoints so the lazy-remote fetch, list and
delete paths apply the same SSRF endpoint checks as the volume and
streaming read paths.
* filer: test lazy fetch honors the guarded remote client builder
Add a regression test that sets BuildGuardedRemoteClient to a rejecting
builder and asserts maybeLazyFetchFromRemote returns no entry without
reaching the remote, covering the SSRF guard wired in the prior commits.
* filer: skip remote client for local-only lazy deletes
maybeDeleteFromRemote resolved and validated the mount's remote client
before checking entry.Remote, so a local-only file (no Remote entry) under
a mount whose endpoint the guard rejects failed to delete: the guard
error aborted the metadata deletion, leaving a file that needs no remote
operation undeletable. Move the local-only check ahead of client
construction so only remote-backed files and directories pay the guard.
* filer: build the guarded remote client inside the lazy singleflight
The lazy fetch and list paths built the guarded client before their
singleflight blocks, so concurrent requests for the same key each
allocated a fresh SDK client and HTTP transport even though only one
remote operation ran. Move client construction inside the singleflight
so the deduplicated operation builds it once, matching the per-request
guard semantics of the sibling streaming paths without the duplicate
transport churn.
* filer: test guarded rejection for the lazy list and delete paths
Add regression tests that set BuildGuardedRemoteClient to a rejecting
builder and assert the lazy list does not reach the remote, a
remote-backed file delete is blocked, and a local-only file under a
rejected mount still deletes (covering the local-only fix).
* filer: decouple lazy guarded-client build from the first caller's context
Building the guarded client inside the singleflight made concurrent
fetches share the first caller's context. If that caller canceled while
endpoint DNS validation was running, the builder returned an error and
published a not-found result to other callers whose contexts were still
valid. Build with context.WithoutCancel so the guard's DNS validation
is not tied to any single caller's cancellation, matching the list
path's existing decoupling for the remote operation itself.
* filer: reject remote-storage confs that dial blocked endpoints at load
The filer's lazy-fetch / lazy-list / remote-delete paths resolve remote
storage clients by name from FilerRemoteStorage.storageNameToConf and
dial them via remote_storage.GetRemoteStorage, which bypasses the SSRF
deny-list the volume server (BuildGuardedRemoteStorageClient) and the
filer's own direct-read path apply. A RemoteConf planted under
/etc/remote with a loopback / private / IMDS S3 endpoint is reloaded into
storageNameToConf on the next metadata-change event and then dialed on
the next cache miss — server-side request forgery from the filer.
Apply the volume server's SSRF deny-list at conf load time, the single
chokepoint that populates storageNameToConf:
- Add RemoteStorageConfValidator, injected into FilerRemoteStorage by
the filer server (the filer package cannot import the server package).
A conf that fails validation is dropped from storageNameToConf, so the
name-based client resolution on the lazy paths returns "not found"
instead of dialing the blocked endpoint.
- Add ValidateRemoteConfForLoad in weed_server, which mirrors
BuildGuardedRemoteStorageClient's gcs credential + endpoint checks
(validateRemoteEndpoint via guardedRemoteClient) without building a
client. allowUntrusted skips the check, mirroring the volume server
opt-out (-filer.allowUntrustedRemoteEndpoints).
- The filer server injects the validator at construction.
A conf whose type dials a fixed provider host (no caller-supplied
endpoint) passes; only caller-influenced endpoints are denied.
* filer: skip DNS resolution in the load-time SSRF validator
ValidateRemoteConfForLoad resolved hostnames during /etc/remote reload,
so a transient DNS failure (2s timeout) dropped the conf from the fresh
map that replaces the live map, disabling a working mount until the next
metadata event. The build-time guard (BuildGuardedRemoteStorageClient)
already re-resolves and re-validates the endpoint at dial time with the
rebinding-safe dialer, so DNS at load is redundant for security.
Split the static checks (scheme, IMDS hostnames, IP-literal blocked
addresses, gcs credentials) into validateRemoteEndpointForLoad, which
does no DNS. Hostname endpoints pass at load and are caught at dial if
they resolve to a blocked address. This preserves fail-fast for
statically-blocked confs (loopback IPs, IMDS hostnames) without letting
transient DNS failures disable mounts.
* filer: accept empty S3 endpoints in the guarded remote client builder
guardedRemoteClient returned ok=true with an empty endpoint for a
standard AWS S3 config (no custom S3Endpoint), so
BuildGuardedRemoteStorageClient and ValidateRemoteConfForLoad rejected
it with "remote endpoint is empty" — breaking standard AWS S3 mounts on
the lazy paths and the sibling streaming read paths that already use the
guarded builder.
An empty endpoint is not caller-supplied: the AWS SDK derives the
regional endpoint from the region, so there is nothing for the SSRF
guard to validate. Return ok=false for empty S3-compatible endpoints so
the builder falls through to the shared unguarded cache, matching the
historical behavior for standard AWS S3.
* filer: accept gcs credentials file paths in the guarded remote client builder
checkGcsCredentials rejected all filesystem paths, so a gcs mount
configured with remote.configure -gcs.appCredentialsFile (which stores
a path in GcsGoogleApplicationCredentials) was rejected by
BuildGuardedRemoteStorageClient with "gcs credentials must be inline
JSON". This broke existing gcs mounts on the volume, filer, and s3
remote-mount read paths that use the guarded builder.
Read and validate the file content instead of rejecting the path,
mirroring what the gcs client itself does in MakeWithHTTPClient. A path
that does not exist or does not contain valid gcs credentials is still
rejected before any client is built. guardedRemoteClient now reads the
file to extract the token exchange URL for the SSRF deny-list, so the
rebinding-safe dialer still guards the token endpoint.
* filer: resolve gcs credential paths and avoid leaking file existence
loadGcsCredentialsContent passed the raw credentials string to os.ReadFile,
so a documented ~/path (as written by remote.configure
-gcs.appCredentialsFile=~/...) was rejected because os.ReadFile does not
expand ~. It also wrapped the os.ReadFile error, which includes the
file path, exposing file existence to a caller who planted a conf with
an arbitrary path.
Resolve the path with util.ResolvePath, matching the gcs client's own
behavior in MakeWithHTTPClient. Return a generic sentinel error on read
failure so the path is not reflected in the error message. The credential
type validation still runs on the file content, so a path that does not
contain valid gcs credentials is rejected before any client is built.
* volume: build the guarded remote storage client through a shared helper
Fold the endpoint validation, credential check and rebinding-safe dialer
that FetchAndWriteNeedle applies before dialing a caller-supplied remote
storage endpoint into a single BuildGuardedRemoteStorageClient helper, so
other callers that dial the same endpoints can reuse it. No behavior
change on this path.
Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN
* filer: build the remote-mount stream client through the guarded helper
streamFromRemote serves a cold remote-only entry straight from its mounted
origin. Build its client through BuildGuardedRemoteStorageClient so the
same endpoint checks the volume server applies cover this read path too.
Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN
* s3: build the remote-mount stream client through the guarded helper
openRemoteStream serves a remote-mounted object straight from its origin
when the local read cannot. Build its client through the same guarded
helper so the endpoint checks apply here as well.
Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN
* volume: accept only static-key gcs credentials on the fetch request
An inline credentials document of a federated type points the SDK at a url,
file or executable of the caller's choosing for the token exchange, so the
request-supplied value is no longer just a key.
* volume: guard the gcs token endpoint like the other remote endpoints
Inline credentials pick where the token request goes, so route the gcs client
through the same deny-list and rebinding-safe dialer used for S3 and azure.
* rust volume: pin that gcs has no credential-driven dial path
* volume: only check gcs credentials on a gcs remote conf
Only the gcs backend reads that field, so another backend carrying a stale
value should not fail the request.
* gcs: load credentials with the type the caller expects
The untyped loader is deprecated because it reads whatever the document
claims to be; callers handling credentials they do not control now name the
types they accept.
* volume: validate replica upload targets in FetchAndWriteNeedle
The replica leg forwarded the fetched needle to a caller-supplied address
without checking it, so a malformed target could redirect the upload to an
unintended host or path. Require each replica target to be a bare host:port
whose host is not loopback / link-local / unspecified, reusing the address
deny-list; cluster peers legitimately sit on private networks, so RFC 1918 /
CGNAT stay allowed and -volume.allowUntrustedRemoteEndpoints still opts out.
Validate every target up front so a bad one fails the request before the local
write, and upload through a client that re-checks the resolved address at
connect time so a replica hostname cannot rebind to a blocked address after
validation. Mirrored in Rust (validation moved ahead of the local write; the
Rust S3 path's connect-time re-check is still a follow-up there).
* volume: only accept inline gcs credentials in FetchAndWriteNeedle
The gcs credentials value on this request could name a local filesystem path,
which the SDK reads from disk. Accept only inline JSON here; the server-side
GOOGLE_APPLICATION_CREDENTIALS env var still supplies a path. The Rust volume
server has no gcs backend, so there is nothing to mirror.
* remote_storage/azure: allow a per-request HTTP client
Thread an optional *http.Client through NewAzBlobClient and add
azure.MakeWithHTTPClient, mirroring the S3 backend. When set, the client
overrides the azblob transport so a caller can pin the dial path. The
existing makers pass nil, so behavior is unchanged.
* volume: extend the remote-endpoint guard to the azure backend
The endpoint validation and rebinding-safe dialer in FetchAndWriteNeedle
covered the S3-SDK backends. The azure backend also dials a caller-supplied
AzureEndpoint, so route both families through a single guardedRemoteClient
helper that returns the endpoint each backend dials and a constructor bound
to the guarded HTTP client. azure is guarded only when AzureEndpoint is set;
an empty endpoint derives the public host from the account.
-volume.allowUntrustedRemoteEndpoints still opts out.
* rust volume: assert the azure endpoint has no remote-client path
The Rust volume server has no azure backend, so make_remote_storage_client
rejects the type before any client is built. Add a regression test pinning
that invariant.
* remote_storage: build S3-compatible clients through one constructor
The eight non-s3 S3-SDK providers each duplicated the AWS session setup
and only the s3 maker could take a custom *http.Client. Route every
S3-compatible type (s3, wasabi, b2, aliyun, tencent, baidu, filebase,
storj, contabo) through MakeWithHTTPClient with a single options table,
and add S3CompatibleEndpoint so callers can resolve the endpoint a given
type dials. No behavior change.
* volume: apply the remote-endpoint check to all S3-compatible providers
FetchAndWriteNeedle validated the endpoint and used the pinned dialer only
for type "s3". Every S3-SDK backend (wasabi, b2, aliyun, tencent, baidu,
filebase, storj, contabo) dials a caller-supplied endpoint through the same
client, so gate on S3CompatibleEndpoint to apply the same check uniformly.
-volume.allowUntrustedRemoteEndpoints still opts out.
* volume: don't route the guarded remote-endpoint client through a proxy
The guarded client exists to dial the validated endpoint directly and
re-check the resolved IP at connect time. With http.ProxyFromEnvironment
set, the dialer only validates the proxy's address while the proxy
re-resolves the endpoint host, which reopens the rebinding window. Drop
the proxy on this path; operators that need one can opt out with
-volume.allowUntrustedRemoteEndpoints.
* volume: decode IPv6 transition addresses in the remote-endpoint guard
checkBlockedIP normalized only ::ffff: mapped IPv4, so NAT64 (64:ff9b::/96),
6to4 (2002::/16), Teredo (2001:0000::/32), and IPv4-compatible (::/96) addresses
that embed an internal IPv4 (loopback, 169.254.169.254, RFC 1918) passed the
endpoint guard even though the plain IPv4 forms are refused. Extract the
embedded IPv4 from those forms and re-check it against the deny list, which
covers both the up-front validation and the dial-time guard. Mirrored in the
Rust volume server.
* volume: require the full NAT64 well-known prefix before decoding
Only 64:ff9b::/96 carries the embedded IPv4 in the low 32 bits, so also require
bytes 4-11 to be zero before treating an address as NAT64; other 64:ff9b:
prefixes place the IPv4 elsewhere and are left untouched. Add public-target
coverage for 6to4, Teredo, and IPv4-compatible so every decoder is exercised on
both a blocked and an allowed destination. Mirrored in the Rust volume server.
* preserve compression state during replication
* explain why ParseUpload skips compression for replica writes
* fix data race on err result in FetchAndWriteNeedle
The local-write and replica-write goroutines all wrote the named err return under an unsynchronized err==nil check. Give each goroutine its own error slot and combine after wg.Wait(): local error wins, then the first replica failure.
* skip redundant decompression of compressed needles during replication
doUploadData decompressed a compressed input only to report the clear-data length on UploadResult.Size, which both replication callers discard. Skip the decompress when IsReplication.
* fix(remote): reject short reads when caching remote objects
A short read from the remote (stale listing size, truncated or flaky
response) was silently zero-padded: the S3 and Azure clients pre-size
the buffer and discard the downloaded byte count, and the chunk is
recorded with the requested size. The cached file then matched the
expected size but its tail was NULL, and the entry was marked cached
so it never re-fetched.
Check the byte count against the requested size in both clients, and
add a backend-agnostic guard in FetchAndWriteNeedle. The cache now
fails loudly and the entry stays remote-only for a later retry.
* fix(remote): match S3 default modes when syncing remote metadata
Remote object listings carry no POSIX mode, so synced entries were
created with a hardcoded 0644. Against a SeaweedFS remote, whose S3
layer writes objects as 0660 and auto-creates directories as 0771
(0660|0111), the mounted copy ended up 0644/0755 and the permissions
visibly diverged from the source.
Default to the S3 modes instead (files 0660, directories 0771). The
filer derives parent-dir modes from the child as fileMode|0111, so
fixing the file default also brings the directories into line.
Directory mtimes still reflect sync time: S3 listings don't enumerate
directories, so the remote's directory timestamps aren't available.
volume: require admin auth and refuse loopback endpoints in FetchAndWriteNeedle
Gate the RPC behind checkGrpcAdminAuth for parity with the rest of the
destructive volume-server RPCs, and reject cluster-internal remote S3
endpoints (loopback / link-local / IMDS / RFC 1918 / CGNAT) before
dialing. Pin the validated address against DNS rebinding by routing the
AWS SDK through an HTTP transport whose DialContext re-resolves the host
and re-applies the deny list on every dial, so an endpoint that resolves
to a public IP at validate-time and then flips to 127.0.0.1 at connect
time is refused. Operators that legitimately fetch from private hosts
can opt out with -volume.allowUntrustedRemoteEndpoints.
* improve large file sync throughput for remote.cache and filer.sync
Three main throughput improvements:
1. Adaptive chunk sizing for remote.cache: targets ~32 chunks per file
instead of always starting at 5MB. A 500MB file now uses ~16MB chunks
(32 chunks) instead of 5MB chunks (100 chunks), reducing per-chunk
overhead (volume assign, gRPC call, needle write) by 3x.
2. Configurable concurrency at every layer:
- remote.cache chunk concurrency: -chunkConcurrency flag (default 8)
- remote.cache S3 download concurrency: -downloadConcurrency flag
(default raised from 1 to 5 per chunk)
- filer.sync chunk concurrency: -chunkConcurrency flag (default 32)
3. S3 multipart download concurrency raised from 1 to 5: the S3 manager
downloader was using Concurrency=1, serializing all part downloads
within each chunk. This alone can 5x per-chunk download speed.
The concurrency values flow through the gRPC request chain:
shell command → CacheRemoteObjectToLocalClusterRequest →
FetchAndWriteNeedleRequest → S3 downloader
Zero values in the request mean "use server defaults", maintaining
full backward compatibility with existing callers.
Ref #8481
* fix: use full maxMB for chunk size cap and remove loop guard
Address review feedback:
- Use full maxMB instead of maxMB/2 for maxChunkSize to avoid
unnecessarily limiting chunk size for very large files.
- Remove chunkSize < maxChunkSize guard from the safety loop so it
can always grow past maxChunkSize when needed to stay under 1000
chunks (e.g., extremely large files with small maxMB).
* address review feedback: help text, validation, naming, docs
- Fix help text for -chunkConcurrency and -downloadConcurrency flags
to say "0 = server default" instead of advertising specific numeric
defaults that could drift from the server implementation.
- Validate chunkConcurrency and downloadConcurrency are within int32
range before narrowing, returning a user-facing error if out of range.
- Rename ReadRemoteErr to readRemoteErr to follow Go naming conventions.
- Add doc comment to SetChunkConcurrency noting it must be called
during initialization before replication goroutines start.
- Replace doubling loop in chunk size safety check with direct
ceil(remoteSize/1000) computation to guarantee the 1000-chunk cap.
* address Copilot review: clamp concurrency, fix chunk count, clarify proto docs
- Use ceiling division for chunk count check to avoid overcounting
when file size is an exact multiple of chunk size.
- Clamp chunkConcurrency (max 1024) and downloadConcurrency (max 1024
at filer, max 64 at volume server) to prevent excessive goroutines.
- Always use ReadFileWithConcurrency when the client supports it,
falling back to the implementation's default when value is 0.
- Clarify proto comments that download_concurrency only applies when
the remote storage client supports it (currently S3).
- Include specific server defaults in help text (e.g., "0 = server
default 8") so users see the actual values in -h output.
* fix data race on executionErr and use %w for error wrapping
- Protect concurrent writes to executionErr in remote.cache worker
goroutines with a sync.Mutex to eliminate the data race.
- Use %w instead of %v in volume_grpc_remote.go error formatting
to preserve the error chain for errors.Is/errors.As callers.
* Boostrap persistent state for volume servers.
This PR implements logic load/save persistent state information for storages
associated with volume servers, and reporting state changes back to masters
via heartbeat messages.
More work ensues!
See https://github.com/seaweedfs/seaweedfs/issues/7977 for details.
* Block RPC operations writing to volume servers when maintenance mode is on.
* Added global http client
* Added Do func for global http client
* Changed the code to use the global http client
* Fix http client in volume uploader
* Fixed pkg name
* Fixed http util funcs
* Fixed http client for bench_filer_upload
* Fixed http client for stress_filer_upload
* Fixed http client for filer_server_handlers_proxy
* Fixed http client for command_fs_merge_volumes
* Fixed http client for command_fs_merge_volumes and command_volume_fsck
* Fixed http client for s3api_server
* Added init global client for main funcs
* Rename global_client to client
* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold
* Reduce the visibility of some functions in the util/http/client pkg
* Added the loadSecurityConfig function
* Use util.LoadSecurityConfiguration() in NewHttpClient func
Chris LuandGitHub
Lisandro PinandGitHub
Aleksey KosovandGitHub
vadimartynovandGitHub