* Name the failure when the source cluster cannot locate a chunk's volume
LookupFileId formatted a nil err into the message it returned, so the only
thing a caller could do with "no locations for this volume" was match on the
text. Return a typed error instead.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Fail a source chunk read on a failure status instead of copying the error page
ReadPart never looked at the response status, so a volume server answering 404
for a needle vacuum had removed came back as a successful read whose body was
the error page. The caller counted those bytes as file content and reported a
size mismatch — a corruption claim about data the source had simply lost — and
a 404 from one replica ended the search instead of trying the next.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Stop retrying a chunk the source cluster can no longer produce
A chunk whose volume vacuum has removed fails the same way on every attempt, but
the retry loop had no way to say so and kept going forever. The sync job holding
it never finished, so it pinned the offset watermark at the event ahead of it and
filer.sync never checkpointed again — alive, quiet, and permanently behind.
Wait the source out for a grace period long enough to cover a volume server
restart or a master failover, then give up and mark the failure permanent.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Let replication continue past an entry whose source data is gone
An entry the source can no longer read holds the sync offset forever: the event
fails on every replay, so the checkpoint never moves past it and every later
event stays uncheckpointed, however long the sync keeps running. Nothing brings
those bytes back, so skip the entry with an error naming it and carry on.
Skip only while the source is demonstrably still serving other chunks. A volume
with no locations reads the same whether it was vacuumed away or every replica is
down, and during a cluster-wide outage that answer comes back for every chunk —
skipping then would drop live files wholesale.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Propagate a missing source chunk instead of waiting when supersession is unverifiable
An incremental sink's dated target keys cannot be mapped back to a source path,
so nothing here can tell a chunk the source lost from one a later version already
replaced. Waiting out the grace period would stall every vacuumed needle for half
an hour; hand the failure to the caller, which has the event's real source key.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Wait out a gone volume once, not once per file it held
A volume vacuum removed took every file it held with it, and each chunk was
timing its own grace period. With a bounded chunk executor those waits serialize,
so one gone volume holding many files stalls the sync for far longer than the
grace period — the wedge again, only slower.
Track the wait per source volume on the sink instead: the first chunk to find it
unlocatable starts the clock, every later chunk inherits it and gives up as soon
as it has run out, and a chunk the source does serve clears it.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
* Probe the source with a read, not a lookup, before writing an entry off
A lookup only proves the source master still has the topology. If every volume
server is unreachable while the master still lists them, the probe passed and the
sink wrote off an entry whose data was merely out of reach. Read the probe chunk
instead, and say in the log that the entry stays unreplicated.
Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK
A Canceled/DeadlineExceeded from the caller's per-request context was
treated like a dead channel: it closed the shared cached ClientConn and
cancelled every other in-flight RPC on it with "the client connection is
closing". Under a burst of concurrent chunk assigns (e.g. a large S3
multipart upload) one slow assign hitting its 10s attempt timeout could
poison the connection for all the rest, cascading into a flood of 500s.
Thread the caller's context into shouldInvalidateConnection and only
invalidate on Canceled/DeadlineExceeded while that context is still live,
which isolates the genuine stale-channel signal (a peer restart behind a
k8s Service VIP). To carry the context, add a ctx parameter to the
existing WithGrpcClient, WithMasterClient, and WithMasterServerClient; the
master assign and volume-lookup paths pass their per-attempt context and
every other caller passes context.Background().
* fix(sync): use per-cluster TLS for HTTP volume connections in filer.sync (#8965)
When filer.sync runs with -a.security and -b.security flags, only gRPC
connections received per-cluster TLS configuration. HTTP clients for
volume server reads and uploads used a global singleton with the default
security.toml, causing TLS verification failures when clusters use
different self-signed certificates.
Load per-cluster HTTPS client config from the security files and pass
dedicated HTTP clients to FilerSource (for downloads) and FilerSink
(for uploads) so each direction uses the correct cluster's certificates.
* fix(sync): address review feedback for per-cluster HTTP TLS
- Add insecure_skip_verify support to NewHttpClientWithTLS and read it
from per-cluster security config via https.client.insecure_skip_verify
- Error on partial mTLS config (cert without key or vice versa)
- Add nil-check for client parameter in DownloadFileWithClient
- Document SetUploader as init-only (same pattern as SetChunkConcurrency)
* filer.sync: support per-cluster mTLS with -a.security and -b.security flags
When syncing between two clusters that use different certificate authorities,
a single security.toml cannot authenticate to both. Add -a.security and
-b.security flags so each filer can use its own security.toml for TLS.
Closes#8481
* security: fatal on failure to read explicitly provided security config
When -a.security or -b.security is specified, falling back to insecure
credentials on read error would silently bypass mTLS. Fatal instead.
* fix(filer.sync): use source filer's fromTsMs flag in initOffsetFromTsMs
A→B was using bFromTsMs and B→A was using aFromTsMs — these were
swapped. Each path should seed the target's offset with the source
filer's starting timestamp.
* security: return error from LoadClientTLSFromFile, resolve relative PEM paths
Change LoadClientTLSFromFile to return (grpc.DialOption, error) so
callers can handle failures explicitly instead of a silent insecure
fallback. Resolve relative PEM paths (grpc.ca, grpc.client.cert,
grpc.client.key) against the config file's directory.
* fix(replication): resume partial chunk reads on EOF instead of re-downloading
When replicating chunks and the source connection drops mid-transfer,
accumulate the bytes already received and retry with a Range header
to fetch only the remaining bytes. This avoids re-downloading
potentially large chunks from scratch on each retry, reducing load
on busy source servers and speeding up recovery.
* test(replication): add tests for downloadWithRange including gzip partial reads
Tests cover:
- No offset (no Range header sent)
- With offset (Range header verified)
- Content-Disposition filename extraction
- Partial read + resume: server drops connection mid-transfer, client
resumes with Range from the offset of received bytes
- Gzip partial read + resume: first response is gzip-encoded (Go auto-
decompresses), connection drops, resume request gets decompressed data
(Go doesn't add Accept-Encoding when Range is set, so the server
decompresses), combined bytes match original
* fix(replication): address PR review comments
- Consolidate downloadWithRange into DownloadFile with optional offset
parameter (variadic), eliminating code duplication (DRY)
- Validate HTTP response status: require 206 + correct Content-Range
when offset > 0, reject when server ignores Range header
- Use if/else for fullData assignment for clarity
- Add test for rejected Range (server returns 200 instead of 206)
* refactor(replication): remove unused ReplicationSource interface
The interface was never referenced and its signature didn't match
the actual FilerSource.ReadPart method.
---------
Co-authored-by: Copilot <copilot@github.com>
* filer.sync: add exponential backoff on unexpected EOF during replication
When the source volume server drops connections under high traffic,
filer.sync retries aggressively (every 1-6s), hammering the already
overloaded source. This adds a longer exponential backoff (10s to 2min)
specifically for "unexpected EOF" errors, reducing pressure on the
source while still retrying indefinitely until success.
Also adds more logging throughout the replication path:
- Log source URL and error at V(0) when ReadPart or io.ReadAll fails
- Log content-length and byte counts at V(4) on success
- Log backoff duration in retry messages
Fixes#8542
* filer.sync: extract backoff helper and fix 2-minute cap
- Extract nextEofBackoff() and isEofError() helpers to deduplicate
the backoff logic between fetchAndWrite and uploadManifestChunk
- Fix the cap: previously 80s would double to 160s and pass the
< 2min check uncapped. Now doubles first, then clamps to 2min.
* filer.sync: log source URL instead of empty upload URL on read errors
UploadUrl is not populated until after the reader is consumed, so the
V(0) and V(4) logs were printing an empty string. Add SourceUrl field
to UploadOption and populate it from the HTTP response in fetchAndWrite.
* filer.sync: guard isEofError against nil error
* filer.sync: use errors.Is for EOF detection, fix log wording
- Replace broad substring matching ("read input", "unexpected EOF")
with errors.Is(err, io.ErrUnexpectedEOF) and errors.Is(err, io.EOF)
so only actual EOF errors trigger the longer backoff
- Fix awkward log phrasing: "interrupted replicate" → "interrupted
while replicating"
* filer.sync: remove EOF backoff from uploadManifestChunk
uploadManifestChunk reads from an in-memory bytes.Reader, so any EOF
errors there are from the destination side, not a broken source stream.
The long source-oriented backoff is inappropriate; let RetryUntil
handle destination retries at its normal cadence.
---------
Co-authored-by: Copilot <copilot@github.com>
* Added global http client
* Added Do func for global http client
* Changed the code to use the global http client
* Fix http client in volume uploader
* Fixed pkg name
* Fixed http util funcs
* Fixed http client for bench_filer_upload
* Fixed http client for stress_filer_upload
* Fixed http client for filer_server_handlers_proxy
* Fixed http client for command_fs_merge_volumes
* Fixed http client for command_fs_merge_volumes and command_volume_fsck
* Fixed http client for s3api_server
* Added init global client for main funcs
* Rename global_client to client
* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold
* Reduce the visibility of some functions in the util/http/client pkg
* Added the loadSecurityConfig function
* Use util.LoadSecurityConfiguration() in NewHttpClient func
* remove old raft servers if they don't answer to pings for too long
add ping durations as options
rename ping fields
fix some todos
get masters through masterclient
raft remove server from leader
use raft servers to ping them
CheckMastersAlive for hashicorp raft only
* prepare blocking ping
* pass waitForReady as param
* pass waitForReady through all functions
* waitForReady works
* refactor
* remove unneeded params
* rollback unneeded changes
* fix
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
Chris LuandGitHub
vadimartynovandGitHub
chrislu
askeipxandGitHub
Konstantin LebedevandGitHub
Chris Lu