* filer: keep lazy remote reads from resurrecting deleted paths
Under a remote mount with filer.remote.sync as write-back, a path that
was deleted or renamed away could come back as a chunkless remote-only
entry: between the local delete and the daemon's remote delete, a store
miss made maybeLazyFetchFromRemote trust a bucket that was behind the
filer. The ghost then outlived the remote object -- HEAD answered 200,
GET failed, and nothing cleaned it up.
The filer now tombstones paths it deletes under a remote mount, learned
both synchronously from its own delete path and from peer metadata
events. The lazy fetch and the lazy listing skip a tombstoned path until
the path is written again, until the mount's persisted write-back sync
offset has passed the delete event (the remote delete has landed), or
until a generous TTL covers a mount without a daemon.
Fixes#11440
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: cover recursive remote deletes with an ancestor tombstone
A recursive delete now records the directory tombstone before walking
children, so a partial traversal or a store that drops the subtree
without listing it still leaves every descendant covered. Directory
tombstones also subsume older descendant entries on add, descendant
adds covered by a standing ancestor are skipped, and an existing
tombstone can be refreshed even at capacity.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: scope remote tombstones to the deleted object's generation
A remote object whose own mtime postdates the local delete is a new
generation, not the one the tombstone hides, so a recreated directory
can surface remote writes made after its delete while old-generation
objects stay hidden. Lazy fetch now stats the remote object before
deciding, listings pass each child's remote mtime, and a sync offset
releases a tombstone once it reaches the delete's own timestamp.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: rebuild remote deletion tombstones after restart
In-memory tombstones are lost on restart while remote write-back
offsets persist, so a filer boot replays the persisted metadata log
from the oldest mount offset and folds deletes back into the tombstone
set through the same event handler. Lazy remote reads hold off while
the replay runs so a pending delete cannot resurrect in the gap.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: release remote tombstones only after their delete event lands
The write-back offset orders against event timestamps, but the synchronous
delete path recorded tombstones with the local clock before its event was
emitted — a later unrelated event could already have pushed the mount's
watermark past that guess, releasing the tombstone before the daemon
applied the delete. Tombstones recorded ahead of their event are now
marked pending and can only be lifted by the event confirming them or by
TTL; event-stamped tombstones release through the offset as before.
The remote-mtime generation bypass is dropped: remote and filer clocks
are independent, and a pending remote delete removes whatever object sits
at the path, so a "newer" remote object would only resurrect as a
phantom. Tombstoned lookups now skip the remote stat entirely.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: drop dir tombstone when recursive delete fails before listing
The ancestor tombstone is recorded before the child listing; if that
listing fails nothing was deleted, and the leftover tombstone would hide
still-existing remote children for the whole TTL. Tombstones for children
already deleted stay, since their remote deletes are still owed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: block lazy remote reads on startup tombstone rebuild
The rebuild gate is now a done-channel set synchronously before the
replay goroutine starts, so no lazy read can slip through in between.
Reads wait on it with context cancellation instead of returning an
empty miss that makes remote-only objects look deleted.
The replay start is floored at now-TTL: mounts without a recorded
write-back offset previously replayed the whole persisted history, and
events older than the TTL would only build already-expired tombstones.
The gate check now runs after the mount lookup so replaying the meta
log's own directory listings does not deadlock on the gate, and the
replay retries with backoff until it succeeds instead of failing open.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: mark restamped tombstone pending until its delete event lands
When a local delete raises an existing tombstone's timestamp, the new
value is only a local clock guess ahead of that delete's event. Leaving
the tombstone un-pending lets a write-back offset release it before the
event is actually consumed, reopening the resurrection window.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: bound tombstone replay to the tombstone TTL
Persisted-log replay retried forever, keeping lazy remote reads gated
indefinitely when the log cannot be read. Cap retries at the tombstone
TTL measured from replay start: past that point every tombstone would
have expired anyway, so opening the gate loses no protection.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: re-check deletion tombstone before persisting lazy fetch
A delete landing while StatFile is in flight passed the earlier
tombstone check but still persisted the fetched entry, resurrecting a
path whose remote delete is pending. Re-check right before CreateEntry.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: retract a lazily persisted entry when a delete raced the insert
The pre-insert tombstone check still leaves a window between the check
and the store insert. Since deletes always record the tombstone before
removing the entry, a tombstone visible right after a successful insert
means the delete already ran: delete the entry back out so the
tombstoned path stays deleted.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: note why the replay deadline can safely open the gate
Deletes made after startup are captured by the live delete and event
paths, so a stalled replay can only be missing pre-restart deletes, all
of which are past the tombstone TTL by the deadline.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: retract only the entry a lazy remote read materialized
Deleting by path after a raced delete could remove a legitimate rewrite
that replaced the fetched entry. Verify the stored entry still matches
the remote object (or the just-created directory shape) before deleting,
and apply the same post-insert check to lazy listing children.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: require full-entry equality before retracting a lazy entry
Remote-only matching still removed a write that had updated the fetched
entry, e.g. appended chunks. Compare the persisted entry against what
this read materialized; any change means a real update owns the path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: add guarded remote-storage client builder hook for lazy fetch
The lazy-remote fetch path (maybeLazyFetchFromRemote) resolved its
remote-storage client through the unguarded shared cache, bypassing the
SSRF chokepoint (BuildGuardedRemoteStorageClient) that the CVE-2026-73080
remediation wired into the volume, filer stream and s3 stream dial paths.
Add a RemoteStorageClientBuilder hook on Filer plus conf-only lookups on
FilerRemoteStorage, and route the lazy fetch through the builder when set
(endpoint deny-list + DNS-rebinding-safe dialer), falling back to the
shared cache otherwise. The filer server wires the builder in a follow-up.
* filer: route lazy directory listing through the guarded remote client
maybeLazyListFromRemote shared the unguarded client resolution of the
fetch path, so a caller-supplied remote endpoint was dialed without the
SSRF deny-list or rebinding-safe dialer. Resolve the conf and build the
client through buildRemoteStorageClient so the same guard covers listing.
* filer: route lazy remote delete through the guarded remote client
maybeDeleteFromRemote issued outbound DELETE/RemoveDirectory requests
through the unguarded client, giving a write-side SSRF to a caller-chosen
endpoint. Resolve the conf and build the client through
buildRemoteStorageClient so the endpoint deny-list and rebinding-safe
dialer apply to the delete path as well.
* filer server: wire the guarded remote client builder into the filer
Set Filer.BuildGuardedRemoteClient to BuildGuardedRemoteStorageClient and
forward AllowUntrustedRemoteEndpoints so the lazy-remote fetch, list and
delete paths apply the same SSRF endpoint checks as the volume and
streaming read paths.
* filer: test lazy fetch honors the guarded remote client builder
Add a regression test that sets BuildGuardedRemoteClient to a rejecting
builder and asserts maybeLazyFetchFromRemote returns no entry without
reaching the remote, covering the SSRF guard wired in the prior commits.
* filer: skip remote client for local-only lazy deletes
maybeDeleteFromRemote resolved and validated the mount's remote client
before checking entry.Remote, so a local-only file (no Remote entry) under
a mount whose endpoint the guard rejects failed to delete: the guard
error aborted the metadata deletion, leaving a file that needs no remote
operation undeletable. Move the local-only check ahead of client
construction so only remote-backed files and directories pay the guard.
* filer: build the guarded remote client inside the lazy singleflight
The lazy fetch and list paths built the guarded client before their
singleflight blocks, so concurrent requests for the same key each
allocated a fresh SDK client and HTTP transport even though only one
remote operation ran. Move client construction inside the singleflight
so the deduplicated operation builds it once, matching the per-request
guard semantics of the sibling streaming paths without the duplicate
transport churn.
* filer: test guarded rejection for the lazy list and delete paths
Add regression tests that set BuildGuardedRemoteClient to a rejecting
builder and assert the lazy list does not reach the remote, a
remote-backed file delete is blocked, and a local-only file under a
rejected mount still deletes (covering the local-only fix).
* filer: decouple lazy guarded-client build from the first caller's context
Building the guarded client inside the singleflight made concurrent
fetches share the first caller's context. If that caller canceled while
endpoint DNS validation was running, the builder returned an error and
published a not-found result to other callers whose contexts were still
valid. Build with context.WithoutCancel so the guard's DNS validation
is not tied to any single caller's cancellation, matching the list
path's existing decoupling for the remote operation itself.
* filer: reject remote-storage confs that dial blocked endpoints at load
The filer's lazy-fetch / lazy-list / remote-delete paths resolve remote
storage clients by name from FilerRemoteStorage.storageNameToConf and
dial them via remote_storage.GetRemoteStorage, which bypasses the SSRF
deny-list the volume server (BuildGuardedRemoteStorageClient) and the
filer's own direct-read path apply. A RemoteConf planted under
/etc/remote with a loopback / private / IMDS S3 endpoint is reloaded into
storageNameToConf on the next metadata-change event and then dialed on
the next cache miss — server-side request forgery from the filer.
Apply the volume server's SSRF deny-list at conf load time, the single
chokepoint that populates storageNameToConf:
- Add RemoteStorageConfValidator, injected into FilerRemoteStorage by
the filer server (the filer package cannot import the server package).
A conf that fails validation is dropped from storageNameToConf, so the
name-based client resolution on the lazy paths returns "not found"
instead of dialing the blocked endpoint.
- Add ValidateRemoteConfForLoad in weed_server, which mirrors
BuildGuardedRemoteStorageClient's gcs credential + endpoint checks
(validateRemoteEndpoint via guardedRemoteClient) without building a
client. allowUntrusted skips the check, mirroring the volume server
opt-out (-filer.allowUntrustedRemoteEndpoints).
- The filer server injects the validator at construction.
A conf whose type dials a fixed provider host (no caller-supplied
endpoint) passes; only caller-influenced endpoints are denied.
* filer: skip DNS resolution in the load-time SSRF validator
ValidateRemoteConfForLoad resolved hostnames during /etc/remote reload,
so a transient DNS failure (2s timeout) dropped the conf from the fresh
map that replaces the live map, disabling a working mount until the next
metadata event. The build-time guard (BuildGuardedRemoteStorageClient)
already re-resolves and re-validates the endpoint at dial time with the
rebinding-safe dialer, so DNS at load is redundant for security.
Split the static checks (scheme, IMDS hostnames, IP-literal blocked
addresses, gcs credentials) into validateRemoteEndpointForLoad, which
does no DNS. Hostname endpoints pass at load and are caught at dial if
they resolve to a blocked address. This preserves fail-fast for
statically-blocked confs (loopback IPs, IMDS hostnames) without letting
transient DNS failures disable mounts.
* filer: accept empty S3 endpoints in the guarded remote client builder
guardedRemoteClient returned ok=true with an empty endpoint for a
standard AWS S3 config (no custom S3Endpoint), so
BuildGuardedRemoteStorageClient and ValidateRemoteConfForLoad rejected
it with "remote endpoint is empty" — breaking standard AWS S3 mounts on
the lazy paths and the sibling streaming read paths that already use the
guarded builder.
An empty endpoint is not caller-supplied: the AWS SDK derives the
regional endpoint from the region, so there is nothing for the SSRF
guard to validate. Return ok=false for empty S3-compatible endpoints so
the builder falls through to the shared unguarded cache, matching the
historical behavior for standard AWS S3.
* filer: add filer.options.disable_remote_storage_deletion for cache-only deletes
Deleting a filer entry under a remote.mount path also deletes the backing
object from the remote store (maybeDeleteFromRemote). Deployments that use a
remote mount as a read-through cache in front of an authoritative,
externally-managed object store cannot allow this: the filer typically holds
read-only credentials, so the remote delete fails and the entire delete
errors out; and even where it would succeed, it destroys data the filer does
not own.
Add filer.options.disable_remote_storage_deletion (default false, so existing
behaviour is unchanged). When enabled, maybeDeleteFromRemote is skipped for
both single-entry and recursive folder deletes: local metadata and cached
chunks are still removed, but the remote object is left intact.
* filer: assert local removal in cache-only recursive delete test
The recursive cache-only delete test only checked that no remote delete
happened; it did not verify the local child and directory entries were
removed. Add FindEntry assertions so a regression that skips local
recursive deletion is caught.
* filer: reload the remote mount mapping when /etc/remote changes
The mapping was only read at startup, so remote.unmount left the mount live
in the filer: the purge that follows the mapping delete then went to the
remote store and wiped every object under the mount.
Rebuild the rules trie and the conf map from scratch on each load, since
ptrie cannot drop a key, and swap them under a lock.
* filer: drop the filer-wide remote deletion switch
With the mapping reloaded on unmount, the purge no longer reaches the remote
store, so there is nothing left for the switch to protect against.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* refactor: extract remote mount resolution into shared helpers
* refactor: share the adaptive remote cache wait policy
* filer: stream cold remote reads from the origin while caching
* s3: stream cold remote reads from the origin instead of 503 retries
* test: cover the S3 origin stream-through path
* remote mounts: match on path components and prefer the longest mount
* fail short origin streams instead of silently truncating
* s3: try the origin before failing a cold read on a local cache error
* s3: gate origin streaming on the entry's resolved version
* return the cache RPC's NotFound as a canonical status and classify it everywhere
* filer: keep multipart-range cold reads on the retry path
* Add remote storage index for lazy metadata pull
Introduces remoteStorageIndex, which maintains a map of filer directory
to remote storage client/location, refreshed periodically from the
filer's mount mappings. Provides lazyFetchFromRemote, ensureRemoteEntryInFiler,
and isRemoteBacked on S3ApiServer as integration points for handler-level
work in a follow-up PR. Nothing is wired into the server yet.
Made-with: Cursor
* Add unit tests for remote storage index and wire field into S3ApiServer
Adds tests covering isEmpty, findForPath (including longest-prefix
resolution), and isRemoteBacked. Also removes a stray PR review
annotation from the index file and adds the remoteStorageIdx field
to S3ApiServer so the package compiles ahead of the wiring PR.
Made-with: Cursor
* Address review comments on remote storage index
- Use filer_pb.CreateEntry helper so resp.Error is checked, not just the RPC error
- Extract keepPrev closure to remove duplicated error-handling in refresh loop
- Add comment explaining availability-over-consistency trade-off on filer save failure
Made-with: Cursor
* Move lazy metadata pull from S3 API to filer
- Add maybeLazyFetchFromRemote in filer: on FindEntry miss, stat remote
and CreateEntry when path is under a remote mount
- Use singleflight for dedup; context guard prevents CreateEntry recursion
- Availability-over-consistency: return in-memory entry if CreateEntry fails
- Add longest-prefix test for nested mounts in remote_storage_test.go
- Remove remoteStorageIndex, lazyFetchFromRemote, ensureRemoteEntryInFiler,
doLazyFetch from s3api; filer now owns metadata operations
- Add filer_lazy_remote_test.go with tests for hit, miss, not-found,
CreateEntry failure, longest-prefix, and FindEntry integration
Made-with: Cursor
* Address review: fix context guard test, add FindMountDirectory comment, remove dead code
Made-with: Cursor
* Nitpicks: restore prev maker in registerStubMaker, instance-scope lazyFetchGroup, nil-check remoteEntry
Made-with: Cursor
* Fix remotePath when mountDir is root: ensure relPath has leading slash
Made-with: Cursor
* filer: decouple lazy-fetch persistence from caller context
Use context.Background() inside the singleflight closure for CreateEntry
so persistence is not cancelled when the winning request's context is
cancelled. Fixes CreateEntry failing for all waiters when the first
caller times out.
Made-with: Cursor
* filer: remove redundant Mode bitwise OR with zero
Made-with: Cursor
* filer: use bounded context for lazy-fetch persistence
Replace context.Background() with context.WithTimeout(30s) and defer
cancel() to prevent indefinite blocking and release resources.
Made-with: Cursor
* filer: use checked type assertion for singleflight result
Made-with: Cursor
* filer: rename persist context vars to avoid shadowing function parameter
Made-with: Cursor
Peter DoddandGitHub
